Introduction

Apache Spark is an open-source distributed general-purpose cluster-computing framework. Originally developed at the University of California, Berkeley’s AMPLab, the Spark codebase was later donated to the Apache Software Foundation, which has maintained it since. Spark provides an interface for programming entire clusters with implicit data parallelism and fault tolerance.
(Reference: https://spark.apache.org/)

Installation

Package requirement :

  • Java (JDK 8)

Package will be installed :

  • Scala
  • Spark2

Verify Java version

Make Sure Java is Installed

If Java (JDK 8) not installed on your machine, follow this INSTRUCTION to install java.

Install Scala

Follow this instructions to download, extract and create symbolic link for scala.

[root@localhost scala]# cd /opt/scala
[root@localhost scala]# wget https://scala-lang.org/files/archive/scala-2.11.8.tgz
tar xvf scala-2.11.8.tgz
[root@localhost scala]# tar xvf scala-2.11.8.tgz
[root@localhost scala]# mv scala-2.11.8 scala
[root@localhost ~]# ln -s /opt/scala/scala /usr/lib/scala

Then add the following line to the /root/.bash_profile file to set Scala environment permanently. Only add the line that marked with the red square.

.bash_profile File

Verify installed scala version with scala -version command.

Scala Installation is Succeed

Install Spark2

Follow this instructions to download, extract and create symbolic link for Spark2.

[root@localhost scala]# cd /opt/spark
[root@localhost spark2]# wget http://d3kbcqa49mib13.cloudfront.net/spark-2.0.0-bin-hadoop2.7.tgz
[root@localhost spark2]# tar xf spark-2.0.0-bin-hadoop2.7.tgz
[root@localhost spark2]# mv spark-2.0.0-bin-hadoop2.7 spark
[root@localhost spark2]# ln -s /opt/spark2/spark /usr/local/spark

Then add the following line to the /root/.bash_profile file to set Spark2 environment permanently. Only add the line that marked with the red square.

.bash_profile File

Verify installed Spark2 version with spark-submit –version command.

Spark2 Installation is Succeed

Leave a Reply

Your email address will not be published. Required fields are marked *


The maximum upload file size: 64 MB.
You can upload: image, audio, video, document, spreadsheet, interactive, text, archive, code, other.