spark 2.0 installation and configuration

Spark 2 Installation  steps on Ubuntu :

Step 1 : Download apache spark

Download Spark-2.0.0-bin-hadoop2.7 prebuilt version and extract in the required location.

http://spark.apache.org/downloads.html

click the link to download http://d3kbcqa49mib13.cloudfront.net/spark-2.0.0-bin-hadoop2.7.tgz

Step 2: Open .bashrc and set the spark_home path

To open bashrc type the command in the terminal

sudo gedit .bashrc

export SPARK_HOME=/SPARKINSTALLATION PATH/

export PATH=$PATH:$SPARK_HOME/bin:$SPARK_HOME/sbin

note: If you installing spark with hadoop ,both sbin directory have start-all.sh . Use ./start-all.sh to start spark daemons

Step 3:

In the extracted spark directory , open conf directory  and do following changes

Rename spark-env.sh.template to spark-env.sh

Add this :  export SPARK_CLASSPATH=/SPARK INSTALALTION DIR/jars/*

Rename spark-default.conf.template to spark-default.conf

From the terminal , start spark services using ./start-all.sh

or

start-master.sh

start-slave.sh

 Step 4: go to web browser and check at localhost:8080

Step 5 :

 try spark-shell or spark-submit to play with spark

Leave a Reply

Your email address will not be published. Required fields are marked *