HOW TO INSTALL APACHE SPARK HOW TO

In this section, you configure the Spark magic that you installed earlier to connect to an Apache Spark cluster. Enter the command below to enable the server extension: jupyter serverextension enable -py sparkmagicĬonfigure Spark magic to connect to HDInsight Spark cluster Jupyter-kernelspec install sparkmagic/kernels/pyspark3kernel Jupyter-kernelspec install sparkmagic/kernels/pysparkkernel Jupyter-kernelspec install sparkmagic/kernels/sparkrkernel Jupyter-kernelspec install sparkmagic/kernels/sparkkernel Then change your working directory to the location identified with the above command.įrom your new working directory, enter one or more of the commands below to install the wanted kernel(s): Kernel Identify where sparkmagic is installed by entering the following command: pip show sparkmagic See also, sparkmagic documentation.Įnsure ipywidgets is properly installed by running the following command: jupyter nbextension enable -py -sys-prefix widgetsnbextension Install Spark magicĮnter the command pip install sparkmagic=0.13.1 to install Spark magic for HDInsight clusters version 3.6 and 4.0. See also, Installing Jupyter using Anaconda. While running the setup wizard, make sure you select the option to add Anaconda to your PATH variable. The Anaconda distribution will install both, Python, and Jupyter Notebook.ĭownload the Anaconda installer for your platform and run the setup. Install Python before you install Jupyter Notebooks. Install Jupyter Notebook on your computer The local notebook connects to the HDInsight cluster.įamiliarity with using Jupyter Notebooks with Spark on HDInsight. For instructions, see Create Apache Spark clusters in Azure HDInsight.

PrerequisitesĪn Apache Spark cluster on HDInsight.

Configure Spark magic to access Spark cluster on HDInsight.įor more information about custom kernels and Spark magic, see Kernels available for Jupyter Notebooks with Apache Spark Linux clusters on HDInsight.

Install the PySpark and Spark kernels with the Spark magic.

There are four key steps involved in installing Jupyter and connecting to Apache Spark on HDInsight. You then connect the notebook to an HDInsight cluster.

HOW TO INSTALL APACHE SPARK HOW TO

In this article, you learn how to install Jupyter Notebook with the custom PySpark (for Python) and Apache Spark (for Scala) kernels with Spark magic.

How to install apache spark