Apache YARN

If the Spark master value is set as YARN-cluster, then the application can be submitted to the cluster and then terminated. The cluster will take care of allocating resources and running tasks. However, if the application master is submitted as YARN-client, then the application stays alive during the life cycle of processing and requests resources from YARN. These are applicable at a larger scale, when integrating with Hadoop YARN. A step-by-step guideline will be provided later in this chapter to configure a single-node YARN cluster for launching your Spark jobs needing minimal resources.

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset