Client

This documentation presents the job and plan client launcher and its command “punchplatform-analytics.sh”.

Job Launch

../../../_images/analytics_client_cluster.png

For a detailed spark cluster description, please read the official spark documentation: Cluster Mode Overview and Spark Standalone Mode. To summarise a spark standalone cluster is composed of two types of persistent JVM:

  • The master which receive requests from user and supervise workers.
  • The workers which launch spark application process.

An PML job is a spark application which is composed of two types of JVM:

  • The driver where the user main java function runs.
  • The executors on which the driver sends tasks.

Cluster Mode

../../../_images/analytics_client_cluster_mode.png
$ punchplatform-analytics.sh --job <job_path> --spark-master <master_url> --deploy-mode cluster

In this mode, all the job is executed inside the spark cluster. The client just submit the job to the master and died.

Client Mode

../../../_images/analytics_client_client_mode.png
$ punchplatform-analytics.sh --job <job_path> --spark-master <master_url> --deploy-mode client

In this mode, the client launch a driver inside the client machine. Next the driver requests for executors to the master. The client stays alive during the job for monitoring and logs redirecting to the console.

Local Mode

../../../_images/analytics_client_local_mode.png
$ punchplatform-analytics.sh --job <job_path> --spark-master local[*] --deploy-mode client

In this mode, the client launch a driver and executors together in an unique JVM inside the client machine. The client stays alive during the job for monitoring and logs redirecting to the console.

Plan Launch

During plan launch, the client generates succesive json jobs and launches them in one of the above mode. The client stays alive until the last job.

$ punchplatform-analytics.sh --plan <plan_path> <template_path> --spark-master <master_url> --deploy-mode cluster
$ punchplatform-analytics.sh --plan <plan_path> <template_path> --spark-master <master_url> --deploy-mode client
$ punchplatform-analytics.sh --plan <plan_path> <template_path> --spark-master local[*] --deploy-mode client