Troubleshooting Spark Cluster

Application data cleanup

Spark keeps its application data in $SPARK_WORKER_DIR.

If you run many Spark applications, those application data can grow heavy. You can configure cleanup of this data in $PUNCHPLATFORM_SPARK_INSTALL_DIR/conf/spark-env.sh.

With the following configuration, app data is kept for 1h.

-Dspark.worker.cleanup.interval=3600 -Dspark.worker.cleanup.appDataTtl=3600