HOWTO activate logging

Why do that

To debug or understand better the behavior of a Punchline using Spark/Pyspark runtime.

Prerequisites

You need a punch-standalone installed with spark. The easiest way to work with Spark is to launch the job in foreground mode by using the punchlinectl command. For example if you have a job defined in application.hjson file, use the following command

punchlinectl start --punchline application.hjson -v

What to do

Configure the Spark log4j.properties

Spark use log4j. It is located in

# You will have to update those files
$PUNCHPLATFORM_LOG4J_CONF_DIR/log4j2-sparkctl.properties
$PUNCHPLATFORM_LOG4J_CONF_DIR/log4j2-sparkctl.xml

Warning

By default the delivered log4j.properties is configured only with ERROR level so as to limit standard output to the most relevant spark outputs.

Important loggers

• org.apache.spark : the Spark loggers
• org.thales.punch : the various punchplatform loggers