Troubleshooting using Storm worker logs¶

Why Do That¶

It can be difficult to understand what is going on when setting up a new Storm punchline. In most cases, the first thing to do is to watch the logs. We will see how it is possible to increase Storm workers log verbosity to properly troubleshoot.

What to do¶

Locating the worker log file¶

On a production cluster:

Open the Storm Web UI (refer to the punchplatform.properties file for locating the hosts and port), then click on the link to the chosen punchline
Identify the storm supervisor host(s) in charge of running the punchline, and the associated slot number, in the "Worker Resources" table in the punchline page of the Storm UI
on this supervisor host, log as sudoer and go to /var/log/punchplatform/storm/workers.artifacts//slot
the logs are in the "worker.log" file. Each time the worker (re)starts, a "Launching worker for " string appears.

On a standalone punch, the 'workers.artifacts' folder is located in <standaloneSetupDir>/external/punch-binaries-6.3.1/apache-storm-2.3.0 folder.

Tuning the log levels¶

By default, when a topology is submitted to a Storm cluster, the logging configuration file loaded is named worker.xml. It can be found on at

Standalone

punch-standalone-X.Y.Z/external/apache-storm-1.2.2/log4j2/worker.xml

Cluster mode (where [setups_root] is defined in punchplatform-deployment.settings)

[setups_root]/apache-storm-1.2.2/log4j2/worker.xml   # often, setups_root is /data/opt

This file is shared across all running Storm topologies, and taken into account dynamically, even by already running topologies, so we definitely do NOT want to modify it we have any other choice. The best approach to debug a topology is to use another logging file that we can modify to be as verbose as desired, by manually running the topology in foreground mode in the operator environment.

To do so, we provide a file called logback-topology.xml. It can be found at the following locations:

Standalone

punch-standalone-X.Y.Z/external/punch-operator-5.3.0/bin/log4j2-topology.xml

Cluster mode

[setups_root]/punchplatform-admin-node-4.0.0-SNAPSHOT/bin/logback-topology.xml

The best way to see what is happening is to run your topology in foreground mode. The logback-topology.xml file will be used by default instead of worker.xml. On either standalone or cluster platform, let us run an example with apache:

cd $PUNCHPLATFORM_CONF_DIR
punchlinectl tenants/mytenant/channels/apache_httpd/input.json

start-foreground topology 'single_topology.json'...
java -server -Xms128m -Xmx128m  -Dlogfile.name=mytenant_apache_httpd_single_ -Dpunchplatform.log.dir=/Users/gmfmi/punchplatform
/standalone/punch-standalone-4.0.1-SNAPSHOT/logs -Dlogging.sensitivity=S3 -Dlog4j.configurationFile=/Users/gmfmi/
punchplatform/standalone/punch-standalone-4.0.1-SNAPSHOT/bin/logback-topology.xml -Dpunchplatform.configurationFile
=/Users/gmfmi/punchplatform/standalone/punch-standalone-4.0.1-SNAPSHOT/conf/punchplatform.properties -cp [jars]
com.thales.services.cloudomc.punchplatform.processor.cli.TopologyCli start  -m local -t single_topology.json

Now, you should see your punchline logs in your interactive terminal. To stop the punchline, simply press Ctrl + C.

Increase verbosity¶

If you need to some logs from lower logging level, you can change logging value set in the logback-topology.xml file. For example, if you want to see TRACE logs from you Storm Spouts, update your settings this way:

<loggers>
    ...
    <logger name="org.thales.punch.storm.spout" level="TRACE"/>
    ...
</loggers>

To get a detailed description of the available loggers, see the logging configuration guide<LoggingConfiguration>{.interpreted-text role="ref"}.