Troubleshooting Storm worker logs¶
Context¶
Sometimes it can be difficult to really understand what is going on when setting up a new Storm topology/channel. In most cases, the first thing to do is watching the logs. We will see how it possible to increase Storm workers log verbosity to properly troubleshoot.
What to do¶
By default, when a topology is submitted to a Storm cluster, the logging
configuration file loaded is named worker.xml
. It can be found on at
Standalone
1 | punchplatform-standalone-X.Y.Z/external/apache-storm-1.2.2/log4j2/worker.xml |
Cluster mode (where [setups_root] is define in punchplatform-deployment.settings)
1 | [setups_root]/apache-storm-1.2.2/log4j2/worker.xml |
This file is shared across all running Storm topologies so we definitely do NOT want to modify it. The best approach to debug a topology is to use another logging file that we can modify to be as verbose as desired.
To do so, we provide a file called logback-topology.xml
. It can be
found at the following locations:
Standalone
1 | punchplatform-standalone-X.Y.Z/external/punchplatform-operator-environment-dataanalytics-5.0.0/bin/log4j2-topology.xml |
Cluster mode
1 | [setups_root]/punchplatform-admin-node-4.0.0-SNAPSHOT/bin/logback-topology.xml |
The best way to see what is happening is to run your topology in
foreground mode. The logback-topology.xml
file will be used by defaut
instead of worker.xml
. On either standalone or cluster platform,
let\'s run an example with apache:
1 2 3 4 5 6 7 8 9 10 | $ cd $PUNCHPLATFORM_CONF_DIR $ punchplatform-channel.sh --configure tenants/mytenant/etc/channel_config/apache_httpd_channel.json $ punchplatform-topology.sh --start-foreground --topology tenants/mytenant/channels/apache_httpd/single_topology.json -m local start-foreground topology 'single_topology.json'... java -server -Xms128m -Xmx128m -Dlogfile.name=mytenant_apache_httpd_single_ -Dpunchplatform.log.dir=/Users/gmfmi/punchplatform /standalone/punchplatform-standalone-4.0.1-SNAPSHOT/logs -Dlogging.sensitivity=S3 -Dlog4j.configurationFile=/Users/gmfmi/ punchplatform/standalone/punchplatform-standalone-4.0.1-SNAPSHOT/bin/logback-topology.xml -Dpunchplatform.configurationFile =/Users/gmfmi/punchplatform/standalone/punchplatform-standalone-4.0.1-SNAPSHOT/conf/punchplatform.properties -cp [jars] com.thales.services.cloudomc.punchplatform.processor.cli.TopologyCli start -m local -t single_topology.json |
Now, you should see your topology\'s logs in this interactive terminal.
To stop the topology, press Ctrl + C
.
Increase verbosity¶
If you need to some logs from lower logging level, you can change
logging value set in the logback-topology.xml
file. For example, if
you want to see TRACE logs from you Storm Spouts, update your settings
this way:
1 2 3 4 5 | <loggers> ... <logger name="com.thales.services.cloudomc.punchplatform.storm.spout" level="TRACE"/> ... </loggers> |
To get a detailed description of available loggers, see the
logging configuration guide<LoggingConfiguration>
{.interpreted-text
role="ref"}.