TROUBLESHOOTING Cope with Out Of Memory java crashes

Warning

This procedure is discouraged, refer to the warning below.

Why do that

Stormlines are java applications that require the right amount of memory to properly run. Some act as proxy-forwarder applications and can run in a limited amount of memory (100Mb). Others run many input processing and output nodes, or imply archiving logic and require more memory (> 1Gb).

In all cases the punch provides an internal mechanism to make stormlines fail fast should they consume too much of their heap memory. Fail fast is key to prevent a stormline to significantly slow down the traffic processing. Stormlines will be quickly restarted. This said, you can detect the restart of stormlines, and you should of course, grant them the right amount of memory.

By default stormlines should stop if their heap memory reach a threshold of 90%. This simply exit with an error log:

message="FATAL ERROR OutOfMemory treshold exceeded" treshold=0,85 used=0.8635

The uptime metric can be used to detect these restarting topologies on your platform.

What to do

Grant enough memory to your topology

Refer to the performance memory<performance_memory> chapter to grant each stormline the memory it needs.

Change the OutOfMemory threshold

You can change the out of memory exit threshold by settings the [punchplatform.oom_exit_threshold] property. For example to use a 95% threshold, add the following to the topology storm_settings section:

"storm_settings" : {
    "topology.worker.childopts" : "-server -Xms100m -Xmx100m -Dpunchplatform.oom_exit_treshold=0.95"
    ...
}

Warning

The default value of this treshold is 0.0. I.e. it is disabled by default. This mechanism is fragile as it leverages a call to System.gc() internally and takes an exit decision if no or not enough memory has been released. In some cases, the gc call returns without freeing any memory because a concurrent gc is running. As a result you can suffer from intempestive exit.