TROUBLESHOOTING memory/JVM usage¶
Why do that¶
Memory (RAM) is a finite resource which impact the PunchPlatform operations:
- it limits what can do the Punchplatform (e.g. number of running topologies)
- it limits the PunchPlatform performance (e.g. not enough RAM leading to cache issues)
How to investigate¶
We detail here what we can do to understand memory usage in a given node.
The Java Virtual Machine (JVM) heap is the area of memory used for dynamic memory allocation.
The following Java command line parameters control Java heap allocation:
- minimum allocation size:
-Xms256mfor a 256 mega)
- maximum allocation size:
-Xmx1gfor a 1024 mega)
These parameters are used at the application launch and they work with any Java application. Here is a usage example:
$ java -Xms128m -Xmx256m -cp /path/a:/path/b HelloWorld.java
To know what is your configuration current default heap size (when -Xms is not specified), use this command:
$ java -XX:+PrintFlagsFinal -version | grep HeapSize
It is a good practice to set the minimum heap allocation size (-Xms) at the maximum heap allocation size (-Xmx) to allocate all memory at the jvm start.
When multiple -Xms or -Xms are specified, only the last specified value is used. This is an undocumented feature of the JVM.
To investigate memory usage of a Java program, multiple solution are available.
Before going furhter, keep in mind that Java creates a huge virtual memory that allows an efficient access to all jar files. This does not correspond to allocated memory, and is mostly shared between all JVMs sharing the same class path. What is more significant are:
- VmHWM: Peak resident set size
- VmRSS: Resident set size
If you are working on a host with a graphical interface or if you are able to
execute and forward X11, the
jconsole command is the starting point.
Ensure Java is installed on your host, the
jconsole should be available as
it is bring by Java itself. To use it, start by finding the process ID of
Java application you would like to monitor and then run:
$ jconsole <pid>
This will open up a GUI application with a lot of charts representing memory and CPU consumption, threads, etc.
An alternative to the Java Console is the Java VisualVM. It can be used in the same conditions with a slightly more friendly used interface.
This tool is not part of the Java binaries, please refer to the official VisualVM documentation for installation details.
When you are setup, run the application. A graphical interface should show up. You will find on the left menu all the Java application running on your host. Double click on the desired one to show how it goes.
Java heap dump¶
When working on a remote server, it is not handy to use the tools describe above.
A better option is to get a JVM heap dump, which is a
.hprof binary file, that
can be analysed from a host with a graphical interface.
To do so, you can use the command line tool called
jcmd. It is a very rich and
powerful tool to help debugging any Java applicaton. One of this feature can help
us to get our dump, run:
1 2 3 4 5 6
$ jcmd <pid> GC.heap_dump <output-file-path> # for example, on a real process $ jcmd 29890 GC.heap_dump /tmp/my-test-dump.hprof 29890: Heap dump file created
Now, transfert this file to a computer with VisualVM installed and import it with the "File > Load" menu.
When using a relative path, the current directory is the one use to initially launch the Java process. To avoid generating the file into an unwanted location, always prefer an absolute path usage.
Keep in mind that this procedure is quite heavy and can impact performance during its execution.
Another way to generate JVM dumps but with no performance issue, is to only dump it if an "Out Of Memory" error is thown. To enable this behaviour, ensure this parameter is passed to the JVM at launch time:
$ java -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=<output-path> ...
Java threads dump¶
If you are working on a server without any way of exporting
.hprof files, a good
starting option of to dump the current JVM threads stack.
To do so, get the JVM process ID and run :
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33
$ jcmd <pid> Thread.print # On real-life example, here is a sample output $ jcmd 29890 Thread.print 29890: 2019-04-03 13:49:37 Full thread dump OpenJDK 64-Bit Server VM (25.191-b12 mixed mode): "RMI Scheduler(0)" #31 daemon prio=9 os_prio=0 tid=0x00007f6d6400a800 nid=0x750d waiting on condition [0x00007f6de0690000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000e19d7478> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) "RMI TCP Accept-0" #29 daemon prio=9 os_prio=0 tid=0x00007f6d6c017800 nid=0x7509 runnable [0x00007f6de0892000] java.lang.Thread.State: RUNNABLE at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409) at java.net.ServerSocket.implAccept(ServerSocket.java:545) at java.net.ServerSocket.accept(ServerSocket.java:513) at sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:52) at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:405) at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:377) at java.lang.Thread.run(Thread.java:748) [...]
The simpliest method to check global RAM usage is via
This dynamically updated virtual file is actually the source of
information displayed by many other memory related tools such as free,
top and ps tools. From the amount of available/free physical memory to
the amount of buffer waiting to be or being written back to disk,
/proc/meminfo has everything you want to know about system memory usage.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47
$ cat /proc/meminfo MemTotal: 8042864 kB MemFree: 126524 kB MemAvailable: 91196 kB Buffers: 3804 kB Cached: 195584 kB SwapCached: 143020 kB Active: 403216 kB Inactive: 388652 kB Active(anon): 348516 kB Inactive(anon): 347760 kB Active(file): 54700 kB Inactive(file): 40892 kB Unevictable: 6524 kB Mlocked: 6524 kB SwapTotal: 8254460 kB SwapFree: 4796476 kB Dirty: 188 kB Writeback: 0 kB AnonPages: 591632 kB Mapped: 6868980 kB Shmem: 100248 kB Slab: 216812 kB SReclaimable: 114056 kB SUnreclaim: 102756 kB KernelStack: 12560 kB PageTables: 66404 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 12275892 kB Committed_AS: 15649752 kB VmallocTotal: 34359738367 kB VmallocUsed: 0 kB VmallocChunk: 0 kB HardwareCorrupted: 0 kB AnonHugePages: 4096 kB CmaTotal: 0 kB CmaFree: 0 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB DirectMap4k: 359180 kB DirectMap2M: 7897088 kB DirectMap1G: 0 kB
Process-specific memory information is also available from
/proc/<pid>/status. This is documented in the proc man page.
Other tools are available that present these info in a friendlier way:
- memstat -p
When a process creates threads, (see /proc/
- It can reach a system limit and cause an out-of-memory message, which is typically logged.
- It allocates a stack for the thread.
To see what is your system threads number limit, you can use this command:
$ sudo sysctl -a | grep kernel.threads-max kernel.threads-max = 125205