Skip to content

TROUBLESHOOTING memory/JVM usage

Why do that

Memory (RAM) is a finite resource which impact the PunchPlatform operations:

  • it limits what can do the Punchplatform (e.g. number of running topologies)
  • it limits the PunchPlatform performance (e.g. not enough RAM leading to cache issues)

How to investigate

We detail here what we can do to understand memory usage in a given node.

Java process

Heap size

The Java Virtual Machine (JVM) heap is the area of memory used for dynamic memory allocation.

The following Java command line parameters control Java heap allocation:

  • minimum allocation size: -Xms<heap size> (e.g. -Xms256m for a 256 mega)
  • maximum allocation size: -Xmx<heap size> (e.g. -Xmx1g for a 1024 mega)

These parameters are used at the application launch and they work with any Java application. Here is a usage example:

1
$ java -Xms128m -Xmx256m -cp /path/a:/path/b HelloWorld.java

To know what is your configuration current default heap size (when -Xms is not specified), use this command:

1
$ java -XX:+PrintFlagsFinal -version | grep HeapSize

It is a good practice to set the minimum heap allocation size (-Xms) at the maximum heap allocation size (-Xmx) to allocate all memory at the jvm start.

When multiple -Xms or -Xms are specified, only the last specified value is used. This is an undocumented feature of the JVM.

Tooling

To investigate memory usage of a Java program, multiple solution are available.

Before going furhter, keep in mind that Java creates a huge virtual memory that allows an efficient access to all jar files. This does not correspond to allocated memory, and is mostly shared between all JVMs sharing the same class path. What is more significant are:

  • VmHWM: Peak resident set size
  • VmRSS: Resident set size
Java Console

If you are working on a host with a graphical interface or if you are able to execute and forward X11, the jconsole command is the starting point.

Ensure Java is installed on your host, the jconsole should be available as it is bring by Java itself. To use it, start by finding the process ID of Java application you would like to monitor and then run:

1
$ jconsole <pid>

This will open up a GUI application with a lot of charts representing memory and CPU consumption, threads, etc.

Java Visualvm

An alternative to the Java Console is the Java VisualVM. It can be used in the same conditions with a slightly more friendly used interface.

This tool is not part of the Java binaries, please refer to the official VisualVM documentation for installation details.

When you are setup, run the application. A graphical interface should show up. You will find on the left menu all the Java application running on your host. Double click on the desired one to show how it goes.

Java heap dump

When working on a remote server, it is not handy to use the tools describe above. A better option is to get a JVM heap dump, which is a .hprof binary file, that can be analysed from a host with a graphical interface.

To do so, you can use the command line tool called jcmd. It is a very rich and powerful tool to help debugging any Java applicaton. One of this feature can help us to get our dump, run:

1
2
3
4
5
6
$ jcmd <pid> GC.heap_dump <output-file-path>

# for example, on a real process
$ jcmd 29890 GC.heap_dump /tmp/my-test-dump.hprof
29890:
Heap dump file created

Now, transfert this file to a computer with VisualVM installed and import it with the "File > Load" menu.

Info

When using a relative path, the current directory is the one use to initially launch the Java process. To avoid generating the file into an unwanted location, always prefer an absolute path usage.

Warning

Keep in mind that this procedure is quite heavy and can impact performance during its execution.

Another way to generate JVM dumps but with no performance issue, is to only dump it if an "Out Of Memory" error is thown. To enable this behaviour, ensure this parameter is passed to the JVM at launch time:

1
$ java -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=<output-path> ...
Java threads dump

If you are working on a server without any way of exporting .hprof files, a good starting option of to dump the current JVM threads stack.

To do so, get the JVM process ID and run :

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
$ jcmd <pid> Thread.print

# On real-life example, here is a sample output
$ jcmd 29890 Thread.print
29890:
2019-04-03 13:49:37
Full thread dump OpenJDK 64-Bit Server VM (25.191-b12 mixed mode):

"RMI Scheduler(0)" #31 daemon prio=9 os_prio=0 tid=0x00007f6d6400a800 nid=0x750d waiting on condition [0x00007f6de0690000]
   java.lang.Thread.State: TIMED_WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x00000000e19d7478> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

"RMI TCP Accept-0" #29 daemon prio=9 os_prio=0 tid=0x00007f6d6c017800 nid=0x7509 runnable [0x00007f6de0892000]
   java.lang.Thread.State: RUNNABLE
    at java.net.PlainSocketImpl.socketAccept(Native Method)
    at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
    at java.net.ServerSocket.implAccept(ServerSocket.java:545)
    at java.net.ServerSocket.accept(ServerSocket.java:513)
    at sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:52)
    at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:405)
    at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:377)
    at java.lang.Thread.run(Thread.java:748)

[...]

Linux

The simpliest method to check global RAM usage is via /proc/meminfo. This dynamically updated virtual file is actually the source of information displayed by many other memory related tools such as free, top and ps tools. From the amount of available/free physical memory to the amount of buffer waiting to be or being written back to disk, /proc/meminfo has everything you want to know about system memory usage.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
$ cat /proc/meminfo 
   MemTotal:        8042864 kB
   MemFree:          126524 kB
   MemAvailable:      91196 kB
   Buffers:            3804 kB
   Cached:           195584 kB
   SwapCached:       143020 kB
   Active:           403216 kB
   Inactive:         388652 kB
   Active(anon):     348516 kB
   Inactive(anon):   347760 kB
   Active(file):      54700 kB
   Inactive(file):    40892 kB
   Unevictable:        6524 kB
   Mlocked:            6524 kB
   SwapTotal:       8254460 kB
   SwapFree:        4796476 kB
   Dirty:               188 kB
   Writeback:             0 kB
   AnonPages:        591632 kB
   Mapped:          6868980 kB
   Shmem:            100248 kB
   Slab:             216812 kB
   SReclaimable:     114056 kB
   SUnreclaim:       102756 kB
   KernelStack:       12560 kB
   PageTables:        66404 kB
   NFS_Unstable:          0 kB
   Bounce:                0 kB
   WritebackTmp:          0 kB
   CommitLimit:    12275892 kB
   Committed_AS:   15649752 kB
   VmallocTotal:   34359738367 kB
   VmallocUsed:           0 kB
   VmallocChunk:          0 kB
   HardwareCorrupted:     0 kB
   AnonHugePages:      4096 kB
   CmaTotal:              0 kB
   CmaFree:               0 kB
   HugePages_Total:       0
   HugePages_Free:        0
   HugePages_Rsvd:        0
   HugePages_Surp:        0
   Hugepagesize:       2048 kB
   DirectMap4k:      359180 kB
   DirectMap2M:     7897088 kB
   DirectMap1G:           0 kB

Process-specific memory information is also available from /proc/<pid>/statm and /proc/<pid>/status. This is documented in the proc man page.

Other tools are available that present these info in a friendlier way:

  • free
  • vmstat
  • top
  • htop
  • memstat -p

When a process creates threads, (see /proc//status ; Treads field):

  • It can reach a system limit and cause an out-of-memory message, which is typically logged.
  • It allocates a stack for the thread.

To see what is your system threads number limit, you can use this command:

1
2
$ sudo sysctl -a | grep kernel.threads-max
kernel.threads-max = 125205