Memory Leak in Docker Container

Since we recently start using docker for product release, I’m dealing with a lot of problems related with docker. One of the headache is sometimes, the running application performs different in docker containers with the one directly runs in local without docker. For me, a more specific example is my webApp, running in gradle with Jetty9 plugins, can finish our load test perfectly only on my ubuntu machine. But it will get memory leak error when runs in docker container. So I decided to look into it by Java profiler.

It’s quite easy to use a GUI profiler to monitor the whole process how an application uses CPU and memory during running time. But it’s a little complicated to attach it in a container. Firstly I tried to use YourKit by its remote profiling option. The basic steps are:

1. Modify Dockerfile to get YourKit installed.

   RUN apt-get update
   RUN apt-get install -y wget
   RUN wget https://www.yourkit.com/download/yjp-2016.02-b29-linux.tar.bz2
   RUN tar xfj yjp-2016.02-b29-linux.tar.bz2
   
   "-agentpath:/usr/share/jetty/yjp-2016.02/bin/linux-x86-64/libyjpagent.so=disablestacktelemetry,disableexceptiontelemetry,delay=10000,sessionname=Jetty", \
   
   EXPOSE 10001

2. After building the image, access into the docker container and attach to JVM:

   docker exec -i -t <containerID> /bin/bash
   <directory with unpacked content>/bin/yjp.sh -attach 

3. In YourKit GUI, link to the host:port.
   But every time when I try to attach to JVM (step 2), I get error below:

   com.sun.tools.attach.AttachNotSupportedException:
      Unable to open socket file:
         target process not responding or HotSpot VM not loaded.

   It should be solved by just switching to the user created the container. But it didn’t work. So I tried to use JProfiler.
   Here are the steps:
4. Modify the Dockerfile to get JProfiler installed:

   RUN \
     apt-get update && \
     apt-get install -y wget && \
     wget http://download-keycdn.ej-technologies.com/jprofiler/jprofiler_linux_9_1_1.tar.gz && \
     tar -xzf jprofiler_linux_9_1_1.tar.gz -C /usr/local
   
   ENV JPAGENT_PATH="-agentpath:/usr/local/jprofiler9/bin/linux-x64/libjprofilerti.so=nowait"
   EXPOSE 8849

   Be careful of the version. The one installed in docker container should be the same version as the one you use to monitor.
5. Execute the command in docker container:

   docker exec -i -t <containerID> /bin/bash
   /usr/local/jprofiler9/bin/jpenable

   That’s it. Now we can link to the remote from anywhere by host:port. The default port is 8849.

But the sad story is after looking into the result of the profiler, I still don’t know how it got memory leaked. Here is a discussion about the similar docker problem, maybe I’m not the only one.
I’ll update once I figured out…