Docker Debugging Skills

Reuse container name

But when we write / debug Dockerfile, we often repeat previous command s, such as docker run --name jstorm-zookeeper zookeeper:3.4, and then the container name conflicts.

$ docker run --name jstorm-zookeeper zookeeper:3.4
...
$ docker run --name jstorm-zookeeper zookeeper:3.4
docker: Error response from daemon: Conflict. The name "/jstorm-zookeeper" is already in use by container xxxxxxxxx

You can add - rm flag when you run docker run, and the container will be destroyed after exit. No manual docker rm CONTAINER required

$ docker run --name jstorm-zookeeper zookeeper:3.4 --rm

# reuse 
$ docker create --name jstorm-zookeeper zookeeper:3.4
$ docker start jstorm-zookeeper

# no error

<!-- more -->

debug Dockerfile

Docker files are not usually written in one go. Sometimes the container starts and crash exits directly, sometimes the build image fails, or we often need debug Dockerfile to verify that the Dockerfile meets our expectations.

If the build fails, you can view the stdout error information directly, split the instructions, and rebuild.

logs View stdout

All contents written to stdout in the container are captured in a history file in the host and can be viewed through docker logs CONTAINER.

$ docker run -d --name jstorm-zookeeper zookeeper:3.4

$ docker logs jstorm-zookeeper
ZooKeeper JMX enabled by default
Using config: /conf/zoo.cfg
2016-12-18 05:55:27,717 [myid:] - INFO  [main:QuorumPeerConfig@124] - Reading configuration from: /conf/zoo.cfg
2016-12-18 05:55:27,725 [myid:] - INFO  [main:DatadirCleanupManager@78] - autopurge.snapRetainCount set to 3
2016-12-18 05:55:27,725 [myid:] - INFO  [main:DatadirCleanupManager@79] - autopurge.purgeInterval set to 0
2016-12-18 05:55:27,726 [myid:] - INFO  [main:DatadirCleanupManager@101] - Purge task is not scheduled.
2016-12-18 05:55:27,728 [myid:] - WARN  [main:QuorumPeerMain@113] - Either no config or no quorum defined in config, running  in standalone mode
2016-12-18 05:55:27,746 [myid:] - INFO  [main:QuorumPeerConfig@124] - Reading configuration from: /conf/zoo.cfg
2016-12-18 05:55:27,747 [myid:] - INFO  [main:ZooKeeperServerMain@96] - Starting server
2016-12-18 05:55:27,766 [myid:] - INFO  [main:Environment@100] - Server environment:zookeeper.version=3.4.9-1757313, built on 08/23/2016 06:50 GMT
2016-12-18 05:55:27,766 [myid:] - INFO  [main:Environment@100] - Server environment:host.name=dbc742dd5688
2016-12-18 05:55:27,767 [myid:] - INFO  [main:Environment@100] - Server environment:java.version=1.8.0_111-internal

Even if the container has exited, it can be seen, so the unexpected exit can be analyzed in this way. These files are kept until the container is deleted through docker rm. The specific path of the file can be obtained by docker inspect CONTAINER.
(Then you can't find these files on osx, because the docker of OSX actually runs in "VM" and doesn't expand, but it can be "VM" TTY on screen ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/tty touch)

When using docker logs, add some parameters to filter logs, and output all logs by default.

Options:
      --details        Show extra details provided to logs
  -f, --follow         Follow log output
      --help           Print usage
      --since string   Show logs since timestamp
      --tail string    Number of lines to show from the end of the logs (default "all")
  -t, --timestamps     Show timestamps

attach view stdout in real time

If you want to view the output of the container in real time, you can use the docker attach CONTAINER command.

By default, stdin, proxy signals are bound, so if you have a ctrl-c container, it usually exits. Many times people do not want to do this, just want to separate, you can ctrl-p ctrl-q.

Execute arbitrary command s

Any command can be executed in the container by docker exec CONTAINER COMMAND, such as cat to debug something.

$ docker run -d --name jstorm-zookeeper zookeeper:3.4

$ docker exec jstorm-zookeeper java -version
openjdk version "1.8.0_111-internal"
OpenJDK Runtime Environment (build 1.8.0_111-internal-alpine-r0-b14)
OpenJDK 64-Bit Server VM (build 25.111-b14, mixed mode)

It is also more convenient to debug the container directly by starting a shell in the container through exec, without having to execute docker exec one by one.

$ docker exec -it jstorm-zookeeper /bin/bash
bash-4.3# pwd
/zookeeper-3.4.9

docker exec can only be used on running containers. If you have stopped exiting, you can't use docker logs.

Rewrite entrypoint and cmd

Each Docker image has entrypoint and cmd, which can be defined in the Docker file or specified at run time. These two concepts are easily confused, and they are tried in different ways.

Entrypoint is more "advanced" than cmd. entrypoint runs as a pid-1 process in a container (docker is not a virtual machine, but an isolated process). The PID of real linux is init.
cmd is only an entrypoint parameter:

<ENTRYPOINT> "<CMD>"

When we do not specify entrypoint, the default is / bin/sh -c. So entrypoint really expresses what the docker should do. Usually you have a shell script to proxy it.
Both entrypoint and cmd can be changed at runtime to see if this setting of entrypoint is elegant and reasonable.

$ docker run -it --name jstorm-zookeeper --entrypoint /bin/bash zookeeper:3.4
bash-4.3# top
Mem: 320212K used, 1725368K free, 89112K shrd, 35532K buff, 130532K cached
CPU:   0% usr   0% sys   0% nic 100% idle   0% io   0% irq   0% sirq
Load average: 0.20 0.06 0.02 5/195 7
  PID  PPID USER     STAT   VSZ %VSZ CPU %CPU COMMAND
    1     0 root     S     6220   0%   0   0% /bin/bash
    7     1 root     R     1516   0%   2   0% top

The content in any docker run command after the image name is passed to entrypoint as a parameter as the content of cmd.

Pause container

With docker pause, all processes in the container can be suspended. This is very useful.

$ docker run -d --name jstorm-zookeeper zookeeper:3.4 && sleep 0.1 && docker pause jstorm-zookeeper && docker logs jstorm-zookeeper
a24405a53ddd9b7d94d9e77fe2b5a67639a251d681aa2f34fcb0cc96f347ba48
jstorm-zookeeper
ZooKeeper JMX enabled by default
Using config: /conf/zoo.cfg
2016-12-18 16:17:47,720 [myid:] - INFO  [main:QuorumPeerConfig@124] - Reading configuration from: /conf/zoo.cfg
2016-12-18 16:17:47,730 [myid:] - INFO  [main:DatadirCleanupManager@78] - autopurge.snapRetainCount set to 3
2016-12-18 16:17:47,730 [myid:] - INFO  [main:DatadirCleanupManager@79] - autopurge.purgeInterval set to 0
2016-12-18 16:17:47,730 [myid:] - INFO  [main:DatadirCleanupManager@101] - Purge task is not scheduled.
2016-12-18 16:17:47,731 [myid:] - WARN  [main:QuorumPeerMain@113] - Either no config or no quorum defined in config, running  in standalone mode
2016-12-18 16:17:47,757 [myid:] - INFO  [main:QuorumPeerConfig@124] - Reading configuration from: /conf/zoo.cfg
2016-12-18 16:17:47,757 [myid:] - INFO  [main:ZooKeeperServerMain@96] - Starting server

$ docker unpause jstorm-zookeeper && docker logs jstorm-zookeeper
jstorm-zookeeper
ZooKeeper JMX enabled by default
Using config: /conf/zoo.cfg
2016-12-18 16:17:47,720 [myid:] - INFO  [main:QuorumPeerConfig@124] - Reading configuration from: /conf/zoo.cfg
2016-12-18 16:17:47,730 [myid:] - INFO  [main:DatadirCleanupManager@78] - autopurge.snapRetainCount set to 3
2016-12-18 16:17:47,730 [myid:] - INFO  [main:DatadirCleanupManager@79] - autopurge.purgeInterval set to 0
2016-12-18 16:17:47,730 [myid:] - INFO  [main:DatadirCleanupManager@101] - Purge task is not scheduled.
2016-12-18 16:17:47,731 [myid:] - WARN  [main:QuorumPeerMain@113] - Either no config or no quorum defined in config, running  in standalone mode
2016-12-18 16:17:47,757 [myid:] - INFO  [main:QuorumPeerConfig@124] - Reading configuration from: /conf/zoo.cfg
2016-12-18 16:17:47,757 [myid:] - INFO  [main:ZooKeeperServerMain@96] - Starting server
2016-12-18 16:18:09,039 [myid:] - INFO  [main:Environment@100] - Server environment:zookeeper.version=3.4.9-1757313, built on 08/23/2016 06:50 GMT
2016-12-18 16:18:09,040 [myid:] - INFO  [main:Environment@100] - Server environment:host.name=a24405a53ddd
2016-12-18 16:18:09,040 [myid:] - INFO  [main:Environment@100] - Server environment:java.version=1.8.0_111-internal
2016-12-18 16:18:09,040 [myid:] - INFO  [main:Environment@100] - Server environment:java.vendor=Oracle Corporation
2016-12-18 16:18:09,040 [myid:] - INFO  [main:Environment@100] - Server environment:java.home=/usr/lib/jvm/java-1.8-openjdk/jre
2016-12-18 16:18:09,040 [myid:] - INFO  [main:Environment@100] - Server environment:java.class.path=/zookeeper-3.4.9/bin/../build/classes:/zookeeper-3.4.9/bin/../build/lib/*.jar:/zookeeper-3.4.9/bin/../lib/slf4j-log4j12-1.6.1.jar:/zookeeper-3.4.9/bin/../lib/slf4j-api-1.6.1.jar:/zookeeper-3.4.9/bin/../lib/netty-3.10.5.Final.jar:/zookeeper-3.4.9/bin/../lib/log4j-1.2.16.jar:/zookeeper-3.4.9/bin/../lib/jline-0.9.94.jar:/zookeeper-3.4.9/bin/../zookeeper-3.4.9.jar:/zookeeper-3.4.9/bin/../src/java/lib/*.jar:/conf:
2016-12-18 16:18:09,040 [myid:] - INFO  [main:Environment@100] - Server environment:java.library.path=/usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64/server:/usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64:/usr/lib/jvm/java-1.8-openjdk/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2016-12-18 16:18:09,041 [myid:] - INFO  [main:Environment@100] - Server environment:java.io.tmpdir=/tmp
2016-12-18 16:18:09,041 [myid:] - INFO  [main:Environment@100] - Server environment:java.compiler=<NA>
2016-12-18 16:18:09,043 [myid:] - INFO  [main:Environment@100] - Server environment:os.name=Linux
2016-12-18 16:18:09,043 [myid:] - INFO  [main:Environment@100] - Server environment:os.arch=amd64
2016-12-18 16:18:09,044 [myid:] - INFO  [main:Environment@100] - Server environment:os.version=4.4.27-moby
2016-12-18 16:18:09,044 [myid:] - INFO  [main:Environment@100] - Server environment:user.name=zookeeper
2016-12-18 16:18:09,044 [myid:] - INFO  [main:Environment@100] - Server environment:user.home=/home/zookeeper
2016-12-18 16:18:09,044 [myid:] - INFO  [main:Environment@100] - Server environment:user.dir=/zookeeper-3.4.9
2016-12-18 16:18:09,057 [myid:] - INFO  [main:ZooKeeperServer@815] - tickTime set to 2000
2016-12-18 16:18:09,057 [myid:] - INFO  [main:ZooKeeperServer@824] - minSessionTimeout set to -1
2016-12-18 16:18:09,058 [myid:] - INFO  [main:ZooKeeperServer@833] - maxSessionTimeout set to -1
2016-12-18 16:18:09,076 [myid:] - INFO  [main:NIOServerCnxnFactory@89] - binding to port 0.0.0.0/0.0.0.0:2181

top and stats get the state of the process in the container

docker top CONTAINER works similarly to top execution in containers.

$ docker top jstorm-zookeeper
PID    USER      TIME  COMMAND
24593  dockrema  0:01  /usr/lib/jvm/java-1.8-openjdk/jre/bin/java -Dzookeeper.log.dir=. .....

$ docker stats jstorm-zookeeper
CONTAINER           CPU %     MEM USAGE / LIMIT       MEM %   NET I/O        BLOCK I/O   PIDS
jstorm-zookeeper    0.00%     24.86 MiB / 1.951 GiB   1.24%   648 B / 648 B  0 B / 0 B   20

View container details through inspect

Details of docker inspect CONTAINER after meals mirror and container. For example:

  • State - the current state of the container

  • LogPath - history(stdout) file path

  • Config.Env - Environmental Variables

  • Network Settings. Ports - Mapping Relations of Ports

Environmental variables are very useful. Many problems are caused by environmental variables.

history view image layers

You can see the instructions, sizes, and hashes created at each level. It can be used to check whether the image meets your expectations.

Keywords: Linux Zookeeper Docker Java jvm

Added by Fsoft on Mon, 15 Jul 2019 22:22:52 +0300