Baike.dev
Log in
Back to tool/Back to issues
#164·tini

not reaping zombie or defunct child processes

Author: bingeramboCreated Aug 10, 2020Updated May 31, 2024

In container, I exec a python process as a child process of tini. as following steps:

  1. run the container from dockerfile

os info：

[root@node3 ~]# uname -a
Linux node3 3.10.0-862.el7.x86_64 #1 SMP Fri Apr 20 16:44:24 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
[root@node3 ~]#
[root@node3 ~]#
[root@node3 ~]#
[root@node3 ~]# cat /etc/redhat-release
CentOS Linux release 7.5.1804 (Core)

docker version:

[root@node3 ~]# docker version
Client:
 Version:           18.09.6
 API version:       1.39
 Go version:        go1.10.8
 Git commit:        481bc77156
 Built:             Sat May  4 02:34:58 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.0
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.4
  Git commit:       4d60db4
  Built:            Wed Nov  7 00:19:08 2018
  OS/Arch:          linux/amd64
  Experimental:     false

docker file

ADD tini /tini
RUN chmod +x /tini
ENV PYTHONPATH=/root/tf/models
WORKDIR /examples

ENTRYPOINT ["/tini", "-g", "-w", "-vvv", "--", "bash"]
CMD ["-c","/usr/bin/python ~/tf/benchmarks/scripts/tf_cnn_benchmarks/tf_cnn_benchmarks.py  --batch_size=64 --model=official_resnet18 --optimizer=momentum  --num_gpus=2 --num_epochs=1 --weight_decay=1e-4 --data_dir=/tmp/imagenet"]

process status:

tini: pid 1 python: pid 8

root@deee30ff32b4:/examples# ps axjf
  PPID    PID   PGID    SID TTY       TPGID STAT   UID   TIME COMMAND
     0    397    397    397 pts/1       410 Ss       0   0:00 bash
   397    410    410    397 pts/1       410 R+       0   0:00  \_ ps axjf
     0      1      1      1 pts/0         8 Ss       0   0:00 /tini -g -w -vvv -- bash -c /usr/bin/python ~/tf/benchmarks/scripts/tf_cnn_benchmarks/tf_cnn_benchmarks.py  --batch_size=64 --model=official_resnet
     1      8      8      1 pts/0         8 Sl+      0   2:31 /usr/bin/python /root/tf/benchmarks/scripts/tf_cnn_benchmarks/tf_cnn_benchmarks.py --batch_size=64 --model=official_resnet18 --optimizer=momentum -
  1. kill the python process

kill python process (the child of tini, and its pid is 8 )

root@deee30ff32b4:/examples#
root@deee30ff32b4:/examples#
root@deee30ff32b4:/examples# kill -9 8
root@deee30ff32b4:/examples#
root@deee30ff32b4:/examples#
root@deee30ff32b4:/examples#
root@deee30ff32b4:/examples#
root@deee30ff32b4:/examples#
root@deee30ff32b4:/examples#
root@deee30ff32b4:/examples# ps axjf
  PPID    PID   PGID    SID TTY       TPGID STAT   UID   TIME COMMAND
     0    397    397    397 pts/1       411 Ss       0   0:00 bash
   397    411    411    397 pts/1       411 R+       0   0:00  \_ ps axjf
     0      1      1      1 pts/0         8 Ss       0   0:00 /tini -g -w -vvv -- bash -c /usr/bin/python ~/tf/benchmarks/scripts/tf_cnn_benchmarks/tf_cnn_benchmarks.py  --batch_size=64 --model=official_resnet
     1      8      8      1 pts/0         8 Zl+      0   4:30 [tf_cnn_benchmar] <defunct>
  1. zombie process exists: the python process is defunct, but not reap by parent process (tini), it become zombie.

the tini and python processes print info:

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
W0810 01:44:12.306231 140481052276544 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
[TRACE tini (1)] No child to reap
Initializing graph
WARNING:tensorflow:From /root/tf/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2238: Supervisor.__init__ (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version.
Instructions for updating:
Please switch to tf.train.MonitoredTrainingSession
W0810 01:44:13.657472 140481052276544 deprecation.py:323] From /root/tf/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2238: Supervisor.__init__ (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version.
Instructions for updating:
Please switch to tf.train.MonitoredTrainingSession
2020-08-10 01:44:13.911021: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0, 1
2020-08-10 01:44:13.911240: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-08-10 01:44:13.911262: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990]      0 1
2020-08-10 01:44:13.911272: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0:   N Y
2020-08-10 01:44:13.911358: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 1:   Y N
2020-08-10 01:44:13.911696: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 30555 MB memory) -> physical GPU (device: 0, name: Tesla V100-SXM2-32GB, pci bus id: 0000:3d:00.0, compute capability: 7.0)
2020-08-10 01:44:13.912073: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 30555 MB memory) -> physical GPU (device: 1, name: Tesla V100-SXM2-32GB, pci bus id: 0000:88:00.0, compute capability: 7.0)
[TRACE tini (1)] No child to reap
[TRACE tini (1)] No child to reap
[TRACE tini (1)] No child to reap
[TRACE tini (1)] No child to reap
INFO:tensorflow:Running local_init_op.
I0810 01:44:17.589881 140481052276544 session_manager.py:491] Running local_init_op.
INFO:tensorflow:Done running local_init_op.
I0810 01:44:17.782803 140481052276544 session_manager.py:493] Done running local_init_op.
Running warm up
[TRACE tini (1)] No child to reap
2020-08-10 01:44:18.530569: I tensorflow/stream_executor/dso_loader.cc:152] successfully opened CUDA library libcublas.so.10.0 locally
[TRACE tini (1)] No child to reap
[TRACE tini (1)] No child to reap
[TRACE tini (1)] No child to reap
[TRACE tini (1)] No child to reap
[TRACE tini (1)] No child to reap
[TRACE tini (1)] No child to reap
[TRACE tini (1)] No child to reap
[TRACE tini (1)] No child to reap
[TRACE tini (1)] No child to reap
[TRACE tini (1)] No child to reap
[TRACE tini (1)] No child to reap
[TRACE tini (1)] No child to reap
[TRACE tini (1)] No child to reap
[TRACE tini (1)] No child to reap
[TRACE tini (1)] No child to reap
[TRACE tini (1)] No child to reap
[TRACE tini (1)] No child to reap
[TRACE tini (1)] No child to reap
[TRACE tini (1)] No child to reap
[TRACE tini (1)] No child to reap
[TRACE tini (1)] No child to reap
[TRACE tini (1)] No child to reap
[TRACE tini (1)] No child to reap
[TRACE tini (1)] No child to reap
[TRACE tini (1)] No child to reap
[TRACE tini (1)] No child to reap
[TRACE tini (1)] No child to reap
[TRACE tini (1)] No child to reap
[TRACE tini (1)] No child to reap
[TRACE tini (1)] No child to reap
[TRACE tini (1)] No child to reap
[TRACE tini (1)] No child to reap
[TRACE tini (1)] No child to reap
[TRACE tini (1)] No child to reap
[TRACE tini (1)] No child to reap
[TRACE tini (1)] No child to reap
[TRACE tini (1)] No child to reap
[TRACE tini (1)] No child to reap
[TRACE tini (1)] No child to reap
[TRACE tini (1)] No child to reap
[TRACE tini (1)] No child to reap
[TRACE tini (1)] No child to reap
[TRACE tini (1)] No child to reap
[TRACE tini (1)] No child to reap
[TRACE tini (1)] No child to reap
[TRACE tini (1)] No child to reap
[TRACE tini (1)] No child to reap
[TRACE tini (1)] No child to reap
[TRACE tini (1)] No child to reap
[TRACE tini (1)] No child to reap
[TRACE tini (1)] No child to reap
[TRACE tini (1)] No child to reap

now I can not remove the zombie process. only reboot the machine.

when the python process suspended, the tini will not reap the child process, and the defunct python process will become zombie .

I'm not sure about either of the following reasons:

  1. the tini did not receive the SIGCHLD signal from the defunct child process. Is the tini process missed to deal with the SIGCHLD ?
  2. as a child process of tini. did the python process not send the SIGCHLD to tini? maybe it is suspended for hardware or program problem

Source: krallin/tini

View original on GitHubView discussion on GitHub