Document of orphanage¶
Overview¶
Let child processes in Python suicide if they became orphans.
Usage¶
from orphanage import exit_when_orphaned
exit_when_orphaned()
Motivation¶
Some application server softwares (e.g. Gunicorn) work on a multiple-process architect which we call the master-worker model. They must clean up the worker processes if the master process is stopped, to prevent them from becoming orphan processes.
In the gevent-integration scene, the worker processes of Gunicorn poll their
ppid
in an user thread (a.k.a greenlet) to be orphan-aware. But the user
thread may be hanged once the master process crashed because of the blocked
writing on a pipe, the communicating channel between master process and
worker processes.
We want to perform this ppid
polling in a real kernel thread. That is the
intent of this library.
Principle¶
This library spawns an internal thread to poll the ppid
at regular
intervals (for now it is one second). Once the ppid
changed, the original
parent process should be dead and the current process should be orphaned. The
internal thread will send SIGTERM
to the current process.
In the plan, the prctl
& SIGHUP
pattern may be introduced in Linux
platforms to avoid from creating threads. For now, the only supported strategy
is the ppid
polling, for being portable.
Alternatives¶
CaoE is an alternative to this library which developed by the Douban Inc. It
uses prctl
and a twice-forking pattern. It has a pure Python implementation
without any C extension compiling requirement. If you don’t mind to twist the
process tree, that will be a good choice too.
Contributing¶
If you want to report bugs or request features, please feel free to open issues on GitHub.
Of course, pull requests are always welcome.
Table of Content¶
Development Guide¶
There is a Makefile
for development in local environment.
See available commands via invoking make help
.
You will need to install pyenv and tox_ globally in your local environment:
brew install pyenv tox
pyenv install 2.7.14 # and also
Requirement¶
The requirement changes of testing and document building environment need to be
included in requirements-test.in
and docs/requirements.in
. You will
need to invoke make deps
to compile them into requirements*.txt
.
Test¶
For running test in all supported Python versions, you will need pyenv:
# Enter the multi-version Python environment (2.7, 3.6, pypy2, pypy3)
pyenv shell 2.7.14:3.6.5:pypy2.7-5.10.0:pypy3.5-5.10.1
make test
For debugging, you may want to test in a specific Python version, such as 2.7:
tox -e py27 # Default pytest options
tox -e py27 -- -vxs --log-cli-level=DEBUG # Custom pytest options
Package¶
For packaging a new distribution, make dist
will be helpful. It assumes you
are using macOS and the Docker for Mac has been installed and started also. The
binary wheel packages for macOS (with your current ABI) and Linux (with
manylinux API) will be present. Using pyenv and bumpversion is a good idea:
# Enter the multi-version Python environment (2.7, 3.6, pypy2, pypy3)
pyenv shell 2.7.14:3.6.5:pypy2.7-5.10.0:pypy3.5-5.10.1
bumpversion minor # Commit and tag a new major/minor/patch release
make dist # Build release packages
make dist options="-b dev0" # Build pre-release packages
Clean up¶
You could clean up the workspace with make clean
. It removes files which
was ignored in the version control except the .tox
.
Debugg C Extension¶
Debugging the C extension of Python needs different toolchains and skills. The
lldb
or gdb
will be useful in that:
tox -e py27 # Run test until it hangs
vim tests/_orphanage_poll.c # Inspect the CFFI generated code
lldb --attach-pid=100001 # Attach to the target process
lldb> breakpoint set -f _orphanage_poll.c -l 434
lldb> continue
lldb> bt all
For unexpected crashing, the coredump will include useful information:
ulimit -c unlimited # Turn on coredump in current shell
tox -e py27 # Run test until it crashes
lldb --core /cores/cores.10 # Open the coredump named with its pid
lldb> bt all # Print the backtrace
API Reference¶
For most users, please use the public API instead of the internal one.
Public API¶
-
orphanage.
exit_when_orphaned
()¶ Let the current process exit when it was orphaned.
Calling multiple times and calling-and-forking are both safe. But this is not a thread safe function. Never call it concurrently.
Internal API¶
-
class
orphanage.poll.
Context
(callbacks=None, suicide_instead=False)¶ The context of orphans polling which acts as the CFFI wrapper.
Caution
It is dangerous to use this class directly except you are familiar with the implementation of CPython and you know what you are doing clearly. It is recommended to use the Public API instead, for most users.
The context must be closed via
close()
or the memory will be leaked.Parameters: callbacks – Optional. The list of callback functions. A callback function will be passed one parameter, the instance of this context. Be careful, never invoking any Python built-in and C/C++ extended functions which use the Py_BEGIN_ALLOW_THREADS
, such asos.close
and all methods on this context, to avoid from deadlock and other undefined behaviors.-
close
()¶ Closes this context and release the memory from C area.
-
start
()¶ Starts the polling thread.
-
stop
()¶ Stops the polling thread.
Don’t forget to release allocated memory by calling
close()
if you won’t use it anymore.
-
trigger_callbacks
()¶ Triggers the callback functions.
This method is expected to be called from C area.
-
-
orphanage.poll.
orphanage_poll_routine_callback
(ptr)¶ The external callback function of CFFI.
This function invokes the
Context.trigger_callbacks()
method.Parameters: ptr – The C pointer of context. Returns: 0
for nonerror calls.
-
orphanage.poll.
perror
(description)¶ Raises a runtime error from the specified description and
errno
.
-
orphanage.poll.
raise_for_return_value
(return_value)¶ Checks the return value from C area.
A runtime error will be raised if the return value is nonzero.