-
Type: Bug
-
Resolution: Done
-
Priority: Major - P3
-
Affects Version/s: None
-
Component/s: Testing Infrastructure
-
Fully Compatible
-
-
Query F (02/01/16)
-
0
The _posixsubprocess C extension module was introduced in CPython 3.2.0 to resolve the issues (e.g. potential to hang) when using the subprocess module from multiple threads concurrently. resmoke.py attempted to work around these thread-safety issues with the subprocess module in Python 2.6 and Python 2.7 by guarding calls to subprocess.Popen() with a lock. However, it turns out that the lock is insufficient for avoiding corruption of Python's internal data structure for thread-local storage in the child process:
[2016/01/13 06:24:33.816] *** Error in `python': free(): invalid size: 0x00007f0044000b20 *** [2016/01/13 06:24:33.818] ======= Backtrace: ========= [2016/01/13 06:24:33.818] /lib64/libc.so.6(+0x730bf)[0x7f00a54f30bf] [2016/01/13 06:24:33.818] /lib64/libc.so.6(+0x7892e)[0x7f00a54f892e] [2016/01/13 06:24:33.818] /lib64/libc.so.6(+0x79636)[0x7f00a54f9636] [2016/01/13 06:24:33.818] /usr/lib64/libpython2.7.so.1.0(PyThread_ReInitTLS+0x4a)[0x7f00a5b530ca] [2016/01/13 06:24:33.818] /usr/lib64/libpython2.7.so.1.0(PyOS_AfterFork+0x10)[0x7f00a5ae5119] [2016/01/13 06:24:33.818] /usr/lib64/libpython2.7.so.1.0(+0x10ef86)[0x7f00a5b53f86] [2016/01/13 06:24:33.818] /usr/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x5d1)[0x7f00a5b0c6e1] [2016/01/13 06:24:33.818] /usr/lib64/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x4fe)[0x7f00a5b1329e] [2016/01/13 06:24:33.818] /usr/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0xc73)[0x7f00a5b0cd83] [2016/01/13 06:24:33.818] /usr/lib64/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x4fe)[0x7f00a5b1329e] [2016/01/13 06:24:33.818] /usr/lib64/libpython2.7.so.1.0(+0xb24ab)[0x7f00a5af74ab] [2016/01/13 06:24:33.818] /usr/lib64/libpython2.7.so.1.0(PyObject_Call+0x46)[0x7f00a5af26c6] [2016/01/13 06:24:33.818] /usr/lib64/libpython2.7.so.1.0(+0xae5aa)[0x7f00a5af35aa] [2016/01/13 06:24:33.818] /usr/lib64/libpython2.7.so.1.0(PyObject_Call+0x46)[0x7f00a5af26c6] [2016/01/13 06:24:33.818] /usr/lib64/libpython2.7.so.1.0(+0xbe7f9)[0x7f00a5b037f9] [2016/01/13 06:24:33.818] /usr/lib64/libpython2.7.so.1.0(+0xbde2a)[0x7f00a5b02e2a] [2016/01/13 06:24:33.818] /usr/lib64/libpython2.7.so.1.0(PyObject_Call+0x46)[0x7f00a5af26c6] [2016/01/13 06:24:33.818] /usr/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x144e)[0x7f00a5b0d55e] [2016/01/13 06:24:33.818] /usr/lib64/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x221)[0x7f00a5b12fc1] [2016/01/13 06:24:33.818] /usr/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0xc73)[0x7f00a5b0cd83] [2016/01/13 06:24:33.818] /usr/lib64/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x221)[0x7f00a5b12fc1] [2016/01/13 06:24:33.818] /usr/lib64/libpython2.7.so.1.0(+0xb24ab)[0x7f00a5af74ab] [2016/01/13 06:24:33.818] /usr/lib64/libpython2.7.so.1.0(PyObject_Call+0x46)[0x7f00a5af26c6] [2016/01/13 06:24:33.818] /usr/lib64/libpython2.7.so.1.0(+0xae5aa)[0x7f00a5af35aa] [2016/01/13 06:24:33.818] /usr/lib64/libpython2.7.so.1.0(PyObject_Call+0x46)[0x7f00a5af26c6] [2016/01/13 06:24:33.818] /usr/lib64/libpython2.7.so.1.0(+0x6e792)[0x7f00a5ab3792] [2016/01/13 06:24:33.818] /usr/lib64/libpython2.7.so.1.0(PyObject_Call+0x46)[0x7f00a5af26c6] [2016/01/13 06:24:33.818] /usr/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x21f5)[0x7f00a5b0e305] [2016/01/13 06:24:33.818] /usr/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x29cc)[0x7f00a5b0eadc] [2016/01/13 06:24:33.818] /usr/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x29cc)[0x7f00a5b0eadc] [2016/01/13 06:24:33.818] /usr/lib64/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x221)[0x7f00a5b12fc1] [2016/01/13 06:24:33.818] /usr/lib64/libpython2.7.so.1.0(+0xb222f)[0x7f00a5af722f] [2016/01/13 06:24:33.818] /usr/lib64/libpython2.7.so.1.0(PyObject_Call+0x46)[0x7f00a5af26c6] [2016/01/13 06:24:33.818] /usr/lib64/libpython2.7.so.1.0(+0xae5aa)[0x7f00a5af35aa] [2016/01/13 06:24:33.818] /usr/lib64/libpython2.7.so.1.0(PyObject_Call+0x46)[0x7f00a5af26c6] [2016/01/13 06:24:33.819] /usr/lib64/libpython2.7.so.1.0(PyEval_CallObjectWithKeywords+0x30)[0x7f00a5b0bc40] [2016/01/13 06:24:33.819] /usr/lib64/libpython2.7.so.1.0(+0x9f25e)[0x7f00a5ae425e] [2016/01/13 06:24:33.819] /lib64/libpthread.so.0(+0x80a4)[0x7f00a58300a4] [2016/01/13 06:24:33.819] /lib64/libc.so.6(clone+0x6d)[0x7f00a55657fd]
The actual error message can vary, but the invalid free() triggers a SIGABRT and causes the child process to exit with a return code of -6:
- "python: free(): invalid pointer"
- "python: double free or corruption"
- "python: munmap_chunk(): invalid pointer"
The version of the subprocess module in Python 3.2 is available as a separate package called subprocess32. Using it will resolve the memory corruption issue because the call to PyOS_AfterFork() (and thus the call to PyThread_ReInitTLS()) is skipped when preexec_fn is None.