Repository navigation
test__xxsubinterpreters is Occasionally Crashing #104341
Description
This appears to have started once gh-104210 was merged.
"Fatal Python error: drop_gil: PyMUTEX_LOCK(gil->switch_mutex) failed"
It may be specific to test.test__xxsubinterpreters.RunStringTests.test_create_thread.
See:
- https://buildbot.python.org/all/#builders/484/builds/3261
- GH-104308: socket.getnameinfo should release the GIL #104307 (comment)
Linked PRs
- gh-104341: Ensure tstate Holding GIL is Always Current #104436
- gh-104341: Adjust tstate_must_exit() to Respect Interpreter Finalization #104437
- gh-104341: Clean Up threading Module #104552
- gh-104341: Fix threading Module Shutdown #104560
- gh-104341: Minor Fixes in the _thread Module #104595
- gh-104341: Wait Completely at threading._shutdown() #104672
- [reverted] gh-104341: Add a Separate "Running" Lock for Each Thread #104754
- [3.12] gh-104341: Add a Separate "Running" Lock for Each Thread (gh-104754) #104817
- gh-104341: Call _PyEval_ReleaseLock() with NULL When Finalizing the Current Thread #105109
- [3.12] gh-104341: Call _PyEval_ReleaseLock() with NULL When Finalizing the Current Thread (gh-105109) #105209
Activity
ericsnowcurrently commentedon May 9, 2023
CC @brandtbucher
ericsnowcurrently commentedon May 9, 2023
I've been able to reproduce the crash locally with
./python -m test -j8 test__xxsubinterpreters --foreverbrandtbucher commentedon May 9, 2023
brandtbucher commentedon May 9, 2023
Also when running
test_threads_joinandtest_threads_join_2intest_threadingwith the same flags.ericsnowcurrently commentedon May 10, 2023
I think I've got this figured out. It looks like the interpreter is getting finalized before the thread is completely done in
thread_run()(in _threadmodule.c). Apparently the thread isn't completely finalized at the pointthread.join()returns. Thus there's a small race there where the runtime may think the thread is done running. It proceeds to clean up the thread state even though the thread is still cleaning itself up.Based on the above, there are two points at which it should have waited:
thread.join()wait_for_thread_shutdown()viaPy_EndInterpreter()(in pylifecycle.c)I'm going to figure out why things didn't wait properly. I suspect it looks something like this:
Py_EndInterpreter()The race lies between steps (8) and (12). If (8) doesn't finish before (12) happens then we'd see the crash we're seeing.
What I don't understand is:
To be honest, I still haven't figured out what we are actually doing in (2) such that it affects
thread.join()andwait_for_thread_shutdown(). However that may be irrelevant. I suspect we are missing a step before (9) where we acquire the subinterpreter's GIL while we're cleaning things up inPy_EndInterpreter().FYI, this doesn't happen when the GIL is shared because (9) is blocked by the GIL being held still in the sub-thread.
ericsnowcurrently commentedon May 12, 2023
This may also be related to (or a variant of) gh-96387.
ericsnowcurrently commentedon May 12, 2023
Hmm, looks like I need to add
_PyInterpreterState_GetFinalizing(), etc. and use it intstate_must_exit()(in ceval_gil.c).ericsnowcurrently commentedon May 12, 2023
FYI, the two fixes I have up for review don't solve the crashes yet, but I'm getting closer.
ericsnowcurrently commentedon May 12, 2023
Currently here's where things are at:
thread_run()(in _threadmodule.c) starts cleaning up_PyThreadState_DeleteCurrent()(pystate.c), which calls_PyEval_ReleaseLock()(ceval_gil.c)drop_gil()(ceval_gil.c), where everything goes smoothly, until...MUTEX_UNLOCK(gil->mutex);successfully releases the lock, but often gets stuck when there are many processes running the testPy_EndInterpreter()(pylifecycle.c), where the main interpreter is destroying the subinterpreter, likely around the point the subinterpreter's GIL is destoyeddrop_gil()subsequently blows up because the thread state is no longer validHere's a detailed breakdown of execution:
(expand)
12 remaining items
gh-104837: Revert "gh-104341: Add a Separate "Running" Lock for Each …
ericsnowcurrently commentedon May 30, 2023
After a conversation with @markshannon, I had an epiphany about how to solve this minimally: gh-105109.
pitrou commentedon May 31, 2023
It seems each interpreter could maintain its current number of threads? It would be decremented at the very end of
thread_runafter everything else ran, givingPy_EndInterpretera reliable way of knowing that there's nothing going on anymore.ericsnowcurrently commentedon May 31, 2023
Yeah, that makes sense.
ericsnowcurrently commentedon May 31, 2023
Hmm, daemon threads would be a problem still, no?
gpshead commentedon Jun 1, 2023
We should just disallow daemon threads from Subinterpreters. (Silently or loudly, no opinion)
ericsnowcurrently commentedon Jun 1, 2023
We already do by default for interpreters created with
Py_NewInterpreterFromConfig(), but now with the oldPy_NewInterpreter().gh-104341: Call _PyEval_ReleaseLock() with NULL When Finalizing the C…
pythongh-104341: Call _PyEval_ReleaseLock() with NULL When Finalizing…
[3.12] gh-104341: Call _PyEval_ReleaseLock() with NULL When Finalizin…
ericsnowcurrently commentedon Jun 1, 2023
I've merged my fix to main and 3.12. We should be good to go now. FYI, I do plan on circling back to the underlying problem in the 3.13 timeframe.
ericsnowcurrently commentedon Jun 2, 2023
FYI, there are other crashes I know about but they are much less common and have a different root cause.
Regarding the additional work I mentioned, I'll open a new issue once I get around to it.