Repository navigation
threading Thread.join should call the OS join API #110829
Description
Feature or enhancement
Proposal:
threading.Thread.join() only waits for the CPython internals to wash its hands of the underlying thread. It doesn't actually wait for the OS thread itself to exit, which in theory happens rapidly as its final internal code completes quickly - but we have no good wait to determine.
Why finally do this now? Now that we're encouraging people to notice and avoid threading existing when os.fork is called in 3.12, a use case has come up for deterministically knowing when the thread is done at the OS level so that the code can proceed with os.fork.
#110510 could use this in an atfork before fork handler for example.
POSIX has pthread_join, we should be able to expose and use via _thread. Windows presumably has an equivalent concept API.
Has this already been discussed elsewhere?
No response given
Links to previous discussion of this feature:
No response
Activity
pitrou commentedon Oct 13, 2023
On Windows, we start the thread with
_beginthreadex, so we should be able to callWaitForSingleObjectto ensure that the thread has properly terminated. See second example here: https://learn.microsoft.com/en-us/cpp/c-runtime-library/reference/beginthread-beginthreadex?view=msvc-170#examplesBut, of course, on Windows the entire fork() problem doesn't exist, so it's less of a problem if we don't provide lesser guarantees.
Note that right now we detach the thread as soon as it is started, so we may want to expose a different internal API that doesn't detach the thread.
pythonGH-110829: Ensure Thread.join() joins the OS thread
pythonGH-110829: Ensure Thread.join() joins the OS thread
pythonGH-110829: Ensure Thread.join() joins the OS thread
pythonGH-110829: Ensure Thread.join() joins the OS thread
ericsnowcurrently commentedon Oct 25, 2023
FWIW, the gap between Python-thread-ends and OS-thread-exits bit me after I added the per-interpreter GIL (PEP 684). We ended up with crash due to that interval (a race in the GIL's
drop_gil(), which I fixed in gh-105109. The linked PRs on gh-104341 include a couple other approaches I tried out to solve the race (with extra complexity), and IIRC at least one of them would have (partially?) dealt with closing the above gap.pitrou commentedon Oct 25, 2023
The API added here is only for the
threadingmodule, though, because it's quite inflexible (see comments in the.hfile).pythonGH-110829: Ensure Thread.join() joins the OS thread
pythonGH-110829: Ensure Thread.join() joins the OS thread
pythonGH-110829: Ensure Thread.join() joins the OS thread
GH-110829: Ensure Thread.join() joins the OS thread (#110848)
hugovk commentedon Nov 9, 2023
Closing as the PR has been merged.
pitrou commentedon Nov 9, 2023
Oops, sorry for forgetting to close this issue!
pythonGH-110829: Ensure Thread.join() joins the OS thread (python#110848
pythonGH-110829: Ensure Thread.join() joins the OS thread (python#110848