Repository navigation
Move _Py_RefTotal to PyInterpreterState #102304
Description
Activity
From #100227 (comment):
FYI, I have a plan for
_Py_RefTotalthat should preserve stable ABI compatibility without breaking interpreter isolation (and without relying on granular locks). Basically:- deprecate
_Py_Reftotalbut keep exporting the symbol (for stable ABI compatibility) - add
_PyRuntimeState.object_state.last_legacy_reftotal - add
_PyRuntimeState.object_state.reftotal - update everywhere to use the
reftotalfield (for simplicity, add#define _Py_RefTotal _PyRuntime.object_state.reftotal)
We would update
_Py_GetRefTotal()to something like this:Py_ssize_t _Py_GetRefTotal(void) { // _Py_GetLegacyRefTotal() returns the value of the actual `_Py_RefTotal` global. Py_ssize_t legacy = _Py_GetLegacyRefTotal(); _PyRuntimeState.object_state.reftotal += legacy - _PyRuntimeState.object_state.last_legacy_reftotal; _PyRuntimeState.object_state.last_legacy_reftotal = legacy; return _PyRuntimeState.object_state.reftotal; }
All this assumes that folks are not using
_Py_RefTotaldirectly.
For a per-interpreter GIL, I see two options:
- add a granular lock around modifying
_PyRuntimeState.object_state.reftotal, which would hurt the performance of incref/decref, but only on debug builds - switch to
PyInterpreterState.object_state.reftotaland aggregate the values from all interpreters in_Py_GetRefTotal()
- deprecate
Alternately, from #100227 (comment):
_Py_RefTotalis only defined ifPy_REF_DEBUGis defined.
All updates to_Py_RefTotalcan be made atomic (which will have terrible performance with multiple parallel interpreters) to preserve the API.I vote to make it atomic instead of per interpreter. I use this while debugging refleaks and making it per interp will effectively make it impossible to use. This is only used in debug builds so I am not too much concerned about performance.
Reacted by Eric Snow62 remaining items
- added a commit that references this issue
on Jun 9, 2023 - added 5 commits that reference this issue
on Jun 9, 2023 - added a commit that references this issue
on Sep 29, 2023
Metadata
Metadata
Assignees
Labels
Projects
- Status
(See gh-100227.)
_Py_RefTotalholds the current global total number of refcounts. It only exists ifPy_REF_DEBUGis defined (implied byPy_DEBUG). It is exposed bysys.gettotalrefcount()and set byPy_INCREF(),Py_DECREF(), etc. and_Py_NewReference().Modications to
_Py_RefTotalare currently protected by the GIL so it should be moved toPyInterpreterState. For various aspects of compatibility, it makes sense to keep the_Py_RefTotalsymbol around (and correct) and keep returning the global total fromsys.gettotalrefcount().Also,
_Py_RefTotalis used by stable ABI extensions only wherePy_REF_DEBUGis defined (unlikely) and only where built against 3.9 or earlier. Just in case, though, we must still keep the global variable around, so any solution here must respect that.Linked PRs