Skip to content

Isolate the _datetime extension module #117398

Closed
Closed
Isolate the _datetime extension module#117398
Labels
extension-modulesC modules in the Modules dirtopic-subinterpreterstype-featureA feature request or enhancement

Description

@neonene
neonene
opened on Mar 31, 2024

Feature or enhancement

Proposal:

I hope this issue will complete _datetime isolation.

My concerns (and answers)

  • Py_MOD_PER_INTERPRETER_GIL_SUPPORTED should be applied in sync with _zoninfo?

    • YES: Possible.

  • Can a module state have a C-API structure, keeping a capsule just for comatibility?

    • NO: Possible, but a user should not touch the module state.

  • C-API supports only the main interpreter? Otherwise, PyInterpreterState is acceptable to point each structure?

    • NO: PyDateTimeAPI cannot emit an error. Also, no PyInterpreterState member is accessible from datetime.h. UPDATE: Seems to be possible by using a global function pointer instead of a function.

Specific issue:

Links to previous discussion of this feature:

Linked PRs (closed)

Details

Linked PRs

Activity

  1. neonene
    added
    type-featureA feature request or enhancement
    on Mar 31, 2024

  2. neonene commented on Mar 31, 2024

    @neonene
    neonene
    on Mar 31, 2024
    ContributorAuthor

    PR #117399 fails except Windows. Is there an easy way to access a PyInterpreterState member? I'll try later allowing C-API only for the main-interpreter with a global variable.

  3. Eclips4
    added
    extension-modulesC modules in the Modules dir
    on Mar 31, 2024
  4. erlend-aasland
    added
    topic-subinterpreters
    on Mar 31, 2024
  5. github-project-automation
    added this to Subinterpreterson Mar 31, 2024
  6. github-project-automation
    moved this to Todo in Subinterpreterson Mar 31, 2024

  7. erlend-aasland commented on Mar 31, 2024

    @erlend-aasland
    erlend-aasland
    on Mar 31, 2024
    Contributor

    I briefly discussed the C API capsule with @pganssle on the 2023 (or was it 2022?) language summit. I also asked the Steering Council to comment about backwards compatibility concerns regarding datetime.h (which is not included by Python.h):

    Putting it in the PyInterpreterState struct is an option, but I'm not sure how much we want to bloat that struct. We probably want to deprecate the current capsulated API and introduce a new capsulated API.

  8. vstinner
    added 2 commits that reference this issue on May 5, 2024

    pythongh-117398: Move types to datetime state

    c9027f2

    pythongh-117398: Move types to datetime state

    8c321ef

  9. vstinner commented on May 5, 2024

    @vstinner
    vstinner
    on May 5, 2024
    Member

    Converting the _datetime extension to multiphase initialization and/or converting static types to heap types is complicated because:

    • <Include/datetime.h> C API
    • datetime capsule (C API)
    • If the _datetime extension is reloaded, the C API expects to meet the same types. Otherwise, PyDateTime_Check() fails and everything else fails.

    It reminds me the very complicated case of the PyAST C API. We managed to convert the _ast extension to heap types and multiphase init by moving state to the interpreter state. Other "trade offs" attemps for the _ast extension ended by introducing crashes which were hard to trigger and hard to fix.

    So I propose this plan to isolate the datetime module:

    • Move references to types to datetime_state structure: I wrote a minimum PR gh-117398: Move types to datetime state #118606 for that
    • Convert static types to heap types, but create them only once, and don't delete them (on purpose, for now).
    • Move datetime_state to PyInterpreterState.
    • At the point, discuss how to deal with remaining issues: multiphase init, C API, capsule, etc.

  10. neonene commented on May 5, 2024

    @neonene
    neonene
    on May 5, 2024
    ContributorAuthor

    Where can we see your rationale for the inflating PyInterpreterState with PyAST C-API? IIUC, the practice is supposed to be bad: #103092 (comment).

    How do you associate the PyAST issue with PyDateTime in terms of the mechanizm and impact?

    @encukou, @ericsnowcurrently

  11. vstinner commented on May 6, 2024

    @vstinner
    vstinner
    on May 6, 2024
    Member

    How do you associate the PyAST issue with PyDateTime in terms of the mechanizm and impact?

    Both provide a C API and we wanted to isolate their C extension.

    Where can we see your rationale for the inflating PyInterpreterState with PyAST C-API?

    I will try to dig into the bug tracker later. In short, there were 2-3 crashes related to the isolation of the _ast extension. Crashes related to the C API.

  12. vstinner
    added a commit that references this issue on May 8, 2024

    pythongh-117398: Move types to datetime state

    dbdd4ac

  13. neonene commented on May 10, 2024

    @neonene
    neonene
    on May 10, 2024
    ContributorAuthor

    Python-ast.c (3.10.0 alpha 0: b1cc6ba)

    int PyAST_Check(PyObject* obj)
{
    astmodulestate *state = get_global_ast_state();
    if (state == NULL) {
        return -1;
    }
    return PyObject_IsInstance(obj, state->AST_type);
}

    Previously, PyAST_Check() imported the _ast module through get_global_ast_state(). The implicit import could get an unsafe pseudo module by replacing __import__() with a lazy import (strongly discouraged since Python 3.3).

    datetime.h

    #define PyDate_Check(op) PyObject_TypeCheck((op), PyDateTimeAPI->DateType)
#define PyDate_CheckExact(op) Py_IS_TYPE((op), PyDateTimeAPI->DateType)

    PyDate_Check() requires the real module to be imported in advance by calling PyCapsule_Import() explicitly.

    At least, the regression case in PyAST C-API above cannot be applied to PyDateTime C-API, I think.

  14. vstinner
    added a commit that references this issue on May 10, 2024

    pythongh-117398: Move types to datetime state

    9a5b155

  15. 62 remaining items

  16. mrahtz
    added a commit that references this issue on Jun 30, 2024

    pythongh-117398: Use the correct module loader for iOS in datetime CA…

    b27cc03
  17. freakboy3742
    added a commit that references this issue on Jul 10, 2024

    [3.13] gh-117398: Use the correct module loader for iOS in datetime C…

    c08a302
  18. noahbkim
    added 5 commits that reference this issue on Jul 11, 2024

    pythongh-117398: Convert datetime.IsoCalendarDate To A Heap Type (pyt…

    1008aee

    pythongh-117398: Add datetime Module State (pythongh-119810)

    73c7377

    pythongh-117398: Use Per-Interpreter State for the _datetime Static T…

    930edaa

    pythongh-117398: Add datetime C-API type check test for subinterprete…

    78a5805

    pythongh-117398: Use the correct module loader for iOS in datetime CA…

    81ee80f
  19. estyxx
    added 8 commits that reference this issue on Jul 17, 2024

    pythongh-117398: Statically Allocate the Datetime C-API (pythonGH-119472

    8affc4c

    pythongh-117398: Add multiphase support to _datetime (pythongh-119373)

    bfd7ade

    pythongh-117398: pythongh-119655: datetime: Init static state once & …

    f11e522

    pythongh-117398: Convert datetime.IsoCalendarDate To A Heap Type (pyt…

    4ae6a15

    pythongh-117398: Add datetime Module State (pythongh-119810)

    d679478

    pythongh-117398: Use Per-Interpreter State for the _datetime Static T…

    16eb250

    pythongh-117398: Add datetime C-API type check test for subinterprete…

    6285938

    pythongh-117398: Use the correct module loader for iOS in datetime CA…

    baacac8
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    extension-modulesC modules in the Modules dirtopic-subinterpreterstype-featureA feature request or enhancement

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions