Skip to content

gh-159111: Update vendored mimalloc to v2.5.2 - #159112

Open
corona10 wants to merge 7 commits into
python:mainfrom
corona10:gh-159111
Open

gh-159111: Update vendored mimalloc to v2.5.2#159112
corona10 wants to merge 7 commits into
python:mainfrom
corona10:gh-159111

Conversation

@corona10

@corona10 corona10 commented Oct 10, 2026 •
edited by bedevere-app Bot
Loading

Copy link
Copy Markdown
Member

@bedevere-app bedevere-app Bot added the type-feature A feature request or enhancement label Oct 10, 2026
@bedevere-app bedevere-app Bot mentioned this pull request Oct 10, 2026
Open
@corona10 corona10 self-assigned this Oct 10, 2026
@corona10 corona10 added the 🔨 test-with-refleak-buildbots Test PR w/ refleak buildbots; report in status section label Oct 10, 2026
@bedevere-bot

bedevere-bot commented Oct 10, 2026

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @corona10 for commit 2649e2b 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F159112%2Fmerge

If you want to schedule another build, you need to add the 🔨 test-with-refleak-buildbots label again.

@bedevere-bot bedevere-bot removed the 🔨 test-with-refleak-buildbots Test PR w/ refleak buildbots; report in status section label Oct 10, 2026
@corona10
corona10 marked this pull request as ready for review October 10, 2026 11:25
@corona10
corona10 requested a review from mpage October 10, 2026 11:25
@corona10 corona10 added the 🔨 test-with-refleak-buildbots Test PR w/ refleak buildbots; report in status section label Oct 10, 2026
@bedevere-bot

bedevere-bot commented Oct 10, 2026

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @corona10 for commit f81e6c2 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F159112%2Fmerge

If you want to schedule another build, you need to add the 🔨 test-with-refleak-buildbots label again.

@bedevere-bot bedevere-bot removed the 🔨 test-with-refleak-buildbots Test PR w/ refleak buildbots; report in status section label Oct 10, 2026
@corona10

corona10 commented Oct 10, 2026

Copy link
Copy Markdown
Member Author

@mpage, could you trigger a free-threading benchmark for this change? :)

@read-the-docs-community

read-the-docs-community Bot commented Oct 10, 2026 •
edited
Loading

Copy link
Copy Markdown

Documentation build overview

📚 cpython-previews | 🛠️ Build #35060762 | 📁 Comparing 91f4f86 against main (88fd5fe)

  🔍 Preview build  

3 files changed
± license.html
± whatsnew/3.16.html
± whatsnew/changelog.html

@corona10 corona10 added the 🔨 test-with-refleak-buildbots Test PR w/ refleak buildbots; report in status section label Oct 10, 2026
@bedevere-bot

bedevere-bot commented Oct 10, 2026

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @corona10 for commit 222601b 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F159112%2Fmerge

If you want to schedule another build, you need to add the 🔨 test-with-refleak-buildbots label again.

@bedevere-bot bedevere-bot removed the 🔨 test-with-refleak-buildbots Test PR w/ refleak buildbots; report in status section label Oct 10, 2026
@corona10

corona10 commented Oct 10, 2026

Copy link
Copy Markdown
Member Author

I will fix MSAN later:

==11983==WARNING: MemorySanitizer: use-of-uninitialized-value
    #0 0x55e368c48377 in _mi_random_next /home/runner/work/cpython/cpython/Objects/mimalloc/random.c:158:3
    #1 0x55e368c48377 in _mi_heap_random_next /home/runner/work/cpython/cpython/Objects/mimalloc/heap.c:281:10
    #2 0x55e368c48377 in mi_heap_main_init /home/runner/work/cpython/cpython/Objects/mimalloc/init.c:220:29
    #3 0x55e368c49eaf in _mi_auto_process_init /home/runner/work/cpython/cpython/Objects/mimalloc/init.c:568:3
    #4 0x7f8aa762aa53 in __libc_start_main (/usr/lib/x86_64-linux-gnu/libc.so.6+0x2aa53) (BuildId: 4ec69afdf3da96ce7388640dcd27c6d0c1b256ef)
    #5 0x55e36886ee24 in _start (/home/runner/work/cpython/cpython/Programs/_freeze_module+0xb4e24) (BuildId: 5e5dc085937d530e3b49b31b09d53deb8664a51d)

  Uninitialized value was stored to memory at
    #0 0x55e368c8b8a0 in chacha_block /home/runner/work/cpython/cpython/Objects/mimalloc/random.c:67:20
    #1 0x55e368c47beb in chacha_next32 /home/runner/work/cpython/cpython/Objects/mimalloc/random.c:83:5
    #2 0x55e368c47beb in _mi_random_next /home/runner/work/cpython/cpython/Objects/mimalloc/random.c:154:22
    #3 0x55e368c47beb in _mi_heap_random_next /home/runner/work/cpython/cpython/Objects/mimalloc/heap.c:281:10
    #4 0x55e368c47beb in mi_heap_main_init /home/runner/work/cpython/cpython/Objects/mimalloc/init.c:220:29
    #5 0x55e368c49eaf in _mi_auto_process_init /home/runner/work/cpython/cpython/Objects/mimalloc/init.c:568:3
    #6 0x7f8aa762aa53 in __libc_start_main (/usr/lib/x86_64-linux-gnu/libc.so.6+0x2aa53) (BuildId: 4ec69afdf3da96ce7388640dcd27c6d0c1b256ef)
    #7 0x55e36886ee24 in _start (/home/runner/work/cpython/cpython/Programs/_freeze_module+0xb4e24) (BuildId: 5e5dc085937d530e3b49b31b09d53deb8664a51d)

  Uninitialized value was stored to memory at
    #0 0x55e368c5d150 in chacha_init /home/runner/work/cpython/cpython/Objects/mimalloc/random.c:109:23
    #1 0x55e368c5d150 in mi_random_init_ex /home/runner/work/cpython/cpython/Objects/mimalloc/random.c:201:3
    #2 0x55e368c47b57 in _mi_random_init /home/runner/work/cpython/cpython/Objects/mimalloc/random.c:205:3
    #3 0x55e368c47b57 in mi_heap_main_init /home/runner/work/cpython/cpython/Objects/mimalloc/init.c:218:7
    #4 0x55e368c49eaf in _mi_auto_process_init /home/runner/work/cpython/cpython/Objects/mimalloc/init.c:568:3
    #5 0x7f8aa762aa53 in __libc_start_main (/usr/lib/x86_64-linux-gnu/libc.so.6+0x2aa53) (BuildId: 4ec69afdf3da96ce7388640dcd27c6d0c1b256ef)
    #6 0x55e36886ee24 in _start (/home/runner/work/cpython/cpython/Programs/_freeze_module+0xb4e24) (BuildId: 5e5dc085937d530e3b49b31b09d53deb8664a51d)

  Uninitialized value was created by an allocation of 'key' in the stack frame
    #0 0x55e368c5cbad in mi_random_init_ex /home/runner/work/cpython/cpython/Objects/mimalloc/random.c:181:3

SUMMARY: MemorySanitizer: use-of-uninitialized-value /home/runner/work/cpython/cpython/Objects/mimalloc/random.c:158:3 in _mi_random_next
Exiting

@corona10
Re-apply CPython-specific changes to mimalloc v2.5.2
836dced 
Most of the previous downstream changes were upstreamed in mimalloc
v2.1.3 - v2.1.7 (heap tags, _mi_heap_init/_mi_tld_init, MI_PRIM_THREAD_ID,
sub-processes, heap walk optimizations, platform fixes) and are dropped.

Remaining CPython-specific changes, all marked with a "CPython" comment:

- QSBR delayed page freeing for free-threaded builds (pythongh-115103,
  pythongh-145615): mi_page_t/mi_heap_t fields and _PyMem_mi_* hooks in
  page.c, heap.c and segment.c.
- debug_offset: preserve the first bytes of a PyObject (ob_tid, refcount
  fields) when filling freed or uninitialized memory in debug builds
  (pythongh-115491).
- Atomic mi_block_t.next loads/stores to avoid TSAN reports (pythongh-129748).
- Relax the owning thread assertion in _mi_free_delayed_block for
  free-threaded builds, since the GC traverses heaps of other threads.
- Call _mi_heap_delayed_free_partial() in mi_heap_visit_blocks()
  (pythongh-112532).
- Only count reclaimed pages up to MI_MEDIUM_OBJ_SIZE_MAX as a
  "right page" in mi_segment_reclaim (pythongh-115103).
- Rename the weak _ZSt15get_new_handlerv symbol and disable the
  mi_is_in_heap_region debug warning.
- Zero-initialize _mi_page_empty instead of a positional initializer.
@corona10
Update the free-threaded build to mimalloc v2.5.2 changes
3d0f073 
mimalloc v2.1.4 replaced the list of abandoned segments with per-arena
bitmaps and introduced sub-processes (mi_subproc_t) to keep abandoned
segments of separate interpreters apart. Use them instead of the
CPython-specific mi_abandoned_pool_t:

- Each interpreter owns a mi_subproc_t created in init_interpreter().
- tstate_mimalloc_bind() sets tld->segments.subproc and uses the
  upstream _mi_heap_init() signature.
- The GC and sys.getallocatedblocks() visit abandoned segments with
  _PyMem_mi_visit_abandoned_blocks(), the upstream cursor API without
  the mi_option_visit_abandoned requirement, while STW.
@corona10
Update NEWS.d
29b2d1b
@corona10
fix MSAN
c3e6efb
@corona10
fix TSAN
2599016
@corona10
fix test_cext
91f4f86
@corona10 corona10 added the 🔨 test-with-refleak-buildbots Test PR w/ refleak buildbots; report in status section label Oct 10, 2026
@bedevere-bot

bedevere-bot commented Oct 10, 2026

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @corona10 for commit 91f4f86 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F159112%2Fmerge

If you want to schedule another build, you need to add the 🔨 test-with-refleak-buildbots label again.

@bedevere-bot bedevere-bot removed the 🔨 test-with-refleak-buildbots Test PR w/ refleak buildbots; report in status section label Oct 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ericsnowcurrently ericsnowcurrently Awaiting requested review from ericsnowcurrently ericsnowcurrently is a code owner
@ZeroIntensity ZeroIntensity Awaiting requested review from ZeroIntensity ZeroIntensity is a code owner
@FFY00 FFY00 Awaiting requested review from FFY00 FFY00 is a code owner
@AA-Turner AA-Turner Awaiting requested review from AA-Turner AA-Turner is a code owner
@erlend-aasland erlend-aasland Awaiting requested review from erlend-aasland erlend-aasland is a code owner
@emmatyping emmatyping Awaiting requested review from emmatyping emmatyping is a code owner
@itamaro itamaro Awaiting requested review from itamaro itamaro is a code owner
@mpage mpage Awaiting requested review from mpage

Assignees

@corona10 corona10

Labels

awaiting core review type-feature A feature request or enhancement

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@corona10 @bedevere-bot