Repository navigation
Mimalloc differences from upstream #113141
Description
Activity
And which commit is our mimalloc baseline code?
I can spend time for implementing a tool that generating diff files and comparing them on the CI level if it looks okay :)Reacted by Sam JamesHmm, how about generating patchfiles and storing them somewhere in the repository for records? :)
That sounds painful to maintain. I would prefer to have a process to upstream changes first (mimalloc), and then apply them downstream (Python).
Here we wanted to move quickly to add mimalloc and fix all portability issues. But we should be more "upstream first" for following changes.
Reacted by Sam JamesAh okay if they accept our changes, it looks reasonable.
I created this issue in anticipation of further mimalloc changes. There will be changes to support GC in the
--disable-gilbuilds and a few for the optimistic dict accesses. We are going to need to continue to move quickly. An upstream first model is not going to work yet.I don't think it will be too hard to pull in new mimalloc versions. I've done it a few times in the nogil forks: diff vs. base to get patch file, patch new version, fix merge conflicts. We can write up the workflow, but I think automation is overkill at this point.
I had a nice chat with Daan (mimalloc author) on Friday and we got a chance to discuss some (but not all) of the mimalloc changes to support GC. I think we'll be able to get changes merged upstream, but for non-trivial changes I think it's helpful to understand and be able to explain how APIs are used in CPython. Even if time were not an issue, I would not want to be in a situation where we propose something upstream and then it turns out we needed something slightly different.
Reacted by Donghee Na, Carl Meyer, Itamar Oren, Erlend E. Aasland and Eric SnowCC @daanx
Reacted by Erlend E. Aasland- added a commit that references this issue
on Apr 12, 2024 - added a commit that references this issue
on May 18, 2024 I have started upstreaming mimalloc specific modifications into mimalloc to make it easier to downstream future releases of mimalloc. The ideal goal would be to enable Python to pull in mimalloc with no further modifications -- let's see how far we can get.
For now, I am documenting here which commits fix particular issues together with the version of mimalloc that will have these commits. (last update: 2024-06-03)
Planned for v2.1.8
Mimalloc changes to support GC
-
gh-112532: Isolate abandoned segments by interpreter #113717 (v2.1.8, various upstream commits, some are: microsoft/mimalloc@d9aa19a (sub-processes), microsoft/mimalloc@8f87455 (visit arena abandoned per sub process), microsoft/mimalloc@855e3b2 (all abandoned block visiting), and a concurrency "leak" fix microsoft/mimalloc@96b69d7)
This now works with the new bitmap based abandoned segments which scales better. -
gh-112532: Improve mimalloc page visiting #114133 (v2.1.8, upstream commit microsoft/mimalloc@f7fe5bf)
(note: would like to check correctness offast_divideand the range possible ofbsize)
Mimalloc changes for lock-free reads (using QSBR)
upstreamed in v2.1.7:
Simple bug fixes
- gh-90815: Fix mimalloc build on WASI #111524 (upstream commit microsoft/mimalloc@e1f6516, v2.1.6)
- gh-111544: Fix mimalloc build on AIX #111593 (upstream commit microsoft/mimalloc@87c4012, v2.1.7)
- gh-111906: Remove
unused functionwarnings during mimalloc build on FREEBSD #111907 (upstream commit microsoft/mimalloc@87c4012, v2.1.7) - gh-90815: Fix mimalloc atomic.h on Windows arm64 #111527 (upstream commit microsoft/mimalloc@c48a212, v2.1.7)
- gh-112027: Don't print mimalloc warning after mmap() call #113372 (upstream commit microsoft/mimalloc@92a8268, v2.1.7)
- gh-117547: Fix mimalloc compile error on OpenBSD #117548 (upstream commit microsoft/mimalloc@87c4012, v2.1.7)
- gh-117755: Fix mimalloc for huge allocation on s390x #117809 (upstream commit microsoft/mimalloc@78418b3, v2.1.7, issue Issue #876: Fix integer overflow on slice_count microsoft/mimalloc#877)
- gh-116984: Make mimalloc header includes relative to the current file #118808 (upstream commit microsoft/mimalloc@381da05, v2.1.7)
Mimalloc changes to support GC
-
gh-112532: Use separate mimalloc heaps for GC objects #113263 (upstream commit microsoft/mimalloc@710d613, v2.1.7) note: renamed
_mi_heap_init_exto_mi_heap_init(and the original heap init tomi_thread_heap_init). -
gh-112532: Tag mimalloc heaps and pages #113742 (upstream commit microsoft/mimalloc@0c4041f, v2.1.7) note: added an error condition if page with a certain custom tag cannot be reclaimed in a heap with the same tag; is EINVAL strong enough? maybe make it EFAULT to abort in such case?. It should not happen though if used properly.
(edit: I made it EFAULT to catch any bugs early) -
[Make the garbage collector thread-safe in
--disable-gilbuilds #112529: Use _PyThread_Id() in mimalloc (free-threaded build) gh-112529: Use _PyThread_Id() in mimalloc (free-threaded build) #115488](gh-112529: Use _PyThread_Id() in mimalloc (free-threaded build) #115488, v2.1.7) (upstream commit microsoft/mimalloc@66052f1)
Reacted by Eli Schwartz, Sam James, Erlend E. Aasland, GalaxySnail, Sam Gross, Eric Snow, Itamar Oren, IceCodeNew, Yupei Liu, eihqnh and 4 more-
- added a commit that references this issue
on Oct 27, 2025 - added a commit that references this issue
on Mar 29, 2026 - added a commit that references this issue
on Apr 12, 2026 We have four CodeQL warnings for bundled mimalloc:
- https://github.com/python/cpython/security/code-scanning/84
- https://github.com/python/cpython/security/code-scanning/85
- https://github.com/python/cpython/security/code-scanning/87
- https://github.com/python/cpython/security/code-scanning/86
I wrote a PR upstream (https://github.com/microsoft/mimalloc/pull/1321) to fix two and note the other two.
This issue tracks divergence in our copy of mimalloc from upstream https://github.com/microsoft/mimalloc. The purpose is to help with upstreaming fixes and also so that our modifications are not lost as we pull in new mimalloc versions.
base version: v2.1.2
Simple bug fixes
unused functionwarnings during mimalloc build on FREEBSD #111907Py_Initializein background thread #123052 (May be specific to CPython's usage)Mimalloc changes to support GC
Mimalloc changes for lock-free reads (using QSBR)
UWP (Universal Windows Platform) Support
mimallocfor UWP #152477