Repository navigation
Show calls to native code and GC in profiles #140643
Description
Feature or enhancement
Without too much effort (and negligible overhead in the profiler), we can add some useful additional functionality to the new sampling profiler:
- Show calls through native code in the collected stacks. Note that this doesn't show symbols for the native code itself (that's a much more invasive, fragile change). Rather we already have enough information when walking the Python stack to detect when we leave and reenter the interpreter (for example, by calling through a builtin function or C extension), and insert a dummy
<native>frame in the collected stack between the Python frames. - Even more usefully, we can show where/when garbage collections are occurring, and denote these with
<GC>dummy frames. Any Python code running in finalizers, GC callbacks, or weakref callbacks will be below these in the stack.
@pablogsal and I discussed this at the sprint, and he seemed excited for it. I implemented most of it there, and will open a PR shortly once I've cleaned things up a bit and added docs.
Linked PRs
Metadata
Metadata
Assignees
Labels
Projects
- Status
Activity
markshannon commentedon Oct 29, 2025
I tried an experiment a while ago to put minimal "interpreter frame"s on the C stack (just a few fields) for builtin functions and link them through the interpreter frame stack. The cost was small (about 3 writes and a read on entry and a read and write on exit).
Benchmarking showed no significant slowdown. Unfortunately, I seem to have neither that code nor results.
Here's a branch (probably from a different experiment) that shows some of the refactoring of the frame necessary.
https://github.com/faster-cpython/cpython/tree/push-extension-frame
I suspect this would be a bit slower than your approach, but a lot easier for existing tools to work with.
pablogsal commentedon Oct 29, 2025
I think @markshannon idea is very interesting and is worth trying. Maybe we could put a pointer to the Cfunction object somewhere in the thread state instead? 🤔
brandtbucher commentedon Oct 30, 2025
I'm not sure how useful that would be though, since we'd only have a pointer to the first native function entered, not everything after that. Unless you're saying that we push this minimal frame on every PyCFunction call, not just ones that leave the interpreter?
Either way, I think I'll just start with dummy
<native>frames in the profiler output and then we can iterate on improving that as a follow up, weighing the cost of the slowdown. Definitely seems interesting, though.GH-140643: Add `<native>` and `<GC>` frames to the sampling profiler (#…
pablogsal commentedon Nov 17, 2025
Let's use another issue for the improved native frame tracking
pythonGH-140643: Add `<native>` and `<GC>` frames to the sampling pro…