Repository navigation
Use PyBytesWriter API in io.RawIOBase.readall #140135
Description
Activity
- changed the title
[-]Use PyBytesWriter API in `io.RawIOBase.readall`[/-][+]Use `PyBytesWriter` API in `io.RawIOBase.readall`[/+]on Oct 14, 2025 cc: @vstinner, @emmatyping . This a very similar optimization to gh-139877.
This has definitely shown up in user profiles although most the time
FileIO.readall(which is an implementation ofRawIOBase.readall) ends up getting calledI like this as an optimization although to me is an intermediate state. I really want to update this to use
bytearray+readinto+take_bytes()which does the same thing with less allocations + copies (self.read()allocates and returns a newbyteswhich is copied into thePyBytesWriterhere). I don't think that should block this optimization though. Changingio.RawIOBaseto defaulting to.readintofrom.readis unfortunately complicated to do without breaking compatibility or making infinite loops.Reacted by Maurycy Pawłowski-Wieroński- added a commit that references this issue
on Oct 15, 2025
Feature or enhancement
Proposal:
There's new
PyBytesWriter()API:There's an ongoing effort of introducing it in various places:
With its
PyList_Appendcpython/Modules/_io/iobase.c
Line 1003 in 2ca3c85
and
PyBytes_Join:https://github.com/python/cpython/blob/2ca3c85054b231505c5e6f5f3209eb5f63942dce/Modules/_io/iobase.c#L1010C14-L1010C26
the
io.RawIOBase.readalllooks like the ideal candidate forPyBytesWriter.Notably,
io.RawIOBaseis used internally across many modules:https://github.com/search?q=repo%3Apython%2Fcpython+io.RawIOBase+path%3A%2F%5ELib%5C%2F%2F&type=code
and some even call
readallexplicitly sometimes:cpython/Lib/gzip.py
Line 545 in 2ca3c85
Has this already been discussed elsewhere?
No response given
Links to previous discussion of this feature:
No response
Linked PRs
PyBytesWriterinio.RawIOBase.readall; 3.98x faster #140139