Pickle ignores custom getstate methods on TextIOWrapper in Python 3.12

Bug report

Bug description:

So I am not entirely sure whether this is unintended behaviour, but it is definitely a noticeable change between 3.11 and 3.12 that is rather unintuitive

import pickle
from io import BytesIO, TextIOWrapper

class EncodedFile(TextIOWrapper):
	def __getstate__(self):
		return "string"
	def __setstate__(self, state):
		pass

file = EncodedFile(BytesIO(b"string"))
pickle.dumps(file)

This works in Python 3.11 and 3.10, but fails in 3.12 with

pickle.dumps(file)
TypeError: cannot pickle 'EncodedFile' instances

CPython versions tested on:

3.10, 3.11, 3.12

Operating systems tested on:

macOS

Linked PRs

Seems like it was added in this PR: https://github.com/python/cpython/pull/101948/files#diff-f6d1380274f6859688c821fced58fb1c829d8acad44ae252ba988e220e1080b3R3240

Interesting! While it makes sense that the original IOBase can not be pickled, I feel like it should be possible for subclasses overriding getstate

So I can verify that overriding reduce and reduce_ex worked for me. I would leave the issue open for now as I do not think getstate and setstate not working is nice - especially since reduce and reduce_ex of IOBase are buried in the C code.

Please feel free to correct me if you think this has been resolved.

Pickling file objects was explicitly forbidden in bpo-10180 by adding __getstate__ methods that raise TypeError. These methods were removed from the C implementation in bpo-33138, because these classes were recognized as non-pickleable by default. Then these classes were made heap types in #101948 (gh-101819), and new __reduce__ and __reduce_ex__ methods that raise TypeError were added, because condition that prevented them from pickling was gone.

The right way is to remove __reduce__ and __reduce_ex__ methods and restore __getstate__ methods. This will make the C implementation consistent with the Python implementation.

It turned out that this issue is a bit more complex. There are more differences between implementations. Classes are not tested separately, existing tests do not cover all cases. I am working on this problem.

pythongh-122559: Synchronize C and Python implementation of the io mo…

I added tests as a guard against such type of regression, but in general there are no guaranties about pickleability (especially with protocols 0 and 1).

In meantime, the workaround is to reset __reduce__ and __reduce_ex__ in your subclass to default:

class EncodedFile(TextIOWrapper):
    ...
    __reduce__ = object.__reduce__
    __reduce_ex__ = object.__reduce_ex__

gh-122559: Synchronize C and Python implementation of the io module a…

[3.13] pythongh-122559: Synchronize C and Python implementation of th…

[3.13] gh-122559: Synchronize C and Python implementation of the io m…

Merge remote-tracking branch 'origin/main' into call-len

pythongh-122559: Synchronize C and Python implementation of the io mo…

Navigation Menu

Repository navigation

Pickle ignores custom getstate methods on TextIOWrapper in Python 3.12 #122559

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Linked PRs

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Pickle ignores custom getstate methods on TextIOWrapper in Python 3.12 #122559

Description

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Linked PRs

Activity

tomasr8 commented on Aug 1, 2024

SimonSorgQC commented on Aug 1, 2024

SimonSorgQC commented on Aug 1, 2024

serhiy-storchaka commented on Aug 2, 2024

serhiy-storchaka commented on Aug 2, 2024

serhiy-storchaka commented on Aug 2, 2024

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions