Repository navigation
TypedDict: __required_keys__ and __optional_keys__ can be wrong in the presence of inheritance #112509
Description
Bug report
Bug description:
from typing import NotRequired, Required, TypedDict
class A(TypedDict):
a: NotRequired[int]
b: Required[int]
class B(A):
a: Required[int]
b: NotRequired[int]
print(B.__required_keys__)
print(B.__optional_keys__)This will print (tried on 3.11 and current-ish main):
frozenset({'b', 'a'})
frozenset({'b', 'a'})
But obviously, a single key should only be either Required or NotRequired, not both. The child class's version should prevail.
@alicederyn and I discovered this while discussing the implementation of PEP-705 (python/typing_extensions#284). cc @davidfstr for PEP 655.
I also see some problems with how type checkers handle this case, but i'll post about that separately.
CPython versions tested on:
3.11, CPython main branch
Operating systems tested on:
macOS
Linked PRs
- gh-112509: Fix keys being present in both required_keys and optional_keys in TypedDict #112512
- [3.12] gh-112509: Fix keys being present in both required_keys and optional_keys in TypedDict (GH-112512) #112530
- [3.11] gh-112509: Fix keys being present in both required_keys and optional_keys in TypedDict (GH-112512) #112531
Activity
pythongh-112509: Fix keys being present in both required_keys and opt…
alicederyn commentedon Nov 29, 2023
This example is illegal according to PEP-589.
Concretely, B cannot be a subclass of A as it "removes" the ability to delete key 'b' (breaking substitutability) and also adds None to the possible values returned by
get('a')(also breaking substitutability).JelleZijlstra commentedon Nov 29, 2023
@alicederyn you may be right that this is illegal at type checking time; I posted python/typing#1516 about this. But as Alex pointed out on the PR, backward compatibility means we can't just prohibit this at runtime. PEP 705 is actually a good illustration of why it's good to be lenient at runtime: it creates a mechanism that does let you combine NotRequired and Required for the same key in a type-safe manner. If we had prohibited that at runtime, it would make it harder for people to use PEP 705 on older Python versions.
gh-112509: Fix keys being present in both required_keys and optional_…
pythongh-112509: Fix keys being present in both required_keys and opt…
pythongh-112509: Fix keys being present in both required_keys and opt…
[3.11] gh-112509: Fix keys being present in both required_keys and op…
[3.12] gh-112509: Fix keys being present in both required_keys and op…
alicederyn commentedon Nov 29, 2023
I wasn't intending to suggest prohibiting it at runtime -- just that changing the runtime behaviour of an unsupported construction seems unnecessary. I don't think there's a downside to changing it per se, but I don't think the previous behaviour was a bug.
Using PEP-705 with a regular TypedDict instead of importing it from typing_extensions will give the wrong behaviour at runtime anyway (the type qualifier will incorrectly end up mixed with the types at best, at worst it will accidentally break which keys are declared as required/not required, depending on what order you use the qualifiers in), so I don't think this change will benefit anyone there.
AlexWaygood commentedon Nov 29, 2023
This is fixed at runtime now, and backported to 3.11+, so I'll close this out now
JelleZijlstra commentedon Nov 30, 2023
@alicederyn that's a fair point, but I think the behavior here (the same key being in both required and optional keys) is quite clearly buggy; there is room for disagreement (or future spec changes) in the type checker behavior; and the fix was simple.
pythongh-112509: Fix keys being present in both required_keys and opt…
pythongh-112509: Fix keys being present in both required_keys and opt…