Repository navigation
annotationlib: ref.evaluate(format=Format.FORWARDREF) returns a ForwardRef with a copied __globals__ that no longer updates #137969
Description
Bug report
Bug description:
This can happen internally in get_annotations and means subsequent attempts to evaluate it will fail even if the names have since been defined.
Underlying logic:
from annotationlib import get_annotations, Format
class Demo:
x: Sequence[undefined]
annos = get_annotations(Demo, format=Format.FORWARDREF)
x_anno = annos['x']
# Try to evaluate the reference, but just give a forwardref if it fails
x_repeat_anno = x_anno.evaluate(format=Format.FORWARDREF)
# The resulting reference no longer shares the globals namespace with the annotate function
print(f"{x_anno.__globals__ is Demo.__annotate__.__globals__ = }") # True
print(f"{x_repeat_anno.__globals__ is Demo.__annotate__.__globals__ = }") # False
# Define the previously undefined attributes
from collections.abc import Sequence
undefined = str
# This means evaluation fails in the second case
print(f"{x_anno.evaluate() = }") # collections.abc.Sequence[str]
print(f"{x_repeat_anno.evaluate() = }") # NameErrorThis evaluate call happens internally if get_annotations has to rely on the fallback behaviour for an unexpected exception, such as an AttributeError:
from annotationlib import get_annotations, Format
import typing
class Works:
a: Sequence[undefined]
b: unknowable
# Intentionally set up an annotation that will raise AttributeError on evaluation
class Fails:
a: Sequence[undefined]
b: typing.doesnotexist
a_works = get_annotations(Works, format=Format.FORWARDREF)['a']
a_fails = get_annotations(Fails, format=Format.FORWARDREF)['a']
# Realise the references
from collections.abc import Sequence
undefined = str
print(f"{a_works.evaluate() = }") # collections.abc.Sequence[str]
print(f"{a_fails.evaluate() = }") # NameErrorThis appears to be caused by the creation of a new globals dict here:
Lines 164 to 167 in e39255e
Commenting this out makes these examples succeed, but obviously breaks type parameters.
CPython versions tested on:
CPython main branch, 3.14
Operating systems tested on:
No response
Linked PRs
- gh-137969: Fix evaluation of
ref.evaluate(format=Format.FORWARDREF)objects #138075 - [3.14] gh-137969: Fix evaluation of
ref.evaluate(format=Format.FORWARDREF)objects (GH-138075) #140929 - Revert "gh-137969: Fix evaluation of
ref.evaluate(format=Format.FORWARDREF)objects (#138075)" #140930 - [3.14] Revert "gh-137969: Fix evaluation of
ref.evaluate(format=Format.FORWARDREF)objects (GH-138075) (#140929)" #140931 - gh-137969: Fix double evaluation of
ForwardRefs which rely on globals #140974 - [3.14] gh-137969: Fix double evaluation of
ForwardRefs which rely on globals (GH-140974) #141527
Activity
dr-carlos commentedon Aug 22, 2025
A simple (possibly naive?) fix for this is just to add a new dictionary to store type param values instead of copying the globals.
Then, a check against this dict is added where globals is accessed later in
evaluate(), logically between the locals and globals.See dr-carlos@f8ac2c0
This seemingly fixes the provided example and works with type parameters (as far as my simple tests have shown).
Happy to open a PR, just wanting to make sure I haven't missed the problem (or created a new one).
DavidCEllis commentedon Aug 22, 2025
Make sure you run the annotationlib tests with:
./python -m test -v test_annotationlibIt looks like your commit breaks the tests as you shadow the
typesmodule with your new variable which breaks the check a few lines up forisinstance(owner, types.ModuleType).I thought I'd tried putting the
type_paramsat the start of the locals and had test failures I didn't have time to investigate, but I think I had some logic in the wrong order.dr-carlos commentedon Aug 22, 2025
Thanks for the heads up re the test command!
I've renamed the variable to something much more sensible (forgot I was going to do that before posting earlier 🤦 )
All tests are now passing. A new test would possibly be warranted for this use case, but that should probably be discussed in some future PR.
JelleZijlstra commentedon Aug 22, 2025
That sounds reasonable, please send a PR and I'll take a look.
gh-137969: Fix evaluation of `ref.evaluate(format=Format.FORWARDREF)`…
pythongh-137969: Fix evaluation of `ref.evaluate(format=Format.FORWAR…
[3.14] gh-137969: Fix evaluation of `ref.evaluate(format=Format.FORWA…
Revert "pythongh-137969: Fix evaluation of `ref.evaluate(format=Forma…
Revert "[3.14] pythongh-137969: Fix evaluation of `ref.evaluate(forma…
Revert "gh-137969: Fix evaluation of `ref.evaluate(format=Format.FORW…
[3.14] Revert "gh-137969: Fix evaluation of `ref.evaluate(format=Form…
JelleZijlstra commentedon Nov 3, 2025
@dr-carlos your fix had to be reverted because of failures on some buildbots. Are you able to make another PR and try to fix the issues?
dr-carlos commentedon Nov 3, 2025
Yep, I'll take a look!
dr-carlos commentedon Nov 4, 2025
It seems that the issue was with the test:
test_re_evaluate_genericstests globals, so uses aglobaldeclaration, but this shadows a name used in other test functions' scopes.Anyway, I've made a PR which re-implements the fix now that #138164 is implemented and includse a working test.
It's a slightly different approach to last time, but I haven't found any issues with it so far, so let me know if you'd prefer it implemented with a separate dict instead of injecting into
locals.JelleZijlstra commentedon Nov 4, 2025
Thanks, I think the difference is that buildbots run the same test twice in the same process. Should be able to repro with something like
./python -m test test_annotationlib test_annotationlib.gh-137969: Fix double evaluation of `ForwardRef`s which rely on globa…
pythongh-137969: Fix double evaluation of `ForwardRef`s which rely on…
Merge branch 'main' into fix-issue21914
[3.14] gh-137969: Fix double evaluation of `ForwardRef`s which rely o…
Merge branch 'main' of github.com:python/cpython into fix-Issue18387
Merge branch 'main' of github.com:python/cpython into fix-issue24255
pythongh-137969: Fix evaluation of `ref.evaluate(format=Format.FORWAR…
Revert "pythongh-137969: Fix evaluation of `ref.evaluate(format=Forma…
pythongh-137969: Fix double evaluation of `ForwardRef`s which rely on…