Repository navigation
Possible regression in Python3.12 tokenizer #105013
Copy link
Copy link
Closed
Labels
interpreter-core(Objects, Python, Grammar, and Parser dirs)(Objects, Python, Grammar, and Parser dirs)stdlibStandard Library Python modules in the Lib/ directoryStandard Library Python modules in the Lib/ directorytype-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error
Description
Bug report
I have encountered a possible regression while I was testing pyrsistent with python3.12_beta1, see tobgu/pyrsistent#275. Tests fail with following error:
/usr/lib/python3.12/site-packages/_pytest/python.py:617: in _importtestmodule
mod = import_path(self.path, mode=importmode, root=self.config.rootpath)
/usr/lib/python3.12/site-packages/_pytest/pathlib.py:564: in import_path
importlib.import_module(module_name)
/usr/lib/python3.12/importlib/__init__.py:90: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
<frozen importlib._bootstrap>:1293: in _gcd_import
???
<frozen importlib._bootstrap>:1266: in _find_and_load
???
<frozen importlib._bootstrap>:1237: in _find_and_load_unlocked
???
<frozen importlib._bootstrap>:841: in _load_unlocked
???
/usr/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:178: in exec_module
exec(co, module.__dict__)
tests/hypothesis_vector_test.py:48: in <module>
PVectorAndLists = st.lists(st.builds(RefCountTracker)).map(
/usr/lib/python3.12/site-packages/hypothesis/strategies/_internal/strategies.py:347: in map
if is_identity_function(pack):
/usr/lib/python3.12/site-packages/hypothesis/internal/reflection.py:638: in is_identity_function
return bool(re.fullmatch(r"lambda (\w+): \1", get_pretty_function_description(f)))
/usr/lib/python3.12/site-packages/hypothesis/internal/reflection.py:432: in get_pretty_function_description
return extract_lambda_source(f)
/usr/lib/python3.12/site-packages/hypothesis/internal/reflection.py:312: in extract_lambda_source
source = inspect.getsource(f)
/usr/lib/python3.12/inspect.py:1274: in getsource
lines, lnum = getsourcelines(object)
/usr/lib/python3.12/inspect.py:1266: in getsourcelines
return getblock(lines[lnum:]), lnum + 1
/usr/lib/python3.12/inspect.py:1241: in getblock
for _token in tokens:
/usr/lib/python3.12/tokenize.py:451: in _tokenize
for token in _generate_tokens_from_c_tokenizer(source, extra_tokens=True):
/usr/lib/python3.12/tokenize.py:542: in _generate_tokens_from_c_tokenizer
for info in c_tokenizer.TokenizerIter(source, extra_tokens=extra_tokens):
E File "<string>", line 1
E lambda l: (l, pvector(l)))
E ^
E SyntaxError: unmatched ')'
It is triggered for this code (see https://github.com/tobgu/pyrsistent/blob/cc90f3e2b339653fde0df422aaf3ccdb3fc1225d/tests/hypothesis_vector_test.py#L48-L49 )
PVectorAndLists = st.lists(st.builds(RefCountTracker)).map(
lambda l: (l, pvector(l)))if I concatenate it to one line, everything works. I also tried to backport fix for #104866 as it sounds relevant, but it didn't solved the issue.
Your environment
- Gentoo
~amd64 - Python 3.12.0b1
Linked PRs
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
interpreter-core(Objects, Python, Grammar, and Parser dirs)(Objects, Python, Grammar, and Parser dirs)stdlibStandard Library Python modules in the Lib/ directoryStandard Library Python modules in the Lib/ directorytype-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error
Activity
dev-python/hypothesis: add link to py3.12 upstream bug
arhadthedev commentedon May 27, 2023
The typo seems to be in pyrsistent itself.
https://github.com/tobgu/pyrsistent/blob/cc90f3e2b339653fde0df422aaf3ccdb3fc1225d/tests/hypothesis_vector_test.py#L49:
arkamar commentedon May 27, 2023
Sorry, I don't follow, where is the typo? Why this works
and this doesn't
btw, the wrapped version works in Python3.10 and Python3.11
arhadthedev commentedon May 27, 2023
Correction: the third parenthesis comes from the line above (found out preparing a PR for pyrsistent):
pablogsal commentedon May 27, 2023
This is not failing on
mainfor me:pablogsal commentedon May 27, 2023
Actually, both versions work with 3.12 (current):
This shows:
pablogsal commentedon May 27, 2023
Can you provide a reproducer that doesn't use any 3rd party stuff that fails with
mainor the tip of3.12?arkamar commentedon May 27, 2023
pyrsistent tests still don't work with the tip of
3.12. However, I was pointed out, that similar errors occurred in hypothesis tests. I dag into it and this should be the reproducer:The issue seems to be related to reflection and it prints out following error with Python3.12
but Python3.11 prints this:
arkamar commentedon May 27, 2023
or this one with just one
lambdaarkamar commentedon May 27, 2023
It works correctly if I concatenate it to one line
pablogsal commentedon May 27, 2023
The regression here is not on the
tokenizebut in theinspectmodule. The tokenize tokenizes the source as it does in 3.11:pythongh-105013: Fix inspect.getsource with parenthesized multiline l…
pablogsal commentedon May 27, 2023
@arkamar can you check if this fixes
pyrsistentwith #105021 ?fixup! pythongh-105013: Fix inspect.getsource with parenthesized mult…
pablogsal commentedon May 27, 2023
Seems that I can execute the test suite correctly with the fix installed:
fixup! pythongh-105013: Fix inspect.getsource with parenthesized mult…
fixup! fixup! pythongh-105013: Fix inspect.getsource with parenthesiz…
fixup! fixup! fixup! pythongh-105013: Fix inspect.getsource with pare…
gh-105013: Fix inspect.getsource with parenthesized multiline lambdas (…
pythongh-105013: Fix inspect.getsource with parenthesized multiline l…
pablogsal commentedon May 27, 2023
Fixed in #105021
mgorny commentedon May 28, 2023
I can confirm that this patch alone is sufficient to resolve the collection error in pyrsistent. Python segfaults while running the tests afterwards but FWICS this is another bug that's already fixed in the 3.12 branch.
arkamar commentedon May 28, 2023
@mgorny thanks for testing.
[3.12] gh-105013: Fix inspect.getsource with parenthesized multiline …