Skip to content

\rs introduced after codec decoding cause SystemError #145234

Closed
Closed
\rs introduced after codec decoding cause SystemError#145234
Labels
interpreter-core(Objects, Python, Grammar, and Parser dirs)topic-parsertype-bugAn unexpected behavior, bug, or error

Description

@StanFromIreland
StanFromIreland
opened on Feb 25, 2026

Bug report

Bug description:

Found by OSS-Fuzz.

The testcase is:

#coding=U7+AA0''

resulting in:

$ ./python -c "import py_compile; py_compile.compile('testcase')"
Sorry: SystemError: Parser/string_parser.c:286: bad argument to internal function

I gather this is because +AA0 decodes to \r in UTF-7, however, _PyTokenizer_translate_newlines is called before the codec decoding so it slips by and blows up (at least it apologizes😆). I think it could be fixed by calling _PyTokenizer_translate_newlines again after decoding if a \r is introduced, WDYT?

cc @lysnikolaou @pablogsal

CPython versions tested on:

CPython main branch

Operating systems tested on:

No response

Linked PRs

Activity

  1. StanFromIreland
    added
    type-bugAn unexpected behavior, bug, or error
    interpreter-core(Objects, Python, Grammar, and Parser dirs)
    topic-parser
    on Feb 25, 2026
  2. gourijain029-del
    added a commit that references this issue on Feb 26, 2026

    pythongh-145234: Fix SystemError in parser when \r is introduced afte…

    ccf6915
  3. pablogsal
    added 4 commits that reference this issue on Feb 26, 2026

    pythongh-145234: Normalize decoded CR in string tokenizer

    d701ca0

    pythongh-145234: Add patch attribution in NEWS entry

    6545f1f

    pythongh-145234: keep tokenizer source pointer after UTF-8 translation

    cb60a16

    gh-145234: Normalize decoded CR in string tokenizer (#145281)

    98b1e51
  4. miss-islington
    added a commit that references this issue on Feb 27, 2026

    pythongh-145234: Normalize decoded CR in string tokenizer (pythonGH-1…

    0142d1c
  5. pablogsal
    added 3 commits that reference this issue on Feb 27, 2026

    [3.14] gh-145234: Normalize decoded CR in string tokenizer (GH-145281) (

    f04aefa

    [3.13] pythongh-145234: Normalize decoded CR in string tokenizer (pyt…

    cf5c995

    [3.13] gh-145234: Normalize decoded CR in string tokenizer (GH-145281) (

    2daece9
  6. pablogsal
    closed this as completedon Feb 28, 2026

  7. pablogsal commented on Feb 28, 2026

    @pablogsal
    pablogsal
    on Feb 28, 2026
    Member

    Thanks for the report @StanFromIreland !

  8. brijkapadia
    added a commit that references this issue on Feb 28, 2026

    pythongh-145234: Normalize decoded CR in string tokenizer (python#145281

    aca1c73
  9. ljfp
    added a commit that references this issue on Apr 25, 2026

    pythongh-145234: Normalize decoded CR in string tokenizer (python#145281

    90949fa
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    interpreter-core(Objects, Python, Grammar, and Parser dirs)topic-parsertype-bugAn unexpected behavior, bug, or error

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions