Repository navigation
[C API] Change PyUnicode_AsUTF8() to return NULL on embedded null characters #111089
Description
I propose to change the PyUnicode_AsUTF8() API to raise an exception and return NULL if the string contains embedded null characters.
If the string contains an embedded null character, the UTF-8 encoded string can be truncated if used with C functions using char* since a null byte is treated as the terminator: marker of the string end. Truncating a string silently is a bad practice and can lead to different bugs including security vulnerabilities.
In practice, the minority of impacted C extensions and impacted users should benefit of such backward incompatible change, since truncating a string silently is a bad practice. Impacted users can use PyUnicode_AsUTF8AndSize(obj, NULL) and just ignore the size if they want to truncate on purpose.
It would address the following "hidden" comment on PyUnicode_AsUTF8():
Use of this API is DEPRECATED since no size information can be
extracted from the returned data.
PyUnicode_AsUTF8String() is part of the limited C API, whereas PyUnicode_AsUTF8() is not.
In the recently added PyUnicode_EqualToUTF8(obj, str), str is treated as not equal if obj contains embedded null characters.
The folllowing functions already raise an exception if the string contains embedded null characters or bytes:
- PyUnicode_AsWideCharString()
- PyUnicode_EncodeLocale()
- PyUnicode_EncodeFSDefault()
- PyUnicode_DecodeLocale(), PyUnicode_DecodeLocaleAndSize()
- PyUnicode_DecodeFSDefaultAndSize()
- PyUnicode_FSConverter()
- PyUnicode_FSDecoder()
PyUnicode_AsUTF8String() returns a bytes object and so the length, so it doesn't raise the exception.
PyUnicode_AsUTF8AndSize() also returns the size and so don't raise on embedded null characters.
Linked PRs
- gh-111089: PyUnicode_AsUTF8() now raises on embedded NUL #111091
- gh-111089: PyUnicode_AsUTF8AndSize() sets size on error #111106
- gh-111089: Add PyUnicode_AsUTF8() to the limited C API #111121
- gh-111089: Use PyUnicode_AsUTF8() in sqlite3 #111122
- gh-111089: Use PyUnicode_AsUTF8() in Argument Clinic #111585
- gh-111089: Add cache to PyUnicode_AsUTF8() for embedded NUL #111587
- gh-111089: Use PyUnicode_AsUTF8() in getargs.c #111620
- gh-111089: Add PyUnicode_AsUTF8Unsafe() function #111672
- gh-111089: Add PyUnicode_AsUTF8NoNUL() function #111688
- gh-111089: Revert PyUnicode_AsUTF8() changes #111833
Activity
pythongh-111089: PyUnicode_AsUTF8() now raises on embedded NUL
pythongh-111089: PyUnicode_AsUTF8() now raises on embedded NUL
pythongh-111089: PyUnicode_AsUTF8AndSize() sets size on error
pythongh-111089: PyUnicode_AsUTF8() now raises on embedded NUL
pythongh-111089: PyUnicode_AsUTF8AndSize() sets size on error
gh-111089: PyUnicode_AsUTF8() now raises on embedded NUL (#111091)
pythongh-111089: Add PyUnicode_AsUTF8() to the limited C API
pythongh-111089: Use PyUnicode_AsUTF8() in sqlite3
gh-111089: Add PyUnicode_AsUTF8() to the limited C API (#111121)
gh-111089: PyUnicode_AsUTF8AndSize() sets size on error (#111106)
gh-111089: Use PyUnicode_AsUTF8() in sqlite3 (#111122)
vstinner commentedon Oct 20, 2023
I added PyUnicode_AsUTF8() to the limited C API and it now raises an exception if the string contains embedded null characters.
48 remaining items
encukou commentedon Nov 8, 2023
Thank you for the revert. Sorry that the issues were only discovered after merge, and after such heated discussion. Maybe there's some process improvement to be made.
I filed this issue for the C-API-WG-to-be: capi-workgroup/api-evolution#39
Feel free to re-frame the issue or propose other solutions.
pythongh-111089: PyUnicode_AsUTF8() now raises on embedded NUL (pytho…
pythongh-111089: Add PyUnicode_AsUTF8() to the limited C API (python#…
pythongh-111089: PyUnicode_AsUTF8AndSize() sets size on error (python…
pythongh-111089: Use PyUnicode_AsUTF8() in sqlite3 (python#111122)
pythongh-111089: Use PyUnicode_AsUTF8() in Argument Clinic (python#11…
pythongh-111089: Use PyUnicode_AsUTF8() in getargs.c (python#111620)
pythongh-111089: Revert PyUnicode_AsUTF8() changes (python#111833)
pythongh-111089: PyUnicode_AsUTF8() now raises on embedded NUL (pytho…
pythongh-111089: Add PyUnicode_AsUTF8() to the limited C API (python#…
pythongh-111089: PyUnicode_AsUTF8AndSize() sets size on error (python…
pythongh-111089: Use PyUnicode_AsUTF8() in sqlite3 (python#111122)
pythongh-111089: Use PyUnicode_AsUTF8() in Argument Clinic (python#11…
pythongh-111089: Use PyUnicode_AsUTF8() in getargs.c (python#111620)
pythongh-111089: Revert PyUnicode_AsUTF8() changes (python#111833)