Skip to content

array: Add 'w' type and deprecate 'u' type. #80480

Closed
Closed
array: Add 'w' type and deprecate 'u' type.#80480
Labels
3.13only security fixesextension-modulesC modules in the Modules dirtype-featureA feature request or enhancement

Description

@methane
methane
opened on Mar 15, 2019
BPO 36299
Nosy @terryjreedy, @ncoghlan, @methane, @skrah, @serhiy-storchaka
PRs
  • bpo-36299: array('u') uses Py_UCS4 instead of Py_UNICODE  #12497
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = None
created_at = <Date 2019-03-15.05:50:02.657>
labels = ['3.8', 'library']
title = "array: Deprecate 'u' type in array module"
updated_at = <Date 2020-04-23.00:47:43.731>
user = 'https://github.com/methane'

    bugs.python.org fields:

    activity = <Date 2020-04-23.00:47:43.731>
actor = 'methane'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2019-03-15.05:50:02.657>
creator = 'methane'
dependencies = []
files = []
hgrepos = []
issue_num = 36299
keywords = ['patch']
message_count = 12.0
messages = ['337967', '338031', '338595', '338598', '338607', '338608', '338609', '338610', '338611', '367000', '367044', '367065']
nosy_count = 5.0
nosy_names = ['terry.reedy', 'ncoghlan', 'methane', 'skrah', 'serhiy.storchaka']
pr_nums = ['12497']
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue36299'
versions = ['Python 3.8']

    Linked PRs

    Activity

    1. methane commented on Mar 15, 2019

      @methane
      methane
      on Mar 15, 2019
      MemberAuthor

      The doc says:

      'u' will be removed together with the rest of the Py_UNICODE API.
      Deprecated since version 3.3, will be removed in version 4.0.
      https://docs.python.org/3/library/array.html

      But DeprecationWarning is not raised yet. Let's raise it.

      • 3.8 -- PendingDeprecationWarning
      • 3.9 -- DeprecationWarning
      • 4.0 or 3.10 -- Remove it.
    2. methane
      added
      3.8 (EOL)end of life
      stdlibStandard Library Python modules in the Lib/ directory
      on Mar 15, 2019

    3. terryjreedy commented on Mar 15, 2019

      @terryjreedy
      terryjreedy
      on Mar 15, 2019
      Member

      '4.0' is a stand-in for 'sometime after 2.7.final', scheduled for Jan 2020. A Pending... for 3.8.0, scheduled for Oct 2019, seems reasonable to me. Perhaps we should have a pydev discussion for the general issue of post 2.7 removals of already deprecated items.

    4. methane commented on Mar 22, 2019

      @methane
      methane
      on Mar 22, 2019
      MemberAuthor

      https://mail.python.org/pipermail/python-dev/2019-March/156807.html

      We may able to convert 'u' to wchar_t to int32_t and un-deprecate it.

    5. methane commented on Mar 22, 2019

      @methane
      methane
      on Mar 22, 2019
      MemberAuthor

      I found converting Py_UNICODE to Py_UCS4 wad happened, and reverted.
      ref: https://bugs.python.org/issue13072

    6. methane
      changed the title [-]Deprecate 'u' type in array module[/-] [+]array: Deprecate 'u' type in array module[/+] on Mar 22, 2019

    7. skrah commented on Mar 22, 2019

      @skrah
      skrahmannequin
      on Mar 22, 2019
      Mannequin

      I think the problem is still whether to use 'u' == UCS2 and 'w' == UCS4 like in PEP-3118.

      For the project I'm currently working on I'd need these for buffer exports:

      >>> from xnd import *
>>> x = xnd(["abc", "xyz"], dtype="fixed_string(10, 'utf16')")
>>> y = xnd(["abc", "xyz"], dtype="fixed_string(10, 'utf32')")
>>> 
>>> memoryview(x)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: type is not supported by the buffer protocol

      The use case is not an array that represents a single utf16 string, but
      an array *of* fixed strings with different encodings.

      So x would be exported with format 'u' and y with format 'w'.

    8. skrah commented on Mar 22, 2019

      @skrah
      skrahmannequin
      on Mar 22, 2019
      Mannequin

      Just to demonstrate what the format would look like, this is working
      for an array of fixed bytes:

      >>> x = xnd([b"123", b"23456"], dtype="fixed_bytes(size=10)")
>>> memoryview(x).format
'10s'

      So the formats in the previous message would be '10u' and '10w'.

    9. serhiy-storchaka commented on Mar 22, 2019

      @serhiy-storchaka
      serhiy-storchaka
      on Mar 22, 2019
      Member

      array('u') is not tied with the legacy Unicode C API. It is possible to use the modern wchar_t based Unicode C API for it. See bpo-36346.

      There are benefits from getting rid of the legacy Unicode C API, but not from array('u').

    10. skrah commented on Mar 22, 2019

      @skrah
      skrahmannequin
      on Mar 22, 2019
      Mannequin

      array() uses struct module characters except for 'u'. PEP-3118 was
      supposed to be implemented in the struct module.

      If array() continues to use 'u', the only sensible thing would be
      to remove (or rename) 'a', 'u' and 'w' from PEP-3118.

    11. skrah commented on Mar 22, 2019

      @skrah
      skrahmannequin
      on Mar 22, 2019
      Mannequin

      The funny thing is that array() already knows this:

      >>> import array
>>> a = array.array("u", "123")
>>> memoryview(a).format
'w'

    12. methane commented on Apr 22, 2020

      @methane
      methane
      on Apr 22, 2020
      MemberAuthor

      I closed #56706 (Py_UNICODE -> Py_UCS4).
      I created #63852 (Py_UNICODE -> wchar_t) instead.

    13. terryjreedy commented on Apr 22, 2020

      @terryjreedy
      terryjreedy
      on Apr 22, 2020
      Member

      Should this issue be closed, possibly as superseded by bpo-36346, the issue for the new PR-19653?

    14. methane commented on Apr 23, 2020

      @methane
      methane
      on Apr 23, 2020
      MemberAuthor

      While array('u') doesn't use deprecated API with #63852, I still don't like 'u' because:

      • I don't have any reason to use platform dependant wchar_t. 1
      • It is not consistent with PEP-3118.

      How about this plan?

      • Add 'w' for Py_UCS4.
      • Deprecate 'u', and remove it in the future.

    15. 20 remaining items

    16. encukou commented on May 9, 2023

      @encukou
      encukou
      on May 9, 2023
      Member

      How about:

      • Add 'w' for Py_UCS4.
      • Wait until all Python versions that don't have w reach end of life
      • Deprecate 'u', and remove it in the future.

      Is there a reason to rush this?

    17. methane commented on May 10, 2023

      @methane
      methane
      on May 10, 2023
      MemberAuthor

      array('u') will cause bugs and almost no valid use case for it.
      So waiting deprecation 5+ years seems too slow.

      On the other hand, I don't hurry about removing it.
      I can wait 3+ releases instead of minimum 2 releases.

      My plan is:

      • Add 'w' and deprecate 'u' in Python 3.13
      • Remove 'u' in Python 3.16+ (Postpone removal if some users still use 'u').
    18. methane
      changed the title [-]array: Deprecate 'u' type in array module[/-] [+]array: Add 'w' type and deprecate 'u' type.[/+] on May 10, 2023
    19. methane
      added a commit that references this issue on Jun 4, 2023

      gh-80480: array: Add 'w' typecode. (#105242)

      1237fb6

    20. vstinner commented on Jun 6, 2023

      @vstinner
      vstinner
      on Jun 6, 2023
      Member

      @methane added array.array('w') format to Python 3.13.

      @methane: Would you mind to also document the addition with versionchanged in https://docs.python.org/dev/library/array.html ? Currently, it's only documented at: https://docs.python.org/dev/whatsnew/3.13.html#array

    21. vstinner commented on Jun 6, 2023

      @vstinner
      vstinner
      on Jun 6, 2023
      Member

      I dislike the fact that array.array('u') is only deprecated in the doc. I would prefer to either emit a DeprecationWarning at runtime, or remove the deprecation. See also issue #105373. I'm fine with really deprecating it if someone wants to submit a PR (@methane ?).

    22. hugovk commented on Jun 6, 2023

      @hugovk
      hugovk
      on Jun 6, 2023
      Member

      @vstinner PR already open at #95760 :)

    23. hugovk
      added a commit that references this issue on Jun 11, 2023

      gh-80480: Emit DeprecationWarning for array's 'u' type code (#95760)

      cc87948

    24. hugovk commented on Jun 11, 2023

      @hugovk
      hugovk
      on Jun 11, 2023
      Member

      array('u') will cause bugs and almost no valid use case for it. So waiting deprecation 5+ years seems too slow.

      On the other hand, I don't hurry about removing it. I can wait 3+ releases instead of minimum 2 releases.

      My plan is:

      • Add 'w' and deprecate 'u' in Python 3.13

      Done in #105242 and #95760.

      • Remove 'u' in Python 3.16+ (Postpone removal if some users still use 'u').

      Let's close this issue and come back to this in four years.

    25. hugovk
      closed this as completedon Jun 11, 2023
    26. skirpichev
      added a commit that references this issue on May 8, 2026

      pythongh-80480: remove deprecated 'u' type code (array)

      2009a2d
    27. vstinner
      added a commit that references this issue on Jun 1, 2026

      gh-80480: Remove deprecated 'u' array type code (#149535)

      46b5e3e

    28. vstinner commented on Jun 1, 2026

      @vstinner
      vstinner
      on Jun 1, 2026
      Member

      The array u type has been removed by the PR gh-149535.

    29. serhiy-storchaka
      added this to Struct, memoryview and array issues 🏗️on Jul 13, 2026
    30. github-project-automation
      moved this to Done in Struct, memoryview and array issues 🏗️on Jul 13, 2026
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      3.13only security fixesextension-modulesC modules in the Modules dirtype-featureA feature request or enhancement

      Projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions