Skip to content

Change shutil.rmtree and os.walk to support very deep hierarchies #89727

Open
Open
Change shutil.rmtree and os.walk to support very deep hierarchies#89727
Labels
3.13only security fixesstdlibStandard Library Python modules in the Lib/ directorytype-featureA feature request or enhancement

Description

@AlexanderPatrakov
AlexanderPatrakov
mannequin
opened on Oct 21, 2021
BPO 45564

Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2021-10-21.22:21:02.146>
labels = ['library', '3.9', 'type-crash']
title = 'shutil.rmtree and os.walk are implemented using recursion, fail on deep hierarchies'
updated_at = <Date 2021-10-21.22:21:02.146>
user = 'https://bugs.python.org/AlexanderPatrakov'

bugs.python.org fields:

activity = <Date 2021-10-21.22:21:02.146>
actor = 'Alexander.Patrakov'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2021-10-21.22:21:02.146>
creator = 'Alexander.Patrakov'
dependencies = []
files = []
hgrepos = []
issue_num = 45564
keywords = []
message_count = 1.0
messages = ['404687']
nosy_count = 1.0
nosy_names = ['Alexander.Patrakov']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue45564'
versions = ['Python 3.9']

Linked PRs

Activity

  1. AlexanderPatrakov commented on Oct 21, 2021

    @AlexanderPatrakov
    AlexanderPatrakovmannequin
    on Oct 21, 2021
    MannequinAuthor

    It is possible to create deep directory hierarchies that cannot be removed via shutil.rmtree or walked via os.walk, because these functions exceed the interpreter recursion limit. This may have security implications for web services (e.g. various webdisks) that have to clean up user-created mess or walk through it.

    [aep@aep-haswell ~]$ mkdir /tmp/badstuff
[aep@aep-haswell ~]$ cd /tmp/badstuff
[aep@aep-haswell badstuff]$ for x in `seq 2048` ; do mkdir $x ; cd $x ; done
[aep@aep-haswell 103]$ cd
[aep@aep-haswell ~]$ python
Python 3.9.7 (default, Oct 10 2021, 15:13:22) 
[GCC 11.1.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import shutil
>>> shutil.rmtree('/tmp/badstuff')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.9/shutil.py", line 726, in rmtree
    _rmtree_safe_fd(fd, path, onerror)
  File "/usr/lib/python3.9/shutil.py", line 663, in _rmtree_safe_fd
    _rmtree_safe_fd(dirfd, fullname, onerror)
  File "/usr/lib/python3.9/shutil.py", line 663, in _rmtree_safe_fd
    _rmtree_safe_fd(dirfd, fullname, onerror)
  File "/usr/lib/python3.9/shutil.py", line 663, in _rmtree_safe_fd
    _rmtree_safe_fd(dirfd, fullname, onerror)
  [Previous line repeated 992 more times]
  File "/usr/lib/python3.9/shutil.py", line 642, in _rmtree_safe_fd
    fullname = os.path.join(path, entry.name)
  File "/usr/lib/python3.9/posixpath.py", line 77, in join
    sep = _get_sep(a)
  File "/usr/lib/python3.9/posixpath.py", line 42, in _get_sep
    if isinstance(path, bytes):
RecursionError: maximum recursion depth exceeded while calling a Python object
>>> import os
>>> list(os.walk('/tmp/badstuff'))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.9/os.py", line 418, in _walk
    yield from _walk(new_path, topdown, onerror, followlinks)
  File "/usr/lib/python3.9/os.py", line 418, in _walk
    yield from _walk(new_path, topdown, onerror, followlinks)
  File "/usr/lib/python3.9/os.py", line 418, in _walk
    yield from _walk(new_path, topdown, onerror, followlinks)
  [Previous line repeated 993 more times]
  File "/usr/lib/python3.9/os.py", line 412, in _walk
    new_path = join(top, dirname)
  File "/usr/lib/python3.9/posixpath.py", line 77, in join
    sep = _get_sep(a)
  File "/usr/lib/python3.9/posixpath.py", line 42, in _get_sep
    if isinstance(path, bytes):
RecursionError: maximum recursion depth exceeded while calling a Python object
>>>
  2. AlexanderPatrakov
    added
    3.9 (EOL)end of life
    stdlibStandard Library Python modules in the Lib/ directory
    type-crashA hard crash of the interpreter, possibly with a core dump
    on Oct 21, 2021
  3. ezio-melotti
    transferred this issue fromon Apr 10, 2022
  4. AlexWaygood
    added
    type-bugAn unexpected behavior, bug, or error
    and removed
    type-crashA hard crash of the interpreter, possibly with a core dump
    on Jul 10, 2022
  5. jonburdo
    added 2 commits that reference this issue on Nov 26, 2022

    pythongh-89727: Fix os.walk RecursionError on deep trees

    d106f5c

    pythongh-89727: Fix os.walk RecursionError on deep trees

    eeca07b

  6. barneygale commented on Nov 27, 2022

    @barneygale
    barneygale
    on Nov 27, 2022
    Contributor

    This also affects pathlib.Path.walk()

  7. jonburdo
    added a commit that references this issue on Nov 30, 2022

    pythongh-89727: Fix os.walk RecursionError on deep trees

    694c71e
  8. jonburdo
    added a commit that references this issue on Dec 15, 2022

    pythongh-89727: Fix os.walk RecursionError on deep trees

    99f4691
  9. zmievsa
    added a commit that references this issue on Dec 16, 2022

    Merge branch 'main' into pythongh-89727/fix-pathlib.Path.walk-recursi…

    7f36140

  10. 101 remaining items

  11. barneygale
    added a commit that references this issue on Jun 1, 2024

    GH-89727: Fix `shutil.rmtree()` recursion error on deep trees (#119808)

    53b1981
  12. miss-islington
    added a commit that references this issue on Jun 1, 2024

    pythonGH-89727: Fix `shutil.rmtree()` recursion error on deep trees (p…

    136d027
  13. barneygale
    added 5 commits that reference this issue on Jun 1, 2024

    pythonGH-89727: Fix `shutil.rmtree()` recursion error on deep trees (p…

    b900fdb

    [3.12] pythonGH-89727: Fix `shutil.rmtree()` recursion error on deep …

    411a75d

    [3.13] GH-89727: Fix `shutil.rmtree()` recursion error on deep trees (G…

    225c3cd

    [3.12] GH-89727: Fix `shutil.rmtree()` recursion error on deep trees (G…

    feb7870

    pythonGH-89727: Fix `shutil.rmtree()` recursion error on deep trees (p…

    39998dd
  14. noahbkim
    added 4 commits that reference this issue on Jul 11, 2024

    pythonGH-89727: Partially fix `shutil.rmtree()` recursion error on de…

    39dffed

    pythonGH-89727: Fix `os.fwalk()` recursion error on deep trees (pytho…

    1d802ba

    pythonGH-89727: Fix FD leak on `os.fwalk()` generator finalization. (p…

    a983848

    pythonGH-89727: Fix `shutil.rmtree()` recursion error on deep trees (p…

    e2e608a
  15. estyxx
    added 4 commits that reference this issue on Jul 17, 2024

    pythonGH-89727: Partially fix `shutil.rmtree()` recursion error on de…

    459d4f3

    pythonGH-89727: Fix `os.fwalk()` recursion error on deep trees (pytho…

    6bb9185

    pythonGH-89727: Fix FD leak on `os.fwalk()` generator finalization. (p…

    984d064

    pythonGH-89727: Fix `shutil.rmtree()` recursion error on deep trees (p…

    24a1a10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    3.13only security fixesstdlibStandard Library Python modules in the Lib/ directorytype-featureA feature request or enhancement

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions