Support for sharing state between pathlib subclasses

Feature or enhancement

This enhancement proposes that we allow state to be shared between related instances of subclasses of pathlib.PurePath and pathlib.Path.

Pitch

Now that #68320 is resolved, users can subclass pathlib.PurePath and pathlib.Path directly:

import pathlib

class MyPath(pathlib.Path):
    def my_custom_method(self):
        pass

etc = MyPath('/etc')
etc_hosts = etc / 'hosts'
etc_hosts.my_custom_method()

However, some user implementations of classes - such as TarPath or S3Path - would require underlying tarfile.TarFile or botocore.Resource objects to be shared between path objects (etc and etc_hosts in the example above). Such sharing of resources is presently rather difficult, as there's no single instance method used to derive new path objects.

This feature request proposes that we add a new PurePath.makepath() method, which is called whenever one path object is derived from another, such as in joinpath(), iterdir(), etc. The default implementation of this method looks something like:

def makepath(self, *args):
    return type(self)(*args)

Users may redefine this method in a subclass, in conjunction with a customized initializer:

class SessionPath(pathlib.Path):
    def __init__(self, *args, session_id):
        super().__init__(*args)
        self.session_id = session_id

    def makepath(self, *args):
        return type(self)(*args, session_id=self.session_id)

etc = SessionPath('/etc', session_id=42)
etc_hosts = etc / 'hosts'
print(etc_hosts.session_id)  # 42

I propose the name "makepath" for this method due to its close relationship with the existing "joinpath" method: a.joinpath(b) == a.makepath(a, b).

Performance

edit: this change has been taken care of elsewhere, and so implementing this feature request should no longer have much affect on performance.

This change will affect the performance of some pathlib operations, because it requires us to remove the _from_parsed_parts() constructor, which is an internal optimization used in cases where path parsing and normalization can be skipped (for example, in the parents sequence). I suggest that, within the standard library, pathlib is not a particularly performance-sensitive module -- few folks reach to pathlib for reason of speed. Within pathlib itself, the savings from optimizing these "pure" methods are usually drowned out by the I/O costs of "impure" methods. With the appeal of this feature in mind, I believe the performance cost is justified.

However, if the performance degradation is considered unacceptable, there's a possible alternative: add a normalize keyword argument to the path initializer and to makepath(). This would require some serious internal surgery to make work, and might be difficult to communicate to users, so at this stage it's not my preferred route forward.

Previous discussion

https://discuss.python.org/t/make-pathlib-extensible/3428/47 (and replies)

Linked PRs

This will be needed to implement zipfile.Path using pathlib. Observe that it already has a _next() method that works exactly like what I'm proposing here:

cpython/Lib/zipfile/_path.py

Lines 296 to 297 in a68e585

    
           def _next(self, at): 
        
               return self.__class__(self.root, at)

Exactly how big would the performance regression be? Have you tried running some benchmarks?

Not yet; I'll do this before I mark the PR as 'ready' and share the numbers.

My instinct is to say that it would be a shame to make pathlib performance worse, since it's already got a reputation in some quarters for having surprisingly bad performance in some ways compared to os.path. But it's hard to evaluate without knowing how big the performance regression would be, and in what situations you'd hit the regression :)

few folks reach to pathlib for reason of speed.

In my opinion, we should be trying to fix this rather than exacerbating the problem :)

Absolutely fair!

Do you know of particular cases of (or complaints about) pathlib being slow? I'm aware that recursive globbing in pathlib can be slow, but not much else. We're increasingly calling os.path functions, some of which have C implementations, which should improve performance a little.

I'm of course keen for pathlib to be as fast as possible, but I don't want to prevent the future addition of an AbstractPath class either.

Do you know of particular cases of (or complaints about) pathlib being slow?

I have no idea how many of these complaints, if any, still apply to the current implementation of pathlib, nor how many of these are realistically resolvable :)

Mega, thank you. I'll study these carefully and write benchmarks.

Some initial thoughts:

The performance cost of constructing Path objects (vs constructing strings) stands out. This would surely benefit from an optimization pass, though it's unlikely we'll ever approach str construction performance.
The change to the parents implementation in GH-100479: Add optional blueprint argument to pathlib.PurePath #100481 may improve performance for some of these - I'll check.
A C implementation of pathlib would be wonderful but I suspect quite difficult! C implementations of select os.path functions would also help.
Caching stat() results would break user code, but there's a possible alternative: we could introduce something like a status() method that returns a new object with exists, is_dir, etc, attributes. Users could call path.status() once and check its attributes, rather than calling path.is_dir(), path.is_symlink(), etc, each of which would trigger their own stat() call.

Merge branch 'main' into pythongh-100479-add-makepath

@AlexWaygood

Hey @AlexWaygood - I've addressed the performance aspects of this elsewhere (#101664, #101665, #101667, #102789), so addressing this feature request is pretty much performance-neutral. Would you be up for reviewing #100481? Thank you!

I'll try to take a look soon!

Merge branch 'main' into pythongh-100479-add-makepath

Merge branch 'main' into pythongh-100479-add-makepath-redux

Merge branch 'main' into pythongh-100479-add-makepath

Merge branch 'main' into pythongh-100479-add-makepath-redux

GH-100479: Add `pathlib.PurePath.with_segments()` (GH-103975)

Implemented in 3.12: #103975 / d00d942

pythonGH-100479: Fix pathlib test failure on WASI

Merge branch 'main' into inlinecomp2

GH-100479: Fix pathlib test failure on WASI (#104215)

pythonGH-100479: Add `pathlib.PurePath.with_segments()` (pythonGH-103975

pythonGH-100479: Fix pathlib test failure on WASI (python#104215)

Navigation Menu

Repository navigation

Support for sharing state between pathlib subclasses #100479

Feature or enhancement

Pitch

Performance

Previous discussion

Linked PRs

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Support for sharing state between pathlib subclasses #100479

Description

Feature or enhancement

Pitch

Performance

Previous discussion

Linked PRs

Activity

barneygale commented on Dec 23, 2022

AlexWaygood commented on Dec 24, 2022

barneygale commented on Dec 24, 2022

AlexWaygood commented on Dec 24, 2022

barneygale commented on Dec 24, 2022

AlexWaygood commented on Dec 24, 2022

barneygale commented on Dec 24, 2022

barneygale commented on Apr 5, 2023

AlexWaygood commented on Apr 5, 2023

barneygale commented on May 5, 2023

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions