Repository navigation
Speed up array appends - #666
harshitgupta31415 wants to merge 1 commit into
Conversation
Wkhanex
left a comment
There was a problem hiding this comment.
AI-assisted validation at 90d15406608c on Python 3.12.14: I checked the fast path against tomllib after 250 appends in each of five parsed layouts (including a trailing comma and an inline comment), then a middle insert, extended-slice deletion, and a nested-value append. The serialized data matched the native expected lists in all five cases. A separate 1,000-append check made zero _reindex() calls, and the 12 tests in test_array_appends.py passed.
A boundary worth retaining in this optimization is the first mutation after the append sequence: it exercises whether the reused index map still points to the formatting groups after fallback operations. The mixed-mutation sequence above stayed synchronized; this is scoped validation of that transition, not a benchmark of overall array performance.
Summary
Closes #665.
Appending to an array currently rebuilds all its indexes, even though the existing values haven't moved.
This keeps those indexes and adds the new one for normal tail appends. Middle inserts, special formatting items, and states where the value list and index map have different lengths still use the old rebuild. The value conversion and formatting code are unchanged.
That makes the index work for N normal appends O(N) instead of O(N²). It reuses the existing map, so there isn't another cache to keep in memory.
I checked comments, whitespace, trailing commas, generators, self-extension, edits after appending, and invalid inputs against the old code. All 550 comparisons matched, including 60 errors and the state left after them. The full suite passed: 1,214 tests. The correctness tests also pass with the old method; only the new no-rebuild check needs this change.
Ruff passed. My local mypy 2.4.0 run reports 74 existing errors on both versions, with no new diagnostics. The repo pins an older mypy version, so I'm not counting this as a clean type-check run.
These timings include converting the data and writing the TOML. The file lists use names from Django's source archive. The wheel case uses the 65 wheel records from NumPy 2.5.4, in the same sort of array-building workflow used by PDM's array helper and pylock export. This is a constructed workflow, not a measurement of PDM itself.
The larger flat arrays improve clearly. Small arrays and wheel records are mixed, so I wouldn't claim this speeds up every document. All the results are below, including the slower ones.
The expanded run also covers generators, nested tables, edits to existing documents, and lots of separate small arrays. Middle/front inserts and parsing an unchanged document are included as controls.
Measurements: Python 3.15.0, Windows, Intel Core i5-13420H. Each row is the median of five paired timing samples on that same CPU pair. Short calls are batched. Only one process runs, allowed on two physical cores at a time, with the pairs rotated. The masks are kept separate because this CPU has different core types. Run order alternates on successive visits to each pair.
Peak figures are medians of five separate tracemalloc runs, not total process RAM. A separate memory run is shared by the 4 timing rows for a workload. Small timing differences on this laptop should not be treated as guaranteed gains or losses.
Base commit: 19edc23. Fixture loading is outside the warm measurements. The Django archive is version 6.1.2, SHA-256 a1e92451ccb8b514e91bbb3b6d186d20b4030558f116b5d9de6535455ff210b7.
All 26 workloads and 104 table rows are included, including slower results.
Full benchmark scripts, fixture details and raw samples are available here.
All complete TOML-writing benchmarks