Repository navigation
Bytecode positions seem way too broad #93691
Description
(Note that dis currently has a bug in displaying accurate location info in the presence of CACHEs. The correct information can be observed by working with co_positions directly or using the code from that PR.)
While developing specialist, I realized that there are lots of common code patterns that produce bytecode with unexpectedly large source ranges. In addition to being unhelpful for both friendly tracebacks (the original motivation) and things like bytecode introspection, I suspect these huge ranges may also be bloating the size of our internal position tables as well.
Consider the following function:
def analyze(path): # 1
upper = lower = total = 0 # 2
with open(path) as file: # 3
for line in file: # 4
for character in line: # 5
if character.isupper(): # 6
upper += 1 # 7
elif character.islower(): # 8
lower += 1 # 9
total += 1 # 10
return lower / total, upper / total # 11
import dis
from pprint import pprint as pp
def pos(p):
return (p.lineno, p.end_lineno, p.col_offset, p.end_col_offset)
pp([(pos(x.positions), x.opname, x.argval) for x in dis.get_instructions(analyze)])Things that should probably span one line at most:
- The first
GET_ITER/FOR_ITERpair span all of lines 4 through 10. - The second
GET_ITER/FOR_ITERpair spans all of lines 5 through 10. - The first
POP_JUMP_FORWARD_IF_FALSEspans all of lines 6 through 9. - The second
POP_JUMP_FORWARD_IF_FALSEspans all of lines 8 through 9. - Ten instructions for
withcleanup each span all of lines 3 through 10.
Things that should probably be artificial:
- A
JUMP_FORWARDspans all of line 7. - The first
JUMP_BACKWARDspans all of line 10. - The second
JUMP_BACKWARDspans all of lines 5 through 10.
Things I don't get:
- A
NOPspans all of lines 4 through 10.
As a result, over half of the generated bytecode for this function claims to span line 9, for instance. Also not shown here: the instructions for building functions and classes have similarly huge spans.
I think this can be tightened up in the compiler by:
- Being more aggressive in calling
SET_LOCon child nodes. - Being more aggressive in calling
UNSET_LOCbefore unconditional jumps.
Linked PRs
- gh-93691: fix too broad source locations of with-statement instructions #120125
- gh-93691: fix too broad source locations of for statement iterators #120330
- [3.13] gh-93691: fix too broad source locations of for statement iterators (GH-120330) #120399
- [3.12] gh-93691: fix too broad source locations of for statement iterators (GH-120330 #120405
- [3.13] gh-93691: fix too broad source locations of with-statement instructions (GH-120125) #123604
- [3.12] gh-93691: fix too broad source locations of with-statement instructions (GH-120125) #123605
Activity
brandtbucher commentedon Jun 10, 2022
I can perhaps see why the argument could be made that we should have the location info for certain constructs span their entire block or jump range, but to me this just feels like the shape of the AST and the design of the compiler are leaking into the bytecode more than is really helpful in practice.
brandtbucher commentedon Jun 10, 2022
@pablogsal @markshannon @iritkatriel
brandtbucher commentedon Jun 10, 2022
As a quick-and-dirty experiment: for the given example function, setting
node.end_lineno = node.linenoon every node before compiling the AST resulted in a 20% reduction in the size ofco_linetable.pablogsal commentedon Jun 10, 2022
I don't think that many of these things were conscious decisions. Originally we added and enabled the infrastructure so the debug information could be propagated and we spent some time doing small optimisations, but we are missing a full pass over the compiler to fix things like this. Additionally, there are many instructions that don't really benefit from having position information because they are either artificial or don't map well to source code.
I think this is a very good find. With what seems like a small tedious amount of work we could reduce substantially the size for some functions, specially in block setup stuff.
Thanks for opening the issue and the insights @brandtbucher, this is very interesting indeed.
markshannon commentedon Jun 13, 2022
I don't think the shape of the AST is "leaking" into the bytecode. The AST defines the locations.
The problem is, IMO, in the design of the compiler. Tracking the "current" location in the compiler only makes sense if the bytecode is produced in a linear fashion, which it clearly isn't.
We should make the location used explicit when generating code, not use the implicit location stored in the compiler.
ADDOP(opcode, oparg,location)6 remaining items
pythongh-93691: fix too broad source locations of for statement itera…
[3.13] gh-93691: fix too broad source locations of for statement iter…
[3.12] pythongh-93691: fix too broad source locations of for statemen…
[3.12] gh-93691: fix too broad source locations of for statement iter…
pythongh-93691: fix too broad source locations of for statement itera…
pythongh-93691: fix too broad source locations of with-statement inst…
pythongh-93691: fix too broad source locations of for statement itera…
pythongh-93691: fix too broad source locations of with-statement inst…
pythongh-93691: fix too broad source locations of for statement itera…
pythongh-93691: fix too broad source locations of with-statement inst…
pythongh-93691: fix too broad source locations of with-statement inst…
[3.12] gh-93691: fix too broad source locations of with-statement ins…
[3.13] gh-93691: fix too broad source locations of with-statement ins…