Repository navigation
Abandoned StreamWriter isn't reliably detected #114914
Description
Bug report
Bug description:
A StreamWriter should be close():d when you are done with it. There is code in the destructor for StreamWriter to detect when this is overlooked and trigger a ResourceWarning. However, the current code often maintains a strong reference to the writer, preventing it from being garbage collected and hence no warning.
Test case:
#!/usr/bin/python3
import asyncio
import gc
import socket
async def handle_echo(reader, writer):
addr = writer.get_extra_info('peername')
print(f"Connection from {addr!r}")
# Forgetting to close the writer
#writer.close()
#await writer.wait_closed()
async def main():
server = await asyncio.start_server(
handle_echo, '127.0.0.1', 8888)
addrs = ', '.join(str(sock.getsockname()) for sock in server.sockets)
print(f'Serving on {addrs}')
client = socket.create_connection(('127.0.0.1', 8888))
client.send(b'a' * 64 * 1024)
async with server:
for i in range(25):
await asyncio.sleep(0.1)
gc.collect()
print('Exiting')
print('Done serving')
client.close()
asyncio.run(main())Test case locks up waiting for the client connection, when instead I would expect a ResourceWarning and everything exiting nicely.
CPython versions tested on:
3.12, CPython main branch
Operating systems tested on:
Linux
Activity
CendioOssman commentedon Feb 2, 2024
The issue is because
StreamReaderProtocolmaintains a strong reference to the writer. The protocol is in turn referenced by the transport, and the transport from the event loop (because it has calledadd_reader()).This chain isn't always active, though, so things sometimes work:
StreamReader, then the transport unregisters from the event loop and the whole group of objects can be garbage collected. This can easily be tested by changing the test case above to send 256 KiB of data instead of 64 KiB.create_server()is used, orstart_tls(). So a fairly standard outgoing connection will not be affected.CendioOssman commentedon Feb 2, 2024
The reference between the protocol and the writer is never actually used for anything, so my suggestion would be to just remove it:
The above change resolves the issue for me here.
gvanrossum commentedon Feb 19, 2024
Yeah, it does look like that
_stream_writerattribute is never used. And it was this way sincestart_server()was added, in 1540b16. This code is over a decade old, so I am willing to believe that it was never thought through carefully.Interestingly, the
ResourceWarningitself was only added 6 months ago (in gh-107650, and backported to 3.12 and 3.11), with a further refinement a few months later in gh-111983 (also backported).Also interesting, your example does not hang for me in 3.11! So what happened to streams.py between 3.11 and 3.12 that affects your example? I don't see anything that looks relevant in the diff for streams.py between 3.11 and 3.12, but the rest of asyncio changed extensively. Let me run a
git bisectcommand.gvanrossum commentedon Feb 19, 2024
And the winner is... gh-98582, the cursed fix to
wait_closed(). :-(In case you don't recall what happened in that PR, in 3.11 and before,
await server.wait_closed()after callingserver.close()was a no-op, due to a bug that made it return immediately if the server was closed. This was fixed, but broke code that had unclosed connections. Your example is one of those -- you're usingasync with serverand there's await_closed()call inAbstractServer.__aexit__().Anyway, I think we should just fix this (and probably backport the fix to 3.12 and 3.11) along the lines of your diff. Do you want to submit a PR?
CendioOssman commentedon Feb 19, 2024
Sure, let me give it a try.
pythongh-114914: Avoid keeping dead StreamWriter alive
gh-114914: Avoid keeping dead StreamWriter alive (#115661)
pythongh-114914: Avoid keeping dead StreamWriter alive (python#115661)
pythongh-114914: Avoid keeping dead StreamWriter alive (python#115661)
pythongh-114914: Avoid keeping dead StreamWriter alive (python#115661)
pythongh-114914: Avoid keeping dead StreamWriter alive (python#115661)
Remove redundant assignment in asyncio.streams.StreamReaderProtocol.c…
Remove redundant assignment in asyncio.streams.StreamReaderProtocol.c…