Repository navigation
AnyIO commit breaks Matplotlib display in separate windows #1235
Description
Activity
Probably because of external event loop integration.
enter_eventloopis only ever called ifkernel.eventloopis set at startup:ipykernel/ipykernel/kernelapp.py
Lines 732 to 736 in a7d66ae
If the
eventloopis set later, e.g. via%matplotlib qt, it is never called.Plus there are 7 uses of
kernel.shell_streamineventloops.pyand theshell_streamwas replaced withshell_portin the AnyIO changes. So evidently this code isn't covered by anyipykerneltests.I've been looking into this today, and it is definitely the case that the way the event loop integration expects to return control has assumptions about socket use that are not satisfied by the changes in how sockets are used.
In particular, the event loop integration pattern is that the event loops block and hold control until there is a message waiting to be received on the shell socket, at which point control is returned to the kernel. But with the new async architecture, there could be any number of events waiting to process, including the coroutines involved in actually handling a message.
It is not hard to test this, because this test:
@pytest.mark.parametrize("gui", [ "qt", "osx", ]) def test_gui_responsive(kernel_client, gui): if gui == 'osx': if sys.platform == 'darwin': pytest.importorskip("matplotlib") else: pytest.skip("osx test only on mac") if gui == 'qt' and len(qt_guis_avail) == 0: pytest.skip("No qt backend available") kc = kernel_client # enable gui msg_id, reply = execute(f"%gui {gui}", kc=kc) assert reply['status'] == 'ok' flush_channels(kc) # kernel should remain responsive (it doesn't) msg_id, reply = execute(f"print(1)", kc=kc, timeout=2) assert reply['status'] == 'ok' flush_channels(kc)
reproduces the hang.
I think the fix is probably to change the
advance_eventloopfunctions to do what the call appears to expect, which is to step the event loop once, never blocking for an extended period waiting for a specific wake event. This is far less efficient than what we had before, but I think is the only way to properly integrate the two full event loops that we have going now since there is no longer a simple, clear signal that there's something waiting to be processed in the kernel's loop.For example, removing the poll here and just returning control on every poll interval enables both the matplotlib window and the kernel to remain responsive with
%gui osx.This seems to stem from a general misunderstanding that we have
advance_eventloopfunctions, when what we actually have is "run eventloop forever until there is something to process in the kernel's loop" functions, and the socket poll we have for that is not an adequate condition for waking for async tasks. I'm not sure there's a way to ask the event loop if there's anything waiting to be processed?Removing the poll greatly simplifies what these eventloop integration functions need to do, though, because they no longer need to hook up any waking mechanism, they can simply run for a finite period and return control. This is simpler, but means that both eventloops block each other for predefined periods of time.
Some of these event loops like qt now have a mechanism for asyncio cooperation, which we might be able to use. Of course, that's much easier with asyncio directly than it is with anyio.
I'm wondering if we could use Trio's guest mode, if it was available in AnyIO.
With the commit before the AnyIO PR on the
mainbranch (830829f) and usingipython 8.23.0andmatplotlib 3.8.4, the displaying of Matplotlib plots in separate windows (e.g. withqtbackend) using jupyter works fine. With the AnyIO commit (772dfb8) onwards the plot windows are no longer displayed.Code to reproduce (I've been using
jupyter qtconsolebutjupyter lab, etc, all show the same):In
ipythonthe same example works fine.