some IMDReader streaming tests time out with imdclient 0.2.4

[orbeckst]

Expected behavior

All IMDReader streaming tests continue to pass, regardless of updates to the upstream dependency imdclient.

Actual behavior

Since imdclient 0.2.4, a number of test fail by hitting the 200s timeout on the tests (see below)

Code to reproduce the behavior

See e.g. PR #5423 , https://github.com/MDAnalysis/mdanalysis/actions/runs/29380161861/job/87743056098?pr=5423

FAILED testsuite/MDAnalysisTests/coordinates/test_imd.py::TestIMDReaderBaseAPI::test_transformations_iter - Failed: Timeout (>200.0s) from pytest-timeout.
FAILED testsuite/MDAnalysisTests/coordinates/test_imd.py::TestIMDReaderBaseAPI::test_iter_as_aux_highf - Failed: Timeout (>200.0s) from pytest-timeout.
FAILED testsuite/MDAnalysisTests/coordinates/test_imd.py::TestIMDReaderBaseAPI::test_iter - Failed: Timeout (>200.0s) from pytest-timeout.
FAILED testsuite/MDAnalysisTests/coordinates/test_imd.py::TestStreamIteration::test_iterate_step - Failed: Timeout (>200.0s) from pytest-timeout.
FAILED testsuite/MDAnalysisTests/coordinates/test_imd.py::TestStreamIteration::test_iterate_twice_sliced_raises_error - Failed: Timeout (>200.0s) from pytest-timeout.
FAILED testsuite/MDAnalysisTests/coordinates/test_imd.py::TestStreamIteration::test_iterate_twice_all_raises_error - Failed: Timeout (>200.0s) from pytest-timeout.
FAILED testsuite/MDAnalysisTests/coordinates/test_imd.py::TestStreamIteration::test_iterate_twice_fi_all_raises_error - Failed: Timeout (>200.0s) from pytest-timeout.
FAILED testsuite/MDAnalysisTests/coordinates/test_imd.py::TestStreamIteration::test_subslice_fi_all_after_iteration_raises_error - Failed: Timeout (>200.0s) from pytest-timeout.
FAILED testsuite/MDAnalysisTests/coordinates/test_imd.py::test_imd_stream_empty - Failed: Timeout (>200.0s) from pytest-timeout.

Background

imdclient 0.2.4 increased a timeout value from 5s to 600s, see issue Becksteinlab/imdclient#111 and PR Becksteinlab/imdclient#120

[orbeckst]

@amruthesht @ljwoods2 do you have an idea of what may lead to these failures?

[orbeckst]

I can confirm that locally the imd tests pass with imdclient 0.2.3 and also pass with 0.2.4 ( pytest -v test_imd.py) (on Python 3.14, macOS, develop MDA 3d3a53e) BUT they take a LONG time

It looks as if these 0.2.3 tests take about 5 seconds to pass. This is suspicious because the imdclient default timeout for 0.2.3 is 5s and the same tests now take a long time (after the imdclient timeout was increased to 600s).

I tried adding timeout=10 to

mdanalysis/testsuite/MDAnalysisTests/coordinates/test_imd.py

Line 213 in 3d3a53e

reader = ref.reader(

but this did not seem to make a difference.

[orbeckst]

Confirmed that with imdclient 0.2.4 one of the tests in question passes after the 600-s timeout value (=10 min):

$ time pytest -v -k test_transformations_iter testsuite/MDAnalysisTests/coordinates/test_imd.py
...
testsuite/MDAnalysisTests/coordinates/test_imd.py::TestIMDReaderBaseAPI::test_transformations_iter PASSED            [100%]
======================================= 1 passed, 70 deselected in 600.58s (0:10:00) =======================================

real	10m0.946s
user	0m0.693s
sys	0m0.147s

[orbeckst]

@amruthesht how do I enforce a short timeout for the tests? This is important and urgent to just get the CI working again.

We then need to understand why we need timeout for tests to pass — what kind of behavior in imdclient has IMDReader relying out for properly ending reading??

[ljwoods2]

@orbeckst I think the problem is that we never call disconnect or close on the dummy IMDServer's socket until after the test ends, during cleanup- so the only way the reader has of knowing the stream has ended during the test is via the timeout.

mdanalysis/testsuite/MDAnalysisTests/coordinates/test_imd.py

Line 438 in 3d3a53e

server.send_frames(1, 5)

So we need two changes: IMDServer needs to call socket.shutdown(socket.SHUTDOWN_WR) as a part of its disconnect() call, and the reader fixtures in these tests need to call server.disconnect() after sending frames through the socket

[ljwoods2]

@orbeckst

imdclient repo side
Becksteinlab/imdclient#149

mda side - can be merged WITHOUT above MR, since disconnect already calls socket.close() which will send EOF in the case of only one process holding the file descriptor to the senders socket, as is always currently true in our tests)
#5443

[orbeckst]

Thanks for quickly digging into this issue. I’m also reviewing the PR on the imdclient side. I might not fully understand how the server is supposed to work so please bear with my questions.