[PR #13356/72eaa429 backport][3.14] Stop handing back pipelined requests the parser buffered

[patchback]

This is a backport of PR #13356 as merged into master (72eaa42).

What do these changes do?

When the pipelined message queue fills, the pure-Python parser buffers the unparsed remainder in self._tail and returns it (http_parser.py, queue-full break). The caller therefore holds a second copy of bytes the parser is already keeping, and the next feed_data() prepends the retained copy onto the returned one.

data_received() discards the returned tail unless the request upgraded, so ordinary pipelining never notices. finish_response() stores it unconditionally, and that is the path taken when a handler answers an Upgrade: request with a normal response. Pipelining more requests than the queue holds behind such a request then makes the server either serve them repeatedly or grow the buffer by a copy of itself per handled request.

Measured on this branch's parent, pure-Python parser, one upgrade request followed by 40 pipelined GETs in a single write:

	before	after
responses	connection never settles; handlers re-run indefinitely	41
distinct paths served	—	41
repeat dispatches	unbounded	0

The fix clears the returned buffer, which is exactly what the incomplete-request-line branch a few lines below already does. The C parser never returned the tail, so the duplication is pure-Python only, and the added parser test passes on the C backend before and after.

The second defect is not parser-specific. finish_response() appends to the message queue without the pause bookkeeping data_received() performs, so _msg_queue_paused stays False, start() never calls _resume_msg_queue_reading(), and nothing ever feeds the parser the remainder it kept. The requests past the queue limit simply go unanswered (33 of 41 above). This half affects the C parser too — arguably more cleanly, since before the fix the pure-Python duplicate kept _message_tail non-empty and so kept pumping. Reverting both source files and running the new functional test under the C extensions fails on its 10s timeout. Pausing in finish_response() lets the existing drain resume them.

Are there changes in behavior for the user?

Requests pipelined behind a declined upgrade past the queue limit are answered instead of being left buffered — that applies to both parsers. With the pure-Python parser they are additionally handled exactly once rather than repeatedly. No API change.

Is it a substantial burden for the maintainers to support this?

No. Five lines of parser change plus the queue-pause call that the neighbouring code path already makes, and both new tests fail without them.

Related issue number

None.

Checklist

• I think the code is well written
• Unit tests for the changes exist
• Documentation reflects the changes — N/A, no user-facing API change
• If you provide code modification, please add yourself to CONTRIBUTORS.txt — already listed
• Add a new news fragment into the CHANGES/ folder

Test run

Both parser backends, C extensions built (make cythonize + editable install):

$ pytest tests/test_http_parser.py tests/test_web_protocol.py \
         tests/test_web_functional.py tests/test_web_server.py
1015 passed, 6 deselected, 3 xfailed          # C extensions
599 passed, 35 skipped, 4 deselected          # AIOHTTP_NO_EXTENSIONS=1

Revert check, with the source changes reverted and the tests kept:

# pure-Python parser
FAILED tests/test_http_parser.py::test_max_msg_queue_size_keeps_tail_to_itself[py-parser]
# C extensions - the stalled-request half
FAILED tests/test_web_functional.py::test_http1_pipelined_behind_declined_upgrade_served_once
       (1 failed in 10.06s - the wait_for timeout, requests stranded)

Under AIOHTTP_NO_EXTENSIONS=1 that functional test does not fail cleanly when reverted: the repeated dispatch keeps the loop busy and the test's own timeout never fires. With the C extensions there is no repeated dispatch, so it fails on the timeout as shown above. It passes in well under a second either way with the change applied.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.26%. Comparing base (09bcaa7) to head (fb534c6).
✅ All tests successful. No failed tests found.

Additional details and impacted files

@@           Coverage Diff           @@
##             3.14   #13360   +/-   ##
=======================================
  Coverage   98.26%   98.26%           
=======================================
  Files         135      135           
  Lines       49184    49218   +34     
  Branches     2637     2639    +2     
=======================================
+ Hits        48329    48363   +34     
  Misses        675      675           
  Partials      180      180           

Flag	Coverage Δ
CI-GHA	98.30% <100.00%> (+<0.01%)	⬆️
OS-Linux	98.08% <100.00%> (+<0.01%)	⬆️
OS-Windows	95.76% <100.00%> (+<0.01%)	⬆️
OS-macOS	97.27% <100.00%> (+<0.01%)	⬆️
Py-3.10	97.47% <100.00%> (+<0.01%)	⬆️
Py-3.11	97.75% <100.00%> (+<0.01%)	⬆️
Py-3.12	97.82% <100.00%> (+<0.01%)	⬆️
Py-3.13	97.80% <100.00%> (-0.01%)	⬇️
Py-3.14	97.90% <100.00%> (-0.02%)	⬇️
Py-3.14t	96.90% <100.00%> (+<0.01%)	⬆️
Py-pypy-3.11	96.72% <100.00%> (+<0.01%)	⬆️
VM-macos	97.27% <100.00%> (+<0.01%)	⬆️
VM-ubuntu	98.08% <100.00%> (+<0.01%)	⬆️
VM-windows	95.76% <100.00%> (+<0.01%)	⬆️
cython-coverage	38.80% <100.00%> (+0.04%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

[codspeed-hq]

Merging this PR will not alter performance

✅ 72 untouched benchmarks
⏩ 7 skipped benchmarks¹

---

_{Comparing patchback/backports/3.14/72eaa429cf89b1e20212590f3e704444239569fb/pr-13356 (fb534c6) with 3.14 (09bcaa7)}

Footnotes

1. 7 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩