MT-7273: Optimize Message parsing, copying and encoding for large messages

[sipho-sarao]

MT-7273: Optimize Message parsing, copying and encoding for wide messages

Problem

A 4096-channel CBF+ ?gain request yields a ~60 KiB KATCP line with ~4100 arguments. Processing these large messages hits noticeable CPU bottlenecks:

• MessageParser.parse: ~7.5 ms
• Message.copy(): ~3.3 ms (re-runs format_argument on already-bytes args)
• Message.__bytes__: ~1.8 ms

For the CBF+ proxy, this cost is incurred on every proxied gain message and dominates request latency investigated in this branch.

Changes (katcp/core.py)

1. MessageParser.parse - If a line contains no backslash, skip per-argument UNESCAPE_RE.sub / SPECIAL_RE.search calls and validate via a single scan over the argument region (excluding space/tab delimiters). Construct the result via a new Message._from_parsed() helper to bypass __init__'s re-validation loops.
2. Message.copy - Use direct slot assignment + list() instead of re-formatting every argument. Preserves existing behavior (does not copy mid).
3. Message.__bytes__ - Avoid regex substitution overhead on unescaped args by checking ESCAPE_RE.search first, and combine empty-argument formatting into the loop pass..

Non-functional changes in this PR:

• tox.ini: Pins coverage<5 (CI fix - build node lacks _sqlite3 needed by coverage ≥5; 4.x is the last major supporting Python 2.7).
• bench/: Added the verification and benchmark scripts described below. These are self-contained scripts to reproduce the performance and equivalence claims of the large-message optimisation, added for reviewers and future regression checks. They locate the repo automatically (or via the KATCP_REPO env var) and can be run on Python 2.7 and 3.

Verification & Benchmarks

• Tests: Core + BNF grammar test suites pass on Python 2.7 (62 tests).
• Fuzzing: 300,000-line differential fuzz test against master (covering heavy escapes, bad names/mids, and whitespace variations). Produced identical outputs, exception types, and error messages.
• Micro-benchmarks (Py2.7 @ 4096 args):
  • Parse: 7.47 ms --> 1.76 ms
  • Copy: 3.30 ms --> 0.013 ms
  • Encode: 1.82 ms --> 0.93 ms
  • (Py3.10 parse drops 4.97 ms --> 1.60 ms)
• End-to-End Latency: Proxied gain round-trip dropped from 34.8 ms to 14.9 ms. Pipelined throughput under burst dropped from 49.5 ms/msg to 15.2 ms/msg.
• Live A/B (cbf_plus_dev_1, 880 serial 4096-channel set-gain messages):
  • Total time: 97.8 s --> 29.7 s
  • p50 latency: 77.8 ms --> 32.7 ms
  • Worst message: 25.3 s --> 60.2 ms
  • Proxy CPU cost: ~50 ms --> ~16 ms / msg (~14% core load down from ~50%)

Note: Small messages also benefit slightly from bypassing constructor validation (1-arg parse drops from 9.2 µs to 4.1 µs). The escape-containing fallback path remains unchanged.

Testing Instructions

Verification tools are included under bench/ (see bench/README.md).

Local repo tests:

cd bench
python ab_compare.py          # Benchmarks master vs working tree via git worktree
python fuzz_parse_branch.py   # Differential fuzz test vs master's parser
python run_core_tests.py      # Core + BNF test suites

[ludwigschwardt]

Interesting! What is the latency after the fix?

[sipho-sarao]

Interesting! What is the latency after the fix?

Great question, here is the latency broken down across three measurement contexts on my devbox environment:

1. Live proxy on my devbox (cbf_plus_dev_1, Py2.7, busy shared system; 4096-channel set-gain, ~20 KiB/message, serial, state-neutral):

Per-message round trip	Before	After
p50	67.8 ms	31.5 ms
p95	143.4 ms	35.0 ms
Worst observed (880-msg run)	25.3 s	60 ms

So ~30 ms per gain-set end-to-end on a loaded devbox, and the long tail is effectively gone. The new p95 sits well below the old median.

2. Controlled proxy chain (client -> proxy -> zero-work device, isolating CAM stack cost with full 60 KiB complex-formatted payloads):

• 34.8 ms --> 14.9 ms per message.

3. Codec CPU cost alone (the specific functions optimized in this PR, Py2.7 @ 4096 args):

• Parse: 7.47 ms --> 1.76 ms
• Handler copy: 3.30 ms --> 0.013 ms
• Encode: 1.82 ms --> 0.93 ms
• Total codec CPU: ~12.6 ms --> ~2.7 ms per message.

What remains after the fix is mostly the product-controller round trip (~10 ms, as it re-parses the line on its side outside of katcp-python) plus residual Tornado/thread plumbing. The KATCP codec is no longer the dominant bottleneck.

Overall throughput for the ticket's 880-message scenario dropped from ~98 s down to ~30 s on my devbox. All numbers are reproducible using the ab_compare.py script in bench/.

[ludwigschwardt]

Just some generic comments for now...

[ludwigschwardt]

What does the search check do that makes it better than before?

[sipho-sarao]

In the vast majority of KATCP arguments and in all arguments of the bulk numeric payloads this PR targets (gains, weights, delays, numeric sensor values), there are no characters needing escaping. I stand to be corrected.

Previously, self.ESCAPE_RE.sub() was executed unconditionally on every single argument, including even when nothing matches (sub() then returns the original object, no new bytes are built), it still pays the substitution's setup overhead for the callable-replacement path. When measured on my devbox Py2.7 proxy runtime, per clean argument: search takes ~278 ns vs sub taking ~457 ns.

search(x) acts as a fast-path pre-check:

1. Short-circuits clean arguments: re.search() does a single C-level scan of the bytes and returns None, avoiding the heavier sub() call entirely, saving ~0.7 ms per 4096-argument message.
2. Local variable binding: Aliasing search = self.ESCAPE_RE.search outside the loop avoids repeated attribute lookups on every iteration across thousands of arguments.

Together with folding the former empty-argument pass (x or b"\\@") into the same loop, this takes the encode step from 1.8 ms down to 0.9 ms per 4096-argument message.

The trade-off: escape-bearing arguments (mainly human-readable descriptions in #help, #sensor-list, or #log informs) take the sub() path as before, paying search + sub (+340 ns measured per escaped argument). Since the guard is per-argument, clean arguments within those same messages still benefit from the fast path.

[ludwigschwardt]

Why would you need special logic to find the KATCP repo?

[sipho-sarao]

Mainly to support the A/B differential scripts (ab_compare.py and fuzz_parse_branch.py) and flexible execution:

1. Explicit checkout target vs site-packages: It guarantees the scripts test the specific source tree you intend (including the temporary git worktree checkout that ab_compare.py generates for the baseline side), preventing Python from silently importing an already-installed katcp from site-packages.
2. Out-of-tree execution: KATCP_REPO=/path/to/katcp-python lets you run the suite against an arbitrary local checkout without copying the bench/ directory over. (For ab_compare.py and fuzz_parse_branch.py, the target must be a git repo since they run git worktree / git show against it to obtain the baseline).
3. Location independence: Discovery is anchored on the script's own __file__, so invocation works from any working directory. git rev-parse --show-toplevel additionally resolves the true repo root rather than assuming bench/'s parent is it.

For a simple single-branch run, sys.path.insert(0, dirname(dirname(__file__))) would do, _repo.py just keeps the discovery consistent between local runs and the git-worktree differential scripts. Happy to drop the rev-parse call and use dirname as the default if preferred.

[ludwigschwardt]

Where does fastkatcp come from?

[sipho-sarao]

Good catch, it doesn't come from anywhere in this repo. fastkatcp was my investigation-phase monkeypatch form of this fix: an import-time patch I used to hot-deploy the optimisations onto my devbox proxy for testing before this branch existed.

The --fast hooks were scaffolding from that phase and are vestigial here. Removed the flag and the import from all four scripts. Thanks for spotting that.

[ludwigschwardt]

Thanks for getting your agent to clean up after itself 😅🙏🏼

It would be instructive to compare the performance of katcp-python to aiokatcp on this front to see whether both have the same issues or if aiokatcp potentially has a better solution.