Split out of PR #43 per review — this is a distinct benchmark-harness data-integrity bug, not the Unicode-detection feature itself.
Bug 1: non-string message content is corrupted before it's ever written to disk.
SessionManager._parse_message (Detection/main_benchmark.py) passes tool_result message content — which Claude Code's JSONL always carries as a list of content blocks — straight into _truncate_content:
def _truncate_content(self, content: str, max_length: int = 10000) -> str:
if isinstance(content, str):
return content[:max_length] + "..." if len(content) > max_length else content
return str(content)[:max_length] + "..." if len(str(content)) > max_length else str(content)
The non-str branch calls str() on the list, which invokes Python's repr() on every element — silently escaping any non-printable Unicode into literal backslash text (e.g. a real hidden-Unicode payload becomes the 10 literal ASCII characters \U000e0069, not the actual codepoint) before json.dump ever writes claude_conversation.json. Irreversible: nothing downstream un-reprs it, including main_detector.py's _convert_conversation_to_messages, which reads that file back unmodified. In a real recorded benchmark run, this hits 8112 of ~19k message contents — every MCP tool output.
Bug 2 (found while reviewing a fix for Bug 1): ToolAnalyzer.analyze_tool_usage silently depends on Bug 1's broken output.
ToolAnalyzer.analyze_tool_usage (main_benchmark.py:458-460) detects failed tool calls by grepping recorded content for repr artifacts — "'is_error': True", "'tool_use_id': '...'" — patterns that only ever existed because the current code stores str() of the raw block list. A correct fix for Bug 1 that returns clean extracted text instead makes this grep always come up empty: failed_tool_ids stays permanently empty, and _validate_expected_tools counts every failed tool call as successful, for every task in tasks.json. A silent regression across the whole benchmark, caused by fixing Bug 1 correctly — the two need to be fixed together, not sequentially.
Also, once a proper extractor is in place, some edge cases need care: dict-shaped content still hitting a str() fallback and re-mangling codepoints; a falsy toolUseResult['result'] overwriting real content from a block's own content field; and a single root-level toolUseResult being applied to every tool_result block when a message contains more than one (e.g. parallel tool calls), rather than just the one it actually corresponds to.
I already have a fix scoped covering all of the above, plus new test coverage for SessionManager and ToolAnalyzer (currently zero coverage on this parsing path). PR to follow.
Split out of PR #43 per review — this is a distinct benchmark-harness data-integrity bug, not the Unicode-detection feature itself.
Bug 1: non-string message content is corrupted before it's ever written to disk.
SessionManager._parse_message(Detection/main_benchmark.py) passestool_resultmessage content — which Claude Code's JSONL always carries as a list of content blocks — straight into_truncate_content:The non-str branch calls
str()on the list, which invokes Python'srepr()on every element — silently escaping any non-printable Unicode into literal backslash text (e.g. a real hidden-Unicode payload becomes the 10 literal ASCII characters\U000e0069, not the actual codepoint) beforejson.dumpever writesclaude_conversation.json. Irreversible: nothing downstream un-reprs it, includingmain_detector.py's_convert_conversation_to_messages, which reads that file back unmodified. In a real recorded benchmark run, this hits 8112 of ~19k message contents — every MCP tool output.Bug 2 (found while reviewing a fix for Bug 1):
ToolAnalyzer.analyze_tool_usagesilently depends on Bug 1's broken output.ToolAnalyzer.analyze_tool_usage(main_benchmark.py:458-460) detects failed tool calls by grepping recorded content for repr artifacts —"'is_error': True","'tool_use_id': '...'"— patterns that only ever existed because the current code storesstr()of the raw block list. A correct fix for Bug 1 that returns clean extracted text instead makes this grep always come up empty:failed_tool_idsstays permanently empty, and_validate_expected_toolscounts every failed tool call as successful, for every task intasks.json. A silent regression across the whole benchmark, caused by fixing Bug 1 correctly — the two need to be fixed together, not sequentially.Also, once a proper extractor is in place, some edge cases need care: dict-shaped content still hitting a
str()fallback and re-mangling codepoints; a falsytoolUseResult['result']overwriting real content from a block's owncontentfield; and a single root-leveltoolUseResultbeing applied to everytool_resultblock when a message contains more than one (e.g. parallel tool calls), rather than just the one it actually corresponds to.I already have a fix scoped covering all of the above, plus new test coverage for
SessionManagerandToolAnalyzer(currently zero coverage on this parsing path). PR to follow.