Skip to content

SessionManager._parse_message repr-mangles non-str tool_result content, and ToolAnalyzer's error detection silently depends on that mangling #46

Description

@Rahul-s-007

Split out of PR #43 per review — this is a distinct benchmark-harness data-integrity bug, not the Unicode-detection feature itself.

Bug 1: non-string message content is corrupted before it's ever written to disk.

SessionManager._parse_message (Detection/main_benchmark.py) passes tool_result message content — which Claude Code's JSONL always carries as a list of content blocks — straight into _truncate_content:

def _truncate_content(self, content: str, max_length: int = 10000) -> str:
    if isinstance(content, str):
        return content[:max_length] + "..." if len(content) > max_length else content
    return str(content)[:max_length] + "..." if len(str(content)) > max_length else str(content)

The non-str branch calls str() on the list, which invokes Python's repr() on every element — silently escaping any non-printable Unicode into literal backslash text (e.g. a real hidden-Unicode payload becomes the 10 literal ASCII characters \U000e0069, not the actual codepoint) before json.dump ever writes claude_conversation.json. Irreversible: nothing downstream un-reprs it, including main_detector.py's _convert_conversation_to_messages, which reads that file back unmodified. In a real recorded benchmark run, this hits 8112 of ~19k message contents — every MCP tool output.

Bug 2 (found while reviewing a fix for Bug 1): ToolAnalyzer.analyze_tool_usage silently depends on Bug 1's broken output.

ToolAnalyzer.analyze_tool_usage (main_benchmark.py:458-460) detects failed tool calls by grepping recorded content for repr artifacts — "'is_error': True", "'tool_use_id': '...'" — patterns that only ever existed because the current code stores str() of the raw block list. A correct fix for Bug 1 that returns clean extracted text instead makes this grep always come up empty: failed_tool_ids stays permanently empty, and _validate_expected_tools counts every failed tool call as successful, for every task in tasks.json. A silent regression across the whole benchmark, caused by fixing Bug 1 correctly — the two need to be fixed together, not sequentially.

Also, once a proper extractor is in place, some edge cases need care: dict-shaped content still hitting a str() fallback and re-mangling codepoints; a falsy toolUseResult['result'] overwriting real content from a block's own content field; and a single root-level toolUseResult being applied to every tool_result block when a message contains more than one (e.g. parallel tool calls), rather than just the one it actually corresponds to.

I already have a fix scoped covering all of the above, plus new test coverage for SessionManager and ToolAnalyzer (currently zero coverage on this parsing path). PR to follow.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions