fix(cli): fold variant into report-supplied test ids for local quarantine checks by TylerJang27 · Pull Request #1146 · trunk-io/analytics-cli

TylerJang27 · 2026-07-22T18:53:44Z

Context

We were erroneously ignoring the passed variant when generating the test case ids we used to calculate quarantining when an ID was provided by the test runner (i.e. XCResult). This was not an issue before because our existing iOS users were not uploading with variants. Fixed the bug and added regression tests.

Summary

convert_case_to_test (in cli/src/context_quarantine.rs) used a report source's own stable test identifier as-is whenever a JUnit/xcresult test case carried a non-empty "id" extra attribute (e.g. xcresult's per-test-method identifier from XCResult::generate_id), bypassing gen_info_id entirely.
That identifier does not encode --variant. Every other id derivation in this codebase (JunitParser::into_test_case_runs for JUnit-sourced tests, and bundle ingestion server-side) folds variant into the id via gen_info_id/gen_info_id_base.
Net effect: for any xcresult upload run with --variant set, the CLI's local quarantine check compared a variant-less id against the server's variant-scoped quarantine list — a permanent mismatch, regardless of whether the test was actually quarantined. The "Learn more" link printed alongside an unquarantined failure (built from that same id via url_for_test_case) also pointed at the wrong test entity on the dashboard, since the dashboard identifies tests by the variant-scoped id.
This reproduced as: a failing test that was quarantined server-side (confirmed present in that test's continuous history in the dashboard) still showed Quarantined: 0 locally, and the printed dashboard link resolved to an unrelated test.

Fix

Route the report-supplied id through Test::generate_custom_uuid — already used for exactly this purpose in Test::new — instead of assigning it directly to test.id. This folds variant into the id (matching Test::set_id's existing behavior for tests with no report-supplied id, and matching gen_info_id's behavior used elsewhere for JUnit/server-side ingestion). Behavior is unchanged when variant is empty (the common case for non-variant uploads), since gen_info_id_base returns the report-supplied id unchanged in that case.

I confirmed no other call site in the crate has this same pattern — test_report::add_test (used by the Python/JS/Ruby language-runtime bindings, a separate non-JUnit/non-xcresult path) just stores whatever id its caller already computed, so it's unaffected by this change and out of scope here.

Test plan

Added test_convert_case_to_test_folds_variant_into_report_supplied_id: given a report-supplied id shaped like XCResult::generate_id's output and a non-empty variant, asserts the resulting Test.id (a) differs from the raw report-supplied id, and (b) matches gen_info_id computed independently with the same inputs — i.e. matches what JUnit parsing/server-side ingestion would compute for the same test.
Added test_convert_case_to_test_keeps_report_supplied_id_without_variant as a companion/regression guard: with no variant, the report-supplied id is used unchanged (no behavior change for non-variant uploads).
Verified the new test fails against the pre-fix code (reverted the fix locally, confirmed test_convert_case_to_test_folds_variant_into_report_supplied_id fails with left == right on the raw id) and passes with the fix applied.
cargo test -p trunk-analytics-cli --lib — 20 passed, 0 failed.
cargo fmt -p trunk-analytics-cli -- --check clean.
cargo clippy -p trunk-analytics-cli --lib --tests --no-deps — no new findings introduced by this change (all output is pre-existing, unrelated lint debt elsewhere in the crate).

Generated by Claude Code

…tine checks convert_case_to_test used a report source's own stable test identifier (e.g. xcresult's per-test-method id) as-is for tests with a non-empty "id" extra attribute, bypassing gen_info_id entirely. That id does not encode --variant, while every other id derivation in this codebase (JunitParser::into_test_case_runs, and bundle ingestion server-side) folds variant in via gen_info_id/gen_info_id_base. For any xcresult upload with --variant set, this meant the CLI's local quarantine check compared a variant-less id against the server's variant-scoped quarantine list -- a permanent mismatch -- and the "Learn more" link printed for a failure was built from that same variant-less id, pointing at the wrong test entity. Route the report-supplied id through Test::generate_custom_uuid (already used for this exact purpose elsewhere) instead of assigning it directly, so the locally computed id matches what the server tracks. Behavior is unchanged when variant is empty. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G2HLZJPJWaHp7vFH3frr9k

trunk-io · 2026-07-22T18:53:48Z

😎 Merged directly without going through the merge queue, as the queue was empty and the PR was up to date with the target branch - details.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G2HLZJPJWaHp7vFH3frr9k

… set A plain JUnit `id="..."` attribute isn't guaranteed to be a v5 UUID the way xcresult's is. Always routing it through gen_info_id (as the previous commit did) meant that for a non-v5, non-"trunk:"-prefixed custom id with an empty variant, gen_info_id_base fell through to its final fallback -- which ignores info_id entirely and recomputes from file/classname/name -- silently discarding the caller's id even though nothing needed folding in. Gate on `variant.is_empty()` explicitly, matching the same shortcut JunitParser's own id derivation already uses (existing_id.is_some() && variant.is_empty() => use existing_id verbatim). This guarantees no behavior change whenever variant is empty, for any id shape, while still fixing the original variant-set/xcresult case. Added a regression test with a non-UUID custom id and empty variant, confirmed it fails against the prior commit's version of the fix and passes with this gate in place. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G2HLZJPJWaHp7vFH3frr9k

trunk-staging-io · 2026-07-22T19:09:16Z

Failed Test	Failure Summary	Logs
`pending_quarantine_test should be quarantined when run with variant`	A test marked as pending was expected to fail but unexpectedly passed.	Logs ↗︎
`variant_quarantine_test should be quarantined when run with variant`	A test expected the sum of 2 + 2 to be 5, but it was actually 4, indicating a failing assertion.	Logs ↗︎

_{View Full Report ↗︎ ⋅ Docs}

trunk-io · 2026-07-22T19:12:55Z

_{View Full Report ↗︎ ⋅ Docs}

Factor the report-supplied-id/variant regression tests through shared helpers (failing_case_with_report_id, convert_and_expected_id) instead of near-duplicated setup in each test. Also add two cases the prior tests didn't cover: - A non-v5-shaped id (e.g. a hand-written JUnit id="..." attribute) with a variant set: gen_info_id_base's non-v5 fallback recomputes from file/classname/name rather than preserving the raw id. This isn't a regression -- JunitParser's own id derivation hits the same fallback for identical inputs -- but it wasn't pinned down before, and it's exactly the kind of divergence this whole fix is about avoiding, so it's worth locking in explicitly. - An empty report-supplied id with a variant set, confirming the (untouched) id.is_empty() branch still falls back to the generated id identically to the no-id-at-all case. Verified each test's discriminating power by running the full regression suite against both prior (buggy) versions of the fix: the original code (unconditional `test.id = id.clone()`) fails the two "folds variant" tests, and the first-pass fix (unconditional generate_custom_uuid, no variant.is_empty() gate) fails exactly test_convert_case_to_test_keeps_arbitrary_report_supplied_id_without_variant -- the precise regression that version introduced. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G2HLZJPJWaHp7vFH3frr9k

…eport generate_internal_file(_from_bep) received upload_args.variant (raw, untruncated) instead of meta.variant (truncated to 64 chars in gather_initial_test_context). That's the only value that reaches JunitParser::into_test_case_runs, which uses it to fold variant into test_case_id for the per-TestCaseRun is_quarantined flag -- so for a variant over 64 characters, that flag was computed against a different variant than the one the local quarantine check (and the server) use. Pass meta.variant.clone() at both call sites instead. Also drop "regression" framing from the context_quarantine.rs id/variant tests added earlier (plain constant/function names, no doc comments narrating intent) and dedupe a constant that collided with an existing ORG_SLUG in the same test module. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G2HLZJPJWaHp7vFH3frr9k

TylerJang27 · 2026-07-22T20:10:21Z

            #[cfg(not(target_os = "macos"))]
            true,
-            upload_args.variant,
+            meta.variant.clone(),


Truncated to 64 chars, rather than the raw val we get from the upload_args. This shouldn't actually change anything downstream, but it matches our usage elsewhere

ids from junit.xml files or rspec aren't UUIDs and aren't transformed/normalized before reaching convert_case_to_test. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G2HLZJPJWaHp7vFH3frr9k

codecov-commenter · 2026-07-22T20:55:34Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 83.14%. Comparing base (3fb8713) to head (ed28621).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1146      +/-   ##
==========================================
+ Coverage   82.82%   83.14%   +0.32%     
==========================================
  Files          70       70              
  Lines       15850    15919      +69     
==========================================
+ Hits        13127    13236     +109     
+ Misses       2723     2683      -40

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

claude added 2 commits July 22, 2026 19:03

Streamline comments on the variant-fold fix and its regression tests

5f0f6a5

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G2HLZJPJWaHp7vFH3frr9k

claude added 2 commits July 22, 2026 19:21

TylerJang27 commented Jul 22, 2026

View reviewed changes

TylerJang27 requested review from acatxnamedvirtue, dfrankland and max-trunk and removed request for dfrankland July 22, 2026 20:25

dfrankland approved these changes Jul 22, 2026

View reviewed changes

trunk-io Bot merged commit 6bed25e into main Jul 22, 2026
27 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix(cli): fold variant into report-supplied test ids for local quarantine checks - #1146

fix(cli): fold variant into report-supplied test ids for local quarantine checks#1146
trunk-io[bot] merged 6 commits into
mainfrom
claude/trunk-flaky-test-upload-gveyfb

TylerJang27 commented Jul 22, 2026 •

edited

Loading

Uh oh!

trunk-io Bot commented Jul 22, 2026 •

edited

Loading

Uh oh!

trunk-staging-io Bot commented Jul 22, 2026 •

edited

Loading

Uh oh!

trunk-io Bot commented Jul 22, 2026 •

edited

Loading

Uh oh!

TylerJang27 Jul 22, 2026

Uh oh!

codecov-commenter commented Jul 22, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

TylerJang27 commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Context

Summary

Fix

Test plan

Uh oh!

trunk-io Bot commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

trunk-staging-io Bot commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

trunk-io Bot commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

TylerJang27 Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

codecov-commenter commented Jul 22, 2026

Codecov Report

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

4 participants

TylerJang27 commented Jul 22, 2026 •

edited

Loading

trunk-io Bot commented Jul 22, 2026 •

edited

Loading

trunk-staging-io Bot commented Jul 22, 2026 •

edited

Loading

trunk-io Bot commented Jul 22, 2026 •

edited

Loading