feat(troubleshoot): OR max-consecutive-system-exceptions playbook + test

[Stefan-Virgil]

What

Adds the maximum consecutive system exceptions Orchestrator playbook and its faithful-replay test — closing a committed gap from the OR Diagnostic Agent backlog.

Playbook — job-consecutive-system-exceptions.md

The maximum number of consecutive system exceptions was reached. Consecutive retry number: <N> is a symptom, not the cause. The real fault is the System Exception that recurs on every transaction (broken selector, app not launching, down dependency) until the REFramework threshold trips. Find the recurring underlying exception in logs/traces and fix that — raising MaxConsecutiveSystemExceptions only delays the abort. Registered in products/orchestrator/summary.md.

Test — rpa-consecutive-system-exceptions

An OrderReconciliation REFramework job aborts after 5 consecutive system exceptions; the logs/traces show a SelectorNotFoundException on "Click Submit" on every transaction. The job Info gives only the abort message, so the root cause is reachable only by reading logs/traces — an agent that stops at the abort (or suggests raising the threshold) scores low. Graded on identifying the recurring selector failure and recommending a selector fix.

Validation

coder-eval -m claude-opus-4-6, --repeats 3 -j 3: 3/3 SUCCESS — skill_triggered = yes and llm_judge = 1.0 on all replicates.

Notes

• Follows the troubleshoot suite contract: exactly skill_triggered + llm_judge.
• Tagged e2e, per tests/tasks/uipath-troubleshoot/CLAUDE.md.

🤖 Generated with Claude Code

[github-actions]

Claude finished @Stefan-Virgil's task in 4m 46s —— View job

---

PR Review: OR max-consecutive-system-exceptions playbook + test

• Read review criteria and project conventions
• Analyze full diff
• Read all changed files in full
• Check playbook content and registration
• Check test task structure and grading
• Verify links, cross-references, and conventions
• Post review findings

---

Summary

Adds a new Orchestrator playbook (job-consecutive-system-exceptions.md) for the "maximum consecutive system exceptions" failure class and a matching faithful-replay e2e test scenario (rpa-consecutive-system-exceptions). The playbook teaches that the abort message is a symptom — the real fault is the recurring System Exception underneath — and the test validates that agents dig into logs/traces rather than stopping at the abort message.

Change-by-Change Review

1. skills/uipath-troubleshoot/references/products/orchestrator/playbooks/job-consecutive-system-exceptions.md

Severity: OK

Well-structured playbook following the established pattern — frontmatter with confidence: medium, clear Context / Investigation / Resolution / Prevention sections. The investigation steps use --output json per convention. The key message ("symptom, not the root cause") is front-loaded. The resolution correctly warns against raising MaxConsecutiveSystemExceptions without fixing the underlying exception. Prose is terse and prescriptive per token-optimization rules.

2. skills/uipath-troubleshoot/references/products/orchestrator/summary.md

Severity: OK

Single row added to the summary table. The relative link [job-consecutive-system-exceptions.md](./playbooks/job-consecutive-system-exceptions.md) resolves correctly. Description is concise and matches the playbook's thesis. Confidence column matches frontmatter. Alphabetical/positional placement at end of table is fine (existing rows are not strictly alphabetical either).

3. tests/.../rpa-consecutive-system-exceptions/task.yaml

Severity: OK

Follows the canonical troubleshoot suite contract exactly:

• task_id: skill-troubleshoot-consecutive-system-exceptions — matches the skill-<domain>-<capability> pattern.
• tags: [uipath-troubleshoot, orchestrator, rpa, e2e, mode:diagnose] — skill tag first, domain tags present, tier is e2e (not smoke, per suite rules), mode:diagnose included.
• success_criteria uses exactly skill_triggered + llm_judge — no forbidden criteria types.
• llm_judge block matches the canonical shape verbatim (prompt, weights, flags).
• run_limits values (task_timeout: 5400, max_turns: 60, turn_timeout: 3600) match the CLAUDE.md-prescribed values.
• sandbox correctly uses python: {}, template_sources with the depth-correct ../../../_shared/mock_template path, and mock_path_dirs: ["m"].
• simulation section follows the established pattern from sibling scenarios.
• No env_packages (forbidden for @uipath/cli).

4. tests/.../rpa-consecutive-system-exceptions/README.md

Severity: OK

Clear documentation of what the scenario covers, how it reproduces, and the distinguishing fingerprint. Correctly notes that fixtures are synthetic (not captured from a real session). The success criteria summary accurately reflects task.yaml.

5. tests/.../rpa-consecutive-system-exceptions/RESOLUTION.md

Severity: OK

Comprehensive ground truth for the LLM judge: identifies the root cause (recurring SelectorNotFoundException on "Click Submit"), the fix (repair selector), and explicitly rules out the threshold-raise and business-exception hypotheses. The investigation summary table with H1/H2/H3 is thorough. Format is consistent with sibling RESOLUTION.md files.

6. tests/.../rpa-consecutive-system-exceptions/data/m/r/manifest.json

Severity: OK

Rules are ordered correctly (specific before generic). The docsai ask passthrough rule is present per suite requirements. unmocked_default returns empty array with exit 0 — consistent with sibling scenarios. The mock dispatch chain (folders → jobs list → jobs get → jobs logs → jobs traces) is logical for this playbook's investigation flow.

One note: the manifest includes two jobs logs rules (lines 8–9) — one with --level Error and one without — both pointing to the same file. This is intentional and correct: it ensures the agent gets the error logs regardless of whether it adds the --level Error filter.

7. tests/.../rpa-consecutive-system-exceptions/data/m/r/*.json (fixture files)

Severity: Low

All fixture files are well-formed JSON with consistent keys and timestamps. No secrets, personal paths, or real email addresses beyond the scrubbed original_email@test.com pattern. The _doc fields in fixture files (e.g., jobs-get-failing.json:4) explicitly state the root cause diagnosis — this matches the established convention in sibling scenarios (e.g., rpa-foreground-already-running fixtures also carry diagnostic _doc hints). Since skill_triggered is required, the agent must invoke the skill to pass, and the test's 3/3 pass rate confirms this works in practice.

8. tests/.../rpa-consecutive-system-exceptions/process/ (project files)

Severity: OK

Minimal REFramework-style project. project.json has correct schema, Main.xaml is a simple sequence with neutral log messages (no diagnosis-leaking names — compliant with the anti-pattern rules in the suite's CLAUDE.md). entry-points.json and project.uiproj are consistent with the main project file.

What's Missing

Nothing identified. CODEOWNERS already covers both skill and test paths. The playbook is registered in summary.md. The test follows the canonical contract. All relative links resolve.

Area Ratings

Area	Status	Notes
Frontmatter	OK	Playbook has valid confidence: medium frontmatter
E2E Tests	OK	Single e2e scenario with canonical skill_triggered + llm_judge; 3/3 pass claimed
Skill Body	OK	Playbook is well-structured, terse, and prescriptive
References & Assets	OK	Summary table updated, link resolves, mock fixtures consistent
Repo Hygiene	OK	CODEOWNERS covered, no secrets, no cross-skill deps, scoped changes

Issues for Manual Review

• Domain accuracy: The playbook states that MaxConsecutiveSystemExceptions is configured via "REFramework Config.xlsx / an asset." Verify this matches the current REFramework template's configuration mechanism — if newer templates use a different config surface, the playbook should reflect that.

Conclusion

Clean PR. The playbook and test follow all suite conventions, the task YAML matches the canonical contract exactly, and the 3/3 pass rate with llm_judge: 1.0 demonstrates the scenario works as intended. No issues found that would block merge.

---

[github-actions]

Claude finished @Stefan-Virgil's task in 2m 1s —— View job

---

Coder-eval task lint (advisory)

1 task YAML changed; verdicts: 0 Critical, 0 High, 0 Medium, 0 Low, 1 OK.

Rubric: .claude/commands/lint-task.md. This check is advisory and never blocks merge.

Evidence of passing run

• ✅ Author confirms passing run: "coder-eval -m claude-opus-4-6, --repeats 3 -j 3: 3/3 SUCCESS — skill_triggered = yes and llm_judge = 1.0 on all replicates."

Per-task lint

tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-consecutive-system-exceptions/task.yaml — verdict: OK

Task follows the canonical troubleshoot suite contract exactly: goal-oriented prompt (no procedure leak), skill_triggered + llm_judge with include_reference: true (ground truth via RESOLUTION.md), run_limits in the top-level block (not under agent:), no sandbox env_packages, and the scenario exercises a distinct failure class (consecutive system exceptions / SelectorNotFoundException) not covered by any of the 5 nearest siblings (queue-items-failing-test, rpa-foreground-misconfigured, rpa-foreground-already-running, rpa-job-killed-by-operator-ui, rpa-job-killed-by-watchdog-account).

Within-PR duplicates

• No duplicate clusters detected (only 1 new task YAML).

Conclusion

• ✅ All changed tasks pass the rubric. Evidence of passing run confirmed.

---