Skip to content

feat(benchmarks): add deterministic Sentry evaluation - #26

Merged
tcballard merged 2 commits into
mainfrom
codex/sentry-benchmark
Jul 31, 2026
Merged

feat(benchmarks): add deterministic Sentry evaluation#26
tcballard merged 2 commits into
mainfrom
codex/sentry-benchmark

Conversation

@tcballard

@tcballard tcballard commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Summary

  • add SentryBench as a deterministic, contract-shaped evaluation of decision-to-code enforcement
  • freeze 80 cases, including 38 seeded violations across supported rule kinds, languages, multi-finding ordering, diff isolation, invalid constraints, fail-closed behaviour, SARIF, gate parity, and repeatability
  • generate a temporary 5,000-decision corpus to verify supported-scale correctness without committing thousands of repetitive fixtures
  • keep enforcement correctness separate from classification/coverage and wall-clock performance
  • run the suite against the published decided CLI as an external process, with no Core imports or model judge
  • move benchmark CI to the verified AsDecided v0.26.0 release binary

Results

All committed correctness floors are 1.0:

  • 80/80 cases passed
  • 100% violation recall
  • 100% clean-case pass rate
  • 100% decision/rule/path/line attribution
  • 100% SARIF accuracy
  • 100% shared-projection sentry / gate --code parity
  • byte-identical repeated JSON
  • zero negative-case violations

Published macOS v0.26.0 binary, 30 iterations:

  • clean full-tree: 7.474 ms median, 8.015 ms p95
  • violating diff: 14.870 ms median, 16.893 ms p95

Timing remains diagnostic and outside the scored metrics.

The 5,000-decision profile separately passes clean/full, violating/full, violating/diff, composed-gate parity, attribution, coverage-accounting, and byte-determinism checks. A representative local run completed the dedicated Sentry profiles in 0.61–0.67 seconds and the composed gate in 1.74 seconds; these timings are reported, not gated.

Contract boundary

decided gate --code does not yet expose Sentry's decision and rule provenance. Parity therefore compares the public outcome, coverage, code, path, and line fields shared by both surfaces. Dedicated Sentry JSON remains authoritative for decision/rule attribution, and the missing composed-gate provenance is recorded as follow-up rather than inferred.

Validation

  • RAC_BIN=/private/tmp/asd-v026-published/decided python3 sentry/run.py --check
  • RAC_BIN=/private/tmp/asd-v026-published/decided python3 sentry/run.py --scale --corpus-size 5000
  • RAC_BIN=/private/tmp/asd-v026-published/decided uv run --extra dev pytest -q — 63 passed
  • fixture corpora and governing decision/requirement artifacts validate with AsDecided v0.26.0
  • shell syntax and staged-diff whitespace checks pass

@tcballard
tcballard merged commit 46477f5 into main Jul 31, 2026
12 checks passed
@tcballard
tcballard deleted the codex/sentry-benchmark branch July 31, 2026 08:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant