Bench: harbor + swe-bench generation mode with runtime hardening (+ a templated dataset card)#17
Conversation
Squashed fork delta off upstream/main, bundling the work that landed on the AletheiaResearch fork via PR #5 and PR #7: - bench mode: drive Harbor as a library and add a swe-bench backend under a single `generate --mode bench` flow (its own dataset, no mode co-mingling). - seed-workspace verifiable tasks: exit-code-reward verifier that runs over a writable snapshot (binary reward via VerificationResult). - bench runtime hardening: purge per-task Docker images after each run (gated by output.keep_bench_images), make Ctrl-C actually stop the run, and render the dataset card from a configurable Jinja template (output.license / card_extra / readme_template).
|
@codex review |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 43c8ce30a6
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
- trace_readme: fix data_files regression (P1) — the templated-card train fallback advertised only top-level *.jsonl, so nested extractions (Cursor's project-relative transcripts) were counted in the card but excluded from HF data_files, publishing an empty/incomplete dataset. Now advertise a recursive <dir>/**/*.jsonl for each data-bearing, non-excluded subdir. - trace_readme: card reward summary now also reads bench's metadata/<stem>.json sidecars (split + numeric reward), not just verification/<stem>.json (P3), so bench datasets show their pass/fail counts instead of just the tag. - bench/backends/base: bound the instances discriminator in source_id (P2) — a large SWE-bench subset no longer leaks its full comma-joined list into filenames / Docker tags / container names past OS/Docker length limits (hash when long; sorted, so listing order is irrelevant). - bench/backends/harbor: key the source download cache on repo too (P2), matching source_id, so two sources with the same spec/version but different repos don't share a cache dir and reuse the wrong tasks. - config: resolve a relative output.readme_template against the config file like prompts_file (P2), not the process CWD. - bench/runner: fail fast when agent.provider=codex + use_host_auth and no API key (P2) — bench backends don't seed the host-auth snapshot into the task container, so this would otherwise launch a silently-unauthenticated run. Full suite 618 passed / 27 skipped; ruff clean.
|
Thanks for the review pass. All six findings verified against the code and addressed in
Each fix has a regression test ( |
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d943875c5f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| NON_DATA_TRACE_DIR_NAMES = frozenset( | ||
| {"partials", "failures", "verification", "sandbox", "__pycache__", "bench"} |
There was a problem hiding this comment.
Because _jsonl_files checks these names against every path component, any real trace data organized under a directory named bench, verification, or sandbox is now silently dropped by teich convert, README generation, and related dataset scans. This is easy to hit with extracted or manually staged datasets, such as a Cursor workspace/repo named bench; the skip should be scoped to known top-level Teich sidecar directories instead of arbitrary nested folders.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Deferring this one — reasoning: the per-component non-data-dir match is pre-existing upstream behavior (it applied to partials/failures before this PR); this PR only expanded the name set. The nested-name exclusion is intentional and documented (# the name is listed here only to defend a nested override), and the cited Cursor scenario doesn't actually trip it — cursor dirs are slugified full paths like c-Users-...-bench, which aren't an exact bench component match. Scoping the skip to top-level would flip that documented behavior and needs to touch ~6 call sites (converter/trace_readme/cli/runner) to be consistent, so it's better as its own PR than folded into this review round. Happy to open that follow-up if you'd like it.
…erred) - bench/backends/base: harvest now deletes any sibling-split copy before writing (P2) — a re-run (no --resume) whose score crosses a routing boundary otherwise left a stale, contradictorily-labeled row that the dataset scanners still read. - bench/backends/base: guard the *default* bench_root against the output dir too (P2), not just an explicit output.bench_dir — e.g. --output bench made the sibling default collide with the dataset and upload raw trials as rows. - trace_readme: count borderline bench tasks in the card reward summary (P3); they were dropped, so mixed runs undercounted and an all-borderline run showed no reward section. Card shows '.. / N borderline' only when present. - config: resolve a relative *local* bench source (./tasks, ../x, ~/x) against the config file like prompts_file (P2), not the process CWD. - bench/backends/swebench: fail fast when the claude-code agent is paired with a non-anthropic api.provider (P2) — the container's CLI has no usable ANTHROPIC_* creds otherwise and would emit empty traces. Deferred: the converter's per-component non-data-dir skip (Codex P2) — pre-existing behavior, the nested-name exclusion is intentional/documented, and top-level-only scoping is a behavior change across ~6 call sites; better as its own PR. Full suite 623 passed / 27 skipped; ruff clean.
|
Round 2 addressed in
Each fix has a regression test. Full suite: 623 passed / 27 skipped; ruff clean. |
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9098444011
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Terminal-bench 2.0 tasks declare [environment].docker_image, so harbor runs them in prebuilt mode: compose pulls that image directly and never builds an hb__<task> image, and harbor's teardown (down --rmi local) does not remove pulled tagged images. The per-task purge only ever removed hb__* names (even harbor's _main_image_name returns the hb__ name in prebuilt mode), so every task leaked its pulled image (up to ~2GB each) on all outcomes, including Ctrl-C. Parse task.toml for the prebuilt image name(s) — [environment].docker_image and a separate [verifier.environment].docker_image — and include them in the purge list. Images declared by more than one task of the source are kept (deduped per task first: one task using the same image for agent + verifier isn't sharing): rmi'ing a shared image can race a concurrent sibling's compose up (No such image) and forces a re-pull per task. Tests that need tomllib (py3.11+) skip on 3.10, where the harbor backend can't run anyway. Squashed cherry-pick of #9 (32d2c67 + review-fix bc16887).
|
Added 76928d7 — a bug fix for per-task Docker image cleanup, squash-picked from AletheiaResearch#9 (reviewed + merged there). Bug: terminal-bench 2.0 tasks declare Fix: parse the task's 🤖 Generated with Claude Code |
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 76928d7c96
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if other != split: | ||
| (cfg.output.traces_dir / other / f"{stem}.jsonl").unlink(missing_ok=True) | ||
| trace_path.parent.mkdir(parents=True, exist_ok=True) | ||
| trace_path.write_text("\n".join(run.native_lines) + "\n", encoding="utf-8") |
There was a problem hiding this comment.
When users run generate --mode bench and then load or convert the dataset through Teich, this writes the JSONL as the raw native trace while keeping split/reward only in output/metadata/<stem>.json. convert_traces_to_training_data() consumes the JSONL trace files, and remote load_traces() does not download those metadata JSON sidecars, so the published bench dataset has no passed/reward fields despite being marked reward-labeled; reward training or filtering will silently lose the verifier labels.
Useful? React with 👍 / 👎.
| return "\n".join(lines) | ||
| if splits: | ||
| return splits | ||
| patterns = ["*.jsonl"] |
There was a problem hiding this comment.
For nested-only extractions such as Cursor project-relative transcripts, this always adds the top-level *.jsonl pattern even when there are no top-level JSONL files. Hugging Face resolves each data_files glob and raises when a listed pattern matches nothing, so the generated dataset card can make the Hub viewer or load_dataset fail for the nested-only case this helper is trying to support; include the top-level glob only when it has at least one match.
Useful? React with 👍 / 👎.
| # uploads verifier sidecars as dataset provenance.) The bench intermediates normally live | ||
| # in a sibling ``bench`` dir; the name is listed here only to defend a nested override. | ||
| NON_DATA_TRACE_DIR_NAMES = frozenset( | ||
| {"partials", "failures", "verification", "sandbox", "__pycache__", "bench"} |
There was a problem hiding this comment.
When extracting nested sessions, for example Cursor project transcripts, Teich preserves the project directory under the output tree; if that project or any parent directory is named bench, sandbox, or verification, this global component-name filter makes _jsonl_files() skip every valid JSONL row below it. That silently removes legitimate sessions from conversion/README/mode detection even though the directory is user data, not a configured artifact directory; exclude the actual artifact locations instead of any matching path component.
Useful? React with 👍 / 👎.
| from harbor.models.environment_type import EnvironmentType | ||
| from harbor.models.trial.config import TaskConfig, TrialConfig | ||
|
|
||
| config = TrialConfig(task=TaskConfig(path=task_dir), trials_dir=trials_dir) |
There was a problem hiding this comment.
When a Harbor bench source is run with timeout_seconds set, this builds a default TrialConfig and never copies Teich's timeout into Harbor's agent/verifier timeout overrides. Harbor therefore uses the task/default timeouts instead of the user's cap, so a long-running or hung Harbor agent can keep running well past timeout_seconds; propagate the configured timeout into the trial config before launching it.
Useful? React with 👍 / 👎.
| "rewards": run.rewards or {}, | ||
| "agent": cfg.get_agent_provider(), | ||
| "trace_file": f"{split}/{stem}.jsonl", | ||
| **run.metadata, |
There was a problem hiding this comment.
For Codex/Hermes native traces, trace_metadata() can return a source key from the recovered session metadata, often null or a provider-local value. Expanding run.metadata after the bench fields overwrites the benchmark source_id(source) written above, so the uploaded sidecar no longer reliably identifies which benchmark source produced the row; merge recovered trace metadata first or protect the reserved provenance keys.
Useful? React with 👍 / 👎.
- bench/runner: exit non-zero when tasks were attempted but nothing harvested — a run where every task fails no longer looks like a successful empty benchmark in automation. Resume 'nothing to do' and legitimately-empty runs stay exit 0. - cli: write the dataset card for a partial bench dataset before exiting on a later-source failure (parity with prompt mode); no auto-publish. - trace_readme: thread excluded_dirs into _split_data_files, so a custom output.failures_dir under traces_dir isn't advertised as a train split (follow-up to the round-1 data_files fix, which only skipped reserved names). - config: also resolve a bare relative local bench source (data/tasks, instances.jsonl) that exists beside the config, not just ./ ../ ~ prefixes (follow-up to the round-2 local-source fix); registry/HF specs untouched. - bench/backends/swebench: chmod the bind-mounted capture tree back to a host-writable mode before the container exits, so the next no-resume run's rmtree() can't fail on root-owned files (native-Linux Docker). Deferred: local SWE-bench custom-instance images (Codex P2) — the fix needs a new bench.sources[].namespace config field (new API surface) and only affects novel unpublished instances; better as its own PR. Full suite 632 passed / 27 skipped; ruff clean.
|
Round 3 addressed in
Full suite: 632 passed / 27 skipped; ruff clean. (Also resolved the now-fixed threads from all three rounds so they collapse; the two deferred threads stay open.) |
|
@codex review |
- swe-bench: add a per-source `namespace` (BenchSource.namespace, default "swebench") wired through make_test_spec + _ensure_instance_image; set it to null to build instance images locally, which is required for custom/unpublished instances (previously the build branch was unreachable). Documented in config.example.yaml / CONFIG_TEMPLATE. (Lands the round-3 deferred item.) - runner: add a rich per-source progress bar (_BenchProgress) showing M/N with a live passed/failed/borderline/error tally, mirroring prompt mode's live display. No-ops when not a terminal (per-task lines still print); never installs a signal handler, so the Ctrl-C teardown and the all-failed/partial-README behavior are unchanged. Counts are tracked even when disabled, so the logic is unit-tested. Adversarially reviewed (interrupt/teardown, concurrency/counting, fallbacks) — no findings. Full suite 635 passed / 27 skipped; ruff clean.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1b3f284a69
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| existing = base.existing_output(cfg, base.bench_stem(source, task.id)) | ||
| if existing is not None: | ||
| out(f"[yellow]bench: skipping {task.id} (already harvested)[/yellow]") | ||
| written.append(existing) |
There was a problem hiding this comment.
When --resume skips an already-harvested task, this appends that old path to written; if every remaining pending task then fails, the final if attempted and not written check never fires, so the command exits 0 and may publish a stale partial dataset even though no attempted task produced a row. Track newly harvested rows separately from resume-skipped rows, or base the all-attempted-failed check only on rows written during this invocation.
Useful? React with 👍 / 👎.
|
|
||
| try: | ||
| written = run_bench(cfg, console=console, resume=resume, refresh=refresh) | ||
| except RuntimeError as exc: |
There was a problem hiding this comment.
In bench mode this branch sits before the prompt-mode KeyboardInterrupt handler below, but it only catches RuntimeError; if the user presses Ctrl-C after some bench tasks were harvested, run_bench re-raises KeyboardInterrupt and the CLI bypasses _try_write_completed_output_metadata, so the completed rows are left without the README/upload hint that prompt mode writes on interruption. Add a KeyboardInterrupt handler here mirroring the prompt-mode cleanup path.
Useful? React with 👍 / 👎.
| "rewards": run.rewards or {}, | ||
| "agent": cfg.get_agent_provider(), | ||
| "trace_file": f"{split}/{stem}.jsonl", | ||
| **run.metadata, |
There was a problem hiding this comment.
For real native traces, trace_metadata() returns converter metadata that already contains keys such as source (often None for Codex session metadata). Because run.metadata is expanded after the bench sidecar fields, those trace keys can overwrite "source": source_id(source) and other reserved provenance fields, leaving output/metadata/<stem>.json without the benchmark source that produced the reward. Merge trace metadata first or nest/filter it so bench fields remain authoritative.
Useful? React with 👍 / 👎.
| # (no effect on swe-bench, which uses Hugging Face's own dataset cache). | ||
| # Keep bench its own project: dedicated output.traces_dir + publish.repo_id. | ||
| bench: | ||
| sources: [] |
There was a problem hiding this comment.
The generated config template has sources: [] followed by indented example list items; if a user follows the comments and uncomments one of those - { ... } lines, the YAML is invalid because the sequence is nested under an already-complete flow-style empty list. Use sources: with the commented items beneath it (and make the same change in config.example.yaml) so the bench examples can be uncommented directly.
Useful? React with 👍 / 👎.
| # uploads verifier sidecars as dataset provenance.) The bench intermediates normally live | ||
| # in a sibling ``bench`` dir; the name is listed here only to defend a nested override. | ||
| NON_DATA_TRACE_DIR_NAMES = frozenset( | ||
| {"partials", "failures", "verification", "sandbox", "__pycache__", "bench"} |
There was a problem hiding this comment.
This skip list is used by conversion, README generation, and mode detection, but bench and sandbox are ordinary project names too; Cursor native extraction preserves the project directory under the output tree, so a workspace named bench writes real traces under output/bench/... that teich convert and the README will now silently ignore even outside bench mode. Prefer excluding the configured bench working directory explicitly (or a less ambiguous marker) instead of filtering every path component with this name.
Useful? React with 👍 / 👎.
| env["OPENROUTER_API_KEY"] = api_key | ||
| base_url = cfg.get_base_url() | ||
| if base_url: | ||
| env["OPENAI_BASE_URL"] = base_url |
There was a problem hiding this comment.
When swe-bench is run with agent.provider: claude-code and an Anthropic-compatible custom api.base_url, this only passes OPENAI_BASE_URL into the container. The Claude runner used elsewhere in this repo sets ANTHROPIC_BASE_URL, which is the variable the Claude CLI path relies on, so this bench container ignores the configured endpoint and uses the default Anthropic API. Set ANTHROPIC_BASE_URL when cfg.api.provider is anthropic (the mirrored harbor helper has the same mapping).
Useful? React with 👍 / 👎.
| "trace_file": f"{split}/{stem}.jsonl", | ||
| **run.metadata, | ||
| } | ||
| meta_path = cfg.output.traces_dir / "metadata" / f"{stem}.json" |
There was a problem hiding this comment.
This writes the only copy of split/reward into metadata/*.json while the harvested JSONL remains a native trace; however load_traces() only downloads JSONL/README/tools files from Hub snapshots and convert_traces_to_training_data() never joins these sidecars, so published bench datasets lose their reward labels when users load them for reward-based filtering or training. Either embed the verifier outcome into the converted row metadata or teach the loader/converter to fetch and merge these sidecars.
Useful? React with 👍 / 👎.
| local = Path(spec).expanduser() | ||
| if local.exists(): | ||
| return local | ||
| cache_dir = base.bench_root(cfg) / "sources" / _source_slug(spec, source.version, source.repo) |
There was a problem hiding this comment.
When a config uses an explicit local Harbor source such as ./tasks or /tmp/tasks and the directory is missing, this falls through into the remote-download path and asks Harbor to download that filesystem path as a dataset ref, producing a misleading network/auth/spec error instead of the actionable local bench task directory not found error. Treat absolute or explicitly path-like specs as local and fail before _fetch_remote().
Useful? React with 👍 / 👎.
… endpoints) Prompt mode threads api.wire_api into the agent (codex config; pi provider_settings via _pi_provider_api -> openai-completions/openai-responses). Bench does neither — it only sets the model prefix + OPENAI_BASE_URL — so a chat-completions-only endpoint (e.g. provider: openai + base_url: api.z.ai + wire_api: chat_completions) is silently hit on /responses by pi/codex and fails, ignoring the config. Rather than silently misbehave, run_bench now raises when the agent is pi or codex, the provider isn't openrouter (whose prefix routing already selects completions), and api.wire_api is a chat-completions variant. Points the user at provider: openrouter or a Responses-API endpoint. Actually threading wire_api into the bench agents is a larger follow-up. Full suite 636 passed / 27 skipped; ruff clean.
…e_url Step 1 of making bench reuse prompt mode's agent config instead of a thinner, diverging copy. New src/teich/agent_cfg.py holds the pure cfg->env/model/url logic: - provider_env_key: the FULL provider->key map (zai->GLM_API_KEY, deepseek, xai, google, ...). Bench previously only knew openai/openrouter/anthropic, so a provider: zai run got no usable key var. - container_base_url: rewrites localhost/127.0.0.1 -> host.docker.internal so a host-local endpoint is reachable from the container. Bench used raw get_base_url(). - bench_auth_env / pi_prefixed_model: the shared credential+base-url env and pi model-prefix rule. swebench (_auth_env/_model_name) and harbor (_agent_auth_env/_bench_model_name) now delegate to it (deleting their duplicated maps), and runner.py's ExternalCliRunner._provider_env_key + _container_base_url delegate too, so it's one source of truth, not a fork. Next: seed pi models.json/settings.json + codex config.toml into the swe-bench container so wire_api/reasoning/etc. are honored too (replacing the fail-fast guard). Full suite 640 passed / 27 skipped; ruff clean.
2 participants
Summary
Adds a benchmark generation mode —
teich generate --mode bench— that runs reward-labeled benchmark tasks through a pluggable backend (harbororswe-bench), harvests each native trace routed by its verifier score intooutput/{passed,failed,borderline}/, and hardens the run for real Docker use (per-task image purge, working Ctrl-C). Everything is opt-in:--mode prompts(the default),github_repo, and all existing behavior are unchanged, bench lives behind theteich[harbor]/teich[swe]extras, and the only new core runtime dependency isjinja2>=3.1(previously dev-only).What's included
1.
teich generate --mode benchteich generategrows a--mode {prompts,bench}flag (defaultprompts) and routes bench through a shared driver + harvest so backends stay thin.src/teich/bench/runner.pyrun_bench(cfg, console, resume, refresh)is imported lazily fromcli.pysoharbor/swebenchstay optional extras; tasks run through a bounded pool sized bymax_concurrency(default1).src/teich/bench/backends/base.pydefines theBenchBackendprotocol (require/tasks/run, whererunreturns a native trace plus a rewards dict) plus reward routing (primary_score,route_split: score1->passed,0/unscored ->failed, anything else ->borderline) andharvest().output/{passed,failed,borderline}/with scores + provenance written tooutput/metadata/<stem>.json; filenames are namespacedbench-<source>-<task>.bench-<task>.jsonl,bench/**is added toUPLOAD_IGNORE_PATTERNS, and a mode guard warns against mixing bench and prompt rows.src/teich/converter.pyexcludes bench / verification / sandbox intermediates from the dataset view, while verifier sidecars undermetadata/are still uploaded as dataset provenance — this is what makes the dataset "verifiable": each row carries its task's reward outcome.src/teich/config.pyaddsbench.sources(per-sourcetype/source/split/instances/repo/version/backend) andoutput.bench_dir(defaults to a siblingbenchdir besidetraces_dir, outside the dataset);config.example.yamldocuments both.--resumeskips already-harvested bench tasks;--refreshre-downloads remoteharborsources only (swe-bench uses Hugging Face's own dataset cache).2. Harbor backend
src/teich/bench/backends/harbor.pyruns Harbor-format tasks by driving theharborpackage as a library (registry spec, local task dir, or git/HF registry) and reads the verifier's reward dict off the trial env.teich[harbor](harbor>=0.15.0, Python >= 3.12).3. SWE-bench backend
SWE-bench ships an evaluation harness, not runnable tasks, so teich runs the agent itself and grades what it produces.
src/teich/bench/backends/swebench.pychecks out the clean repo@ base_commitinside swebench's prebuilt instance image, renders a thin agent layer on top (src/teich/bench/templates/agent.dockerfile.j2), takes the agent'sgit diffas the candidate patch, and grades it withswebench.run_instance; theresolvedverdict becomes the reward (resolved->passed, unresolved ->failed).teich[swe](swebench>=4.1.0);teich[bench]=teich[harbor,swe].4. Ctrl-C actually stops a run
The driver ran tasks in a
ThreadPoolExecutor, butSIGINTonly reaches the main thread andwith ThreadPoolExecutordoesshutdown(wait=True), so Ctrl-C looked stopped while Docker ground on in the workers.max_concurrency == 1now runs each task inline on the main thread, so Ctrl-C propagates straight into the Docker call and stops cleanly (the smoke-test /--resumepath).max_concurrency > 1cancels queued tasks (cancel_futures) and lets the few already inside a Docker call finish before exit (threads can't be signalled).--resume" hint.5. Per-task Docker image purge
harbor's teardown runs
compose down --rmi local, which does not remove its custom-taggedhb__<task>image, so per-task images accumulated and filled the disk.finally, gated byoutput.keep_bench_images(defaultfalse= purge).hb__<task>read off the trial env, with a deterministicsanitize("hb__" + task)fallback covering the error path where the trial never returned.6. Templated dataset card
src/teich/trace_readme.pyis reworked to render the dataset card fromsrc/teich/templates/dataset_card.md.j2off a presentation-only context dict — byte-identical to the previous hand-built card, locked by a golden test.output.licenseadds an SPDX id to the frontmatter.output.card_extramerges arbitrary extra frontmatter keys (reserved keys dropped, dumped viayaml.safe_dump).output.readme_templatepoints at a user.j2to fully override the card (rendered with the same context); afield_validatorerrors at config load if the file is missing.size_categoriesis auto-populated from the dataset row count using HF's standard buckets (n<1K...n>1T).jinja2>=3.1is promoted from a dev-only to a core dependency; both templates ship in the wheel.Validation
uv run --extra dev python -m pytest -q-> 612 passed, 27 skippeduv run --extra dev python -m pytest -q tests/test_bench.py tests/test_swebench.py tests/test_trace_readme.py tests/test_cli.py tests/test_converter.py-> 168 passed, 2 skipped (bench 31, swe-bench 21, trace_readme 30, cli 29, converter 59)test_rewards_resolved,test_rewards_unresolved_routes_to_failed,test_rewards_resolved_routes_to_passedtest_rendered_card_matches_golden_byte_for_byte, plustest_size_category_bucket_boundaries,license,card_extra, and custom-template-override testsuv run --extra dev ruff check src/teich/bench src/teich/trace_readme.py src/teich/config.py src/teich/cli.py-> All checks passeduv build --wheel-> builtteich-0.2.8-py3-none-any.whl; confirmedteich/templates/dataset_card.md.j2andteich/bench/templates/agent.dockerfile.j2are packagedCompatibility
--mode prompts(the default),github_repo, and every existing config key behave exactly as before; bench is opt-in behindteich[harbor]/teich[swe]/teich[bench], the newoutputknobs all default to today's behavior (keep_bench_imagesdefaults to purge, the card renders byte-identically), and the only new core runtime dependency isjinja2>=3.1(previously dev-only).Note: the offline surface (dataset loading, Dockerfile render, reward mapping, backend selection/registration) is unit-tested above; the swe-bench Docker build -> run agent -> grade path is integration-validated only, since CI has no Docker.