diff --git a/.dockerignore b/.dockerignore index 884b27b4..7ffd6643 100644 --- a/.dockerignore +++ b/.dockerignore @@ -28,3 +28,7 @@ build/ dist/ .idea/ .vscode/ + +# Cargo build output — the image's sandbox-builder stage recompiles run-confined; +# copying host target/ (large) would only bloat the build context. +**/target/ diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md new file mode 100644 index 00000000..3627b824 --- /dev/null +++ b/ARCHITECTURE.md @@ -0,0 +1,284 @@ +# AutoProver — Architecture & High-Level Design + +> A companion to the [README](README.md). The README tells you how to *run* AutoProver; +> this document explains how it is *built* — the major subsystems, the abstractions that +> hold them together, and the data flow through a run. + +## 1. What it is + +AutoProver (internally "AI Composer") is a **multi-agent pipeline that turns a Solidity +codebase + a design document into verified formal specifications.** Given a project root, a +main contract, and a system description, it drives a fleet of LLM agents to analyze the +system, formulate properties, author CVL (Certora Verification Language) specs, and run the +Certora Prover in a feedback loop until the specs verify (or the agent gives up with a +reason). + +The same generic pipeline also powers two sibling workflows — **Foundry test generation** +and **NatSpec greenfield code generation** — which reuse the shared analysis/extraction/ +reporting machinery behind a backend protocol. + +## 2. Design philosophy + +A handful of decisions shape the whole codebase: + +- **Generic driver, pluggable backends.** One backend-agnostic driver + ([composer/pipeline/core.py](composer/pipeline/core.py)) owns the steps that are the same + for everyone (system analysis, property extraction, caching, report assembly). Anything + backend-specific (how a spec is authored and verified) is contributed through a small + protocol. Adding the Foundry backend required *no* changes to the shared steps. +- **Phase-chain immutability.** Each phase produces an immutable object that is the + constructor input to the next: `Backend → PreparedSystem → Formalizer`. The *existence* of + a `Formalizer` proves that system analysis and preparation already succeeded — ordering is + a type-level dependency, not a call-order convention, so there is no half-initialized state. +- **Agents are graphs; everything is checkpointed.** Every agent is a LangGraph state graph + built from a reusable framework ([graphcore/](graphcore/)). State, conversations, and + intermediate results are persisted to Postgres so any run can be resumed, time-traveled, + inspected, or replayed. +- **Tight LLM↔tool loops with hard validation gates.** Agents don't emit free text that is + hoped to be correct — every CVL write passes the Certora type-checker, every spec is run + through the actual prover, and a separate "judge" agent adjudicates whether a property was + legitimately handled. Invalid output is rejected at the tool boundary with actionable + feedback. +- **Deterministic, LLM-free replay for tests.** A "tape" system records real LLM responses + per task and replays them, letting the entire pipeline run end-to-end in CI with no API + calls. + +## 3. The big picture + +``` + ┌──────────────────────────────────────────────┐ + CLI / TUI │ GENERIC PIPELINE DRIVER │ + (console_autoprove, │ composer/pipeline/core.py │ + tui_autoprove) │ │ + │ │ 1. System Analysis ───────► SourceApplication│ + │ builds context │ 2. backend.prepare_system ─► PreparedSystem │ + ▼ │ 3. prepare_formalization ∥ property extraction│ + AIComposerContext │ 4. per-component formalize (parallel) │ + (LLM, RAG, prover opts, │ 5. backend-agnostic Report │ + VFS, handler factory) └───────────────┬────────────────────────────────┘ + │ │ PipelineBackend protocol + │ ┌─────────────────┼─────────────────┐ + ▼ ▼ ▼ ▼ + ┌─────────┐ Prover backend Foundry backend NatSpec workflow + │ graphcore│ (CVL specs + (.t.sol tests + (greenfield stubs + │ agent fw │ Certora Prover) forge test) + CVL) + └────┬─────┘ + │ each agent = LangGraph state graph + tools + ▼ + ┌──────────────────────────────────────────────────────────────────┐ + │ Tools: filesystem/VFS · CVL author+typecheck · prover · RAG │ + │ search · knowledge base · human-in-loop · memory │ + └──────────────────────────────────────────────────────────────────┘ + │ + ▼ + ┌──────────────────────────────────────────────────────────────────┐ + │ Persistence (Postgres): rag_db · langgraph_store_db · │ + │ langgraph_checkpoint_db · memory_tool_db · audit_db │ + └──────────────────────────────────────────────────────────────────┘ +``` + +## 4. The generic pipeline driver + +[composer/pipeline/core.py](composer/pipeline/core.py) is the spine. `run_pipeline()` executes +five steps and never inspects anything backend-specific: + +1. **System analysis** (shared). Runs `run_component_analysis` to produce a + `SourceApplication` — the contracts, components, external actors, and their interactions. + Always yields the same type regardless of backend. +2. **`backend.prepare_system(analyzed)`** — the backend's transform. The prover backend lifts + the source app into a *harnessed* application (generating harness contracts for external + dependencies); the Foundry backend is an identity transform. +3. **`prepared.prepare_formalization()` runs concurrently with property extraction.** Neither + depends on the other, so the prover's expensive AutoSetup/summary/invariant work overlaps + with per-component property inference. Property extraction fans out one agent per component, + bounded by a semaphore (`--max-concurrent`). +4. **Per-component formalization** (parallel). For each component's properties, the backend's + `Formalizer.formalize()` is invoked. Results are cached by the backend's result type. +5. **Report assembly** (shared, best-effort). The driver collects per-component verdicts via a + backend-supplied `fetch_verdicts` callback, an LLM groups properties into semantic clusters, + and a coverage check validates the result. A failure here never fails the run. + +### The backend contract + +A backend implements `PipelineBackend[P, FormT, H, A]` plus three phase objects: + +| Object | Responsibility | Prover impl | Foundry impl | +|---|---|---|---| +| `PipelineBackend` | Phase-enum map, analysis prompt, artifact store, `prepare_system` | [spec/source/pipeline.py](composer/spec/source/pipeline.py) | [foundry/pipeline.py](composer/foundry/pipeline.py) | +| `PreparedSystem` | Holds the located main contract; builds the `Formalizer` | harness-lifted app + AutoSetup/invariants | identity | +| `Formalizer` | `formalize()` one component; `fetch_verdicts`; `finalize` | `batch_cvl_generation` + prover | `batch_foundry_test_generation` + `forge test` | + +The phase enum (`CorePhases`) lets each backend label its own phases while the driver tags the +three universal ones (analysis / extraction / formalization) for the UI and the replay tapes. + +## 5. The agent framework (graphcore + workflow) + +Every LLM agent in the system is a LangGraph state graph, constructed through a reusable, +type-safe framework. + +- **[graphcore/](graphcore/)** is a standalone library (its own package, used by other Certora + products too). It provides a fluent `Builder[State, Context, Input]` that binds a model, + state type, tools, system/initial prompt templates, and an optional summarization config, + then compiles a checkpointed `StateGraph`. The agent loop is the standard + *LLM → tool calls → tool results → LLM …* cycle, terminating when the model produces a + final structured output instead of more tool calls. +- **Tools** are Pydantic models with `run()` implementations and mixins for injecting graph + state / tool-call id. Tools return either a string, a `Command` that merges into state, or an + `interrupt()` that pauses the graph for human input. +- **[composer/workflow/](composer/workflow/)** wires graphcore to this application: model + capability parsing ([llm.py](composer/workflow/llm.py) — thinking budgets, interleaved + thinking, memory-tool beta, cache control), Postgres checkpointer + store services + ([services.py](composer/workflow/services.py)), and summarization tuned for long spec runs + ([summarization.py](composer/workflow/summarization.py)). +- **[composer/io/](composer/io/)** is the execution and observability layer. `run_task` + ([multi_job.py](composer/io/multi_job.py)) wraps each schedulable unit with a handler factory, + an optional concurrency semaphore, and lifecycle hooks. Graph execution emits an immutable + event stream (Start / StateUpdate / Checkpoint / CustomUpdate / End) onto a lock-free queue; + a background drainer dispatches events to the active IO handler (console or TUI). Nested + sub-agent graphs are tracked transparently so handlers can reconstruct the full call path. + +### State, context, and editing + +- `AIComposerState` ([core/state.py](composer/core/state.py)) extends LangGraph's message state + with a virtual filesystem (VFS), validation results, and the working spec being edited. +- `AIComposerContext` ([core/context.py](composer/core/context.py)) is the run-scoped dependency + container: the bound LLM, RAG connection, prover options, and VFS materializer. +- Spec edits go through `replace_unique` ([core/edit.py](composer/core/edit.py)) — a surgical, + match-exactly-once text replacement that fails with actionable guidance on ambiguity, keeping + the model honest about what it's changing and saving tokens. + +## 6. The prover (default) backend — phase by phase + +Implemented under [composer/spec/source/](composer/spec/source/). The phases map onto the +README's Phase 0–5: + +- **System analysis** ([spec/system_analysis.py](composer/spec/system_analysis.py), + [system_model.py](composer/spec/system_model.py)) — an agent reads the design doc and source + to produce a `SourceApplication` of contracts → components, each with entry points, state + variables, interactions, and requirements. +- **Harness setup** ([spec/source/harness.py](composer/spec/source/harness.py)) — a classifier + agent categorizes external contracts (singleton / multiple / dynamic, ERC20s, interfaces) and + generates harness contracts, lifting the system to a `HarnessedApplication`. +- **AutoSetup** ([spec/source/autosetup.py](composer/spec/source/autosetup.py)) — shells out to + the [certora_autosetup/](certora_autosetup/) package to compile the project and produce a + prover `compilation_config.conf` plus summaries for known externals. +- **Custom summaries** ([spec/source/summarizer.py](composer/spec/source/summarizer.py)) — + generates CVL summaries for ERC20s and external interfaces. +- **Structural invariants** ([spec/source/struct_invariant.py](composer/spec/source/struct_invariant.py)) + — a two-agent loop: one proposes invariants, a judge accepts/rejects each (not structural / + not inductive / unlikely to hold / …). Survivors become `certora/specs/invariants.spec`, + importable by later phases. +- **Per-component property extraction** ([spec/prop_inference.py](composer/spec/prop_inference.py)) + — multi-round agent producing `PropertyFormulation`s (attack vectors, safety properties, + invariants), optionally refined interactively or against a threat model. +- **CVL generation** ([spec/cvl_generation.py](composer/spec/cvl_generation.py), + [spec/source/author.py](composer/spec/source/author.py)) — the core feedback loop. The agent + authors CVL with `put_cvl`/`edit_cvl` (type-checked on every write), runs the prover via a + `verify_spec` tool, analyzes any counterexamples, and revises. A property-feedback judge + validates coverage and adjudicates the agent's objections (e.g. "this property is vacuous + because…"). Output is a `GeneratedCVL` carrying the spec, skipped properties with reasons, + the property→rule mapping, and the final prover run link. + +### Outputs and artifacts + +`ArtifactStore` ([spec/artifacts.py](composer/spec/artifacts.py), prover subclass in +[spec/source/](composer/spec/source/)) owns the on-disk layout under the project's `certora/`: +specs in `certora/specs/`, configs in `certora/confs/`, per-spec metadata (properties, +property→rules, commentary) in `certora/properties/`, and a final `certora/ap_report/report.json`. + +## 7. Caching & resumption + +Two complementary mechanisms: + +- **Phase cache** (`--cache-ns`). `WorkflowContext` / `CacheKey` + ([spec/context.py](composer/spec/context.py)) form a hierarchical, type-parameterized cache + in the LangGraph store: `None → SourceApplication → Properties → ComponentGroup → CVLGeneration`. + Each key incorporates a hash of its inputs, so changing the project, doc, or contract + invalidates exactly the affected subtree. Repeated runs skip completed phases. +- **Checkpointing** (`--thread-id` / `--checkpoint-id`). LangGraph persists graph state after + every node to `langgraph_checkpoint_db`, enabling crash recovery and "time travel" — resuming + from any prior checkpoint, even a non-latest one. + +## 8. Prover integration + +[composer/prover/](composer/prover/) abstracts running the Certora Prover. `run_prover` +([core.py](composer/prover/core.py)) spawns `certoraRun`, streams stdout, and resolves results +through either a **local** results path or a **cloud** path ([cloud.py](composer/prover/cloud.py), +polled via job URL) depending on `ProverOptions.cloud`. Violated rules are fed to a +counterexample analyzer ([analysis.py](composer/prover/analysis.py)) whose findings go back to +the authoring agent. A callback protocol streams per-rule outcomes to the UI and the audit DB. + +## 9. Knowledge: RAG + curated KB + +- **RAG** ([composer/rag/](composer/rag/)) — the CVL manual, chunked and embedded + (`nomic-embed-text-v1.5`) into `rag_db` (pgvector, with a ChromaDB fallback). Agents query it + for CVL syntax/semantics during authoring. Read-only at runtime; rebuilt offline from the docs. +- **Curated KB** ([composer/kb/](composer/kb/)) — ~30 hand-written articles on CVL pitfalls + (vacuity traps, ghost semantics, summary misapplication…) stored in the LangGraph store and + searched semantically by symptom. Agents can also contribute new articles (`KBPut`). + +## 10. Alternate backends & workflows + +- **Foundry backend** ([composer/foundry/](composer/foundry/)) — same driver, but formalizes + properties as Solidity `.t.sol` tests verified by `forge test` instead of CVL + prover. + Verdicts come from test pass/expected-failure status. Entry points: + [cli/console_foundry.py](composer/cli/console_foundry.py), `tui_foundry.py`. +- **NatSpec** ([composer/spec/natspec/](composer/spec/natspec/)) — a *greenfield* workflow + (its own asyncio orchestrator, not the generic driver) that goes from a design doc to Solidity + interfaces, stub implementations, and CVL. A semaphore-serialized "semantic registry" + ([natspec/registry.py](composer/spec/natspec/registry.py)) lets parallel agents share and + reuse generated state fields without conflicting. + +## 11. CLI, TUI & observability + +- **Entry points** ([composer/cli/](composer/cli/)) — `console-autoprove` (headless, prints to + stdout; best for `print`/log debugging) and `tui-autoprove` (Textual UI with per-phase panels + and live prover-output logs). Both build the context and call the same pipeline; CLI args are + parsed by introspecting `Annotated` protocol definitions in [composer/input/](composer/input/). +- **TUI** ([composer/ui/](composer/ui/)) — observes the pipeline purely through the IO event + stream, rendering per-task lanes, tool calls, and streamed prover output. +- **Diagnostics** ([composer/diagnostics/](composer/diagnostics/) + [scripts/](scripts/)) — + per-phase timing/token aggregation; `snapshot_viewer.py` replays a single agent's conversation + by mnemonic; `traceDump.py` renders a full run to HTML; `autoprove_cache_explorer.py` inspects + (and edits) cached phase results and agent memories. + +## 12. Testing — deterministic replay tapes + +[composer/testing/](composer/testing/) solves end-to-end testing without an LLM. A +`TapeRecorder` captures real `AIMessage` responses **per task, in call order** (including +out-of-graph calls like CEX analysis and interleaved sub-agents). `HarnessFakeLLM` replays +them by routing on the active task id (a `ContextVar` set by `run_task`). Tapes are +human-editable Python modules with embedded JSON; smoke scenarios live in +[test_scenarios/](test_scenarios/). Recording is curated into a clean tape before use (see the +`generate-tape` and `inspect-run` skills). + +## 13. Persistence map + +All state lives in Postgres (a single `pgvector/pgvector:pg16` container provisions all five): + +| Database | Holds | +|---|---| +| `rag_db` | CVL-manual embeddings for RAG search | +| `langgraph_store_db` | LangGraph document/index store — phase cache, KB articles | +| `langgraph_checkpoint_db` | Per-node workflow checkpoints (resume / time-travel) | +| `memory_tool_db` | Hierarchical LLM context memory (per agent) | +| `audit_db` | Run history, VFS snapshots, prover results, summaries (resumption + `traceDump`) | + +## 14. Top-level packages at a glance + +| Package | Role | +|---|---| +| [composer/](composer/) | The application — pipeline, agents, backends, tools, UI | +| [graphcore/](graphcore/) | Reusable LangGraph agent-building framework (separate package) | +| [certora_autosetup/](certora_autosetup/) | Solidity project analysis, compilation, harness/conf generation (Phase 1) | +| [analyzer/](analyzer/) | Standalone counterexample analyzer (also used inline by the prover backend) | +| [sanity_analyzer/](sanity_analyzer/) | Diagnoses unsatisfiable / sanity-failed prover runs | +| [scripts/](scripts/) | Docker entry, DB/RAG setup, trace & snapshot debugging tools | +| [tests/](tests/), [test_scenarios/](test_scenarios/) | Unit tests and tape-based smoke scenarios | + +--- + +*This document is a high-level map. For runtime/setup details see [README.md](README.md) and +[AICOMPOSER_INFRA.md](AICOMPOSER_INFRA.md); for the canonical contract between the driver and a +backend, read the module docstring and protocols in +[composer/pipeline/core.py](composer/pipeline/core.py).* diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 00000000..e662afc0 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,29 @@ +# Repository conventions for coding agents + +Guidance for AI coding agents (and humans) working in this repo. Keep changes consistent +with these unless a maintainer says otherwise. + +## Python + +### Do NOT use `from __future__ import annotations` + +Never add `from __future__ import annotations` to a module. If you touch a file that has it +and it's reasonable to do so, remove it. + +Why: +- It's a dead-end feature. PEP 563 (the string-annotations behaviour this import enables) was + never made the default and has effectively been superseded by PEP 649 / PEP 749 (lazy + evaluation) from Python 3.14 on. Relying on the `__future__` behaviour is betting on a path + the language is moving away from — it will change/break under you in future versions. +- It doesn't actually solve the problem it's reached for. Stringizing *all* annotations breaks + anything that introspects them at runtime — pydantic, dataclasses, `typing.get_type_hints`, + and our own annotation-driven graph wiring (see `composer/rustapp/_llm_agent.py`, which had + to stay eager precisely because stringized `NotRequired[T]` broke pydantic unwrapping). It + trades one set of problems for a subtler set. + +What to do instead (we target Python 3.12+): +- Modern syntax works at runtime without the future import: `X | None`, `list[str]`, + `dict[str, int]`, `tuple[int, ...]`, PEP 695 generics (`class Foo[T]:` / `def f[T]()`). +- For a genuine forward reference (a name not yet defined where the annotation is evaluated — + e.g. a dataclass field typed as a class defined later in the file), quote just that one + annotation: `backend: "RustBackend"`. Quote the specific ref; don't stringize the whole module. diff --git a/composer/crucible_launch.py b/composer/crucible_launch.py new file mode 100644 index 00000000..4bc14986 --- /dev/null +++ b/composer/crucible_launch.py @@ -0,0 +1,22 @@ +"""Console entry points for the Crucible (Solana fuzzing) application. + +Crucible is now a *pure-Rust app* (``docs/rust-pure-app.md``): the ``crucible_app`` wheel + its +descriptor define everything (the shared fixture, crate deliverable, workspace prep, sandbox +grants, verdict summary), so these are the same two-line shims echoprover uses — no +Crucible-specific Python package. ``import composer.bind`` runs first (inside +``composer.rustapp.cli``) for the import-time DI / test-tape bootstrap. +""" + +from composer.rustapp.cli import console_main, tui_main + +_MODULE = "crucible_app" + + +def console_crucible() -> int: + """Run the Crucible application in console mode.""" + return console_main(_MODULE) + + +def tui_crucible() -> int: + """Run the Crucible application in the Textual TUI.""" + return tui_main(_MODULE) diff --git a/composer/foundry/pipeline.py b/composer/foundry/pipeline.py index ef7c323e..2ff056c4 100644 --- a/composer/foundry/pipeline.py +++ b/composer/foundry/pipeline.py @@ -106,7 +106,7 @@ class FoundryPhase(enum.Enum): TEST_GENERATION = "test_generation" REPORT = "report" -class FoundryFormalizer(Formalizer[GeneratedFoundryTest]): +class FoundryFormalizer(Formalizer[GeneratedFoundryTest, ContractComponentInstance]): def __init__(self, conf: _ForgeRunConfig): super().__init__(GeneratedFoundryTest, "foundry") self.conf = conf @@ -138,11 +138,11 @@ async def fetch_verdicts(self, inp: ReportComponentInput[GeneratedFoundryTest]) return await _foundry_verdicts(inp) @dataclass -class FoundrySystem(PreparedSystem[GeneratedFoundryTest]): +class FoundrySystem(PreparedSystem[GeneratedFoundryTest, ContractComponentInstance]): form: FoundryFormalizer @override - async def prepare_formalization(self, run: PipelineRun) -> Formalizer[GeneratedFoundryTest]: + async def prepare_formalization(self, run: PipelineRun) -> Formalizer[GeneratedFoundryTest, ContractComponentInstance]: return self.form @dataclass @@ -166,7 +166,7 @@ async def prepare_system( self, analyzed: SourceApplication, run: PipelineRun[FoundryPhase, None] - ) -> PreparedSystem[GeneratedFoundryTest]: + ) -> PreparedSystem[GeneratedFoundryTest, ContractComponentInstance]: return FoundrySystem( main_instance( analyzed, run.source diff --git a/composer/io/context.py b/composer/io/context.py index c2fa1ba3..d6a8e326 100644 --- a/composer/io/context.py +++ b/composer/io/context.py @@ -138,6 +138,23 @@ def emit_custom_event(payload: Mapping[str, Any]): raise ValueError("No IO handler installed") curr_io[0].push(ProgressEvent(dict(payload))) + +def push_custom_update( + payload: Mapping[str, Any], *, thread_id: str, checkpoint_id: str = "" +) -> bool: + """Push a ``CustomUpdate`` to the current ``with_handler`` scope's queue so it + reaches ``EventHandler.handle_event`` — the out-of-graph analogue of + ``get_stream_writer()``. Used by backends (e.g. the Rust IoC loop) that emit + domain events *between* graph calls, where ``get_stream_writer()`` is unavailable. + Returns ``False`` (event dropped) if no handler scope is installed.""" + curr_io = _io_handler.get() + if curr_io is None: + return False + curr_io[0].push( + CustomUpdate(payload=dict(payload), thread_id=thread_id, checkpoint_id=checkpoint_id) + ) + return True + async def run_graph[S: StateLike, C: StateLike | None, I: StateLike]( graph: CompiledStateGraph[S, C, I, Any], ctxt: C, diff --git a/composer/pipeline/cli.py b/composer/pipeline/cli.py index c398bf9f..073a076e 100644 --- a/composer/pipeline/cli.py +++ b/composer/pipeline/cli.py @@ -20,6 +20,7 @@ CorePipelineResult ) from composer.spec.artifacts import ArtifactIdentifier +from composer.spec.system_model import FeatureUnit from composer.spec.service_host import ModelProvider from composer.spec.system_analysis import SolidityIdentifier from .core import PipelineBackend, run_pipeline @@ -113,10 +114,10 @@ class StagedPipeline: root_key: str class Continuation[P: enum.Enum, H](Protocol): - async def __call__[FormT: BackendResult, A: ArtifactIdentifier]( + async def __call__[FormT: BackendResult, A: ArtifactIdentifier, U: FeatureUnit]( self, env: ServiceHost, - backend: PipelineBackend[P, FormT, H, A] + backend: PipelineBackend[P, FormT, H, A, U] ) -> CorePipelineResult[FormT]: ... @@ -248,9 +249,9 @@ async def cli_pipeline[P: enum.Enum, H]( relative_path=init_source.relative_path ) - async def cont[FormT: BackendResult, A: ArtifactIdentifier]( + async def cont[FormT: BackendResult, A: ArtifactIdentifier, U: FeatureUnit]( env: ServiceHost, - backend: PipelineBackend[P, FormT, H, A] + backend: PipelineBackend[P, FormT, H, A, U] ) -> CorePipelineResult[FormT]: full_ctx = WorkflowContext.create( services=conns.memory, diff --git a/composer/pipeline/core.py b/composer/pipeline/core.py index 97ce1b0b..5fe7ab57 100644 --- a/composer/pipeline/core.py +++ b/composer/pipeline/core.py @@ -17,10 +17,9 @@ import enum import logging from dataclasses import dataclass -from typing import Protocol +from typing import Protocol, Any, cast from abc import ABC, abstractmethod -from pydantic import BaseModel from composer.io.multi_job import TaskInfo from composer.spec.artifacts import ArtifactStore @@ -28,7 +27,7 @@ WorkflowContext, CacheKey, Properties, ComponentGroup, SourceCode ) from composer.spec.system_model import ( - SourceApplication, ContractInstance, ContractComponentInstance, AnyApplication + SourceApplication, FeatureUnit ) from composer.spec.types import PropertyFormulation, ArtifactIdentifier from composer.spec.system_analysis import run_component_analysis @@ -40,6 +39,10 @@ from composer.spec.source.report.schema import RuleName, ReportBackend from composer.spec.source.report import build as report_build from composer.spec.source.task_ids import SYSTEM_ANALYSIS_TASK_ID, REPORT_TASK_ID +# The ecosystem seam supplies the domain-specific front half (analyzed model type, prompts, +# analysis validation, unit enumeration). ``main_instance`` moved here too and is re-exported +# so existing EVM backends keep doing ``from composer.pipeline.core import main_instance``. +from composer.pipeline.ecosystem import Ecosystem, EVM, main_instance from .ptypes import ( BackendJob, BackendResult, ComponentOutcome, CorePhases, CorePipelineResult, Delivered, GaveUp, PipelineRun, SystemAnalysisSpec ) @@ -49,18 +52,22 @@ _log = logging.getLogger(__name__) @dataclass -class Formalizer[FormT: BackendResult](ABC): +class Formalizer[FormT: BackendResult, U: FeatureUnit](ABC): """Immutable, fully constructed by prepare_formalization. Carries the prover's config/resources/prover_tool/invariant-results (or nothing, for foundry) as constructor - state — never set post-hoc. `FormT: ReportableResult` is what makes the report a core step.""" + state — never set post-hoc. `FormT: ReportableResult` is what makes the report a core step. + + Generic over ``U``, the *formalized unit* type it consumes (EVM's ``ContractComponentInstance``, + a Rust backend's ``FeatureUnit``): the backend works with its concrete unit — reading its + members without casts — while the driver stays unit-agnostic.""" formalized_type: type[FormT] backend_tag: ReportBackend - + @abstractmethod async def formalize( self, label: str, - feat: ContractComponentInstance, + feat: U, props: list[PropertyFormulation], ctx: WorkflowContext[FormT], run: PipelineRun @@ -77,20 +84,22 @@ async def fetch_verdicts(self, inp: ReportComponentInput[FormT]) -> dict[RuleNam off-thread. Foundry: read straight off inp.formalized.result.""" ... - async def finalize(self, outcomes: list[ComponentOutcome[FormT]], run: PipelineRun) -> None: + async def finalize(self, outcomes: list[ComponentOutcome[FormT, U]], run: PipelineRun) -> None: """Emit any backend-specific run-level artifacts from the full outcome set (prover: components_to_prover_runs.json). Default: none.""" return None @dataclass -class PreparedSystem[FormT: BackendResult](ABC): - main: ContractInstance +class PreparedSystem[FormT: BackendResult, U: FeatureUnit](ABC): + # The located "main" unit is ecosystem-specific (EVM ContractInstance, Solana program, …); + # only the ecosystem's own ``units()`` consumes it, so the base keeps it untyped. + main: Any @abstractmethod - async def prepare_formalization(self, run: PipelineRun) -> Formalizer[FormT]: ... + async def prepare_formalization(self, run: PipelineRun) -> Formalizer[FormT, U]: ... -class PipelineBackend[P: enum.Enum, FormT: BackendResult, H, A: ArtifactIdentifier](Protocol): +class PipelineBackend[P: enum.Enum, FormT: BackendResult, H, A: ArtifactIdentifier, U: FeatureUnit](Protocol): @property def backend_guidance(self) -> str: ... @@ -106,35 +115,26 @@ def artifact_store(self) -> ArtifactStore[A, FormT]: ... async def prepare_system( self, analyzed: SourceApplication, run: PipelineRun[P, H] - ) -> PreparedSystem[FormT]: ... + ) -> PreparedSystem[FormT, U]: ... - def to_artifact_id(self, c: ContractComponentInstance) -> A: ... + def to_artifact_id(self, c: U) -> A: ... # ---- shared helpers (the de-duplicated cache keys + batch) ------------------- PROPERTIES_KEY = CacheKey[None, Properties]("properties") -def main_instance(app: AnyApplication, source: SourceCode) -> ContractInstance: - """Locate the application's main contract — the one whose solidity identifier matches - ``source.contract_name`` — and return a ``ContractInstance`` pointing at it. Backends call this - from ``prepare_system`` to seed the per-component loop; component analysis should already have - guaranteed the contract is present (via ``expected_main_id``).""" - for i, c in enumerate(app.contract_components): - if c.solidity_identifier == source.contract_name: - return ContractInstance(i, app) - raise ValueError(f"main contract {source.contract_name!r} not found in analyzed application") - - @dataclass -class _Batch(BackendJob): +class _Batch[U: FeatureUnit](BackendJob[U]): feat_ctx: WorkflowContext[ComponentGroup] -def _component_cache_key(c: ContractComponentInstance) -> CacheKey[Properties, ComponentGroup]: - return CacheKey(string_hash("|".join([c.app.model_dump_json(), str(c.ind), str(c._contract.ind)]))) +def _component_cache_key(c: FeatureUnit) -> CacheKey[Properties, ComponentGroup]: + # ``cache_material`` is the ecosystem-agnostic view of what identifies a unit; EVM's + # implementation reproduces the previous inline key (app JSON | ind | contract ind) exactly. + return CacheKey(string_hash(c.cache_material())) -def _batch_cache_key[FormT: BaseModel](props: list[PropertyFormulation]) -> CacheKey[ComponentGroup, FormT]: +def _batch_cache_key(props: list[PropertyFormulation]) -> CacheKey[ComponentGroup, Any]: return CacheKey(string_hash("|".join(p.model_dump_json() for p in props))) @@ -146,28 +146,33 @@ def formalize_task_id(idx: int) -> str: return f"formalize-{idx}" # ---- the driver -------------------------------------------------------------- -async def run_pipeline[P: enum.Enum, FormT: BackendResult, H, A: ArtifactIdentifier]( - backend: PipelineBackend[P, FormT, H, A], +async def run_pipeline[P: enum.Enum, FormT: BackendResult, H, A: ArtifactIdentifier, U: FeatureUnit]( + backend: PipelineBackend[P, FormT, H, A, U], run: PipelineRun[P, H], *, interactive: bool = False, threat_model: Document | None = None, max_bug_rounds: int = 3, + ecosystem: Ecosystem[Any, Any, Any] = EVM, ) -> CorePipelineResult[FormT]: + # ``ecosystem`` supplies the domain-specific front half; it defaults to ``EVM``, which + # reproduces the previous hardcoded Solidity behavior exactly, so EVM callers (and cli.py) + # need pass nothing. Non-EVM backends (e.g. the Rust/Crucible backend) pass ``ecosystem=SOLANA``. spec, phases = backend.analysis_spec, backend.core_phases source = run.source - # 1. System analysis (shared primitive, backend-parameterized; always yields SourceApplication). + # 1. System analysis (shared primitive; the ecosystem supplies the analyzed model type, + # prompts, validation, and front-matter — EVM reproduces prior behavior exactly). analyzed = await run.runner( TaskInfo(SYSTEM_ANALYSIS_TASK_ID, "System Analysis", phases["analysis"]), lambda: run_component_analysis( - ty=SourceApplication, child_ctxt=run.ctx.child(CacheKey(spec.analysis_key)), - input=source, env=run.env, extra_input=[ - f"The main entry point of this application has been explicitly identified as {source.contract_name} at relative path {source.relative_path}. " - "Your output MUST contain an explicit contract instance with this solidity identifier.", - *spec.extra_input - ], + ty=ecosystem.system_model, child_ctxt=run.ctx.child(CacheKey(spec.analysis_key)), + input=source, env=run.env, + extra_input=[*ecosystem.analysis_extra_input(source), *spec.extra_input], expected_main_id=source.contract_name, + system_template=ecosystem.analysis_prompts.system, + initial_template=ecosystem.analysis_prompts.initial, + validate=ecosystem.validate_analysis, ), ) if analyzed is None: @@ -180,14 +185,21 @@ async def run_pipeline[P: enum.Enum, FormT: BackendResult, H, A: ArtifactIdentif # this preserves the prover's autosetup ∥ bug-analysis overlap, generically. formalizer_task = asyncio.create_task(prepared.prepare_formalization(run)) - batches = await _extract_all(prepared.main, backend.backend_guidance, run, - phases["extraction"], interactive, threat_model, max_bug_rounds) + # Extraction yields ``FeatureUnit`` batches (the ecosystem is invariant in its unit type and + # callers pass a concrete chain, so it can't be tied to the backend's ``U`` at the signature). + # The paired backend guarantees these units are its ``U``; widen once, here, honestly. + batches: list[_Batch[U]] = cast( + "list[_Batch[U]]", + await _extract_all( + prepared.main, backend.backend_guidance, run, + phases["extraction"], interactive, threat_model, max_bug_rounds, ecosystem), + ) formalizer = await formalizer_task if not batches: raise ValueError("No properties extracted from any component.") # 4. Per-component formalization. Caching is core-owned, keyed by the backend's result type. - async def _run(batch: _Batch) -> ComponentOutcome[FormT]: + async def _run(batch: _Batch[U]) -> ComponentOutcome[FormT, U]: result_key = backend.to_artifact_id(batch.feat) backend.artifact_store.write_properties(result_key, batch.props) child : WorkflowContext[FormT] = await batch.feat_ctx.child( @@ -196,11 +208,11 @@ async def _run(batch: _Batch) -> ComponentOutcome[FormT]: cached_result: FormT | None = await child.cache_get(formalizer.formalized_type) result : FormT | GaveUp if cached_result is None: - label = f"{batch.feat.component.name} ({len(batch.props)} properties)" + label = f"{batch.feat.display_name} ({len(batch.props)} properties)" result : FormT | GaveUp = await run.runner( TaskInfo( - formalize_task_id(batch.feat.ind), - f"{batch.feat.component.name} ({len(batch.props)} properties)", + formalize_task_id(batch.feat.unit_index), + label, phases["formalization"] ), lambda: formalizer.formalize(label, batch.feat, batch.props, child, run), @@ -228,7 +240,7 @@ async def _run(batch: _Batch) -> ComponentOutcome[FormT]: # never fails the run. inputs = [ ReportComponentInput( - name=o.feat.component.name, + name=o.feat.display_name, props=o.props, formalized=o.result if isinstance(o.result, Delivered) else None, ) @@ -251,32 +263,67 @@ async def _run(batch: _Batch) -> ComponentOutcome[FormT]: return _tally(outcomes) async def _extract_all[P: enum.Enum, H]( - main: ContractInstance, backend_guidance: str, run: PipelineRun[P, H], + main: Any, backend_guidance: str, run: PipelineRun[P, H], phase: P, interactive: bool, threat_model: Document | None, max_rounds: int, -) -> list[_Batch]: + ecosystem: Ecosystem[Any, Any, Any], +) -> list[_Batch[FeatureUnit]]: prop_ctx = run.ctx.child(PROPERTIES_KEY) - async def _one(idx: int) -> _Batch | None: - feat = ContractComponentInstance(_contract=main, ind=idx) - feat_ctx = await prop_ctx.child(_component_cache_key(feat), - {"component": feat.component.model_dump()}) + async def _unit_ctx(unit: FeatureUnit) -> WorkflowContext[ComponentGroup]: + return await prop_ctx.child(_component_cache_key(unit), unit.context_tag()) + + async def _extract(unit: FeatureUnit) -> tuple[WorkflowContext[ComponentGroup], list[PropertyFormulation]]: + ctx = await _unit_ctx(unit) props = await run.runner( - TaskInfo(extract_task_id(idx), feat.component.name, phase), + TaskInfo(extract_task_id(unit.unit_index), unit.display_name, phase), lambda conv: run_property_inference( - feat_ctx, run.env, feat, refinement=conv if interactive else None, - threat_model=threat_model, max_rounds=max_rounds, backend_guidance=backend_guidance), + ctx, run.env, unit, refinement=conv if interactive else None, + threat_model=threat_model, max_rounds=max_rounds, backend_guidance=backend_guidance, + system_template=ecosystem.property_prompts.system, + initial_template=ecosystem.property_prompts.initial), ) - return _Batch(feat, props, feat_ctx) if props else None + return ctx, props + + # Both strategies reduce to a list of (unit, properties, unit-context) triples; the ecosystem + # differs only in the *fan direction*. Global extraction infers once over the whole program + # and fans each property into its own unit (one prop per batch); per-component infers per unit + # (concurrently) and keeps that unit's properties grouped. The ``_Batch`` build below is shared. + # (Extraction is unit-agnostic — over the ``FeatureUnit`` protocol; run_pipeline's single cast + # reunites the batches with the paired backend's concrete unit type ``U``.) + triples: list[tuple[FeatureUnit, list[PropertyFormulation], WorkflowContext[ComponentGroup]]] + if ecosystem.global_extraction: + assert ecosystem.extraction_unit is not None + ext_unit = ecosystem.extraction_unit(main) + ext_ctx, props = await _extract(ext_unit) + if ecosystem.collapse_units: + # One whole-program batch: every invariant is formalized into a single harness + run + # (docs/crucible-unit-granularity.md §3). Empty props → no batch (nothing to formalize). + triples = [(ext_unit, props, ext_ctx)] if props else [] + else: + assert ecosystem.property_unit is not None + triples = [] + for i, prop in enumerate(props): + unit = ecosystem.property_unit(main, prop, i) + triples.append((unit, [prop], await _unit_ctx(unit))) + else: + units = ecosystem.units(main) + extracted = await asyncio.gather(*[_extract(u) for u in units]) + triples = [(u, props, ctx) for u, (ctx, props) in zip(units, extracted) if props] - got = await asyncio.gather(*[_one(i) for i in range(len(main.contract.components))]) - return [b for b in got if b is not None] + return [_Batch(unit, props, ctx) for unit, props, ctx in triples] -def _tally[FormT: BackendResult](outcomes: list[ComponentOutcome[FormT]]) -> CorePipelineResult[FormT]: +def _tally[FormT: BackendResult, U: FeatureUnit]( + outcomes: list[ComponentOutcome[FormT, U]] +) -> CorePipelineResult[FormT]: failures: list[str] = [] for o in outcomes: if isinstance(o.result, BaseException): - failures.append(f"{o.feat.component.name}: {o.result}") + failures.append(f"{o.feat.display_name}: {o.result}") elif isinstance(o.result, GaveUp): - failures.append(f"{o.feat.component.name}: GAVE_UP: {o.result.reason}") - return CorePipelineResult(len(outcomes), sum(len(o.props) for o in outcomes), outcomes, failures) + failures.append(f"{o.feat.display_name}: GAVE_UP: {o.result.reason}") + # The rollup is unit-agnostic; widen the concrete-unit outcomes to the protocol for storage. + return CorePipelineResult( + len(outcomes), sum(len(o.props) for o in outcomes), + cast(list[ComponentOutcome[FormT, FeatureUnit]], outcomes), failures, + ) diff --git a/composer/pipeline/ecosystem.py b/composer/pipeline/ecosystem.py new file mode 100644 index 00000000..c57338b6 --- /dev/null +++ b/composer/pipeline/ecosystem.py @@ -0,0 +1,324 @@ +"""The ecosystem seam — the *front half* of the pipeline made parametric over the +blockchain/source domain (see ``docs/ecosystem-abstraction.md``). + +An **ecosystem** bundles everything the shared analysis + property-extraction steps need +that is domain-specific: the system-model type they produce, the analysis/property prompt +templates, connectivity validation, the main-unit locator, and the per-unit enumeration. +It factors into a **language** facet (the conventions for reading the *analyzed* program's +source — Solidity, Rust — shared across chains that use the same language) and the **chain** +facet (the platform model + prompts). The language here is that of the *code under analysis*, +not the language the AutoProver backend is implemented in (see :class:`Language`). + +Phase 1 introduces the seam and captures today's behavior as ``EVM = SOLIDITY ⊕ evm`` — +a *move*, not a rewrite: the existing `SourceApplication`, prompt templates, +`_validate_connectivity`, `main_instance`, and unit enumeration become the EVM ecosystem's +members. The driver defaults to ``EVM``, so existing applications are unchanged. Solana / +Soroban chains and the prompt-fragment split land in later phases. +""" + +from dataclasses import dataclass +from typing import Any, Callable, Literal + +from composer.spec.context import SourceCode +from composer.spec.code_explorer import CODE_EXPLORER_SYS_PROMPT +from composer.spec.system_analysis import _validate_connectivity +from composer.spec.system_model import ( + AnyApplication, + BaseApplication, + ContractComponentInstance, + ContractInstance, + FeatureUnit, + SolidityIdentifier, + SourceApplication, +) +from composer.spec.solana.model import ( + SolanaApplication, + SolanaInstructionInstance, + SolanaInvariantUnit, + SolanaProgramInstance, +) +from composer.spec.types import PropertyFormulation +from composer.spec.util import FS_FORBIDDEN_READ + +LanguageTag = Literal["solidity", "rust"] +ChainTag = Literal["evm", "solana", "soroban"] + + +@dataclass(frozen=True) +class PromptPair: + """A (system prompt, initial prompt) template-name pair for one agent.""" + + system: str + initial: str + + +@dataclass(frozen=True) +class Language: + """The language of the **code being analyzed** — a facet of the ecosystem, shared by every + chain whose programs are written in it (e.g. the ``rust`` facet is shared by Solana and + Soroban). It drives how the shared front half *reads* the target's source (fs-exclusion + pattern, code-explorer prompt, failure modes). + + This is emphatically **not** the language the AutoProver backend is *implemented* in: a + backend implemented as a Rust wheel (:mod:`composer.rustapp`) may analyze Solidity + (``echoprover`` → EVM) or Rust (Crucible → Solana). The implementation language is not + associated with the ecosystem; only the analyzed-source language is. + + Its members are captured here for the seam; consumers (the entry point's ``forbidden_read``, + the ``code_explorer`` prompt) are rewired to read from it in a later phase, when a + non-Solidity analyzed language first needs them.""" + + name: LanguageTag + default_forbidden_read: str + code_explorer_prompt: str + # The j2 partial with this language's failure modes (overflow, panics, …). Reserved for + # the prompt-fragment split; unused while prompts are still monolithic. + failure_modes_partial: str | None = None + + +@dataclass(frozen=True) +class Ecosystem[App: BaseApplication, Main, Unit: FeatureUnit]: + """A resolved ecosystem = a chain that carries its language. The driver consumes it to + drive the shared front half without hardcoding any one domain. + + Generic over ``App`` (the analyzed system-model type), ``Main`` (the located main-unit + wrapper), and ``Unit`` (the per-unit item the extraction phase iterates). A backend is + paired with an ecosystem by these types: ``run_pipeline`` ties + ``PipelineBackend[..., App, Main, Unit]`` to ``Ecosystem[App, Main, Unit]``, so the analyzed + model, the main-unit, and the per-unit values flow through without casts. EVM binds + ``(SourceApplication, ContractInstance, ContractComponentInstance)``; Solana binds its own.""" + + name: ChainTag + language: Language + #: The pydantic model the analysis phase produces. + system_model: type[App] + #: Prompts for the system-analysis agent. + analysis_prompts: PromptPair + #: Prompts for the per-component property-inference agent. + property_prompts: PromptPair + #: Connectivity/shape validation of the analyzed model (retry feedback on failure). + #: Typed over ``BaseApplication`` (not ``App``): the validator receives the produced model + #: and narrows internally (as ``_validate_connectivity`` does), and this keeps it assignable + #: to ``run_component_analysis``'s ``validate`` parameter without a contravariance clash. + validate_analysis: Callable[[BaseApplication, SolidityIdentifier | None], str | None] + #: Locate the target unit (the "main contract"/program) in the analyzed model. + locate_main: Callable[[App, SourceCode], Main] + #: Enumerate the per-unit items the extraction phase infers properties for. + units: Callable[[Main], list[Unit]] + #: Domain-specific front-matter appended to the analysis input (was hardcoded in the driver). + analysis_extra_input: Callable[[SourceCode], list[str | dict]] + + # -- Extraction strategy (docs/crucible-unit-granularity.md) ------------------------- + #: When True, the driver runs ONE whole-program property extraction (context = + #: ``extraction_unit(main)``) and fans each resulting property out into its own unit + #: via ``property_unit`` — one harness + verdict per property. When False (the EVM + #: default) it extracts per ``units(main)`` (one batch per component). Solana uses + #: global extraction so the fuzzer gets whole-program invariants, one run per invariant. + global_extraction: bool = False + #: The whole-program context the global extraction reads (only used when global_extraction). + extraction_unit: Callable[[Main], FeatureUnit] | None = None + #: Build the per-property unit from (main, property, index) (only used when global_extraction + #: and NOT collapse_units). + property_unit: Callable[[Main, PropertyFormulation, int], FeatureUnit] | None = None + #: When True (with global_extraction), keep all extracted properties in a SINGLE batch on the + #: whole-program ``extraction_unit`` — one formalization, one harness, one run covering every + #: invariant (docs/crucible-unit-granularity.md §3; prototype of the "single whole-program + #: unit" collapse). When False, fan each property into its own ``property_unit``. + collapse_units: bool = False + + +# --------------------------------------------------------------------------- +# main-unit location (moved out of pipeline.core; re-exported there for callers) +# --------------------------------------------------------------------------- + + +def main_instance(app: AnyApplication, source: SourceCode) -> ContractInstance: + """Locate the application's main contract — the one whose solidity identifier matches + ``source.contract_name`` — and return a ``ContractInstance`` pointing at it. Backends call + this from ``prepare_system`` to seed the per-component loop; component analysis should + already have guaranteed the contract is present (via ``expected_main_id``).""" + for i, c in enumerate(app.contract_components): + if c.solidity_identifier == source.contract_name: + return ContractInstance(i, app) + raise ValueError(f"main contract {source.contract_name!r} not found in analyzed application") + + +# --------------------------------------------------------------------------- +# The EVM ecosystem (= today's behavior) +# --------------------------------------------------------------------------- + + +def _evm_units(main: ContractInstance) -> list[ContractComponentInstance]: + return [ + ContractComponentInstance(_contract=main, ind=i) + for i in range(len(main.contract.components)) + ] + + +def _evm_analysis_extra_input(source: SourceCode) -> list[str | dict]: + return [ + f"The main entry point of this application has been explicitly identified as " + f"{source.contract_name} at relative path {source.relative_path}. " + "Your output MUST contain an explicit contract instance with this solidity identifier." + ] + + +SOLIDITY = Language( + name="solidity", + default_forbidden_read=FS_FORBIDDEN_READ, + code_explorer_prompt=CODE_EXPLORER_SYS_PROMPT, +) + +EVM: Ecosystem[SourceApplication, ContractInstance, ContractComponentInstance] = Ecosystem( + name="evm", + language=SOLIDITY, + system_model=SourceApplication, + analysis_prompts=PromptPair( + "application_analysis_system.j2", "application_analysis_prompt.j2" + ), + property_prompts=PromptPair( + "property_analysis_system_prompt.j2", "property_analysis_prompt.j2" + ), + validate_analysis=_validate_connectivity, + locate_main=main_instance, + units=_evm_units, + analysis_extra_input=_evm_analysis_extra_input, +) + + +# --------------------------------------------------------------------------- +# The RUST language facet (shared by Solana and, later, Soroban) +# --------------------------------------------------------------------------- + +#: Cargo/Anchor project layout: hide build output, VCS, lockfiles, and the JS side; keep the +#: crate sources and `tests/`. (Contrast the Foundry-shaped ``FS_FORBIDDEN_READ``.) +RUST_FORBIDDEN_READ = r"(^target/.*)|(^\.git.*)|(^node_modules/.*)|(.*\.lock$)" +# The pipeline generates hundreds of MB of build/scratch *inside the workdir* mid-run, which the +# source tools' file-listing would otherwise pull into LLM context and blow the model's window: +# • ``.sandbox_cargo`` — the command sandbox's private CARGO_HOME (docs/command-sandbox.md §3); +# a build fills it with the *entire* cargo registry (~19.5k files, ~520 MB). +# • ``.sandbox_tmp`` — the sandbox's private linker TMPDIR. +# • nested ``target/`` — cargo build output below the root (e.g. the generated +# ``fuzz//target``, ~4k files, ~900 MB); the top-level ``^target/`` above misses it. +# These are never source, so they are never readable by the source tools (belt-and-suspenders with +# each run's own cleanup: a re-run or cached CI workspace can leave them behind). +RUST_FORBIDDEN_READ = ( + RUST_FORBIDDEN_READ + r"|(^\.sandbox_cargo/.*)|(^\.sandbox_tmp/.*)|(.*/target/.*)" +) + +RUST_CODE_EXPLORER_PROMPT = """\ +You are a code-exploration assistant analyzing Rust source for on-chain programs (e.g. Solana +/ Anchor). You have file tools (list_files, get_file, grep_files) to explore the project. +Answer the question concretely, citing the relevant items: instruction handlers, account +validation structs (e.g. Anchor `#[derive(Accounts)]`), account/state types, PDA seed +derivations, signer/owner checks, and cross-program invocations. Quote the exact Rust snippets +that establish or omit a check; do not speculate about code you have not read. +""" + +RUST = Language( + name="rust", + default_forbidden_read=RUST_FORBIDDEN_READ, + code_explorer_prompt=RUST_CODE_EXPLORER_PROMPT, + failure_modes_partial="rust/_failure_modes.j2", +) + + +# --------------------------------------------------------------------------- +# The Solana chain (RUST ⊕ solana) +# --------------------------------------------------------------------------- + + +def _solana_validate(app: BaseApplication, expected_main: SolidityIdentifier | None) -> str | None: + """Connectivity/shape validation for a ``SolanaApplication`` (retry feedback on failure). + Mirrors the EVM ``_validate_connectivity`` structure: unique program identifiers, unique + instruction slugs within a program, the expected main program present, CPI targets known.""" + if not isinstance(app, SolanaApplication): + return None + errors: list[str] = [] + known_programs: set[str] = set() + for prog in app.programs: + if prog.program_identifier in known_programs: + errors.append(f"Duplicate program identifier: {prog.program_identifier}") + known_programs.add(prog.program_identifier) + slug_origin: dict[str, str] = {} + from composer.spec.util import slugify_filename + + for ins in prog.instructions: + slug = slugify_filename(ins.name) + if slug in slug_origin: + errors.append( + f"Instructions {slug_origin[slug]!r} and {ins.name!r} in {prog.name} " + f"reduce to the same filename slug {slug!r}; give them more-distinct names." + ) + slug_origin[slug] = ins.name + # CPI targets may be well-known external programs (SPL Token, System, …) + # that are not declared in the model; we do not flag those. A future + # policy can require known_programs | known_authorities | an allowlist. + if expected_main is not None and expected_main not in known_programs: + errors.append( + f"Expected a program with identifier {expected_main!r}; declared programs: " + f"{sorted(known_programs) or '(none)'}." + ) + if not errors: + return None + if len(errors) == 1: + return errors[0] + return "Multiple validation errors; fix all before resubmitting:\n" + "\n".join(f"- {e}" for e in errors) + + +def _solana_locate_main(app: SolanaApplication, source: SourceCode) -> SolanaProgramInstance: + for i, prog in enumerate(app.programs): + if prog.program_identifier == source.contract_name: + return SolanaProgramInstance(i, app) + raise ValueError(f"main program {source.contract_name!r} not found in analyzed application") + + +def _solana_units(main: SolanaProgramInstance) -> list[SolanaInstructionInstance]: + # Per-instruction units — unused under global extraction (kept for a per-instruction + # fallback and to satisfy the ecosystem's Unit type). + return [SolanaInstructionInstance(i, main) for i in range(len(main.program.instructions))] + + +def _solana_extraction_unit(main: SolanaProgramInstance) -> SolanaProgramInstance: + # The whole program is the extraction context; SolanaProgramInstance is itself a + # FeatureUnit, so the driver reads it directly to propose whole-program invariants. + return main + + +def _solana_property_unit( + main: SolanaProgramInstance, prop: PropertyFormulation, ind: int +) -> SolanaInvariantUnit: + return SolanaInvariantUnit(ind, main, prop) + + +def _solana_analysis_extra_input(source: SourceCode) -> list[str | dict]: + return [ + f"The main program of this application has been explicitly identified as " + f"{source.contract_name} at relative path {source.relative_path}. " + "Your output MUST contain a program whose program_identifier is this exact identifier." + ] + + +SOLANA: Ecosystem[SolanaApplication, SolanaProgramInstance, SolanaInstructionInstance] = Ecosystem( + name="solana", + language=RUST, + system_model=SolanaApplication, + analysis_prompts=PromptPair("solana/analysis_system.j2", "solana/analysis_prompt.j2"), + property_prompts=PromptPair("solana/property_system.j2", "solana/property_prompt.j2"), + validate_analysis=_solana_validate, + locate_main=_solana_locate_main, + units=_solana_units, + analysis_extra_input=_solana_analysis_extra_input, + # Fuzzing wants whole-program invariants — docs/crucible-unit-granularity.md. Prototype: + # collapse all invariants into ONE whole-program harness + run (§3), instead of one per + # invariant, now that finding-level attribution recovers which property a counterexample hits. + global_extraction=True, + extraction_unit=_solana_extraction_unit, + property_unit=_solana_property_unit, + collapse_units=True, +) + + +#: Registry of available ecosystems, keyed by chain tag. Heterogeneous in ``App``/``Main``/``Unit`` +#: (each chain has its own model), hence ``Ecosystem[Any, Any, Any]``. +ECOSYSTEMS: dict[ChainTag, Ecosystem[Any, Any, Any]] = {"evm": EVM, "solana": SOLANA} diff --git a/composer/pipeline/ptypes.py b/composer/pipeline/ptypes.py index e4adb62b..27031790 100644 --- a/composer/pipeline/ptypes.py +++ b/composer/pipeline/ptypes.py @@ -13,7 +13,7 @@ ) from composer.spec.service_host import ServiceHost from composer.spec.system_model import ( - ContractComponentInstance + FeatureUnit, ) from composer.spec.types import PropertyFormulation, FormalResult from composer.spec.source.report.collect import ReportableResult @@ -71,8 +71,12 @@ class SystemAnalysisSpec: @dataclass -class BackendJob: - feat: ContractComponentInstance +class BackendJob[U: FeatureUnit]: + # ``U`` is the *formalized unit* type the paired backend consumes (EVM's + # ``ContractComponentInstance``, Solana's invariant unit, …) — see ``FeatureUnit``. The shared + # driver is generic over it; a backend narrows it to its concrete unit and reads its members + # without casts. + feat: U props: list[PropertyFormulation] @dataclass(frozen=True) @@ -94,14 +98,16 @@ def run_link(self) -> str | None: return self.result.output_link @dataclass -class ComponentOutcome[FormT: BackendResult](BackendJob): +class ComponentOutcome[FormT: BackendResult, U: FeatureUnit](BackendJob[U]): result: Delivered[FormT] | GaveUp | BaseException @dataclass class CorePipelineResult[FormT: BackendResult]: + # The result rollup is unit-agnostic — it only reads ``feat.display_name`` (a ``FeatureUnit`` + # member) — so it stays mono in the unit type and widens the outcomes to the protocol. n_components: int n_properties: int - outcomes: list[ComponentOutcome[FormT]] + outcomes: list[ComponentOutcome[FormT, FeatureUnit]] failures: list[str] __all__ = [ diff --git a/composer/rag/db.py b/composer/rag/db.py index 1dc3d4c3..fa001ef2 100644 --- a/composer/rag/db.py +++ b/composer/rag/db.py @@ -49,6 +49,15 @@ DEFAULT_CONNECTION: str = f"postgresql://rag_user:rag_password@{_RAG_HOST}:{_RAG_PORT}/rag_db" SANITY_DEFAULT_CONNECTION: str = f"postgresql://extended_rag_user:rag_password@{_RAG_HOST}:{_RAG_PORT}/rag_db" FOUNDRY_DEFAULT_CONNECTION: str = f"postgresql://foundry_rag_user:rag_password@{_RAG_HOST}:{_RAG_PORT}/rag_db" +CRUCIBLE_DEFAULT_CONNECTION: str = f"postgresql://crucible_rag_user:rag_password@{_RAG_HOST}:{_RAG_PORT}/rag_db" + +# Logical knowledge-base tag -> default DB connection, for the generic RAG importer +# (composer.scripts.rag_import). The tag is the one a manifest carries (== a wheel's +# `rag_db_default`), so import target and runtime search tools resolve by one name. Only corpora +# on the JSON-manifest path live here; the CVL/Foundry builders still use their constants above. +KNOWLEDGE_BASES: dict[str, str] = { + "crucible_kb": CRUCIBLE_DEFAULT_CONNECTION, +} type _RagHeader = str | None diff --git a/composer/rag/import_format.py b/composer/rag/import_format.py new file mode 100644 index 00000000..fc47904d --- /dev/null +++ b/composer/rag/import_format.py @@ -0,0 +1,65 @@ +"""The common JSON manifest format for RAG corpora (see ``docs/rag-import-format.md``). + +A *producer* parses a corpus's native docs and emits one :class:`RagManifest` as JSON; the shared +importer (:mod:`composer.scripts.rag_import`) reads that manifest and owns everything downstream — +chunking, embedding, ``part`` numbering, and the dual-path DB ingestion. So these models are the +seam between the two halves, and they are deliberately free of any RAG-stack imports (no +``composer.rag.db``, no spaCy): a producer needs only these classes to emit a corpus. + +The cut is *logical sections*, not pre-chunked rows — a producer decides section boundaries (the +one genuinely corpus-specific editorial choice) and the importer sub-splits each section by +length. See the design doc for why this is the right cut. +""" + +import enum +from typing import Literal + +from pydantic import BaseModel, Field + +#: The schema version the importer understands. Bumped only on a breaking change; the importer +#: refuses a manifest whose ``version`` it doesn't recognize rather than mis-ingesting it. +SCHEMA_VERSION = 1 + + +class BlockKind(str, enum.Enum): + """The two content kinds a section is built from.""" + + TEXT = "text" + CODE = "code" + + +class Block(BaseModel): + """One ordered piece of a section: prose (``text``) or a code sample (``code``). + + Maps 1:1 onto the two ``BlockBuilder`` operations the importer drives — ``text`` → + ``append_text`` (spaCy-split prose), ``code`` → ``add_code`` (gets a ```` tag + assigned by the importer, so producers never touch the tag scheme).""" + + kind: Literal["text", "code"] + body: str + + +class Section(BaseModel): + """A logical documentation section: a header path plus its ordered content blocks. + + ``headers`` is the ``h1..h6`` path (the importer left-packs and truncates to 6, matching the + DB's ``_normalize_head``). Both retrieval indexes are keyed off this path.""" + + headers: list[str] + blocks: list[Block] = Field(default_factory=list) + + +class RagManifest(BaseModel): + """A whole corpus: metadata + an ordered list of sections, serialized as one JSON document.""" + + #: Schema version; must match :data:`SCHEMA_VERSION`. Defaults so a hand-written manifest can + #: omit it, but the importer still validates any value present. + version: int = SCHEMA_VERSION + #: Logical corpus tag — the *same* string a wheel declares as ``rag_db_default`` and that + #: ``rag_env.py`` resolves to search tools. The importer resolves it to a DB connection via + #: ``composer.rag.db.KNOWLEDGE_BASES`` (overridable by ``--output``). + knowledge_base: str + #: Free-text provenance (source repo/commit/glob). For logs only — not persisted per row + #: (the DB schema is header-only). + source: str | None = None + sections: list[Section] = Field(default_factory=list) diff --git a/composer/rustapp/__init__.py b/composer/rustapp/__init__.py new file mode 100644 index 00000000..4489ce7c --- /dev/null +++ b/composer/rustapp/__init__.py @@ -0,0 +1,137 @@ +"""Host for AutoProver applications whose backend is *implemented* in Rust (PyO3). + +"Rust" here is the **backend implementation** language — the wheel is compiled Rust — and it is +orthogonal to the ecosystem, i.e. the language of the *code being analyzed*: a Rust-implemented +backend may analyze Solidity (``echoprover`` selects ecosystem ``evm``) or Rust (Crucible selects +``solana``). The analyzed-source language rides on the ecosystem (see +``composer.pipeline.ecosystem.Language``); this host never assumes it from the fact that the +backend is a Rust wheel — it reads it from ``app.ecosystem.language``. + +This package is the Python side of the seam described in +``docs/rust-backend-api.md``. A Rust application is a wheel built with +``autoprover-sdk`` (see ``rust/``) exposing a small, synchronous, JSON FFI surface +— a **passive service** the pipeline drives: + + descriptor() -> str # the AppDescriptor (declarative spine) + validate_preconditions(args_json) -> str|None + units(input_json) -> str # the report rows / validation units + author_prompt(input_json, failure_json|None) -> str + judge_prompt(input_json, spec) -> str|None + compile(input_json, spec, workdir, sandbox_json) -> str # BLOCKING (run-confined) + validate(input_json, spec, unit, workdir, sandbox_json) -> str # BLOCKING (run-confined) + finalize(outcomes_json) -> str|None + +The host loads that module, synthesizes the pipeline's phase enum from the +descriptor, and wraps the module in a :class:`PipelineBackend` whose ``formalize`` +runs the author→compile→judge→validate loop (:mod:`composer.rustapp.adapter`) — +Python owns the loop and every LLM turn; the two blocking callouts run the toolchain +via ``run-confined``. No IoC ``resume`` protocol and no ``pyo3-async`` bridge. + +Entry points: + +* :func:`composer.rustapp.cli.tui_main` / ``console_main`` — a complete runnable + application from a module name (the descriptor drives argparse, the entry point, + the frontend, and ``main()``). This is the whole vertical. +* :func:`composer.rustapp.host.build_application` — synthesize the phase enum, + labels, section order and backend factory for a frontend / ``main()``. +* :func:`composer.rustapp.entry.rust_entry_point` — the async entry point context + manager (services + ``WorkflowContext``), yielding the Executor. +* :func:`composer.rustapp.host.run_rust_pipeline` — headless: build the backend + from a module name and run the shared driver directly. +""" + +from composer.rustapp.descriptor import ( + AppDescriptor, + ArgDefault, + ArgSpec, + ArtifactLayout, + CoreSlot, + DeliverableMode, + EventKind, + PhaseSpec, + SetupSpec, +) +from composer.rustapp.result import RustArtifact, RustFormalResult +from composer.rustapp.adapter import ( + RustBackend, + RustFormalizer, + RustPreparedSystem, + as_report_backend, + author_and_compile, + make_emitter, + unique_slugs, +) +from composer.rustapp.store import RustArtifactStore +from composer.rustapp.host import ( + BackendOptions, + RustApplication, + StoreFactory, + build_application, + build_backend, + build_core_phases, + build_phase_enum, + load_descriptor, + load_module, + resolve_ecosystem, + run_application, + run_rust_pipeline, +) +from composer.rustapp.entry import ( + EnvBuilder, + RustRunner, + build_arg_parser, + build_neutral_env, + rust_entry_point, +) +from composer.rustapp.frontend import ( + GenericRustApp, + GenericRustConsoleHandler, + GenericRustTaskHandler, +) + +# NOTE: composer.rustapp.cli is intentionally NOT imported here — it runs +# `import composer.bind` (import-time DI / test-tape bootstrap), which the +# built-in apps only trigger from their `main` modules. Import it explicitly: +# from composer.rustapp.cli import tui_main, console_main + +__all__ = [ + "AppDescriptor", + "ArgDefault", + "ArgSpec", + "ArtifactLayout", + "CoreSlot", + "DeliverableMode", + "EventKind", + "PhaseSpec", + "SetupSpec", + "RustArtifact", + "RustFormalResult", + "RustBackend", + "RustFormalizer", + "RustPreparedSystem", + "as_report_backend", + "author_and_compile", + "make_emitter", + "unique_slugs", + "RustArtifactStore", + "RustApplication", + "BackendOptions", + "StoreFactory", + "build_application", + "build_backend", + "build_core_phases", + "build_phase_enum", + "load_descriptor", + "load_module", + "resolve_ecosystem", + "run_application", + "run_rust_pipeline", + "EnvBuilder", + "RustRunner", + "build_arg_parser", + "build_neutral_env", + "rust_entry_point", + "GenericRustApp", + "GenericRustConsoleHandler", + "GenericRustTaskHandler", +] diff --git a/composer/rustapp/adapter.py b/composer/rustapp/adapter.py new file mode 100644 index 00000000..8d85b29d --- /dev/null +++ b/composer/rustapp/adapter.py @@ -0,0 +1,767 @@ +"""Adapter: wrap a Rust wheel (a :class:`~autoprover_sdk.Backend`) as a +:class:`~composer.pipeline.core.PipelineBackend`. + +The Rust wheel is a **passive service** (``docs/rust-backend-api.md``): Python owns the +author→compile→judge→validate loop and every LLM turn, and calls the wheel's pure callouts +(``descriptor`` / ``units`` / ``author_prompt`` / ``judge_prompt`` / ``finalize``) plus the two +blocking ones (``compile`` / ``validate``) that run the toolchain via ``run-confined``. There is +no IoC ``resume`` loop and no ``Effects`` protocol. + +Three phase objects mirror the CVL / foundry backends: + +* :class:`RustBackend` — ``PipelineBackend`` (guidance, phases, store, ``prepare_system``). +* :class:`RustPreparedSystem` — builds the formalizer (thin; no app-specific setup). +* :class:`RustFormalizer` — ``formalize`` runs the loop; ``fetch_verdicts`` reads the verdicts + ``validate`` baked into the result. + +App-specific orchestration (a shared setup artifact, workspace prep, crate assembly) is +descriptor-driven here — no per-application Python package (``docs/rust-pure-app.md``): the wheel +declares ``setup`` / ``workspace_prep`` / ``deliverable_mode=callout`` / ``finalize`` and the +generic :class:`RustPreparedSystem` runs them. +""" + +import asyncio +import json +import logging +from dataclasses import dataclass, field +from pathlib import Path +from typing import Any, Awaitable, Callable, NotRequired, cast, get_args, override + +from pydantic import Field +from graphcore.graph import FlowInput, tool_state_update +from graphcore.tools.schemas import WithAsyncDependencies, WithInjectedId +from langgraph.graph import MessagesState +from langgraph.types import Command + +from composer.io.multi_job import TaskInfo +from composer.pipeline.core import ( + CorePhases, + Formalizer, + GaveUp, + PipelineRun, + PreparedSystem, + SystemAnalysisSpec, +) +from composer.pipeline.ecosystem import Ecosystem +from composer.sandbox.command import DEFAULT_TIMEOUT_S +from composer.sandbox.config import SandboxConfig +from composer.rustapp.descriptor import AppDescriptor +from composer.rustapp.result import RustArtifact, RustFormalResult +from composer.spec.artifacts import ArtifactStore +from composer.spec.context import WorkflowContext +from composer.spec.source.report.collect import ReportComponentInput, Verdict +from composer.spec.graph_builder import bind_standard, run_to_completion +from composer.ui.tool_display import tool_display +from composer.spec.source.report.schema import Outcome, ReportBackend, RuleName +from composer.spec.system_model import BaseApplication, FeatureUnit +from composer.spec.types import PropertyFormulation +from composer.spec.util import slugify_filename, uniq_thread_id + +_log = logging.getLogger(__name__) + +# Author→compile revise budget (was the Rust sessions' SETUP/PC_MAX_ATTEMPTS). +DEFAULT_MAX_ATTEMPTS = 7 + +# Derived from the ReportBackend literal so the two can't drift (single source of truth). +_REPORT_BACKENDS: frozenset[str] = frozenset(get_args(ReportBackend.__value__)) + + +def as_report_backend(tag: str) -> ReportBackend: + """Validate a wheel's free-form ``backend_tag`` against the closed report set.""" + if tag not in _REPORT_BACKENDS: + raise ValueError( + f"unknown report backend_tag {tag!r}; expected one of {sorted(_REPORT_BACKENDS)}" + ) + return cast(ReportBackend, tag) + + +# --------------------------------------------------------------------------- +# The LLM authoring turn — the Rust backend's binding of the shared agent primitive +# (bind_standard / run_to_completion), the peer of composer/foundry/author.py. Python +# runs this; the backend only supplies the prompt (its author_prompt/judge_prompt +# callouts). It is the "author" step of the author→compile→judge→validate loop. +# --------------------------------------------------------------------------- + +# Neutral fallback system prompt. The backend's prompt payload carries the task-specific +# `instruction` and MAY carry its own `system` prompt; when it doesn't, this applies. It +# conveys only the tool-using-agent + result-tool contract — no domain/language specifics +# (those belong in the backend's prompt). +_DEFAULT_SYS_PROMPT = ( + "You are an authoring agent. Use the available tools to explore the target " + "program's source and any reference material, then produce the requested artifact. " + "When done, call the `result` tool with your complete final answer as a single " + "string — the artifact source only, with no surrounding prose or code fences." +) + + +def _split_prompt(messages: Any) -> tuple[str | None, str]: + """Split a backend prompt payload into ``(system, instruction)``. + + The payload is a bare instruction string, or a dict carrying ``instruction`` and + (optionally) a backend-defined ``system`` prompt. ``system`` is ``None`` when the + backend doesn't supply one (the caller falls back to :data:`_DEFAULT_SYS_PROMPT`).""" + if isinstance(messages, dict): + return messages.get("system"), messages.get("instruction") or json.dumps(messages) + return None, messages + + +# NOTE: `bind_standard` introspects this state class's ``__annotations__`` at runtime to +# unwrap ``result: NotRequired[T]`` — so the annotation must stay a real object, not a +# string. This is a concrete reason the repo bans ``from __future__ import annotations`` +# (see CLAUDE.md); stringized annotations would break the unwrap here. +class _LlmState(MessagesState): + result: NotRequired[str] + + +# In-loop-judge author state: `reviewed_text`/`review_ok` record the last draft the author sent +# to `request_review` and the judge's verdict on it; the result gate (`_review_gate`) blocks +# finalization until the submitted draft is one the judge accepted. +class _JudgedAuthorState(MessagesState): + result: NotRequired[str] + reviewed_text: NotRequired[str] + review_ok: NotRequired[bool] + + +class _LlmInput(FlowInput): + pass + + +# A callable that reviews a candidate artifact: ``(draft) -> (accepted, feedback)``. The host +# builds it (see :func:`_make_judge_hook`) around the wheel's ``judge_prompt``. +type _JudgeHook = Callable[[str], Awaitable[tuple[bool, str]]] + + +# Appended to the system prompt when the judge runs in-loop, so the author knows the review +# protocol and the finalize gate. Kept generic (no backend/domain specifics). +_REVIEW_PROTOCOL = ( + "\n\nBefore you finalize, a reviewer must accept your work. Call the `request_review` tool " + "with the exact artifact text you intend to submit; if the review is REJECTED, revise and " + "call `request_review` again. Only call `result` with a draft the reviewer ACCEPTED — the " + "`result` call is rejected otherwise." +) + + +def _review_gate(state: _JudgedAuthorState, result_value: str) -> str | None: + """Block ``result`` unless the submitted draft is exactly the one the judge accepted.""" + if state.get("review_ok") and state.get("reviewed_text") == result_value: + return None + return ( + "Not accepted yet: call `request_review` with this exact draft and get an ACCEPTED " + "review before calling `result` (revise and re-review if it was REJECTED)." + ) + + +@tool_display("Requesting review", "Review") +class _RequestReview(WithInjectedId, WithAsyncDependencies[Command, "_JudgeHook"]): + """Ask the reviewer to evaluate your current draft against the task's criteria. Returns the + verdict and any feedback; if REJECTED, revise and call this again before finalizing.""" + + draft: str = Field( + description="The complete candidate artifact source to review — the exact text you intend " + "to submit via `result`, no surrounding prose or code fences." + ) + + @override + async def run(self) -> Command: + with self.tool_deps() as judge: + ok, feedback = await judge(self.draft) + verdict = "ACCEPTED" if ok else "REJECTED" + content = f"Review {verdict}.\n\n{feedback}" if feedback else f"Review {verdict}." + return tool_state_update( + tool_call_id=self.tool_call_id, content=content, + reviewed_text=self.draft, review_ok=ok, + ) + + +async def run_llm_agent( + env: Any, messages: Any, *, recursion_limit: int, backend_name: str = "rust", + turn_label: str = "authoring", judge: "_JudgeHook | None" = None, memory_tool: Any = None, + exclude_tools: frozenset[str] = frozenset(), +) -> str: + """Run one bounded, tool-enabled turn and return its final text. ``turn_label`` + names the turn's role ("authoring" / "judge") for the UI/log panel. + + Binds the env's tool belt (source navigation + RAG search over the backend's + knowledge base) and a result tool, and runs an agent to completion — so the + prompt can pull in framework docs / read the program. Must run inside a + ``with_handler`` scope (the caller wraps it in ``run.runner``). + + When ``judge`` is given, the turn becomes an in-loop-review author (docs/crucible-judge-in-loop.md): + a ``request_review`` tool runs the judge in-session and ``result`` is gated on an accepted draft, + so the author self-revises against feedback. ``memory_tool`` (when given) is added to the belt so + facts persist across turns/components. ``exclude_tools`` drops named tools from the belt (used to + clamp the review sub-agent's exploration — docs/crucible-judge-cost.md §3).""" + tools = [t for t in (getattr(env, "all_tools", None) or env.rag_tools) if t.name not in exclude_tools] + if memory_tool is not None: + tools.append(memory_tool) + system, instruction = _split_prompt(messages) + doc = "Your complete final answer as a single string (e.g. the authored source file)." + if judge is None: + builder = bind_standard(env.builder_heavy(), _LlmState, doc=doc) + state_input: FlowInput = _LlmInput(input=[]) + else: + builder = bind_standard(env.builder_heavy(), _JudgedAuthorState, doc=doc, validator=_review_gate) + tools = [*tools, _RequestReview.bind(judge).as_tool("request_review")] + system = (system or _DEFAULT_SYS_PROMPT) + _REVIEW_PROTOCOL + state_input = _LlmInput(input=[]) + graph: Any = ( + builder + .with_input(_LlmInput) + .with_sys_prompt(system or _DEFAULT_SYS_PROMPT) + .with_initial_prompt(instruction) + .with_tools(tools) + .compile_async() + ) + res = await run_to_completion( + graph, + state_input, + thread_id=uniq_thread_id(f"{backend_name}-llm"), + recursion_limit=recursion_limit, + description=f"{backend_name} {turn_label} turn", + ) + result = res.get("result") + return result if isinstance(result, str) else json.dumps(result) + + +# --------------------------------------------------------------------------- +# Shared loop helpers (used by RustFormalizer.formalize and app setup artifacts). +# --------------------------------------------------------------------------- + +def make_emitter() -> Callable[[str, dict], None]: + """A ``emit(kind, payload)`` that streams a domain event to the current task's panel. + Routes out-of-graph (the loop isn't inside a LangGraph run) via ``push_custom_update``, + keyed by the active ``run_task`` id — the same routing the old ``RealEffects.emit`` used.""" + from composer.diagnostics.timing import get_current_task_id + from composer.io.context import push_custom_update + + def emit(kind: str, payload: dict) -> None: + push_custom_update({"type": kind, **payload}, thread_id=get_current_task_id() or "rust") + + return emit + + +def _strip_fence(text: str) -> str: + """Strip a leading/trailing ``​```lang`` code fence if the model wrapped its answer + (the authored artifact is written verbatim into a source file, so a fence would break it).""" + t = text.strip() + if t.startswith("```"): + first_nl = t.find("\n") + body = t[first_nl + 1 :] if first_nl != -1 else t + return body.removesuffix("```").rstrip().removesuffix("```").rstrip() + return t + + +def unique_slugs(props: list[PropertyFormulation]) -> list[str]: + """One unique kebab slug per property (basis for its unit/feature name). Titles are unique + at extraction; a slug collision (punctuation/casing) gets a numeric suffix.""" + slugs: list[str] = [] + seen: dict[str, int] = {} + for p in props: + base = slugify_filename(p.title) or "inv" + n = seen.get(base, 0) + seen[base] = n + 1 + slugs.append(base if n == 0 else f"{base}_{n}") + return slugs + + +def _first_line(s: str) -> str: + return next((ln for ln in s.splitlines() if ln.strip()), "").strip() + + +def _parse_judge(review: str) -> tuple[bool, str]: + """Interpret a judge reply as (accept, feedback). Accepts a JSON ``{accept, feedback}`` (what + the Crucible judge emits) or a plain reply led by ``ACCEPT`` / ``REJECT``.""" + try: + obj = json.loads(review) + if isinstance(obj, dict): + return bool(obj.get("accept")), str(obj.get("feedback", "")) + except (json.JSONDecodeError, ValueError): + pass + return (not review.strip().upper().startswith("REJECT")), review + + +async def _author_turn( + module: Any, input_json: str, failure: dict | None, *, env: Any, recursion_limit: int, + backend_name: str, judge: "_JudgeHook | None" = None, memory_tool: Any = None, +) -> str: + """One authoring turn: render the backend's prompt (with any prior failure as revise + context), run the tool-enabled LLM agent, and strip a code fence off the result. When + ``judge`` is given, the author reviews and self-revises in-session (docs/crucible-judge-in-loop.md).""" + prompt = json.loads( + module.author_prompt(input_json, json.dumps(failure) if failure is not None else None) + ) + reply = await run_llm_agent( + env, prompt, recursion_limit=recursion_limit, backend_name=backend_name, + judge=judge, memory_tool=memory_tool, + ) + return _strip_fence(reply) + + +# The review sub-agent gets the program API + fixture in its prompt and shares the run memory, so +# it doesn't need the expensive `code_explorer` exploration sub-agent — direct file reads +# (`get_file`/`grep`) cover its spot-checks. Dropping it is the bulk of the review cost +# (docs/crucible-judge-cost.md §3): each `code_explorer` call is itself a multi-call sub-agent. +_JUDGE_EXCLUDE_TOOLS = frozenset({"code_explorer"}) + + +async def _judge_turn( + module: Any, input_json: str, spec: str, *, env: Any, recursion_limit: int, backend_name: str, + emit: Callable[[str, dict], None] | None = None, memory_tool: Any = None, +) -> tuple[bool, str]: + """Optional LLM review of a spec: ``(accept, feedback)``. ``(True, "")`` when the backend + declares no judge (``judge_prompt`` → ``None``, the default). When a review actually runs, + emit a ``judge`` event carrying the verdict so the frontend surfaces accept/reject.""" + jp = module.judge_prompt(input_json, spec) + if not jp: + return True, "" + review = await run_llm_agent( + env, json.loads(jp), recursion_limit=recursion_limit, + backend_name=backend_name, turn_label="judge", memory_tool=memory_tool, + exclude_tools=_JUDGE_EXCLUDE_TOOLS, + ) + ok, feedback = _parse_judge(review) + if emit is not None: + emit("judge", { + "line": "reviewer accepted the tests" if ok + else f"reviewer rejected — revising: {_first_line(feedback)}", + "outcome": "GOOD" if ok else "BAD", + }) + return ok, feedback + + +def _make_judge_hook( + module: Any, input_json: str, *, env: Any, recursion_limit: int, backend_name: str, + emit: Callable[[str, dict], None] | None, memory_tool: Any, +) -> "_JudgeHook": + """Wrap the wheel's judge as a ``(draft) -> (accepted, feedback)`` callable for the in-loop + ``request_review`` tool. Reuses :func:`_judge_turn` so the verdict event still fires.""" + async def judge(draft: str) -> tuple[bool, str]: + return await _judge_turn( + module, input_json, draft, env=env, recursion_limit=recursion_limit, + backend_name=backend_name, emit=emit, memory_tool=memory_tool, + ) + return judge + + +async def author_and_compile( + module: Any, + input_dict: dict, + *, + env: Any, + sandbox_dict: dict, + workdir: Path, + recursion_limit: int, + backend_name: str, + emit: Callable[[str, dict], None], + max_attempts: int = DEFAULT_MAX_ATTEMPTS, + command_sem: asyncio.Semaphore | None = None, +) -> str | GaveUp: + """Author an artifact's spec, gate it with the backend's ``compile`` (retry on failure) and + optional ``judge``. Returns the compiled spec text, or :class:`GaveUp`. Used for artifacts + that have no report units to validate — e.g. Crucible's shared setup fixture (a compile-only + gate). The component path fuses the build gate into ``validate`` instead (see + :meth:`RustFormalizer.formalize`).""" + input_json = json.dumps(input_dict) + sandbox_json = json.dumps(sandbox_dict) + failure: dict | None = None + for _ in range(max_attempts): + spec = await _author_turn( + module, input_json, failure, env=env, recursion_limit=recursion_limit, backend_name=backend_name + ) + result = json.loads( + await _run_blocking( + lambda: module.compile(input_json, spec, str(workdir), sandbox_json), command_sem + ) + ) + if result.get("status") != "ok": + errors = result.get("errors", "") + failure = {"draft": spec, "errors": errors} + emit("build_output", {"line": _first_line(errors) or "build failed; revising"}) + continue + ok, feedback = await _judge_turn( + module, input_json, spec, env=env, recursion_limit=recursion_limit, + backend_name=backend_name, emit=emit, + ) + if not ok: + failure = {"draft": spec, "errors": feedback, "kind": "judge"} + continue + return spec + return GaveUp(reason=f"{backend_name}: did not pass compile/judge in {max_attempts} attempts") + + +async def _run_blocking(thunk: Callable[[], str], sem: asyncio.Semaphore | None) -> str: + """Run a blocking wheel call (``compile``/``validate`` — they spawn ``run-confined`` and + release the GIL) off the event loop, serialized by ``sem`` when the backend shares one + workdir/crate across concurrent units.""" + if sem is not None: + async with sem: + return await asyncio.to_thread(thunk) + return await asyncio.to_thread(thunk) + + +def _confined_target(root: Path, rel: str) -> Path: + """Join a wheel-supplied relative path under ``root``, rejecting absolute paths / ``..`` + traversal — mirrors the Rust ``confined_join`` so host-written deliverable/prep files stay + inside the project (the wheel is trusted, but defense-in-depth is cheap).""" + p = Path(rel) + if p.is_absolute() or ".." in p.parts: + raise ValueError(f"unsafe file path {rel!r}: absolute or traverses outside the workdir") + return root / p + + +async def run_workspace_prep( + module: Any, + input_dict: dict, + *, + workdir: Path, + sandbox: SandboxConfig | None, + command_timeout_s: int, +) -> None: + """Execute the wheel's pure ``workspace_prep`` plan (``docs/rust-pure-app.md`` §4): write the + declared files (path-confined), then — only when a sandbox is enabled, so a later + confined+offline build finds its deps warm — ``cargo fetch`` each ``warm_dirs`` and build the + named program via the shared Solana build capability. + + Network stays Python-owned and the posture is unchanged: fetches run *unconfined* (a fetch + executes no untrusted code), the code-executing build runs *confined + offline* + (``build_program`` handles both). The wheel supplies only file contents + which dirs/program — + never a command line.""" + plan = json.loads(module.workspace_prep(json.dumps(input_dict))) + for rel, contents in (plan.get("files") or {}).items(): + target = _confined_target(workdir, rel) + target.parent.mkdir(parents=True, exist_ok=True) + target.write_text(contents) + + warm_dirs = plan.get("warm_dirs") or [] + build_prog = plan.get("build_program") + if not warm_dirs and not build_prog: + return + + from composer.spec.solana.build import build_program, warm_cargo_cache + + if warm_dirs and sandbox is not None and sandbox.enabled: + # Warm into the SAME private CARGO_HOME the confined offline build will read. + from composer.sandbox.recipes import sandbox_cargo_home + + cargo_home = sandbox_cargo_home(str(workdir)) + for d in warm_dirs: + await warm_cargo_cache( + _confined_target(workdir, d), cargo_home=cargo_home, timeout_s=command_timeout_s + ) + if build_prog: + await build_program(str(workdir), build_prog, timeout_s=command_timeout_s, sandbox=sandbox) + + +# --------------------------------------------------------------------------- +# The formalizer. +# --------------------------------------------------------------------------- + +class RustFormalizer(Formalizer[RustFormalResult, FeatureUnit]): + """Drives a Rust :class:`~autoprover_sdk.Backend` through the author→compile→judge→validate + loop. Ecosystem-agnostic: the unit is any :class:`FeatureUnit`, marshalled via + ``feature_json()``.""" + + def __init__( + self, + module: Any, + descriptor: AppDescriptor, + *, + sandbox: SandboxConfig | None = None, + command_timeout_s: int = DEFAULT_TIMEOUT_S, + command_sem: asyncio.Semaphore | None = None, + max_attempts: int = DEFAULT_MAX_ATTEMPTS, + context_extra: dict | None = None, + setup_result: str | None = None, + ): + super().__init__(RustFormalResult, as_report_backend(descriptor.backend_tag)) + self._module = module + self._descriptor = descriptor + self._sandbox = sandbox + self._command_timeout_s = command_timeout_s + self._command_sem = command_sem + self._max_attempts = max_attempts + # Injected into every component's ``AuthorInput.context`` (declared-arg values + the + # compiled setup artifact under its ``context_key``); the prepared system assembles it. + self._context_extra = context_extra or {} + # The compiled setup spec (Crucible's fixture), forwarded to ``finalize`` so a + # callout-mode wheel can render the whole deliverable. + self._setup_result = setup_result + + # -- hooks an application backend may override ------------------------- + + def _context(self, run: PipelineRun) -> dict: + """The ``AuthorInput.context`` blob for a component. The program plus whatever the + prepared system injected (declared args + the setup artifact under its context key).""" + return {"program": str(run.source.contract_name), **self._context_extra} + + def _before_formalize(self, feat: FeatureUnit, slugs: list[str]) -> None: + """Place any crate scaffolding before compile/validate. Base: nothing (the wheel + materializes its crate per confined run via the ``files`` map).""" + return None + + def _sandbox_spec(self, workdir: Path) -> dict: + if self._sandbox is None or not self._sandbox.enabled: + return {"run_confined": None, "timeout_s": self._command_timeout_s} + return self._sandbox.backend_spec(workdir, timeout_s=self._command_timeout_s) + + # -- the loop ---------------------------------------------------------- + + @override + async def formalize( + self, + label: str, + feat: FeatureUnit, + props: list[PropertyFormulation], + ctx: WorkflowContext[RustFormalResult], + run: PipelineRun, + ) -> RustFormalResult | GaveUp: + workdir = Path(run.source.project_root) + slugs = unique_slugs(props) + self._before_formalize(feat, slugs) + + input_dict = { + "kind": "component", + "program": str(run.source.contract_name), + "component": feat.feature_json(), + "props": [ + {"title": p.title, "sort": p.sort, "description": p.description, "slug": s} + for p, s in zip(props, slugs) + ], + "context": self._context(run), + } + input_json = json.dumps(input_dict) + sandbox_dict = self._sandbox_spec(workdir) + sandbox_json = json.dumps(sandbox_dict) + emit = make_emitter() + units = json.loads(self._module.units(input_json)) + + # When the wheel supplies a judge for this input, it runs in-loop: a `request_review` tool + # inside the author session, which self-revises against feedback and can only finalize an + # accepted draft (docs/crucible-judge-in-loop.md). The author and judge share the run + # memory across components. Probe the pure callout — `judge_prompt` returns None exactly + # when there is no judge for this kind, so no review machinery is bound then. + has_judge = self._module.judge_prompt(input_json, "") is not None + memory_tool = ctx.get_memory_tool() if has_judge else None + judge_hook = _make_judge_hook( + self._module, input_json, env=run.env, recursion_limit=ctx.recursion_limit, + backend_name=self._descriptor.name, emit=emit, memory_tool=memory_tool, + ) if has_judge else None + + # Fused author → validate loop: validate's build IS the compile gate (no separate dry-run + # per component — that ~2×'d the e2e). The units share one build, so a BuildFailed from any + # unit re-authors the whole spec. + failure: dict | None = None + for _ in range(self._max_attempts): + spec = await _author_turn( + self._module, input_json, failure, env=run.env, + recursion_limit=ctx.recursion_limit, backend_name=self._descriptor.name, + judge=judge_hook, memory_tool=memory_tool, + ) + + # Each report unit declares the *target* that validates it (its own name by default; + # e.g. Crucible shares one `c_invariants` target across all its units). Run each + # DISTINCT target once; the backend returns a verdict per unit it covers — it owns + # attribution (how a failure maps to units), the host records verbatim. + targets = list(dict.fromkeys(u.get("target") or u["unit"] for u in units)) + prop_of = {u["unit"]: u["property"] for u in units} + + verdicts: dict[str, dict] = {} + property_units: list[tuple[str, list[str]]] = [] + build_failed: str | None = None + for target in targets: + res = json.loads( + await _run_blocking( + lambda target=target, spec=spec: self._module.validate( + input_json, spec, target, str(workdir), sandbox_json + ), + self._command_sem, + ) + ) + if res.get("kind") == "build_failed": + build_failed = res.get("errors", "") + break + for unit, verdict in res["verdicts"]: + verdicts[unit] = verdict + prop = prop_of.get(unit, unit) + property_units.append((prop, [unit])) + detail = verdict.get("detail") + line = f'{prop}: {verdict.get("outcome")}' + emit( + "verdict", + {"outcome": verdict.get("outcome"), "name": prop, + "line": f"{line} — {detail}" if detail else line}, + ) + if build_failed is not None: + failure = {"draft": spec, "errors": build_failed} + emit("build_output", {"line": _first_line(build_failed) or "build failed; revising"}) + continue + return RustFormalResult(artifact_text=spec, units=property_units, verdicts=verdicts) + + return GaveUp( + reason=f"{self._descriptor.name}: did not compile/pass judge in {self._max_attempts} attempts" + ) + + @override + async def fetch_verdicts( + self, inp: ReportComponentInput[RustFormalResult] + ) -> dict[RuleName, Verdict]: + formalized = inp.formalized + if formalized is None: + return {} + return { + unit: Verdict( + outcome=Outcome(v["outcome"]), + line=v.get("line"), + duration_seconds=v.get("duration_seconds"), + unit_file=v.get("unit_file") or formalized.unit_file, + message=v.get("detail"), + ) + for unit, v in formalized.result.verdicts.items() + } + + @override + async def finalize(self, outcomes, run: PipelineRun) -> None: + from composer.pipeline.core import Delivered + + components = [] + for o in outcomes: + res = o.result + entry: dict = {"name": o.feat.display_name, "delivered": isinstance(res, Delivered)} + if isinstance(res, Delivered): + # A callout-mode wheel renders the whole deliverable from these (Crucible: folds + # each section into the shared crate, keyed by its property_units feature). + entry["unit_file"] = res.unit_file + entry["run_link"] = res.run_link + entry["artifact_text"] = res.result.artifact_text + entry["property_units"] = res.result.property_units() + components.append(entry) + payload = { + "program": str(run.source.contract_name), + "components": components, + "setup": self._setup_result, + } + raw = await asyncio.to_thread(self._module.finalize, json.dumps(payload)) + if not raw: + return + files: dict[str, str] = json.loads(raw) + root = Path(run.source.project_root) + for rel, contents in files.items(): + target = _confined_target(root, rel) + target.parent.mkdir(parents=True, exist_ok=True) + target.write_text(contents) + + +@dataclass +class RustPreparedSystem(PreparedSystem[RustFormalResult, FeatureUnit]): + """Generic prepared system, descriptor-driven: run the wheel's workspace prep, author the + optional shared ``setup`` artifact, and build a formalizer carrying the injected context. + + Fully expresses what Crucible used to need a subclass for (``docs/rust-pure-app.md``): the + shared fixture, the harness warm + ``.so`` build, per-run serialization, and the + context-thread of the fixture + declared args.""" + + backend: "RustBackend" + analyzed: BaseApplication | None = None + + @override + async def prepare_formalization(self, run: PipelineRun) -> Formalizer[RustFormalResult, FeatureUnit]: + b = self.backend + descriptor = b.descriptor + workdir = Path(run.source.project_root) + program = str(run.source.contract_name) + # One shared crate / target dir → serialize the toolchain runs (declared by the wheel). + command_sem = asyncio.Semaphore(1) if descriptor.serialize_toolchain else None + + analyzed_json = self.analyzed.model_dump(mode="json") if self.analyzed is not None else {} + prep_input = { + "kind": "setup", "program": program, "component": analyzed_json, + "props": [], "context": {}, + } + + # 1. Workspace prep: write the wheel's manifest, warm deps, build the program. + await run_workspace_prep( + b.module, prep_input, workdir=workdir, + sandbox=b.sandbox, command_timeout_s=b.command_timeout_s, + ) + + # 2. Every component's context = declared args + (optionally) the compiled setup artifact. + context_extra: dict = dict(b.declared_args) + setup_result: str | None = None + if descriptor.setup is not None: + sandbox_dict = ( + b.sandbox.backend_spec(workdir, timeout_s=b.command_timeout_s) + if (b.sandbox is not None and b.sandbox.enabled) + else {"run_confined": None, "timeout_s": b.command_timeout_s} + ) + emit = make_emitter() + fixture = await run.runner( + TaskInfo( + f"{descriptor.name}-setup", descriptor.setup.label, + cast(Any, b._phase)[descriptor.setup.phase_key], + ), + lambda: author_and_compile( + b.module, prep_input, env=run.env, sandbox_dict=sandbox_dict, + workdir=workdir, recursion_limit=run.ctx.recursion_limit, + backend_name=descriptor.name, emit=emit, command_sem=command_sem, + ), + ) + if isinstance(fixture, GaveUp): + raise RuntimeError(f"{descriptor.name} setup gave up: {fixture.reason}") + setup_result = fixture + context_extra[descriptor.setup.context_key] = fixture + + return RustFormalizer( + b.module, b.descriptor, sandbox=b.sandbox, + command_timeout_s=b.command_timeout_s, + command_sem=command_sem, context_extra=context_extra, setup_result=setup_result, + ) + + +@dataclass +class RustBackend: + """A :class:`PipelineBackend` backed by a Rust wheel. Structurally satisfies the protocol — + the driver never imports it. Ecosystem-agnostic: it locates the main and marshals units + through the resolved ``ecosystem`` + the ``FeatureUnit`` protocol. + + Subclass (or replace via ``backend_cls``) when the app needs non-generic prep — e.g. + Crucible's shared fixture + harness crate.""" + + module: Any + descriptor: AppDescriptor + _phase: type + _core_phases: CorePhases + artifact_store: ArtifactStore[Any, RustFormalResult] + ecosystem: Ecosystem[Any, Any, Any] + # Wall-clock ceiling for a single compile/validate (a first build can be minutes). + command_timeout_s: int = DEFAULT_TIMEOUT_S + # How to confine every toolchain run (docs/command-sandbox.md). None → unsandboxed. + sandbox: SandboxConfig | None = None + # Parsed values of the descriptor's declared CLI args, injected into every component's + # ``AuthorInput.context`` (e.g. Crucible's ``fuzz_timeout``). Set by the entry point. + declared_args: dict[str, Any] = field(default_factory=dict) + + @property + def backend_guidance(self) -> str: + return self.descriptor.backend_guidance + + @property + def analysis_spec(self) -> SystemAnalysisSpec: + return SystemAnalysisSpec(self.descriptor.analysis_key, "rust-properties") + + @property + def core_phases(self) -> CorePhases: + return self._core_phases + + async def prepare_system( + self, analyzed: BaseApplication, run: PipelineRun + ) -> PreparedSystem[RustFormalResult, FeatureUnit]: + return RustPreparedSystem( + self.ecosystem.locate_main(analyzed, run.source), self, analyzed + ) + + def to_artifact_id(self, c: FeatureUnit) -> RustArtifact: + return RustArtifact( + c.slug, + self.descriptor.artifact_layout.artifact_prefix, + self.descriptor.artifact_layout.artifact_extension, + ) diff --git a/composer/rustapp/cli.py b/composer/rustapp/cli.py new file mode 100644 index 00000000..05d3671f --- /dev/null +++ b/composer/rustapp/cli.py @@ -0,0 +1,126 @@ +"""Generic ``main()`` glue for a Rust application. + +Two shapes, differing only in who owns the event loop (identical to the built-in +apps' ``composer/cli/*.py``): + +* :func:`tui_main` — the pipeline runs as a background worker inside the Textual + app, streaming into it. +* :func:`console_main` — the pipeline runs directly, printing on completion. + +A Rust application ships a two-line CLI: + + from composer.rustapp.cli import tui_main + def main() -> int: + return tui_main("my_app") + +``import composer.bind`` runs first (import-time DI / test-tape bootstrap), exactly +as the built-in ``main()``s require. +""" + +import asyncio + +import composer.bind as _ # noqa: F401 (side-effecting DI/tape bootstrap; must load first) + +from composer.diagnostics.timing import RunSummary +from composer.pipeline.core import CorePipelineResult +from composer.rustapp.adapter import as_report_backend +from composer.rustapp.entry import EnvBuilder, rust_entry_point +from composer.rustapp.frontend import GenericRustApp, GenericRustConsoleHandler +from composer.rustapp.host import build_application +from composer.rustapp.result import RustFormalResult +from composer.rustapp.results import format_verdict_lines, summarize_verdicts + + +def _event_kinds(app) -> set[str]: + return {e.kind for e in app.descriptor.event_kinds} + + +def _notice_kinds(app) -> set[str]: + return {e.kind for e in app.descriptor.event_kinds if e.notice} + + +def _component_label(app) -> str: + """The counts-block noun for one formalized unit ("Components" / "Instructions").""" + return (app.descriptor.component_noun or "component").capitalize() + "s" + + +def _verdict_lines(app, result: CorePipelineResult[RustFormalResult]) -> list[str]: + """Per-unit verdict tally + listing when the results carry verdicts; empty otherwise + (a run-service backend, or a wheel that bakes none).""" + return format_verdict_lines( + summarize_verdicts(result, as_report_backend(app.descriptor.backend_tag)) + ) + + +async def _tui_main(module_name: str, *, env_builder: EnvBuilder | None = None) -> int: + summary = RunSummary() + app_meta = build_application(module_name) + async with rust_entry_point(app_meta, summary, env_builder=env_builder) as pipeline: + tui = GenericRustApp( + phase_labels=app_meta.phase_labels, + section_order=app_meta.section_order, + header_text=app_meta.header_text, + event_kinds=_event_kinds(app_meta), + notice_kinds=_notice_kinds(app_meta), + ) + result: CorePipelineResult[RustFormalResult] | None = None + + async def work(): + nonlocal result + try: + result = await pipeline(tui.make_handler) + noun = (app_meta.descriptor.component_noun or "component") + msg = ( + f"{app_meta.name} complete: {result.n_components} {noun}s, " + f"{result.n_properties} properties" + ) + if tally := summarize_verdicts( + result, as_report_backend(app_meta.descriptor.backend_tag) + ).tally: + msg += f" — {tally}" + if result.failures: + msg += f", {len(result.failures)} failures" + tui.notify(msg) + except Exception as exc: # noqa: BLE001 — surface to the UI, don't crash the loop + tui.notify(f"Pipeline failed: {exc}", severity="error") + finally: + tui._pipeline_done = True + + tui.set_work(work) + await tui.run_async() + print(summary.format()) + if result is not None: + for line in _verdict_lines(app_meta, result): + print(line) + for f in result.failures: + print(f" FAILED: {f}") + return 0 + + +async def _console_main(module_name: str, *, env_builder: EnvBuilder | None = None) -> int: + summary = RunSummary() + app_meta = build_application(module_name) + async with rust_entry_point(app_meta, summary, env_builder=env_builder) as run: + result = await run(GenericRustConsoleHandler(_event_kinds(app_meta)).make_handler) + print(f"\n{'=' * 60}") + print(summary.format()) + print(f"\n {_component_label(app_meta)}: {result.n_components}") + print(f" Properties: {result.n_properties}") + for line in _verdict_lines(app_meta, result): + print(line) + if result.failures: + print(f" Failures: {len(result.failures)}") + for f in result.failures: + print(f" - {f}") + print(f"{'=' * 60}") + return 0 + + +def tui_main(module_name: str, *, env_builder: EnvBuilder | None = None) -> int: + """Run ``module_name`` as a Textual TUI application. Blocks until the run ends.""" + return asyncio.run(_tui_main(module_name, env_builder=env_builder)) + + +def console_main(module_name: str, *, env_builder: EnvBuilder | None = None) -> int: + """Run ``module_name`` in console (no-TUI) mode. Blocks until the run ends.""" + return asyncio.run(_console_main(module_name, env_builder=env_builder)) diff --git a/composer/rustapp/descriptor.py b/composer/rustapp/descriptor.py new file mode 100644 index 00000000..dc6a9ee2 --- /dev/null +++ b/composer/rustapp/descriptor.py @@ -0,0 +1,141 @@ +"""Python mirror of the Rust ``AppDescriptor`` (see ``autoprover-sdk``). + +These pydantic models are the Python side of the descriptor ABI. They are parsed +from the JSON a Rust wheel returns from ``descriptor()`` and consumed by the host +to synthesize the phase enum, argparse, frontend and artifact store. Keep the +field names in lockstep with ``rust/autoprover-sdk/src/lib.rs``. +""" + +import enum +from typing import Literal + +from pydantic import BaseModel, Field + +#: Ecosystem/chain tag. Mirrors ``composer.pipeline.ecosystem.ChainTag`` (kept local so this +#: ABI-mirror module stays decoupled from the pipeline); the host resolves it against the +#: ecosystem registry. +ChainTag = Literal["evm", "solana", "soroban"] + + +class CoreSlot(str, enum.Enum): + """Which driver-tagged core phase a declared phase fills.""" + + ANALYSIS = "analysis" + EXTRACTION = "extraction" + FORMALIZATION = "formalization" + REPORT = "report" + + +class DeliverableMode(str, enum.Enum): + """How the source deliverable is written (mirrors the Rust ``DeliverableMode``). + + ``per_component`` (default): the generic store writes one ``{prefix}_{slug}.{ext}`` file per + component. ``callout``: the store writes no per-component source; the wheel's ``finalize`` + renders the whole deliverable (e.g. Crucible's one shared crate).""" + + PER_COMPONENT = "per_component" + CALLOUT = "callout" + + +class SetupSpec(BaseModel): + """A shared setup artifact authored once before per-component formalization (Crucible's + shared fixture). The host runs the author→compile loop for a ``kind="setup"`` input under + ``phase_key`` and threads the compiled spec into each component's context under + ``context_key``. Mirrors the Rust ``SetupSpec``.""" + + phase_key: str + label: str + context_key: str + + +class PhaseSpec(BaseModel): + """One task-grouping phase; ``key`` becomes the synthesized enum member name.""" + + key: str + label: str + order: int = 0 + core_slot: CoreSlot | None = None + + +class ArgDefault(BaseModel): + """Tagged default value for a declared CLI argument.""" + + kind: Literal["str", "int", "bool"] + value: str | int | bool | None = None + + +class ArgSpec(BaseModel): + """A CLI flag the generic entry point adds beyond the positional inputs.""" + + flag: str + help: str + default: ArgDefault + required: bool = False + + +class EventKind(BaseModel): + """A domain event kind the frontend should render (see ``Command::Emit``). + + ``notice`` events are surfaced as a persistent, always-visible callout (plus a toast) + rather than a line in the collapsible per-task events log — for one-shot important + results such as a per-unit verdict. Defaults to ``False`` so wheels built before + the field existed still load.""" + + kind: str + label: str + notice: bool = False + + +class ArtifactLayout(BaseModel): + """Project-root-relative deliverable layout.""" + + deliverable_dir: str + internal_dir: str + report_dir: str + artifact_dir: str + artifact_prefix: str + artifact_extension: str + property_suffix: str + #: Under ``callout`` deliverable mode, the project-relative primary deliverable path, + #: ``{program}``-templated (Crucible: ``fuzz/{program}/src/main.rs``). Used only as each + #: component's report link; ``None`` in ``per_component`` mode. + deliverable_primary: str | None = None + + +class AppDescriptor(BaseModel): + """The complete declaration a Rust wheel exports.""" + + name: str + header_text: str + #: The ecosystem (chain) whose system model / prompts the shared front half uses. The + #: host resolves it against ``composer.pipeline.ecosystem.ECOSYSTEMS``. Defaults to + #: ``"evm"`` so wheels built before this field existed keep working. + ecosystem: ChainTag = "evm" + backend_tag: str + backend_guidance: str + analysis_key: str + phases: list[PhaseSpec] + args: list[ArgSpec] = Field(default_factory=list) + rag_db_default: str | None = None + event_kinds: list[EventKind] = Field(default_factory=list) + artifact_layout: ArtifactLayout + #: Optional shared-setup step run before per-component formalization (see :class:`SetupSpec`). + setup: SetupSpec | None = None + #: How the source deliverable is written (see :class:`DeliverableMode`). + deliverable_mode: DeliverableMode = DeliverableMode.PER_COMPONENT + #: Serialize the blocking toolchain callouts on one semaphore — set when the app shares a + #: single build dir / target across units. + serialize_toolchain: bool = False + #: Default to the fail-closed ``launcher`` sandbox provider (still overridable by + #: ``COMPOSER_SANDBOX_PROVIDER``). Set by any wheel that runs untrusted native toolchains. + confine_by_default: bool = False + #: Human noun for one formalized unit in the console/TUI summary ("instruction" for + #: Crucible). ``None`` → "component". + component_noun: str | None = None + + def ordered_phases(self) -> list[PhaseSpec]: + return sorted(self.phases, key=lambda p: (p.order, p.key)) + + def core_slot_map(self) -> dict[CoreSlot, str]: + """The declared phase ``key`` for each core slot it fills.""" + return {p.core_slot: p.key for p in self.phases if p.core_slot is not None} diff --git a/composer/rustapp/entry.py b/composer/rustapp/entry.py new file mode 100644 index 00000000..c9371f0a --- /dev/null +++ b/composer/rustapp/entry.py @@ -0,0 +1,417 @@ +"""Generic async entry point for a Rust application. + +Mirrors ``composer/foundry/entry.py``'s shape — parse args → open DB / store / +checkpointer / logging → yield a closure the caller drives with a handler factory +— but is *descriptor-driven*: the CLI flags, precondition validation, and report +tag all come from the Rust wheel's ``AppDescriptor`` instead of being hard-coded. + +The imperative service wiring (Postgres pools, the async tool context, the thread +logger, ``WorkflowContext``) stays Python and is essentially identical to the +foundry entry point — that shell is irreducibly async and is not something Rust +owns (see ``docs/rust-applications.md`` §4.2). What Rust contributes here is only +declarative: the arg schema and the ``validate_preconditions`` hook. + +The env built here is *neutral*: the standard source-navigation toolset +(``code_explorer`` + fs tools) with no RAG surface. A backend that wants a RAG +database can supply its own env builder via ``env_builder=``. +""" + +import argparse +import asyncio +import hashlib +import json +import os +import pathlib +import sys +import uuid +from contextlib import asynccontextmanager +from typing import Any, AsyncIterator, Awaitable, Callable, cast + +from langgraph.store.base import BaseStore + +from composer.core.user import user_data_ns +from composer.diagnostics.logging_setup import setup_autoprove_logging +from composer.diagnostics.timing import RunSummary, install_run_summary +from composer.input.parsing import add_protocol_args +from composer.input.types import ( + DEFAULT_RECURSION_LIMIT, + ExtendedModelOptions, +) +from composer.io.multi_job import HandlerFactory, TaskInfo, run_task +from composer.io.thread_logging import default_logging_ns, thread_logger +from composer.kb.knowledge_base import DefaultEmbedder +from composer.pipeline.core import CorePipelineResult +from composer.rag.models import get_model +from composer.rustapp.descriptor import ArgDefault, ArgSpec +from composer.rustapp.host import RustApplication, build_application, run_application +from composer.rustapp.result import RustFormalResult +from composer.sandbox.config import SandboxConfig +from composer.spec.context import SourceCode, SourceFields, WorkflowContext +from composer.spec.service_host import ModelProvider, PureServiceHost, ServiceHost +from composer.spec.source.design_doc_finder import ( + DESIGN_DOC_DISCOVERY_TASK_ID, + discovery_cache_key, + resolve_design_doc, +) +from composer.spec.source.source_env import ( + build_basic_source_tools, + build_source_tools, +) +from composer.spec.system_model import SolidityIdentifier +from composer.spec.util import FS_FORBIDDEN_READ +from composer.ui.tool_display import async_tool_context +from composer.workflow.services import llm_factory, standard_connections +from composer.llm.registry import get_provider_for + +# A caller-supplied env builder, for backends that want a custom tool/RAG surface. +EnvBuilder = Callable[..., ServiceHost] + +# The Executor a frontend drives. +RustRunner = Callable[ + [HandlerFactory], Awaitable[CorePipelineResult[RustFormalResult]] +] + + +def build_neutral_env( + *, + model_provider: ModelProvider, + project_root: str, + store: BaseStore, + source_question_ns: tuple[str, ...], + recursion_limit: int, + forbidden_read: str = FS_FORBIDDEN_READ, +) -> ServiceHost: + """A source-navigation env with no RAG surface — the same ``code_explorer`` + + fs tools the built-in backends use for analysis/authoring. ``forbidden_read`` + is the ecosystem's fs-exclusion default (Cargo layout for Rust, Foundry for EVM).""" + basic = build_basic_source_tools(root=project_root, forbidden_read=forbidden_read) + full = build_source_tools( + basic, model_provider, store, source_question_ns, recursion_limit=recursion_limit + ) + return PureServiceHost(models=model_provider, rag_tools=(), sort="existing").bind_source_tools( + full + ) + + +def _default_env_builder(rag_db: str | None) -> EnvBuilder: + """The env builder for a wheel that declares no custom ``env_builder``: neutral (source tools + only) when the descriptor sets no ``rag_db_default``, otherwise the same source tools plus the + declared corpus's RAG search tools (replaces the old per-app ``build_crucible_env``).""" + if not rag_db: + return build_neutral_env + + def builder( + *, + model_provider: ModelProvider, + project_root: str, + store: BaseStore, + source_question_ns: tuple[str, ...], + recursion_limit: int, + forbidden_read: str = FS_FORBIDDEN_READ, + ) -> ServiceHost: + from composer.tools.rag_env import build_rag_tools + + basic = build_basic_source_tools(root=project_root, forbidden_read=forbidden_read) + full = build_source_tools( + basic, model_provider, store, source_question_ns, recursion_limit=recursion_limit + ) + return PureServiceHost( + models=model_provider, rag_tools=build_rag_tools(rag_db), sort="existing" + ).bind_source_tools(full) + + return builder + + +def _build_confinement(app: RustApplication, args: dict) -> SandboxConfig: + """The default command-sandbox config for a wheel that sets ``confine_by_default`` — the + fail-closed ``launcher`` provider (overridable by ``COMPOSER_SANDBOX_PROVIDER``), with the + wheel's ``sandbox_grants`` (extra read-only paths / env names) unioned in. Python owns the + policy; the wheel only *declares* the grants (``docs/rust-pure-app.md`` §5.2).""" + from composer.sandbox.config import SandboxConfig + from composer.sandbox.recipes import DEFAULT_ENV_PASSTHROUGH + + grants = json.loads(app.module.sandbox_grants(json.dumps(args))) + extra_ro = tuple(pathlib.Path(p) for p in grants.get("extra_ro", [])) + extra_env = tuple(grants.get("extra_env", [])) + provider = os.environ.get("COMPOSER_SANDBOX_PROVIDER", "launcher") + return SandboxConfig( + provider=provider, + extra_ro=extra_ro, + env_passthrough=DEFAULT_ENV_PASSTHROUGH + extra_env, + ) + + +def _user_ns(*parts: str | None) -> tuple[str, ...]: + return user_data_ns() + tuple(p for p in parts if p) + + +def _root_cache_key( + project_root: str, system_doc_path: pathlib.Path, relative_path: str, contract_name: str +) -> str: + doc_hash = hashlib.sha256(system_doc_path.read_bytes()).hexdigest() + combined = "|".join([project_root, doc_hash, relative_path, contract_name]) + return hashlib.sha256(combined.encode()).hexdigest()[:16] + + +def _add_declared_args(parser: argparse.ArgumentParser, specs: list[ArgSpec]) -> list[str]: + """Add the descriptor's declared flags; return their argparse dests.""" + dests: list[str] = [] + for spec in specs: + dest = spec.flag.lstrip("-").replace("-", "_") + dests.append(dest) + d: ArgDefault = spec.default + if d.kind == "bool": + parser.add_argument( + spec.flag, dest=dest, action="store_true", + default=bool(d.value), help=spec.help, + ) + elif d.kind == "int": + parser.add_argument( + spec.flag, dest=dest, type=int, default=d.value, + required=spec.required, help=spec.help, + ) + else: # str + parser.add_argument( + spec.flag, dest=dest, type=str, default=d.value, + required=spec.required, help=spec.help, + ) + return dests + + +def _discovery_phase(app: RustApplication) -> Any: + """The phase to tag the design-doc-discovery task with: a descriptor phase keyed + ``discover_design_doc`` if the app declares one (a dedicated UI section), else the + first ordered phase (so a generic wheel still groups it somewhere sensible).""" + ordered = app.descriptor.ordered_phases() + key = "discover_design_doc" + if any(p.key == key for p in ordered): + return app.phase[key] + return app.phase[ordered[0].key] + + +@asynccontextmanager +async def rust_entry_point( + app: RustApplication, + summary: RunSummary, + *, + argv: list[str] | None = None, + env_builder: EnvBuilder | None = None, + run_pipeline_fn: Callable[..., Awaitable[CorePipelineResult[RustFormalResult]]] | None = None, +) -> AsyncIterator[RustRunner]: + """Parse args, open services, and yield the Executor for ``app``. + + Pass a pre-built :class:`RustApplication` (from :func:`build_application`) so the + backend and the frontend share one phase enum. ``argv`` overrides ``sys.argv`` + (useful in tests); ``env_builder`` overrides :func:`build_neutral_env`.""" + descriptor = app.descriptor + parser = argparse.ArgumentParser(description=f"{descriptor.name} — AutoProver (Rust backend)") + add_protocol_args(parser, ExtendedModelOptions) + parser.add_argument( + "--recursion-limit", type=int, default=DEFAULT_RECURSION_LIMIT, + help=f"Max graph iterations (default: {DEFAULT_RECURSION_LIMIT})", + ) + parser.add_argument("project_root", help="Project root") + parser.add_argument("main_contract", help="Main contract as path:ContractName") + parser.add_argument( + "system_doc", nargs="?", default=None, + help="Path to the design document (text or PDF); auto-discovered if omitted", + ) + parser.add_argument("--max-concurrent", type=int, default=4, help="Max concurrent agents (default: 4)") + parser.add_argument("--cache-ns", default=None, help="Cache namespace (enables cross-run caching)") + parser.add_argument("--memory-ns", default=None, help="Memory namespace (default: thread id)") + parser.add_argument("--interactive", action="store_true", help="Interactively refine extracted properties") + parser.add_argument("--max-bug-rounds", type=int, default=3, help="Max bug-extraction rounds per component (default: 3)") + declared_dests = _add_declared_args(parser, descriptor.args) + + args = parser.parse_args(argv) + + project_root = pathlib.Path(args.project_root).resolve() + main_path, contract_name = args.main_contract.split(":", 1) + contract_name = SolidityIdentifier(contract_name) + full_path = pathlib.Path(main_path).resolve() + if not full_path.is_relative_to(project_root): + parser.error(f"Invalid path: {full_path} not under project root {project_root}") + relative_path = str(full_path.relative_to(project_root)) + + # Rust-owned precondition validation (cf. foundry's foundry.toml check). + declared_args = {d: getattr(args, d) for d in declared_dests} + err = app.validate_preconditions( + { + "project_root": str(project_root), + "main_contract": args.main_contract, + "system_doc": args.system_doc or "", + **declared_args, + } + ) + if err: + parser.error(err) + + # The ecosystem's fs-exclusion default (Cargo layout for Rust, Foundry for EVM). + forbidden_read = app.ecosystem.language.default_forbidden_read + model = get_model() + + thread_id = f"{descriptor.name}_{uuid.uuid4().hex[:12]}" + text_log, events_log = setup_autoprove_logging(str(project_root), thread_id) + print(f"{descriptor.name} logs: {text_log}\n events: {events_log}", file=sys.stderr) + install_run_summary(summary) + + # argparse Namespace duck-types the ModelConfiguration protocol (the model flags come from + # ExtendedModelOptions); the built-in entries cast their args the same way. + tiered = get_provider_for(tiered=cast(Any, args)) + discovery_phase = _discovery_phase(app) + + async with ( + standard_connections(provider=tiered.provider_kind, embedder=DefaultEmbedder(model)) as conns, + async_tool_context(), + thread_logger( + conns.store, + { + "root_thread_id": thread_id, + "workflow": descriptor.name, + "memory_ns": args.memory_ns if args.memory_ns is not None else thread_id, + }, + default_logging_ns(uid=None), + run_id=summary.run_id, + ), + ): + model_provider = ModelProvider( + heavy_model=tiered.heavy, + lite_model=tiered.lite, + checkpointer=conns.checkpointer, + ) + # The design-doc finder works off the source fields alone; its cache namespace + # is doc-independent (keyed by project/contract), so it's built up front. + init_source = SourceFields( + project_root=str(project_root), + contract_name=contract_name, + relative_path=relative_path, + forbidden_read=forbidden_read, + ) + disc_cache_ns: tuple[str, ...] | None = ( + _user_ns( + args.cache_ns, "discovery", + discovery_cache_key(str(project_root), relative_path, str(contract_name)), + ) + if args.cache_ns is not None + else None + ) + disc_ctx = WorkflowContext.create( + services=conns.memory, thread_id=thread_id, store=conns.store, + recursion_limit=args.recursion_limit, memory_namespace=args.memory_ns, + cache_namespace=disc_cache_ns, + ) + semaphore = asyncio.Semaphore(args.max_concurrent) + + async def runner(handler: HandlerFactory) -> CorePipelineResult[RustFormalResult]: + # 1. Resolve the design doc: use the supplied path, else discover one as a + # visible task (needs the handler scope, which only exists here). + if args.system_doc is not None: + sys_path = pathlib.Path(args.system_doc) + else: + sys_path = await run_task( + factory=handler, + info=TaskInfo( + task_id=DESIGN_DOC_DISCOVERY_TASK_ID, + label="Design Doc Discovery", + phase=discovery_phase, + ), + fn=lambda: resolve_design_doc( + source=init_source, uploader=conns.uploader, + models=model_provider, disc_ctx=disc_ctx, + ), + semaphore=semaphore, + ) + + content = await conns.uploader.get_document(sys_path) + if content is None: + raise ValueError(f"cannot read design document: {sys_path}") + + # 2. Doc-dependent construction. The root cache key hashes the doc bytes, so + # a discovered doc and a supplied one produce an identical key. + root_key = _root_cache_key( + str(project_root), sys_path, relative_path, str(contract_name) + ) + cache_root: tuple[str, ...] | None = ( + _user_ns(args.cache_ns, root_key) if args.cache_ns is not None else None + ) + source_input = SourceCode( + content=content, + project_root=str(project_root), + contract_name=contract_name, + relative_path=relative_path, + forbidden_read=forbidden_read, + ) + source_question_ns = _user_ns("source_agent", "cache", root_key) + # Descriptor-driven env: neutral, or (if the wheel declares a RAG corpus) the same + # source tools plus that corpus's search tools. A wheel can still force its own via + # ``env_builder=``. + builder = env_builder or _default_env_builder(app.descriptor.rag_db_default) + env = builder( + model_provider=model_provider, + project_root=str(project_root), + store=conns.indexed_store, + source_question_ns=source_question_ns, + recursion_limit=args.recursion_limit, + forbidden_read=forbidden_read, + ) + ctx = WorkflowContext.create( + services=conns.memory, + thread_id=thread_id, + store=conns.store, + recursion_limit=args.recursion_limit, + cache_namespace=cache_root, + memory_namespace=args.memory_ns, + ) + + # Thread the declared CLI args into the backend (→ every component's context) and, + # if the wheel asks to be confined by default, build the launcher policy with its + # declared sandbox grants. Both are inert for a wheel that declares neither. + app.options.declared_args = declared_args + if app.options.sandbox is None and app.descriptor.confine_by_default: + app.options.sandbox = _build_confinement( + app, + { + "project_root": str(project_root), + "main_contract": args.main_contract, + "system_doc": args.system_doc or "", + **declared_args, + }, + ) + + # 3. A backend that needs a bespoke store/pipeline (e.g. Crucible's crate + # store) supplies run_pipeline_fn; everything else uses the generic host. + if run_pipeline_fn is not None: + return await run_pipeline_fn( + source_input=source_input, ctx=ctx, handler_factory=handler, + env=env, args=args, + ) + return await run_application( + app, + source_input=source_input, + ctx=ctx, + handler_factory=handler, + env=env, + max_concurrent=args.max_concurrent, + max_bug_rounds=args.max_bug_rounds, + interactive=args.interactive, + ) + + yield runner + + +def build_arg_parser(app: RustApplication) -> argparse.ArgumentParser: + """Build (but do not run) the descriptor-driven argument parser — exposed for + tests and ``--help`` introspection without opening any service.""" + parser = argparse.ArgumentParser(description=f"{app.descriptor.name} — AutoProver (Rust backend)") + add_protocol_args(parser, ExtendedModelOptions) + parser.add_argument("--recursion-limit", type=int, default=DEFAULT_RECURSION_LIMIT) + parser.add_argument("project_root") + parser.add_argument("main_contract") + parser.add_argument("system_doc", nargs="?", default=None) + parser.add_argument("--max-concurrent", type=int, default=4) + parser.add_argument("--cache-ns", default=None) + parser.add_argument("--memory-ns", default=None) + parser.add_argument("--interactive", action="store_true") + parser.add_argument("--max-bug-rounds", type=int, default=3) + _add_declared_args(parser, app.descriptor.args) + return parser diff --git a/composer/rustapp/frontend.py b/composer/rustapp/frontend.py new file mode 100644 index 00000000..c37ac6a1 --- /dev/null +++ b/composer/rustapp/frontend.py @@ -0,0 +1,152 @@ +"""Generic frontend for a Rust application. + +Both frontends are thin, descriptor-driven subclasses of the shared bases: + +* :class:`GenericRustApp` — a ``MultiJobApp`` TUI whose phase labels / section + order come from the descriptor. +* :class:`GenericRustConsoleHandler` — the stdout ``HandlerFactory``. + +Domain-event rendering is *data-driven* by the descriptor's ``event_kinds``: a +Rust ``Command::Emit`` becomes a ``{"type": kind, ...}`` custom-stream payload, +which the handler writes to the task's log if ``kind`` is a declared event kind. +No per-application Python subclass is needed — the same generic handler renders +any Rust app's events (see ``docs/rust-applications.md`` §4.4). +""" + +import json +from collections.abc import Set as AbstractSet +from typing import Any, override + +from rich.text import Text +from textual.containers import VerticalScroll +from textual.widgets import Collapsible, RichLog + +from composer.io.event_handler import EventHandler, NullEventHandler +from composer.io.multi_job import TaskInfo +from composer.ui.multi_console_handler import MultiJobConsoleHandler +from composer.ui.multi_job_app import MultiJobApp, MultiJobTaskHandler, TaskHost +from composer.ui.tool_display import ToolDisplayConfig + + +def _render_event(payload: dict) -> str: + """A one-line rendering of an emit payload: prefer a ``line`` field, else a + compact JSON of everything but the discriminating ``type``.""" + if isinstance(payload.get("line"), str): + return payload["line"] + rest = {k: v for k, v in payload.items() if k != "type"} + return json.dumps(rest) if rest else "" + + +# Glyph for a notice event that carries a neutral ``outcome`` (e.g. a verdict). Mirrors +# the report/TUI GOOD→✓ / BAD→✗ vocabulary so a result scans at a glance. +_OUTCOME_GLYPH: dict[str, str] = { + "GOOD": "✓", # ✓ + "BAD": "✗", # ✗ + "TIMEOUT": "⧖", # ⧖ + "ERROR": "!", + "UNKNOWN": "?", +} + + +def _notice_headline(payload: dict) -> str: + """The persistent-callout headline for a notice event: its one-line rendering, + prefixed with an outcome glyph when the payload carries one.""" + body = _render_event(payload) + glyph = _OUTCOME_GLYPH.get(payload.get("outcome", "")) if isinstance(payload.get("outcome"), str) else None + return f"{glyph} {body}" if glyph else body + + +class GenericRustTaskHandler(MultiJobTaskHandler[None], NullEventHandler): + """Per-task handler that streams the app's declared domain events into a + collapsible log under the task panel.""" + + def __init__( + self, + task_id: str, + label: str, + panel: VerticalScroll, + host: TaskHost, + tool_config: ToolDisplayConfig, + event_kinds: set[str], + notice_kinds: AbstractSet[str] = frozenset(), + ): + super().__init__(task_id, label, panel, host, tool_config) + self._event_kinds = event_kinds + # Notice kinds are surfaced as a persistent callout (post_notice) instead of a + # buried log line; ``event_kinds`` here is the streaming (non-notice) remainder. + self._notice_kinds = notice_kinds + self._event_log: RichLog | None = None + + def format_hitl_prompt(self, ty: None) -> list[Text | str]: + raise NotImplementedError("Rust applications do not use HITL interrupts") + + async def _ensure_event_log(self) -> RichLog: + if self._event_log is None: + log = RichLog(highlight=True, markup=False) + log.styles.min_height = 12 + self._event_log = log + await self._mount_to(self._panel, Collapsible(log, title="Events")) + return self._event_log + + @override + async def handle_event(self, payload: dict, path: list[str], checkpoint_id: str) -> None: + kind = payload.get("type") + if kind in self._notice_kinds: + # A one-shot important result — a persistent callout in the panel + a toast, + # visible without expanding the events log. + await self.post_notice(_notice_headline(payload)) + elif kind in self._event_kinds: + log = await self._ensure_event_log() + log.write(f"[{kind}] {_render_event(payload)}") + + +class GenericRustApp(MultiJobApp[Any, GenericRustTaskHandler]): + """Textual TUI for a Rust application.""" + + def __init__( + self, + *, + phase_labels: dict[Any, str], + section_order: list[str], + header_text: str, + event_kinds: set[str], + notice_kinds: AbstractSet[str] = frozenset(), + ): + super().__init__( + phase_labels=phase_labels, section_order=section_order, header_text=header_text + ) + self._event_kinds = event_kinds + self._notice_kinds = notice_kinds + + def create_task_handler( + self, panel: VerticalScroll, info: TaskInfo[Any] + ) -> GenericRustTaskHandler: + return GenericRustTaskHandler( + info.task_id, info.label, panel, self, ToolDisplayConfig(), + self._event_kinds, self._notice_kinds, + ) + + def create_event_handler( + self, handler: GenericRustTaskHandler, info: TaskInfo[Any] + ) -> EventHandler: + return handler + + +class GenericRustConsoleHandler(MultiJobConsoleHandler[Any]): + """Stdout ``HandlerFactory`` for a Rust application.""" + + def __init__(self, event_kinds: set[str]): + super().__init__() + self._event_kinds = event_kinds + + @override + async def handle_event(self, payload: dict, path: list[str], checkpoint_id: str) -> None: + kind = payload.get("type") + if kind in self._event_kinds: + self._output(f"[{self._label(path)}] {kind}: {_render_event(payload)}") + + @override + async def handle_progress_event(self, payload: dict) -> None: + # Rust applications stream everything through Command::Emit (the custom + # channel); there are no progress-channel events. + pass diff --git a/composer/rustapp/host.py b/composer/rustapp/host.py new file mode 100644 index 00000000..1622175a --- /dev/null +++ b/composer/rustapp/host.py @@ -0,0 +1,299 @@ +"""Assemble a Rust wheel into a runnable AutoProver application. + +The declarative descriptor lets a single host synthesize what a hand-written +application spells out (phase enum, core-phase mapping, artifact store, labels, +section order) and hand the driver a ready :class:`PipelineBackend`. + +* :func:`run_rust_pipeline` — the pipeline wrapper (build backend + ``PipelineRun`` + + call the shared driver). This is the piece a generic entry point calls. +* :func:`build_application` — bundle everything a frontend / ``main()`` needs + (the synthesized phase enum, labels, section order, and a backend factory). + +Applications that need a non-default store or backend class (e.g. Crucible's crate +store) pass ``store_factory`` / ``backend_cls``; the **same** phase enum is shared +by the frontend and the pipeline. +""" + +import asyncio +import enum +import importlib +from dataclasses import dataclass, field +from typing import Any, Callable, cast + +from composer.io.multi_job import HandlerFactory +from composer.pipeline.core import ( + CorePhases, + CorePipelineResult, + PipelineRun, + run_pipeline, +) +from composer.pipeline.ecosystem import ECOSYSTEMS, Ecosystem +from composer.rustapp.adapter import RustBackend +from composer.rustapp.descriptor import AppDescriptor, CoreSlot +from composer.rustapp.result import RustFormalResult +from composer.rustapp.store import RustArtifactStore +from composer.sandbox.command import DEFAULT_TIMEOUT_S +from composer.sandbox.config import SandboxConfig +from composer.spec.artifacts import ArtifactStore +from composer.spec.context import SourceCode, WorkflowContext +from composer.spec.service_host import ServiceHost + +#: Build an artifact store for a run from the source + descriptor. +StoreFactory = Callable[[SourceCode, AppDescriptor], ArtifactStore[Any, RustFormalResult]] + + +@dataclass +class BackendOptions: + """Mutable run options closed over by :meth:`RustApplication.make_backend`. + + The CLI can adjust these (e.g. the sandbox) after building the application but + before :func:`run_application`, keeping one phase enum. Backend-specific tuning knobs + (e.g. a fuzz budget) travel as descriptor-declared args in :attr:`declared_args`. + """ + + command_timeout_s: int = DEFAULT_TIMEOUT_S + sandbox: SandboxConfig | None = None + #: Parsed values of the descriptor's declared CLI args, threaded into the backend and + #: injected into every component's ``AuthorInput.context``. Set by the entry point. + declared_args: dict[str, Any] = field(default_factory=dict) + + +def load_module(module_name: str) -> Any: + """Import a Rust application's compiled module by name (e.g. ``"echoprover"``).""" + return importlib.import_module(module_name) + + +def load_descriptor(module: Any) -> AppDescriptor: + """Parse a module's ``descriptor()`` JSON into an :class:`AppDescriptor`.""" + return AppDescriptor.model_validate_json(module.descriptor()) + + +def resolve_ecosystem(descriptor: AppDescriptor) -> Ecosystem[Any, Any, Any]: + """Resolve the descriptor's declared ecosystem against the registry.""" + eco = ECOSYSTEMS.get(descriptor.ecosystem) + if eco is None: + raise ValueError( + f"application {descriptor.name!r} selects ecosystem {descriptor.ecosystem!r}, " + f"which is not registered. Available: {sorted(ECOSYSTEMS)}." + ) + return eco + + +def build_phase_enum(descriptor: AppDescriptor) -> type[enum.Enum]: + """Synthesize the pipeline's phase enum from the descriptor. Safe: the code + only ever uses phase members for ``.name`` and as dict keys (no isinstance / + identity checks against a static class).""" + ordered = descriptor.ordered_phases() + name = "".join(part.capitalize() for part in descriptor.name.split("_")) + "Phase" + # enum.Enum's functional API is typed as returning an ``Enum`` instance, not the new + # class; it does return a class at runtime. + return cast(type[enum.Enum], enum.Enum(name, {p.key: p.key for p in ordered})) + + +def build_core_phases( + descriptor: AppDescriptor, phase: type[enum.Enum] +) -> CorePhases: + """Map the descriptor's core-slot declarations onto the synthesized enum. Every + slot must be filled — the driver tags all four.""" + slot_to_key = descriptor.core_slot_map() + missing = [s.value for s in CoreSlot if s not in slot_to_key] + if missing: + raise ValueError( + f"descriptor {descriptor.name!r} is missing core phase(s): {missing}. " + "Every application must map analysis/extraction/formalization/report." + ) + return CorePhases( + analysis=phase[slot_to_key[CoreSlot.ANALYSIS]], + extraction=phase[slot_to_key[CoreSlot.EXTRACTION]], + formalization=phase[slot_to_key[CoreSlot.FORMALIZATION]], + report=phase[slot_to_key[CoreSlot.REPORT]], + ) + + +def _default_store(source: SourceCode, descriptor: AppDescriptor) -> RustArtifactStore: + return RustArtifactStore( + source.project_root, + descriptor.artifact_layout, + deliverable_mode=descriptor.deliverable_mode, + program=str(source.contract_name), + ) + + +def build_backend( + module: Any, + descriptor: AppDescriptor, + source: SourceCode, + *, + phase: type[enum.Enum] | None = None, + core_phases: CorePhases | None = None, + store_factory: StoreFactory | None = None, + backend_cls: type[RustBackend] = RustBackend, + options: BackendOptions | None = None, +) -> RustBackend: + """Construct a :class:`RustBackend` (phase enum + core phases + store + ecosystem). + + Prefer :func:`build_application` + :meth:`RustApplication.make_backend` so the + frontend and pipeline share one phase enum. This is the headless path. + """ + opts = options or BackendOptions() + ph = phase if phase is not None else build_phase_enum(descriptor) + core = core_phases if core_phases is not None else build_core_phases(descriptor, ph) + sf = store_factory or _default_store + return backend_cls( + module=module, + descriptor=descriptor, + _phase=ph, + _core_phases=core, + artifact_store=sf(source, descriptor), + ecosystem=resolve_ecosystem(descriptor), + command_timeout_s=opts.command_timeout_s, + sandbox=opts.sandbox, + declared_args=opts.declared_args, + ) + + +async def run_rust_pipeline( + module_name: str, + source_input: SourceCode, + ctx: WorkflowContext[None], + handler_factory: HandlerFactory, + env: ServiceHost, + *, + max_concurrent: int = 4, + max_bug_rounds: int = 3, + interactive: bool = False, +) -> CorePipelineResult[RustFormalResult]: + """Build the backend from ``module_name`` and run the shared driver — the Rust + analogue of ``run_autoprove_pipeline`` / ``run_foundry_pipeline``. + + This synthesizes a *fresh* phase enum internal to the backend. It is the right + entry for headless callers whose handler ignores phases; for a TUI/console + frontend, build a :class:`RustApplication` once and use :func:`run_application` + so the frontend's labels and the backend's phases share one enum object.""" + app = build_application(module_name) + return await run_application( + app, + source_input, + ctx, + handler_factory, + env, + max_concurrent=max_concurrent, + max_bug_rounds=max_bug_rounds, + interactive=interactive, + ) + + +async def run_application( + app: "RustApplication", + source_input: SourceCode, + ctx: WorkflowContext[None], + handler_factory: HandlerFactory, + env: ServiceHost, + *, + max_concurrent: int = 4, + max_bug_rounds: int = 3, + interactive: bool = False, +) -> CorePipelineResult[RustFormalResult]: + """Run a pre-built :class:`RustApplication`. The backend is constructed from the + app's already-synthesized phase enum, so the ``TaskInfo`` phases the driver emits + are the *same* enum members the frontend's ``phase_labels`` are keyed by — the + identity the frontend's label lookup relies on.""" + backend = app.make_backend(source_input) + run = PipelineRun( + ctx=ctx, source=source_input, _handler_factory=handler_factory, + _semaphore=asyncio.Semaphore(max_concurrent), env=env, + ) + return await run_pipeline( + backend, run, ecosystem=app.ecosystem, interactive=interactive, threat_model=None, max_bug_rounds=max_bug_rounds + ) + + +@dataclass +class RustApplication: + """Everything a frontend / ``main()`` needs, synthesized from the descriptor. + + ``phase_labels`` is keyed by the synthesized enum members (member identity + drives the frontend's label lookup), and ``section_order`` lists every phase + label in declared order — the two inputs a ``MultiJobApp`` frontend consumes. + + ``options`` is mutable so the CLI can apply parsed flags (timeouts, sandbox) + before :func:`run_application` without rebuilding the phase enum. + """ + + descriptor: AppDescriptor + module: Any + ecosystem: Ecosystem[Any, Any, Any] + phase: type[enum.Enum] + core_phases: CorePhases + phase_labels: dict[Any, str] + section_order: list[str] + options: BackendOptions = field(default_factory=BackendOptions) + store_factory: StoreFactory = field(default=_default_store) + backend_cls: type[RustBackend] = RustBackend + + @property + def name(self) -> str: + return self.descriptor.name + + @property + def header_text(self) -> str: + return self.descriptor.header_text + + def validate_preconditions(self, args: dict) -> str | None: + """Delegate to the Rust precondition hook; return an error string or None.""" + import json + + return self.module.validate_preconditions(json.dumps(args)) + + def make_backend(self, source: SourceCode) -> RustBackend: + """Build the backend for this run — same phase enum as :attr:`phase_labels`.""" + return build_backend( + self.module, + self.descriptor, + source, + phase=self.phase, + core_phases=self.core_phases, + store_factory=self.store_factory, + backend_cls=self.backend_cls, + options=self.options, + ) + + +def build_application( + module_name: str, + *, + store_factory: StoreFactory | None = None, + backend_cls: type[RustBackend] = RustBackend, + command_timeout_s: int = DEFAULT_TIMEOUT_S, + sandbox: SandboxConfig | None = None, +) -> RustApplication: + """Load a Rust wheel and synthesize a :class:`RustApplication`. + + ``store_factory`` / ``backend_cls`` let an application supply a specialized + store or prepared-system path (Crucible) while keeping one phase enum for the + frontend and the pipeline. + """ + module = load_module(module_name) + descriptor = load_descriptor(module) + ecosystem = resolve_ecosystem(descriptor) + phase = build_phase_enum(descriptor) + core = build_core_phases(descriptor, phase) + ordered = descriptor.ordered_phases() + phase_labels = {phase[p.key]: p.label for p in ordered} + section_order = [p.label for p in ordered] + + return RustApplication( + descriptor=descriptor, + module=module, + ecosystem=ecosystem, + phase=phase, + core_phases=core, + phase_labels=phase_labels, + section_order=section_order, + options=BackendOptions( + command_timeout_s=command_timeout_s, + sandbox=sandbox, + ), + store_factory=store_factory or _default_store, + backend_cls=backend_cls, + ) diff --git a/composer/rustapp/result.py b/composer/rustapp/result.py new file mode 100644 index 00000000..43385c8f --- /dev/null +++ b/composer/rustapp/result.py @@ -0,0 +1,68 @@ +"""The Rust backend's result type (``FormT``) and artifact identifier. + +``RustFormalResult`` is a plain pydantic model — the design's rule is that the +result stays Python-native so the driver's type-keyed cache +(``cache_get(formalizer.formalized_type)`` / ``cache_put``) round-trips it +unchanged. Rust returns its fields as JSON (a ``Formalized``); the adapter +validates them into this model. It satisfies both ``FormalResult`` +(``artifact_text`` / ``commentary`` / ``property_units()``) and +``ReportableResult`` (``skipped`` / ``property_units()`` / ``output_link``) +structurally. +""" + +from dataclasses import dataclass + +from pydantic import BaseModel, Field + +from composer.spec.cvl_generation import SkippedProperty + + +class RustFormalResult(BaseModel): + """A successful Rust formalization. ``units`` holds the property→unit-names + map as JSON-friendly lists; ``property_units()`` re-tuples it for the + protocols. The field is *not* named ``property_units`` to avoid clashing with + that required method.""" + + commentary: str = "" + artifact_text: str = "" + units: list[tuple[str, list[str]]] = Field(default_factory=list) + skipped: list[SkippedProperty] = Field(default_factory=list) + output_link: str | None = None + # Per-unit verdicts baked in at formalize time by a self-contained backend + # (unit name -> the Rust ``Verdict`` dict: {outcome, line?, duration_seconds?, + # unit_file?}). Empty for run-service-backed backends (they use fetch_verdicts). + verdicts: dict[str, dict] = Field(default_factory=dict) + + def property_units(self) -> list[tuple[str, list[str]]]: + return [(title, list(names)) for title, names in self.units] + + @classmethod + def from_formalized(cls, formalized: dict) -> "RustFormalResult": + """Build from a Rust ``Formalized`` dict (the payload of ``Command::Publish``).""" + return cls( + commentary=formalized.get("commentary", ""), + artifact_text=formalized.get("artifact_text", ""), + units=[(t, list(u)) for t, u in formalized.get("property_units", [])], + skipped=[SkippedProperty(**s) for s in formalized.get("skipped", [])], + output_link=formalized.get("output_link"), + verdicts=dict(formalized.get("verdicts", {})), + ) + + +@dataclass(frozen=True) +class RustArtifact: + """Artifact identifier for a Rust backend — ``{prefix}_{slug}.{extension}``. + Prefix/extension come from the descriptor's ``ArtifactLayout`` so naming lives + in one place.""" + + slug: str + prefix: str + extension: str + + @property + def stem(self) -> str: + return f"{self.prefix}_{self.slug}" + + @property + def artifact_file(self) -> str: + return f"{self.stem}.{self.extension}" diff --git a/composer/rustapp/results.py b/composer/rustapp/results.py new file mode 100644 index 00000000..95f85be1 --- /dev/null +++ b/composer/rustapp/results.py @@ -0,0 +1,103 @@ +"""Human-facing rollup of a Rust backend's per-unit verdicts for the console / TUI. + +The canonical results artifact is ``report.json`` (the shared report phase). But — as with the +CVL and Foundry backends — the console/TUI otherwise surface only a counts block, so a completed +run reads as "success" with no visible verdicts. This turns the per-unit verdicts baked into the +pipeline result (:attr:`RustFormalResult.verdicts`, published by ``validate``) into a compact +tally + per-unit listing, using the report's own outcome labels so the wording matches the HTML +report. + +Backend-agnostic: the outcome wording is parametrized by the descriptor's ``backend_tag`` (was +hard-coded ``crucible`` when this lived in ``composer/crucible/results.py``), so any Rust app whose +results carry verdicts gets the same summary. +""" + +from collections import Counter +from dataclasses import dataclass + +from composer.pipeline.core import CorePipelineResult, Delivered +from composer.rustapp.result import RustFormalResult +from composer.spec.source.report.render import outcome_label +from composer.spec.source.report.schema import Outcome, ReportBackend + +# Tally display order — mirrors render.py's ``_OUTCOME_ORDER`` so the console and the HTML report +# list outcomes in the same sequence. +_ORDER = [Outcome.GOOD, Outcome.BAD, Outcome.TIMEOUT, Outcome.ERROR, Outcome.UNKNOWN] + +# Per-verdict glyph — mirrors the TUI's lifecycle indicators (✓/✗) so a GOOD/BAD scans at a glance. +_GLYPH: dict[Outcome, str] = { + Outcome.GOOD: "✓", + Outcome.BAD: "✗", + Outcome.TIMEOUT: "⧖", + Outcome.ERROR: "!", + Outcome.UNKNOWN: "?", +} + + +@dataclass(frozen=True) +class UnitVerdict: + """One unit's outcome: its display name and the neutral ``Outcome``.""" + + name: str + outcome: Outcome + + +@dataclass(frozen=True) +class VerdictSummary: + """The delivered units' verdicts, in pipeline order, plus the report backend tag for wording.""" + + verdicts: list[UnitVerdict] + backend_tag: ReportBackend + + @property + def counts(self) -> dict[Outcome, int]: + """Occurrence count per outcome, in display order, omitting absent outcomes.""" + c = Counter(v.outcome for v in self.verdicts) + return {o: c[o] for o in _ORDER if c.get(o)} + + @property + def tally(self) -> str: + """A one-line ``"10 No counterexample, 1 Counterexample"`` summary (backend labels).""" + return ", ".join( + f"{n} {outcome_label(self.backend_tag, o)}" for o, n in self.counts.items() + ) + + +def _parse_outcome(raw: str) -> Outcome: + try: + return Outcome(raw) + except ValueError: + return Outcome.UNKNOWN + + +def summarize_verdicts( + result: CorePipelineResult[RustFormalResult], backend_tag: ReportBackend +) -> VerdictSummary: + """Extract the per-unit verdicts baked into a completed run's ``outcomes``. + + Only *delivered* units carry a verdict; give-ups / exceptions are already surfaced in + ``result.failures`` and skipped here. Each delivered unit bakes a single verdict, so we + read the one entry (falling back to UNKNOWN if a delivered result somehow carries none).""" + verdicts: list[UnitVerdict] = [] + for o in result.outcomes: + if not isinstance(o.result, Delivered): + continue + baked = o.result.result.verdicts + outcome = ( + _parse_outcome(next(iter(baked.values()))["outcome"]) if baked else Outcome.UNKNOWN + ) + verdicts.append(UnitVerdict(o.feat.display_name, outcome)) + return VerdictSummary(verdicts, backend_tag) + + +def format_verdict_lines(summary: VerdictSummary, *, indent: str = " ") -> list[str]: + """The ``Verdicts:`` tally line plus a per-unit listing, in the console counts-block style. + Empty when no unit was delivered (the counts/failures block already conveys that).""" + if not summary.verdicts: + return [] + lines = [f"{indent}Verdicts: {summary.tally}"] + for v in summary.verdicts: + lines.append( + f"{indent} {_GLYPH[v.outcome]} {v.name} — {outcome_label(summary.backend_tag, v.outcome)}" + ) + return lines diff --git a/composer/rustapp/store.py b/composer/rustapp/store.py new file mode 100644 index 00000000..542d958b --- /dev/null +++ b/composer/rustapp/store.py @@ -0,0 +1,63 @@ +"""Artifact store for a Rust application. + +A thin :class:`composer.spec.artifacts.ArtifactStore` subclass: the base already +writes everything identical across backends (``properties.json``, +``commentary.md``, the property→units map, ``token_usage.json``) and materializes +the artifact bytes from ``result.artifact_text``. All this subclass supplies is +the on-disk layout, taken from the descriptor's :class:`ArtifactLayout`. +""" + +from pathlib import Path +from typing import override + +from composer.rustapp.descriptor import ArtifactLayout, DeliverableMode +from composer.rustapp.result import RustArtifact, RustFormalResult +from composer.spec.artifacts import ArtifactStore +from composer.spec.util import ensure_dir + + +class RustArtifactStore(ArtifactStore[RustArtifact, RustFormalResult]): + """Persist a Rust backend's deliverables under the descriptor's layout. + + ``deliverable_mode`` selects how the *source* deliverable is written: + ``per_component`` (the default) writes one ``{prefix}_{slug}.{ext}`` file per component from + its ``artifact_text``; ``callout`` writes no per-component source — the wheel's ``finalize`` + renders the whole deliverable (e.g. Crucible's one shared crate). Either way the shared + metadata (``commentary.md`` / the property→units map) is written per component.""" + + def __init__( + self, + project_root: str, + layout: ArtifactLayout, + *, + deliverable_mode: DeliverableMode = DeliverableMode.PER_COMPONENT, + program: str = "", + ): + self._layout = layout + self._deliverable_mode = deliverable_mode + self._program = program + super().__init__( + project_root, + layout.property_suffix, + deliverable_dir=layout.deliverable_dir, + internal_dir=layout.internal_dir, + report_dir=layout.report_dir, + ) + + @override + def _artifact_dir(self) -> Path: + return ensure_dir(Path(self._project_root) / self._layout.artifact_dir) + + @override + def write_artifact(self, i: RustArtifact, artifact: RustFormalResult) -> Path: + """In ``callout`` mode, write only the shared metadata and return the (whole-deliverable) + report link — the source files come from the wheel's ``finalize``. Otherwise defer to the + base one-file-per-component writer.""" + if self._deliverable_mode is not DeliverableMode.CALLOUT: + return super().write_artifact(i, artifact) + self._write_commentary(i.stem, artifact.commentary) + self._write_property_map( + i.stem, self._property_suffix, {k: v for k, v in artifact.property_units()} + ) + primary = self._layout.deliverable_primary + return Path(primary.format(program=self._program) if primary else self._layout.deliverable_dir) diff --git a/composer/sandbox/__init__.py b/composer/sandbox/__init__.py new file mode 100644 index 00000000..ff1decda --- /dev/null +++ b/composer/sandbox/__init__.py @@ -0,0 +1,62 @@ +"""Run local commands, optionally confined to an unprivileged in-kernel sandbox. + +A backend-agnostic home for the ``RunCommand`` execution primitive +(:func:`run_local_command`) and the swappable sandbox seam +(``docs/command-sandbox.md``). It lives outside ``rustapp`` so **any** backend — +Rust-IoC *or* Python — can run untrusted native code (a compiler, a fuzzer) +confined: no network, no inherited secrets, only its own inputs on disk. + +- :mod:`composer.sandbox.policy` — the tool-agnostic seam: :class:`SandboxPolicy` + (confinement intent), :class:`SandboxProvider` (maps policy+command → a + :class:`LaunchSpec`), the ``none`` passthrough, the provider registry, and the + fail-closed helpers. +- :mod:`composer.sandbox.launcher` — the ``run-confined`` launcher provider + (Landlock + seccomp). Import it to register the ``"launcher"`` provider; the + *seam* deliberately never imports a concrete mechanism. +- :mod:`composer.sandbox.command` — :func:`run_local_command`, the single choke + point that materializes files into a workdir and runs a command there. +""" + +from composer.sandbox.command import ( + DEFAULT_TIMEOUT_S, + NOT_FOUND_EXIT, + CommandResult, + UnsafePath, + run_local_command, +) +from composer.sandbox.config import SandboxConfig +from composer.sandbox.policy import ( + Availability, + LaunchSpec, + NoneProvider, + SandboxPolicy, + SandboxProvider, + SandboxUnavailable, + ensure_available, + get_provider, + register_provider, +) +from composer.sandbox.recipes import DEFAULT_ENV_PASSTHROUGH, rust_build_policy + +__all__ = [ + # command runner + "run_local_command", + "CommandResult", + "UnsafePath", + "DEFAULT_TIMEOUT_S", + "NOT_FOUND_EXIT", + # sandbox seam + "SandboxPolicy", + "SandboxProvider", + "LaunchSpec", + "Availability", + "NoneProvider", + "SandboxUnavailable", + "get_provider", + "register_provider", + "ensure_available", + # config + recipes + "SandboxConfig", + "rust_build_policy", + "DEFAULT_ENV_PASSTHROUGH", +] diff --git a/composer/sandbox/command.py b/composer/sandbox/command.py new file mode 100644 index 00000000..d3720965 --- /dev/null +++ b/composer/sandbox/command.py @@ -0,0 +1,156 @@ +"""The local-command runner behind the ``RunCommand`` effect. + +A single choke point: materialize a set of files into a workdir, run a command +over them (as a child process, **never** a shell), and capture the result. The +trusted Python build steps (the Solana sBPF build / IDL step) route through here. + +(A Rust backend's own ``compile``/``validate`` toolchain runs no longer go through +this: they spawn the ``run-confined`` launcher directly from the wheel via +``autoprover_sdk::run_confined`` — see ``docs/rust-backend-api.md``. This runner and the +launcher share the same :mod:`composer.sandbox.policy` seam, which is why it lives in +:mod:`composer.sandbox` rather than under ``rustapp``.) + +Optional confinement is applied via a :class:`~composer.sandbox.policy.SandboxProvider` ++ :class:`~composer.sandbox.policy.SandboxPolicy` (``docs/command-sandbox.md``): +``None`` / the ``none`` provider is a passthrough; the ``launcher`` provider wraps +the argv in ``run-confined`` (Landlock + seccomp) and is fail-closed. + +**Trust boundary** (``docs/command-sandbox.md`` §2): the *caller* — a trusted Rust +decider or a trusted Python build step — supplies ``program`` and ``args``; only +file *contents* may derive from LLM output. We enforce path confinement here +(no absolute paths, no ``..`` traversal) in addition to whatever the provider does. +""" + +import asyncio +import logging +import os +from dataclasses import dataclass +from pathlib import Path, PurePosixPath + +from composer.sandbox.policy import ( + NoneProvider, + SandboxPolicy, + SandboxProvider, + ensure_available, +) + +_log = logging.getLogger(__name__) + +# Generous default; individual callers (a fuzz run vs a quick dry-run) pass their own. +DEFAULT_TIMEOUT_S = 600 + +# Exit code we synthesize when the binary isn't on PATH (mirrors shells' 127). +NOT_FOUND_EXIT = 127 + + +class UnsafePath(ValueError): + """A requested file path is absolute or escapes the workdir.""" + + +@dataclass(frozen=True) +class CommandResult: + exit_code: int + stdout: str + stderr: str + + def as_observation(self) -> dict: + """The ``Observation::CommandResult`` payload the IoC loop feeds back to Rust.""" + return { + "exit_code": self.exit_code, + "stdout": self.stdout, + "stderr": self.stderr, + } + + +def _confined_target(workdir: Path, rel: str) -> Path: + """Resolve ``rel`` under ``workdir``, rejecting absolute paths / ``..`` escapes.""" + p = PurePosixPath(rel) + if p.is_absolute() or ".." in p.parts: + raise UnsafePath( + f"file path {rel!r} is absolute or traverses outside the workdir" + ) + target = workdir / p + # Belt-and-suspenders: the resolved path must still live under the workdir. + try: + target.resolve().relative_to(workdir.resolve()) + except ValueError as e: + raise UnsafePath(f"file path {rel!r} resolves outside the workdir") from e + return target + + +async def run_local_command( + program: str, + args: list[str], + files: dict[str, str], + *, + workdir: Path, + timeout_s: int = DEFAULT_TIMEOUT_S, + sem: asyncio.Semaphore | None = None, + provider: SandboxProvider | None = None, + policy: SandboxPolicy | None = None, + env_overlay: dict[str, str] | None = None, +) -> CommandResult: + """Write ``files`` into ``workdir``, then run ``program args`` there and capture output. + + ``workdir`` persists across calls (a session materializes its crate once and + runs several commands against it). Concurrency is bounded by ``sem`` when + given — important because fuzzers are resource-hungry. + + ``provider`` selects the sandbox mechanism (``docs/command-sandbox.md``); with + the default (``None`` → the ``none`` passthrough) the command runs exactly as + before. A real provider maps ``policy`` → a confined launch and is **fail-closed** + (raises :class:`~composer.sandbox.policy.SandboxUnavailable` if it can't confine, + rather than running unsandboxed). The untrusted ``files`` are materialized by + trusted Python (path-confined via ``_confined_target``) and then the command runs + — as one unit under ``sem`` when given, so that when callers share a workdir the + file-write and the run don't interleave (a concurrent caller can't overwrite these + files between our write and our run). Path confinement complements the sandbox. + + ``env_overlay`` sets extra env vars on the child on top of what it would otherwise + inherit — used by the *unsandboxed* prep steps (e.g. `cargo fetch` with a per-run + ``CARGO_HOME``); the sandboxed path's env is fully governed by the provider/policy. + """ + prov: SandboxProvider = provider if provider is not None else NoneProvider() + ensure_available(prov) # fail-closed: raises before running if it can't confine + spec = prov.wrap(policy if policy is not None else SandboxPolicy(), program, list(args)) + child_env = dict(spec.env) if spec.env is not None else None + if env_overlay: + # Overlay onto the effective env (the inherited parent env when the provider + # didn't set one — i.e. the `none`/unsandboxed path). + child_env = {**(child_env if child_env is not None else os.environ), **env_overlay} + + async def _run() -> CommandResult: + # Materialize files + launch as one unit so a shared workdir stays consistent + # for this command's duration (see `sem`). + workdir.mkdir(parents=True, exist_ok=True) + for rel, contents in files.items(): + target = _confined_target(workdir, rel) + target.parent.mkdir(parents=True, exist_ok=True) + target.write_text(contents) + try: + proc = await asyncio.create_subprocess_exec( + *spec.argv, + cwd=str(workdir), + env=child_env, + stdout=asyncio.subprocess.PIPE, + stderr=asyncio.subprocess.PIPE, + ) + except FileNotFoundError: + return CommandResult(NOT_FOUND_EXIT, "", f"{spec.argv[0]}: not found on PATH") + try: + out_b, err_b = await asyncio.wait_for( + proc.communicate(), timeout=timeout_s + ) + except asyncio.TimeoutError: + proc.kill() + await proc.wait() + return CommandResult(-1, "", f"command timed out after {timeout_s}s") + rc = proc.returncode if proc.returncode is not None else -1 + return CommandResult( + rc, out_b.decode(errors="replace"), err_b.decode(errors="replace") + ) + + if sem is not None: + async with sem: + return await _run() + return await _run() diff --git a/composer/sandbox/config.py b/composer/sandbox/config.py new file mode 100644 index 00000000..8b9824a2 --- /dev/null +++ b/composer/sandbox/config.py @@ -0,0 +1,98 @@ +"""Runtime selection of the command-sandbox provider + policy. + +A backend constructs a :class:`SandboxConfig` (usually via :meth:`from_env`) and +hands it to the command path (``RealEffects`` / ``build_program``), which turns it +into a concrete ``(provider, policy)`` per command via :meth:`resolve_provider` and +:meth:`build_policy`. Keeping selection here — rather than in :func:`run_local_command` +— means the runner stays mechanism-agnostic (``docs/command-sandbox.md`` §4/§7). + +The library default provider is ``"none"`` (passthrough). Backends that run +untrusted native code (Crucible) construct a config with ``provider="launcher"`` +by default; override with ``COMPOSER_SANDBOX_PROVIDER=none`` for trusted-input dev. +""" + +import os +from dataclasses import dataclass +from pathlib import Path + +from composer.sandbox.policy import SandboxPolicy, SandboxProvider, ensure_available, get_provider +from composer.sandbox.recipes import DEFAULT_ENV_PASSTHROUGH, rust_build_policy + +_ENV_VAR = "COMPOSER_SANDBOX_PROVIDER" + + +@dataclass(frozen=True) +class SandboxConfig: + """Which provider to use + the inputs for building its policy.""" + + provider: str = "none" + extra_ro: tuple[Path, ...] = () + extra_rw: tuple[Path, ...] = () + env_passthrough: tuple[str, ...] = DEFAULT_ENV_PASSTHROUGH + offline: bool = True # sandbox has no network → force cargo offline (§5) + mem_bytes: int | None = None + cpu_seconds: int | None = None + nproc: int | None = None + fsize_bytes: int | None = None + + @classmethod + def from_env(cls, **overrides) -> "SandboxConfig": + """Read the provider from ``$COMPOSER_SANDBOX_PROVIDER`` (default ``none``); + remaining fields come from ``overrides`` (e.g. a backend's ``extra_ro``).""" + return cls(provider=os.environ.get(_ENV_VAR, "none"), **overrides) + + @property + def enabled(self) -> bool: + return self.provider != "none" + + def resolve_provider(self) -> SandboxProvider: + # Importing the launcher module registers the "launcher" provider; the seam + # itself never imports a concrete mechanism (docs/command-sandbox.md §6). + if self.provider == "launcher": + import composer.sandbox.launcher # noqa: F401 + return get_provider(self.provider) + + def build_policy(self, workdir: str | Path) -> SandboxPolicy: + """The concrete policy for a command running in ``workdir``. The ``none`` + provider ignores the policy, so a bare :class:`SandboxPolicy` suffices there.""" + if not self.enabled: + return SandboxPolicy() + return rust_build_policy( + workdir, + extra_ro=self.extra_ro, + extra_rw=self.extra_rw, + env_passthrough=self.env_passthrough, + offline=self.offline, + mem_bytes=self.mem_bytes, + cpu_seconds=self.cpu_seconds, + nproc=self.nproc, + fsize_bytes=self.fsize_bytes, + ) + + def backend_spec(self, workdir: str | Path, *, timeout_s: int) -> dict: + """The ``Sandbox`` JSON a Rust backend's ``compile``/``validate`` consume to build + their own ``run-confined`` launch (`autoprover_sdk::Sandbox`). Python keeps ownership + of the confinement *intent* (this policy); the backend only assembles it into an argv. + + For a real provider this resolves the ``run-confined`` path and is **fail-closed** + (``ensure_available`` raises if the launcher can't confine here). The ``none`` provider + yields ``run_confined=None`` — the backend runs the command directly (trusted input).""" + if not self.enabled: + return {"run_confined": None, "timeout_s": timeout_s} + provider = self.resolve_provider() + ensure_available(provider) # fail-closed: raise before any untrusted code runs + policy = self.build_policy(workdir) + return { + "run_confined": getattr(provider, "binary", None), + "ro": [str(p) for p in policy.ro_paths], + "rw": [str(p) for p in policy.rw_paths], + "allow_env": [f"{k}={v}" for k, v in policy.env_allowlist.items()], + "network": policy.network, + "rlimits": { + "mem_bytes": policy.mem_bytes, + "cpu_seconds": policy.cpu_seconds, + "nproc": policy.nproc, + "fsize_bytes": policy.fsize_bytes, + }, + "timeout_s": timeout_s, + } diff --git a/composer/sandbox/launcher.py b/composer/sandbox/launcher.py new file mode 100644 index 00000000..0aff4bce --- /dev/null +++ b/composer/sandbox/launcher.py @@ -0,0 +1,114 @@ +"""The ``run-confined`` launcher provider — the first real :class:`SandboxProvider`. + +Maps a tool-agnostic :class:`SandboxPolicy` to an invocation of the ``run-confined`` +trusted Rust binary (``rust/run-confined``), which applies Landlock + seccomp + +rlimits + a scrubbed env to itself and then ``execve``s the command +(``docs/command-sandbox.md`` §6). This module is deliberately *separate* from the +:mod:`composer.sandbox.policy` seam: importing it registers the ``"launcher"`` +provider, so the seam never imports a concrete mechanism. + +``wrap`` is pure argv construction (unit-testable, no subprocess); ``available`` +shells out to ``run-confined --probe`` once to confirm the kernel supports Landlock +(fail-closed otherwise). +""" + +import os +import shutil +import subprocess +from pathlib import Path + +from composer.sandbox.policy import ( + Availability, + LaunchSpec, + SandboxPolicy, + register_provider, +) + +_BIN_NAME = "run-confined" +_PROBE_TIMEOUT_S = 10 + + +def _resolve_binary() -> str | None: + """Locate the ``run-confined`` binary: ``$RUN_CONFINED_BIN`` → ``PATH`` → the + dev build under ``rust/target/release`` (repo-relative). ``None`` if unbuilt.""" + override = os.environ.get("RUN_CONFINED_BIN") + if override and Path(override).is_file(): + return override + on_path = shutil.which(_BIN_NAME) + if on_path: + return on_path + # Dev fallback: composer/sandbox/launcher.py → repo root is parents[2]. + repo_root = Path(__file__).resolve().parents[2] + cand = repo_root / "rust" / "target" / "release" / _BIN_NAME + return str(cand) if cand.is_file() else None + + +class LauncherProvider: + """Confines commands via the ``run-confined`` launcher (Landlock + seccomp).""" + + name = "launcher" + + def __init__(self, binary: str | None = None): + # Resolve at construction so `available()` and `wrap()` agree on the path; + # tests pass an explicit binary to keep `wrap()` golden-testable offline. + self._binary = binary if binary is not None else _resolve_binary() + + @property + def binary(self) -> str | None: + return self._binary + + def available(self) -> Availability: + if self._binary is None: + return Availability( + ok=False, + reason=( + f"{_BIN_NAME} binary not found; build rust/run-confined " + f"(cargo build -p run-confined --release) or set RUN_CONFINED_BIN" + ), + ) + try: + proc = subprocess.run( + [self._binary, "--probe"], + capture_output=True, + text=True, + timeout=_PROBE_TIMEOUT_S, + ) + except OSError as e: + return Availability(ok=False, reason=f"{_BIN_NAME} --probe could not run: {e}") + if proc.returncode != 0: + reason = proc.stderr.strip() or f"{_BIN_NAME} --probe reported no Landlock support" + return Availability(ok=False, reason=reason) + return Availability(ok=True) + + def wrap(self, policy: SandboxPolicy, program: str, args: list[str]) -> LaunchSpec: + """Build the ``run-confined … -- program args`` argv from ``policy``. + + Emits ``--allow-env NAME=VALUE`` (explicit values — the allowlist holds only + benign build vars, never secrets). ``env`` stays ``None``: the launcher + inherits AutoProver's env but scrubs it to the allowlist for the child, so + the child's environment is fully determined by the flags.""" + argv: list[str] = [self._binary or _BIN_NAME] + for p in policy.ro_paths: + argv += ["--ro", str(p)] + for p in policy.rw_paths: + argv += ["--rw", str(p)] + for name, value in policy.env_allowlist.items(): + argv += ["--allow-env", f"{name}={value}"] + if policy.network: + argv.append("--allow-network") + if policy.mem_bytes is not None: + argv += ["--rlimit-as", str(policy.mem_bytes)] + if policy.cpu_seconds is not None: + argv += ["--rlimit-cpu", str(policy.cpu_seconds)] + if policy.nproc is not None: + argv += ["--rlimit-nproc", str(policy.nproc)] + if policy.fsize_bytes is not None: + argv += ["--rlimit-fsize", str(policy.fsize_bytes)] + argv += ["--", program, *args] + return LaunchSpec(argv=tuple(argv), env=None) + + +# Registering on import keeps the `composer.sandbox.policy` seam free of any +# concrete-mechanism import. Consumers (RealEffects, tests) `import` this module to +# make ``get_provider("launcher")`` resolvable. +register_provider("launcher", LauncherProvider) diff --git a/composer/sandbox/policy.py b/composer/sandbox/policy.py new file mode 100644 index 00000000..67c12f87 --- /dev/null +++ b/composer/sandbox/policy.py @@ -0,0 +1,165 @@ +"""The command-sandbox provider seam (``docs/command-sandbox.md`` §4, §7). + +Every ``RunCommand`` invocation compiles and/or runs *untrusted native code* (an +LLM-authored harness, a user program's ``build.rs``), so it must run confined — +no network, no inherited secrets, only its own inputs on the filesystem. This +module is the **tool-agnostic isolation layer** that makes that confinement +*swappable*: + +- :class:`SandboxPolicy` — the confinement *intent* (rw/ro paths, env allowlist, + network on/off, resource caps). It names **no mechanism**, so swapping the + sandbox tool never changes the policy. +- :class:`SandboxProvider` — maps a policy + a command to a concrete + :class:`LaunchSpec` (the argv/env to actually exec). The mechanism (a + Landlock+seccomp launcher, or an off-the-shelf tool like ``landrun`` / + ``sandlock``) lives entirely behind this protocol. + +Because :func:`composer.sandbox.command.run_local_command` will depend only on +this seam — never on a concrete tool — a provider can be swapped without touching +the command runner, ``RealEffects``, or the escape-test gate. It lives outside +``rustapp`` so Python-based backends can use it too, not just the Rust-IoC ones. + +This module ships the policy, the protocol, the ``none`` passthrough provider, +and the provider registry. The ``run-confined`` launcher provider (Landlock + +seccomp) registers itself under ``"launcher"`` when +:mod:`composer.sandbox.launcher` is imported (typically via +:meth:`composer.sandbox.config.SandboxConfig.resolve_provider`). + +**Trust boundary** (``docs/command-sandbox.md`` §7.2): the policy and the emitted +``LaunchSpec`` are authored by trusted Python — never the LLM, which controls only +file *contents*. +""" + +from collections.abc import Callable, Mapping +from dataclasses import dataclass, field +from pathlib import Path +from typing import Protocol, runtime_checkable + + +class SandboxUnavailable(RuntimeError): + """A sandbox provider was requested but cannot confine the command. + + Raised (fail-closed) instead of silently running unconfined — untrusted input + must never run without the sandbox (``docs/command-sandbox.md`` §7). Carries the + provider name + a human reason so the caller can surface a prominent message. + """ + + def __init__(self, provider: str, reason: str): + self.provider = provider + self.reason = reason + super().__init__(f"command sandbox provider {provider!r} is unavailable: {reason}") + + +@dataclass(frozen=True) +class SandboxPolicy: + """The confinement *intent* — tool-agnostic (``docs/command-sandbox.md`` §7). + + Every :class:`SandboxProvider` consumes *this* shape, so a mechanism swap needs + no policy change. ``program``/``args`` are passed per-call to + :meth:`SandboxProvider.wrap`, not stored here. Resource caps default to ``None`` + (unset); a provider maps them to its own limit mechanism (rlimits for the + launcher). + """ + + rw_paths: tuple[Path, ...] = () # writable: the workdir (+ any scratch) + ro_paths: tuple[Path, ...] = () # read+exec: toolchains, crucible checkout, /usr… + env_allowlist: Mapping[str, str] = field(default_factory=dict) + network: bool = False # egress allowed? default off + mem_bytes: int | None = None # RLIMIT_AS + cpu_seconds: int | None = None # RLIMIT_CPU + nproc: int | None = None # RLIMIT_NPROC + fsize_bytes: int | None = None # RLIMIT_FSIZE + + +@dataclass(frozen=True) +class LaunchSpec: + """How :func:`run_local_command` should actually launch the (confined) command. + + ``argv`` is the full argument vector to exec; ``env`` is the environment to pass + (``None`` = inherit the parent's, i.e. today's unconfined behavior). Both are + authored by trusted code, never the LLM. + """ + + argv: tuple[str, ...] + env: Mapping[str, str] | None = None + + +@dataclass(frozen=True) +class Availability: + """Result of :meth:`SandboxProvider.available` — whether the provider can + actually confine here (e.g. the launcher probes the kernel's Landlock ABI).""" + + ok: bool + reason: str = "" + + +@runtime_checkable +class SandboxProvider(Protocol): + """Maps a :class:`SandboxPolicy` + a command to a concrete :class:`LaunchSpec`. + + The one seam every sandbox mechanism implements. Implementations are pure with + respect to :meth:`wrap` (argv construction only — no subprocess), so they are + trivially unit-testable; the actual confinement happens in the launched process. + """ + + name: str + + def available(self) -> Availability: + """Whether this provider can confine a command in the current environment.""" + ... + + def wrap(self, policy: SandboxPolicy, program: str, args: list[str]) -> LaunchSpec: + """Translate ``policy`` into how to launch ``program args`` confined.""" + ... + + +class NoneProvider: + """Passthrough — **no confinement**. Exec the command directly, inheriting the + environment: byte-for-byte today's behavior. + + An *explicit, logged* choice for the trusted EVM/Foundry callers and + trusted-input dev runs. It is never reached as a silent fallback from a failed + real sandbox (``docs/command-sandbox.md`` §7) — the caller selects it on purpose. + """ + + name = "none" + + def available(self) -> Availability: + return Availability(ok=True) + + def wrap(self, policy: SandboxPolicy, program: str, args: list[str]) -> LaunchSpec: + # Policy is intentionally ignored: this provider provides no isolation. + return LaunchSpec(argv=(program, *args), env=None) + + +# Provider registry. The ``launcher`` factory is registered by importing +# :mod:`composer.sandbox.launcher` (see :meth:`SandboxConfig.resolve_provider`). +_PROVIDERS: dict[str, Callable[[], SandboxProvider]] = { + "none": NoneProvider, +} + + +def register_provider(name: str, factory: Callable[[], SandboxProvider]) -> None: + """Register a provider factory under ``name`` (used by later steps to add the + launcher / off-the-shelf providers without this module importing them).""" + _PROVIDERS[name] = factory + + +def get_provider(name: str) -> SandboxProvider: + """Construct the provider registered under ``name``. Raises ``ValueError`` for an + unknown name (a config error, distinct from a provider being *unavailable*).""" + try: + factory = _PROVIDERS[name] + except KeyError: + raise ValueError( + f"unknown sandbox provider {name!r}; known: {sorted(_PROVIDERS)}" + ) from None + return factory() + + +def ensure_available(provider: SandboxProvider) -> None: + """Fail-closed check: raise :class:`SandboxUnavailable` unless ``provider`` can + confine here. Call before running untrusted input under a real provider.""" + avail = provider.available() + if not avail.ok: + raise SandboxUnavailable(provider.name, avail.reason) diff --git a/composer/sandbox/recipes.py b/composer/sandbox/recipes.py new file mode 100644 index 00000000..edb78701 --- /dev/null +++ b/composer/sandbox/recipes.py @@ -0,0 +1,158 @@ +"""Ready-made :class:`SandboxPolicy` recipes. + +The seam (:mod:`composer.sandbox.policy`) is mechanism- *and* workload-agnostic; +this module holds opinionated builders for common workloads. :func:`rust_build_policy` +covers "compile and/or run Rust" (``cargo build-sbf``, ``cargo build``, ``crucible +run``): it grants the workdir read-write, the discoverable Rust/Solana toolchains +read-only, the device nodes the toolchain needs, and an env allowlist — with the +network off. Any Rust backend reuses it; Crucible adds its own paths via ``extra_ro``. + +Paths are included only if they exist, so the same recipe works across machines +with different toolchain layouts (and the escape-test gate can prove exactly what +was and wasn't granted). +""" + +import os +import shutil +from pathlib import Path + +from composer.sandbox.policy import SandboxPolicy + +# Benign build vars passed through to the child (values read from the current env). +# Never secrets — the whole point is that secrets are *not* inherited. +DEFAULT_ENV_PASSTHROUGH: tuple[str, ...] = ( + "PATH", + "HOME", + "TERM", + "LANG", + "LC_ALL", + "USER", + "LOGNAME", + "TMPDIR", + "CARGO_HOME", + "RUSTUP_HOME", + "SSL_CERT_FILE", + "SSL_CERT_DIR", +) + +# Read-only system directories the toolchain + its dynamic linker need. ``/etc`` is +# included because glibc NSS (``getpwuid`` via ``getuser``, CA-cert lookup) reads +# ``/etc/passwd`` / ``/etc/nsswitch.conf``; it holds no AutoProver secret (those are +# in the scrubbed env and in files we never grant). The escape gate must therefore +# probe a *planted* host file / the parent's environ, not ``/etc/passwd``. +_SYSTEM_RO: tuple[str, ...] = ("/usr", "/bin", "/sbin", "/lib", "/lib64", "/etc") + +# Device nodes the toolchain opens (rw so ``/dev/null`` writes work). Granting the +# node files — not the whole ``/dev`` tree; ``mknod`` stays blocked (no capability). +_DEV_NODES: tuple[str, ...] = ( + "/dev/null", + "/dev/zero", + "/dev/full", + "/dev/random", + "/dev/urandom", + "/dev/tty", +) + + +def sandbox_cargo_home(workdir: str | Path) -> Path: + """The **private, per-run `CARGO_HOME`** for a sandboxed build, under the workdir. + + Why a private cargo home rather than the shared `~/.cargo`: + + An offline `cargo build` doesn't just *read* the cache — it *writes* to `CARGO_HOME` + (extracts crate sources into `registry/src`, takes `.package-cache` locks). To let + the confined build do that we'd have to grant `CARGO_HOME` read-write. But the same + build runs **untrusted `build.rs`/proc-macro code**, so a writable *shared* cargo + home is a cross-run attack surface: a malicious build could overwrite an extracted + source under `registry/src` and poison a *later* run that compiles that crate (cargo + checksums the downloaded `.crate`, but trusts an already-extracted `registry/src`). + + A per-run home under the (already-writable, per-run) workdir removes that: any write + the untrusted build makes touches only this run's throwaway cache, never a shared one. + The cost is that deps are fetched per run (the warm step downloads into this home); + a shared *read-only* index/cache to avoid re-download is a deferred optimization + (command-sandbox.md §11 item 5). + """ + return Path(workdir).resolve() / ".sandbox_cargo" + + +def rust_build_policy( + workdir: str | Path, + *, + extra_ro: tuple[Path, ...] = (), + extra_rw: tuple[Path, ...] = (), + env_passthrough: tuple[str, ...] = DEFAULT_ENV_PASSTHROUGH, + offline: bool = True, + mem_bytes: int | None = None, + cpu_seconds: int | None = None, + nproc: int | None = None, + fsize_bytes: int | None = None, +) -> SandboxPolicy: + """Build a network-off policy for compiling/running Rust in ``workdir``. + + Grants: ``workdir`` + the device nodes (+ ``extra_rw``) read-write; the Rust + (``RUSTUP_HOME``/``CARGO_HOME``) and Solana platform-tool directories, the system + dirs, and ``extra_ro`` read-only. Non-existent paths are dropped. + + With ``offline`` (the default — the sandbox has no network, §5), ``CARGO_NET_OFFLINE=1`` + is set in the child env. That one var forces *every* cargo invocation offline, + including the nested ``cargo`` that ``crucible run`` spawns to build the harness — + so the deps must already be warm in ``CARGO_HOME`` (see :func:`warm_cargo_cache`, + run *outside* the sandbox first). + """ + home = Path.home() + rustup = Path(os.environ.get("RUSTUP_HOME", home / ".rustup")) + cargo = Path(os.environ.get("CARGO_HOME", home / ".cargo")) + + ro_candidates: list[Path] = [Path(p) for p in _SYSTEM_RO] + ro_candidates += [ + rustup, + cargo, + # cargo-build-sbf's downloaded sBPF platform-tools (layout varies by version). + home / ".cache" / "solana", + home / ".local" / "share" / "solana", + ] + ro_candidates += list(extra_ro) + # Absolute paths only: the launcher opens each relative to *its* cwd (the workdir), + # so a relative grant would resolve wrong. resolve() also canonicalizes symlinks. + ro_paths = tuple(p.resolve() for p in ro_candidates if p.exists()) + + dev = tuple(Path(d).resolve() for d in _DEV_NODES if Path(d).exists()) + wd = Path(workdir).resolve() + rw_paths = (wd, *dev, *(p.resolve() for p in extra_rw)) + + env = {name: os.environ[name] for name in env_passthrough if name in os.environ} + if offline: + env["CARGO_NET_OFFLINE"] = "1" + # A private temp dir UNDER the (writable) workdir, so tools that need scratch space + # — notably the linker, which writes to $TMPDIR (default /tmp) during `cargo build` — + # work without granting the shared /tmp (which may hold host/other-run secrets and + # would defeat the escape test). Created here so $TMPDIR points at an existing dir. + sandbox_tmp = wd / ".sandbox_tmp" + sandbox_tmp.mkdir(parents=True, exist_ok=True) + for var in ("TMPDIR", "TMP", "TEMP"): + env[var] = str(sandbox_tmp) + + # Point CARGO_HOME at a PRIVATE per-run cargo home under the workdir (see + # sandbox_cargo_home for the reasoning). The shared ~/.cargo stays read-only (its + # `bin/cargo` is still on PATH; we only redirect where cargo *writes*). Copy the + # user's global cargo config in so registry mirrors / build settings still apply. + cargo_home = sandbox_cargo_home(wd) + cargo_home.mkdir(parents=True, exist_ok=True) + shared_cargo = Path(os.environ.get("CARGO_HOME", Path.home() / ".cargo")) + for cfg in ("config.toml", "config"): + src = shared_cargo / cfg + if src.is_file() and not (cargo_home / cfg).exists(): + shutil.copy(src, cargo_home / cfg) + env["CARGO_HOME"] = str(cargo_home) + + return SandboxPolicy( + rw_paths=rw_paths, + ro_paths=ro_paths, + env_allowlist=env, + network=False, + mem_bytes=mem_bytes, + cpu_seconds=cpu_seconds, + nproc=nproc, + fsize_bytes=fsize_bytes, + ) diff --git a/composer/scripts/init-db.sql b/composer/scripts/init-db.sql index 89cf7b20..474f3f32 100644 --- a/composer/scripts/init-db.sql +++ b/composer/scripts/init-db.sql @@ -11,6 +11,9 @@ DO $$ BEGIN IF NOT EXISTS (SELECT FROM pg_catalog.pg_roles WHERE rolname = 'foundry_rag_user') THEN CREATE USER foundry_rag_user WITH PASSWORD 'rag_password'; END IF; + IF NOT EXISTS (SELECT FROM pg_catalog.pg_roles WHERE rolname = 'crucible_rag_user') THEN + CREATE USER crucible_rag_user WITH PASSWORD 'rag_password'; + END IF; IF NOT EXISTS (SELECT FROM pg_catalog.pg_roles WHERE rolname = 'langgraph_store_user') THEN CREATE USER langgraph_store_user WITH PASSWORD 'langgraph_store_password'; END IF; @@ -56,6 +59,11 @@ CREATE SCHEMA IF NOT EXISTS foundry_rag AUTHORIZATION foundry_rag_user; GRANT USAGE ON SCHEMA extensions TO foundry_rag_user; ALTER ROLE foundry_rag_user IN DATABASE rag_db SET search_path = foundry_rag, extensions; +-- crucible rag (Solana fuzzing knowledge base) +CREATE SCHEMA IF NOT EXISTS crucible_rag AUTHORIZATION crucible_rag_user; +GRANT USAGE ON SCHEMA extensions TO crucible_rag_user; +ALTER ROLE crucible_rag_user IN DATABASE rag_db SET search_path = crucible_rag, extensions; + \c langgraph_store_db CREATE EXTENSION IF NOT EXISTS vector; diff --git a/composer/scripts/rag_import.py b/composer/scripts/rag_import.py new file mode 100644 index 00000000..26b0c417 --- /dev/null +++ b/composer/scripts/rag_import.py @@ -0,0 +1,166 @@ +"""Generic RAG importer — ingest any corpus described by a common JSON manifest. + +This is the shared back half of the RAG build, factored out of the old per-corpus builders (see +``docs/rag-import-format.md``): it reads one or more :class:`~composer.rag.import_format.RagManifest` +documents and owns everything downstream — length-bounded chunking (``BlockBuilder``), embedding, +``part`` numbering, and the *dual-path* ingestion every corpus wants: + +* ``add_chunks_batch`` — length-bounded embedded chunks for **vector** (semantic) search; +* ``add_manual_section`` — the full section for **keyword** search + exact ``get_section``. + +Both paths are always populated — there is no per-section knob; "ingest a corpus" means feed both +indexes. A *producer* does the corpus-specific parsing and emits the manifest; this module is +corpus-agnostic. Crucible's corpus is a committed manifest (``rust/crucible-app/crucible_kb.rag.json``). + +Run under the ragbuild uv group (has spaCy + sentence-transformers):: + + uv run --isolated --group ragbuild python -m composer.scripts.rag_import \\ + corpus.rag.json [more.rag.json ...] [--output ] [--max-length N] [--print] +""" + +import argparse +import asyncio +import logging +import pathlib +from collections import defaultdict + +import spacy + +from composer.rag.db import KNOWLEDGE_BASES, get_rag_db +from composer.rag.import_format import RagManifest, Section, SCHEMA_VERSION +from composer.rag.models import get_model +from composer.rag.text import code_ref_tag +from composer.rag.types import BlockChunk +from composer.scripts.text_processors import BlockBuilder, BuilderConfig + +logging.basicConfig(level=logging.INFO, format="%(asctime)s - %(levelname)s - %(message)s") +logger = logging.getLogger(__name__) + +_BATCH_SIZE = 50 + + +def _manual_chunk(sec: Section) -> BlockChunk: + """One full-section chunk (code as ```` tags) for keyword / get-section.""" + parts: list[str] = [] + code_refs: list[str] = [] + for b in sec.blocks: + if b.kind == "code": + parts.append(code_ref_tag(len(code_refs))) + code_refs.append(b.body) + else: + parts.append(b.body) + return BlockChunk(headers=list(sec.headers), part=0, code_refs=code_refs, chunk="\n\n".join(parts)) + + +def _embedded_chunks(sec: Section, config: BuilderConfig) -> list[BlockChunk]: + """Length-bounded embedded chunks for vector search, via the shared builder.""" + builder = BlockBuilder(header=list(sec.headers), config=config) + for b in sec.blocks: + if b.kind == "code": + builder.add_code(b.body) + else: + builder.append_text(b.body, is_structured_boundary=True, unbreakable=False) + return list(builder.finish()) + + +def _load_manifest(path: pathlib.Path) -> RagManifest: + manifest = RagManifest.model_validate_json(path.read_text()) + if manifest.version != SCHEMA_VERSION: + raise SystemExit( + f"{path}: unsupported manifest version {manifest.version} (this importer speaks " + f"v{SCHEMA_VERSION}). Regenerate the manifest with a matching producer." + ) + return manifest + + +def _resolve_output(manifest: RagManifest, override: str | None) -> str: + if override: + return override + conn = KNOWLEDGE_BASES.get(manifest.knowledge_base) + if conn is None: + raise SystemExit( + f"no connection registered for knowledge_base {manifest.knowledge_base!r} " + f"(known: {sorted(KNOWLEDGE_BASES)}). Add it to composer.rag.db.KNOWLEDGE_BASES " + f"or pass --output ." + ) + return conn + + +def _print_manifest(manifest: RagManifest) -> None: + """Dry-run: render each section's full-section chunk to stdout, no DB writes.""" + print(f"=== knowledge_base: {manifest.knowledge_base} (source: {manifest.source})") + for s in manifest.sections: + print(f"\n#### {' / '.join(h for h in s.headers if h)}") + print(_manual_chunk(s).chunk[:500]) + + +async def _ingest( + db, manifest: RagManifest, config: BuilderConfig, seen_paths: dict[tuple[str, ...], int] +) -> tuple[int, int]: + """Ingest one manifest's sections into ``db``, feeding both indexes. ``seen_paths`` is shared + across manifests targeting the same DB so the ``manual_sections`` ``(headers, part)`` unique + key never collides.""" + buffer: list[BlockChunk] = [] + n_docs = n_manual = 0 + for s in manifest.sections: + buffer.extend(_embedded_chunks(s, config)) + if len(buffer) >= _BATCH_SIZE: + await db.add_chunks_batch(buffer) + n_docs += len(buffer) + buffer = [] + manual = _manual_chunk(s) + key = tuple(manual.headers) + manual.part = seen_paths.get(key, 0) + seen_paths[key] = manual.part + 1 + await db.add_manual_section(manual) + n_manual += 1 + if buffer: + await db.add_chunks_batch(buffer) + n_docs += len(buffer) + return n_docs, n_manual + + +async def _async_main(args: argparse.Namespace) -> None: + manifests = [_load_manifest(f) for f in args.files] + + if args.print: + for m in manifests: + _print_manifest(m) + return + + config = BuilderConfig(nlp=spacy.load("en_core_web_sm"), max_length=args.max_length) + model = get_model() + + # Group by resolved target so manifests sharing a DB share one connection + one part counter. + groups: dict[str, list[RagManifest]] = defaultdict(list) + for m in manifests: + groups[_resolve_output(m, args.output)].append(m) + + for output, group in groups.items(): + db = await get_rag_db(output, model) + seen_paths: dict[tuple[str, ...], int] = {} + n_docs = n_manual = 0 + for m in group: + d, mn = await _ingest(db, m, config, seen_paths) + n_docs += d + n_manual += mn + logger.info( + "ingested %d embedded chunk(s) + %d manual section(s) from %d manifest(s) into %s", + n_docs, n_manual, len(group), output, + ) + + +def main() -> None: + parser = argparse.ArgumentParser(description="Ingest a RAG corpus from one or more JSON manifests.") + parser.add_argument("files", nargs="+", type=pathlib.Path, help="RAG manifest JSON files.") + parser.add_argument("--max-length", type=int, default=2000, help="Soft cap on embedded-chunk length (chars).") + parser.add_argument( + "--output", "-o", default=None, + help="RAG DB connection string. Overrides the manifest's knowledge_base -> connection lookup.", + ) + parser.add_argument("--print", action="store_true", help="Dry-run: print sections, no DB writes.") + asyncio.run(_async_main(parser.parse_args())) + + +if __name__ == "__main__": + main() diff --git a/composer/spec/prop_inference.py b/composer/spec/prop_inference.py index 3edc92c0..635137ab 100644 --- a/composer/spec/prop_inference.py +++ b/composer/spec/prop_inference.py @@ -21,7 +21,7 @@ from composer.spec.context import WorkflowContext, CacheKey, ComponentGroup from composer.spec.graph_builder import bind_standard, run_to_completion from composer.spec.types import PropertyFormulation -from composer.spec.system_model import ContractComponentInstance +from composer.spec.system_model import FeatureUnit from composer.tools.thinking import RoughDraftState, get_rough_draft_tools from composer.spec.service_host import Sort, ServiceHost from composer.io.conversation import ConversationContextProvider @@ -101,13 +101,14 @@ class RefinementState(MessagesState): def _get_initial_prompt( - context: ContractComponentInstance, + context: FeatureUnit, sort: Sort, prev_results: list[_AgentRoundResult], backend_guidance: str, + template: str = "property_analysis_prompt.j2", ) -> str: return load_jinja_template( - "property_analysis_prompt.j2", + template, context=context, backend_guidance=backend_guidance, sort=sort, @@ -279,12 +280,14 @@ def validate(_state: Any, result: _AgentRoundResult) -> str | None: async def _run_bug_round( env: ServiceHost, - component: ContractComponentInstance, + component: FeatureUnit, front_matter_items: Sequence[str | dict], ctx: WorkflowContext[_AgentResult], round: int, prev: list[_AgentRoundResult], backend_guidance: str, + system_template: str = "property_analysis_system_prompt.j2", + initial_template: str = "property_analysis_prompt.j2", ) -> _AgentRoundWithHistory: round_ctx = ctx.child(agent_round_key(round)) if (cached := await round_ctx.cache_get(_AgentRoundWithHistory)) is not None: @@ -305,13 +308,13 @@ class ST(MessagesState, RoughDraftState): ).with_input( BugAnalysisInput ).with_initial_prompt( - _get_initial_prompt(component, env.sort, prev, backend_guidance) + _get_initial_prompt(component, env.sort, prev, backend_guidance, initial_template) ).with_tools( get_rough_draft_tools(ST) ).with_tools( env.analysis_tools ).with_sys_prompt_template( - "property_analysis_system_prompt.j2", sort=env.sort + system_template, sort=env.sort ).compile_async() flow_input: BugAnalysisInput = BugAnalysisInput( @@ -341,11 +344,13 @@ class ST(MessagesState, RoughDraftState): async def _run_bug_analysis_inner( agent_component_analysis: WorkflowContext[_AgentResult], env: ServiceHost, - component: ContractComponentInstance, + component: FeatureUnit, extra_input: Sequence[str | dict], threat_model: Document | None, max_rounds: int, backend_guidance: str, + system_template: str = "property_analysis_system_prompt.j2", + initial_template: str = "property_analysis_prompt.j2", ) -> _AgentResult: if (cached := await agent_component_analysis.cache_get(_AgentResult)) is not None: return cached @@ -367,7 +372,7 @@ async def _run_bug_analysis_inner( for i in range(0, max_rounds): next_result = await _run_bug_round( env, component, front_matter_items, agent_component_analysis, i, prev_rounds, - backend_guidance, + backend_guidance, system_template, initial_template, ) if len(next_result.items) == 0: assert last_round_convo is not None @@ -389,12 +394,15 @@ async def _run_bug_analysis_inner( async def run_property_inference( ctx: WorkflowContext[ComponentGroup], env: ServiceHost, - component: ContractComponentInstance, + component: FeatureUnit, extra_input : Sequence[str | dict] = tuple(), threat_model: Document | None = None, refinement: ConversationContextProvider | None = None, max_rounds: int = 3, backend_guidance: str = CERTORA_BACKEND_GUIDANCE, + *, + system_template: str = "property_analysis_system_prompt.j2", + initial_template: str = "property_analysis_prompt.j2", ) -> list[PropertyFormulation]: """ Extract security properties for a component. @@ -419,6 +427,8 @@ async def run_property_inference( threat_model, max_rounds=max_rounds, backend_guidance=backend_guidance, + system_template=system_template, + initial_template=initial_template, ) if refinement is None: to_ret = agent_attempt.items diff --git a/composer/spec/solana/__init__.py b/composer/spec/solana/__init__.py new file mode 100644 index 00000000..63f53c3d --- /dev/null +++ b/composer/spec/solana/__init__.py @@ -0,0 +1,9 @@ +"""Solana ecosystem: system model + (in the ecosystem module) prompts and wiring. + +The Solana chain of the ecosystem abstraction (see docs/ecosystem-abstraction.md). This +package holds the Solana-native system model the shared analysis phase produces +(``SolanaApplication``) and the index-wrapper instances the driver iterates +(``SolanaProgramInstance`` / ``SolanaInstructionInstance``, the latter satisfying the +``FeatureUnit`` protocol). The ecosystem object that binds these + the Rust language facet + +the Solana prompts lives in ``composer/pipeline/ecosystem.py``. +""" diff --git a/composer/spec/solana/build.py b/composer/spec/solana/build.py new file mode 100644 index 00000000..b9b219bf --- /dev/null +++ b/composer/spec/solana/build.py @@ -0,0 +1,133 @@ +"""Build a Solana program to sBPF (and, optionally, its IDL). + +The shared "Solana build capability" (``docs/crucible-application.md`` §5.1): +``source → .so [+ IDL]``. It is deliberately backend-agnostic — the Crucible +backend calls it in *no-munge* mode (build the program as-is), and a future +Certora-Prover/CVLR backend will call it in *munge-and-rebuild* mode (rewrite the +source first). Both route through the same :func:`run_local_command` choke point +the ``RunCommand`` effect uses, so phase-6 sandboxing (§7.4) wraps one path. +""" + +import logging +from dataclasses import dataclass +from pathlib import Path + +from composer.sandbox.command import CommandResult, run_local_command +from composer.sandbox.config import SandboxConfig + +_log = logging.getLogger(__name__) + +DEFAULT_BUILD_TIMEOUT_S = 600 + + +class BuildError(RuntimeError): + """A build step (``cargo-build-sbf`` / ``anchor idl``) failed.""" + + def __init__(self, step: str, result: CommandResult): + self.step = step + self.result = result + # Keep the tail of stderr — the actionable part of a cargo error. + super().__init__(f"{step} failed (exit {result.exit_code}):\n{result.stderr[-2000:]}") + + +@dataclass(frozen=True) +class BuiltProgram: + """Where the build left its outputs.""" + + program: str + so_path: Path + idl_path: Path | None + + +async def warm_cargo_cache( + manifest_dir: str | Path, + *, + cargo_binary: str = "cargo", + cargo_home: str | Path | None = None, + timeout_s: int = DEFAULT_BUILD_TIMEOUT_S, +) -> CommandResult: + """Populate ``CARGO_HOME`` with the deps declared in ``manifest_dir/Cargo.toml``. + + Run **outside** any sandbox (with network) so a later *sandboxed*, + ``CARGO_NET_OFFLINE`` build finds every dep warm (``docs/command-sandbox.md`` §5). + ``cargo fetch`` downloads but never runs build scripts, so no untrusted code + executes here — the code-exec build happens confined + offline. Best-effort: a + fetch failure is logged (the offline build will surface a hard error if a dep is + genuinely missing), not raised. + + ``cargo_home`` fetches into a specific (per-run, private) ``CARGO_HOME`` — it must + be the *same* home the sandboxed build will use, or the offline build won't find + the deps. Defaults to the ambient ``CARGO_HOME`` when omitted. + """ + overlay = {"CARGO_HOME": str(cargo_home)} if cargo_home is not None else None + res = await run_local_command( + cargo_binary, ["fetch"], {}, workdir=Path(manifest_dir), + timeout_s=timeout_s, env_overlay=overlay, + ) + if res.exit_code != 0: + _log.warning( + "cargo fetch in %s failed (exit %s); a sandboxed offline build may fail. stderr:\n%s", + manifest_dir, + res.exit_code, + res.stderr[-500:], + ) + return res + + +async def build_program( + project_root: str | Path, + program: str, + *, + build_binary: str = "cargo-build-sbf", + anchor_binary: str = "anchor", + with_idl: bool = False, + timeout_s: int = DEFAULT_BUILD_TIMEOUT_S, + sandbox: SandboxConfig | None = None, +) -> BuiltProgram: + """Compile ``program`` in the workspace at ``project_root`` to + ``target/deploy/.so``. If ``with_idl``, also try ``anchor idl build`` + (best-effort — not every project has an ``Anchor.toml``; the same-version + harness path depends on the program crate directly and needs no IDL). + + ``cargo-build-sbf`` runs the user program's ``build.rs`` natively, so it is + confined by ``sandbox`` when one is supplied (``docs/command-sandbox.md``); + ``None`` runs it unsandboxed (trusted input only). + + Raises :class:`BuildError` if the ``.so`` is not produced. + """ + root = Path(project_root) + provider = policy = None + if sandbox is not None and sandbox.enabled: + provider = sandbox.resolve_provider() + policy = sandbox.build_policy(root) + # Warm the registry with network BEFORE the sandboxed, offline build (§5), + # into the SAME private CARGO_HOME the sandboxed build will read (the policy's). + await warm_cargo_cache(root, cargo_home=policy.env_allowlist.get("CARGO_HOME"), timeout_s=timeout_s) + + res = await run_local_command( + build_binary, [], {}, workdir=root, timeout_s=timeout_s, provider=provider, policy=policy + ) + so = root / "target" / "deploy" / f"{program}.so" + if res.exit_code != 0 or not so.is_file(): + raise BuildError("cargo-build-sbf", res) + + idl_path: Path | None = None + if with_idl: + out_rel = f"target/idl/{program}.json" + idl_res = await run_local_command( + anchor_binary, ["idl", "build", "-o", out_rel], {}, workdir=root, + timeout_s=timeout_s, provider=provider, policy=policy, + ) + candidate = root / out_rel + if idl_res.exit_code == 0 and candidate.is_file(): + idl_path = candidate + else: + _log.warning( + "IDL build did not produce %s (exit %s); continuing without an IDL " + "(same-version harnesses depend on the program crate directly). stderr tail:\n%s", + candidate, + idl_res.exit_code, + idl_res.stderr[-500:], + ) + + return BuiltProgram(program=program, so_path=so, idl_path=idl_path) diff --git a/composer/spec/solana/model.py b/composer/spec/solana/model.py new file mode 100644 index 00000000..7e2a8d1b --- /dev/null +++ b/composer/spec/solana/model.py @@ -0,0 +1,274 @@ +"""The Solana system model — the standalone analog of the EVM ``SourceApplication``. + +Where the EVM model is contracts → components with storage variables and external functions, +Solana is **programs → instructions** that operate on **accounts passed in by the caller** +(there is no per-contract owned storage; state lives in accounts the instruction validates +and mutates). The model captures that shape natively — accounts + their signer/owner/PDA +constraints, cross-program invocations (CPIs), and the authorities involved — rather than +reusing the EVM field names. + +``SolanaApplication`` is what the shared analysis phase produces (it is a ``BaseApplication`` +so ``run_component_analysis`` accepts it). ``SolanaProgramInstance`` / ``SolanaInstructionInstance`` +are the driver's ``Main`` / ``Unit``: thin index wrappers over the model, with the instruction +instance satisfying the ecosystem-agnostic ``FeatureUnit`` protocol so the shared driver's +cache keys / task ids / labels work unchanged. +""" + +from dataclasses import dataclass +from functools import cached_property +from typing import Literal + +from pydantic import BaseModel, Field + +from composer.spec.system_model import BaseApplication +from composer.spec.types import PropertyFormulation +from composer.spec.util import slugify_filename + +#: How an account is expected to be supplied to an instruction. Drives the "missing signer / +#: owner check" and "account substitution" reasoning in the property prompt. +AccountRole = Literal["signer", "writable", "readonly", "pda", "program", "sysvar"] + + +class AccountConstraint(BaseModel): + """One account an instruction expects in its accounts context, plus the constraints the + program is responsible for enforcing on it.""" + + name: str = Field(description="The account's name in the instruction's accounts struct/context.") + account_type: str = Field( + description="The account's declared type (e.g. 'Signer', 'Account', 'Program', " + "'SystemAccount', 'UncheckedAccount', a PDA of some seeds)." + ) + roles: list[AccountRole] = Field( + default_factory=list, + description="Roles this account plays: signer / writable / readonly / pda / program / sysvar.", + ) + constraints: list[str] = Field( + default_factory=list, + description="Validations the program must enforce on this account — e.g. Anchor " + "constraints (has_one, seeds+bump, address, owner), an explicit owner/signer check, or " + "a documented invariant. Empty means the program performs no checks (often a finding).", + ) + + +class CpiCall(BaseModel): + """A cross-program invocation the instruction makes.""" + + target_program: str = Field(description="The program invoked (name or program id).") + description: str = Field(description="What the CPI does and any authority/PDA-signer it uses.") + + +class SolanaInstruction(BaseModel): + """A single instruction (entry point) of a program.""" + + name: str = Field(description="The instruction's snake_case name (its handler function).") + description: str = Field(description="What the instruction does, at the behavioral level (not how).") + accounts: list[AccountConstraint] = Field( + default_factory=list, description="The accounts the instruction takes and their constraints." + ) + signers: list[str] = Field( + default_factory=list, + description="Which accounts must sign (authorities/owners the instruction authenticates).", + ) + cpis: list[CpiCall] = Field( + default_factory=list, description="Cross-program invocations this instruction performs." + ) + args: list[str] = Field( + default_factory=list, description="The instruction's non-account arguments (name & type)." + ) + requirements: list[str] = Field( + description="Natural-language behavioral requirements — the instruction's specification." + ) + + +class SolanaProgram(BaseModel): + """A concrete on-chain program in the system.""" + + name: str = Field( + description="A short conceptual name for the program, used to refer to it across the system." + ) + program_identifier: str = Field( + pattern=r"^[a-zA-Z_][a-zA-Z0-9_]*$", + description="The program's Rust crate/module identifier as it appears in source. A valid " + "Rust identifier (snake_case).", + ) + program_id: str | None = Field( + default=None, description="The on-chain program id (base58), if declared (e.g. declare_id!)." + ) + description: str = Field(description="The program's role in the system.") + instructions: list[SolanaInstruction] = Field(description="The program's instructions.") + account_types: list[str] = Field( + default_factory=list, + description="The account/state types this program owns and derives (PDAs), name & purpose.", + ) + + +class SolanaAuthority(BaseModel): + """An external actor: a signer/authority, an off-chain keypair, or another program the + system interacts with but does not itself implement.""" + + name: str = Field(description="A short unique identifier for this authority/actor.") + description: str = Field(description="A short technical description.") + assumptions: list[str] = Field( + default_factory=list, description="Assumptions about this actor's behavior/trust." + ) + + +type SolanaComponent = SolanaProgram | SolanaAuthority + + +class SolanaApplication(BaseApplication[SolanaComponent]): + """A Solana application: a set of programs (+ external authorities).""" + + @cached_property + def programs(self) -> list[SolanaProgram]: + return [c for c in self.components if isinstance(c, SolanaProgram)] + + @cached_property + def authorities(self) -> list[SolanaAuthority]: + return [c for c in self.components if isinstance(c, SolanaAuthority)] + + +# --------------------------------------------------------------------------- +# Index wrappers — the driver's Main (program) and Unit (instruction). +# --------------------------------------------------------------------------- + + +@dataclass +class SolanaProgramInstance: + """The located target program — the ecosystem's ``Main``. + + Also serves as the **whole-program extraction unit** (satisfies + ``composer.spec.system_model.FeatureUnit``): under the global extraction strategy + (docs/crucible-unit-granularity.md) it is the context the property phase reads to + propose whole-program invariants, before those invariants fan out into + :class:`SolanaInvariantUnit`\\ s.""" + + ind: int + app: SolanaApplication + + @property + def program(self) -> SolanaProgram: + return self.app.programs[self.ind] + + # -- FeatureUnit protocol (whole-program extraction context) ------------------------ + @property + def display_name(self) -> str: + return self.program.name + + @property + def slug(self) -> str: + return slugify_filename(self.program.name) + + @property + def unit_index(self) -> int: + return self.ind + + def cache_material(self) -> str: + return "|".join([self.app.model_dump_json(), str(self.ind), "program"]) + + def context_tag(self) -> dict: + return {"program": self.program.model_dump()} + + def feature_json(self) -> dict: + return { + "program": self.program.name, + "instructions": [i.model_dump(mode="json") for i in self.program.instructions], + } + + +@dataclass +class SolanaInvariantUnit: + """One whole-program invariant — a ``Unit`` for the global extraction strategy. + + Produced by fanning the invariants out of a single whole-program extraction, so each + invariant gets its own harness fn + fuzz run + report row (satisfies + ``composer.spec.system_model.FeatureUnit``). The invariant travels in the formalize + batch's ``props``; ``feature_json`` carries the whole-program API so the test author + can drive any ``action_*`` in the sequence it asserts over.""" + + ind: int + _program: SolanaProgramInstance + invariant: PropertyFormulation + + @property + def app(self) -> SolanaApplication: + return self._program.app + + @property + def program(self) -> SolanaProgram: + return self._program.program + + # -- FeatureUnit protocol ----------------------------------------------------------- + @property + def display_name(self) -> str: + return self.invariant.title + + @property + def slug(self) -> str: + return slugify_filename(self.invariant.title) + + @property + def unit_index(self) -> int: + return self.ind + + def cache_material(self) -> str: + return "|".join( + [self.app.model_dump_json(), str(self._program.ind), str(self.ind), self.invariant.title] + ) + + def context_tag(self) -> dict: + return {"invariant": self.invariant.model_dump(mode="json"), "program": self.program.name} + + def feature_json(self) -> dict: + return { + "program": self.program.name, + "instructions": [i.model_dump(mode="json") for i in self.program.instructions], + } + + +@dataclass +class SolanaInstructionInstance: + """One instruction of the target program — the ecosystem's ``Unit`` (satisfies + ``composer.spec.system_model.FeatureUnit``).""" + + ind: int + _program: SolanaProgramInstance + + @property + def app(self) -> SolanaApplication: + return self._program.app + + @property + def program(self) -> SolanaProgram: + return self._program.program + + @property + def instruction(self) -> SolanaInstruction: + return self.program.instructions[self.ind] + + # -- FeatureUnit protocol ----------------------------------------------------------- + @property + def display_name(self) -> str: + return self.instruction.name + + @property + def slug(self) -> str: + return slugify_filename(self.instruction.name) + + @property + def unit_index(self) -> int: + return self.ind + + def cache_material(self) -> str: + return "|".join([self.app.model_dump_json(), str(self.ind), str(self._program.ind)]) + + def context_tag(self) -> dict: + return {"instruction": self.instruction.model_dump()} + + def feature_json(self) -> dict: + # The unit's semantic content for a backend marshalling it across a boundary: + # the instruction, tagged with its program (a Solana backend needs both). + return { + "program": self.program.name, + "instruction": self.instruction.model_dump(mode="json"), + } diff --git a/composer/spec/solana/null_backend.py b/composer/spec/solana/null_backend.py new file mode 100644 index 00000000..8b1df96e --- /dev/null +++ b/composer/spec/solana/null_backend.py @@ -0,0 +1,173 @@ +"""A null Solana backend — records extracted properties without verifying them. + +It satisfies the full ``PipelineBackend`` contract over the Solana ecosystem's +``(SolanaApplication, SolanaProgramInstance, SolanaInstructionInstance)`` triple, but its +``formalize`` just echoes the extracted properties into a trivial result and its +``fetch_verdicts`` returns nothing. + +**Role:** a **test double** for the Solana front half (analysis + property extraction) +without a real verifier — see ``tests/test_solana_gate.py``. Production Solana +verification is :mod:`composer.crucible` (Crucible fuzzer backend). +""" + +import enum +import json +from dataclasses import dataclass +from pathlib import Path +from typing import override + +from pydantic import BaseModel, Field + +from composer.pipeline.core import ( + CorePhases, + Formalizer, + GaveUp, + PipelineRun, + PreparedSystem, + SystemAnalysisSpec, +) +from composer.spec.artifacts import ArtifactStore +from composer.spec.context import WorkflowContext +from composer.spec.cvl_generation import SkippedProperty +from composer.spec.solana.model import ( + SolanaApplication, + SolanaInstructionInstance, + SolanaProgramInstance, +) +from composer.spec.source.report.collect import ReportComponentInput, Verdict +from composer.spec.source.report.schema import RuleName +from composer.spec.system_model import FeatureUnit +from composer.spec.types import PropertyFormulation +from composer.spec.util import ensure_dir + +SOLANA_NULL_GUIDANCE: str = """\ +These properties are recorded by a null backend (no verification is performed). Extract +properties a Solana verification tool could plausibly check: account/state invariants, access +control (signer/owner/authority), PDA-derivation correctness, and arithmetic safety. Freely +state universally-quantified properties. +""" + + +class SolanaPhase(enum.Enum): + ANALYSIS = "analysis" + EXTRACTION = "extraction" + FORMALIZATION = "formalization" + REPORT = "report" + + +class NullResult(BaseModel): + """A trivial formalization result: it just carries the properties back out.""" + + commentary: str = "" + property_rules: list[tuple[str, list[str]]] = Field(default_factory=list) + skipped: list[SkippedProperty] = Field(default_factory=list) + + def property_units(self) -> list[tuple[str, list[str]]]: + return [(t, list(u)) for t, u in self.property_rules] + + @property + def artifact_text(self) -> str: + return json.dumps( + {"commentary": self.commentary, "properties": self.property_units()}, indent=2 + ) + + @property + def output_link(self) -> str | None: + return None + + +@dataclass(frozen=True) +class NullArtifact: + slug: str + + @property + def stem(self) -> str: + return f"null_{self.slug}" + + @property + def artifact_file(self) -> str: + return f"{self.stem}.json" + + +class NullSolanaArtifactStore(ArtifactStore[NullArtifact, NullResult]): + def __init__(self, project_root: str): + super().__init__( + project_root, + "property_units", + deliverable_dir="certora/solana_null", + internal_dir=".certora_internal/solana_null", + report_dir="certora/solana_null/reports", + ) + + @override + def _artifact_dir(self) -> Path: + return ensure_dir(Path(self._project_root) / "certora/solana_null/artifacts") + + +class NullSolanaFormalizer(Formalizer[NullResult, FeatureUnit]): + def __init__(self) -> None: + # Reuses the ``"crucible"`` report backend (the real Solana verifier this null backend + # models); its results are all-UNKNOWN, so the label choice is provenance only. + super().__init__(NullResult, "crucible") + + @override + async def formalize( + self, + label: str, + feat: FeatureUnit, + props: list[PropertyFormulation], + ctx: WorkflowContext[NullResult], + run: PipelineRun, + ) -> NullResult | GaveUp: + return NullResult( + commentary=f"Null formalization of instruction {feat.display_name} " + f"({len(props)} properties recorded, unverified).", + property_rules=[(p.title, [p.title]) for p in props], + ) + + @override + async def fetch_verdicts( + self, inp: ReportComponentInput[NullResult] + ) -> dict[RuleName, Verdict]: + return {} + + +@dataclass +class NullSolanaPrepared(PreparedSystem[NullResult, FeatureUnit]): + form: NullSolanaFormalizer + + @override + async def prepare_formalization( + self, run: PipelineRun + ) -> Formalizer[NullResult, FeatureUnit]: + return self.form + + +@dataclass +class NullSolanaBackend: + """``PipelineBackend[SolanaPhase, NullResult, None, NullArtifact, SolanaApplication, + SolanaProgramInstance, SolanaInstructionInstance]`` — structural.""" + + artifact_store: NullSolanaArtifactStore + backend_guidance = SOLANA_NULL_GUIDANCE + analysis_spec = SystemAnalysisSpec("solana-analysis", "solana-properties") + core_phases = CorePhases( + { + "analysis": SolanaPhase.ANALYSIS, + "extraction": SolanaPhase.EXTRACTION, + "formalization": SolanaPhase.FORMALIZATION, + "report": SolanaPhase.REPORT, + } + ) + + async def prepare_system( + self, analyzed: SolanaApplication, run: PipelineRun[SolanaPhase, None] + ) -> PreparedSystem[NullResult, FeatureUnit]: + # Use the Solana ecosystem's locate_main so the backend and ecosystem agree on the + # target program (imported lazily to avoid an import cycle with pipeline.ecosystem). + from composer.pipeline.ecosystem import SOLANA + + return NullSolanaPrepared(SOLANA.locate_main(analyzed, run.source), NullSolanaFormalizer()) + + def to_artifact_id(self, c: FeatureUnit) -> NullArtifact: + return NullArtifact(c.slug) diff --git a/composer/spec/source/pipeline.py b/composer/spec/source/pipeline.py index c6ac5120..aea1603d 100644 --- a/composer/spec/source/pipeline.py +++ b/composer/spec/source/pipeline.py @@ -98,7 +98,7 @@ def _lift_harnessed( @dataclass -class ProverRunner(Formalizer[GeneratedCVL]): +class ProverRunner(Formalizer[GeneratedCVL, ContractComponentInstance]): """Immutable formalizer: per-batch CVL generation against a fixed prover config + resource set (already including ``invariants.spec`` when there are structural invariants), plus the in-memory invariant result for the report.""" @@ -149,7 +149,7 @@ async def fetch_verdicts( return await self._fetch(inp) @override - async def finalize(self, outcomes: list[ComponentOutcome[GeneratedCVL]], run: PipelineRun) -> None: + async def finalize(self, outcomes: list[ComponentOutcome[GeneratedCVL, ContractComponentInstance]], run: PipelineRun) -> None: # components_to_prover_runs.json: {run_key (slug): prover /output/ link}. runs: dict[str, str] = { ComponentSpec(o.feat.slugified_name).run_key: o.result.run_link @@ -164,7 +164,7 @@ async def finalize(self, outcomes: list[ComponentOutcome[GeneratedCVL]], run: Pi @dataclass -class ProverPrepared(PreparedSystem[GeneratedCVL]): +class ProverPrepared(PreparedSystem[GeneratedCVL, ContractComponentInstance]): """Post-harness system: holds the harnessed app + prover tool, and runs the prover-only pre-formalization fan-out in ``prepare_formalization``.""" _store: ProverArtifactStore @@ -175,7 +175,7 @@ class ProverPrepared(PreparedSystem[GeneratedCVL]): _analyzed: SourceApplication @override - async def prepare_formalization(self, run: PipelineRun) -> Formalizer[GeneratedCVL]: + async def prepare_formalization(self, run: PipelineRun) -> Formalizer[GeneratedCVL, ContractComponentInstance]: # AutoSetup (+ custom summaries) ∥ structural-invariant formulation; both # depend only on the harnessed app, so they run concurrently. (setup_config, resources), invariants = await asyncio.gather( @@ -287,7 +287,7 @@ class ProverBackend: async def prepare_system( self, analyzed: SourceApplication, run: PipelineRun[AutoProvePhase, None], - ) -> PreparedSystem[GeneratedCVL]: + ) -> PreparedSystem[GeneratedCVL, ContractComponentInstance]: sys_desc = await run.runner( TaskInfo(HARNESS_TASK_ID, "Harness Creation", AutoProvePhase.HARNESS), lambda: run_harness_creation(run.ctx, run.source, run.env, analyzed), diff --git a/composer/spec/source/report/collect.py b/composer/spec/source/report/collect.py index ddff5b8e..08b5064b 100644 --- a/composer/spec/source/report/collect.py +++ b/composer/spec/source/report/collect.py @@ -69,10 +69,14 @@ class Verdict: line: int | None = None duration_seconds: float | None = None unit_file: str | None = None + #: Human-readable explanation of a non-GOOD outcome (a counterexample / assertion message for + #: a BAD, error text for an ERROR). Provenance/diagnostics only; ``None`` when the backend + #: gives no detail (the prover/foundry fetchers don't). + message: str | None = None def merge(self, other: "Verdict | None") -> "Verdict": """Combine two results for one unit within a run: higher-priority outcome wins, - line/duration/unit_file kept from whichever side has them.""" + line/duration/unit_file/message kept from whichever side has them.""" if other is None: return self hi, lo = ( @@ -85,6 +89,7 @@ def merge(self, other: "Verdict | None") -> "Verdict": hi.line if hi.line is not None else lo.line, hi.duration_seconds if hi.duration_seconds is not None else lo.duration_seconds, hi.unit_file or lo.unit_file, + hi.message or lo.message, ) @@ -162,7 +167,7 @@ def _ref(unit_name: str) -> RuleRef: if key not in rules_by_key: rules_by_key[key] = RuleVerdict( name=unit_name, spec_file=key[0], outcome=v.outcome, line=v.line, - duration_seconds=v.duration_seconds, prover_link=run_link, + duration_seconds=v.duration_seconds, prover_link=run_link, message=v.message, ) # A referenced unit with no verdict still needs an (UNKNOWN) entry to render. diff --git a/composer/spec/source/report/render.py b/composer/spec/source/report/render.py index 25986c51..e7f1ab59 100644 --- a/composer/spec/source/report/render.py +++ b/composer/spec/source/report/render.py @@ -57,6 +57,10 @@ Outcome.GOOD: "Successful test", Outcome.BAD: "Failing test", Outcome.ERROR: "Error", Outcome.TIMEOUT: "Timeout", Outcome.UNKNOWN: "Unknown", }, + "crucible": { + Outcome.GOOD: "No counterexample", Outcome.BAD: "Counterexample", Outcome.ERROR: "Error", + Outcome.TIMEOUT: "Timeout", Outcome.UNKNOWN: "Unknown", + }, } _GROUP_LABELS: dict[ReportBackend, dict[GroupStatus, str]] = { "prover": { @@ -67,6 +71,10 @@ GroupStatus.GOOD: "All tests passing", GroupStatus.BAD: "Has failing test", GroupStatus.PARTIAL: "Partial", GroupStatus.UNKNOWN: "No results", }, + "crucible": { + GroupStatus.GOOD: "No counterexamples", GroupStatus.BAD: "Has counterexample", + GroupStatus.PARTIAL: "Partial", GroupStatus.UNKNOWN: "No results", + }, } @@ -89,6 +97,10 @@ class ReportTerms(TypedDict): title="Foundry test report", unit_singular="test", unit_plural="tests", unit_cap="Test", outcomes_label="Test outcomes", ), + "crucible": ReportTerms( + title="Crucible fuzzing report", unit_singular="property", unit_plural="properties", + unit_cap="Property", outcomes_label="Property outcomes", + ), } # Chip display order for the header outcome counts. @@ -123,6 +135,9 @@ class RowView(TypedDict): line: int | None link: LinkView descriptions: list[str] + #: Backend diagnostic for a non-GOOD row (e.g. the fuzzer's counterexample / failed-assertion + #: message). ``None`` when the backend supplied none. + message: str | None class GroupView(TypedDict): @@ -151,6 +166,16 @@ class ReportTemplateParams(TypedDict): _REPORT_TEMPLATE = TypedTemplate[ReportTemplateParams]("autoprove_report.html.j2") +def outcome_label(backend: ReportBackend, outcome: Outcome) -> str: + """The human word an auditor reads for an ``Outcome`` under a backend (e.g. a + ``crucible`` ``GOOD`` → "No counterexample", a ``prover`` ``GOOD`` → "Verified"). + + The report's HTML render is the primary consumer, but the console/TUI verdict + rollups reuse this so the same run reads with one vocabulary everywhere — this is + the single place the per-backend wording lives.""" + return _OUTCOME_LABELS[backend][outcome] + + def _is_url(link: str) -> bool: return link.startswith("http://") or link.startswith("https://") @@ -216,6 +241,7 @@ def _group_view( "line": rule.line if rule else None, "link": _link_view(rule.prover_link if rule else None), "descriptions": descriptions[ref], + "message": rule.message if rule else None, }) return { "slug": group.slug, diff --git a/composer/spec/source/report/schema.py b/composer/spec/source/report/schema.py index a791ac7e..e269678c 100644 --- a/composer/spec/source/report/schema.py +++ b/composer/spec/source/report/schema.py @@ -84,6 +84,11 @@ class RuleVerdict(BaseModel): line: int | None = None duration_seconds: float | None = None prover_link: str | None = None + message: str | None = Field( + default=None, + description="Human-readable explanation of a non-GOOD outcome (e.g. the fuzzer's " + "counterexample / failed-assertion message). Diagnostics only; may be absent.", + ) @property def ref(self) -> RuleRef: @@ -151,10 +156,13 @@ class CoverageReport(BaseModel): warnings: list[str] = Field(default_factory=list) -type ReportBackend = Literal["prover", "foundry"] +type ReportBackend = Literal["prover", "foundry", "crucible"] """Which pipeline produced this report. Provenance only — every backend fills the same fields; -this tag just lets the renderer pick the right outcome labels ("Verified" vs "Successful test") -for a report.json it reads cold.""" +this tag just lets the renderer pick the right outcome labels ("Verified" vs "Successful test" +vs "No counterexample") for a report.json it reads cold. The backends are the CVL prover +(``"prover"``), Foundry (``"foundry"``), and the Rust/Crucible fuzzer (``"crucible"``, produced by +``composer.rustapp``/``composer.crucible``). The set is closed: every backend lives in this repo, +so a new one adds its literal here (and its labels in ``report/render.py``).""" class AutoProverReport(BaseModel): diff --git a/composer/spec/system_analysis.py b/composer/spec/system_analysis.py index 7666c90a..e2105394 100644 --- a/composer/spec/system_analysis.py +++ b/composer/spec/system_analysis.py @@ -1,4 +1,4 @@ -from typing import NotRequired, Any +from typing import NotRequired, Any, Callable from graphcore.graph import MessagesState, FlowInput @@ -95,8 +95,16 @@ async def run_component_analysis[T: BaseApplication]( env: ServiceHost, extra_input: list[str | dict], expected_main_id: SolidityIdentifier | None = None, + *, + system_template: str = "application_analysis_system.j2", + initial_template: str = "application_analysis_prompt.j2", + validate: Callable[[BaseApplication, SolidityIdentifier | None], str | None] = _validate_connectivity, ) -> T | None: - """Analyze application components from a system doc and optionally source code.""" + """Analyze application components from a system doc and optionally source code. + + The prompt templates and connectivity ``validate`` are parameters so the ecosystem + (``composer.pipeline.ecosystem``) can supply domain-specific ones; the defaults are the + EVM/Solidity values, so existing callers are unaffected.""" if (cached := await child_ctxt.cache_get(ty)) is not None: return cached @@ -112,7 +120,7 @@ class AnalysisInput(RoughDraftState, FlowInput): def _validation_wrapper( _: Any, app: BaseApplication ) -> str | None: - return _validate_connectivity(app, expected_main_id) + return validate(app, expected_main_id) b = bind_standard( builder=env.builder_lite(), @@ -121,12 +129,12 @@ def _validation_wrapper( ).with_input( AnalysisInput ).with_sys_prompt_template( - "application_analysis_system.j2", + system_template, sort=env.sort, ).with_tools( [memory, *get_rough_draft_tools(AnalysisState), *env.analysis_tools] ).with_initial_prompt_template( - "application_analysis_prompt.j2", + initial_template, sort=env.sort, ) diff --git a/composer/spec/system_model.py b/composer/spec/system_model.py index 7bdb4059..68042b9c 100644 --- a/composer/spec/system_model.py +++ b/composer/spec/system_model.py @@ -1,10 +1,49 @@ from dataclasses import dataclass -from typing import Literal +from typing import Literal, Protocol, runtime_checkable from pydantic import BaseModel, Field from functools import cached_property from composer.spec.util import slugify_filename from .types import ComponentName, SolidityIdentifier, ContractName + +@runtime_checkable +class FeatureUnit(Protocol): + """The per-unit interface the shared pipeline driver needs, independent of ecosystem. + + Each ecosystem's unit wrapper (EVM's :class:`ContractComponentInstance`, Solana's + instruction instance, …) satisfies it; ecosystem-specific code — prompts, backends, + formalizers — works with the concrete unit type. This keeps the driver's per-unit cache + keys, task ids, labels, and context tags ecosystem-agnostic.""" + + @property + def display_name(self) -> str: + """Human label for tasks / report rows.""" + ... + + @property + def slug(self) -> str: + """Filesystem-safe slug used as an artifact-id base.""" + ... + + @property + def unit_index(self) -> int: + """Stable per-run index used in task ids.""" + ... + + def cache_material(self) -> str: + """Stable string identifying this unit, hashed into its per-unit cache key.""" + ... + + def context_tag(self) -> dict: + """The tag persisted alongside this unit's workflow context.""" + ... + + def feature_json(self) -> dict: + """The unit's semantic content as a JSON-able dict, for a backend that + marshals it across a boundary (e.g. a Rust wheel's ``FormalizeInput.component``). + The shape is ecosystem-specific; the paired backend knows how to read it.""" + ... + type ContractSort = Literal["dynamic", "singleton", "multiple"] @@ -92,7 +131,10 @@ class SourceExternalActor(ExternalActor): type SystemComponent = ExternalActor | ExplicitContract -class BaseApplication[T : SystemComponent](BaseModel): +# Bound is ``BaseModel`` (not ``SystemComponent``) so non-EVM ecosystems can parameterize it +# with their own component unions (e.g. Solana's programs/authorities); the EVM subclasses below +# still pin the concrete ``SystemComponent`` union. +class BaseApplication[T : BaseModel](BaseModel): application_type: str = Field(description="A concise, description of the type of application (AMM/Liquidity Provider/etc.)") description: str = Field(description="A description of the application's main functionality (2 - 3 sentences max)") components : list[T] = Field(description="The system components (explicit contract & external actors) that comprise this application") @@ -234,6 +276,28 @@ def slugified_name(self) -> str: ``_validate_connectivity``), so no disambiguation is needed here.""" return slugify_filename(self.component.name) + # -- FeatureUnit protocol (the ecosystem-agnostic view the driver consumes) --------- + @property + def display_name(self) -> str: + return self.component.name + + @property + def slug(self) -> str: + return self.slugified_name + + @property + def unit_index(self) -> int: + return self.ind + + def cache_material(self) -> str: + return "|".join([self.app.model_dump_json(), str(self.ind), str(self._contract.ind)]) + + def context_tag(self) -> dict: + return {"component": self.component.model_dump()} + + def feature_json(self) -> dict: + return self.component.model_dump(mode="json") + @staticmethod def from_app( app: AnyApplication, diff --git a/composer/templates/autoprove_report.html.j2 b/composer/templates/autoprove_report.html.j2 index 894c6c9c..d7fa97f9 100644 --- a/composer/templates/autoprove_report.html.j2 +++ b/composer/templates/autoprove_report.html.j2 @@ -177,6 +177,16 @@ section.gaps h4 { } ul.claims { margin: 4px 0 0 18px; padding: 0; font-size: 13px; } ul.claims li { margin-bottom: 4px; } +.finding { + margin-top: 6px; + padding: 4px 8px; + background: var(--bg-pre); + border-radius: 3px; + font-family: ui-monospace, SFMono-Regular, "SF Mono", Menlo, monospace; + font-size: 12px; + white-space: pre-wrap; + word-break: break-word; +} .sort-tag { display: inline-block; padding: 1px 7px; @@ -263,6 +273,7 @@ footer.report-foot {
    {% for d in row.descriptions %}
  • {{ d }}
    • {% endfor %}
{%- elif row.descriptions %}{{ row.descriptions[0] }} {%- else %}(no inferred description){%- endif %} + {%- if row.message %}
{{ row.message }}
{%- endif %} {%- if has_links %} diff --git a/composer/templates/rust/_failure_modes.j2 b/composer/templates/rust/_failure_modes.j2 new file mode 100644 index 00000000..99d71091 --- /dev/null +++ b/composer/templates/rust/_failure_modes.j2 @@ -0,0 +1,15 @@ +Rust language-level failure modes (shared by every Rust on-chain ecosystem — Solana, Soroban, …): + +- **Integer overflow / underflow.** On-chain Rust builds usually enable `overflow-checks`, so an + overflow *panics* (aborting the transaction) rather than wrapping — a denial-of-service / + griefing vector, not silent corruption. But code that uses `wrapping_*`, `unchecked_*`, `as` + casts (truncation), or arithmetic in a non-checked build can silently produce wrong values. + Look for arithmetic on balances, amounts, indices, and lengths. +- **Panics as aborts.** `unwrap()`, `expect()`, `panic!`, array indexing out of bounds, and + `unreachable!` all abort the transaction. An attacker-controllable input that forces one is a + DoS / stuck-funds vector (e.g. an operation that can be made to always panic). +- **Lossy conversions / precision.** `as` casts between integer widths truncate; integer + division rounds toward zero. Rounding direction that favors the caller over the protocol + (or vice versa) across two code paths is an arbitrage / value-leak vector. +- **Unchecked results.** An ignored `Result`/error (`let _ = ...`, `.ok()`, `?` swallowed) that + should have aborted lets execution continue in an inconsistent state. diff --git a/composer/templates/solana/_failure_modes.j2 b/composer/templates/solana/_failure_modes.j2 new file mode 100644 index 00000000..0da21b80 --- /dev/null +++ b/composer/templates/solana/_failure_modes.j2 @@ -0,0 +1,30 @@ +Solana platform failure modes (the account/PDA/CPI model — reason about these specifically): + +- **Missing signer check.** An instruction that mutates state or moves value on behalf of an + authority but never checks that authority `is_signer` (or omits the Anchor `Signer<'info>` / + `has_one` / `#[account(signer)]` constraint) lets anyone act as that authority. +- **Missing owner check.** An account deserialized as a program type without verifying its + `owner` is the expected program lets an attacker substitute a look-alike account they created + under a program they control (`UncheckedAccount` / `AccountInfo` without checks is the smell; + Anchor `Account<'info, T>` checks owner, raw does not). +- **Account substitution / confused deputy.** Two accounts that must be related (e.g. a vault and + its authority, a token account and its mint/owner) but whose relationship is not constrained + (`has_one`, `constraint = a.key() == b.x`, `address = ...`) let an attacker pass a mismatched + pair — draining a different user's vault, minting to the wrong mint, etc. +- **Unvalidated PDA seeds / bump.** A PDA account not re-derived and checked against its expected + `seeds` + canonical `bump` lets an attacker pass an arbitrary account in its place; a mutable + bump or non-canonical bump enables collisions/hijacking. +- **Arbitrary CPI / program substitution.** Invoking a program taken from an unchecked account + (not pinned to a known program id) lets an attacker redirect the CPI to malicious code; missing + checks on the callee for token/system operations. +- **Signer-seed / authority misuse in CPI.** A PDA that signs a CPI (`invoke_signed`) with seeds + an attacker can influence, or over-broad authority, lets value move without proper authorization. +- **Lamport / rent draining & account close bugs.** Manual lamport transfers that don't preserve + rent exemption, or `close` handling that doesn't zero data / send to the right account, enable + reinitialization attacks or fund theft. +- **Duplicate mutable accounts.** Passing the same account twice where the program assumes two + distinct accounts (e.g. `from`/`to`) can double-count or bypass a balance check. +- **Missing reinitialization / init guards.** An `init`-like path callable twice, or state that + can be re-initialized after close, resets ownership/config. +- **Sysvar / clock spoofing.** Reading a sysvar (clock, rent) from a passed account rather than the + canonical sysvar lets an attacker forge time/rent values. diff --git a/composer/templates/solana/analysis_prompt.j2 b/composer/templates/solana/analysis_prompt.j2 new file mode 100644 index 00000000..9dc5c16a --- /dev/null +++ b/composer/templates/solana/analysis_prompt.j2 @@ -0,0 +1,74 @@ +# Background + +{% if sort == "greenfield" %} +You have been provided a system/design document describing a Solana application. No +implementation exists yet, and the document may vary in technical rigor. +{% else %} +You have been provided a system/design document describing a Solana application, plus access to +the Rust source (native and/or Anchor) that implements it. +{% endif %} + +# Task + +Analyze this design document {% if sort != "greenfield" %}and implementation {% endif %}and extract a holistic, structured model of the +application. Your model describes the **programs** {% if sort == "greenfield" %}that need to be written{% else %}present in the implementation{% endif %}, the +**instructions** each program exposes, the **accounts** each instruction operates on (with the +checks the program must enforce), and any **external authorities/actors** involved. Produce +natural-language descriptions — do *NOT* write Rust or pseudo-code. + +## Application Description + +Provide an "Application Type" — a broad category (e.g. "AMM", "Staking", "NFT marketplace", +"Escrow") — and a short (2–3 sentence) description that refines it. Then a list of "System +Components": each is either a **Program** or an **External Authority**. + +### External Authorities + +Any actor not part of the core application the programs interact with: an admin/authority +keypair, an off-chain signer, or a third-party program (e.g. an oracle) the system does not +itself implement. The **SPL Token program, the System program, and other standard Solana +programs are external authorities** unless the application implements them. Give each a short +name, a description of its role, and the assumptions/trust placed in it. + +### Programs + +Each program the application implements. For each program provide: + +- `name`: a short conceptual label (e.g. "Vault", "Staking Pool"). +- `program_identifier`: the program's Rust crate/module identifier as it appears (or will + appear) in source — a valid Rust identifier (snake_case, matching `[a-zA-Z_][a-zA-Z0-9_]*`). +- `program_id`: the on-chain base58 program id if declared (e.g. via `declare_id!`), else null. +- `description`: the program's role in the application. +- `account_types`: the account/state types this program owns and derives (PDAs) — name & purpose. +- `instructions`: the program's instructions (entry points), described below. + +#### Instructions + +Each instruction (the program's entry points / handlers). For each: + +- `name`: the instruction's snake_case name. +- `description`: what it does behaviorally (not how). +- `args`: its non-account arguments (name & type). +- `accounts`: every account it takes in its accounts context. For each account give its `name`, + its `account_type` (e.g. `Signer`, `Account`, `Program`, `SystemAccount`, + `UncheckedAccount`, a PDA of specific seeds), its `roles` (signer / writable / readonly / pda / + program / sysvar), and the `constraints` the program is responsible for enforcing on it + (signer/owner checks, `has_one`, `seeds`+`bump`, `address`, or documented invariants). If an + account has no enforced constraints, record an empty list — that is itself important signal. +- `signers`: which accounts must sign (the authorities the instruction authenticates). +- `cpis`: cross-program invocations it performs (target program + what it does / signer used). +- `requirements`: the instruction's behavioral specification, stated as requirements ("The + instruction must ..."). + +## On Interactions + +Enumerate programs and authorities as comprehensively as the inputs allow. Halt transitive +exploration at external-authority boundaries — do not speculate about what those actors do +internally. When an instruction CPIs into another program the application implements, that +callee is a Program (a component), not an external authority. + +# Memory + +You have a memory tool that may include prior progress on this task. Assume the documentation / +source has NOT changed since the memories were written; you need not re-verify memorized facts +and may proceed with them as if you derived them yourself. diff --git a/composer/templates/solana/analysis_system.j2 b/composer/templates/solana/analysis_system.j2 new file mode 100644 index 00000000..b59bbf88 --- /dev/null +++ b/composer/templates/solana/analysis_system.j2 @@ -0,0 +1,28 @@ +You are an experienced system architect well-versed in Solana on-chain programs (native and +Anchor). You are analyzing a Solana application's design document {% if sort != "greenfield" %}and its Rust implementation {% endif %}to extract a +structured model of its behavior. This model guides the {% if sort != "greenfield" %}verification{% else %}implementation{% endif %} of the application. + +## Behavior + +- After any collection of data, use your adaptive thinking capabilities to consider all of your + information before proceeding. +- Ground any results you produce in verifiable information as much as possible. +- When resolving ambiguity during your work, favor self-consistency. + +## Tools + +- You have access to a "memory" tool, presented as a filesystem-like abstraction. Whenever you + synthesize new knowledge or reach a conclusion, record it (and your reasoning) with the tool. +- On a multi-step task, track progress in `/memories/progress.md` (current step + task data). +{% with draft_subject = "your response" %} +{% include "rough_draft_protocol.j2" %} +{% endwith %} +{% if sort != "greenfield" %} +{% with source_tools_has_explorer = sort == "existing" %} +{% include "source_tools_system_prompt.j2" %} +{% endwith %} +When reading Rust/Anchor source, pay attention to the instruction handlers, their +`#[derive(Accounts)]` account-validation structs (and the constraints on each account: +`signer`, `mut`, `has_one`, `seeds`/`bump`, `owner`, `address`), the account/state types the +program declares, and any cross-program invocations. +{% endif %} diff --git a/composer/templates/solana/instruction_context.j2 b/composer/templates/solana/instruction_context.j2 new file mode 100644 index 00000000..19e7fb36 --- /dev/null +++ b/composer/templates/solana/instruction_context.j2 @@ -0,0 +1,52 @@ +This task targets the security properties of the instruction `{{ context.instruction.name }}` +of the program `{{ context.program.name }}` (`{{ context.program.program_identifier }}`). + +This instruction is described as: + +{{ context.instruction.description }} + +The requirements for this instruction are: +{% for req in context.instruction.requirements %} +* {{ req }} +{% endfor %} +{% if context.instruction.args %} + +Non-account arguments: {% for a in context.instruction.args %}`{{ a }}`{% if not loop.last %}, {% endif %}{% endfor %}. +{% endif %} + +Accounts this instruction takes, and the checks the program is responsible for enforcing on each: +{% for acc in context.instruction.accounts %} +* `{{ acc.name }}` — {{ acc.account_type }}{% if acc.roles %} [{{ acc.roles | join(", ") }}]{% endif %}. + {% if acc.constraints %}Program-enforced constraints: {{ acc.constraints | join("; ") }}.{% else %}No constraints were recorded for this account — consider whether a signer/owner/relationship check is required here.{% endif %} +{% endfor %} +{% if context.instruction.signers %} + +Accounts that must sign: {{ context.instruction.signers | join(", ") }}. +{% endif %} +{% if context.instruction.cpis %} + +Cross-program invocations this instruction performs: +{% for cpi in context.instruction.cpis %} +* → `{{ cpi.target_program }}`: {{ cpi.description }} +{% endfor %} +{% endif %} + +The `{{ context.program.name }}` program is part of a `{{ context.app.application_type }}` application. +{% if context.program.account_types %} +It owns / derives these account (state) types: {{ context.program.account_types | join(", ") }}. +{% endif %} + +{% set siblings = [] %} +{% for p in context.app.programs %}{% if p.program_identifier != context.program.program_identifier %}{% set _ = siblings.append(p) %}{% endif %}{% endfor %} +{% if siblings %} +Other programs in the application: +{% for p in siblings %} +* `{{ p.name }}` (`{{ p.program_identifier }}`): {{ p.description }} +{% endfor %} +{% endif %} +{% if context.app.authorities %} +External authorities / actors the system interacts with: +{% for a in context.app.authorities %} +* {{ a.name }}: {{ a.description }} +{% endfor %} +{% endif %} diff --git a/composer/templates/solana/program_context.j2 b/composer/templates/solana/program_context.j2 new file mode 100644 index 00000000..03cb993d --- /dev/null +++ b/composer/templates/solana/program_context.j2 @@ -0,0 +1,21 @@ +This task targets whole-program security invariants of the program `{{ context.program.name }}` +(`{{ context.program.program_identifier }}`), part of a `{{ context.app.application_type }}` application. + +{% if context.program.account_types %} +The program owns / derives these account (state) types: {{ context.program.account_types | join(", ") }}. +{% endif %} + +The program exposes these instructions (the actions a fuzzer will drive in random sequences): +{% for ins in context.program.instructions %} +### `{{ ins.name }}` +{{ ins.description }} +{% if ins.requirements %}Requirements: +{% for req in ins.requirements %} - {{ req }} +{% endfor %}{% endif %} +{% if ins.args %}Non-account arguments: {% for a in ins.args %}`{{ a }}`{% if not loop.last %}, {% endif %}{% endfor %}. +{% endif %}Accounts (and the checks the program must enforce): +{% for acc in ins.accounts %} - `{{ acc.name }}` — {{ acc.account_type }}{% if acc.roles %} [{{ acc.roles | join(", ") }}]{% endif %}{% if acc.constraints %}; constraints: {{ acc.constraints | join("; ") }}{% endif %}. +{% endfor %}{% if ins.signers %}Must sign: {{ ins.signers | join(", ") }}. +{% endif %}{% if ins.cpis %}CPIs: {% for cpi in ins.cpis %}→ `{{ cpi.target_program }}` ({{ cpi.description }}){% if not loop.last %}; {% endif %}{% endfor %}. +{% endif %} +{% endfor %} diff --git a/composer/templates/solana/property_prompt.j2 b/composer/templates/solana/property_prompt.j2 new file mode 100644 index 00000000..3c6435f2 --- /dev/null +++ b/composer/templates/solana/property_prompt.j2 @@ -0,0 +1,114 @@ + +You are performing a security review of a Solana program as a whole, to produce **program-level +invariants** for a coverage-guided fuzzer. The fuzzer drives *random sequences* of the program's +instructions and checks each property after every action — so the properties you write must hold +across ANY reachable sequence of actions, not only within a single instruction. + + + +{% include "solana/program_context.j2" %} + + +{% if prior_properties %} + +You are running iteratively. {{ prior_properties | length }} prior round(s) have already analyzed +this program. Their findings and reasoning are below. The properties listed here MUST NOT be +re-proposed — including reframings of the same underlying claim as a different sort. Your task this +round is to surface security properties that prior rounds *missed*. + +{% for round_result in prior_properties %} +### Round {{ loop.index }} extracted properties: +{% for p in round_result.items %} +- [{{ p.sort }}] `{{ p.title }}`: {{ p.description }} +{% endfor %} + +### Round {{ loop.index }} reasoning: +{{ round_result.reasoning }} + +{% endfor %} + +Treat the prior rounds' reasoning as the authoritative record of what has been considered. If a +prior round is silent on an angle, that angle is fair game. If the meaningful surface is already +covered, returning an empty list is the right answer. + +{% endif %} + + +Input: {% if sort != "greenfield" %}the Rust/Anchor implementation of a Solana program{% else %}a design document for a Solana program{% endif %}, and a description of its instructions. +Output: a list of whole-program "security properties" a fuzzer can check after every action. + +Follow these steps exactly: + +## Step 1 + +Analyze the provided {% if sort != "greenfield" %}implementation{% else %}design{% endif %} and formulate a set of security +properties that are PLAUSIBLE for this program. Prefer **cross-instruction** properties — ones a +random sequence of actions could violate (conservation, monotonicity, access control that must hold +no matter what came before) — over ones local to a single call. Each description must be precise and +concise, and phrased as something checkable against on-chain state after any action. + +Security properties fall into three categories: + +1. **Invariants** — representational invariants that should always hold for the program's accounts/state, + across every reachable state. + Good: "The vault's token balance always equals the sum of all depositors' recorded shares." + Good: "A config PDA's `admin` field is only ever the key that signed the last `set_admin`." + Bad: "The accounts should be correct" (overly broad). + +2. **Safety properties** — concrete statements about what should not be possible in a correct program, + over any action sequence. + These SHOULD include intended-normal-behavior properties, not only immediately-security-critical ones. + Good: "Only the account stored in `vault.authority` (and it must sign) can ever reduce the vault balance." + Good: "`initialize` can succeed at most once for a given config PDA, no matter the call order." + Good: "No sequence of actions lets a user withdraw more than they have deposited." + Bad: "A user should not be able to hack the program" (overly broad). + +3. **Attack vectors** — plausible issues/edge cases exploitable to the protocol's detriment. You do + NOT need evidence they exist, only that they are PLAUSIBLE given the {% if sort != "greenfield" %}implementation{% else %}design{% endif %}. + Good: "If `vault` is an `UncheckedAccount` whose owner is not checked, an attacker can pass a + look-alike account they control and redirect the withdrawal." + Good: "If the `authority` account is not constrained to sign, anyone can invoke `withdraw` as + the authority (missing signer check)." + Good: "If the fee PDA's `bump` is taken from the instruction rather than re-derived, an attacker + can substitute a different PDA." + Bad: "The program could be exploited somehow" (overly broad). + +When reasoning about attack vectors and the checks each account requires, consider these failure modes: + +{% include "rust/_failure_modes.j2" %} + +{% include "solana/_failure_modes.j2" %} + +{{ backend_guidance }} + +NB: you do **NOT** need to formalize the properties yourself; limit your analysis to properties that +can reasonably be formalized by the downstream verification tool. + +## Step 2 +Write a rough-draft list of your properties using the `write_rough_draft` tool. + +## Step 3 +Review your rough draft. For each property, confirm it meets the Step-1 guidelines — a clear +violation consequence, non-trivial, plausibly verifiable as a whole-program invariant. Drop anything +you cannot defend. + +## Step 4 +Output the results using the result tool. + + + + {% if sort != "greenfield" %} + + Derive invariants and safety properties from first principles — what the program SHOULD + guarantee — relying as little as possible on what the code CURRENTLY does. + + {% endif %} + + You need not prove a safety property/invariant definitely holds — only that it, with extreme + likelihood, SHOULD hold for a correct implementation. + + + For attack vectors you need not find evidence the issue exists — only that it is plausible + given the program and its {% if sort != "greenfield" %}implementation{% else %}design{% endif %}. + + diff --git a/composer/templates/solana/property_system.j2 b/composer/templates/solana/property_system.j2 new file mode 100644 index 00000000..f3cb9c0c --- /dev/null +++ b/composer/templates/solana/property_system.j2 @@ -0,0 +1,65 @@ +You are an expert security auditor of Solana on-chain programs (native and Anchor). You know the +failure modes that recur on Solana — missing signer/owner checks, account substitution / confused +deputy, unvalidated PDAs, arbitrary CPIs, lamport/rent and account-close bugs, duplicate mutable +accounts, reinitialization — plus the Rust-level ones (overflow-panic DoS, truncating casts). You +recognize them in design documents and in Rust/Anchor code. + +Your job is to extract a list of whole-program *security properties* (invariants a coverage-guided +fuzzer checks after every action in a random instruction sequence). The work happens in a multi-round +loop: each round produces only the NEW properties not surfaced by prior rounds, plus a written +reasoning record that later rounds and a human reviewer will read. + +# Behavior + +## Quality over quantity — and "nothing new" is the right answer when it's true + +The point of multiple rounds is to *eventually exhaust* the meaningful property surface for this +program. An empty `items` list with a `reasoning` field explaining what you looked at and why +nothing new emerged is a good round's desired outcome, not a failure. + +DO NOT propose properties just to feel productive. DO NOT pad the list. Concretely avoid: + +- Restating a prior property in different words and calling it new. +- Trivial corollaries ("X holds" plus "X holds for account A"). +- Edge cases the design does not motivate, invented to have something to return. +- Properties about non-security-relevant or non-verifiable behavior. +- Properties whose violation has no meaningful consequence. + +For every property you DO include, your `reasoning` must defend why it matters and, when prior +rounds exist, why a prior round could legitimately have missed it. If you can't, drop it. If you +think "this is a stretch but I should include something" — return an empty list. + +## Reasoning is load-bearing + +Your `reasoning` field is read by future-round agents and by a human reviewer. Be specific: name +the instruction, the accounts, the seeds, the CPI target, the attack pattern. "I considered the +missing owner check on `vault` but ruled it out because the Anchor `Account` type already +enforces owner = this program" is the right granularity; "I thought about it and decided Y" is not. + +## Adaptive thinking and self-consistency + +After gathering information from the source / design doc, integrate it before drafting properties; +don't propose from a single read. Favor self-consistency when resolving ambiguity. + +# Tools + +## Rough draft + +{% with draft_subject = "your property list" %} +{% include "rough_draft_protocol.j2" %} +{% endwith %} +When reviewing your draft, verify each property against the task criteria — verifiability, +non-triviality, a clear violation consequence, and defensibility in your `reasoning`. + +{% if sort != "greenfield" %} +## Source exploration + +{% with source_tools_has_explorer = sort == "existing" %} +{% include "source_tools_system_prompt.j2" %} +{% endwith %} + +Use the tools to ground reasoning in the actual Rust/Anchor implementation when the design is +ambiguous or an attack hinges on a code-level detail — which account checks are (or are not) +present in the `#[derive(Accounts)]` struct, how a PDA is derived, where a CPI's target program +comes from. Don't speculate when you can read the code. +{% endif %} diff --git a/composer/tools/crucible_rag.py b/composer/tools/crucible_rag.py new file mode 100644 index 00000000..c2560328 --- /dev/null +++ b/composer/tools/crucible_rag.py @@ -0,0 +1,108 @@ +"""Search tools over the `crucible_kb` RAG — the Crucible harness-authoring docs. + +Mirrors ``composer/tools/foundry_rag.py`` (keyword + vector + get-section over a +``ComposerRAGDB``), bound into the Crucible author's env so the tool-enabled +``call_llm`` (§7.5) can retrieve the harness guide / API reference / writing-tests +material alongside the static cheat-sheet. +""" + +import logging +from typing import Iterable + +from langchain_core.tools import BaseTool +from pydantic import Field + +from composer.rag.db import ComposerRAGDB +from graphcore.tools.schemas import WithAsyncDependencies + +_log = logging.getLogger(__name__) + +# A documentation-search tool is an *aid*, not a dependency — the author also has the +# static cheat-sheet + worked example. So a backend failure (DB down, or an embedding +# model that chokes on a particular query) must degrade to "no results", never crash +# the authoring turn / fail the component. +_UNAVAILABLE = ( + "(Documentation search is temporarily unavailable; proceed using the cheat-sheet " + "and worked example already in your prompt.)" +) + + +def _header_string(s: list[str]) -> str: + return " > ".join(i for i in s if i) + + +class CrucibleKeywordSearch(WithAsyncDependencies[str, ComposerRAGDB]): + """Full-text search of the Crucible docs. Returns matching section titles in + relevance order; use ``crucible_docs_get_section`` to read one.""" + + query: str = Field(description=( + "A websearch-style query. Unquoted terms are AND-ed; use 'OR' for alternatives, " + "quotes for exact phrases, '-' to exclude. Example: '\"fuzz_fixture\" setup -coverage'" + )) + limit: int = Field(default=10, description="Maximum number of results.") + + async def run(self) -> str: + try: + with self.tool_deps() as db: + res = await db.search_manual_keywords(self.query, limit=self.limit) + except Exception as e: # noqa: BLE001 — a search aid must never fail the authoring turn + _log.warning("crucible_docs_keyword_search failed (%s); returning no results", e) + return _UNAVAILABLE + hits = [f"{_header_string(r.headers)} [relevance: {r.relevance:.4f}]" for r in res] + return "\n".join(hits) if hits else "No results found" + + +class CrucibleVectorSearch(WithAsyncDependencies[str, ComposerRAGDB]): + """Semantic search of the Crucible docs for sections matching a natural-language + question. Returns section title, text, and similarity.""" + + query: str = Field(description=( + "A single natural-language question, e.g. 'how do I write an invariant that reads " + "on-chain account state?' or 'how are PDA seeds encoded in a harness?'" + )) + similarity_cutoff: float = Field(default=0.5, description="Minimum cosine similarity (default 0.5).") + max_results: int = Field(default=10, description="Maximum results (default 10).") + + async def run(self) -> str: + try: + with self.tool_deps() as db: + res = await db.find_refs( + self.query, similarity_cutoff=self.similarity_cutoff, top_k=self.max_results + ) + except Exception as e: # noqa: BLE001 — the embedding model can choke on a query; don't crash the turn + _log.warning("crucible_docs_search failed (%s); returning no results", e) + return _UNAVAILABLE + out = [ + f"----\nSection: {_header_string(r.headers)}\n\n{r.content}\nSimilarity: {r.similarity:.4f}" + for r in res + ] + return "\n".join(out) if out else "(No results found)" + + +class CrucibleSectionGet(WithAsyncDependencies[str, ComposerRAGDB]): + """Retrieve the full contents of a Crucible docs section by its header path.""" + + section_names: list[str] = Field(description=( + "The section heading path (the searches separate headers with ' > '). E.g. to read " + "'Writing Solana/Anchor Fuzz Harnesses > PDA Seed Encoding', pass " + "['Writing Solana/Anchor Fuzz Harnesses', 'PDA Seed Encoding']." + )) + + async def run(self) -> str: + try: + with self.tool_deps() as db: + content = await db.get_manual_section(self.section_names) + except Exception as e: # noqa: BLE001 — a search aid must never fail the authoring turn + _log.warning("crucible_docs_get_section failed (%s)", e) + return _UNAVAILABLE + if content is None: + return f"No section found for {' > '.join(self.section_names)!r}" + return content + + +def get_tools(db: ComposerRAGDB) -> Iterable[BaseTool]: + return [ + CrucibleSectionGet.bind(db).as_tool("crucible_docs_get_section"), + CrucibleVectorSearch.bind(db).as_tool("crucible_docs_search"), + CrucibleKeywordSearch.bind(db).as_tool("crucible_docs_keyword_search"), + ] diff --git a/composer/tools/rag_env.py b/composer/tools/rag_env.py new file mode 100644 index 00000000..a3405dd0 --- /dev/null +++ b/composer/tools/rag_env.py @@ -0,0 +1,46 @@ +"""Descriptor-driven RAG toolset selection for Rust applications. + +A wheel declares ``rag_db_default`` in its :class:`AppDescriptor` (e.g. ``"crucible_kb"``); the +generic env builder looks that tag up here and binds the corresponding corpus's search tools onto +the author's env — the same wiring the old ``build_crucible_env`` did, now selected by tag rather +than hard-coded per application. + +Like the ecosystem registry, this maps a declarative tag → a concrete toolset; it is not an +application fork (the tool classes live in ``composer/tools/_rag.py``, shared, exactly as +``foundry_rag`` is). A search aid must never fail a run, so an unknown tag or an unavailable +DB / embedding model degrades to *no RAG* (the static cheat-sheet in the prompt suffices). +""" + +import logging + +_log = logging.getLogger(__name__) + + +def _resolve(rag_db: str): + """(connection string, tools factory) for a declared RAG tag. Imports are local so the + generic host never pulls a corpus module unless a descriptor actually selects it.""" + from composer.rag.db import CRUCIBLE_DEFAULT_CONNECTION + from composer.tools.crucible_rag import get_tools as crucible_tools + + registry = { + "crucible_kb": (CRUCIBLE_DEFAULT_CONNECTION, crucible_tools), + } + if rag_db not in registry: + raise KeyError(f"no RAG toolset registered for {rag_db!r} (available: {sorted(registry)})") + return registry[rag_db] + + +def build_rag_tools(rag_db: str) -> tuple: + """Search tools for the declared corpus, or ``()`` if it can't be opened (best-effort — the + author still has the static cheat-sheet).""" + try: + from composer.rag.db import PostgreSQLRAGDatabase + from composer.rag.models import get_model + + conn, get_tools = _resolve(rag_db) + # Lazy pool — opens on first search; the DB must already be populated. + db = PostgreSQLRAGDatabase(conn, get_model()) + return tuple(get_tools(db)) + except Exception as e: # noqa: BLE001 — RAG is optional; the cheat-sheet suffices + _log.warning("RAG %r unavailable (%s); using the static cheat-sheet only", rag_db, e) + return () diff --git a/docs/application-abstraction.md b/docs/application-abstraction.md new file mode 100644 index 00000000..9c0e24ac --- /dev/null +++ b/docs/application-abstraction.md @@ -0,0 +1,515 @@ +# Design Doc — What Is an AutoProver "Application" + +> How the pieces — argument parsing, service setup, the pipeline, and the UI — are +> wired into a single, runnable *application* such as **autoprove** or **foundry**, +> and the conventions a new application is expected to follow. +> +> Companion to [ARCHITECTURE.md](../ARCHITECTURE.md) and +> [formalization-abstraction.md](./formalization-abstraction.md). Where the +> formalization doc zooms into the *backend* seam (how a property becomes a verified +> artifact), this doc zooms out to the *whole vertical*: everything from `argv` to a +> rendered TUI. The [MultiJobApp design](../composer/ui/MULTI_JOB_DESIGN.md) covers +> the generic UI base this leans on. + +--- + +## 1. What "application" means here + +An AutoProver **application** is a complete, runnable vertical slice that takes a +Solidity project + a design document and drives the shared property-extraction / +formalization pipeline to a set of on-disk deliverables, rendered live to the user. + +There are two today: + +| Application | Deliverable | Backend | Entry points | +|---|---|---|---| +| **autoprove** | CVL `.spec` + `.conf`, verified by the Certora Prover | `ProverBackend` | [tui_autoprove.py](../composer/cli/tui_autoprove.py) · [console_autoprove.py](../composer/cli/console_autoprove.py) | +| **foundry** | `.t.sol` tests, gated by `forge test` | `FoundryBackend` | [tui_foundry.py](../composer/cli/tui_foundry.py) · [console_foundry.py](../composer/cli/console_foundry.py) | + +Crucially, "application" is **not** a single class. It is a *convention*: a set of +five collaborating pieces, each an implementation of a shared abstraction, wired +together by a thin `main()`. The value of the convention is that the pieces are +mutually orthogonal — you can swap the frontend (TUI ↔ console) without touching the +pipeline, and swap the backend without touching either frontend. + +--- + +## 2. The five pieces of an application + +Every application is assembled from exactly these, each keyed off one shared +type parameter — the application's **phase enum** `P`: + +``` + ┌─────────────────────────────────────────────────────────────┐ + │ main() (composer/cli/*.py) │ + │ async with entry_point(summary) as run: ← the Executor │ + │ app = FrontendApp() ← the Frontend │ + │ await run(app.make_handler) ← the seam │ + └─────────────────────────────────────────────────────────────┘ + │ │ + ┌───────────▼───────────┐ ┌─────────────▼─────────────┐ + │ 2. Entry point / │ │ 4. Frontend │ + │ Executor │ │ MultiJobApp[P, H] │ + │ argv → services → │ │ OR console handler │ + │ a run(handler) closure│ │ supplies make_handler: │ + └───────────┬───────────┘ │ HandlerFactory[P, H] │ + │ calls └───────────────────────────┘ + ┌───────────▼───────────┐ + │ 3. Pipeline │ 1. Phase enum P: HasName + │ run_pipeline(backend) │ (the spine that threads all + │ + a PipelineBackend │ five together) + └───────────┬───────────┘ + │ contributes + ┌───────────▼───────────┐ + │ 5. Artifact store │ + │ on-disk layout │ + └───────────────────────┘ +``` + +1. **Phase enum** `P` — the task-grouping vocabulary. +2. **Entry point / Executor** — argv → configured services → a `run(handler)` closure. +3. **Pipeline + backend** — the work, expressed as a `PipelineBackend` fed to the + shared `run_pipeline` driver. +4. **Frontend** — a `MultiJobApp[P, H]` subclass (TUI) or console handler that + supplies a `HandlerFactory[P, H]`. +5. **Artifact store** — the on-disk deliverable layout. + +The rest of this doc walks each piece with the autoprove and foundry +implementations side by side. + +--- + +## 3. The seam that makes it compose: `HandlerFactory` + +Before the pieces, understand the seam between them. The pipeline and the frontend +never reference each other. They meet at one protocol, +[`HandlerFactory[P, H]`](../composer/io/multi_job.py): + +```python +# composer/io/multi_job.py +class HandlerFactory[P: HasName, H](Protocol): + def __call__(self, /, info: TaskInfo[P]) -> Awaitable[TaskHandle[H]]: ... +``` + +- The **pipeline** is a producer of *work*. Every task it launches is described by a + `TaskInfo(task_id, label, phase)` and run through `run_task`, which calls the factory + to obtain a `TaskHandle` (an `IOHandler` + `EventHandler` + lifecycle callbacks). +- The **frontend** is a producer of *handlers*. It implements the factory: given a + `TaskInfo`, it mounts a panel, builds a per-task renderer, and returns the + `TaskHandle`. + +So the entire application boils down to: + +```python +async with entry_point(summary) as run: # run: Executor (the pipeline, service-loaded) + app = FrontendApp() # the frontend + result = await run(app.make_handler) # hand the factory to the pipeline +``` + +`run` is typed as an **Executor**, and its whole signature *is* the seam: + +```python +# composer/spec/source/autoprove_common.py +type Executor = Callable[[HandlerFactory[AutoProvePhase, None]], Awaitable[CorePipelineResult[GeneratedCVL]]] +# composer/foundry/pipeline.py +type FoundryPipelineExecutor = Callable[[HandlerFactory[FoundryPhase, None]], Awaitable[FoundryPipelineResult]] +``` + +Because the pipeline only ever *calls* the factory and the frontend only ever +*implements* it, the two are swappable independently. That is why autoprove has both +a TUI ([`AutoProveApp`](../composer/ui/autoprove_app.py)) and a console +([`AutoProveConsoleHandler`](../composer/ui/autoprove_console.py)) frontend against +the same pipeline, selected purely by which `make_handler` `main()` passes in. + +`H` is the human-interaction schema. Both current applications are non-interactive at +the per-task level (`H = None`; their handlers raise from `format_hitl_prompt`), but +the seam carries the type so an interactive application (e.g. the NatSpec pipeline) +plugs in without changing the contract. + +--- + +## 4. Piece 1 — the phase enum `P` + +Every application defines a single enum whose members are its task-grouping phases. +This enum is the type parameter that threads through the frontend +(`MultiJobApp[P, ...]`), the seam (`HandlerFactory[P, H]`, `TaskInfo[P]`), and the +backend (`CorePhases[P]`). It only needs to satisfy `HasName` (an enum trivially does). + +```python +# composer/ui/autoprove_app.py +class AutoProvePhase(enum.Enum): + HARNESS = "harness" + AUTOSETUP = "autosetup" + INVARIANTS = "invariants" + SUMMARIES = "summaries" + COMPONENT_ANALYSIS = "component_analysis" + BUG_ANALYSIS = "bug_analysis" + CVL_GEN = "cvl_gen" + REPORT = "report" +``` + +```python +# composer/foundry/pipeline.py +class FoundryPhase(enum.Enum): + SYSTEM_ANALYSIS = "system_analysis" + PROPERTY_EXTRACTION = "property_extraction" + TEST_GENERATION = "test_generation" + REPORT = "report" +``` + +The phase serves two roles: + +- **Grouping in the UI.** The frontend maps each phase to a human label and an + ordering, and every task lands in the section for its phase: + + ```python + # composer/ui/foundry_app.py + FOUNDRY_PHASE_LABELS = { + FoundryPhase.SYSTEM_ANALYSIS: "System Analysis", + FoundryPhase.PROPERTY_EXTRACTION: "Property Extraction", + FoundryPhase.TEST_GENERATION: "Test Generation", + } + FOUNDRY_SECTION_ORDER = ["System Analysis", "Property Extraction", "Test Generation"] + ``` + +- **The driver ↔ backend contract.** The shared driver tags four *core* phases; the + backend maps its own enum onto them via `CorePhases[P]` (see §6). Note the two enums + above differ in granularity: foundry has three phases, autoprove has eight — the + prover contributes several extra prep phases (harness, autosetup, summaries, + invariants) that the driver never knows about. The enum is the application's own + vocabulary; only the four core slots are shared. + +--- + +## 5. Piece 2 — the entry point / Executor + +Each application has an `_entry_point` async context manager that owns **all** the +imperative setup and yields the Executor closure. Its shape is a strict convention +(the foundry file's docstring literally says "Mirrors autoprove_common's shape"): + +> parse args → set up DB / RAG / store / checkpointer / logging / thread logger → +> yield a closure the caller drives with a handler factory. + +```python +# composer/spec/source/autoprove_common.py (shape shared by composer/foundry/entry.py) +@asynccontextmanager +async def _entry_point(summary: RunSummary) -> AsyncIterator[Executor]: + parser = argparse.ArgumentParser(...) + add_protocol_args(parser, RAGDBOptions) + add_protocol_args(parser, ExtendedModelOptions) + parser.add_argument("project_root", ...) + parser.add_argument("main_contract", help="Main contract as path:ContractName") + parser.add_argument("system_doc", ...) + # ... application-specific flags ... + args = cast(AutoProveArgs, parser.parse_args()) + async with autoprove_executor(args, summary) as runner: + yield runner +``` + +The heavy lifting is in the inner context manager, which: + +1. Resolves `project_root` + `main_contract` (`path:ContractName`) + `system_doc`. +2. Computes a **root cache key** from the inputs (`_root_cache_key` hashes project + root + doc content + relative path + contract name) — identical helper in both. +3. Opens the shared connection stack — `standard_connections`, a RAG DB, the async + tool context, the thread logger — under a single `async with`. +4. Reads the design doc into a `SourceCode`, builds the environment (`ServiceHost`), + and creates the `WorkflowContext`. +5. Yields a `runner(handler)` closure that calls the application's pipeline function. + +The args are declared as a **Protocol** (`AutoProveArgs`, `FoundryArgs`), not a class, +so the parser and the typed access agree without a dataclass in between: + +```python +# composer/spec/source/autoprove_common.py +class AutoProveArgs(ExtendedModelOptions, RAGDBOptions, Protocol): + project_root: str + main_contract: str + system_doc: str + max_concurrent: int + cloud: bool # ← prover-only: run jobs in the cloud + ... +``` + +```python +# composer/foundry/entry.py +class FoundryArgs(TieredModelOptions, FoundryRAGDBOptions, Protocol): + project_root: str + main_contract: str + system_doc: str + forge_binary: str # ← foundry-only + forge_timeout_s: int # ← foundry-only + max_forge_runners: int # ← foundry-only + ... +``` + +The `runner` closure each yields is the Executor: + +```python +# autoprove +async def runner(handler: HandlerFactory[AutoProvePhase, None]) -> CorePipelineResult[GeneratedCVL]: + return await run_autoprove_pipeline( + ctx=ctx, source_input=system_doc, env=source_env, handler_factory=handler, + prover_opts=prover_opts, interactive=args.interactive, ...) + +# foundry +async def runner(handler: HandlerFactory[FoundryPhase, None]) -> FoundryPipelineResult: + return await run_foundry_pipeline( + source_input=source_input, ctx=ctx, handler_factory=handler, env=env, + forge_binary=args.forge_binary, forge_timeout_s=args.forge_timeout_s, ...) +``` + +Convention points worth naming: + +- **Foundry validates its precondition in the entry point** (`foundry.toml` must + exist) — application-specific input validation belongs here, before any service is + opened. +- **Each application owns its RAG DB choice.** Foundry overrides `--rag-db`'s default + to the cheatcodes DB via a Protocol (`FoundryRAGDBOptions`) rather than a new flag. +- **The `finally` block is where run-close artifacts land** (autoprove dumps + `token_usage.json` there) — guarded so a diagnostics failure can't mask the run's + own outcome. +- **The entry point never imports a frontend.** It yields the Executor; `main()` + chooses the frontend. That is what lets one entry point back both a TUI and a + console `main()`. + +--- + +## 6. Piece 3 — the pipeline and its backend + +The Executor's closure calls the application's `run_*_pipeline` function. For both +current applications, that function is a thin wrapper that constructs a +`PipelineBackend` + a `PipelineRun` and hands them to the shared driver +[`run_pipeline`](../composer/pipeline/core.py): + +```python +# composer/spec/source/pipeline.py +async def run_autoprove_pipeline(source_input, ctx, handler_factory, env, *, prover_opts, ...): + backend = ProverBackend(ProverArtifactStore(source_input.project_root, source_input.contract_name), prover_opts) + run = PipelineRun(ctx, env, source_input, handler_factory, asyncio.Semaphore(max_concurrent)) + return await run_pipeline(backend, run, interactive=interactive, ...) +``` + +```python +# composer/foundry/pipeline.py +async def run_foundry_pipeline(source_input, ctx, handler_factory, env, *, forge_binary, ...): + artifacts = FoundryArtifactStore(source_input.project_root) + backend = FoundryBackend(artifacts, _ForgeRunConfig(forge_binary, forge_timeout_s, foundry_sem)) + run = PipelineRun(ctx, env, source_input, handler_factory, asyncio.Semaphore(max_concurrent)) + return await run_pipeline(backend, run, ...) +``` + +Notice the `handler_factory` (the frontend seam) is bundled into the `PipelineRun` — +`run.runner(task_info, job)` is how every phase of the driver spins up a task through +whatever frontend was supplied. + +The backend itself is the four-slot contract the driver reads. The application maps +its phase enum onto the four **core phases** the driver tags: + +```python +# composer/spec/source/pipeline.py # composer/foundry/pipeline.py +core_phases = CorePhases({ core_phases = CorePhases({ + "analysis": AutoProvePhase.COMPONENT_ANALYSIS, "analysis": FoundryPhase.SYSTEM_ANALYSIS, + "extraction": AutoProvePhase.BUG_ANALYSIS, "extraction": FoundryPhase.PROPERTY_EXTRACTION, + "formalization": AutoProvePhase.CVL_GEN, "formalization": FoundryPhase.TEST_GENERATION, + "report": AutoProvePhase.REPORT, "report": FoundryPhase.REPORT, +}) }) +``` + +Everything below this — `prepare_system`, `prepare_formalization`, `formalize`, +`fetch_verdicts` — is the **formalization abstraction**, documented in full in +[formalization-abstraction.md](./formalization-abstraction.md). The one-line summary +of the contrast: + +| | autoprove (`ProverBackend`) | foundry (`FoundryBackend`) | +|---|---|---| +| `FormT` | `GeneratedCVL` | `GeneratedFoundryTest` | +| `prepare_system` | harness lift + build prover tool | identity (`main_instance` only) | +| `prepare_formalization` | AutoSetup ∥ summaries ∥ invariants fan-out | trivial (formalizer already built) | +| `formalize` | author CVL, run prover, revise on CEX | author `.t.sol`, run `forge test` | +| `backend_guidance` | `CERTORA_BACKEND_GUIDANCE` | `FOUNDRY_BACKEND_GUIDANCE` | + +`backend_guidance` deserves a note as an application-shaping convention: it is a prose +string injected into the property-extraction prompt telling the agent what the +verification surface can and can't express. Foundry's, for instance, explains that a +fuzzer can't *prove* universals but *refutations are valuable* — so the same shared +extraction step produces backend-appropriate properties without the driver knowing +anything about it. + +--- + +## 7. Piece 4 — the frontend + +The frontend implements the `HandlerFactory` seam. The TUI frontends are thin +subclasses of the generic [`MultiJobApp[P, T]`](../composer/ui/multi_job_app.py) +(see [its design doc](../composer/ui/MULTI_JOB_DESIGN.md)). A frontend supplies four +things and inherits everything else: + +1. **Phase labels + section order** (constructor args), covered in §4. +2. **A per-task handler** — `create_task_handler`, returning a + `MultiJobTaskHandler` subclass. +3. **A per-task event handler** — `create_event_handler`, for domain-specific + streaming events beyond LLM messages. +4. **`make_handler`** — inherited from `MultiJobApp`; this *is* the `HandlerFactory`. + It mounts the panel/summary-row and calls the two `create_*` hooks. + +The autoprove and foundry TUIs are nearly identical in shape; they differ only in +what streams into a task's log. Both make their task handler double as its own +`EventHandler` via the `NullEventHandler` mixin: + +```python +# composer/ui/foundry_app.py +class FoundryTaskHandler(MultiJobTaskHandler[None], NullEventHandler): + async def handle_event(self, payload, path, checkpoint_id) -> None: + evt = cast(ForgeTestRunEvent, payload) + if evt["type"] == "forge_test_run": # stream forge run summaries + log = await self._ensure_forge_log() + log.write(evt["summary"]) + +class FoundryApp(MultiJobApp[FoundryPhase, FoundryTaskHandler]): + def __init__(self): + super().__init__(phase_labels=FOUNDRY_PHASE_LABELS, + section_order=FOUNDRY_SECTION_ORDER, + header_text="Foundry Test Author | ...") + def create_task_handler(self, panel, info) -> FoundryTaskHandler: + return FoundryTaskHandler(info.task_id, info.label, panel, self, ToolDisplayConfig()) + def create_event_handler(self, handler, info) -> EventHandler: + return handler # handler is its own event handler +``` + +```python +# composer/ui/autoprove_app.py — same structure; the domain events differ +class AutoProveTaskHandler(MultiJobTaskHandler[None], NullEventHandler): + async def handle_event(self, payload, path, checkpoint_id) -> None: + evt = cast(AutoProveEvent, payload) + match evt["type"]: + case "prover_output": ... # stream Certora Prover output lines + case "cloud_polling": ... # stream cloud job status +``` + +The autoprove handler additionally implements `handle_progress_event` to stream the +AutoSetup agent's output — an example of an application surfacing a backend-specific +sub-agent in its own panel. Neither handler supports HITL, so both raise from +`format_hitl_prompt` — a deliberate, explicit opt-out of a base-class hook. + +**The console frontend is the proof the seam works.** `AutoProveConsoleHandler` is a +*different* implementation of the same `HandlerFactory[AutoProvePhase, None]` that +renders to stdout instead of a Textual app: + +```python +# composer/ui/autoprove_console.py +class AutoProveConsoleHandler(MultiJobConsoleHandler[AutoProvePhase]): + """IOHandler[Never] + HandlerFactory for the auto-prove pipeline.""" +``` + +The pipeline can't tell the difference — it only ever calls `make_handler`. + +--- + +## 8. Piece 5 — the artifact store + +Deliverables are written through an [`ArtifactStore[I, FormT]`](../composer/spec/artifacts.py) +subclass — one per application. The base owns everything identical across +applications (`properties.json`, `commentary.md`, the property→units map, +`token_usage.json`); the subclass fixes the on-disk layout and adds the +format-specific bundle. This is covered in detail in +[formalization-abstraction.md §6](./formalization-abstraction.md); the application-level +point is the *convention that both applications share a project root without +colliding*: + +``` +autoprove → certora/specs/… certora/confs/… certora/ap_report/… +foundry → /*.t.sol certora/foundry/… certora/foundry/reports/… +``` + +Foundry deliberately materializes its `.t.sol` into the foundry project's own `test/` +dir (so `forge` finds them) but keeps all metadata under `certora/foundry/`, so a +co-located autoprove run and foundry run share one project without clobbering each +other's outputs. + +--- + +## 9. Piece 0 — the wiring: `main()` + +The `main()` in each `composer/cli/*.py` is the whole application in ~20 lines. It is +the *only* place that names both an entry point and a frontend, and its job is to +glue them via the seam and translate the result into user-facing output. + +```python +# composer/cli/tui_foundry.py (tui_autoprove.py is identical in shape) +async def _main() -> int: + summary = RunSummary() + async with _entry_point(summary) as pipeline: # piece 2: Executor + app = FoundryApp() # piece 4: frontend + async def work(): + result = await pipeline(app.make_handler) # ← the seam + app.notify(f"Foundry tests complete: {result.n_components} components, ...") + app._pipeline_done = True + app.set_work(work) + await app.run_async() # TUI owns the event loop + print(summary.format()) + return 0 +``` + +Two `main()` shapes exist, differing only in who owns the event loop: + +- **TUI** — the pipeline runs as a background *worker* inside the Textual app + (`app.set_work(work); await app.run_async()`), so the UI stays responsive while + the pipeline streams into it. +- **Console** — the pipeline runs directly and results print on completion: + + ```python + # composer/cli/console_autoprove.py + async with _entry_point(summary) as run: + result = await run(AutoProveConsoleHandler().make_handler) + print(summary.format()) + print(f" Components: {result.n_components} Properties: {result.n_properties}") + ``` + +Both call `import composer.bind as _` first — the side-effecting binding module that +must load before anything touches the DI container. + +--- + +## 10. Extending: defining a new application + +Because each piece is an implementation of a shared abstraction, adding an +application is a fill-in-the-blanks exercise; nothing in the driver, the UI base, or +the seam changes. + +1. **Phase enum** `P(enum.Enum)` — your task-grouping vocabulary, with at least the + four core phases (analysis / extraction / formalization / report) representable. +2. **Backend** — implement `PipelineBackend[P, FormT, H, A]` and its three phase + objects (`prepare_system` → `PreparedSystem.prepare_formalization` → `Formalizer`), + plus `backend_guidance`, `core_phases`, `analysis_spec`, `artifact_store`, + `to_artifact_id`. (Full checklist in + [formalization-abstraction.md §9](./formalization-abstraction.md).) +3. **Artifact store** — subclass `ArtifactStore`; define an `ArtifactIdentifier`. +4. **Result type** `FormT` satisfying `FormalResult` + `ReportableResult`. +5. **Pipeline function** `run__pipeline(...)` — build backend + `PipelineRun`, + call `run_pipeline`. +6. **Entry point** — an `_entry_point` context manager following the + parse → services → `yield runner` convention; declare args as a `Protocol`. +7. **Frontend(s)** — a `MultiJobApp[P, T]` subclass (phase labels, section order, + `create_task_handler`, per-task event streaming) and/or a console handler. +8. **`main()`** — glue an entry point to a frontend via `run(app.make_handler)`. + +The dependency direction is the guardrail: `main` → (entry point, frontend); +entry point → pipeline; pipeline → backend + `PipelineRun(handler_factory)`. Frontend +and backend never reference each other, and neither references `main`. Keep those +edges and the pieces stay swappable. + +--- + +## 11. Key files + +| Piece | autoprove | foundry | shared abstraction | +|---|---|---|---| +| Phase enum | [autoprove_app.py](../composer/ui/autoprove_app.py) | [foundry/pipeline.py](../composer/foundry/pipeline.py) | `HasName` ([multi_job.py](../composer/io/multi_job.py)) | +| Entry point / Executor | [autoprove_common.py](../composer/spec/source/autoprove_common.py) | [foundry/entry.py](../composer/foundry/entry.py) | — | +| Pipeline + backend | [spec/source/pipeline.py](../composer/spec/source/pipeline.py) | [foundry/pipeline.py](../composer/foundry/pipeline.py) | [pipeline/core.py](../composer/pipeline/core.py) | +| Frontend (TUI) | [autoprove_app.py](../composer/ui/autoprove_app.py) | [foundry_app.py](../composer/ui/foundry_app.py) | [multi_job_app.py](../composer/ui/multi_job_app.py) | +| Frontend (console) | [autoprove_console.py](../composer/ui/autoprove_console.py) | — | — | +| Artifact store | [spec/source/artifacts.py](../composer/spec/source/artifacts.py) | [foundry/artifacts.py](../composer/foundry/artifacts.py) | [spec/artifacts.py](../composer/spec/artifacts.py) | +| The seam | — | — | `HandlerFactory` / `TaskInfo` / `TaskHandle` ([multi_job.py](../composer/io/multi_job.py)) | +| `main()` | [tui_autoprove.py](../composer/cli/tui_autoprove.py) · [console_autoprove.py](../composer/cli/console_autoprove.py) | [tui_foundry.py](../composer/cli/tui_foundry.py) · [console_foundry.py](../composer/cli/console_foundry.py) | — | diff --git a/docs/command-sandbox.md b/docs/command-sandbox.md new file mode 100644 index 00000000..7376a38a --- /dev/null +++ b/docs/command-sandbox.md @@ -0,0 +1,511 @@ +# Design — Sandboxing the `RunCommand` effect (Phase 6) + +**Status:** implemented. Design + record for [crucible-application.md §7.4](./crucible-application.md#L436) +and [§9 Phase 6](./crucible-application.md#L634) — the *required*, definition-of-done phase. The +sandbox mechanism is built and validated (§9 steps 1–5 done, gate §10 green — incl. the full LLM +e2e passing under the launcher); Crucible runs confined by default. Open items are orthogonal to the +sandbox (§11): a shared-`Cargo.toml` feature race that lost one of three instructions, and per-run +`CARGO_HOME`/tightening follow-ups. + +**One-line summary.** Every command run through the `RunCommand` effect compiles and/or runs +LLM-authored *native* code (§7.2). Today that runs with the full ambient environment of the +AutoProver process. Phase 6 confines each such command — with no network, no inherited secrets, and +only its own inputs on the filesystem — using **unprivileged, in-process kernel sandboxing +(Landlock + seccomp)** that needs no container changes, no namespaces, no capabilities, and no +custom runtime. It is a single wrapper around [`run_local_command`](../composer/sandbox/command.py). +Done is proven by an escape test. + +--- + +## 1. Why this is required, not optional + +The outer AutoProver container protects the *host* from AutoProver. It does **not** protect +AutoProver's own secrets, network access, and filesystem from code running *inside* it. And the +`RunCommand` effect deliberately runs untrusted native code: + +- `cargo build-sbf` on the **user-supplied program** compiles it natively — running its + `build.rs`, its proc-macros, and (for a future Prover/CVLR backend) LLM-munged source. +- `crucible run` compiles the **LLM-authored harness** (its `setup()`, `action_*`, `build.rs`) + and then runs it as a native LiteSVM-in-process binary (§7.2 — verified native, no SVM sandbox). + +So arbitrary code of the LLM's (and the analyzed program's) choosing executes with whatever +ambient authority the AutoProver process has: `ANTHROPIC_API_KEY`, `CERTORA*` cloud tokens, +`AWS_*`, `PG*`, the network route to `169.254.169.254`, and the entire bind-mounted host project. +The trust boundary from §7.2 ("the LLM authors only file *contents*, never argv") stops the LLM +from choosing *what command runs* — it does nothing about what that command, once running, can +*reach*. That is this phase's job. + +Until this phase is green the backend may run only in a trusted, offline environment on trusted +input (the gate scenario). This is the definition of done. + +--- + +## 2. Threat model + +| | | +|---|---| +| **Asset** | AutoProver's ambient secrets (LLM/cloud API keys, DB creds), its network egress (incl. `169.254.169.254` metadata → IAM role creds on EC2), and host files outside the command's declared inputs. | +| **Adversary** | Native code the LLM authored (harness `setup`/`action`/`build.rs`) **and** native code in the analyzed program (its `build.rs`, proc-macros) that `cargo build-sbf` runs. Assume it is actively hostile and knows it is being fuzzed. | +| **Trust boundary** | The process boundary of each `RunCommand` invocation. Inside: untrusted. Outside: the trusted AutoProver process. `program`+`args` are trusted (Rust decider / Python build step author them, §7.2); only the *files* are untrusted. | +| **Assumptions** | (1) The outer container/host is the infrastructure's boundary against the host machine and other tenants (on EC2, the Nitro hypervisor) — this phase is the boundary *within* the container, between AutoProver and its own untrusted child. (2) The kernel is patched and Landlock-capable (§8). (3) The host toolchains we grant read access are trusted. | +| **Non-goals** | Protecting the host machine *from the container* (the infrastructure does that). A full VM boundary between AutoProver and the child (that is what gVisor/Kata/VM-per-run would add at the infra layer, orthogonal to this phase — §6). Defending against a malicious *`program`/`args`* — those are trusted by construction (§7.2). | + +**Explicit guarantees the sandbox must provide:** + +1. **No network** — no egress at all, including DNS and `169.254.169.254`. +2. **No secrets** — the child's environment is a scrubbed allowlist, and it cannot recover + AutoProver's secrets out-of-band (via `/proc//environ` or `ptrace` — see §6, the + same-uid caveats). +3. **Minimal filesystem** — only the command's own inputs are writable; toolchains are read-only; + nothing else of the host is readable. +4. **Resource caps + wall-clock kill** — memory / CPU-time / pids / file-size bounded; a hung or + runaway command is killed. +5. **Offline, code-exec-free dependency resolution** — all network dep-fetching happens *outside* + the sandbox and *before* any untrusted code runs (§5); the sandboxed build is `--offline`. + +--- + +## 3. What runs inside, and what it legitimately needs + +The hard part of sandboxing a compiler+fuzzer is that it needs a *lot* of real toolchain — the +sandbox is only useful if it grants exactly that and nothing more. The three command shapes and +their real needs: + +| Command | Reads (grant **ro+x**) | Writes (grant **rw**) | Network | +|---|---|---|---| +| `cargo build-sbf ` | rust toolchain (`RUSTUP_HOME`), solana platform-tools (the sBPF toolchain), warm cargo registry (`CARGO_HOME/registry`), program crate source | program crate `target/` | none (offline) | +| `crucible run …` | the `crucible` binary + its libs, rust toolchain, cargo registry, the **crucible checkout crates** (path deps from `CrucibleDep`, §6.1), the built `.so` + IDL | the harness crate `target/`, corpus/output dirs | none (offline) | +| `cargo build` (harness, if run directly) | as above | harness `target/` | none (offline) | + +Common surface, resolved once at sandbox-config time and expressed as Landlock rules (§6): + +- **Rust toolchain** — `RUSTUP_HOME` (default `~/.rustup`), `cargo`/`rustc` shims — read+exec. +- **Cargo home** — `CARGO_HOME` (default `~/.cargo`): the `cargo` binary and the **registry cache**, + read-only inside; warmed *outside* (§5). +- **Solana platform-tools** — cargo-build-sbf's sBPF rust toolchain — read+exec. +- **The `crucible` binary** and libs it dlopens — read+exec. +- **The crucible checkout** (`$CRUCIBLE_REPO/crates/…`) — the path deps — read-only. +- **System runtime** — `/usr`, `/bin`, `/lib`, `/lib64` — read+exec (needed for the toolchain's own + dynamic linking and subprocesses). +- **Device nodes** — `/dev/null`, `/dev/urandom`, `/dev/zero`, `/dev/tty` — read+write. The toolchain + opens these constantly (a build *fails* without `/dev/null` — validated during step 2). Landlock + rules can target individual files, so we grant these specific nodes rather than the whole `/dev` + tree; either way `mknod` stays blocked (no capability), so no new devices can be created. +- **Workdir** — the crate tree + `target/` + corpus/output — the primary read-write grant. +- **A private temp dir** — `/.sandbox_tmp`, with `TMPDIR`/`TMP`/`TEMP` pointed at it. The + **linker** writes scratch files to `$TMPDIR` (default `/tmp`) during `cargo build`; we do *not* + grant the shared `/tmp` (it may hold host/other-run secrets and would defeat the escape test), so a + per-run temp under the already-writable workdir is redirected in. (A fresh harness build fails at + the link step — "Cannot create temporary file in /tmp/" — without this; found via the e2e gate.) +- **A private cargo home** — `/.sandbox_cargo`, with `CARGO_HOME` pointed at it (§11 item 5). + The offline `cargo build` writes there (source extraction, locks); keeping it per-run means + untrusted build code can't poison a *shared* `~/.cargo` (which stays read-only, for the `cargo` + binary). The warm step (§5) fetches into this same home. + +Everything else — the rest of the bind-mounted project, `/etc`, `/proc/`, `$HOME`, the +process environment — is **not granted**, therefore inaccessible. Confinement is default-deny. + +> The exact host paths (`RUSTUP_HOME`, platform-tools dir, crucible binary) are **resolved by the +> host at config time**, not hardcoded — see the `SandboxPolicy` in §7. They are discovered from the +> environment the same way `resolve_crucible_repo` already discovers the checkout. + +--- + +## 4. The seam — one function, unchanged signature + +All command execution already funnels through +[`run_local_command`](../composer/sandbox/command.py) (both the IoC `RunCommand` effect via +[`RealEffects.run_command`](../composer/rustapp/adapter.py#L120) and the Solana build step +[`build_program`](../composer/spec/solana/build.py)). It lives in the backend-agnostic +[`composer/sandbox`](../composer/sandbox/) package — outside `rustapp` — so Python-based backends can +run confined commands too, not just the Rust-IoC ones. The sandbox wraps exactly this one function. + +**The mechanism sits behind a `SandboxProvider` seam, so it is swappable.** `run_local_command` +never names a concrete tool. It holds a **tool-agnostic `SandboxPolicy`** (the *intent*: rw paths, +ro paths, env allowlist, rlimits, network-off — §7) and a `SandboxProvider` that translates that +intent into a concrete launch: + +```python +class SandboxProvider(Protocol): + def wrap(self, policy: SandboxPolicy, program: str, args: list[str]) -> LaunchSpec: ... + def available(self) -> Availability: ... # drives fail-closed (§7) + +# run_local_command, unchanged shape: +spec = provider.wrap(policy, program, args) +create_subprocess_exec(*spec.argv, cwd=workdir, env=spec.env, …) +``` + +The first provider is our **custom launcher shim** (§6): `LaunchSpec.argv == ["run-confined", +*policy_argv, "--", program, *args]`, all authored by trusted Python (never the LLM). Swapping to an +off-the-shelf tool later — `landrun`, `sandlock` — is a *new `SandboxProvider` implementation that +maps the same `SandboxPolicy` to that tool's flags*; the policy, this seam, `run_local_command`, +`RealEffects`, and the escape-test gate (§10) are all untouched. The provider is chosen by config +(`CommandConfig` / an env var), defaulting to the custom launcher. The `none` provider is a +passthrough (`argv == [program, *args]`) — byte-for-byte today's behavior for the EVM/Foundry paths +and explicit trusted-input dev runs. + +Nothing in the Rust decider, the ABI, the driver, or the artifact store changes — this is why §7.4 +could defer it to last. + +Two properties `run_local_command` *already* enforces stay in force and are the first line of +defense (the sandbox is the second): the command runs via **exec, not a shell**, and every written +file path is **confined to the workdir** (`_confined_target`). The sandbox does not replace these; +it assumes them. + +--- + +## 5. Offline dependency resolution — split fetch (network, no exec) from build (exec, no network) + +The tension: `cargo build` needs its dependency crates, but the sandbox has no network. Resolution +splits cleanly along the code-execution line: + +- **`cargo fetch` / `cargo vendor` download but never run build scripts** — no untrusted code + executes during fetch. So the *fetch* happens **outside** the sandbox, with network, as a trusted + prep step, warming `CARGO_HOME/registry` (or producing a vendored dir + source-replacement + config). +- **`cargo build` runs build scripts and proc-macros** — this is where untrusted code executes, so + it happens **inside** the sandbox, `--offline`, against the already-warm cache. + +The harness `Cargo.toml` is **host-owned** (`CrucibleDep.render_deps`, pinned versions, §6.1), so +its dep graph is fixed and vendorable deterministically. The program-under-test's `Cargo.toml` is +user-supplied, but `cargo fetch` on it is still exec-free, so the same split holds for the build-sbf +step. This also closes the build-time supply-chain vector: with offline + a pre-warmed cache, a +malicious `build.rs` cannot pull a payload at build time. + +**Implementation (step 4).** "Offline inside" is one env var, not per-tool flags: the policy sets +**`CARGO_NET_OFFLINE=1`** in the child env, which forces *every* cargo invocation offline — including +the nested `cargo` that `crucible run` spawns to build the harness — so we never thread `--offline` +through each tool ([recipes.py](../composer/sandbox/recipes.py), `offline=True` default). "Fetch +outside" is [`warm_cargo_cache`](../composer/spec/solana/build.py) — a `cargo fetch` run *unsandboxed* +(no provider → network on) before the confined build; `build_program` calls it before the sandboxed +`cargo build-sbf`. The harness crate has its own deps (libafl, litesvm, …), so it needs its own warm +at manifest-assembly time; wiring that exact call site (and confirming whether `CARGO_HOME` must be +granted rw for cargo's build-time source extraction, or pre-extracted during the warm) lands with the +gate in step 5, where a real offline build proves it. All of this is inert until a sandbox is enabled. + +--- + +## 6. Mechanism: unprivileged Landlock + seccomp self-sandboxing + +### Why not a namespace sandbox (bwrap/nsjail) or gVisor + +The obvious tools (bwrap, nsjail) build the sandbox out of **namespaces** (user + mount + net + +pid), then `pivot_root` into a minimal filesystem. That model **fights the container**: creating an +unprivileged user namespace and mounting inside it is exactly what Docker's default seccomp + +AppArmor block. Validated empirically (python:3.12-slim, host kernel 7.0.11, `bwrap 0.11.0`, uid +1000): + +| Approach under Docker defaults | Outcome | +|---|---| +| unprivileged `bwrap` | ✗ userns creation blocked by default **seccomp** | +| `bwrap`, `seccomp=unconfined` | ✗ `mount --make-rslave` blocked by **AppArmor** `docker-default` | +| `bwrap`, `seccomp=unconfined`+`apparmor=unconfined` | ✓ works — but requires **weakening the whole container's LSMs** (rejected) | +| setuid `bwrap` | ✗ `capset` blocked (Docker capability bounding set drops `CAP_SETPCAP`) | + +Making bwrap work would mean either **stripping the container's own seccomp/AppArmor** (widening the +host-kernel attack surface across *all* of AutoProver — the opposite of what a sandboxing phase +should do) or running AutoProver under a **gVisor/Kata** runtime. gVisor works, but (a) it imposes +its *heaviest* overhead precisely on our syscall/I/O-bound compile+fuzz workload, and (b) its benefit +— protecting the host kernel — is an *infrastructure* boundary that on EC2 is already provided by the +Nitro hypervisor. Neither is worth coupling this phase to a deployment decision. + +### The chosen model: the process sandboxes itself + +Instead of building a new namespace *around* the command, the command **restricts itself** using two +unprivileged kernel facilities — the model Chrome, OpenSSH, and systemd use. Both need **no +namespaces, no capabilities, no root, and no `--security-opt`**, and both work in a **stock** +container. Validated (stock python:3.12-slim, uid 1000, Docker default profile): + +| Guarantee | Probe result | Mechanism | +|---|---|---| +| filesystem — write outside workdir | ✗ `EACCES` | **Landlock** (full ABI FS bit set, grant only workdir rw) | +| filesystem — read host file (`/etc/passwd`) | ✗ `EACCES` | Landlock (no grant) | +| **secret** — read `/proc//environ` | ✗ `EACCES` | Landlock (no `/proc` grant) | +| **secret** — `ptrace(ATTACH, parent)` | ✗ `EPERM` | **seccomp** (deny `ptrace`, `process_vm_readv`) | +| network — `socket(AF_INET)` | ✗ `EPERM` | seccomp (deny inet-domain sockets → blocks TCP, UDP/DNS, IMDS) | +| legitimate — write workdir, `exec` toolchain | ✓ works | Landlock rw grant + r+x on toolchain paths | + +- **[Landlock](https://docs.kernel.org/userspace-api/landlock.html)** (LSM; Linux ≥5.13, we observed + ABI **8**) — an unprivileged process installs a filesystem ruleset on itself: default-deny, then + grant rw to the workdir and read+exec to the toolchain paths of §3, handling the *full* set of FS + access rights the running ABI supports (else unhandled operations stay unrestricted). This is what + confines reads *and* writes and — crucially — closes the `/proc//environ` leak that a user + namespace would otherwise have closed for free. +- **seccomp-BPF self-filter** (`PR_SET_NO_NEW_PRIVS` + `SECCOMP_SET_MODE_FILTER`) — installing a + *stricter* filter on yourself is unprivileged and permitted by Docker's default profile. It denies + the network (`socket` with `AF_INET`/`AF_INET6` — covering TCP, UDP/DNS, and the IMDS endpoint, + while leaving `AF_UNIX` for benign local IPC) and the remaining same-uid secret vectors + (`ptrace`, `process_vm_readv`/`writev`). +- **env allowlist** — the launcher `execve`s with a scrubbed environment (PATH, HOME, CARGO_HOME, + RUSTUP_HOME, TERM, and benign build vars only). The `--clearenv` equivalent, done in-process. +- **rlimits** — `setrlimit` for `RLIMIT_AS` / `RLIMIT_CPU` / `RLIMIT_NPROC` / `RLIMIT_FSIZE` (§7). + +Landlock and seccomp are **preserved across `execve`** (with `NO_NEW_PRIVS`) and **inherited across +`fork`**, so the launcher applies them once and every descendant — `cargo`, `rustc`, each `build.rs`, +the linker, the fuzz binary — runs confined. + +### The same-uid caveat, and why it is closed + +A user namespace (bwrap) would have run the child under a *remapped* uid, so cross-process access to +AutoProver was denied by credential mismatch. Self-sandboxing keeps the child at AutoProver's **own +uid**, so the two out-of-band secret vectors must be closed *explicitly* — and are: `/proc// +environ` by **not granting `/proc`** in the Landlock ruleset (proven `EACCES`), and `ptrace`/ +`process_vm_readv` by the **seccomp deny-list** (proven `EPERM`). These are the only same-uid vectors +to AutoProver's memory/env; both verified closed in the stock container. + +### The launcher: a custom shim over audited crates (not hand-rolled primitives) + +The first `SandboxProvider` (§4) is a small **trusted Rust launcher** (`run-confined`) that applies the +four confinements to itself, then `execve`s the command. It does **not** hand-write raw seccomp BPF +or raw Landlock syscalls — it composes two mature, permissively-licensed crates: + +- **[`landlock`](https://crates.io/crates/landlock)** — the reference Rust binding; does ABI + negotiation and the full FS access-right set (the fiddly part §11 Q1 warns about). +- **[`seccompiler`](https://crates.io/crates/seccompiler)** — the seccomp-BPF compiler from **AWS + Firecracker**; we hand it a small allow/deny policy, not raw bytecode. + +plus `setrlimit` and an env allowlist. So the security-sensitive primitives are audited upstream; +our code is the glue + the policy. We build Rust already, so this adds no new toolchain. + +### Alternatives considered — and why the seam stays swappable (§4) + +Two off-the-shelf tools do essentially this model. Neither is adopted *now*, but the `SandboxProvider` +seam means either can be dropped in later as a new provider mapping the same `SandboxPolicy`: + +- **[`landrun`](https://github.com/zouuup/landrun)** (Go CLI, **MIT**, mature ~2.2k★, FS floor 5.13): + excellent for Landlock FS + env, and the reference for our CLI shape. But it blocks network via + **Landlock network rules (TCP-only, kernel ≥6.7)** — it does **not** block UDP/DNS, and degrades + fail-open on older kernels — and has no rlimits. It would need a seccomp companion anyway, so it + doesn't save the hard part. +- **[`sandlock`](https://github.com/multikernel/sandlock)** (Python+Rust, Landlock+seccomp): the + closest match to our full model, but requires **kernel ≥6.12 (Landlock ABI v6)** — above Amazon + Linux 2023's 6.1 — and ships an **unstated license** plus more surface than we need (MITM proxy, + COW, notification supervisor). A strong candidate to revisit *if* the kernel-floor and license + questions are resolved and reviewers prefer an off-the-shelf boundary. + +The custom launcher wins for now on **kernel floor** (5.13, because we block network with seccomp not +Landlock), **license clarity**, and **minimal surface** — while the provider seam keeps the door open +to swap in `sandlock`/`landrun` with no change to the policy or the gate. + +### The chief advantage: deployment-independence + +Because it needs nothing from the container, the same code path runs identically on a dev laptop, +self-managed EC2, ECS, EKS, and even Fargate, and under `runc` or gVisor alike. **It decouples Phase +6 from the open deployment/tenancy questions** — those can be settled later as an *infrastructure* +hardening decision (VM-per-run / gVisor / IMDSv2 hop-limit / least-privilege IAM), layered *on top* +of this in-process boundary, not blocking it. + +**Residual risk:** a Landlock/seccomp bypass or a kernel LPE would let the child reach the container +(and then only as far as the infrastructure boundary allows — the container, or on EC2 the Nitro +VM). Named; mitigated by keeping the kernel patched, by the env/network already being denied, and by +the orthogonal infra hardening above for higher-trust-risk deployments. + +--- + +## 7. Resource limits, and the config surface + +**Resource caps** are `setrlimit` calls the launcher makes on itself before `execve` (lowering your +own limits is unprivileged; inherited by all descendants): `RLIMIT_AS` (address space / memory-ish), +`RLIMIT_CPU` (CPU-seconds — a wall-clock-independent bound), `RLIMIT_NPROC` (fork-bomb guard), +`RLIMIT_FSIZE` (disk-fill guard). `RLIMIT_AS` is crude (address space, not RSS) but dependency-free; +a **cgroup v2** scope (`memory.max`, `pids.max`, `cpu.max`) is the robust upgrade if the container +grants writable cgroup delegation — note it, defer it. The existing asyncio `wait_for(..., +timeout_s)` in `run_local_command` stays the primary wall-clock kill. + +The confinement *intent* is a **tool-agnostic** policy object (the same one every `SandboxProvider` +consumes, §4) — deliberately naming no mechanism, so a future provider swap needs no policy change: + +```python +@dataclass(frozen=True) +class SandboxPolicy: + rw_paths: tuple[Path, ...] # the workdir (+ any writable scratch) + ro_paths: tuple[Path, ...] # toolchains, crucible checkout, platform-tools, /usr… + env_allowlist: Mapping[str, str] # PATH, HOME, CARGO_HOME, RUSTUP_HOME, TERM, … + network: bool = False # egress allowed? default off + mem_bytes: int = ... + cpu_seconds: int = ... + nproc: int = ... + fsize_bytes: int = ... + # program + args come per-call from run_local_command +``` + +**Provider selection is separate config, not part of the policy** — a `CommandConfig.sandbox_provider` +knob (`"launcher"` = the custom Rust shim, default; `"none"` = passthrough; later `"landrun"` / +`"sandlock"`), overridable by env var. `run_local_command` gains `policy: SandboxPolicy | None` + +the resolved provider (default provider `"none"` when no policy, so existing callers and the EVM path +are unchanged). `RealEffects` builds the policy from a host-resolved config (toolchain paths +discovered like `resolve_crucible_repo` already does), and `build_program` uses the same. + +**Fail-closed.** Before running under a real sandbox provider, `provider.available()` is checked +(for the launcher: kernel Landlock ABI present). If it isn't — or the provider cannot apply its +confinement — the command **refuses to run** rather than silently executing unconfined. The failure +is a **prominent, actionable message** naming the reason ("the command sandbox requires a +Landlock-capable kernel (Linux ≥5.13); this backend cannot run without it — see +docs/command-sandbox.md §8"). The `none` provider is a *separate*, explicit, logged choice for the +trusted EVM/Foundry callers and trusted-input dev runs — never reached as a fallback from a failed +sandbox setup. + +--- + +## 8. Platform requirements — Linux with Landlock; nothing else supported + +Landlock and seccomp are **Linux** facilities. This backend is supported only on a Linux host with a +**Landlock-capable kernel (≥5.13; ≥6.7 adds Landlock network rules as defense-in-depth)** — which +AutoProver's own container already provides (Amazon Linux 2023 = 6.1, recent Ubuntu, and the dev +container all qualify). **macOS is not a supported configuration** (team decision): there is no +Landlock, and no macOS-native equivalent is planned. A Mac developer runs this backend the way +AutoProver already runs — inside the Linux container. + +If the sandbox cannot be established (non-Linux host, or a kernel without Landlock), the run +**fails immediately** with the §7 fail-closed message. This is the one uniform response everywhere +the sandbox is unavailable: refuse to run, loudly, rather than run untrusted native code unconfined. + +--- + +## 9. Implementation plan + +1. **The `SandboxProvider` seam + `SandboxPolicy`** — *done* ([composer/sandbox/policy.py](../composer/sandbox/policy.py)): + the tool-agnostic policy (§7), the `SandboxProvider` protocol (`wrap` → `LaunchSpec`, `available`), + the `none` passthrough provider, the name registry, and `ensure_available` / `SandboxUnavailable`. + Pure, unit-tested. **This is the isolation layer that makes the mechanism swappable** — everything + else depends only on this interface, never on a concrete tool. Lives in the backend-agnostic + [`composer/sandbox`](../composer/sandbox/) package (with `run_local_command`), not under `rustapp`. +2. **The custom launcher provider** — *done*: the `run-confined` **trusted Rust binary** + ([rust/run-confined](../rust/run-confined)) + the `LauncherProvider` + ([composer/sandbox/launcher.py](../composer/sandbox/launcher.py)) that maps a + `SandboxPolicy` to its argv. `run-confined --ro … --rw … --allow-env NAME[=VAL]… + --rlimit-* … [--allow-network] -- ` sets rlimits + `NO_NEW_PRIVS`, builds the + Landlock ruleset (best-effort ABI negotiation, full FS bit set, deny-by-default + §3 grants) via + the [`landlock`](https://crates.io/crates/landlock) crate, builds the seccomp filter (deny inet + sockets + ptrace/process_vm_*) via [`seccompiler`](https://crates.io/crates/seccompiler), applies + both, then `execve`s the command with an env scrubbed to the allowlist. `--probe` reports the + kernel Landlock ABI and drives `available()` → fail-closed (§7). Enforcement smoke-tested on the + host (write-outside / `/etc/passwd` / `/proc//environ` / inet-socket all denied; workdir + write, AF_UNIX, and toolchain `exec` allowed); argv mapping golden-tested. Full escape gate is + step 5. +3. **Thread `policy` + provider through `run_local_command`** — *done*: the runner accepts + `provider`/`policy` (default `None` → the `none` passthrough, byte-for-byte today's behavior) and + is fail-closed via `ensure_available`. A `SandboxConfig` ([composer/sandbox/config.py](../composer/sandbox/config.py)) + selects the provider (`$COMPOSER_SANDBOX_PROVIDER`, default `none`) and builds the policy via the + `rust_build_policy` recipe ([composer/sandbox/recipes.py](../composer/sandbox/recipes.py) — the + workdir and `/dev` nodes rw; discovered rust/cargo/platform-tool and system dirs ro, incl. `/etc` + for NSS; env allowlist; network off). Threaded through `RealEffects` and `RustBackend`/`RustFormalizer` + ([composer/rustapp/adapter.py](../composer/rustapp/adapter.py)), `build_program` + ([composer/spec/solana/build.py](../composer/spec/solana/build.py)), and the Crucible pipeline + (which adds the crucible checkout + binary to `extra_ro`). Integration-tested: `run_local_command` + under the launcher denies out-of-workdir reads and network while allowing the workdir + toolchain. +4. **Offline prep (§5)** — *done*: `warm_cargo_cache` (a `cargo fetch` run outside the sandbox, + network on) warms the registry, and the policy sets `CARGO_NET_OFFLINE=1` so the confined build — + and the nested cargo `crucible run` spawns — run offline. Wired into `build_program`; the + harness-dir warm is `CrucibleArtifactStore.warm_dependencies`, called from `prepare_formalization` + after the manifest is placed when a sandbox is on. `CARGO_HOME` is granted rw (the crucible policy) + so cargo can extract crate sources offline. +5. **The escape-test gate (§10)** — *done*, and **Crucible's default provider is now `launcher`** + (`_crucible_sandbox`; override with `COMPOSER_SANDBOX_PROVIDER=none`). Validated: + - **Part A (escape suite) — green** ([tests/test_sandbox_escape.py](../tests/test_sandbox_escape.py)): + a `rustc`-compiled malicious program run through the real launcher has every vector *denied* + (secret env, `/proc//environ`, host file outside the workdir, external TCP, and + `169.254.169.254`), with an unconfined control confirming the leaks would otherwise happen. + - **Part B — green**: a real `cargo-build-sbf` of `solana_vault` under the launcher (offline, + confined) produces the `.so` ([tests/test_crucible_sandbox_gate.py](../tests/test_crucible_sandbox_gate.py) + — this caught the relative-policy-path bug; grants must be absolute), and a real + `crucible run --dry-run` under the launcher builds the harness *offline* and runs LiteSVM + (`Harness validation passed!`). + - **Full LLM vertical — green**: the e2e gate (`tests/test_crucible_e2e_gate.py`) passes under the + launcher (`COMPOSER_SANDBOX_PROVIDER=launcher`): analysis → 23 properties → shared fixture + authored → per-instruction harness build + fuzz, all confined + offline, with **all three + instructions (initialize / deposit / withdraw) delivered with fuzz verdicts** (`BAD` — + counterexamples found). Getting here required the `/tmp` fix below and the shared-crate + concurrency fix (§11 item 8); before the latter, `initialize` was dropped to a `Cargo.toml` + feature race. + + **Root cause found via the gate:** every fresh harness build initially failed at the *link* step — + `Cannot create temporary file in /tmp/: Permission denied` (the linker's `$TMPDIR` scratch, which + the policy didn't grant). A link failure reads as "could not compile", so the LLM kept rewriting a + fine fixture. Fixed by redirecting `TMPDIR` to a private `/.sandbox_tmp` (§3) rather than + granting the shared `/tmp`. The `RunCommand` failure logging added alongside the authoring + improvements is what surfaced it. + +Each step is behind the seam, so the earlier Phase 1–5 gates keep passing. **Prerequisite of the +flip:** `run-confined` must be on PATH (built into the Docker image; `cargo build -p run-confined +--release` in dev) — otherwise Crucible fail-closes (§7/§8). A later off-the-shelf swap +(`landrun`/`sandlock`) is *only* a new step-2-style provider — the seam, policy, and gate are +untouched. + +--- + +## 10. The gate — an escape test + +A new expensive gate (`tests/test_crucible_sandbox_gate.py`) with two halves: + +**A. Escape attempts are denied.** Author a *malicious* harness whose `setup()` (and a companion +`build.rs`) attempts, and record the outcome of each: + +- **Read a planted secret** — set a fake `ANTHROPIC_API_KEY=canary-` in AutoProver's env + before the run; the harness (a) reads `std::env::var("ANTHROPIC_API_KEY")` and (b) tries to read + it out-of-band from `/proc//environ`, writing whatever it finds to a workdir file. Assert + the value is **absent** by both routes. +- **Read a host file outside the workdir** — the harness tries to open `/etc/passwd` and a file in + the real project root *outside* the granted workdir. Assert **permission denied**. +- **Reach the network** — the harness tries to connect to an external host **and** to + `http://169.254.169.254/latest/meta-data/` (the EC2 metadata endpoint → IAM creds). Assert **both + fail** (socket creation denied). + +The harness must not be able to fail the assertions silently — it writes each probe's result into +the workdir (allowed) and the test reads them back, asserting every probe reports *denied*. + +**B. The legitimate path still works.** The existing `solana_vault` gate ([§8](./crucible-application.md#L545)) +passes **unchanged** under the launcher provider — the shared fixture is authored, the `.so` builds, +tests compile and fuzz, verdicts are produced. This proves the sandbox grants exactly the toolchain +the real work needs and nothing more. + +Because the gate is written against the `SandboxProvider` seam (§4), not a specific tool, it doubles +as the **conformance test any future provider must pass** — swapping in `landrun`/`sandlock` means +re-running this same gate green, nothing more. + +Only when both halves are green may the backend run on untrusted input (the §9 definition of done). + +--- + +## 11. Open questions + +1. **Landlock ABI coverage / negotiation.** The launcher must handle the full FS access-right set of + the *running* kernel's ABI (unhandled rights stay unrestricted) with best-effort fallback on older + kernels. The `landlock` crate does this; confirm the minimum supported ABI on our target AMIs and + what "best-effort" degrades to (e.g. pre-ABI-3 has no `TRUNCATE` handling). +2. **AF_UNIX / netlink allowance.** The seccomp filter denies `AF_INET`/`AF_INET6` but allows + `AF_UNIX`. Confirm the toolchain (cargo jobserver, rustc, linker) needs nothing more; if a + benign `AF_NETLINK` use surfaces, decide whether to allow it (it can read but not egress). +3. **rlimits vs cgroup v2 (§7).** Is `RLIMIT_AS` enough to contain a memory-hungry fuzzer, or do we + need cgroup `memory.max` (and thus writable cgroup delegation in the container) sooner? +4. **Cache warming cost (§5).** Per-run `cargo fetch` adds latency; is a shared, pre-warmed + read-only registry volume worth it for CI throughput? +5. **Per-run `CARGO_HOME` — done.** An offline `cargo build` *writes* to `CARGO_HOME` (extracts crate + sources, takes locks), and that build runs untrusted `build.rs`/proc-macro code — so a writable + *shared* `~/.cargo` was a cross-run poisoning surface (overwrite an extracted `registry/src` to + hit a later run). Fixed: `rust_build_policy` points `CARGO_HOME` at a **private per-run dir under + the workdir** (`sandbox_cargo_home` → `/.sandbox_cargo`), the warm step (`warm_cargo_cache`, + unsandboxed) fetches *into that same home*, and the shared `~/.cargo` is granted read-only (for the + `cargo` binary) — so untrusted writes touch only the run's throwaway cache. Validated: a fresh + fetch into an empty private home + a confined offline build succeed. **Remaining cost:** deps are + re-fetched per run (no shared writable cache); a shared *read-only* index/cache to avoid the + re-download is the deferred optimization. +6. **Off-the-shelf provider swap (deferred, seam is ready — §4/§6).** `sandlock` (needs kernel + ≥6.12; unstated license) or `landrun` (+ a seccomp companion for UDP/DNS + rlimits) could replace + the custom launcher as a new `SandboxProvider` if reviewers prefer an off-the-shelf boundary. Blocked + today on the kernel-floor (target AMI ≥6.12?) and license questions; revisit once those resolve. + The provider seam + the gate-as-conformance-test (§10) make the swap mechanical. +7. **Infra-layer hardening (orthogonal, non-blocking).** Independent of this in-process boundary, + deployments running genuinely untrusted programs should also apply the standard EC2 hardening — + least-privilege instance IAM role, IMDSv2 with hop limit 1, egress-restricted security group, and + (if desired) VM-per-run or a gVisor runtime. Decide per deployment when the tenancy model is + settled; none of it blocks Phase 6. +8. **Shared-`Cargo.toml`/`main.rs` race (crucible backend) — fixed.** The per-component sessions + share one `fuzz//` crate; concurrent runs raced on both files (the observed + "package does not contain this feature: `c_`" that dropped `initialize`, and a latent + `main.rs` clobber). Fixed two ways: `prepare_component` now reserves Cargo features + **cumulatively** (the manifest only grows, so no feature is lost), and per-component command runs + are **serialized + atomic** (`run_local_command` materializes files and runs as one unit under a + `Semaphore(1)` shared by `RustFormalizer`), while the LLM authoring turns still run concurrently. + The remaining parallelism win — concurrent *builds/fuzzing* — needs a crate-per-component (§10 Q1); + deferred. diff --git a/docs/crucible-application.md b/docs/crucible-application.md new file mode 100644 index 00000000..7a0b3c26 --- /dev/null +++ b/docs/crucible-application.md @@ -0,0 +1,744 @@ +# Proposal — A Solana Verification Application (Crucible backend) + +> A plan to stand up a new AutoProver **application** that authors properties for **Solana** +> programs and checks them with **[Crucible](https://github.com/asymmetric-research/crucible)**, +> a coverage-guided fuzzer for Solana. The application pairs the **`solana` ecosystem** (the +> analysis/extraction front half, already built — [ecosystem-abstraction.md](./ecosystem-abstraction.md) +> §8.1, Phase 4) with a **new Crucible backend**, implemented as a **Rust application** on the +> PyO3 framework ([rust-applications.md](./rust-applications.md) · +> [rust-formalization-backends.md](./rust-formalization-backends.md)). +> +> Companion to [application-abstraction.md](./application-abstraction.md) (the five pieces of an +> application) and [formalization-abstraction.md](./formalization-abstraction.md) (the backend +> seam). The **Foundry** application is the closest existing analog and the running reference +> throughout: like Foundry, Crucible authors a source-language artifact, gates it with an +> external local CLI, and produces *refutation-oriented* verdicts. **Status: proposal / for +> review.** + +--- + +## 1. What we are building, in one paragraph + +AutoProver already has two orthogonal axes ([ecosystem-abstraction.md §2](./ecosystem-abstraction.md)): +the **ecosystem** (front half — how we model and reason about a domain) and the **backend** (back +half — how a property becomes a checked artifact). The `solana` ecosystem front half exists and is +gated: on a real Anchor program it analyzes the program into instructions and extracts sane, +Solana-native properties, today terminating in a `NullSolanaBackend` that only records them. **This +project replaces that null backend with a real one: a Crucible backend that turns each extracted +property into a Crucible fuzz harness, runs the fuzzer, and reports pass/refuted.** Per the design +decision in [rust-applications.md](./rust-applications.md), the backend is written **in Rust** as a +PyO3 wheel implementing the `autoprover-sdk` traits, consumed by the generic Python host — so the +new AutoProver application, `crucible`, is *`ecosystem="solana"` + a Crucible backend wheel* and +needs (ideally) zero bespoke Python. + +``` + ┌──────────── solana ecosystem (DONE) ───────────┐ ┌──── Crucible backend (NEW) ────┐ +program ─analyze─▶ SolanaApplication ─extract─▶ properties ─formalize─▶ fuzz harness ─verdicts─▶ report + (SolanaProgram / SolanaInstruction, (signer/owner, (a Crucible crate: fixture + (crash → BAD, + accounts, PDAs, CPI — model.py) PDA, overflow…) actions + invariant tests) clean → GOOD*) + run: `crucible run … --timeout` + *GOOD = no violation found within the fuzzing budget (bounded, not a proof) +``` + +--- + +## 2. What already exists (and is reused unchanged) + +| Piece | Where | Status | +|---|---|---| +| `solana` ecosystem: `SolanaApplication` model (programs / instructions / account constraints / CPI / authorities) | [composer/spec/solana/model.py](../composer/spec/solana/model.py) | **Done** (Phase 4) | +| `RUST` language facet (Cargo `forbidden_read`, Rust `code_explorer` prompt, `rust/_failure_modes.j2`) + `SOLANA` chain (validate / `locate_main` / `units`) | [composer/pipeline/ecosystem.py](../composer/pipeline/ecosystem.py) · `composer/templates/{rust,solana}/…` | **Done** (Phase 4) | +| Solana analysis + property-extraction prompts | `composer/templates/solana/…` | **Done** (Phase 4) | +| The Rust-application framework: `AppDescriptor`, `Application`/`FormalizeSession` traits, the IoC `Command`/`Observation` ABI, `export_app!` | [rust/autoprover-sdk/src/lib.rs](../rust/autoprover-sdk/src/lib.rs) | **Done** | +| The generic Python host: enum/argparse/entry-point/frontend synthesis, `resolve_ecosystem`, the IoC effect loop, the `RustBackend` adapter | [composer/rustapp/](../composer/rustapp/) (`host.py`, `entry.py`, `loop.py`, `adapter.py`, `descriptor.py`) | **Done** | +| Ecosystem selection by descriptor (`AppDescriptor.ecosystem`, registry lookup) | [rust/autoprover-sdk/src/lib.rs](../rust/autoprover-sdk/src/lib.rs) · [composer/rustapp/host.py](../composer/rustapp/host.py) | **Done** (Phase 3) | +| A reusable null backend + Anchor `solana_vault` scenario + live gate | [composer/spec/solana/null_backend.py](../composer/spec/solana/null_backend.py) · [test_scenarios/solana_vault/](../test_scenarios/solana_vault/) · [tests/test_solana_gate.py](../tests/test_solana_gate.py) | **Done** | + +The net: **the entire front half and the entire Rust-app shell are already built and gated.** This +project is squarely a *backend* effort (formalization-abstraction.md §9's checklist), plus the +Crucible-specific infrastructure that a fuzzing backend needs but the prover/Foundry backends did +not. That new infrastructure — §7 — is the real content of the plan. + +--- + +## 3. What Crucible is (and why Foundry is the right mental model) + +Crucible is a **coverage-guided fuzzing framework for Solana programs** (LibAFL + LiteSVM). You +declare a program's actions, write invariants, and the fuzzer searches randomly generated action +sequences for violations. The relevant facts for backend design: + +- **The artifact is a Rust *fuzz-harness crate*, not a spec file.** A harness (`fuzz//`) + is a standalone Cargo workspace with: + - `src/main.rs` — a `#[derive(Clone)]` **fixture** with a `setup()` (loads the program `.so`, + creates accounts, runs init instructions in dependency order), `action_*` methods (one per + instruction, with `#[range(..)]`-constrained fuzz params), an optional `after_action` hook, and + one or more **tests**: `#[invariant_test]` fns (stateful, checked after every action) and/or + `#[crucible_fuzz]` fns (single-operation, random inputs). Invariants use `fuzz_assert_*!` macros + (which record a violation instead of aborting the process). + - `Cargo.toml` — a `[[bin]]` plus a **feature per test** (`crucible run ` requires the + feature name to equal the test name). + - `idls/.json` — the program IDL, from which `crucible-idl-gen` generates typed + `instruction`/`accounts`/`state` bindings at compile time (`raw_call()` is the fallback when no + IDL exists). +- **It is invoked as a local CLI, and verdicts come off its output** (the user's chosen model — like + `forge test`): `crucible run --release --timeout `. Structured stdout: + `[FUZZ_PULSE]` progress, `[FUZZ_FINDING] reproduces:true summary:` on a violation (a crash + file + `.meta.json` action sequence written alongside), `[FUZZ_ERROR]` on fatal setup error. + `--dry-run` compiles + runs one iteration to validate the harness. There is **no run service and + no run link** — pass/fail is local. +- **Verdicts are refutation-oriented, exactly like Foundry.** A crash *refutes* a property (BAD); a + clean run to the timeout means *no violation found within the budget* (GOOD\*, bounded — not a + proof). So the Crucible `backend_guidance` is nearly Foundry's: write universals freely, their + refutations are valuable, but skip properties a fuzzer can't meaningfully sample (off-chain + events, pure hash-collision resistance). + +Side-by-side, the three backends: + +| | prover (CVL) | **Foundry** | **Crucible (new)** | +|---|---|---|---| +| Artifact | `.spec` (+ `.conf`) | `.t.sol` | a **fuzz-harness crate** (`main.rs` + `Cargo.toml` feature + `idls/`) | +| Gate | Certora Prover (cloud, run link) | `forge test` (local) | `crucible run --timeout` (local) | +| Verdict source | prover output service | local exit/parsed output | parsed `[FUZZ_FINDING]` / crash dir | +| Verdict meaning | proof / CEX | bounded refutation | bounded refutation | +| Prep pre-work | AutoSetup ∥ summaries ∥ invariants | none (identity) | **build program `.so` + IDL + author the shared fixture/actions** | +| Deliverable granularity | one `.spec` per component | one `.t.sol` per component | **one *test* (feature+fn) per component, in one shared crate** | + +The last row is the crux and the source of most new infrastructure (§7.1): unlike the prover and +Foundry, a Crucible deliverable is **one multi-file crate shared across all components**, not one +file per component. + +--- + +## 4. The application: `crucible` + +Following the naming of `foundry` (an app named for its tool), the application is **`crucible`**: +`ecosystem="solana"` + the Crucible backend wheel. Mapped onto +[application-abstraction.md](./application-abstraction.md)'s five pieces, everything but the backend +comes free from the Rust-app host: + +| Piece | Source | Crucible specifics | +|---|---|---| +| Phase enum `P` | host, synthesized from `descriptor.phases` | `Analysis → Extraction → BuildHarness(setup) → Formalization → Report` (Build/setup is a UI-only phase, cf. autoprove's harness phase) | +| Entry point / Executor | host (`_generic_entry_point`) | positional `project_root main_program system_doc`; declared args: `--crucible-binary`, `--fuzz-timeout`, `--fuzz-cores`, `--stateful`, `--max-actions`; `validate_preconditions` checks the toolchain + a Cargo/Anchor project (§6) | +| Pipeline wrapper | host (`run_rust_pipeline`) | passes `ecosystem=SOLANA` (resolved from `descriptor.ecosystem="solana"`) | +| **Backend** | **Rust wheel (new)** | **the whole of §5** | +| Frontend | host (`GenericRustApp` / console) | `event_kinds`: `fuzz_pulse` (coverage/exec-rate), `fuzz_finding` (crash), `build_output` (cargo/build-sbf) | +| Artifact store | host shell + Rust formatter | **needs the multi-file-crate extension, §7.1** | +| `main()` | host | unchanged | + +So the deliverables of *this* project are: the Rust wheel (§5), the ABI/host extensions a fuzzing +backend forces (§7), the toolchain/preconditions (§6), and a scenario + gate (§8). + +--- + +## 5. The Crucible backend, method by method + +The backend is a Rust `Application` (in a new crate, e.g. `rust/crucible-app/`) that decides via a +`FormalizeSession`. The three phase objects of the formalization abstraction +([formalization-abstraction.md §2](./formalization-abstraction.md)) map as follows. **The governing +idea mirrors the CVL backend's structural-invariant pattern**: author the expensive, program-wide +scaffold *once* in `prepare_formalization`, then have each per-component `formalize` contribute only +its own test — just as CVL builds `invariants.spec` once and each per-component spec `import`s it. + +### 5.1 `prepare_system` — build the program + IDL + fixture skeleton + +Roughly Foundry's identity transform, but with real pre-work because a Crucible harness must +*compile against a built program*: + +1. `locate_main` (from the `SOLANA` chain) picks the target `SolanaProgramInstance`. +2. Build the program to sBPF: `cargo build-sbf` (or `anchor build`) → `target/deploy/.so`. +3. Generate/collect the IDL: `anchor idl build` (or convert an existing one) → + `fuzz//idls/.json`; the harness uses `crucible_idl_gen::declare_fuzz_program!`. + +This is naturally a **UI-only "BuildHarness" phase** with its own `TaskInfo`. Its outputs (the `.so` +path, the IDL, the program id) flow into the next phase as immutable state. + +> **Build the Solana build step as shared, reusable infrastructure — not Crucible-specific.** Steps +> 2–3 (`source → .so + IDL`, version-aware per §6.1) are needed by *every* Solana backend, and a +> future Certora-Prover-style Solana backend will go further and **munge the source and rebuild** it +> (harness lift, mocks, `cvlr` hooks) — the exact analog of the EVM CVL backend's `prepare_system` +> harness-lift ([formalization-abstraction.md §4.1](./formalization-abstraction.md)). So factor a +> reusable "Solana build" capability — `source → [optional munge] → .so + IDL` — that Crucible calls +> in its *no-munge* mode and a Prover backend calls in its *munge-and-rebuild* mode, mirroring how the +> EVM backends share solc/harness tooling. A **user-supplied prebuilt `.so`** is then just an optional +> fast-path for the no-munge (Crucible) case; it does not remove the pipeline, since other backends +> must rebuild, so it is a minor optimization at most. + +### 5.2 `prepare_formalization` — author the shared fixture + actions (once) + +The single most important step, and the biggest LLM authoring job. Using the `SolanaApplication` +model + source, author the **shared harness scaffold** that every per-component test reuses: + +- the `#[derive(Clone)] struct

Fixture { ctx, program_id, … }` and its `#[fuzz_fixture] setup()` + (init order, admin/authority whitelists, PDA seed encoding, token accounts — the *Harness Guide* + is essentially the agent's playbook here); +- one `action_*` per instruction (typed `ctx.program(..).call(..).accounts(..).signers(..).send()`, + or `raw_call` when no IDL), with `#[range(..)]` bounds inferred from the property set; +- an `after_action` hook if useful. + +Gate: `crucible run --dry-run` must succeed (harness compiles + `setup()` runs one +iteration). This is the "loud, fail-fast setup" the Harness Guide demands, surfaced as a hard +validation gate exactly like the CVL prover gate. The returned `Formalizer` carries the built `.so`, +the IDL, the compiled scaffold, and the run config as immutable state. + +> This is the direct analog of CVL's `prepare_formalization`: expensive, program-wide, run once, +> and *its output is a shared precondition for every component* — here the fixture/actions rather +> than `invariants.spec`. + +### 5.3 `formalize` — per component, author one Crucible test (the IoC loop) + +For each extracted component (a `SolanaInstructionInstance`, or property group — see §9 Q1), author +**one Crucible test** encoding that component's properties, sharing the scaffold from §5.2: + +- pick the test form per property kind: `#[invariant_test]` for conservation/solvency/consistency + invariants (checked after each action), `#[crucible_fuzz]` for single-instruction properties + (overflow, access-control-on-one-call); +- write the `fuzz_assert_*!` checks against **on-chain** state (`read_anchor_account`), not local + mirrors; +- register the test's feature in `Cargo.toml` (feature name = test name); +- gate with `crucible run --dry-run` (compiles + one iteration), then run a bounded + `crucible run --release --timeout [--stateful]`. + +The **verification loop is the Rust IoC decider** ([rust-formalization-backends.md](./rust-formalization-backends.md) +Tier 2): the `FormalizeSession` emits `CallLlm` to draft/revise the test, a "run Crucible" effect to +compile+fuzz, interprets the parsed result (compile error → revise; `[FUZZ_FINDING]` → the property +is *refuted*, record the minimized action sequence via `crucible tmin` and either publish-as-refuted +or, if the harness itself is wrong, revise; clean timeout → publish-as-held), and terminates with +`Publish`/`GiveUp`. Python owns every effect; Rust only decides. The "run Crucible" step is a +**general run-a-command-over-files effect** (§7.2): crucially, the *Rust decider* fixes the command +line and the LLM only ever authors file *contents*. + +### 5.4 `fetch_verdicts` — refuted vs held, off the local result + +Like Foundry's `_foundry_verdicts` (read pass/fail straight off the stored result, no run service): +each test's outcome is known at `formalize` time (a `[FUZZ_FINDING]` was emitted or not), so it is +baked into the published `Formalized` and `fetch_verdicts` maps it to `Verdict{ BAD (refuted, with +the crash's action sequence / line) | GOOD (no violation within budget) | ERROR (build failed) | +TIMEOUT }`. `run_link` is `None` (no run service). + +### 5.5 `finalize` — assemble/emit the buildable crate + +The one hook that sees all outcomes at once: ensure `Cargo.toml` lists every generated test's +feature and write the crash artifacts / a `{test → crash sequence}` map under the metadata dir. If +the per-component artifact model can't produce a single compilable `main.rs` incrementally, `finalize` +is where the shared scaffold + all per-component test fns are stitched into the final crate (§7.1). + +--- + +## 6. Toolchain & preconditions + +A fuzzing backend has heavier local prerequisites than the prover (which offloads to the cloud) or +Foundry (just `forge`). `validate_preconditions` (a **sync** Rust hook, run before any service opens +— [rust-applications.md §4.2](./rust-applications.md)) should check, with actionable error messages: + +- **`crucible`** on `PATH` (or `--crucible-binary`); `crucible --version`. +- **The Solana/Anchor build toolchain**: `cargo build-sbf` (Solana CLI) and/or `anchor`, plus a + `rust-toolchain.toml` compatible with the target program (the examples pin one). +- **A buildable target program**: a Cargo/Anchor workspace with the program under `programs//` + (mirror Foundry's `foundry.toml`-exists precondition). +- **Version skew** between the program's Solana deps and Crucible's — the docs explicitly call this + out and offer `crucible-idl-gen` (IDL → types without a crate dep) as the escape hatch. The backend + should prefer the standalone-IDL path to stay robust across program toolchains. + +These are environment facts the gate (§8) must provision, analogous to the prover's `solc`/AutoSetup +and Foundry's `forge`. + +### 6.1 Version compatibility (Crucible / Solana-Anchor / Rust) + +Unlike `forge` (one self-contained binary) or the cloud prover (server-pinned), a Crucible run is a +**version matrix**, and getting it wrong shows up as a compile error deep in the fuzz phase. This is +the Rust/Solana analog of the `solc`-version pinning the autoprove pipeline already needs (the Counter +scenario's `pragma ^0.8.29` vs a 0.8.21 default — [ecosystem-abstraction.md §10](./ecosystem-abstraction.md)), +but with more moving parts. Three axes: + +- **A Crucible version pins a whole stack.** The harness depends on `crucible-fuzzer` / + `crucible-test-context` / `crucible-idl-gen`, and a given Crucible release pins `litesvm`, + `anchor-lang`, `solana-*`, and `solana-sbpf` (in Crucible's workspace today: `litesvm = 0.9`, + `anchor-lang = 1.0.1`, `solana-* = 3.0`, `solana-sbpf = 0.13`). The generated harness must depend on + the *matching* set or it won't build. +- **The target program's Solana version may differ from the fuzzer's.** Crucible's own docs call this + out and provide the decoupling: `crucible-idl-gen`'s `declare_fuzz_program!` generates typed bindings + **from the IDL, with no crate dependency on the program**, so the harness need not compile against the + program's Solana version. Prefer this path always. (Orthogonal axis: `litesvm` must still be able to + *load and execute* the program's compiled `.so` — i.e. loader/sBPF compatibility between the + program's build and the fuzzer's `litesvm`.) +- **Two Rust toolchains, not one.** The **harness** is a *native* build — Crucible forces + `RUSTUP_TOOLCHAIN=stable` ([try_cargo_build, lib.rs:2031](../../crucible/crates/crucible-fuzz-cli/src/lib.rs#L2031)) + — so it needs a host `stable` recent enough for that Crucible version's MSRV. The **program `.so`** is + an *sBPF* build via `cargo build-sbf` / `anchor build`, which uses Solana's **platform-tools** bundled + Rust (pinned per Solana/Anchor version, driven by the project's `rust-toolchain.toml` / `Anchor.toml`), + not host stable. Both must be present and mutually compatible. + +What this means for the backend: + +1. **Make the version explicit and selectable**, not "whatever's on `PATH`": a `--crucible-version` + (release tag / git ref) plus detection of the program's Solana/Anchor version from its + `Cargo.toml` / `Anchor.toml` / `rust-toolchain.toml`. `validate_preconditions` resolves these to a + concrete, compatible combination up front and fails fast (with the required versions) on a mismatch + or a missing toolchain. +2. **The backend owns the pin.** Because the backend (not the LLM) authors `Cargo.toml` (§7.2/§7.4), it + generates the harness manifest from a small **compatibility table** — `Crucible version → { crucible + crate refs, litesvm, anchor, solana-*, min host rustc }` — so version selection is one trusted lookup, + not something the LLM can perturb. +3. **Support a *curated set*, via per-version sandbox images.** §7.4 already requires an offline, + vendored build inside the sandbox; make the sandbox image the unit of version support — one immutable + image per supported `(Crucible × Solana/Anchor)` combo, each carrying the matching `crucible` binary, + the host `stable` toolchain, the Solana platform-tools, and the vendored crate set. "Support a new + version" then means "add a vetted image to the matrix," which also bounds the combinatorics — we + support a known list, not arbitrary versions. +4. **Record the resolved versions** in the deliverable (the generated `Cargo.toml` pins are part of it) + and **fold them into the formalize cache key**, so a result built against Crucible vX / Solana vY is + not silently reused when a different combination is selected (cf. the CVL backend threading `config` + through its result for reproducibility, [formalization-abstraction.md §5](./formalization-abstraction.md)). + +--- + +## 7. New infrastructure a fuzzing backend forces + +The prover and Foundry backends fit the existing seam cleanly. Crucible stresses five assumptions +that were previously EVM/prover/Foundry-shaped. These are the genuinely new build items. + +### 7.1 Multi-file, one-crate-shared-across-components deliverable + +**The problem.** `ArtifactStore` and `AppDescriptor.artifact_layout` assume *one deliverable file per +component*: `_.` ([formalization-abstraction.md §6](./formalization-abstraction.md); +the layout fields in [autoprover-sdk](../rust/autoprover-sdk/src/lib.rs) are `artifact_prefix` + +`artifact_extension`). A Crucible deliverable is **one Cargo crate** whose `main.rs` holds a *shared* +fixture/actions plus *one test fn per component*, with a *per-component feature* in a shared +`Cargo.toml`. There is no clean "one file per component." + +**A constraint that decides it (found while building phase 1).** Crucible's CLI hardcodes the harness +binary name: `crucible run` builds `cargo build --features ` and then runs the single +`target//invariant_test` binary (`find_fuzz_binary` in +[crucible-fuzz-cli/src/lib.rs](../../crucible/crates/crucible-fuzz-cli/src/lib.rs)). So the tempting +"one `[[bin]]` per component" layout is a **dead end** — `crucible run` would never execute those +bins. A component must instead be selected by a **Cargo feature** that gates its test inside the one +`invariant_test` binary. That is exactly how Crucible's own examples work (`escrow` = one `[[bin]]`, +one `#[invariant_test]`, one feature). + +**The model, therefore.** One crate per program, one `[[bin]] invariant_test`, and **per component: a +Cargo feature whose name is the component's test fn**, all sharing the fixture/actions: + +```rust +fuzz// + Cargo.toml // [[bin]] invariant_test; [features] one per component (c_ = []) + src/main.rs // shared #[fuzz_fixture] + action_* ; then, per component, verbatim: + // #[invariant_test] fn c_(fx) { … } (NOT user-#[cfg]-wrapped) +``` + +The gating is done *by Crucible's macros*, not by us: `#[invariant_test] fn ` expands to a +`main()` behind `#[cfg(feature = "")]` (verified in +[crucible-invariant-macro/src/lib.rs](../../crucible/crates/crucible-invariant-macro/src/lib.rs)). So +the load-bearing rule is **the test fn's name equals its Cargo feature** (`c_`); building +`--features c_` keeps exactly one `main()`, and `crucible run c_` runs it. The +store therefore emits each section *verbatim* and only declares the features — wrapping a section in +its own `#[cfg]` (or a fn-name/feature mismatch) desyncs the macro's gate and the crate loses its +`main` (a mistake the phase-2 gate caught). The fn must live at crate root in `main.rs`, not a +submodule, since the macro generates the entrypoint there. + +**Implication for the store.** This is a **Crucible-specific `ArtifactStore`** (not the generic +single-file `RustArtifactStore`): each component's `artifact_text` is its `#[invariant_test]` fn, +and the store **assembles** `src/main.rs` (shared fixture + all fns) + `Cargo.toml` (the feature +list) from the shared scaffold plus every component. The base `ArtifactStore` still writes the +per-component metadata (`properties.json`, `commentary.md`, the property→units map) under +`certora/crucible/`, so only the deliverable bundle is bespoke. The generic host keeps the +single-file store for backends that fit it (echoprover); the Crucible wheel opts into the crate store. + +### 7.2 A general "run a local command over a set of files" effect + +Today the IoC vocabulary ([autoprover-sdk](../rust/autoprover-sdk/src/lib.rs) · [loop.py](../composer/rustapp/loop.py)) +is prover-specific: `RunProver { spec: String }` + `RunFeedback`, a *single* spec string checked by +*the* verifier. That shape is wrong here for two independent reasons: + +1. **There is no single "verifier for an ecosystem."** We intend multiple backends per ecosystem + (Crucible is one Solana backend; others will follow), each driving its own tool(s) — `crucible`, + `cargo build-sbf`, `anchor idl`, and whatever a future backend needs. The effect must therefore be + **backend-agnostic**: *the Rust decider names the command*, rather than the framework hardcoding a + per-ecosystem prover. +2. **A harness is a multi-file crate,** not one `spec: String`. + +So replace the prover-specific pair with **one general effect** — materialize a set of files, run a +command over them, return the output — which the run-Crucible, build-`.so`, and IDL steps all reuse: + +```rust +// Command (Rust → Python) +RunCommand { + program: String, // e.g. "crucible" ── authored by the Rust decider + args: Vec, // e.g. ["run","vault","inv","--release","--timeout","60"] + files: BTreeMap, // workdir-relative path → contents (merged into the session workdir) +} +// Observation (Python → Rust) +CommandResult { exit_code: i32, stdout: String, stderr: String } +``` + +`RealEffects` gains a runner that writes `files` into a per-session sandbox workdir, executes +`program`+`args` there **bounded by a semaphore + timeout** (exactly Foundry's `_ForgeRunConfig` +discipline — critical because `crucible --cores` is greedy), tees stdout/stderr to the frontend +(the `fuzz_pulse`/`build_output` event kinds), and returns the result for the Rust decider to parse +(`[FUZZ_FINDING]`, cargo errors, …). It is additive — new `Command`/`Observation` variants + one new +`Effects` method with a default — so the prover/echoprover path is untouched, and any future +CLI-gated backend reuses it. + +#### The trust boundary: Rust owns argv, the LLM owns only file *contents* + +The parties differ in trust: the **driver** (Python) and the **backend wheel** (compiled Rust) are +trusted; the **LLM** is not. The invariant to enforce — and the reason `program`/`args` are separate +structured fields, never a shell string: + +> **The LLM never influences the command line. It authors only the *contents* of input files.** + +Concretely, the LLM's *only* output channel is `CallLlm` replies. The Rust decider parses those +replies into file **contents** and places them in `files` under **paths it chooses**; it constructs +`program`/`args` from its own compiled logic. The LLM has no path into `program`, `args`, or the file +*paths*. Python then enforces this defensively: + +- **Exec, not shell.** Run via `asyncio.create_subprocess_exec(program, *args)` — never + `create_subprocess_shell` / a shell string. File contents can't inject argv even in principle. +- **Path confinement.** Every `files` key must be a relative path that stays inside the session + workdir (reject `..` / absolute paths), so the LLM's contents can't land at `~/.bashrc` etc. +- **Program allowlist (optional, defense-in-depth).** The descriptor can declare the binaries a + backend is permitted to invoke, so even a buggy wheel can't launch an arbitrary program. + +One honest caveat this rule does **not** cover — and it is bigger than it looks. It is tempting to +think the SVM sandboxes the LLM's code, but it does not, and this was **verified against Crucible's +source**, not assumed: + +- The **harness** (fixture `setup()` + `action_*` + invariant fns — the LLM-authored part) is built + with plain `cargo build --release --features ` for the **host** target (`RUSTUP_TOOLCHAIN=stable`), + *not* `cargo build-sbf` — a **native binary** (`try_cargo_build`, + [crucible-fuzz-cli/src/lib.rs:2031](../../crucible/crates/crucible-fuzz-cli/src/lib.rs#L2031)). +- The CLI executes that binary **directly and unwrapped**: `Command::new(&binary_path).…status()` + ([lib.rs:531](../../crucible/crates/crucible-fuzz-cli/src/lib.rs#L531), + [:727](../../crucible/crates/crucible-fuzz-cli/src/lib.rs#L727)) — no launcher, no isolation. +- **LiteSVM is a linked-in library, not a boundary** (`litesvm = "0.9.0"`, a struct field + `TestContext { pub svm: LiteSVM, … }`, + [crucible-test-context/src/lib.rs:1326](../../crucible/crates/crucible-test-context/src/lib.rs#L1326)); + it is "a lightweight Solana VM that runs inside your tests" — [litesvm.com](https://www.litesvm.com/docs/getting-started)). + It interprets the **program-under-test's sBPF bytecode** in-process — sandboxing the user's `.so`, + not the native harness around it. +- A scan of the whole dependency tree for sandbox tech (seccomp / landlock / namespaces / nsjail / + bubblewrap / gvisor / wasmtime / rlimit / chroot / unshare / capabilities) found **none**. + +So the LLM-authored code runs **natively, with full process privileges**, at two points: **build +time** (`build.rs` / proc-macros during `cargo build`) and **run time** (`setup()`, every +`action_*`, the invariant fns — only the *instructions they submit* execute as sandboxed sBPF). The +argv boundary is therefore necessary but nowhere near sufficient — `crucible run` on LLM-authored +source is arbitrary native code execution regardless. The whole build+fuzz must run in a real sandbox +that **we** provide (Crucible provides none); how, is §7.4. + +### 7.3 Verdict semantics + report nouns for a fuzzer + +Reuse Foundry's refutation semantics wholesale, but the report should read in Solana/fuzzing nouns +("program"/"instruction"/"fuzz test"/"violation" vs "contract"/"rule"). This is +[ecosystem-abstraction.md §11 Q3](./ecosystem-abstraction.md) (report labels by ecosystem) surfacing +for real; a small `ecosystem`/`backend_tag`-driven noun map suffices. Minor. + +### 7.4 Sandboxing the untrusted build+fuzz + +Because the LLM-authored harness runs as native code (§7.2), each build+fuzz must execute in an +isolation boundary *we* provide — the outer AutoProver container protects the host *from* AutoProver, +but not AutoProver's own secrets/network *from* code running inside it. **The scope is every step that +compiles or runs untrusted Rust — not just the harness fuzz.** That includes the shared Solana build +(§5.1): `cargo build-sbf` on *user-supplied* program source (and, for a Prover-style backend, on +*LLM-munged* source) runs that source's `build.rs`/proc-macros natively too. So the sandbox wraps +`cargo build-sbf`, `cargo build` (harness), and `crucible run` alike. The sandbox must guarantee, at +minimum: + +- **No network** (blocks exfiltration and the cloud metadata endpoint alike); +- **A clean, secret-free environment** — none of `ANTHROPIC_API_KEY` / `CERTORA` / `AWS_*` / `PG*` + inherited; only benign build vars; +- **A minimal filesystem** — a scratch workdir (rw) holding only the step's inputs (the program + source crate for a build step; the harness crate + built `.so` + IDL for a fuzz step), nothing else + of the host; +- **Resource caps + a supervisor-enforced wall-clock kill** (CPU / memory / pids / disk); +- **An offline, vendored build** — the *backend* (not the LLM) owns `Cargo.toml` and forbids + `build.rs`, so the dependency set is fixed and pre-vendored, and `cargo build` runs with no + network (this also closes the build-time supply-chain vector). + +**Mechanism — decided direction (built last, in §9 Phase 6; required for done).** + +- **Unprivileged in-kernel self-sandboxing: [Landlock](https://docs.kernel.org/userspace-api/landlock.html) + (filesystem) + seccomp (network + `ptrace`) + env allowlist + rlimits.** The command restricts + *itself* via a small trusted launcher shim, the way Chrome/OpenSSH/systemd sandbox untrusted work — + needing **no namespaces, no capabilities, no custom runtime, and no container changes**. Step 1 of + Phase 6 established empirically that the namespace tools (bwrap/nsjail) *cannot* run under the + container's default seccomp/AppArmor without either weakening the whole container's LSMs (rejected) + or adopting a gVisor/Kata runtime (heavy on our compile+fuzz workload, and its host-kernel + protection is an infra concern already covered by the EC2 Nitro hypervisor). Landlock (ABI ≥ the + kernel's; we observed 8) grants rw to the workdir and ro+x to the toolchain paths and denies all + else — closing host-file reads *and* the same-uid `/proc//environ` leak; seccomp denies + inet-domain sockets (TCP/UDP/DNS/IMDS) and `ptrace`/`process_vm_readv`; the launcher `execve`s with + a scrubbed env. All four guarantees were verified in a **stock** container as a non-root user. Its + chief virtue: it is **deployment-independent** — identical on a laptop, self-managed EC2, ECS, EKS, + Fargate, and under `runc` or gVisor — so it does not couple this phase to any deployment/tenancy + decision. The launcher is a small Rust shim over audited crates (`landlock` + Firecracker's + `seccompiler`), and sits behind a **swappable `SandboxProvider` seam**: an off-the-shelf tool + (`landrun`, `sandlock`) can replace it later as a new provider mapping the same policy, with no + change to the seam or the escape-test gate. Full detail + the validation matrix: + [command-sandbox.md §6](./command-sandbox.md). +- **macOS: not supported (team decision).** Landlock/seccomp are Linux facilities; there is no macOS + equivalent planned. If the sandbox cannot be established (non-Linux, or a kernel without Landlock) + the run **fails immediately with a prominent message** rather than running untrusted native code + unconfined (see [command-sandbox.md §8](./command-sandbox.md)). A Mac developer runs this backend + the way AutoProver already runs — inside the Linux container. +- **Infra hardening is orthogonal (non-blocking).** VM-per-run / gVisor / IMDSv2-hop-limit / + least-privilege IAM are deployment-layer defenses that stack *on top* of the in-process boundary; + they are decided per deployment once the tenancy model settles, and do not block Phase 6. + +The sandbox sits entirely behind the `RunCommand` effect (§7.2): `RealEffects` launches `program`+`args` +in the sandbox instead of directly, so nothing in the Rust decider, the driver, or the ABI changes when +it lands. That seam is what lets it be **built last** (§9 Phase 6) without disturbing the earlier +phases — but it is **required, not optional**: the backend is not considered done until *every* +`RunCommand` invocation is sandboxed, and until then the backend may run only in a trusted, offline +environment on trusted input (the gate scenario, §8), never on an untrusted program. See the +definition of done in §9. + +### 7.5 Giving the LLM the Crucible documentation + +Authoring a Crucible harness demands framework-specific knowledge the base model won't reliably have: +the `TestContext` builder chain, the `#[fuzz_fixture]`/`action_*`/`#[invariant_test]` conventions, the +`fuzz_assert_*!` macros, `crucible-idl-gen` usage, and — above all — the *Harness Guide*'s blocker→fix +playbook (admin whitelists, PDA seed encoding, init order, the Anchor error-code tables). We need to +put that in front of the author. + +**Precedent.** Each backend already ships a domain knowledge base as a `ComposerRAGDB` (pgvector): +autoprove's CVL manual, Foundry's cheatcode DB. The pattern is a `populate__rag.sh` that +collects the source docs, chunks them by markdown header, embeds them (`DefaultEmbedder`), and ingests +into a dedicated Postgres DB ([scripts/populate_foundry_rag.sh](../scripts/populate_foundry_rag.sh)); +the DB is exposed as keyword + vector **search tools** ([foundry_rag.py](../composer/tools/foundry_rag.py)) +bound into the author's env, and selected per-app via `AppDescriptor.rag_db_default` / `--rag-db`. + +**A blocker specific to the Rust-app path.** The Foundry/CVL authors are agent loops with those RAG +tools bound. The Rust IoC `call_llm` effect is **not** — it is a single, tool-less turn +(`model.ainvoke([HumanMessage(content)])`, [adapter.py:75](../composer/rustapp/adapter.py#L75)). So the +Crucible author cannot *search* a RAG DB mid-loop until we change one of: + +- **(a) tool-enabled `call_llm`** — `RealEffects.call_llm` runs a bounded agent turn with the + ecosystem/backend `rag_tools` (+ source tools) bound, returning final text. Closest to how the + Foundry/CVL authors work, and a **general** rustapp-framework improvement (any Rust backend whose LLM + needs tools benefits), not Crucible-specific. +- **(b) a retrieval effect** — `SearchDocs { query } → Observation::Docs`, so the Rust decider + explicitly retrieves and injects results into its next prompt. More IoC-pure; keeps `call_llm` simple. +- **(c) Python pre-injects** the docs into the `call_llm` messages — no tool-calling at all. + +**Recommendation: design the knowledge *seam* for large corpora now; fill it cheaply for Crucible.** +Crucible's own docs are small, but a **Certora Prover / CVLR backend for Solana is on the roadmap**, +and it will carry a documentation set the size of today's CVL manual — far too large to inject. If we +ship Crucible on static injection and stop, we will have to retrofit tool-based RAG into the Rust-app +framework for CVLR. So make the *framework* RAG-capable from the start, even though Crucible starts +small: + +1. **Build (a) — tool-enabled `call_llm` — now, as a shared rustapp capability (not Crucible-specific).** + `RealEffects.call_llm` runs a bounded agent turn whose tool belt the host assembles from: the + backend's knowledge-base **search tools** (standard keyword + vector search over the `ComposerRAGDB` + named by the descriptor), the shared **learned-KB tools** (`kb_tools`), and **source-navigation** + tools. This is exactly how the CVL and Foundry authors already work; it scales to any corpus size + and is backend-agnostic, so **CVLR-Solana ships only a large `cvlr_manual` DB and reuses the + mechanism with zero framework change.** (Chosen over (b) because it directly reuses the existing RAG + tool implementations and the agent-with-tools machinery; over (c) because (c) cannot scale to CVLR.) +2. **Knowledge is per-*backend*, not per-ecosystem.** Crucible and CVLR are both Solana backends but + have entirely different manuals, so the selector is already the right grain: + `AppDescriptor.rag_db_default` names the wheel's own DB (`crucible_kb` vs `cvlr_manual`). Corpus size + is invisible to the framework. (This mirrors [ecosystem-abstraction.md §2](./ecosystem-abstraction.md)'s + "multiple backends per ecosystem" — knowledge rides the backend axis, prompts ride the ecosystem + axis.) +3. **Static injection is a Crucible *content* shortcut layered on top — not the mechanism.** Because + Crucible's docs are small, its wheel *may additionally* inject a compact **harness cheat-sheet** + (assertion macros, the `TestContext` builder chain, fixture/action conventions, the Anchor + error-code table) + one or two curated example harnesses (`examples/staking`, `examples/escrow`) for + the always-needed basics, while the same tool surface serves the rest. CVLR simply skips injection + and relies on the search tools. So Crucible's `crucible_kb` can even start nearly empty without + blocking the framework work. +4. **Ingestion** — Crucible's corpus ships as a committed manifest + (`rust/crucible-app/crucible_kb.rag.json`, generated from the crucible repo's `docs/`) and is + imported by the generic `composer.scripts.rag_import`; a CVL-manual-style corpus for CVLR later. +5. **Learned knowledge (orthogonal, shared).** The Harness Guide's blockers are often *program-specific* + (which keypair is admin, string-vs-binary PDA seeds, init order) — discovered during authoring, in + no manual. Persist them via the existing learned-KB / memory store (`kb_tools` `KBPut`, + [knowledge_base.py](../composer/kb/knowledge_base.py)) so a later component's author reuses what an + earlier one learned about the same program. Generic; benefits every backend. + +--- + +## 8. The gate + +Mirror the Solana front-half gate ([tests/test_solana_gate.py](../tests/test_solana_gate.py)) and the +autoprove end-to-end test, but run the **real Crucible backend** on the existing +[`solana_vault`](../test_scenarios/solana_vault/) Anchor scenario (it already has three instructions +and a clear invariant — balance = deposits − withdrawals, only-authority-withdraws, no underflow): + +1. **Build + dry-run gate (cheap, no LLM):** provision `crucible` + the sBPF toolchain; assert + `prepare_system`/`prepare_formalization` build the `.so`, generate the IDL, author a fixture, and + pass `crucible run --dry-run`. This alone validates §6 + §7.1. +2. **Full live gate (expensive):** real LLM authoring loop → for each component, a compiling Crucible + test that runs to a bounded timeout; assert at least one property is expressed and the pipeline + reaches a report. Bonus signal: seed a *known bug* into a `solana_vault` variant (e.g. an + unchecked withdraw) and assert Crucible **refutes** the relevant property (a `[FUZZ_FINDING]`), and + `crucible tmin` minimizes it — the fuzzing analog of a prover CEX. + +Provisioning notes carry over from prior gates: `env -u CERTORA`, testcontainers Postgres, a +deterministic embedder. New: the Solana build toolchain + `crucible` binary must be on `PATH` in the +gate environment (§6). Because fuzzing is nondeterministic, gate on *"a violation was found for the +seeded bug within N seconds"* with a generous budget, not on an exact crash sequence. + +--- + +## 9. Phased plan + +Each phase has a concrete gate, in the style of [ecosystem-abstraction.md §10](./ecosystem-abstraction.md). + +1. **Toolchain + preconditions + build/IDL pre-work.** `validate_preconditions` (§6) including + version resolution (§6.1) — a `--crucible-version`, detection of the program's Solana/Anchor + toolchain, and the compatibility-table lookup that pins the harness manifest; the `prepare_system` + build-`.so`-and-IDL step (via `crucible-idl-gen`); the general `RunCommand`/`CommandResult` ABI + + `Effects` addition (§7.2) with a Python `RealEffects` runner (exec-not-shell, path-confined, + semaphore-bounded). **Gate:** given `solana_vault`, the pipeline resolves a concrete version combo, + builds the `.so`, emits an IDL, and a hand-written trivial fixture passes `crucible run --dry-run` — + no LLM, no property authoring. +2. **Deliverable model (§7.1).** A Crucible-specific `ArtifactStore` + harness assembler that write a + *compilable* crate: one `[[bin]] invariant_test`, per component a feature whose name is the test fn + (macro self-gated), shared fixture + all fns in one `src/main.rs`. **Gate met:** a hand-authored + fixture + one test written through the store assembles a crate that `crucible run vault c_deposit + --dry-run` accepts; metadata lands under `certora/crucible/` and a co-located EVM `certora/specs/` + deliverable is left untouched. +3. **The fixture-authoring loop (shared fixture/actions) + the knowledge seam (§7.5).** Implemented as + a new IoC hook: `Application::new_setup_session` (SDK) drives a Rust decider through the effect loop + — `CallLlm` (author a `Fixture` from the analyzed model + a harness cheat-sheet, reading source via + tools) → `RunCommand` (assemble the crate with a `c_probe` test, `crucible run --dry-run`) → publish + / revise / give up. Built the **tool-enabled `call_llm`** framework change (host-assembled tool belt: + source + RAG + learned-KB) so a future CVLR-Solana backend reuses it unchanged; the §7.5 cheat-sheet + is embedded by the decider (a `crucible_kb` RAG DB is layered on at packaging). **Gate met:** with a + real model (`tests/test_crucible_setup_gate.py`), the agent read the vault source (`list_files` / + `get_file`) and authored a clean `Fixture` (`#[fuzz_fixture] setup()` + actions, no test fn) that + passed `crucible run --dry-run` with no human edits — green in ~37s. (Pipeline store-selection + + driving the setup session from `prepare_formalization` is packaging, Phase 5.) +4. **`formalize` per-component tests + verdicts.** `new_session` is now a per-component IoC decider: + author one test fn `c_` (CallLlm, against the shared fixture) → fuzz it (RunCommand: `crucible + run --mode explore --timeout`) → bake a verdict — clean run to budget = GOOD, `[FUZZ_FINDING]` = + BAD, compile error = revise/give-up. Verdicts ride a new `Formalized.verdicts` field (a + self-contained backend publishes them directly; `fetch_verdicts` returns them without an FFI call). + **Gate met (core):** with a real model, the agent authored a genuine invariant (reads `VaultState` + on-chain, `fuzz_assert_le!` on the recorded balance) that compiled, fuzzed to a 15s timeout with no + violation, and published GOOD with a correct property→unit map (`tests/test_crucible_formalize_gate.py`, + ~59s). The BAD path is verified at the state-machine level; a **live seeded-bug refutation gate** + (§8.2 — a buggy vault variant refuted + `tmin`-minimized) remains a follow-up. Per-run crate + assembly via `finalize`/the store and the fuzz-nondeterminism caching question (§10 Q4) land with + packaging (Phase 5). +5. **Package as the `crucible` application.** Done: `composer/crucible/pipeline.py` + (`run_crucible_pipeline` / `build_crucible_backend`) assembles the `RustBackend` with the + `CrucibleArtifactStore` + host-resolved `CrucibleDep` (§6.1) + build/fuzz timeouts; the adapter's + `prepare_formalization` drives the setup session and threads the fixture/config into per-component + `formalize`; the entry point takes the ecosystem's `forbidden_read` (Cargo) and gained a + `run_pipeline_fn` hook; `composer/crucible/cli.py` + the `console-crucible` script are the thin + console glue. Manifest pre-placement (the decider can't render host-resolved deps) is generic guarded + store hooks (`write_setup_manifest` / `prepare_component`). **Gate met:** one `run_crucible_pipeline` + call on `solana_vault` with a real model ran the whole vertical (`tests/test_crucible_e2e_gate.py`, + ~21 min) — analyzed 3 instructions, extracted 28 properties, authored the shared fixture, and + produced per-instruction fuzz verdicts (deposit/withdraw delivered with BAD — the fuzzer found + counterexamples; initialize gave up cleanly after failing to compile, surfaced as a handled + failure), then a report. **Knowledge base — built.** The Crucible corpus ships as a committed + manifest (`rust/crucible-app/crucible_kb.rag.json`) imported by the shared + `composer/scripts/rag_import.py` (drives `BlockBuilder`; both `add_chunks_batch` + + `add_manual_section`) into a `crucible_kb` `ComposerRAGDB` (a `crucible_rag_user` + schema in `rag_db`, added to `init-db.sql`); `composer/tools/crucible_rag.py` exposes keyword / + vector / get-section tools, which `build_crucible_env` binds into `env.rag_tools` (gracefully + degrading to the cheat-sheet if the embedder is absent). Populated end-to-end (126 embedded + 126 + manual sections) with on-target retrieval (e.g. "read on-chain account state in an invariant" → + *TestContext / Account Reading* + *Writing-Harnesses / Deserialize On-Chain State*). This is the + template for CVLR-Solana's `cvlr_manual`. (Runtime binding needs `sentence-transformers` in the app + venv; absent it, `build_crucible_env` falls back to the cheat-sheet, which carried phases 3–5.) + Remaining polish: a TUI entry, wiring the `--fuzz-timeout`/`--crucible-version` args through, and + **verdict triage** (a BAD may be a true bug or an over-strict invariant — §10 Q4). The application + still runs on **trusted input only** until Phase 6. +6. **Sandbox every `RunCommand` (required — §7.4) — done.** All command execution (`cargo build-sbf`, + the harness `cargo build`, `crucible run`) runs behind a trusted launcher shim (`run-confined`) + applying **Landlock (filesystem) + seccomp (network + `ptrace`) + env allowlist + rlimits** — + network-off, a clean/secret-free env, a workdir-only writable filesystem, offline build; no + container changes (full design + validation: [command-sandbox.md](./command-sandbox.md)). Built as + a swappable `SandboxProvider` seam ([composer/sandbox/](../composer/sandbox/)); **Crucible's default + provider is `launcher`** (fail-closed if `run-confined` is absent; `COMPOSER_SANDBOX_PROVIDER=none` + to opt out). **Gate green:** the escape suite denies every vector (planted secret env, + `/proc//environ`, a host file outside the workdir, external TCP and `169.254.169.254`), a + real `cargo-build-sbf` and `crucible run --dry-run` of `solana_vault` succeed confined and offline, + and **the full LLM e2e passes under the launcher** (`test_crucible_e2e_gate.py`: shared fixture + authored, **all three instructions — initialize / deposit / withdraw — delivered with fuzz + verdicts**) with zero sandbox denials. Getting the vertical green surfaced + fixed two bugs: a + sandbox policy gap (the linker's `/tmp` scratch → a private per-run `TMPDIR`), and a shared-crate + concurrency race that had dropped one instruction (now cumulative Cargo features + serialized + atomic per-component builds — command-sandbox.md §11 item 8 / §10 Q1). +7. **Polish (optional).** Report nouns (§7.3); coverage surfacing (`--coverage`/LCOV as a report + attachment); stateful-mode tuning; crash-artifact rendering in the frontend. + +**Definition of done — met.** Phase 6 is green: every command run via the `RunCommand` effect +(`cargo-build-sbf`, the harness `cargo build`, `crucible run`) executes inside the sandbox by +default, verified by the escape test + a real confined build/dry-run, with the full vertical showing +zero sandbox denials. The backend may now be pointed at untrusted programs (on a Linux/Landlock host +with `run-confined` present; fail-closed otherwise). Remaining work is orthogonal: the optional +Phase 7 polish, and improving LLM fixture-authoring reliability (§10 Q3) — a quality issue, not a +safety one. + +--- + +## 10. Open questions + +1. **Unit granularity for a fuzzer.** — **RESOLVED (commit fd51700; see + [crucible-unit-granularity.md](./crucible-unit-granularity.md)).** Neither (a) per-instruction + nor (b) a single opaque whole-program harness: Solana now uses **global extraction with + per-invariant units** — one whole-program property pass proposes cross-instruction invariants, + each of which fans out to its own harness fn + fuzz run + report row (via the ecosystem's + `global_extraction` / `extraction_unit` / `property_unit` hooks). This matches the fuzzer's + whole-program semantics and Foundry's stateful-fuzz model while keeping per-invariant + attribution. Original framing retained below for history: the `units` were per-instruction, but + a Crucible invariant is inherently *cross-instruction* (checked over a random action sequence). +2. **Sandbox specifics (required work — §7.4, §9 Phase 6).** The isolation *requirements* and the + mechanism are decided — **unprivileged Landlock + seccomp self-sandboxing** (filesystem, network, + env, rlimits) behind the `RunCommand` seam, needing no container changes — and building it is an + in-scope, definition-of-done phase (not deferrable); the detailed design + the step-1 validation + matrix is [command-sandbox.md](./command-sandbox.md). macOS is **not supported** (Linux/Landlock + only → hard fail; command-sandbox.md §8). Open items within the phase are minor (Landlock ABI + negotiation on target AMIs; `AF_UNIX`/`AF_NETLINK` allowance — command-sandbox.md §11). + Infra-layer hardening (VM-per-run / gVisor / IMDSv2 / least-priv IAM) is orthogonal and + non-blocking, decided per deployment. +3. **Fixture authoring difficulty.** `setup()` for real DeFi programs is hard (init order, admin + whitelists, account patching — the Harness Guide is a long playbook). The setup session *looked* + like the failure point, but the live gate revealed the real blocker was a sandbox `/tmp` gap (the + linker's scratch), not authoring — a link failure reads as "could not compile," so the model + churned on a fine fixture; once fixed, the fixture authored and the e2e passed. The authoring loop + was hardened anyway (crucible-app decider): more attempts (4→7); **compiler errors extracted + + front-loaded** in the revise prompt; a concise **"PROGRAM API FACTS"** block (crate id — the real + dep crate, not the `#[program] mod` name — `declare_id`, each instruction's snake→Pascal + `instruction::`/`accounts::` names + args + accounts); a full **worked example** fixture; explicit + **Anchor path conventions**; and `RunCommand` **failure logging** (which surfaced the `/tmp` bug). + A multi-round/self-critique loop remains the next lever if genuine authoring difficulty recurs. +4. **Nondeterminism in verdicts + caching.** A clean fuzz run is "no bug found in N seconds," not a + stable fact — re-running may differ. How do we cache a `formalize` result keyed by the property + batch when the *verdict* isn't deterministic? Likely cache the authored harness (deterministic) + but re-fuzz on demand, or record the seed (`--seed`) for reproducibility. Decide in Phase 4. +5. **Whole-crate deliverable vs the host's single-file layout.** §7.1 (a) vs (b) — a real fork in how + much the generic Rust-app host must grow. Prototype (a) in Phase 2 before committing. +6. **Coverage as a first-class signal.** Crucible emits LCOV and edge counts. Should low coverage + *downgrade* a GOOD verdict (i.e. "held, but the fuzzer barely explored this path")? Powerful but + out of scope for v1; note it and defer (Phase 7). +7. **Is one `rag_db` + standard keyword/vector search enough for CVLR? (§7.5)** The tool-enabled + `call_llm` binds a *single* descriptor-named `ComposerRAGDB` with the standard search tools — which + covers Crucible and, we expect, CVLR-Solana (whose CVL manual is the same *shape* as today's CVL + DB). But the current CVL backend also exposes richer surfaces (`cvl_research`, multiple + `cvl_manual_*` tools). Decide whether the descriptor should let a backend declare **multiple KBs / + bespoke knowledge tools**, or whether one DB + standard search suffices; keep it to one DB until + CVLR proves it needs more. + +## 11. Key files + +| Concern | File | +|---|---| +| The backend seam (the contract to implement) | [docs/formalization-abstraction.md](./formalization-abstraction.md) · [composer/pipeline/core.py](../composer/pipeline/core.py) | +| The Rust-app framework (backend in Rust) | [docs/rust-applications.md](./rust-applications.md) · [docs/rust-formalization-backends.md](./rust-formalization-backends.md) | +| The Rust SDK ABI to extend (§7.2) | [rust/autoprover-sdk/src/lib.rs](../rust/autoprover-sdk/src/lib.rs) | +| The Python host / IoC loop / adapter to extend | [composer/rustapp/host.py](../composer/rustapp/host.py) · [composer/rustapp/loop.py](../composer/rustapp/loop.py) · [composer/rustapp/adapter.py](../composer/rustapp/adapter.py) | +| The Solana ecosystem (reused front half) | [composer/spec/solana/model.py](../composer/spec/solana/model.py) · [composer/pipeline/ecosystem.py](../composer/pipeline/ecosystem.py) · `composer/templates/{rust,solana}/…` | +| Closest backend precedent (local-CLI, refutation) | [composer/foundry/pipeline.py](../composer/foundry/pipeline.py) · [composer/foundry/runner.py](../composer/foundry/runner.py) | +| Knowledge/RAG precedent to mirror (§7.5) | [composer/tools/foundry_rag.py](../composer/tools/foundry_rag.py) · [scripts/populate_foundry_rag.sh](../scripts/populate_foundry_rag.sh) · [composer/kb/knowledge_base.py](../composer/kb/knowledge_base.py) | +| The Crucible backend crate (new) | `rust/crucible-app/` | +| Crucible knowledge base (new) | `crucible_kb` RAG DB · committed manifest `rust/crucible-app/crucible_kb.rag.json` + `composer/scripts/rag_import.py` (new) | +| Scenario + gate | [test_scenarios/solana_vault/](../test_scenarios/solana_vault/) · `tests/test_crucible_gate.py` (new) | +| Crucible itself (docs) | `/home/eric/src/crucible/docs/` (harness-guide, writing-tests, cli-reference, remote-fuzzing) | diff --git a/docs/crucible-demo.md b/docs/crucible-demo.md new file mode 100644 index 00000000..ada4985a --- /dev/null +++ b/docs/crucible-demo.md @@ -0,0 +1,214 @@ +# Running the Crucible demo (solana_vault) + +Step-by-step instructions for a human to run the Crucible (Solana fuzzing) backend +end-to-end on the bundled `solana_vault` Anchor scenario. This is the full vertical: +sBPF build → Solana analysis + property extraction → shared-fixture setup → +per-instruction test authoring + fuzzing → report. + +Budget ~15–20 minutes of wall-clock for a full run, plus one-time setup (toolchain + +a ~0.5 GB model download on first run). It makes real, paid LLM calls. + +--- + +## 1. Prerequisites (one-time) + +### 1a. External toolchain on `PATH` + +The pipeline shells out to these; install them first and confirm they resolve: + +```bash +which crucible # the Crucible fuzzer CLI +which cargo-build-sbf # Solana platform-tools (builds the program to sBPF) +``` + +- `cargo-build-sbf` comes with the Solana platform tools / Anchor toolchain. +- `crucible` is the fuzzer CLI (installed to `~/.cargo/bin` in a typical setup). + +### 1b. A local `crucible` checkout + +The generated harness crate path-depends on Crucible's own crates, so you need a +local clone and must point `CRUCIBLE_REPO` at it. It must contain +`crates/crucible-fuzzer`: + +```bash +export CRUCIBLE_REPO=/path/to/crucible # e.g. ~/src/crucible +ls "$CRUCIBLE_REPO/crates/crucible-fuzzer" # must exist +``` + +There is **no default** — the run errors clearly if `CRUCIBLE_REPO` is unset. + +### 1c. An Anthropic API key + +```bash +export ANTHROPIC_API_KEY=sk-ant-... +``` + +### 1d. Python environment (the `ml` extra is required) + +The pipeline builds a real sentence-transformers embedder for the indexed store, so +the venv must include the `ml` extra **and** a torch flavor (`cpu` for a GPU-less +box, `cuda` for a GPU box — they are mutually exclusive). Keep your existing +`certora-cli` selection. This mirrors the container's `uv sync` line: + +```bash +# GPU-less host: +uv sync --extra cpu --extra ml --extra certora-cli --group apps +# GPU host: swap --extra cpu for --extra cuda +``` + +The **`apps` group** declares the `crucible_app` and `echoprover` wheels as editable +path dependencies (`[tool.uv.sources]`), so `uv sync --group apps` **builds them via +maturin and keeps them** — it no longer prunes the wheels the way a bare `uv sync` +did with the old out-of-band `maturin develop`. The group is deliberately outside the +default groups, so the container image (no Rust toolchain) never tries to compile it. + +### 1e. Auto-rebuild on Rust changes (one-time hook install) + +Install the maturin import hook into the venv **once**: + +```bash +python -m maturin_import_hook site install +``` + +After that, with the venv **activated** (so `maturin` is on `PATH`), editing anything +under `rust/crucible-app` transparently recompiles `crucible_app` on the next `import` +— no manual `maturin` step. As a fallback (e.g. running without an activated venv), you +can still force a rebuild explicitly: + +```bash +uv run --no-sync maturin develop --release -m rust/crucible-app/Cargo.toml +``` + +### 1f. Build the sandbox launcher (`run-confined`) + +The default provider confines every external command (Landlock + seccomp) and is +**fail-closed** — it refuses to run if the binary is missing. Build it once: + +```bash +cd rust && cargo build -p run-confined --release && cd .. +ls rust/target/release/run-confined # must exist +``` + +To demo **without** confinement (trusted-input dev), skip this and set +`COMPOSER_SANDBOX_PROVIDER=none` in step 3. + +### 1g. (Optional) Populate the RAG knowledge base + +Improves the model's Crucible-specific grounding. The pipeline falls back to a static +cheat-sheet if it's absent, so you can skip this for a first demo. The corpus ships as a +committed manifest (`rust/crucible-app/crucible_kb.rag.json`) — no crucible checkout needed — +imported by the generic importer under the `ragbuild` dependency group: + +```bash +uv sync --extra cpu --extra ml --extra certora-cli --group ragbuild --group apps +uv run --group ragbuild python -m composer.scripts.rag_import \ + rust/crucible-app/crucible_kb.rag.json +``` + +--- + +## 2. Start Postgres + +The CLI stores conversation memory + LangGraph checkpoints in the composer Postgres: + +```bash +docker compose -f scripts/docker-compose.yml up -d +``` + +--- + +## 3. (Recommended) Clean the scenario directory + +Earlier runs leave large generated dirs (`.sandbox_cargo/`, `target/`, `fuzz/`, …) in +the scenario. They no longer break a run (the source tools exclude them), but a clean +dir builds faster: + +```bash +rm -rf test_scenarios/solana_vault/{.sandbox_cargo,.sandbox_tmp,target,corpus,output,fuzz,certora,.certora_internal} +``` + +--- + +## 4. Run the demo + +```bash +env -u CERTORA_DEV_MODE -u CERTORA -u CERTORA_DISABLE_POPUP -u CERTORAKEY \ + -u CERTORA_DISABLE_AUTO_CACHE -u CERTORA_DISABLE_NOTIFICATION \ + COMPOSER_SANDBOX_PROVIDER=launcher \ + CRUCIBLE_REPO="$CRUCIBLE_REPO" \ + uv run --no-sync console-crucible \ + test_scenarios/solana_vault \ + test_scenarios/solana_vault/programs/vault/src/lib.rs:vault \ + test_scenarios/solana_vault/system.md \ + --max-bug-rounds 1 --fuzz-timeout 30 +``` + +Positional arguments: + +1. `project_root` — the scenario root. +2. `main_contract` — `path:ProgramName`. Here the program/crate name is `vault` (the + crate, **not** the `#[program]` module `vault_program`). +3. `system_doc` — the design document (text or PDF) the analysis reads. + +Useful flags (`console-crucible --help` lists all): + +- `--max-bug-rounds N` — property-extraction rounds per instruction; `1` for a shorter + demo, default `3`. +- `--fuzz-timeout SECONDS` — per-property fuzzing budget (default 30). +- `--max-concurrent N` — concurrent agents (default 4). +- `--interactive` — pause to refine extracted properties before formalization. +- `--heavy-model` / `--lite-model` — default `claude-opus-4-6` / `claude-sonnet-4-6`. + +The `env -u CERTORA_*` unsets are **required** — those variables otherwise interfere +with the run. + +> **First run downloads the embedder.** On the first `get_model()` call the pipeline +> fetches `nomic-ai/nomic-embed-text-v1.5` (~0.5 GB, `trust_remote_code=True`) from +> Hugging Face — one-time, needs internet. + +--- + +## 5. What you'll see + +At startup it prints the log paths (`.certora_internal/autoProve/.events.jsonl` +and a text log). Then phases stream to the console: + +1. **sBPF build** — `cargo-build-sbf` compiles the program; the harness loads the `.so`. +2. **System Analysis** — the model maps the program's instructions/accounts. +3. **Property extraction** — per instruction (`initialize`, `deposit`, `withdraw`, …). +4. **Build Harness** — the shared-fixture setup session authors `main.rs` + fixture and + validates it with `crucible run … --dry-run` (labeled e.g. `crucible authoring turn`). +5. **Per-component authoring + fuzzing** — a test per property, gated by `crucible run`. +6. **Report** — a property-keyed report is assembled and written. + +Deliverables land under `test_scenarios/solana_vault/certora/crucible/` (per-component +tests, `commentary.md`, the property→tests map) plus the report; a run summary +(components / properties / failures) prints at the end. + +--- + +## 6. Faster smoke check (no LLM, ~1 min) + +To confirm the toolchain + sandbox + harness-build path works before a full paid run: + +```bash +CRUCIBLE_REPO="$CRUCIBLE_REPO" COMPOSER_SANDBOX_PROVIDER=launcher \ + uv run --no-sync python -m pytest tests/test_crucible_sandbox_gate.py -q +``` + +--- + +## 7. Troubleshooting + +| Symptom | Cause | Fix | +|---|---|---| +| `NotImplementedError: Sentence transformers not available` | venv synced without the `ml` extra | `uv sync --extra cpu --extra ml --extra certora-cli --group apps` | +| `ModuleNotFoundError: No module named 'crucible_app'` | synced without the `apps` group | re-sync with `--group apps` (1d) | +| `maturin not found` when a Rust edit should have rebuilt | import hook can't see `maturin` on `PATH` | activate the venv (or use `uv run`) so `.venv/bin` is on `PATH`; re-import | +| `FileNotFoundError: crucible checkout not configured` | `CRUCIBLE_REPO` unset / wrong | point it at a clone containing `crates/crucible-fuzzer` | +| sandbox "provider unavailable" / fail-closed | `run-confined` not built | build it (1f), or set `COMPOSER_SANDBOX_PROVIDER=none` | +| Postgres connection errors | DB not up | `docker compose -f scripts/docker-compose.yml up -d` | +| `Failed to spawn: pyright` (only when validating) | `uv sync` dropped the `ci` group | add `--group ci --group test` to the sync (keep `--group apps`) | + +> After **any** `uv sync`, re-run the wheel build (1e) — this is the most common +> foot-gun. diff --git a/docs/crucible-foundry-parity.md b/docs/crucible-foundry-parity.md new file mode 100644 index 00000000..ed795f76 --- /dev/null +++ b/docs/crucible-foundry-parity.md @@ -0,0 +1,217 @@ +# Crucible → Foundry parity: gap analysis + +A feature-by-feature comparison of the **Crucible** (Solana fuzzing) backend against +the mature **Foundry** (EVM) backend, to identify the work remaining to bring Crucible +to parity. Scope is strictly *Foundry* parity — several capabilities exist only on the +**autoprove/prover** path and are absent from *both* Foundry and Crucible; those are +called out as "parity (neither has it)" so they are not mistaken for Crucible gaps. + +## TL;DR + +Crucible is functionally end-to-end and shares the report schema, grouping, caching, +console frontend, precondition checks, and per-component deliverables with Foundry. The +**real Foundry-parity gaps are two, plus two smaller ones**: + +1. **No TUI + no live progress telemetry** (Foundry has `tui-foundry` and streams + `forge_test_run` summaries; Crucible has no `tui-crucible`, and its declared event + kinds are never emitted). — *largest gap* +2. **No design-doc auto-discovery** (Foundry accepts an optional `system_doc` and + discovers one; Crucible requires it). +3. Per-component **status artifact** (Foundry writes `*.status.json`; Crucible doesn't). +4. **Build concurrency** (Foundry runs forge processes in parallel; Crucible serializes + on one shared harness crate). + +Everything else is either at parity or is a Crucible-specific rough edge unrelated to +Foundry (dead tuning flags, hardcoded toolchain versions) — see §3. + +--- + +## Parity scorecard + +| Capability | Foundry | Crucible | Status | +|---|---|---|---| +| Console entry point | `console-foundry` | `console-crucible` | ✅ parity | +| **TUI entry point** | `tui-foundry` (`FoundryApp`) | `tui-crucible` (`GenericRustApp`) | ✅ parity — *done (2c0f693)* | +| **Progress telemetry** | emits `forge_test_run` summaries | emits `fuzz_pulse`/`fuzz_finding`/`build_output` (post-hoc, per run) | ✅ parity — *done (2c0f693)* | +| **Design-doc auto-discovery** | `system_doc` optional → discovery phase | `system_doc` optional → discovery phase | ✅ parity — *done (55a5959)* | +| Per-component status artifact | `*.status.json` | commentary + property→tests only | ⚠️ minor gap | +| Build/verify concurrency | `--max-forge-runners` parallel | serialized (`Semaphore(1)`, shared crate) | ⚠️ perf gap | +| Shared `report.json` + backend labels | ✅ | ✅ (`crucible` labels wired) | ✅ parity | +| Verdict model (GOOD/BAD only) | GOOD/BAD | GOOD/BAD | ✅ parity | +| Per-component deliverables | commentary/properties/property-tests | commentary/properties/property-tests (+crate) | ✅ parity | +| Upfront precondition validation | lazy (foundry.toml at first run) | eager (`validate_preconditions`: bins + Cargo.toml) | ✅ Crucible ahead | +| cache-ns / memory-ns / result cache | ✅ | ✅ | ✅ parity | +| RAG env | Foundry cheatcode DB | `crucible_kb` DB (optional) | ✅ parity (different DBs) | +| ap-trail / run index (run_id) | ✅ | ✅ | ✅ parity | +| Test coverage | 1 arg-parser test | 7 gates + unit tests | ✅ Crucible ahead | +| `--interactive` (HITL refinement) | flag forwarded; handler **raises NotImplementedError** | flag forwarded; handler **raises NotImplementedError** | ➖ parity (neither services it; autoprove-only) | +| `threat_model` | plumbed but forced `None`, no flag | hardcoded `None`, no flag | ➖ parity (neither; autoprove-only) | +| `write_job_info` / token-usage ledger | not called (`at_exit=None`) | not called (no `at_exit`) | ➖ parity (autoprove-only) | +| Auto HTML report | no (manual `autoprove-report-render`) | no (same manual CLI) | ➖ parity | +| `finalize` run-level artifact | not overridden (no-op) | not implemented (no-op) | ➖ parity | + +Legend: ❌ real gap · ⚠️ minor/perf gap · ✅ parity · ➖ parity because *neither* has it (a +capability that exists only on autoprove/prover). + +--- + +## 1. Real Foundry-parity gaps (the work) + +### 1.1 TUI frontend + progress telemetry — **DONE (commit 2c0f693)** + +> **Resolved.** The decider now issues `Command::Emit` from the setup and per-component +> sessions (`fuzz_pulse` before each fuzz run + on a clean hold, `fuzz_finding` on a +> counterexample, `build_output` on (re)compiles / dry-run outcomes); `RealEffects.emit` +> was fixed to route out-of-graph via `push_custom_update` (it previously dropped every +> event through a stale `get_stream_writer()` call); and `tui-crucible` is wired +> (`GenericRustApp`). Events are **post-hoc per run** (matching Foundry's +> `forge_test_run` summaries) — true intra-run live streaming would need the +> `RunCommand` effect to stream subprocess output and is deferred. Original analysis +> retained below for context. + +### ~~1.1 TUI frontend + live progress telemetry — largest gap, L~~ (superseded) + +Foundry ships a Textual TUI (`tui-foundry` → `composer/cli/tui_foundry.py`, `FoundryApp` +in `composer/foundry/foundry_app.py:114`) that streams each `forge test` run's summary +into a per-task collapsible panel (`ForgeTestRunEvent`, `composer/foundry/runner.py:73-79`; +rendered `foundry_app.py:79-97`). Crucible has: + +- **No `tui-crucible` script** (`pyproject.toml` has only `console-crucible`). +- **No emitted events.** `rust/crucible-app/src/lib.rs` declares three event kinds + (`fuzz_pulse`, `fuzz_finding`, `build_output`, `lib.rs:666-670`) but contains **no + `Command::Emit`** anywhere — the sessions only issue `CallLlm`/`RunCommand`/`Publish`/ + `GiveUp`. So there is no live fuzzing pulse, no streamed findings, no build output + panel. Crash detection is a post-hoc string match on `[FUZZ_FINDING]` in captured + stdout (`lib.rs:598`), never surfaced live. + +The generic machinery exists and is reachable (`GenericRustApp` in +`composer/rustapp/frontend.py:77`; a generic `tui_main` in `composer/rustapp/cli.py:92`), +so this is two pieces of work: +- **(a) Wire a `tui-crucible` entry** that builds the app with the crucible env + (`build_crucible_env`) + `run_crucible_pipeline` (mirror how `console_crucible` + injects `run_pipeline_fn`). Small once (b) exists. +- **(b) Emit events from the Rust decider** (`Command::Emit` in the setup/per-component + sessions of `lib.rs`) for build output and fuzzing progress/findings, so the TUI (and + console) have something to show. This is the substantive part. + +Note: HITL/interactive refinement is *not* part of this gap — Foundry's own handler +raises `NotImplementedError` for HITL (`foundry_app.py:74`), exactly like the Rust +frontend (`frontend.py:58`). Neither backend services interactive refinement. + +### 1.2 Design-doc auto-discovery — **DONE (commit 55a5959)** + +> **Resolved.** `system_doc` is now optional in the generic rust entry; when omitted, +> `resolve_design_doc` runs as a visible "Design Doc Discovery" task (a dedicated +> UI-only descriptor phase), and the doc-dependent construction moved into the runner +> so a discovered and a supplied doc share the same cache key. Original analysis below. + +### ~~1.2 Design-doc auto-discovery — M~~ (superseded) + +Foundry (and autoprove) make `system_doc` optional and, when omitted, run a "Design Doc +Discovery" task via the shared `cli_pipeline` (`composer/pipeline/cli.py:194-218`, +`resolve_design_doc` in `composer/spec/source/design_doc_finder.py`). Crucible goes +through the generic Rust entry (`composer/rustapp/entry.py`), where `system_doc` is a +**required positional** (`entry.py:149`) and `design_doc_finder` is never imported. So a +Crucible run must always be handed an explicit design doc. + +Work: integrate `resolve_design_doc` into `rust_entry_point` (make `system_doc` +`nargs="?"` and run discovery when absent), reusing `design_doc_finder`. The +`DesignDocChosenEvent` UI is already consumed by autoprove/foundry frontends and could +be surfaced once the TUI (1.1) exists. + +### 1.3 Per-component status artifact — **S** + +Foundry writes a `*.status.json` per component (`ComponentTestStatus`: +pass/expected-failure/skip, `composer/foundry/artifacts.py:100-126`). Crucible writes +`commentary.md` + the property→tests map (`composer/crucible/store.py:113-117`) but no +equivalent status file. The baked verdicts land in `report.json`, so this is largely +redundant — decide whether Crucible needs a parallel status artifact for tooling that +reads per-component status off disk. + +### 1.4 Build/verify concurrency — **M (design)** + +Foundry runs multiple `forge test` processes concurrently, gated by `--max-forge-runners` +(`composer/foundry/pipeline.py:192-203`). Crucible **serializes** all harness builds/fuzz +runs via a single `command_sem = Semaphore(1)` (`composer/crucible/backend.py`) because +every component shares one harness crate (`fuzz//`). This is a throughput gap on +multi-invariant programs. + +**Investigated; deferred (see docs/crucible-unit-granularity.md §7).** The obvious +"crate-per-component" fix does not work: separate `target/` dirs recompile the heavy deps +N× (a regression), and a shared `CARGO_TARGET_DIR` collides on Crucible's hardcoded +`invariant_test` binary name. The only clean path is `crucible run --binary-in` (build +serially into one crate, fuzz binaries in parallel) — a non-trivial formalize-phase +redesign whose payoff only materializes at production fuzz budgets (60–300 s), so it's +deferred until then. + +--- + +## 2. Already at parity (including limitations Foundry shares) + +These are **not** Crucible gaps — Foundry lacks them too (they exist only on the +autoprove/prover path). Listed so they aren't mistaken for work: + +- **Interactive HITL refinement** — both backends forward `--interactive` to the driver + but their handlers raise `NotImplementedError` (`foundry_app.py:74`, `frontend.py:58`). +- **Threat model** — Foundry forces `threat_model=None` (`foundry/entry.py:103`); Crucible + hardcodes `None` (`rustapp/host.py:207`). Only autoprove exposes `--threat-model`. +- **`write_job_info` / token- & prover-usage ledger** — called only by autoprove + (`autoprove_common.py:109`); Foundry passes `at_exit=None`, Crucible has no `at_exit`. + Both only `print(RunSummary.format())`. +- **Auto HTML report** — neither renders HTML in-pipeline; both rely on the shared + `autoprove-report-render` CLI over `report.json`. +- **`finalize` run-level artifact / `extra_report_inputs`** — neither overrides these + (both no-op); only the prover uses them. +- **Verdict granularity** — both emit only GOOD/BAD (never ERROR/TIMEOUT). For a + coverage-guided fuzzer, "ran to the fuzzing budget with no counterexample ⇒ GOOD" + (`lib.rs:604`) is the intended semantics, not a misclassification; a build failure + becomes a `GiveUp`/failure rather than a verdict, matching Foundry's compile-failure + handling. + +--- + +## 3. Crucible-specific rough edges (remaining work beyond Foundry parity) + +Independent of Foundry, these are worth closing for a production-quality backend: + +- ~~**Inert tuning flags.**~~ **DONE (266f421)** — `--fuzz-cores`, `--stateful`, and + `--crucible-version` were parsed but never threaded to `crucible run`; removed from the + descriptor (only `--fuzz-timeout` remains, which is honored). Re-add each with its wiring + if/when the fuzz command grows the corresponding knob. +- **Hardcoded toolchain versions.** anchor 1.0.1 / solana 3.0 / libafl 0.15.1 are pinned + in `composer/crucible/harness.py:32-43` ("Hardcoded for now"); `--crucible-version` was + meant to drive a version table (see docs/crucible-toolchain-versioning.md) but is inert. +- **No persisted usage ledger.** `ArtifactStore.write_token_usage` exists + (`artifacts.py:112`) but has **zero callers**; a Crucible run persists no token/cost + record (Foundry is the same, but this matters for demo/observability). +- **Provisional deliverable layout.** The descriptor `artifact_layout` is marked + "provisional" (`lib.rs:671`) and is superseded by the hand-written `CrucibleArtifactStore`. +- **Verdict metadata.** `line` / `duration_seconds` are always `None` on Crucible verdicts + (`lib.rs:316`), so the report's timing/line columns are blank. + +--- + +## 4. Where Crucible already exceeds Foundry + +- **Eager precondition validation** — `validate_preconditions` checks required binaries + (`crucible`, `cargo-build-sbf`, `anchor`) and `Cargo.toml` up front (`lib.rs:685-721`); + Foundry only discovers a bad `foundry.toml` lazily at first `forge test`. +- **Test coverage** — 7 gates (build/dry-run, setup, formalize, e2e, sandbox, solana) plus + harness unit tests, vs Foundry's single arg-parser regression test. +- **Command sandboxing** — every external command is confined (Landlock+seccomp), + fail-closed; Foundry runs `forge` unconfined. + +--- + +## 5. Suggested prioritization + +1. ~~**1.1 emit decider events + `tui-crucible`**~~ — **DONE (2c0f693)**. +2. ~~**1.2 design-doc discovery**~~ — **DONE (55a5959)**. +3. ~~**§3 inert flags**~~ — **DONE (266f421)**, removed. (Version-table work remains, + tracked in docs/crucible-toolchain-versioning.md.) +4. **1.4 crate-per-component** (concurrency) and **1.3 status artifact** — lower priority; + 1.4 is a larger design change tracked separately. + +Not recommended as "parity" work (they'd be new capability for *both* backends, better +scoped as cross-backend features): interactive HITL, threat-model input, usage ledger, +auto-HTML. diff --git a/docs/crucible-judge-cost.md b/docs/crucible-judge-cost.md new file mode 100644 index 00000000..a1d352c8 --- /dev/null +++ b/docs/crucible-judge-cost.md @@ -0,0 +1,112 @@ +# Plan — reducing the Crucible judge's runtime cost + +**Status:** proposed. The Crucible reviewer/judge turn (added in the formalization loop — see +`docs/rust-backend-api.md` §4) is correct and e2e-verified, but it roughly **5×'d** the end-to-end +wall-clock (a `solana_vault` gate went from ~22 min to **1:53:04**). This plan measures where the +cost goes, contrasts the approach with the CVL and Foundry backends' judges, and proposes a phased +reduction that keeps the judge's catch rate. + +## 1. Where the cost goes (measured, from the e2e run) + +| Signal (one `solana_vault` gate, 13 properties) | Author | Judge | +|---|---|---| +| LLM turns | 23 | **22** | +| `code_explorer` sub-agent calls | 59 | **53** | +| Model tier | heavy (Opus) | **heavy (Opus)** | + +Three drivers, in order of impact: + +1. **The judge re-explores the program from scratch.** 53 Code Explorer sub-agents inside judge + turns — almost as many as authoring's 59 — even though the program API (`api_facts`), the + fixture, and the source were *already* gathered during authoring. Each `code_explorer` is itself + a multi-call sub-agent, so this is the real multiplier. +2. **The judge runs on the heavy model** (`run_llm_agent` → `env.builder_heavy()`) for a bounded + review task. +3. **It roughly doubles the turns per component** (22 judge ≈ 23 author), and each judge + *rejection* (~6 in the run) adds a full author+judge re-cycle — all on each component's critical + path. + +## 2. How CVL and Foundry judge today — and why Crucible costs more + +All three backends run essentially the same *kind* of review (a "are these tests/specs meaningful +evidence?" pass with the full `source_tools + rag_tools` belt on the heavy model). The cost gap is +**architectural**, not prompt size. + +| Aspect | CVL / prover (`property_feedback_judge`) | Foundry (`feedback_tool`) | Crucible (`judge_prompt`) | +|---|---|---|---| +| **Driver** | author-invoked **tool** (in-graph) | author-invoked **tool** (in-graph) | **host-driven** turn (out-of-graph) | +| **Cadence** | when the author calls it; feedback can be addressed *without* re-invoking | when the author calls it | **unconditional — every author attempt**, before validate | +| **Context continuity** | shares the run's memory (`ctx.get_memory_tool()`) + the current spec (`get_cvl`); accumulates facts across rounds | shares the run's memory + `get_test`; prior-round conclusions assumed still valid | **fresh, stateless turn** — no memory tool, no authoring context handed in | +| **Source exploration** | has the belt, but works mostly from the passed artifact + memory | has the belt (source + rag) | has the belt and **re-derives everything** each turn | +| **Model** | heavy | heavy | heavy | +| **Acceptance** | author must clear feedback before `result` | judge must stamp the buffer before `result` | host parses `{accept, feedback}`; re-authors on reject | + +The decisive differences: + +- **In-graph, author-invoked (CVL/Foundry) vs. host-driven, unconditional (Crucible).** Because the + CVL/Foundry judge is a tool the author calls *when it is ready*, it runs deliberately (often once) + and inside the authoring conversation. Crucible's judge — by the **passive-service design** + (`docs/rust-pure-app.md`: the wheel is stateless pure callouts, Python owns the loop) — is a + *separate* turn the host fires after **every** author attempt. +- **Memory / context continuity.** CVL and Foundry judges attach `ctx.get_memory_tool()` and are + handed the artifact under review, so they don't re-mine framework facts (the Foundry prompt even + says prior-round memory "MAY be assumed still valid"). Crucible's judge turn binds only + `all_tools` (= `source_tools + rag_tools`, **no memory**) and gets no authoring context, so it + re-explores the program from zero — the 53 `code_explorer` calls. + +So Crucible's cost is largely the price of statelessness: the same review, but re-derived from +scratch, unconditionally, on the heavy model. **Most of the plan below is about recovering the +context-sharing that CVL/Foundry get for free from being in-graph — within the service-shaped +constraint that the wheel stays a set of pure callouts.** + +## 3. Plan + +### Phase 1 — close the statelessness gap (biggest win, lowest risk) +Give the judge what authoring already knows, so it stops re-deriving: +- **Inject the gathered context into the judge prompt** — the `api_facts` block and the fixture are + already in `input`; pass them in (they already are, partly) and add the program's key source + facts, so the judge reasons from them instead of re-exploring. +- **Restrict the judge's tools + recursion.** Drop or tightly cap `code_explorer` for the judge + turn (keep a bounded `get_file`/`grep` for spot-checks) and lower its `recursion_limit`. This is + where the 53 explorations collapse. **[Done]** the review sub-agent now drops `code_explorer` + (`_JUDGE_EXCLUDE_TOOLS` in `composer/rustapp/adapter.py`); the recursion cap is still open. +- **Give the judge the run's memory tool** (the CVL/Foundry lever) so facts verified for one + property carry to the next instead of being re-derived per component. **[Done]** in the in-loop + refactor (`docs/crucible-judge-in-loop.md`). + +### Phase 2 — fix the cadence (match the author-invoked pattern) +- **Don't judge unconditionally every attempt.** Today every attempt is author→judge→validate, so a + build-fail re-author re-judges. Only judge new *logic* (`failure is None or + failure.kind == "judge"`) — a mechanical compile fix doesn't need re-review. (The `TEST_CHEAT_SHEET` + lamports fix already reduces build-fail retries.) +- **Cap judge rounds** (e.g. accept after one revise) so a stubborn judge can't stack cycles — the + bounded analogue of CVL/Foundry's "address feedback without re-invoking." + +### Phase 3 — make it tunable (escape hatch + measurement knob) +- Descriptor flag / CLI arg for judge **enable** and **model tier**. Lets a run trade cost for rigor + (off for quick iteration, on for a final gate), and is how we A/B the phases. +- **Optional divergent lever: judge on the lite model.** Unlike Phases 1–2 (which align Crucible with + CVL/Foundry), CVL and Foundry both judge on the *heavy* model — so dropping Crucible to Sonnet is a + deliberate divergence. Cheap, but validate the catch rate before making it the default; prefer it as + an opt-down once Phases 1–2 land. + +## 4. Validation + +Each phase re-runs `test_crucible_e2e_gate` and compares **wall-clock + verdict parity**: do the +12 GOOD / 1 BAD outcomes hold, and does the judge still catch the fee-oracle class (now that the +`TEST_CHEAT_SHEET` fix is in)? The gate is ~2 h and paid, so iterate Phases 1–2 on a **smaller +scenario** (fewer properties) first and use the full gate only to confirm. + +## 5. Architectural note + +The judge is currently a separate host-driven turn because the current host loop runs it that way — +**not** because the passive-service design forbids the CVL/Foundry `feedback_tool` (author-invoked, +in-graph) pattern. The author loop already runs in Python (`run_llm_agent`), so the host *could* bind +a judge tool into it that reuses the wheel's `judge_prompt` — a host-side change that leaves the wheel +API untouched. That alternative is written up in **`docs/crucible-judge-in-loop.md`**. + +The two paths trade off effort vs. shape: this doc's Phases 1–2 recover the bulk of the in-graph +efficiency (shared context, memory continuity, deliberate cadence) with a localized change to the +current single-shot author; the in-loop proposal removes the cost at its architectural source but +refactors the authoring loop. A reasonable sequence is Phase 1 now, converging toward the in-loop +design. diff --git a/docs/crucible-judge-in-loop.md b/docs/crucible-judge-in-loop.md new file mode 100644 index 00000000..8b71be06 --- /dev/null +++ b/docs/crucible-judge-in-loop.md @@ -0,0 +1,129 @@ +# Proposal (alternate) — move the Crucible judge into an author-loop tool + +**Status:** implemented (host-side). This is the **alternate** to the phased mitigation in +`docs/crucible-judge-cost.md`, and it is now the *only* judge architecture — the previous +host-driven, per-attempt judge turn has been removed. The judge becomes a **tool the rustapp author +agent calls**, matching how the Foundry (`feedback_tool`) and CVL (`property_feedback_judge`) backends +already work, removing the statelessness and the unconditional cadence at their root (the judge ~5×'d +the e2e; see the cost doc §1). + +**What shipped:** a `request_review` tool (`composer/rustapp/adapter.py`) bound into the author agent +**whenever the wheel supplies a judge for the input** (detected by probing the pure `judge_prompt` +callout — it returns `None` exactly when there is no judge), running that `judge_prompt` as an +in-session sub-agent; a `bind_standard` `_review_gate` validator that blocks `result` until the +submitted draft was accepted; the run memory tool shared across author/judge/components; and the host +loop no longer runs a separate `_judge_turn`. `crucible_app` has a component judge, so it always runs +in-loop; `echoprover` (no judge) keeps the single-shot author. The final validation below (e2e +wall-clock + verdict parity) is still pending. + +## 1. Why this is possible (correcting the cost doc's §5) + +The cost doc claimed the passive-service design *prevents* the Foundry pattern. That's wrong. The +Rust wheel is passive, but the **author loop already runs in Python**: `_author_turn` → +`run_llm_agent` → `bind_standard(...).with_tools(...)` + `run_to_completion` is a full tool-enabled +agent. The wheel only supplies the *prompt string* (`author_prompt`); Python owns the agent +machinery and the tool belt. So Python can bind a judge/`request_review` tool into that loop that +invokes the wheel's existing `judge_prompt` — a **host-side change; the wheel API does not change**. +The judge is pure LLM review (no toolchain), so nothing about the sandbox boundary ("the LLM controls +file contents, never argv") is in the way — that constraint only pins `compile`/`validate`, which stay +host-driven. + +## 2. The design + +Model it on Foundry's author (`composer/foundry/author.py`): a stateful +write → review → revise → publish agent, with the judge behind a tool and a completion gate. + +**For a judge-enabled wheel, the authoring turn becomes a richer agent** (today it's a single-shot +`doc`-style agent whose final answer *is* the artifact). It gains: + +- **A draft buffer** — `put_spec` / `get_spec` tools (the peers of Foundry's `put_test_raw` / + `get_test`). The author writes its candidate spec into agent state instead of returning it as the + final answer. +- **A `request_review` tool** (the peer of `feedback_tool`). When called, it runs the **judge + sub-agent** built from the wheel's `judge_prompt(input, current_draft)`, with the run's memory + (`ctx.get_memory_tool()`), `get_spec`, and a *bounded* source/rag belt; it returns the feedback and + records whether the **current** draft was accepted. This is the direct analogue of + `_build_feedback_thunk`. +- **A completion gate** — a `bind_standard` `validator` (the mechanism CVL/Foundry use, e.g. + `did_rough_draft_read`) that **rejects finalization unless the judge accepted the current draft**. + The author must clear review before it can publish, so it can't skip the judge. + +**The host loop simplifies.** `RustFormalizer.formalize` / `author_and_compile` drop the separate +`_judge_turn`; they run the (judge-integrated) author once to get a **judge-accepted** draft, then +`compile` → `validate` as today. A build failure still re-invokes the author (the host loop), but a +*judge* rejection is now handled **inside** the author session — the author self-revises against the +feedback rather than being re-invoked fresh. + +**The wheel is unchanged.** `judge_prompt(input, spec)` is reused verbatim as the sub-agent's prompt; +no new callout. A descriptor flag (e.g. `judge_in_loop: bool`) can gate the behavior so the host +knows whether to bind the tool. + +**Non-judge wheels are unaffected.** `echoprover` returns `judge_prompt → None`; the author keeps its +current single-shot `doc` shape and none of the buffer/review/gate machinery is bound. + +### Sketch + +``` +run_llm_agent(env, author_prompt, ..., judge=JudgeSpec(module, input)) # judge optional + bind_standard(builder_heavy, ST, validator=require_judge_accepted) + .with_tools(bounded_source + rag + memory + [put_spec, get_spec, + request_review, result]) + # request_review -> run_to_completion(judge sub-agent from module.judge_prompt(input, get_spec()), + # tools = memory + get_spec + bounded source/rag) + # result -> allowed only if the last request_review accepted the current draft +returns: the judge-accepted spec +``` + +## 3. Why it fixes the cost (vs. the phased plan) + +| Cost driver (cost doc §1) | Phased plan | In-loop judge | +|---|---|---| +| Judge re-explores from scratch (53 `code_explorer`) | inject context + restrict tools + add memory | **memory shared in-session + author's context already present** → same effect, structurally | +| Judge runs every attempt (unconditional) | skip on compile-retries, cap rounds | **subsumed** — the author calls review deliberately, and self-revises in-session; no per-attempt host judge | +| Judge on heavy model | optional opt-down to lite (divergent) | orthogonal — can still restrict the sub-agent's tools; heavy matches CVL/Foundry | + +Net: the in-loop design *is* the endpoint the phased plan approximates. It recovers the in-graph +efficiency (shared memory, deliberate cadence, self-revision) that CVL/Foundry get for free — and +gives one mental model across all three backends. + +## 4. What this makes vestigial + +- `_judge_turn` / `_parse_judge` (host-side judge turn) — replaced by the `request_review` tool. +- The `FailureKind::Judge` re-author path (`judge_revise_suffix`) — the author now self-revises, so a + judge rejection no longer re-enters `author_prompt`. `FailureKind` collapses back toward compile-only + (keep it for build failures). +- The host-emitted `judge` verdict event — replaced by the review tool's own events (as Foundry's + feedback UI does). + +These were added for the current host-driven judge; adopting this proposal would remove or repurpose +them. + +## 5. Risks / tradeoffs + +- **More machinery in the *generic* host.** The `rustapp` author gains a buffer + review tool + gate, + moving it closer to Foundry's bespoke author. Genericity is preserved (all driven by the wheel's + `judge_prompt`), but the shared loop is heavier and less "single-shot". +- **Loop-ownership shift.** The author/judge micro-loop moves from the host into the agent; the + `docs/rust-backend-api.md` "Python owns the loop" story now applies to author→compile→validate, with + the author owning the inner review cycle. Compile/validate stay host-driven. +- **Gaming the judge.** The author decides *when* to review; the completion gate (must be accepted + before `result`) is what prevents a perfunctory pass — same safeguard Foundry relies on. +- **Bigger, higher-risk change** than the phased plan's localized tweaks; needs its own e2e validation. + +## 6. Rollout + +1. Add the buffer + `request_review` + gate to `run_llm_agent` behind a `judge` parameter; wire the + judge sub-agent from `module.judge_prompt`. Add the run memory tool to the author loop. +2. Gate on a descriptor flag (`judge_in_loop`) so it can be enabled per wheel and A/B'd against the + current host-driven judge. +3. Remove `_judge_turn` from the host loop for in-loop wheels; keep the compile/validate host steps. +4. Validate on `test_crucible_e2e_gate`: **wall-clock + verdict parity** (12 GOOD / 1 BAD hold; the + fee-oracle class still caught), iterating on a smaller scenario first (the gate is ~2 h / paid). + +## 7. Recommendation + +If we only want the cost down with minimal risk, the phased plan (cost doc) is the pragmatic path. If +we're willing to refactor the authoring loop, this proposal is the better long-term shape: it unifies +Crucible with CVL/Foundry, deletes the bespoke host-driven judge plumbing, and removes the cost at its +architectural source. A reasonable middle path is to ship phased **Phase 1** now (immediate relief) +and treat this proposal as the target the loop converges to. diff --git a/docs/crucible-toolchain-versioning.md b/docs/crucible-toolchain-versioning.md new file mode 100644 index 00000000..f6194708 --- /dev/null +++ b/docs/crucible-toolchain-versioning.md @@ -0,0 +1,212 @@ +# Design options — Crucible toolchain versioning & container packaging + +**Status:** options for discussion. No decision yet. Sibling of +[crucible-application.md §6.1](./crucible-application.md) (version compatibility) — this doc +zooms in on *how we package the toolchains and pick versions*, which §6.1 flagged and left open +("a version table replaces the hardcoding later"). + +--- + +## 1. The problem + +To fuzz a Solana program with Crucible we do two builds, both version-sensitive: + +- **Build the program to sBPF** with `cargo-build-sbf` (from the Solana **platform-tools**, which + bundle their own sBPF `rustc`). +- **Assemble + build a native harness crate** ([composer/crucible/harness.py](../composer/crucible/harness.py)) + that **path-depends on the program crate** (`features = ["no-entrypoint"]`) and also depends on the + **Crucible crates**, **`anchor-lang`**, the **`solana-*`** crates, and **`libafl`**; then run it via + `crucible`. + +The binding constraint is the harness↔program link: **one version of `anchor-lang` and of the +`solana-*` crates must satisfy both the program and the harness** (two versions of `anchor-lang` in +one dependency graph don't link). So per target program, several axes must line up at once: + +| Axis | Set by | Notes | +|---|---|---| +| host `rustc` | `rust-toolchain.toml` / ambient | builds the harness natively + runs `cargo` | +| sBPF `rustc` + Solana SDK | **platform-tools** version | builds the program `.so` | +| `solana-*` crates | program's `Cargo.lock` | harness must match | +| `anchor-lang` | program's `Cargo.lock` / `Anchor.toml` | harness must match | +| `crucible` crates + CLI | our choice | must be compatible with the above | +| `libafl` | `crucible` pins it | rides the crucible version | + +So a run needs a **mutually-compatible `(rustc, platform-tools, solana, anchor, crucible)` combo** +that also *matches the program's* solana/anchor — the §6.1 compatibility matrix. + +**Where we are today:** the harness pins `anchor-lang 1.0.1` / `solana 3.0` / `libafl 0.15.1` and the +Crucible crates via a **local checkout path-dep** (`CrucibleDep`, hardcoded), and the AutoProver +container ships **none** of this toolchain (no Rust, no platform-tools, no `anchor`, no `crucible`, no +checkout, and the `crucible_app` wheel isn't in `uv sync`). So "package the toolchain" and "pick the +versions" are the same conversation. + +--- + +## 2. What every option must solve (cross-cutting) + +Independent of packaging, these are needed by all three options and should be built once, shared: + +1. **Version detection** — read the program's `rust-toolchain.toml` (host `rustc`), `Cargo.lock` + (authoritative `solana-*` / `anchor-lang` versions), and `Anchor.toml` (anchor CLI) to determine + its required combo. Edge cases: workspaces, unpinned deps, no lockfile. +2. **A compatibility table** (§6.1) — the single source of truth mapping *program versions → + a compatible `crucible` version + the toolchain set*. Every option consults it; they differ only + in what they *do* with the result. +3. **Sourcing the Crucible crates** — today a local checkout. Versioning needs a stable story: git + tags/commits, or published crates.io releases, or vendored-per-combo. (Open: are the crucible + crates published, or checkout-only?) +4. **Offline + the sandbox** — the confined build is offline ([command-sandbox.md §5](./command-sandbox.md)); + the trusted warm/fetch (network) must have *the resolved combo's* deps available. A bigger version + matrix means more to vendor/provision. +5. **Two Rust toolchains** — host (harness + `cargo`) and sBPF (platform-tools). Both must be present + and version-correct; they are installed and selected differently (`rustup` vs `agave`/`solana-install`). + +--- + +## 3. Prior art in AutoProver (how the EVM backends handle this) + +AutoProver already ships two EVM toolchain strategies. Neither faces Crucible's *coupled* matrix, but +both are concrete precedents for the packaging axis — and one maps to each end of the options below. + +### 3.1 CVL / Certora-Prover path — "bake every version, select by name" + +- The image installs **every released `solc`** as `solcX.Y` (`0.8.29` → `solc8.29`), checksum-verified + against Solidity's official binary index, via [scripts/install_solc.py](../scripts/install_solc.py) + at build time (a `COPY` + `RUN` in [scripts/Dockerfile](../scripts/Dockerfile)); a bare `solc` + symlink points at a default (`0.8.29`). +- The compiler is chosen by an explicit **`--solc-version` arg** (default `8.29`) threaded through the + pipeline ([cli/tui_pipeline.py](../composer/cli/tui_pipeline.py#L85) → [prover/runner.py](../composer/prover/runner.py#L112) + `--solc `; the LLM's prover tool also names `solcX.Y`, [tools/prover.py](../composer/tools/prover.py#L71)). + It is **caller-specified, not auto-detected** from the pragma. +- The verification itself runs in the **Certora cloud** — the Dockerfile explicitly bundles no local + prover ("Cloud Certora Prover only"). +- **Shape:** one version axis (`solc`), all versions baked into one image, per-run selection by name. + This is essentially the image-baked end of **Options 1–2**, and operational proof that "bake all + + select" works in this project today. + +### 3.2 Foundry / `forge` path — "delegate to the tool's own version manager" + +- The backend ([composer/foundry/](../composer/foundry/)) shells out to `forge test --json` on a + staged draft test in the user's project ([composer/foundry/runner.py](../composer/foundry/runner.py)) + and parses the JSON; publish requires a green *unseeded* run. +- Version management is **forge's**: forge reads `foundry.toml`/pragma and uses its own solc manager + (`svm`) to fetch the matching compiler, and resolves libraries its own way. AutoProver never picks + or installs `solc` for this path. +- **`forge` is *not* in the Dockerfile** (no `forge`/`foundryup`/`svm`). So the Foundry backend is + **not packaged in the container today** — the *same gap* Crucible has; forge just self-manages once + it's present. +- **Shape:** delegate to the ecosystem's own version manager. This is the **Option-3** pattern — + except Crucible has no single umbrella manager, so it would compose the ecosystem's own managers + (`rustup`, `agave`/`solana-install`, `avm`, plus a `crucible` fetch) to play forge's role. + +### 3.3 The difference that makes Crucible harder + +EVM has **no harness↔contract link coupling**: `forge` (or the prover) compiles the contract and the +test with *one* `solc` — there is no separate test artifact that must *link the contract's compiled +bindings at a matching dependency version*. Crucible's harness **path-depends on the program crate**, +which is exactly what turns one knob (`solc`) into the five-axis coupled matrix of §1. So the `solc` +single-knob model transfers to Crucible only if the compatible matrix turns out small; otherwise +Crucible is closer to the forge/version-manager model. + +### 3.4 A fourth angle the prover suggests + +The prover keeps the image light (`solc` only) by offloading the heavy tool to the **Certora cloud**. +The Crucible analog — a fuzzing **service** the container calls rather than an in-image toolchain — +is a distinct packaging option worth keeping in view: it sidesteps in-image versioning entirely, at +the cost of standing up and *itself* versioning a service. Not fleshed out here, but noted so it isn't +lost. + +--- + +## 4. The options + +### Option 1 — One blessed combo per image (pin-and-bump) + +Bake exactly one mutually-compatible toolchain set into the image (host `rustc` + platform-tools + +`anchor` + `crucible` CLI + a crucible checkout at a pinned ref + the pinned harness deps). The image +*is* the version; there's no runtime selection. Programs that don't match the blessed combo are +**rejected fast** with a clear message. Bumping = cutting a new dated/tagged image. + +- **Selection:** none at runtime — detection is used only to *fail fast* on mismatch. +- **Crucible crates:** a checkout pinned to a tag/commit, baked in. +- **Offline:** one vendored dep set — the simplest possible. +- **Pros:** smallest image; fully deterministic + reproducible; cleanest offline story; least to build + and reason about; matches today's hardcoded pins, so it's the shortest path to *runnable-in-container*. +- **Cons:** supports a single combo only; version drift means real programs (on other anchor/solana) + don't run; we must track upstream and re-bless; it punts the matrix entirely. +- **Best when:** early days, a narrow set of target programs, "get it running" is the priority. + +### Option 2 — A matrix of blessed images, selected per program + +CI builds **N images**, each a blessed combo (the tag encodes it, e.g. +`autoprover-crucible:sol3.0-anchor1.0-cruX`), enumerated from the compatibility table. A dispatcher +detects the program's versions, maps them to a tag, and runs that image. Each image is internally an +Option-1 image (single combo, reproducible); the *fleet* covers the matrix. + +- **Selection:** at dispatch time — pick the image; nothing version-variable inside it. +- **Crucible crates:** each image pins its combo's ref. +- **Offline:** each image vendors its own combo — clean, per image. +- **Pros:** covers the matrix while keeping each image simple + reproducible; no in-container version + juggling; adding a combo = one table row + one CI build; a bad combo can't affect the others. +- **Cons:** needs a build-matrix + image registry + an image-selection layer; N images to build, + store, and refresh on every crucible/solana/anchor bump; a program on an un-built combo isn't + supported until CI adds it; leans hard on reliable version detection. +- **Best when:** several distinct program families/versions, CI + a registry are available, and + reproducibility matters. + +### Option 3 — Thin base + on-demand toolchain provisioning + +Ship a thin base image; a **trusted provisioning step** at run/job start installs the exact needed +versions — `rustup` (host toolchain), `agave`/`solana-install` (platform-tools), `avm` (anchor), +and a `crucible` fetch/build at the resolved ref — keyed by the detected versions + the table, cached +on a persistent volume. One image, arbitrary combos, provision-what-you-need. + +- **Selection:** at runtime — resolve, then provision. +- **Crucible crates:** fetched at the resolved ref on demand (cached). +- **Offline:** provisioning is the network-on trusted pre-step (like the sandbox's `cargo fetch` + warm); the confined build still runs offline against the provisioned set. +- **Pros:** one image; supports the full (long-tail) matrix without pre-building every cell; only + materializes what's used; extends to new versions with no image rebuild. +- **Cons:** per-run (or per-new-combo) provisioning latency + network; a version-manager layer to + build and maintain; cache/volume lifecycle; the most moving parts at run time; weakest + reproducibility guarantee (provisioning can drift unless carefully pinned). +- **Best when:** a wide/long-tail of program versions, and run-time flexibility is worth the + provisioning cost. + +--- + +## 5. Suggested path (not a decision) + +- **Start with Option 1.** It's the smallest lift, matches the current hardcoded pins, and gets + Crucible *runnable in the container* — while forcing us to build the two shared pieces (§2.1 + detection + §2.2 the compatibility table) that 2 and 3 also need. Even Option 1 needs detection, to + fail fast on a mismatch instead of producing a confusing link error. +- **Evolve to Option 2** as the target-program set widens. Because each matrix cell *is* an Option-1 + image, this is additive (new table rows + CI builds), not a rewrite. +- **Hold Option 3** for when pre-building every matrix cell becomes impractical (a genuine long tail), + or when a customer's exact combo can't be predicted ahead of time. +- Each end has **direct in-project precedent** (§3): Options 1–2 are the baked-`solc` model, Option 3 + is the `forge`/`svm` delegate-to-a-version-manager model — so neither is a leap into the unknown. +- **The seam that keeps this evolvable:** separate **version resolution** (detect → table → combo) + from **provisioning** (bake / select-image / install-on-demand). Resolution is shared code; only + the provisioning backend swaps. This mirrors the ecosystem/backend split the project already uses, + and the `SandboxProvider` seam pattern from Phase 6. + +--- + +## 6. Open questions + +1. **Are the Crucible crates published (crates.io / a private registry), or checkout-only?** Decides + how a combo sources them (tag vs release vs vendored) — and how heavy Option 3's fetch is. +2. **Exact-match vs compatible-range** — does the harness need the *exact* `solana`/`anchor` the + program uses, or a semver-compatible range? Fewer, wider combos vs many exact ones — this sizes the + whole matrix (and Options 2 vs 3). +3. **Detection reliability** — is `Cargo.lock` always present/authoritative? How do we handle + workspaces, path/git deps, and programs that don't pin a `rust-toolchain`? +4. **Budgets** — image size + registry cost (favors 3) vs run-time provisioning latency (favors 1/2). +5. **Compatibility-table ownership** — who curates it and at what cadence as crucible / solana / + anchor release? This is the recurring maintenance cost common to all three. +6. **`crucible_app` wheel packaging** — orthogonal but adjacent: the maturin wheel must be built into + whichever image(s) we ship (needs the host Rust toolchain in the build layer), and the + `run-confined` sandbox launcher likewise (already a Dockerfile stage). Fold this into whichever + option is chosen. diff --git a/docs/crucible-unit-granularity.md b/docs/crucible-unit-granularity.md new file mode 100644 index 00000000..14144726 --- /dev/null +++ b/docs/crucible-unit-granularity.md @@ -0,0 +1,240 @@ +# Crucible unit granularity: per-instruction vs whole-program (global) fuzzing scenarios + +> **Status: implemented (commit fd51700), then collapsed to a single harness (prototype).** +> Solana first used global extraction with per-invariant fuzz units (recommendation §6). It now +> **collapses all invariants into one whole-program harness + run** — the "single whole-program +> unit" option §3/§6 identified and deferred *for attribution reasons that finding-level +> observability now covers* (each `fuzz_assert` is tagged with its property title, so a +> counterexample names the property it refutes). This is the `Ecosystem.collapse_units` path +> (`composer/pipeline/ecosystem.py` → `SOLANA`; `_extract_all` in `core.py`) with a single +> `c_invariants` fn (`rust/crucible-app/src/lib.rs`). Trade-off realized: one build + one fuzz run +> for the whole program (vs N). Attribution is preserved: each property is still its own report +> row (`c_`) sharing the single fuzz `target` (`Unit.target` in the SDK), and the host +> (`_attribute_run`) pins a counterexample to the property the finding names (each assertion is +> tagged with its property title), holding the rest GOOD — so a single run yields per-property +> verdicts. Measured e2e: **0:59:22** for the whole vault (13 invariants), vs 1:33:20 for the +> per-invariant in-loop run and 1:53:04 for the original — one build + one campaign instead of 13. +> The rest of this note is the original decision record. + +Design note for [crucible-application.md §10 Q1](./crucible-application.md) — "are we +using the right unit granularity for a fuzzer?" It describes moving Crucible's +scenario generation from **per-instruction** units to a **whole-program (global)** +context, the trade-offs, and how each option lines up with the Foundry and CVL/prover +backends. No code change yet — this is to decide the shape. + +## 1. Background: what a "unit" is in each backend + +The shared driver (`composer/pipeline/core.py`) fans out over +`ecosystem.units(main)`, and for each unit runs **property extraction** (with that +unit's context) → **formalization** (author + gate an artifact) → a per-unit +**verdict** in the report. What a "unit" *is* differs by ecosystem: + +| Backend | Unit (`ecosystem.units`) | Granularity | Source | +|---|---|---|---| +| CVL / prover (EVM) | `ContractComponentInstance` | per **component** — a semantic cluster of the contract's behavior, produced by system analysis | `_evm_units` | +| Foundry (EVM) | `ContractComponentInstance` | per **component** (same as CVL) | `_evm_units` | +| **Crucible (Solana)** | `SolanaInstructionInstance` | per **instruction** — one unit per entry point, a flat list, **no grouping** | `_solana_units` | + +So EVM already does a coarsening step — the analysis groups related functions into a +handful of named `ContractComponent`s (`composer/spec/system_model.py:61`). Solana does +**not**: `SolanaProgram.instructions` is a flat list (`composer/spec/solana/model.py:99`) +and `_solana_units` emits one unit per instruction (`composer/pipeline/ecosystem.py`). +Crucible therefore sits at the *finest* granularity of the three. + +## 2. What is already global today (important) + +A subtlety that reframes the question: **the Crucible harness is already +whole-program.** The setup session authors one shared `Fixture` with `action_*` +methods that drive the *entire* program, and the fuzzer runs in `explore` mode — +`crucible run --mode explore` drives a **random sequence of +actions across all instructions** and evaluates the test after each step +(`rust/crucible-app/src/lib.rs`, `TEST_CHEAT_SHEET`). + +What is *per-instruction* today is only: + +1. **Property selection** — `run_property_inference` is called with a single + `SolanaInstructionInstance` as context, so the model proposes properties framed + around *that instruction* (`composer/pipeline/core.py` `_extract_all`). +2. **The authored test** — one `#[invariant_test] fn c_` per instruction, + asserting that instruction's properties (against global state, over the global + action sequence). +3. **The verdict / report row** — one GOOD/BAD per instruction. + +So the fuzzing *engine* is already global; the *authoring and bookkeeping* are +sharded per instruction. The open question is really: **should property generation +and the harness be framed globally, or kept sharded per instruction?** + +## 3. The proposed change: global (whole-program) scenarios + +Generate properties and harnesses from a **whole-program view** — the model sees the +entire program (all instructions, accounts, PDAs, authorities, cross-instruction +flows) at once and proposes **global invariants** ("total deposits always equal vault +balance", "only the admin can ever change the fee", "no sequence of actions drains a +user's escrow") rather than instruction-local properties. The fuzzer explores action +sequences (as it already does) and checks these invariants after every step. + +Concretely, in the current architecture this is a small set of localized changes: + +- **`ecosystem.units` for Solana** returns **one whole-program unit** (the + `SolanaProgramInstance` itself), instead of one per instruction. The driver then + fans out over a single unit. +- **Property extraction context** becomes the whole program (it already receives the + analyzed model as front-matter; the *unit* context widens from one instruction to + all). The prompt shifts from "properties of this instruction" to "global invariants + of this program". +- **Harness authoring** produces a set of global `#[invariant_test]` functions in one + crate (the fixture is unchanged — it is already whole-program). +- **Verdict / report** granularity becomes per-**invariant** (or one program-level + row) rather than per-instruction. + +A middle option (call it **grouped**) mirrors EVM: have Solana analysis group +instructions into a few semantic **components** (like `ContractComponent`) and fan out +per component. That lands between per-instruction and whole-program. + +There are thus three points on the spectrum: + +``` +per-instruction (today) → per-component (EVM-style grouping) → whole-program (global) + finest, most fan-out middle coarsest, one unit +``` + +## 4. Pros and cons + +### Global (whole-program) — pros +- **Matches the tool's semantics.** A coverage-guided fuzzer explores the whole state + machine; invariants are inherently cross-instruction. Framing them globally removes + an artificial instruction boundary and lets the model express properties that *span* + instructions (deposit-then-withdraw conservation, auth escalation across a sequence). +- **Fewer, higher-value properties.** Per-instruction extraction tends to produce + near-duplicate or trivially-local assertions for each entry point; a global view + yields a smaller set of meaningful system invariants (closer to how an auditor + writes them). +- **No redundant re-fuzzing.** N per-instruction tests each fuzz the *same* global + action space for the full budget; one global harness fuzzes it once, so the time + budget buys deeper exploration instead of N shallow re-runs of the same sequence. +- **Cheaper authoring.** One harness authored/validated instead of N (each currently + pays a compile + dry-run + fuzz cycle — the dominant cost). +- **Cleaner story for the shared-crate serialization.** Per-instruction units share one + harness crate and therefore serialize their builds today (parity gap #4 / + command-sandbox.md §10); with one unit there is nothing to serialize. + +### Global — cons +- **Loses per-instruction fan-out and its parallelism/caching.** The driver's + per-unit concurrency and result cache key off units; one unit means no per-unit + parallelism (though today the shared crate already serializes builds, so little is + lost in practice) and coarser caching. +- **Coarser attribution.** A per-instruction verdict tells you *which entry point* a + counterexample implicates; a single global run says "some sequence violated invariant + X" and leans on the counterexample trace for locality. Mitigated by keeping + per-**invariant** units (still global context, but each invariant is its own report + row) rather than collapsing to one. +- **Bigger single harness.** One crate with many invariants is a larger authoring + target; a compile error blocks all of them (vs isolating a failure to one + instruction's test today). +- **Diverges from the generic driver's per-component assumption.** The EVM backends + fan out per component; making Solana emit one unit is fine (the driver is + unit-agnostic) but the report/labels read "1 component", which is a cosmetic mismatch + with the "instructions" framing. + +### Per-instruction (today) — pros +- Reuses the generic fan-out, caching, and per-unit report rows unchanged; isolates + authoring failures; gives instruction-level attribution. + +### Per-instruction — cons +- The artificial boundary described above: duplicated/local properties, N× redundant + fuzzing of the same global space, N authoring cycles, and it can't express + cross-instruction invariants naturally. + +## 5. Comparison to Foundry and CVL + +| Dimension | CVL / prover | Foundry | Crucible today (per-instruction) | Crucible global (proposed) | +|---|---|---|---|---| +| Unit of fan-out | per component | per component | per **instruction** | per **program** (or per invariant) | +| Property framing | per-component rules | per-component; `test_*` per-function + `invariant_*` whole-contract | per-instruction | **whole-program invariants** | +| Execution model | symbolic — reasons over **all** states, no sequences | concrete: property fuzzing (per-function) **and** stateful invariant fuzzing (random call sequences, whole-contract) | concrete: `explore`-mode random action sequences (whole-program) | same engine, global framing | +| Cross-unit invariants | natural (any state) | natural for `invariant_*` (stateful) | awkward — split across instruction units | **natural** | +| Attribution on failure | the violated rule | the failing test / invariant | the instruction's test | the violated invariant (+ trace) | + +The key alignment: **Foundry's `invariant_*` stateful fuzzing is already +whole-contract** — its runner calls all functions in random sequences and checks +invariants globally (`composer/templates/foundry_property_generation_system_prompt.j2`: +"An `invariant_*` function is run by foundry's *stateful* fuzzer"). Crucible's +`explore` mode is the direct analogue. So a **global** Crucible framing makes Crucible +match Foundry's *fuzzing* model, while the *authoring fan-out* (per component) is a +separate axis that Foundry keeps per-component mainly because it also emits +per-function `test_*` properties — which Crucible does not. + +CVL/prover is the outlier: no action sequences at all (symbolic over all states), so +its per-component split is about *proof modularity*, not scenario construction — not a +useful precedent for how a fuzzer should be scoped. + +Net: the fuzzing backends (Foundry stateful, Crucible) both *want* whole-program +scenario semantics; only Crucible currently imposes a per-instruction authoring shard +that neither the engine nor the Foundry precedent requires. + +## 6. Recommendation + +Move Crucible to a **global scenario context** for property generation and harness +authoring, but keep **per-invariant units** rather than collapsing to a single opaque +run — i.e. the model sees the whole program and proposes a set of global invariants, +and each invariant is a unit (its own harness fn + report row). This preserves +attribution and the report's per-row structure while fixing the artificial +instruction boundary and the N× redundant fuzzing. + +Staging (each independently shippable): + +1. **Widen the extraction context** to whole-program (prompt + the unit the driver + passes to `run_property_inference`) while still emitting per-instruction units — a + low-risk first step that improves property quality without changing fan-out. +2. **Switch `ecosystem.units` (Solana) to per-invariant** (extract global invariants + first, then fan out over them), or to a single whole-program unit if per-invariant + proves awkward. This is the `units` / `render_unit` hook §10 Q1 anticipated. +3. **Re-fuzz vs cache** (ties into §10 Q4): with one global action space, cache the + authored harness (deterministic) and re-fuzz on demand / record the seed — more + important now that a single budget covers the whole program. + +Interactions to keep in view: per-invariant units mean N harnesses build + fuzz +serially on the shared crate (see §8 on why that's hard to parallelize), and it pairs +naturally with coverage-as-signal (§10 Q6). + +## 7. Concurrency: why per-invariant fuzzing serializes (and the only clean fix) + +With N per-invariant harnesses the build+fuzz runs serialize (a single build semaphore +in `CrucibleFormalizer`, because they share one crate `fuzz//`). Making them +parallel is **harder than it looks**, and the obvious "one crate per invariant" does +NOT work: + +- **Separate `target/` per crate** → every crate recompiles the heavy `litesvm` / + `libafl` / `solana` deps from scratch (minutes, ~900 MB each). N× the build work — a + net regression, not a win. +- **Shared `CARGO_TARGET_DIR`** → the deps compile once, BUT Crucible hardcodes the + harness binary name (`[[bin]] name = "invariant_test"`, and `find_fuzz_binary` expects + it), so every crate compiles to the *same* `…/target/release/invariant_test` — the + builds clobber each other and concurrent runs race on the binary. **Unusable.** + +The only clean path to parallel fuzzing is **`crucible run --binary-in`**: build each +invariant's binary *serially* (shared crate/target, cheap incremental, no collision), +copy it to a per-invariant path, then fuzz all binaries *in parallel* via `--binary-in`. +That requires splitting build from fuzz and orchestrating it in the Crucible backend +(not the per-batch decider, which does author→build→fuzz→publish as one unit) — a +non-trivial formalize-phase redesign. + +**Decision: deferred.** The serialized-fuzz cost is small at the e2e's 12 s budget (the +27-min e2e is dominated by LLM authoring + incremental builds, which this doesn't +parallelize), and only becomes the bottleneck at **production fuzz budgets** (60–300 s), +where N × serial fuzzing dominates. Revisit then, as a scoped `--binary-in` project. +This supersedes the "crate-per-component" framing of parity gap #4 / command-sandbox.md +§10 — the parallelism lever is fuzz-via-`--binary-in`, not per-component crates. + +## 8. Open sub-questions for this change +- **Per-invariant vs one-unit:** is the extra report granularity of per-invariant units + worth a second extraction round (invariants first, then fan-out)? Prototype both. +- **How many invariants** should the model target for a program, and does the fixed + fuzz budget get split per-invariant or shared? (Fewer, deeper is the fuzzer's + preference.) +- **Attribution:** is the counterexample trace enough to locate the offending + instruction, or do we still want the model to tag each invariant with the + instructions it stresses? +- **EVM symmetry:** should Solana analysis gain a `ContractComponent`-style grouping so + the *grouped* middle option is available, or is whole-program sufficient for Solana's + typically-smaller instruction sets? diff --git a/docs/ecosystem-abstraction.md b/docs/ecosystem-abstraction.md new file mode 100644 index 00000000..ed6daf10 --- /dev/null +++ b/docs/ecosystem-abstraction.md @@ -0,0 +1,468 @@ +# Proposal — The Ecosystem Abstraction (EVM, Solana, Soroban) + +> A proposal to make AutoProver's shared pipeline parametric over an **ecosystem** — +> the blockchain/source domain being analyzed — selected by an application-level parameter +> that picks the right system model, prompts, source conventions, and validation. Today the +> "generic" pipeline is generic over the *backend* (how a property becomes a verified +> artifact) but silently hardwired to Solidity for everything else. This introduces a second, +> orthogonal axis — and factors it further into a **language** facet (Solidity, Rust) and a +> **chain** facet (EVM, Solana, Soroban), so the Rust-specific prompts and source conventions +> are written once and shared between Solana and Soroban while their blockchain-specific +> models and failure modes stay separate. +> +> Companion to [formalization-abstraction.md](./formalization-abstraction.md) (the backend +> seam), [application-abstraction.md](./application-abstraction.md) (the five pieces of an +> application), and [rust-applications.md](./rust-applications.md) (the Rust app framework +> the first Solana/Soroban backends will likely use). Status: **proposal / for review.** + +--- + +## 1. Problem & motivation + +We want to author properties for **Solana** programs (Rust/Anchor) and **Soroban** contracts +(Rust/soroban-sdk), not just EVM/Solidity. The pipeline advertises itself as backend-agnostic, +and it is — `run_pipeline` is generic over the result type `FormT` +([formalization-abstraction.md](./formalization-abstraction.md)). But "backend-agnostic" is +not "domain-agnostic." An audit of the shared spine (see §3) shows Solidity/EVM assumptions +baked into three shared places the driver owns: + +1. the **system model** — the pydantic types the analysis phase produces; +2. the **prompts** — the analysis and property-extraction templates; +3. a few **source conventions** — the fs-exclusion default, the "main contract" locator. + +None of this is tooling (no `solc`/`slither` in the shared steps — it's pure LLM + generic +file reading), so the work is types + prompts + a small driver generalization, not new +analysis engines. + +We propose making this a first-class **ecosystem** parameter, so `evm` reproduces today's +behavior exactly and `solana` / `soroban` slot in beside it without forking the pipeline — +and, because two of those three are Rust, factoring the ecosystem so the Rust-specific parts +are shared rather than copied (§2.1). + +--- + +## 2. Two orthogonal axes + +The pipeline has a front half and a back half joined by *properties*: + +``` + ┌─────────── ECOSYSTEM owns ───────────┐ ┌──────── BACKEND owns ────────┐ +source ─analyze─▶ SystemModel ─extract─▶ properties ─formalize─▶ artifact ─verdicts─▶ + (how we MODEL and REASON about (how a property becomes a + the domain: contracts vs programs, checkable, verified artifact: + storage vs accounts, reentrancy CVL+prover / foundry / a Rust + vs missing-signer) Solana verifier) + └──────── SHARED: report ────────┘ +``` + +- **Ecosystem** = the *front half*: the system-model types, the analysis + property-extraction + prompts, source conventions, and connectivity validation. +- **Backend** = the *back half*: `prepare_system` → `Formalizer` (`formalize` / `fetch_verdicts`), + documented in [formalization-abstraction.md](./formalization-abstraction.md). +- **Report** stays shared and neutral. + +The axes are conceptually independent — but they meet at the analyzed model: the backend's +`prepare_system(analyzed: App)` consumes the ecosystem's `App` type. So a **backend is written +against an ecosystem's model** (the CVL prover backend needs `SourceApplication`; a Solana +verifier needs `SolanaApplication`). An application picks a **(ecosystem, backend) pair that +agree on `App`.** That pairing is the one coupling to make explicit (§6). + +| | EVM | Solana | Soroban | +| --- | --- | --- | --- | +| Compatible backends | `prover` (CVL), `foundry` | a Rust verifier (via `rustapp`) | a Rust verifier (via `rustapp`) | +| System model | `SourceApplication` (contracts) | `SolanaApplication` (programs) — new | `SorobanApplication` (contracts) — new | +| Unit of extraction | contract component | instruction / account-validation group | contract function | + +### 2.1 Ecosystems factor into a *language* facet and a *chain* facet + +Solana and Soroban are both **Rust**; EVM is **Solidity**. Much of what the front half needs +is fixed by the *source language* — how to read/navigate it, the project layout to exclude +(Cargo vs Foundry), the language-level failure modes (Rust: integer overflow, `panic!` / +`unwrap` / `expect` aborts, ownership) — and is identical across every chain that uses that +language. The rest is fixed by the *chain/platform*: the system model, the storage and +authorization semantics, the platform failure modes. So an ecosystem is a **composition**: + +```text + ecosystem = language facet ⊕ chain facet + evm = solidity ⊕ evm + solana = rust ⊕ solana ┐ share the SAME rust language facet + soroban = rust ⊕ soroban ┘ (prompts, fs conventions, overflow/panic) +``` + +| Facet | Owns | `solidity` | `rust` (shared by `solana` + `soroban`) | +| --- | --- | --- | --- | +| **Language** | fs-exclusion default, `code_explorer` prompt, source-navigation framing, the language-level failure-mode prompt fragment | assembly / delegatecall, checked-arith caveat | integer overflow/underflow, `panic!`/`unwrap`/`expect` aborts, ownership/borrow | +| **Chain** | system-model type, connectivity validation, main-unit locator + units, the platform failure-mode prompt fragment, SDK conventions | contracts, storage, ERC standards | *differs per chain* — see §8 | + +The payoff (the Soroban question directly): **the Rust language facet is authored once and +shared by both Solana and Soroban** — the Rust source conventions, the `code_explorer` prompt, +and the Rust failure-mode prompt fragment. Only the chain facet differs between them. + +One honest caveat: the sharing is **not strictly hierarchical.** Solana and Soroban share the +Rust *language*, but Soroban's *model* — a contract that owns typed storage, with explicit +authorization and cross-contract calls — is closer to EVM's than to Solana's (programs +operating on externally-passed accounts). So facets are best expressed as **composable prompt +fragments keyed by concern** (§4.1), not a rigid two-level inheritance: `soroban` pulls the +`rust` language fragments *and* may reuse EVM-flavored "contract-owns-storage / authorization" +analysis fragments, while `solana` does not. + +--- + +## 3. What is Solidity-specific today (audit) + +Condensed from the audit; all in the *shared* pipeline, not the backends. + +| Concern | Where | Ecosystem-specific? | +|---|---|---| +| System-model types (`ExplicitContract`, `solidity_identifier` + regex, `ContractComponent.state_variables`/`external_entry_points`, `ContractSort`, EOA `ExternalActor`) | [system_model.py](../composer/spec/system_model.py) | **Yes** — EVM-shaped | +| Driver pins the model: `run_component_analysis(ty=SourceApplication)`, `prepare_system(analyzed: SourceApplication)`, `main_instance` matching `solidity_identifier`, `_extract_all` → `ContractComponentInstance`, the "explicit contract instance with this solidity identifier" `extra_input` | [core.py](../composer/pipeline/core.py) | **Yes** — hardcoded | +| Analysis prompts ("Smart Contracts", `solidity_identifier` block, `ContractSort` deploy semantics, ERC20/4626/721) | `application_analysis_system.j2` / `application_analysis_prompt.j2` | **Yes** — heavy rewrite | +| Property-extraction prompts (reentrancy, oracle manipulation, MEV, storage layout, checked arithmetic/`uint256`) | `property_analysis_system_prompt.j2` / `property_analysis_prompt.j2` | **Yes** — failure-mode vocabulary | +| Connectivity validation (contract/component/actor shape) | `_validate_connectivity` in [system_analysis.py](../composer/spec/system_analysis.py) | **Yes** — structure reusable, types EVM | +| fs-exclusion default (`lib/`, `test/`, `.sol` carve-out) | `FS_FORBIDDEN_READ` [util.py:59](../composer/spec/util.py) | **Yes** — but already a per-input param | +| `code_explorer` prompt ("smart contract source code") | [code_explorer.py](../composer/spec/code_explorer.py) | Cosmetic | +| Source tools (`fs_tools`, `code_explorer`, `code_document_ref`) | [source_env.py](../composer/spec/source/source_env.py) | **No** — language-neutral, read Rust fine | +| `backend_guidance` ("what's expressible downstream") | [prop_inference.py](../composer/spec/prop_inference.py) | **No** — already backend-supplied | +| Report (`Verdict`, `RuleName`, `unit_file`, `Outcome`) | [report/collect.py](../composer/spec/source/report/collect.py) | **No** — neutral | + +Two useful facts fell out: `run_component_analysis` is *already* generic (`[T: BaseApplication]`) +— only the driver pins `SourceApplication`; and `SolidityIdentifier`'s regex already accepts +Rust identifiers, so it's not a hard blocker (just misnamed). + +--- + +## 4. The seam: a `Language` facet and a `Chain` facet + +Two small protocols, composed. The **chain** carries its **language**; the driver consumes the +composed ecosystem (which is just a resolved chain). Sketch (illustrative, not final signatures): + +```python +# composer/pipeline/ecosystem.py +type LanguageTag = Literal["solidity", "rust"] +type ChainTag = Literal["evm", "solana", "soroban"] + +@dataclass(frozen=True) +class PromptPair: + system: str # j2 template name + initial: str + +class Language(Protocol): + """Shared by every chain that uses this source language.""" + name: LanguageTag + default_forbidden_read: str # Cargo layout vs Foundry layout + code_explorer_prompt: str # "Rust source" vs "Solidity source" + failure_modes_partial: str # j2 partial: language-level failure modes (overflow, panics …) + +class Chain[App: BaseApplication, Main, Unit](Protocol): + name: ChainTag + language: Language # <-- the shared facet (RUST for solana AND soroban) + + # --- domain model --- + system_model: type[App] # the analyzed pydantic type + def validate_analysis(self, app: App, expected_main: str | None) -> list[str]: ... + def locate_main(self, app: App, source: SourceCode) -> Main: ... + def units(self, main: Main) -> list[Unit]: ... + + # --- prompts (chain templates that compose in the language partials, §4.1) --- + analysis_prompts: PromptPair + property_prompts: PromptPair + +type Ecosystem = Chain # an ecosystem is a chain that carries its language +``` + +`Main`/`Unit` generalize today's `ContractInstance` / `ContractComponentInstance` — thin index +wrappers over `App` that the driver hands to the backend (`to_artifact_id`, `prepare_system`) and +to property inference. For EVM they *are* those types unchanged. + +A registry selects by chain tag; `RUST` and `SOLIDITY` are the shared language singletons: + +```python +RUST = _RustLanguage(...) # authored ONCE +SOLIDITY = _SolidityLanguage(...) +ECOSYSTEMS: dict[ChainTag, Ecosystem] = {"evm": EVM, "solana": SOLANA, "soroban": SOROBAN} +# where SOLANA.language is RUST and SOROBAN.language is RUST — the same object. +``` + +### 4.1 Prompt composition — how the Rust prompts are shared + +Prompts are **assembled from fragments with Jinja2 includes/inheritance**, not duplicated. A +chain's property template pulls the shared language fragment, then adds its own: + +```jinja +{# composer/templates/solana/property_prompt.j2 (soroban/ is identical but for the last include) #} +{% extends "property_prompt_base.j2" %} +{% block failure_modes %} + {% include "rust/_failure_modes.j2" %} {# SHARED: overflow, panics, unwrap — solana AND soroban #} + {% include "solana/_failure_modes.j2" %} {# chain-specific: signer/owner/PDA/CPI checks #} +{% endblock %} +``` + +So `rust/_failure_modes.j2`, the Rust `code_explorer` prompt, and the Cargo `forbidden_read` are +authored once as the `RUST` language and referenced by both `solana` and `soroban`; Soroban swaps +only the final `{% include "soroban/_failure_modes.j2" %}` and its chain facet. The base template +(`property_prompt_base.j2`) holds the ecosystem-neutral skeleton — the invariant / safety / +attack-vector framing, the output contract — so *that* is shared across all three. + +--- + +## 5. Selection: an application parameter + +The ecosystem is chosen per application, threaded to `run_pipeline` alongside the backend. + +- **Built-in apps.** `run_autoprove_pipeline` / `run_foundry_pipeline` pass `ecosystem=EVM` + explicitly (one line; both are EVM). +- **Rust apps.** The [`AppDescriptor`](./rust-applications.md) gains an `ecosystem` field + (default `"evm"`); the host resolves `ECOSYSTEMS[descriptor.ecosystem]` and passes it into + `run_application`. This adds one string field to `rust/autoprover-sdk` and one lookup in + `composer/rustapp/host.py`. + +```python +# composer/rustapp/descriptor.py +class AppDescriptor(BaseModel): + ... + ecosystem: ChainTag = "evm" # "evm" | "solana" | "soroban" +``` + +So a Solana application is `ecosystem="solana"` + a Solana backend wheel, and a Soroban +application is `ecosystem="soroban"` + a Soroban backend wheel. Nothing else in the app shell +changes — the generic entry point/frontend already synthesize from the descriptor, and both Rust +chains transparently pick up the shared `RUST` language facet. + +--- + +## 6. Driver generalization + +Localized changes; the phase chain and concurrency are untouched. + +**`run_pipeline`** ([core.py](../composer/pipeline/core.py)) takes `ecosystem: Ecosystem` and +stops hardcoding EVM: + +```python +async def run_pipeline[P, FormT, H, A, App]( + backend: PipelineBackend[P, FormT, H, A, App], + run: PipelineRun[P, H], + ecosystem: Ecosystem[App, ...], + *, ... +): + analyzed = await run.runner(..., lambda: run_component_analysis( + ty=ecosystem.system_model, # was: SourceApplication + prompts=ecosystem.analysis_prompts, # was: hardcoded templates + validate=ecosystem.validate_analysis, # was: _validate_connectivity + expected_main_id=source.contract_name, ...)) + prepared = await backend.prepare_system(analyzed, run) + ... + main = ecosystem.locate_main(analyzed, run.source) # was: main_instance(...) by solidity_identifier + batches = await _extract_all(ecosystem.units(main), ecosystem.property_prompts, ...) +``` + +**`run_component_analysis`** ([system_analysis.py](../composer/spec/system_analysis.py)) — +already generic over `T`; additionally accept the prompt pair + validation function instead of +importing `_validate_connectivity` and hardcoding template names. + +**`run_property_inference`** ([prop_inference.py](../composer/spec/prop_inference.py)) — accept +the ecosystem's property prompt pair and a generic `Unit` (it already takes `backend_guidance` +as a param, so the "expressible downstream" axis stays backend-owned; the "failure modes in this +domain" axis moves into the ecosystem's prompt). + +**`PipelineBackend` / `SystemAnalysisSpec`** — add the `App` type parameter so +`prepare_system(analyzed: App)` and `to_artifact_id(unit: Unit)` type-check against the paired +ecosystem. `SystemAnalysisSpec` keeps `analysis_key` + `extra_input` (backend/app-owned); the +analyzed *type* and templates move to the ecosystem. + +--- + +## 7. The EVM ecosystem (= today, zero behavior change) + +`EVM` = the `SOLIDITY` language facet ⊕ the `evm` chain facet, a faithful capture of current +behavior, so autoprove/foundry are byte-for-byte unchanged: + +```python +SOLIDITY = _Language( + name="solidity", + default_forbidden_read=FS_FORBIDDEN_READ, + code_explorer_prompt=CODE_EXPLORER_SYS_PROMPT, + failure_modes_partial="solidity/_failure_modes.j2", +) + +EVM = _Chain( + name="evm", + language=SOLIDITY, + system_model=SourceApplication, + analysis_prompts=PromptPair("application_analysis_system.j2", "application_analysis_prompt.j2"), + property_prompts=PromptPair("property_analysis_system_prompt.j2", "property_analysis_prompt.j2"), + validate_analysis=_validate_connectivity, # moved, not rewritten + locate_main=main_instance, # moved, not rewritten + units=lambda main: [ContractComponentInstance(_contract=main, ind=i) + for i in range(len(main.contract.components))], +) +``` + +The migration is a *move*, not a rewrite: existing types, prompts, and functions become the EVM +ecosystem's members (the current monolithic prompts stay as-is at first; splitting out a +`solidity/_failure_modes.j2` partial can wait until Soroban wants to reuse EVM fragments, §8). +This is the safety property of the proposal — the refactor is provably behavior-preserving for +EVM before any Rust chain adds anything. + +--- + +## 8. The Rust chains: Solana and Soroban (shared language facet) + +Both are `RUST` ⊕ their chain facet. The **`RUST` language facet is authored once** and both +reuse it verbatim: + +```python +RUST = _Language( + name="rust", + # Cargo layout: exclude target/ and .git; KEEP tests/ (unlike Foundry) and the crate sources. + default_forbidden_read=r"(^target/.*)|(^\.git.*)|(.*\.lock$)", + code_explorer_prompt=RUST_CODE_EXPLORER_PROMPT, # "Rust source … modules/traits/impls" + failure_modes_partial="rust/_failure_modes.j2", # overflow/underflow, panic!/unwrap/expect, ownership +) +``` + +`rust/_failure_modes.j2` is the concrete answer to "share Rust prompts between Solana and +Soroban": it is `{% include %}`d by both chains' property templates (§4.1). Each chain then +supplies only its own model + validation + platform fragment. + +### 8.1 Solana chain (`RUST ⊕ solana`) + +- **System model** (`composer/spec/solana/model.py`, new) — `SolanaApplication` with `Program` + (program id / `crate::module`), `Instruction` (entry points), `AccountGroup` / account + constraints (Solana accounts are **passed in**, not owned storage), and CPI targets / signers + in place of EOA `ExternalActor`. +- **Platform failure fragment** (`solana/_failure_modes.j2`) — missing signer/owner checks, + account substitution / confused-deputy, unvalidated PDA seeds, arbitrary CPI, lamport/rent + draining, missing Anchor constraints (`has_one`, `constraint`, `seeds`/`bump`). +- **`locate_main` / `units`** — main = the target program; units = its instructions (or + account-validation structs). + +### 8.2 Soroban chain (`RUST ⊕ soroban`) + +- **System model** (`composer/spec/soroban/model.py`, new) — `SorobanApplication` with `Contract` + (`#[contract]`), `ContractFunction` (`#[contractimpl]` entry points), and **typed contract + storage** (`instance` / `persistent` / `temporary`, each with TTL/archival) — i.e. the contract + **owns** its state, closer to EVM than to Solana. Authorization is explicit (`require_auth` / + `require_auth_for_args`); cross-contract calls go through generated clients; custom types are + `#[contracttype]`. +- **Platform failure fragment** (`soroban/_failure_modes.j2`) — missing/incorrectly-scoped + `require_auth`, storage-durability misuse (temporary vs persistent) and TTL/archival + (entry-expiration) bugs, unchecked cross-contract results / reentrancy, replay. The Rust + overflow/panic modes come from the shared `rust/_failure_modes.j2` — **not repeated here.** +- **`locate_main` / `units`** — main = the target contract; units = its contract functions. +- **Fragment reuse across the *chain* axis, too.** Because Soroban's model is + contract-owns-typed-storage with explicit authorization, its *analysis* prompt can + `{% include %}` EVM-flavored fragments about storage and authorization that Solana cannot — + the non-hierarchical sharing noted in §2.1. This is exactly why fragments beat rigid inheritance. + +The Solana/Soroban *backends* (formalization) are out of scope here — each plugs in via +[rust-applications.md](./rust-applications.md) and pairs with its chain's `App` model. + +--- + +## 9. What stays shared and unchanged + +- Source tools (`fs_tools`, `code_explorer`, `code_document_ref`) — already language-neutral; the + only ecosystem input is the `forbidden_read` default and the explorer prompt string. +- The report (`collect` / `Verdict` / schema) — neutral; `ReportBackend` already widened. +- Caching, the multi-round property loop, interactive refinement, `run_to_completion`, the whole + agent plumbing. +- The backend seam and the Rust app framework — a Solana or Soroban verifier is "just another + backend." + +--- + +## 10. Phased plan + +*Status: Phases 1–4 implemented (EVM preserved; Solana front half verified by a live gate). +Phase 5 (Soroban) and Phase 6 (verification backends) remain.* + +1. ✅ **Done — Extract `Language` + `Chain` + `EVM` (= `SOLIDITY ⊕ evm`), behavior-preserving.** + Added `composer/pipeline/ecosystem.py`; moved the `SourceApplication` reference, template + names, `_validate_connectivity`, `main_instance`, and unit-enumeration into `EVM`, and the + `forbidden_read`/`code_explorer` defaults into `SOLIDITY`. Threaded `ecosystem` through + `run_pipeline` / `run_component_analysis` / `run_property_inference`. Prompts stay monolithic + (no fragment split yet). **Gate met:** the autoprove end-to-end integration test passes + identically on this commit and its pre-refactor parent (real Postgres + live prover), and EVM + reproduces the prior template names / validator / analysis front-matter verbatim. + > **Env note (orthogonal to the refactor):** running that gate surfaced a pre-existing solc + > provisioning issue — the Counter scenario pins `pragma ^0.8.29` but the environment's default + > `solc` was 0.8.21, so the prover couldn't compile it (manifesting as an "exhausted tape"). The + > fix is environmental (point `solc` at ≥0.8.29; versioned `solc8.29` was already present), not a + > tape re-record. Worth pinning a matching `solc` in the gate's prover config so it's robust. +2. ✅ **Done — Add the `App` type parameter.** `PipelineBackend[P, FormT, H, A, App]` with + `prepare_system(analyzed: App)`; `Ecosystem` is generic over `App` (`EVM: Ecosystem[SourceApplication]`); + `run_pipeline` takes `ecosystem: Ecosystem[App]` explicitly and the Phase 1 + `cast(SourceApplication, analyzed)` is gone. `SystemAnalysisSpec` was intentionally **not** + parameterized — it carries only `analysis_key` + `extra_input`, no `App`-typed member. + **Gate met:** pyright reports 0 errors on the touched files (pairing type-checks, no cast). +3. ✅ **Done — Wire selection into `rustapp`.** `AppDescriptor.ecosystem: ChainTag` (Rust SDK, + default `"evm"`, + the Python mirror) and `resolve_ecosystem()` in `host.py` (registry lookup, + clear error for an unregistered chain), threaded through `build_application` / + `run_application` / `run_rust_pipeline` in place of the hardcoded `EVM`. **Gate met:** rustapp + tests + pyright green; only `evm` resolves for now (Solana/Soroban register in Phases 4–5). +4. ✅ **Done — Author the `RUST` language facet + the Solana chain.** This phase also performed + the driver's `Unit`/`Main` generalization deferred from Phase 2 (a `FeatureUnit` protocol in + `system_model`, threaded as type params through `Ecosystem`/`PreparedSystem`/`PipelineBackend`/ + `Formalizer`/`run_pipeline`; EVM stays bound to `ContractComponentInstance`/`ContractInstance` + with byte-identical cache keys). Added the standalone `SolanaApplication` model + (`composer/spec/solana/model.py`), the `RUST` language (Cargo `forbidden_read`, Rust + `code_explorer` prompt, `rust/_failure_modes.j2`) and the `SOLANA` chain (validate / + `locate_main` / `units`). The fragment-composition convention is realized with `{% include %}` + of the shared `rust/_failure_modes.j2` + `solana/_failure_modes.j2` into the Solana templates + (a neutral `property_prompt_base.j2` proved unnecessary — the Solana templates are + self-contained and just pull the fragments). A reusable `NullSolanaBackend` + an Anchor + `solana_vault` scenario back the gate. **Gate met:** a live-LLM run + (`tests/test_solana_gate.py`) analyzed the vault into 3 instructions and extracted 27 sane, + Solana-native properties (signer/owner checks, PDA + canonical-bump derivation, System-Program + substitution, reinit-once, arithmetic overflow) — no prover needed. +5. **Author the Soroban chain, reusing `RUST`.** `SorobanApplication` model + Soroban prompts; + the *only* new prompt content is `soroban/_failure_modes.j2` and the Soroban analysis template + — the Rust language fragments are inherited. **Gate:** the shared `rust/_failure_modes.j2` is + referenced by both chains and appears in neither chain's own fragment (no duplication). +6. **Solana / Soroban backends** — separate efforts, via the Rust framework. + +--- + +## 11. Open questions + +1. **Is `Unit` uniform enough across ecosystems?** EVM's unit is a contract component; Solana's + might be an instruction *or* an account-validation struct. If per-unit shape diverges too much, + `run_property_inference` may need the ecosystem to own unit rendering (a `render_unit(unit) -> dict` + hook) rather than a shared template variable. Decide when authoring the Solana property prompt. +2. **One backend, multiple ecosystems?** Could a single Rust backend serve both (unlikely given the + `App` pairing)? Keep the pairing explicit for now; revisit if a genuinely cross-domain backend + appears. +3. **Report labels by ecosystem.** Should the report say "program"/"instruction" vs + "contract"/"rule"? Today `backend_tag` drives labels; an `ecosystem` tag on the report may be + the cleaner source for domain nouns. Minor; defer. +4. **Prompt template packaging.** Templates live in a shared dir; the fragment convention needs + per-facet subdirs (`rust/`, `solidity/`, `solana/`, `soroban/`, plus shared `*_base.j2`) and a + Jinja loader that resolves includes across them. Confirm the loader supports `{% extends %}` / + `{% include %}` across subdirs (it should — it's stock Jinja). +5. **Fragment granularity.** Is a single `_failure_modes.j2` partial per facet the right grain, or + do we want finer per-concern fragments (storage / authorization / arithmetic) so Soroban can + pull the EVM *storage/auth* fragments (§8.2) without the EVM *contract-deployment* ones? Start + coarse (one partial per facet); split a fragment only when a second consumer wants part of it. +6. **`SolidityIdentifier` naming.** Rename the shared field/type to an ecosystem-neutral + `SourceIdentifier`, or leave it (regex already fits Rust)? Cosmetic; a rename touches many + annotations — sequence it after Phase 1. + +--- + +## 12. Key files + +| Concern | File | +|---|---| +| Driver to generalize | [composer/pipeline/core.py](../composer/pipeline/core.py) | +| Ecosystem seam (new) | `composer/pipeline/ecosystem.py` | +| System analysis (accept ecosystem) | [composer/spec/system_analysis.py](../composer/spec/system_analysis.py) | +| Property inference (accept ecosystem) | [composer/spec/prop_inference.py](../composer/spec/prop_inference.py) | +| EVM system model (→ EVM ecosystem) | [composer/spec/system_model.py](../composer/spec/system_model.py) | +| Analysis / property prompts (EVM) | `composer/templates/application_analysis_*.j2` · `property_analysis_*.j2` | +| fs-exclusion default | [composer/spec/util.py](../composer/spec/util.py) | +| Source tools (unchanged) | [composer/spec/source/source_env.py](../composer/spec/source/source_env.py) · [code_explorer.py](../composer/spec/code_explorer.py) | +| Language facets + shared prompt fragments (new) | `composer/templates/{rust,solidity}/_failure_modes.j2` · `*_base.j2` | +| Solana chain model / prompts (new) | `composer/spec/solana/…` · `composer/templates/solana/…` | +| Soroban chain model / prompts (new) | `composer/spec/soroban/…` · `composer/templates/soroban/…` | +| Ecosystem selection in Rust apps | [composer/rustapp/descriptor.py](../composer/rustapp/descriptor.py) · [host.py](../composer/rustapp/host.py) · [rust/autoprover-sdk/src/lib.rs](../rust/autoprover-sdk/src/lib.rs) | +| The backend seam (unchanged) | [docs/formalization-abstraction.md](./formalization-abstraction.md) | diff --git a/docs/formalization-abstraction.md b/docs/formalization-abstraction.md new file mode 100644 index 00000000..de1c37bd --- /dev/null +++ b/docs/formalization-abstraction.md @@ -0,0 +1,586 @@ +# Design Doc — The Formalization Abstraction + +> Detailed design of how AutoProver turns *extracted properties* into *verified +> artifacts*, the abstraction that makes it backend-agnostic, and a concrete +> walk-through of the CVL (Certora Prover) implementation. +> +> Companion to [ARCHITECTURE.md](../ARCHITECTURE.md). Where that document maps the +> whole system, this one zooms into a single seam: the contract between the generic +> pipeline driver and a verification backend. + +--- + +## 1. Problem & motivation + +The pipeline has two kinds of work: + +- **Shared work** that is identical no matter what you generate — analyze the system + into components, infer a list of properties per component, cache expensive results, + and assemble a final report. +- **Backend-specific work** — *how* a property becomes a checkable artifact, and *how* + that artifact's pass/fail verdict is obtained. For the CVL backend this is "author a + `.spec`, run the Certora Prover, revise on counterexamples." For the Foundry backend + it is "write a `.t.sol`, run `forge test`." + +The **formalization abstraction** is the seam between the two. It lets the driver in +[composer/pipeline/core.py](../composer/pipeline/core.py) own all the shared work while +delegating every backend-specific decision through a small, typed protocol. The CVL and +Foundry backends are two implementations of that protocol; the driver never imports +either. + +### Design goals + +1. **The driver inspects nothing backend-specific.** It moves opaque `FormT` values + around; only the backend ever looks inside them. +2. **No half-initialized state.** Each phase yields an immutable object that is the + constructor input to the next, so ordering is enforced by the type system, not by + call-order discipline. +3. **One result type threads everything.** A single generic parameter `FormT` keys the + cache, the artifact store, the verdict fetcher, and the report — so they cannot drift + out of agreement. +4. **Concurrency is structural.** Pre-formalization setup overlaps property extraction; + per-component formalization fans out — all expressed in the driver, inherited by every + backend for free. + +--- + +## 2. The phase chain + +Formalization is the tail of a three-link immutable chain. Each arrow is a method whose +return type is the input to the next link: + +``` + PipelineBackend ──prepare_system──▶ PreparedSystem ──prepare_formalization──▶ Formalizer + (config, analysis spec, (.main: located main (formalize / verdicts / + artifact store) contract; backend setup) report inputs / finalize) +``` + +The driver ([core.py:230](../composer/pipeline/core.py)) sequences them: + +```python +# 1. shared: analyze source → SourceApplication (always the same type) +analyzed = await run.runner(TaskInfo(SYSTEM_ANALYSIS_TASK_ID, ...), lambda: run_component_analysis(...)) + +# 2. backend transform: prover lifts to a harnessed app; foundry is identity +prepared = await backend.prepare_system(analyzed, run) + +# 3. pre-formalization setup runs CONCURRENTLY with property extraction +formalizer_task = asyncio.create_task(prepared.prepare_formalization(run)) +batches = await _extract_all(prepared.main, backend.backend_guidance, run, ...) +formalizer = await formalizer_task + +# 4. per-component formalization (parallel), cache-wrapped by the driver +settled = await asyncio.gather(*[_run(b) for b in batches], return_exceptions=True) + +# 5. shared: build + persist the report from the outcomes + backend verdicts +report = await build_report(..., fetch_verdicts=formalizer.fetch_verdicts) +``` + +The key structural point: `prepare_formalization` is launched as a task *before* +extraction is awaited. For the CVL backend, that is what overlaps the slow AutoSetup / +summary / structural-invariant work with per-component property inference — and it falls +out of the driver generically, so Foundry gets the same overlap with zero extra code. + +--- + +## 3. The contract + +Three protocols + two abstract bases define the entire seam. All live in +[composer/pipeline/core.py](../composer/pipeline/core.py) and +[composer/spec/types.py](../composer/spec/types.py). + +### 3.1 The result type: `FormT` + +Everything is generic over one type variable, the *backend result*. It is the +intersection of two narrow protocols: + +```python +# composer/pipeline/core.py +class BackendResult(FormalResult, ReportableResult, Protocol): ... +``` + +- **`FormalResult`** ([types.py:43](../composer/spec/types.py)) — what *persistence* + needs: `property_units()`, `commentary`, `artifact_text`. +- **`ReportableResult`** ([report/collect.py:25](../composer/spec/source/report/collect.py)) + — what the *report* needs: `skipped`, `property_units()`, `output_link`. + +A backend's concrete result (for CVL, `GeneratedCVL`) structurally satisfies both. The +driver only ever holds it as an opaque `FormT`; it never reads a field. + +### 3.2 `Formalizer[FormT]` — the heart of the abstraction + +```python +# composer/pipeline/core.py +@dataclass +class Formalizer[FormT: BackendResult](ABC): + formalized_type: type[FormT] # the concrete result class — the cache get/put key + backend_tag: ReportBackend # "prover" | "foundry", stamped into the report + + @abstractmethod + async def formalize(self, label, feat, props, ctx, run) -> FormT | GaveUp: ... + + @abstractmethod + async def fetch_verdicts(self, inp: ReportComponentInput[FormT]) -> dict[RuleName, Verdict]: ... + + def extra_report_inputs(self) -> list[ReportComponentInput[FormT]]: + return [] # synthetic report rows; default none + + async def finalize(self, outcomes, run) -> None: + return None # run-level artifacts from the full outcome set; default none +``` + +The contract is deliberately small — one required producer (`formalize`), one required +reader (`fetch_verdicts`), and two optional hooks. Crucially, a `Formalizer` is +**immutable and fully constructed** by `prepare_formalization`: it carries its prover +config, resources, and tool as constructor state, never set post-hoc. By the time +`formalize` runs, every dependency is already present. + +### 3.3 `PreparedSystem[FormT]` — the formalizer factory + +```python +# composer/pipeline/core.py +@dataclass +class PreparedSystem[FormT: BackendResult](ABC): + main: ContractInstance # located main contract + @abstractmethod + async def prepare_formalization(self, run) -> Formalizer[FormT]: ... +``` + +### 3.4 The outcome types the driver produces + +```python +@dataclass(frozen=True) +class Delivered[FormT]: # a success + the path it was written to + result: FormT + deliverable: Path + +class GaveUp(BaseModel): # the single unified give-up signal + reason: str + +@dataclass +class ComponentOutcome[FormT](BackendJob): + result: Delivered[FormT] | GaveUp | BaseException # success / declined / crashed +``` + +`ComponentOutcome` is a closed sum of the three things that can happen to one component: +it was `Delivered`, the agent `GaveUp` with a reason, or it raised. The driver's `_tally` +([core.py:351](../composer/pipeline/core.py)) folds these into the run result; the report +phase renders each. + +--- + +## 4. The CVL backend, method by method + +The prover implementation lives in +[composer/spec/source/pipeline.py](../composer/spec/source/pipeline.py). It declares: + +```python +@dataclass +class ProverBackend: # PipelineBackend[AutoProvePhase, GeneratedCVL, None, ComponentSpec] + backend_guidance = CERTORA_BACKEND_GUIDANCE + core_phases = CorePhases({"analysis": ..., "extraction": ..., "formalization": AutoProvePhase.CVL_GEN}) + analysis_spec = SystemAnalysisSpec("source-analysis") + artifact_store: ProverArtifactStore + _prover_opts: ProverOptions +``` + +So `FormT = GeneratedCVL`, the artifact id type `A = ComponentSpec`, and the phase enum is +`AutoProvePhase`. + +### 4.1 `prepare_system` — harness lift + +```python +async def prepare_system(self, analyzed: SourceApplication, run) -> PreparedSystem[GeneratedCVL]: + sys_desc = await run.runner(TaskInfo(HARNESS_TASK_ID, ...), lambda: run_harness_creation(...)) + harnessed = _lift_harnessed(analyzed, sys_desc) # SourceApplication → HarnessedApplication + prover_tool = get_prover_tool(run.env.llm_heavy(), run.source.contract_name, + run.source.project_root, prover_opts=self._prover_opts) + return ProverPrepared(main_instance(harnessed, run.source), self.artifact_store, + sys_desc, harnessed, prover_tool, self._prover_opts, analyzed) +``` + +It runs the harness-classification agent, folds the generated harnesses back into the app +as a `HarnessedApplication` ([pipeline.py:69](../composer/spec/source/pipeline.py)), builds +the shared `verify_spec` prover tool once, and packages everything the next phase needs into +an immutable `ProverPrepared`. (Foundry's `prepare_system` is an identity transform — no +harness, no tool.) + +### 4.2 `prepare_formalization` — the concurrent setup fan-out + +This is where the CVL backend does its expensive pre-work, and it is the richest method in +the abstraction. From [pipeline.py:178](../composer/spec/source/pipeline.py): + +```python +async def prepare_formalization(self, run) -> Formalizer[GeneratedCVL]: + # AutoSetup (+ custom summaries) ∥ structural-invariant formulation — both depend only + # on the harnessed app, so they run concurrently. + (setup_config, resources), invariants = await asyncio.gather( + self._autosetup(run), self._invariants(run), + ) + + invariant = None + if invariants.inv: + inv_props = [PropertyFormulation(title=inv.name, description=inv.description, sort="invariant") + for inv in invariants.inv] + self._store.write_properties(InvariantSpec(), inv_props) + + # Generate invariants.spec ONCE, with cache short-circuit + inv_cvl_ctx = run.ctx.child(INV_CVL_KEY) + cached = await inv_cvl_ctx.cache_get(GeneratedCVL) + if cached is not None: + inv_cvl = cached + else: + inv_result = await run.runner(TaskInfo(INVARIANT_CVL_TASK_ID, ...), + lambda: batch_cvl_generation(inv_cvl_ctx.abstract(CVLGeneration), + setup_config.prover_config, inv_props, None, resources, self._prover_tool, ...)) + if isinstance(inv_result, GaveUp): + raise RuntimeError(f"Structural invariant CVL generation gave up: {inv_result.reason}") + inv_cvl = inv_result + await inv_cvl_ctx.cache_put(inv_cvl) + + inv_path = self._store.write_artifact(InvariantSpec(), inv_cvl) + # Append invariants.spec to the resource set so EVERY per-component spec imports it. + resources = [*resources, CVLResource(path=inv_path, required=False, + description="Structural invariants that may be assumed as preconditions", sort="import")] + invariant = (inv_props, Delivered(inv_cvl, inv_path)) + + return ProverRunner(GeneratedCVL, "prover", self._store, self._prover_tool, + setup_config.prover_config, resources, invariant, make_prover_fetcher()) +``` + +Three things worth calling out: + +- **Concurrency inside the method.** AutoSetup+summaries and invariant *formulation* are + independent, so they `gather`. This nests under the driver-level overlap (this whole + method already runs concurrently with property extraction). +- **Structural invariants are formalized eagerly, here, not per-component.** They are + generated once into `invariants.spec`, then injected into `resources` so every later + per-component spec can `import` them as preconditions. The invariant CVL goes through the + exact same `batch_cvl_generation` path that components do (with `component=None`). +- **The returned `ProverRunner` is fully loaded.** Its config, resource set (now including + `invariants.spec`), prover tool, and the in-memory invariant result are all constructor + fields. `formalize` adds nothing — it only *reads* them. + +### 4.3 `formalize` — per-component authoring + verification loop + +The `Formalizer.formalize` impl is a thin adapter; all the work is in +`batch_cvl_generation`: + +```python +# ProverRunner.formalize (pipeline.py:114) +async def formalize(self, label, feat, props, ctx, run) -> GeneratedCVL | GaveUp: + return await batch_cvl_generation( + ctx.abstract(CVLGeneration), self._prover_config, props, feat, + self._resources, self._prover_tool, run.env, label, run.source, SPECS_DIR) +``` + +`batch_cvl_generation` ([author.py:334](../composer/spec/source/author.py)) builds a +dedicated LLM agent graph and runs it to a fixpoint. The agent is given: + +- the property batch + component context, rendered into the prompt; +- the resource set as `import` views, with paths made relative to `certora/specs/` so the + prover resolves CVL `import`s correctly ([author.py:349](../composer/spec/source/author.py)); +- a tool belt: CVL authoring tools, the `verify_spec` prover tool, config-edit tools, and the + completion/give-up/expectation tools (`PublishResultTool`, `GiveUpTool`, + `ExpectRuleFailure`/`ExpectRulePassage`). + +Two **hard validation gates** must both pass before the agent may publish +([author.py:404](../composer/spec/source/author.py)): + +```python +required_validations=[FEEDBACK_VALIDATION_KEY, PROVER_VALIDATION_KEY] +``` + +- the **prover gate** — the spec must actually run; +- the **feedback gate** — a separate `property_feedback_judge` agent + ([composer/spec/feedback.py](../composer/spec/feedback.py)) adjudicates whether each + property is genuinely covered, and the author may file evidence-backed `Rebuttal`s + (typecheck failure / counterexample / manual citation / reasoned) against prior feedback. + +The agent loop ends in exactly one of two states, mapped onto the abstraction's +`FormT | GaveUp` ([author.py:413](../composer/spec/source/author.py)): + +```python +if res_state["failed"]: + return GaveUp(reason=res_state["result"]) +return GeneratedCVL(commentary=..., cvl=..., skipped=..., property_rules=..., + config=res_state["config"], final_link=res_state.get("prover_link")) +``` + +Note `config` and `final_link` are captured into the result. That is deliberate: a later +cache hit skips the prover entirely, so the result must carry enough to rebuild +`certora/confs/` and keep the run link without re-running anything. + +### 4.4 `extra_report_inputs` — folding in the invariants + +Per-component outcomes are assembled by the driver. The structural invariants are a +*synthetic* component the backend contributes ([pipeline.py:136](../composer/spec/source/pipeline.py)): + +```python +def extra_report_inputs(self) -> list[ReportComponentInput[GeneratedCVL]]: + if self._invariant is None: + return [] + inv_props, inv = self._invariant + return [ReportComponentInput(name="Structural Invariants", props=inv_props, formalized=inv)] +``` + +This is the report-side payoff of formalizing invariants in `prepare_formalization`: the +in-memory `Delivered[GeneratedCVL]` is replayed straight into the report with no special +casing in the driver. + +### 4.5 `fetch_verdicts` — pass/fail per rule + +```python +async def fetch_verdicts(self, inp) -> dict[RuleName, Verdict]: + return await self._fetch(inp) # make_prover_fetcher(): queries ProverOutputUtility off-thread +``` + +The fetcher resolves each spec's prover run (via `inp.formalized.run_link`) and rolls per-rule +outcomes into `Verdict`s. The `collect` step +([report/collect.py:104](../composer/spec/source/report/collect.py)) then keys rules by +`(unit_file, name)` so a structural invariant imported into several component specs collapses +to one entry, and uses `Verdict.merge` (priority `BAD > ERROR > TIMEOUT > UNKNOWN > GOOD`) to +roll up multiple results for one rule. Foundry's fetcher instead reads pass/fail straight off +the result with no run service — same protocol, different source. + +### 4.6 `finalize` — run-level artifact + +```python +async def finalize(self, outcomes, run) -> None: + runs = {ComponentSpec(o.feat.slugified_name).run_key: o.result.run_link + for o in outcomes if isinstance(o.result, Delivered) and o.result.run_link} + if self._invariant and self._invariant[1].run_link: + runs[InvariantSpec().run_key] = self._invariant[1].run_link + self._store.write_component_runs(runs) # → components_to_prover_runs.json +``` + +`finalize` is the one hook that sees the *entire* outcome set at once — used here to emit the +`{spec → prover-run link}` map. Foundry omits it (default no-op). + +--- + +## 5. The result type as the central key + +`GeneratedCVL` ([cvl_generation.py:125](../composer/spec/cvl_generation.py)) is the concrete +`FormT`. It satisfies `BackendResult` structurally — note nothing declares +`class GeneratedCVL(BackendResult)`; the protocols match by shape: + +```python +class GeneratedCVL(BaseModel): + commentary: str + cvl: str + skipped: list[SkippedProperty] = Field(default_factory=list) + property_rules: list[PropertyRuleMapping] = Field(default_factory=list) + config: dict | None = None + final_link: str | None = None + + def property_units(self) -> list[tuple[str, list[str]]]: # FormalResult + ReportableResult + return [(m.property_title, m.rules) for m in self.property_rules] + + @property + def artifact_text(self) -> str: # FormalResult: bytes to write + return self.cvl + + @property + def output_link(self) -> str | None: # ReportableResult: run link + return _output_link(self.final_link) # /jobStatus/ → /output/ +``` + +The same value is the key for four otherwise-independent subsystems, which is what keeps them +from disagreeing: + +| Consumer | Uses | Via | +|---|---|---| +| **Cache** | the *type* `GeneratedCVL` | `formalizer.formalized_type` → `cache_get`/`cache_put` | +| **Artifact store** | `artifact_text`, `commentary`, `property_units()` | `ArtifactStore.write_artifact` | +| **Report** | `skipped`, `property_units()`, `output_link` | `ReportableResult` | +| **Run-link map** | the persisted `final_link` | `finalize` | + +--- + +## 6. Persistence: the artifact store + +`Delivered` pairs a result with the path it was written to — and those two always travel +together because the path *exists only because the result did* +([core.py:98](../composer/pipeline/core.py)). The write happens in the driver's `_run` +closure: + +```python +backend.artifact_store.write_properties(result_key, batch.props) # before generation +... +Delivered(result, backend.artifact_store.write_artifact(result_key, result)) # after success +``` + +The store is generic over `(ArtifactIdentifier, FormalResult)` +([artifacts.py:32](../composer/spec/artifacts.py)). The base writes everything that is +identical across backends — `properties.json`, `commentary.md`, the +`{property title → demonstrating units}` map — keyed off the identifier's `stem`. The CVL +subclass [ProverArtifactStore](../composer/spec/source/artifacts.py) adds the +CVL-specific bundle: it overrides `write_artifact` to also emit a `.conf` (base config + +fixed run overlay) alongside the `.spec`. + +The artifact id is itself a small sum type, so naming conventions live in one place rather +than being interpolated at call sites: + +```python +@dataclass(frozen=True) +class ComponentSpec: # autospec_.spec + slug: str + @property + def stem(self): return f"autospec_{self.slug}" + @property + def run_key(self): return self.slug + +@dataclass(frozen=True) +class InvariantSpec: # invariants.spec + @property + def stem(self): return "invariants" +``` + +`ProverBackend.to_artifact_id(component)` maps a component instance to its `ComponentSpec`; +the driver uses it both to write properties before generation and to write the artifact after. + +The resulting on-disk layout (all under the project's `certora/`): + +``` +certora/specs/autospec_.spec # per-component CVL +certora/specs/invariants.spec # structural invariants (imported by the above) +certora/confs/.conf # prover config per spec +certora/properties/.properties.json # inferred properties +certora/properties/.property_rules.json # property → [rule names] +certora/properties/.commentary.md # author's commentary +certora/ap_report/report.json # final cross-referenced report +.certora_internal/autoProve/components_to_prover_runs.json # finalize() output +``` + +--- + +## 7. Caching wraps formalization (driver-owned) + +A backend never writes cache logic — the driver does, keyed by +`formalizer.formalized_type` ([core.py:271](../composer/pipeline/core.py)): + +```python +async def _run(batch): + result_key = backend.to_artifact_id(batch.feat) + backend.artifact_store.write_properties(result_key, batch.props) + child = await batch.feat_ctx.child(_batch_cache_key(batch.props), {...}) + cached = await child.cache_get(formalizer.formalized_type) # ← type comes from the formalizer + if cached is None: + result = await run.runner(TaskInfo(formalize_task_id(...)), + lambda: formalizer.formalize(label, batch.feat, batch.props, child, run)) + if not isinstance(result, GaveUp): + await child.cache_put(result) + else: + result = cached + ... +``` + +The cache key is the hash of the *property batch* (`_batch_cache_key`), under the component's +context, under the `properties` context — the hierarchical scheme described in +[ARCHITECTURE.md §7](../ARCHITECTURE.md). Because the result type carries `config` and +`final_link`, a cache hit can rebuild the `.conf` and keep the run link without touching the +prover. The structural-invariant CVL has its own cache short-circuit inside +`prepare_formalization` (`INV_CVL_KEY`, §4.2) for the same reason. + +--- + +## 8. Failure handling + +The abstraction encodes three distinct failure modes, each handled differently: + +| Failure | Representation | Driver behavior | +|---|---|---| +| Agent declines a component | `formalize` returns `GaveUp(reason)` | recorded as a `ComponentOutcome`, surfaced in `failures`, rendered in report as a gap; **not cached** | +| Component crashes | `formalize` raises | `asyncio.gather(..., return_exceptions=True)` captures it into `ComponentOutcome.result` | +| Invariant CVL gives up | `prepare_formalization` raises `RuntimeError` | **fatal** — invariants are a shared precondition, so the whole run aborts | +| Report build fails | exception in `build_report` | best-effort: logged, run still succeeds ([core.py:318](../composer/pipeline/core.py)) | + +The asymmetry is intentional: a single component giving up is a normal, reportable outcome, +but the shared invariant spec failing would silently weaken every downstream component, so it +fails loud. + +--- + +## 9. Extending: what a new backend must provide + +To add a backend you implement the protocol and the three phase objects — nothing in the +driver changes. The Foundry backend +([composer/foundry/pipeline.py](../composer/foundry/pipeline.py)) is the proof: it reuses +system analysis, property extraction, caching, and the report, and contributes only: + +| Abstraction member | CVL backend | Foundry backend | +|---|---|---| +| `FormT` | `GeneratedCVL` | `GeneratedFoundryTest` | +| `prepare_system` | harness lift + prover tool | identity | +| `prepare_formalization` | AutoSetup ∥ summaries ∥ invariants | trivial (pre-built formalizer) | +| `formalize` | author CVL, run prover, revise | author tests, run `forge test` | +| `fetch_verdicts` | query prover output off-thread | read ran/expected tests off the result | +| `extra_report_inputs` | synthetic "Structural Invariants" | none | +| `finalize` | `components_to_prover_runs.json` | none | +| artifact bundle | `.spec` + `.conf` | `.t.sol` + metadata | + +A backend author's checklist: + +1. Define a result type satisfying `FormalResult` + `ReportableResult` (`artifact_text`, + `commentary`, `property_units()`, `skipped`, `output_link`). +2. Subclass `ArtifactStore` for the on-disk bundle; define an `ArtifactIdentifier` sum type. +3. Implement `PipelineBackend` (`prepare_system`, `to_artifact_id`, `backend_guidance`, + `core_phases`, `analysis_spec`, `artifact_store`). +4. Implement `PreparedSystem.prepare_formalization` returning a fully-constructed + `Formalizer`. +5. Implement `Formalizer.formalize` + `fetch_verdicts`; override `extra_report_inputs` / + `finalize` only if needed. + +--- + +## 10. End-to-end trace (CVL backend) + +Putting it together, one run of the CVL backend over a component: + +``` +run_autoprove_pipeline +└─ run_pipeline(ProverBackend, run) + 1. run_component_analysis ──────────────▶ SourceApplication + 2. ProverBackend.prepare_system + run_harness_creation ─▶ SystemDescriptionHarnessed + _lift_harnessed ─▶ HarnessedApplication + get_prover_tool ─▶ verify_spec tool + ▶ ProverPrepared(main=located main contract, ...) + 3. ┌ create_task: ProverPrepared.prepare_formalization + │ gather( _autosetup → (config, [summaries]) , _invariants → [BaseInvariant] ) + │ batch_cvl_generation(component=None) ─▶ invariants.spec (cached under INV_CVL_KEY) + │ resources += invariants.spec + │ ▶ ProverRunner(config, resources, invariant, fetch) + └ _extract_all ─▶ [ _Batch(component, props) , ... ] # runs concurrently with the above + 4. for each batch (parallel, semaphore-bounded): + write_properties(ComponentSpec(slug), props) + cache_get(GeneratedCVL)? ── hit ─▶ reuse + └ miss ─▶ ProverRunner.formalize + batch_cvl_generation(component=feat) + author CVL ⇄ verify_spec ⇄ feedback judge (loop) + gates: PROVER_VALIDATION + FEEDBACK_VALIDATION + ─▶ GeneratedCVL | GaveUp + cache_put(result) + write_artifact ─▶ autospec_.spec (+ .conf) ⇒ Delivered(result, path) + ─▶ ComponentOutcome + ProverRunner.finalize(outcomes) ─▶ components_to_prover_runs.json + 5. build_report( per-component inputs + extra_report_inputs(), + fetch_verdicts=ProverRunner.fetch_verdicts ) ─▶ certora/ap_report/report.json +``` + +--- + +## 11. Key files + +| Concern | File | +|---|---| +| Driver + abstraction definitions | [composer/pipeline/core.py](../composer/pipeline/core.py) | +| Result protocols (`FormalResult`, `ArtifactIdentifier`) | [composer/spec/types.py](../composer/spec/types.py) | +| `ReportableResult`, `Verdict`, `VerdictFetcher`, `collect` | [composer/spec/source/report/collect.py](../composer/spec/source/report/collect.py) | +| CVL backend (the three phase objects) | [composer/spec/source/pipeline.py](../composer/spec/source/pipeline.py) | +| CVL authoring agent (`batch_cvl_generation`) | [composer/spec/source/author.py](../composer/spec/source/author.py) | +| CVL result type (`GeneratedCVL`) | [composer/spec/cvl_generation.py](../composer/spec/cvl_generation.py) | +| Artifact store base / CVL subclass | [composer/spec/artifacts.py](../composer/spec/artifacts.py) · [composer/spec/source/artifacts.py](../composer/spec/source/artifacts.py) | +| Foundry backend (contrast) | [composer/foundry/pipeline.py](../composer/foundry/pipeline.py) | +``` diff --git a/docs/pr-split-plan.md b/docs/pr-split-plan.md new file mode 100644 index 00000000..4e5d726b --- /dev/null +++ b/docs/pr-split-plan.md @@ -0,0 +1,112 @@ +# PR split plan: `eric/crucible` → 4 stacked PRs + +The `eric/crucible` branch is large (62 commits, ~94 files, +13.5k lines over +`origin/master`). To make it reviewable it is split into **4 stacked PRs**, merged +in order, each independently compilable and gated. + +- **History model:** squash-per-PR — each stacked branch is one clean, self-contained + commit built from the final file state for that layer (granular phase history is + not preserved). +- **Execution posture:** build the 4 stacked local branches, verify each compiles + + passes its gate, then hand off for review before any push. + +## Why this order + +The import graph is cleanly layered (verified): + +- `composer.sandbox` is standalone (imports neither `rustapp` nor `crucible`). +- `composer.rustapp` imports `composer.sandbox`. +- `composer.crucible` imports `rustapp`, `sandbox`, and the ecosystem. +- `composer.pipeline.ecosystem` / `spec.solana` import none of the above. + +So the dependency order is: ecosystem → rust framework → sandbox mechanism → crucible. +This also mirrors the phases the work was actually built in, and front-loads the +behavior-preserving refactor (PR 1) while isolating the security-sensitive +`run-confined` mechanism (PR 3) for focused review. + +## The stack + +### PR 1 — Ecosystem abstraction (EVM + Solana front-half) + +Runtime `Ecosystem`/`Language` seam; the driver generalized over `Unit`/`Main`; EVM +reproduces today's behavior exactly; Solana added as a second ecosystem, proven by +analysis + property extraction against a null (no-verifier) backend. + +- **Code:** `composer/pipeline/ecosystem.py`, `composer/pipeline/core.py` + + `ptypes.py`, `composer/spec/system_model.py` (`FeatureUnit`), + `composer/spec/prop_inference.py`, `composer/spec/system_analysis.py`, + `composer/spec/solana/model.py`, `composer/spec/solana/null_backend.py`, + `composer/templates/solana/*` +- **Docs:** `ARCHITECTURE.md`, ecosystem proposal +- **Gate:** existing EVM autoprove tape (Counter) still green **+** `test_solana_gate` + (front-half) +- **Depends on:** `origin/master` + +### PR 2 — Rust application framework (PyO3) + +The generic wheel host + the command-runner / sandbox **seam** (unconfined `none` +provider only — no confinement yet). + +- **Code:** `composer/rustapp/*`, `rust/` workspace (`Cargo.toml`, `autoprover-sdk`, + `example-app`/echoprover), the `RunCommand` effect, + `composer/sandbox/{policy,command,config}.py` (seam + `none` provider), + `pyproject.toml` / `uv.lock`, `composer/templates/rust/*` +- **Docs:** `rust-applications.md`, `rust-formalization-backends.md` +- **Gate:** `test_rustapp` (echoprover decider round-trip; sandbox is a passthrough here) +- **Depends on:** PR 1 + +### PR 3 — Command sandbox (Landlock + seccomp confinement) + +The security-sensitive confinement mechanism, isolated for focused review. + +- **Code:** `composer/sandbox/launcher.py` + `recipes.py` (offline resolution, private + CARGO_HOME / TMPDIR), `rust/run-confined`, `scripts/Dockerfile` + `.dockerignore` +- **Docs:** `docs/command-sandbox.md` +- **Gate:** `test_sandbox_escape` (all vectors denied + unconfined control) + + `test_crucible_sandbox_gate` (legit build under launcher) +- **Depends on:** PR 2 + +### PR 4 — Crucible backend (capstone) + +The Solana verification application, wiring PRs 1–3 together. + +- **Code:** `composer/crucible/*`, `rust/crucible-app`, `test_scenarios/solana_vault`, + crucible RAG (committed manifest `rust/crucible-app/crucible_kb.rag.json` + shared + `composer/scripts/rag_import.py` + `composer/rag/import_format.py`, `composer/tools/crucible_rag.py`, + `composer/rag/db.py`), `ReportBackend` "crucible" + + render labels + `as_report_backend`, sandbox default → launcher for crucible +- **Docs:** crucible proposal / application / toolchain-versioning +- **Gate:** `test_crucible_gate`, `test_crucible_setup_gate`, + `test_crucible_formalize_gate`, **`test_crucible_e2e_gate`** +- **Depends on:** PR 1, 2, 3 + +## Cross-cutting files (the one real hazard) + +A few files are touched by more than one layer; their *final* form assumes later work +exists, so they can't be assigned wholesale to one PR. Ship an **intermediate form in +the earlier PR, final form in the owning PR**: + +| File | Earlier PR gets… | Owning PR finalizes… | +|---|---|---| +| `composer/rustapp/adapter.py` | PR 2: thin generic adapter using `cast(ReportBackend, tag)` | PR 4: swap to `as_report_backend` | +| `composer/spec/source/report/schema.py` + `render.py` | — (stays master's `prover`/`foundry`) | PR 4: close to `{…, crucible}` + crucible labels | +| `composer/pipeline/ecosystem.py` | PR 1: whole file incl. `RUST_FORBIDDEN_READ` (a regex string, no import dep) | — | + +Everything else is disjoint by directory and maps cleanly. + +## Invariant held during execution + +Each branch must **compile + pass its own gate**, not just the final one. The verified +import layering is what makes that achievable. + +## Execution recipe (per PR, when greenlit) + +For each PR, branching off the previous (PR 1 off `origin/master`): + +1. `git checkout -b ` +2. `git checkout eric/crucible -- ` +3. Adjust cross-cutting files to their intermediate form (see table). +4. Run that PR's gate; fix until green. +5. Squash-commit with a descriptive message. + +Result: 4 stacked local branches to review before any push. diff --git a/docs/rag-import-format.md b/docs/rag-import-format.md new file mode 100644 index 00000000..86a7be61 --- /dev/null +++ b/docs/rag-import-format.md @@ -0,0 +1,299 @@ +# Design Doc — A common JSON format for RAG entries + a single importer + +> Today every documentation corpus that feeds a search tool ships its **own** RAG builder: +> a bespoke Python module that parses that corpus's native format *and* talks to the RAG +> database, plus a shell wrapper. Adding a new corpus (notably: a new Rust application that +> wants its own `*_kb`) means writing another builder wired to the DB. +> +> This proposes splitting that seam: a **producer** emits a corpus as a common JSON document, +> and one shared **importer** ingests any such document into the RAG DB. The DB coupling, +> chunking, embedding, batching and dual-path ingestion move into the importer — once — and a +> producer shrinks to "parse my docs → emit JSON", with no dependency on the RAG stack. +> +> **Scope:** for now this mechanism is adopted **only for Crucible**. The Foundry and CVL +> builders stay exactly as they are; they are candidate future adopters (§5), not part of this +> change. The format is nonetheless designed to be general, so migrating them later needs no +> schema change. +> +> Companion to [rust-applications.md](./rust-applications.md) (the descriptor-driven app model) +> and [rust-backend-api.md](./rust-backend-api.md) (the wheel FFI surface). The `knowledge_base` +> tag defined here is the *same* tag a wheel already declares as +> [`rag_db_default`](../composer/rustapp/descriptor.py). + +--- + +## 1. What's actually shared today — and what isn't + +Three builders exist, one per corpus: + +| Builder | Source format | Ingests | +| --- | --- | --- | +| [`ragbuild.py`](../composer/scripts/ragbuild.py) | CVL-manual HTML (docutils) | vector + manual | +| [`foundry_ragbuild.py`](../composer/scripts/foundry_ragbuild.py) | Foundry cheatcode HTML fragments | vector only | +| `crucible_ragbuild.py` (now replaced — see §7) | Crucible markdown | vector + manual | + +Each had a shell wrapper ([`populate_rag.sh`](../scripts/populate_rag.sh), +[`populate_foundry_rag.sh`](../scripts/populate_foundry_rag.sh), `populate_crucible_rag.sh`), and +each pins a default connection constant (`DEFAULT_CONNECTION`, `FOUNDRY_DEFAULT_CONNECTION`, +`CRUCIBLE_DEFAULT_CONNECTION`, `SANITY_DEFAULT_CONNECTION`) in +[`composer/rag/db.py`](../composer/rag/db.py). (This section describes the state that *motivated* +the change; Crucible's builder + wrapper have since been replaced — §7 — while the CVL and Foundry +ones remain.) + +The important observation: **only the first column differs.** Everything downstream is already +common code the three builders call into: + +- the chunk model [`BlockChunk`](../composer/rag/types.py) — a header path (`h1..h6`), a `part` + index, `code_refs`, and a `chunk` body with `` placeholders; +- the length-bounded splitter `BlockBuilder` / `BuilderConfig` + ([`text_processors.py`](../composer/scripts/text_processors.py)), driven by spaCy; +- the **dual-path ingestion** on [`ComposerRAGDB`](../composer/rag/db.py): + `add_chunks_batch` (embedded chunks → vector search) **and** `add_manual_section` (the full + section → keyword search / `get_section`); +- embedding, batching, and `part`-numbering of repeated header paths. + +So each builder re-implements *source parsing* and then hand-rolls the same orchestration around +the same shared primitives. The bespoke part is small; the boilerplate around it is duplicated +per corpus and, critically, **carries a hard dependency on the RAG DB and the heavy `ragbuild` +uv group** (spaCy + sentence-transformers). A new Rust app that just wants to contribute a corpus +inherits all of that — which is what makes Crucible the natural first case to lift onto a generic +mechanism (the other two builders already exist and work, so they can migrate later or never). + +### The right cut: logical sections, not pre-chunked rows + +The seam should fall **between parsing and chunking**, not after chunking. A producer emits +*logical sections* — a header path plus an ordered list of prose/code blocks — exactly the +shape the former `crucible_ragbuild`'s internal `_Section` already had. The importer owns everything from +there down: running `BlockBuilder` to produce length-bounded embedded chunks, assembling the +full-section manual chunk, assigning `` tags, numbering `part`s, embedding, and +batching. + +Why this cut and not "emit finished `BlockChunk`s": + +- **Chunking is common, tuned, and heavy.** Length-bounding needs spaCy and a shared + `max_length`. Keeping it in the importer means producers need neither spaCy nor + sentence-transformers — a Rust app can emit the JSON from a trivial script (or from the wheel + itself; see §6) with no RAG dependencies. +- **Code-ref tagging is a footgun.** `crucible_ragbuild` manually tracked a `code_refs` list and + emits `` tags in lockstep; getting that wrong orphans a ref. Producers should + never see the tag scheme — they just say "this block is code." +- **`part` numbering is global.** The manual-section table is unique on + `(h1..h6, part)`; repeated header paths must bump `part`. That's a whole-corpus concern the + importer is positioned to own; a producer emitting isolated chunks can't. + +The one thing this cut asks of producers is that they decide **section boundaries** — how blocks +group into sections. That is exactly where the genuinely corpus-specific editorial judgment +lives (e.g. Foundry merges signature/description/parameters/returns into one summary section and +gives each example its own; see §5), and it is expressed simply by *how the producer lays out +sections*. Within a section, the importer still sub-splits by length. + +--- + +## 2. The JSON format + +A corpus is one **manifest** document: metadata plus an ordered list of sections. Proposed +schema (v1), mirrored by a pydantic model the way +[`descriptor.py`](../composer/rustapp/descriptor.py) mirrors the Rust `AppDescriptor`: + +```jsonc +{ + "version": 1, + "knowledge_base": "crucible_kb", // logical KB tag (== descriptor rag_db_default) + "source": "crucible@a1b2c3d docs/*.md", // free-text provenance, for logs only + "sections": [ + { + "headers": ["Writing Fuzz Harnesses", "PDA Seed Encoding"], + "blocks": [ + { "kind": "text", "body": "Seeds are encoded as ..." }, + { "kind": "code", "body": "let (pda, bump) = Pubkey::find_program_address(...);" }, + { "kind": "text", "body": "The bump is then ..." } + ] + }, + { + "headers": ["Writing Fuzz Harnesses", "PDA Seed Encoding", "Example"], + "blocks": [ { "kind": "code", "body": "..." } ] + } + ] +} +``` + +Field notes: + +- **`version`** — schema version; the importer rejects unknown majors. Lets the format evolve + without silently mis-ingesting old files. +- **`knowledge_base`** — the logical corpus tag. This is the *same* string a wheel declares as + `rag_db_default` and that [`rag_env.py`](../composer/tools/rag_env.py) resolves to search + tools. Making producer, importer, and runtime agree on one tag is a real simplification: it + becomes the single key naming a corpus end to end. The importer resolves it to a connection + string via a registry (§4), overridable by `--output`. +- **`source`** — provenance for logging/traceability only. The RAG schema is header-only + ([`documents`](../composer/rag/db.py) / `manual_sections` store `content + h1..h6`), so this + is **not** persisted per row; it just lands in the importer's log line. (If we later want + per-row provenance we'd extend the DB schema — out of scope for v1.) +- **`headers`** — the section's header path, ≤ 6 entries (the `h1..h6` columns). Empty/trailing + levels are fine; the importer left-packs and truncates exactly as `_normalize_head` does. +- **`blocks`** — ordered `{ "kind": "text" | "code", "body": "..." }`. Prose is fed to + `append_text` (spaCy-split, structured boundary); code is fed to `add_code` and gets a + `` tag automatically. This is the whole content model — it maps 1:1 onto the two + `BlockBuilder` operations, so nothing about a section is unrepresentable. + +**Every section feeds both retrieval paths.** There is no knob for this: the importer always +ingests a section into *both* the vector index (semantic search over length-bounded embedded +chunks) and the manual index (keyword search + exact `get_section` over the whole section). The +two indexes answer different questions — "what passage *means* this?" vs. "which sections +*contain* this term, and give me one in full" — and Crucible's search tools bind all three +retrieval styles ([`crucible_rag.py`](../composer/tools/crucible_rag.py)). So populating both is +what Crucible's builder already did, and what "ingest a corpus" means here. + +Deliberately **not** in the schema: `part` (importer-assigned), `code_refs` / `` +tags (importer-assigned), `max_length` / chunking knobs (importer flags — a cross-corpus tuning +concern, not corpus data), and any ingest-path selector (there is only one behaviour: both). + +--- + +## 3. The importer + +One module, `composer/scripts/rag_import.py`, that factored the shared orchestration out of the +former `crucible_ragbuild`'s `_async_main` (which was already 90% of this) and generalized it over +the manifest — minus the markdown parser, which stays in the producer: + +``` +uv run --isolated --group ragbuild python -m composer.scripts.rag_import \ + corpus.rag.json [more.rag.json ...] [--output ] [--max-length N] [--print] +``` + +Behaviour: + +1. **Load + validate** each manifest against the pydantic model (clear errors on a malformed + file, before any DB write). +2. **Resolve the target** once per manifest: `--output` if given, else the connection registered + for `knowledge_base` (§4). Refuse to run if neither resolves. +3. For each section, feed **both** indexes: + - **vector** → build a `BlockBuilder` from the blocks, buffer the resulting `BlockChunk`s, + flush via `add_chunks_batch` at `_BATCH_SIZE`; + - **manual** → assemble one full-section `BlockChunk` (code as `` tags), assign + its `part` from a per-header-path counter, `add_manual_section`. +4. **`--print`** — dry-run: render sections/chunks to stdout, no DB writes (parity with every + builder's existing `--print`). + +That is the orchestration the former `crucible_ragbuild` hand-rolled, now reusable by any producer +that emits the manifest. Note it needs the `ragbuild` uv group (spaCy + sentence-transformers) — but +now *only the importer* does; producers don't. + +--- + +## 4. Connection resolution + +The importer resolves a manifest's `knowledge_base` tag to a DB connection via a small registry, +overridable by `--output`: + +```python +KNOWLEDGE_BASES: dict[str, str] = { + "crucible_kb": CRUCIBLE_DEFAULT_CONNECTION, +} +``` + +For now it holds only `crucible_kb` — the one corpus on this path. This is the same registry idea +as [`rag_env.py`](../composer/tools/rag_env.py) (tag → search tools) and the ecosystem registry: +a declarative tag resolved to a concrete resource, not a fork. The existing +`*_DEFAULT_CONNECTION` constants in `db.py` stay put; if CVL/Foundry ever migrate onto this path, +their tags join the registry then — and ideally the two registries share the tag namespace, so a +corpus's *import* target and its *runtime* search tools resolve by one name. + +--- + +## 5. Does the format generalize? (Foundry / CVL — future adopters, not now) + +Only Crucible moves onto this mechanism now. But to be sure we aren't designing a Crucible-shaped +format by accident, it's worth checking the format could absorb the *other* corpora later — the +harder one being Foundry, whose builder does real editorial grouping, not just parsing: it merges +`signature`/`description`/`parameters`/`returns` into **one** summary chunk keyed by the cheatcode +name, gives `Examples`/`Gotchas` their **own** chunks, and drops `Related Cheatcodes`. None of +that needs new schema — it's all just *how a producer lays out sections*: + +- Summary → one section `headers: ["Cheatcodes", ""]` whose `blocks` are the merged, + labelled content (`"Signature:\n..."`, `"Description:\n..."`, the parameter list as text, …). +- Each example/gotcha → its own section `headers: ["Cheatcodes", "", "Examples"]`. +- `Related Cheatcodes` → simply not emitted. + +The genuinely Foundry-specific pieces — the `.mdx → .html` conversion +([`foundry_process.py`](../composer/scripts/foundry_process.py)) and the table-to-parameter-list +translation — would stay in a producer. So the section-layout model absorbs a non-trivial corpus +cleanly; the design isn't Crucible-only. + +One aside worth recording (but **out of scope**): a future Foundry migration would incidentally +fix a latent gap. The current `foundry_ragbuild.py` only populates the vector index, so Foundry's +`foundry_cheatcodes_keyword_search` / `..._get_section` tools query a `manual_sections` table that +is never written. Because this importer always feeds both indexes (§2), routing Foundry through it +would finally populate that table. Not a reason to migrate now — just a note for whoever does. + +--- + +## 6. What this gives Rust applications (the motivating case) + +Under the descriptor model a Rust app is a wheel + a declarative `AppDescriptor`; it already +names its corpus via `rag_db_default`. The missing piece is *contributing the corpus content* +without writing composer-resident Python glued to the RAG DB. Two levels: + +- **Level 1 (done for Crucible):** the app ships a `.rag.json` (built however it likes — a + script in the app's own repo, checked-in output, a CI artifact) and the generic `rag_import.py` + ingests it. Crucible ships its corpus as the committed + [`rust/crucible-app/crucible_kb.rag.json`](../rust/crucible-app/crucible_kb.rag.json) — no + crucible checkout at build or run time. Composer ships the importer and the schema, nothing + corpus-specific. +- **Level 2 (optional, natural follow-on):** add a wheel FFI callout — `rag_entries() -> str` + returning the manifest JSON — so RAG content becomes part of the app package exactly like + `descriptor()`. The importer could then ingest straight from a loaded wheel + (`rag_import --from-wheel crucible_app`), and a Rust app contributes a corpus with **zero** + Python. This is out of scope for v1 but is the reason the manifest is self-describing + (`knowledge_base` inside the document, not a CLI arg): a wheel can emit a complete, resolvable + corpus with no external metadata. + +--- + +## 7. What was built (Crucible) + +1. Manifest model [`composer/rag/import_format.py`](../composer/rag/import_format.py) + the + `KNOWLEDGE_BASES` registry (seeded with `crucible_kb`) in + [`composer/rag/db.py`](../composer/rag/db.py). +2. The generic importer [`composer/scripts/rag_import.py`](../composer/scripts/rag_import.py) (§3). +3. The Crucible corpus is generated **once** from the crucible repo's `docs/` and committed as + [`rust/crucible-app/crucible_kb.rag.json`](../rust/crucible-app/crucible_kb.rag.json) + (126 sections). A checked-in artifact — no crucible checkout is needed to build or run the app. +4. The container populates it at `setup-db` time: the Dockerfile copies the manifest to + `$AUTOPROVE_HOME/crucible_kb.rag.json`, and + [`scripts/autoprove-entrypoint.sh`](../scripts/autoprove-entrypoint.sh) runs `rag_import` into + the `crucible_rag` schema (alongside the CVL `rag_db` build). The demo (§1g of + [crucible-demo.md](./crucible-demo.md)) imports the same committed manifest. +5. The Crucible-specific RAG scripts are gone — the old `crucible_ragbuild.py` and + `populate_crucible_rag.sh` are deleted. Composer ships only the generic importer + schema. + +**Regenerating the manifest.** It was produced by a small markdown→manifest producer (parse ATX +headers + fenced code into sections — the logic mirrors §2 and is preserved in git history at the +commit that removed it). To refresh after a crucible docs update, restore or re-derive that +producer, point it at the crucible `docs/`, and re-commit the JSON. Because the corpus changes +rarely, this is an occasional manual step, not part of any build. + +**Untouched:** `foundry_ragbuild.py`, `ragbuild.py` (CVL), their wrappers, and `refresh_rag.sh`. +No runtime code changes — the search tools, `rag_env.py`, and the DB API are the same. + +--- + +## 8. Alternatives considered + +- **Emit finished `BlockChunk`s in the JSON (cut below chunking).** Rejected: pushes spaCy + + the `max_length` policy + `` tagging + global `part` numbering into every + producer, re-duplicating the heavy, error-prone parts and re-coupling producers to the RAG + stack. The whole point is to keep producers dependency-free. +- **A plugin/entry-point registry of builders** (each corpus registers a `build()` callable) — + removes the shell duplication but keeps every builder coupled to the DB and the `ragbuild` + group, and gives Rust apps nothing (still composer-resident Python per corpus). A data format + is a stronger boundary than a code interface here: it's inspectable, diffable, cacheable, and + producible without the RAG stack. +- **Persist richer per-row metadata** (source URL, doc version, tags). Deferred: the current DB + schema is header-only, so v1 keeps provenance at the manifest level (logs only). Revisit with + a schema change if retrieval ever needs to filter on it. +- **One physical DB per corpus vs. one shared DB.** Orthogonal to this proposal — the + `KNOWLEDGE_BASES` registry expresses whatever the deployment already does (today: shared + `rag_db`, distinct roles; `extended_rag_db` separate). The format doesn't dictate topology. +``` diff --git a/docs/rust-applications.md b/docs/rust-applications.md new file mode 100644 index 00000000..3d464e52 --- /dev/null +++ b/docs/rust-applications.md @@ -0,0 +1,324 @@ +# Design Doc — Rust Applications via PyO3 + +> How to stand up a whole new AutoProver *application* — phase enum, entry point, pipeline, +> frontend, artifact store, and `main()` — when the formalization backend is written in Rust. +> Where does the Python/Rust line fall across the five pieces, and how little bespoke Python +> can an extension author get away with writing? +> +> Companion to [application-abstraction.md](./application-abstraction.md) (the five pieces of +> an application) and [rust-formalization-backends.md](./rust-formalization-backends.md) (the +> backend seam in Rust, incl. the Tier-2 inversion-of-control authoring loop). This doc +> assumes both: it reuses the backend design wholesale and only addresses the *rest of the +> vertical* an application needs. + +--- + +## 1. The dividing line + +An application is [five pieces plus a `main()`](./application-abstraction.md): a phase enum, +an entry point/Executor, a pipeline+backend, a frontend, and an artifact store. The instinct +"write the application in Rust" is the wrong framing, because these pieces are not +symmetric — they fall cleanly into two camps: + +- **Imperative async Python shell** — the entry point (live Postgres pools, a RAG DB, a + background event drainer, `ContextVar`-scoped handler installation, the `composer.bind` + import-time DI/monkeypatch bootstrap), the frontend (a Textual TUI / stdout console owning + the event loop), and `main()`. None of this is data; it is service lifecycle and UI. It + **must stay Python**. +- **Pure logic + declarations** — the backend's decisions (covered in the backend doc), the + on-disk artifact *format*, and a handful of *declarations* (what phases exist, what CLI + flags exist, what the deliverable layout is). This is what Rust is good for. + +So the same principle that governs the backend seam governs the whole application: + +> **Rust declares and decides; Python wires and does.** Rust never owns an event loop, a DB +> connection, a Textual widget, or an `async with`. It contributes data (a descriptor) and +> pure step functions (the backend). Python owns every imperative, stateful, async edge. + +Under that principle, "a Rust application" is really **a Rust backend + a Rust +*descriptor*, both consumed by a reusable Python host.** The rest of this doc defines that +descriptor and the host. + +--- + +## 2. Where each piece lands + +| Piece | Camp | Rust contributes | Python owns | +|---|---|---|---| +| **Phase enum `P`** | declaration | list of phase names + which map to the 4 core slots | synthesizes the `enum.Enum` at runtime from that list | +| **Entry point / Executor** | Python shell | a declarative arg schema, a sync precondition-validation hook, a RAG-DB default | argparse, all service setup, `WorkflowContext`, `composer.bind`, the `runner` closure | +| **Pipeline wrapper** | Python shell | — | the thin `run__pipeline` that builds backend + `PipelineRun` + calls `run_pipeline` | +| **Backend** | Rust core | the `PipelineBackend`/`PreparedSystem`/`Formalizer` logic (see [backend doc](./rust-formalization-backends.md)) | the adapter shim + the Tier-2 effect loop | +| **Frontend** | Python shell | domain-event *content* (pushed across FFI), phase labels + section order | the `MultiJobApp`/console handler, `get_stream_writer()` emission, all rendering | +| **Artifact store** | mixed | the format-specific bundle formatter, the deliverable layout paths | the `ArtifactStore` shell + base writes (`properties.json`, `commentary.md`, …) | +| **`main()`** | Python shell | — | the ~20-line glue; a *generic* one parameterized by the descriptor | + +The striking result of grounding this in the code: **only the backend and the artifact +*formatter* are genuinely Rust; everything else is either a declaration Rust hands over as +data, or Python the extension author does not have to write at all** — if we build the host +in §3. + +--- + +## 3. The `AppDescriptor` and the generic host + +The leverage move is to make the entire non-backend surface **declarative**, so a single +reusable Python "app host" can synthesize the phase enum, the argparse, the entry point, the +frontend, and `main()` from one struct the Rust wheel exports. + +### 3.1 What Rust exports + +```rust +pub struct AppDescriptor { + name: String, // "myprover" + header_text: String, // TUI header + phases: Vec, // ordered; each: { key, label, core_slot: Option } + args: Vec, // extra CLI flags beyond the 3 positional inputs + rag_db_default: Option, // override the --rag-db default (cf. foundry cheatcodes DB) + event_kinds: Vec, // domain events the frontend should render (see §4.4) + artifact_layout: ArtifactLayout, // deliverable dir conventions (see §4.6) + // the backend itself: + backend: RustBackendHandle, // constructs the PipelineBackend (backend doc) +} + +enum CoreSlot { Analysis, Extraction, Formalization, Report } + +struct PhaseSpec { key: String, label: String, order: u32, core_slot: Option } +struct ArgSpec { flag: String, help: String, default: ArgDefault, required: bool } +``` + +The whole thing serializes to JSON at load time; the four `CoreSlot` values are exactly the +[`CorePhases`](../composer/pipeline/core.py) TypedDict keys the driver tags, so `phases` both +defines the UI vocabulary and supplies the `core_phases` mapping. + +### 3.2 What the Python host does with it + +```python +# composer/rustapp/host.py (NEW, write-once, reused by every Rust app) +def build_application(desc: dict) -> Application: + Phase = enum.Enum(f"{desc['name']}Phase", # ← synthesized enum, safe (§4.1) + {p["key"]: p["key"] for p in desc["phases"]}) + labels = {Phase[p["key"]]: p["label"] for p in desc["phases"]} + core = CorePhases({slot: Phase[p["key"]] # analysis/extraction/formalization/report + for p in desc["phases"] if (slot := p.get("core_slot"))}) + return Application( + phase=Phase, labels=labels, section_order=[...], + make_entry_point=lambda summary: _generic_entry_point(desc, Phase, core, summary), + make_frontend=lambda: _GenericRustApp(desc, Phase, labels), # MultiJobApp subclass + make_backend=lambda store, run: RustBackendAdapter(desc["backend"], core, store), + ) +``` + +An extension author then ships **only** the Rust wheel (backend + descriptor) plus a +one-line registration; the generic `main()`/entry point/frontend come from the host. That is +the end state: **zero bespoke Python per application.** + +> **Scope decision.** The generic host is net-new infrastructure. It is worth building only +> once we expect ≥2 Rust applications. For the first one, hand-writing the ~100 lines of +> Python glue (per [application-abstraction.md §10](./application-abstraction.md)) around the +> Rust backend is faster and lower-risk. §7 phases this. + +--- + +## 4. Piece by piece + +### 4.1 Phase enum — Rust declares, Python synthesizes + +Grounded in the code: `P` is used at runtime **only** for `.name` (logging/labels, +[multi_job.py](../composer/io/multi_job.py)) and as a hashable dict key (`phase_labels`, +`CorePhases`). There are no `isinstance` checks on phases, no enum-identity comparisons, and +the generic bound is `enum.Enum`, not the concrete class (and PEP 695 bounds aren't enforced +at runtime anyway). So `enum.Enum(f"{name}Phase", {...})` synthesized from the descriptor is +safe — the *only* rule is that `phase_labels` must be keyed by those same synthesized members +(member identity drives the lookup at [multi_job_app.py](../composer/ui/multi_job_app.py)), +which §3.2 satisfies by construction. + +### 4.2 Entry point / Executor — stays Python, Rust declares its inputs + +The entry point is irreducibly imperative async Python: it opens `standard_connections` (four +Postgres-backed pools — checkpointer, store, pgvector indexed store, memory backend), a RAG +DB, the async tool context, and a thread logger under a nested `async with`; builds the +`ServiceHost` env + `WorkflowContext`; reads the design doc via the async `FileUploader`; and +runs `import composer.bind` for its import-time DI registration and test-tape monkeypatching +([autoprove_common.py](../composer/spec/source/autoprove_common.py), +[composer/bind.py](../composer/bind.py)). **A Rust backend slots in only at the leaf** — the +`Formalizer` the pipeline eventually calls. + +What Rust *does* contribute here is purely declarative, consumed by the generic entry point: + +- **Extra CLI args** — the `args: Vec` become `parser.add_argument(...)` calls; the + three positional inputs (`project_root`, `main_contract`, `system_doc`) are always present. + This replaces the per-app `AutoProveArgs`/`FoundryArgs` Protocol. +- **A precondition-validation hook** — foundry validates `foundry.toml` exists *in the entry + point before opening services* ([application-abstraction.md §5](./application-abstraction.md)). + Rust exposes a **sync** `fn validate_preconditions(args_json) -> Result<(), String>` the + generic entry point calls right after arg parsing. Sync is fine — it's pure filesystem + checks, no async needed. +- **The RAG-DB default** — `rag_db_default` overrides `--rag-db`'s default, exactly as + `FoundryRAGDBOptions` does today, without a new flag. + +### 4.3 Pipeline wrapper + backend — see the backend doc + +The `run__pipeline` wrapper stays a thin Python function (build backend + `PipelineRun` ++ call `run_pipeline`); the generic host provides it. The backend itself — `prepare_system`, +`prepare_formalization`, `formalize` (incl. the Tier-2 LLM authoring loop), `fetch_verdicts`, +`finalize` — is the subject of [rust-formalization-backends.md](./rust-formalization-backends.md) +and is not re-derived here. The one link back: the descriptor's `core_phases` mapping (§3.1) +is the backend's `core_phases` slot; the phase enum synthesized in §4.1 is what stamps +`TaskInfo`. + +### 4.4 Frontend — Python renders; Rust supplies event *content* + +The frontend is a Textual `MultiJobApp` / stdout console — it owns the event loop and every +widget, and it must stay Python. A generic `MultiJobApp` subclass driven by the descriptor's +`phase_labels`/`section_order` handles the structure for free. + +The subtlety is **domain event streaming**. Applications stream backend-specific events to +their task panels (`forge_test_run`, `prover_output`, `cloud_polling`). Grounded in the code, +these are emitted by calling LangGraph's `get_stream_writer()(payload)` from *inside a running +graph node/tool*, or `emit_custom_event(payload)` off-graph — both require being inside the +async `with_handler` scope, and both take a plain `dict` with a discriminating `type` key +([foundry/runner.py](../composer/foundry/runner.py), +[composer/prover/runner.py](../composer/prover/runner.py), +[composer/io/context.py](../composer/io/context.py)). **Rust cannot call `get_stream_writer()`.** + +The clean resolution reuses the backend doc's Tier-2 machinery. The Python effect loop that +drives the Rust decider *is* inside the handler scope, so add one fire-and-forget command to +the vocabulary: + +```rust +Command::Emit { kind: String, payload: Json }, // Rust → Python: "stream this to my panel" +``` + +The Python driver, being in-scope, does `get_stream_writer()({"type": cmd.kind, **cmd.payload})` +and immediately resumes the Rust decider with `Observation::Ack`. This mirrors exactly how the +Certora prover surfaces stdout lines through the `ProverEventCallbacks` shim today — Rust +controls the *content*, Python does the emission. The descriptor's `event_kinds` tells the +generic frontend how to render each `type` (which panel/log, and whether it's plain text), so +even the `handle_event` body is generated, not hand-written. + +### 4.5 Artifact store — Python shell, Rust formatter + +Same split as the backend doc: the `ArtifactStore` subclass stays a Python shell (the base +writes the shared `properties.json` / `commentary.md` / property→units map / `token_usage.json`), +and Rust supplies the format-specific bundle formatter and the deliverable-layout paths (the +descriptor's `artifact_layout`). This is where an app declares that, e.g., its deliverables go +under `/*.t.sol` with metadata under `certora//` — the "share a project root +without colliding" convention ([application-abstraction.md §8](./application-abstraction.md)). + +### 4.6 `main()` — generic, parameterized by the descriptor + +The two `main()` shapes (TUI worker vs console-direct) are ~20 lines each and differ only in +who owns the event loop. The host provides both, parameterized by the descriptor; the +extension author picks TUI/console at the CLI entry point. The one invariant to preserve: +`import composer.bind as _` runs first (import-time DI/tape bootstrap). + +--- + +## 5. New infrastructure this requires + +Most of the application already composes for free; the genuinely net-new pieces are: + +1. **The `AppDescriptor` JSON schema** — the versioned contract between a Rust wheel and the + host (phases, args, event kinds, artifact layout). Shared with the backend doc's + marshalling/ABI schemas. +2. **The generic Python host** (`composer/rustapp/host.py`) — synthesizes enum + argparse + + entry point + frontend + `main()` from a descriptor. This is the reusable payoff; write it + once. +3. **A generic `MultiJobApp` subclass** whose `create_task_handler` / `handle_event` are + data-driven by `event_kinds`, rather than a hand-written subclass per app. +4. **The `Command::Emit` extension** to the Tier-2 effect vocabulary (§4.4), so a Rust + backend can stream domain events without touching `get_stream_writer()`. +5. **A registration entry point** — how a Rust wheel advertises its descriptor to the host + (e.g. a Python `[project.entry-points]` group, or an explicit `register(desc)` call). + +Everything else — the driver, the seam (`HandlerFactory`), the `MultiJobApp` base, caching, +the report — is inherited unchanged. + +--- + +## 6. Hypothetical: a Rust "MyProver" application end to end + +Putting §§3–4 together, standing up a Rust application called `myprover`: + +``` +myprover (Rust wheel) composer host (Python, reusable) +├─ AppDescriptor { build_application(desc): +│ name: "myprover", Phase = enum.Enum("MyproverPhase", …) ← §4.1 +│ phases: [Analyze→analysis, Extract→ labels/section_order from desc +│ extraction, Prove→formalization, core_phases from desc.core_slot +│ Report→report, +Setup(no slot)], _generic_entry_point(desc): ← §4.2 +│ args: [--solver-timeout, --parallel], argparse from desc.args +│ rag_db_default: "myprover_kb", validate_preconditions(args) ↺ Rust (sync) +│ event_kinds: [solver_line, proof_step], open Postgres pools / RAG / ctx / bind +│ artifact_layout: certora/myproof/…, yield runner(handler): +│ } RustBackendAdapter(desc.backend, …) ← backend doc +├─ validate_preconditions() (sync) _GenericRustApp(desc): ← §4.4 +├─ backend: PipelineBackend phase panels from labels +│ ├─ prepare_system / prepare_formalization handle_event dispatches by event_kind +│ └─ formalize: Tier-2 decider ─────┐ generic main() (TUI or console) ← §4.6 +│ emits Command::Emit{solver_line}│ +└─ artifact bundle formatter └──────▶ Python effect loop calls + get_stream_writer()({type:"solver_line",…}) +``` + +The author writes Rust (a backend + a descriptor) and *nothing else*: no argparse, no Textual +code, no entry-point service wiring, no `main()`. The `Setup` phase with no core slot shows an +app contributing its own UI-only phase, exactly as autoprove's harness/autosetup phases do. + +--- + +## 7. Work breakdown + +### Phase A — one Rust application, hand-written Python glue +Prove the vertical before generalizing. Reuse the backend milestones from +[rust-formalization-backends.md §6](./rust-formalization-backends.md), then hand-write the +~100 lines of app glue per [application-abstraction.md §10](./application-abstraction.md): +a concrete phase enum, a concrete `_entry_point`, a concrete `MultiJobApp` subclass, a +`run__pipeline`, and both `main()`s. Wire domain events through a `ProverCallbacks`-style +shim. This ships a working Rust application and surfaces the real friction. + +### Phase B — extract the `AppDescriptor` + generic host +Once Phase A works, factor the hand-written glue into the descriptor schema (§3.1) and the +generic host (§5.1–3). Migrate the Phase-A app onto it; a second Rust app should then need +zero bespoke Python. + +### Phase C — event-emission ergonomics +Add `Command::Emit` (§4.4) and the data-driven `handle_event`, so streaming domain events is +declarative rather than a per-app Python callback. + +--- + +## 8. Open questions + +1. **Build the host, or hand-write glue per app?** The generic host pays off only at ≥2 Rust + apps. Until then Phase A's hand-written glue is cheaper. Decide based on the roadmap. +2. **Interactive applications (`H ≠ None`).** Both current apps are non-interactive + (`H = None`; handlers raise on HITL). An interactive Rust app would need the backend to + *await* a human response mid-loop — which, unlike everything else here, genuinely needs the + Tier-3 async bridge (or an `Emit`-style command whose observation is the human's answer, + routed through the Python HITL handler). The `Emit`-with-response route keeps us bridge-free + and is preferred. +3. **Descriptor registration.** Python entry-points group vs explicit `register()` — how does + the host discover a Rust wheel's descriptor, and how are version mismatches surfaced at + load time? +4. **Event rendering richness.** `event_kinds` as "write this text to this log" covers the + current apps. If an app wants a custom widget (progress bar, table), does it fall back to a + hand-written `MultiJobApp` subclass, or do we grow the descriptor? Start with text; grow + only on demand. + +--- + +## 9. Key files + +| Concern | File | +|---|---| +| The five pieces of an application | [application-abstraction.md](./application-abstraction.md) | +| The Rust backend seam (incl. Tier-2 loop) | [rust-formalization-backends.md](./rust-formalization-backends.md) | +| Phase-enum runtime use (`.name`, dict key) | [composer/io/multi_job.py](../composer/io/multi_job.py) · [composer/ui/multi_job_app.py](../composer/ui/multi_job_app.py) | +| Entry point / service wiring (must stay Python) | [composer/spec/source/autoprove_common.py](../composer/spec/source/autoprove_common.py) · [composer/foundry/entry.py](../composer/foundry/entry.py) | +| DI / tape bootstrap | [composer/bind.py](../composer/bind.py) | +| Domain-event emission (`get_stream_writer` / `emit_custom_event`) | [composer/io/context.py](../composer/io/context.py) · [composer/prover/runner.py](../composer/prover/runner.py) · [composer/foundry/runner.py](../composer/foundry/runner.py) | +| Frontend base + seam | [composer/ui/multi_job_app.py](../composer/ui/multi_job_app.py) · [composer/io/multi_job.py](../composer/io/multi_job.py) | +| `CorePhases` / driver | [composer/pipeline/core.py](../composer/pipeline/core.py) | +| Generic host (NEW) | `composer/rustapp/host.py` | diff --git a/docs/rust-backend-api.md b/docs/rust-backend-api.md new file mode 100644 index 00000000..e2b0247b --- /dev/null +++ b/docs/rust-backend-api.md @@ -0,0 +1,321 @@ +# A service-shaped backend API (no IoC loop, no async runtime in the wheel) + +Design note. A concrete alternative to the IoC decider loop +([rust-ioc-loop.md](./rust-ioc-loop.md)): recast the Rust backend as a **passive service** +that the **Python core pipeline** drives through the whole author→compile→judge→validate +loop. The backend answers pure questions (prompts, RAG) and owns the two "run the local +toolchain" steps (`compile`, `validate`) — which invoke `cargo`/`crucible` **directly**, each +through the `run-confined` sandbox launcher. No `RustSession`/`resume`, no +`Command`/`Observation` protocol, and no async runtime in the wheel. + +## 1. The reframing + +The backend's real job in formalization is small: + +- **know what to say to the model** — the authoring prompt and the optional judging prompt; +- **compile a candidate spec** — build it via the toolchain and report failures verbatim so + the model can fix them; +- **validate the spec** — run the checker (Crucible) and turn its output into verdicts. + +Everything else — running the LLM agent, retrying, streaming events, caching, and building the +RAG search tools — is generic orchestration the pipeline already does for the CVL/Foundry +backends. So the backend stops being a *driver* (the `resume` state machine) and becomes a +*service* the driver consults. (RAG is unchanged: the author's knowledge-base search tools are +still built the way they are today — e.g. Crucible's external `crucible_kb` corpus, imported +from a committed manifest via `composer.scripts.rag_import` — not supplied through this API.) + +The two slow steps are different from the pure ones: `compile` and `validate` actually run +untrusted native tools, so they **execute inside the sandbox**. The backend does that itself +by spawning [`run-confined`](../rust/run-confined/src/main.rs) — the standalone launcher that +applies Landlock + seccomp + rlimits + an env allowlist to itself and then `execve`s the tool +(fail-closed). The backend authors the command line; **Python authors the confinement policy** +(which paths are writable/readable, the private `CARGO_HOME`, caps) and hands it in, because +that is environment/recipe knowledge that lives in `composer/sandbox` +([command-sandbox.md](./command-sandbox.md)). + +## 2. The backend API + +Two tiers: + +- **Pure** callouts (`json → json`, no I/O): metadata, `units`, prompts. Same shape as today's + `descriptor`/`fetch_verdicts`. +- **Blocking** callouts: `compile`/`validate` spawn `run-confined` and wait. They are ordinary + synchronous functions that **release the GIL** while the child runs, so Python calls them + with `await asyncio.to_thread(...)` — non-blocking to the event loop, **no tokio, no + `pyo3-async`** (see §5). They are kept **clearly separate** (compile the whole spec once; + validate one unit at a time) — no fusing the dry-run into the first fuzz. + +```rust +pub trait Backend: Send + Sync + 'static { + // ---- metadata (unchanged, pure) -------------------------------------- + fn descriptor(&self) -> AppDescriptor; + fn validate_preconditions(&self, args: &Value) -> Result<(), String> { Ok(()) } + + // ---- authoring (pure) ------------------------------------------------ + /// The units this input formalizes — one per property — each a property title and its + /// backend-specific unit name (Crucible: `c_`, the test-fn / feature selector). + /// Pure and *pre-authoring*: the author prompt requires exactly these fn names, the + /// host validates each unit, and it is the report's property→unit map. `kind="setup"` + /// (the fixture) has no units. + fn units(&self, input: &AuthorInput) -> Vec; // {property, unit} + /// Instruction (+ optional system prompt) to author `input.kind`'s spec — covering all + /// its units. `failure = Some(..)` on a re-author after a compile failure or a judge + /// rejection, so one function covers the initial draft and every revision. + fn author_prompt(&self, input: &AuthorInput, failure: Option<&Failure>) -> Prompt; + /// Optional LLM review of a compiled spec, before validation. `None` (the default — + /// what every backend returns today) skips judging; `Some(prompt)` runs a judge turn + /// whose structured verdict (accept / reject + feedback) the host feeds back as a + /// `Failure` on reject. Present in the API so a backend can opt in without a reshape. + fn judge_prompt(&self, input: &AuthorInput, spec: &str) -> Option { None } + + // ---- gating: run the toolchain directly, inside run-confined --------- + /// Compile/typecheck the whole spec ONCE (every unit shares one build): materialize it + /// into `workdir`, build under `sandbox`, and report success or the errors to hand back + /// to the model. BLOCKING — releases the GIL while `run-confined` runs. + fn compile(&self, input: &AuthorInput, spec: &str, workdir: &Path, sandbox: &Sandbox) + -> CompileResult; // Ok | Failed { errors } + + /// Validate ONE unit against the (already-compiled) spec and bake its verdict — run the + /// checker for that unit only. Per-unit so the host owns enumeration and scheduling + /// (and can fan out); the backend never discovers units here. BLOCKING (GIL released). + fn validate(&self, input: &AuthorInput, spec: &str, unit: &str, + workdir: &Path, sandbox: &Sandbox) -> Verdict; // GOOD/BAD/… + + // ---- assembly (pure) ------------------------------------------------- + fn finalize(&self, outcomes: &Value) -> BTreeMap { BTreeMap::new() } +} +``` + +Supporting types: + +```rust +struct AuthorInput { kind: String, program: String, component: Value, props: Vec, context: Value } +struct Prompt { system: Option, instruction: String } +struct Failure { errors: String } // compile stderr or judge feedback, fed back to the model +enum CompileResult { Ok, Failed { errors: String } } +struct Unit { property: String, unit: String } // report row + fuzz target (e.g. c_) + +/// The confinement policy, authored by Python (never the LLM). The backend maps it to a +/// `run-confined` argv and prepends the launcher; `None` = the trusted/`none` path (exec +/// directly, no launcher). See composer/sandbox/policy.py::SandboxPolicy. +struct Sandbox { + run_confined: Option, // launcher path; None → run unconfined (trusted) + rw: Vec, ro: Vec, // Landlock grants (workdir, toolchains, checkout) + allow_env: Vec, // env allowlist ("NAME" or "NAME=VAL") + network: bool, + rlimits: Rlimits, // as / cpu / nproc / fsize + timeout_s: u64, +} +``` + +`kind` lets one backend author more than one thing with the same primitives — Crucible's +program-wide **fixture** (`kind="setup"`) and each unit's **tests** (`kind="component"`) are +both "author a spec, then `compile` it"; only the component kind has a `validate` step, and +the fixture's compiled output feeds the components' `context`. + +### How `compile`/`validate` reach the sandbox — a shared SDK helper + +Both `compile` and `validate` run their tool the same way, so `autoprover-sdk` provides **one +helper** every backend calls — the launcher contract lives in exactly one place: + +```rust +// in autoprover-sdk +pub fn run_confined( + sandbox: &Sandbox, program: &str, args: &[String], + files: &BTreeMap, workdir: &Path, timeout: Duration, +) -> io::Result; // { exit_code, stdout, stderr } +``` + +It (1) materializes `files` into `workdir`, path-confining each write (reject absolute / `..`, +as `composer.sandbox.command._confined_target` does); (2) builds the argv +`[run_confined, , "--", program, *args]` — or `[program, *args]` when +`sandbox.run_confined` is `None` (the trusted/`none` path); (3) spawns it (std `Command`), +waits with `timeout`, and captures the streams. The resulting launch is exactly what the +Python `launcher` provider builds today — + +```text +run-confined --rw --rw <.sandbox_cargo> --ro --ro + --allow-env PATH --allow-env CARGO_HOME=… [--allow-network] + --rlimit-as … -- crucible run --release --mode explore --timeout N +``` + +— just assembled in the SDK next to the backend that owns the command. So `compile` builds the +`{program, args, files}` for a dry-run and calls `run_confined`; `validate` does the same for +one unit's fuzz run; neither re-implements sandbox plumbing. The **command after `--` is +backend-authored**; the **policy flags before it are Python-authored** (`Sandbox`); Landlock +grants only the `--rw` workdir, so even a bad file path can't escape. + +## 3. The Python side + +The formalizer is one generic loop in the core pipeline: + +```python +async def formalize(mod, input, env, sandbox, *, workdir, max_attempts, emit) -> Formalized | GaveUp: + units = mod.units(input) # ← backend (pure): the fuzz targets + spec, failure = None, None + for _ in range(max_attempts): # author → compile → judge (retry) + prompt = mod.author_prompt(input, failure) # ← backend (pure) + spec = await run_llm_agent(env, prompt, tools=env.rag_tools + env.source_tools) # ← Python: LLM + r = await asyncio.to_thread(mod.compile, input, spec, workdir, sandbox) # ← backend runs run-confined + if r.failed: + failure = Failure(r.errors); emit("build", r.errors); continue + if (jp := mod.judge_prompt(input, spec)) is not None: # ← backend (pure); default None → skip + review = await run_llm_agent(env, jp, structured=JUDGE) # ← Python: LLM + if not review.accept: + failure = Failure(review.feedback); continue + break + else: + return GaveUp(f"did not pass compile/judge in {max_attempts} attempts") + + # validate each unit — host owns enumeration + scheduling (serial today; see §5). + async def check(u): + v = await asyncio.to_thread(mod.validate, input, spec, u.unit, workdir, sandbox) + emit("verdict", {"unit": u.unit, **v}) # live per-unit notice + return u, v + results = [await check(u) for u in units] # or asyncio.gather to fan out + return Formalized(artifact_text=spec, + property_units=[(u.property, [u.unit]) for u, _ in results], + verdicts={u.unit: v for u, v in results}) +``` + +- The **author system prompt is already backend-definable** (`_llm_agent._split_prompt`), so + `Prompt.system` drops in. +- **RAG is unchanged**: the host builds the author's knowledge-base search tools as it does + today (Crucible's external `crucible_kb`, imported from the committed + `rust/crucible-app/crucible_kb.rag.json` via `composer.scripts.rag_import`) and passes them in + `env.rag_tools`. Moving the corpus into the wheel is a possible later step, not part of this change. +- **Sandbox policy** is built once by Python (`SandboxConfig.build_policy(workdir)`, the + existing recipe) and passed straight through as `Sandbox` — Python keeps ownership of the + *intent*; the backend only assembles it into a `run-confined` argv. +- **Events + caching are Python's** (it knows the phase and each result), so the `Emit` + command and the `Emitter` shim disappear and the pipeline's result cache subsumes the loop's + scratch cache. +- `fetch_verdicts` disappears for self-contained backends — verdicts come from `validate`. + +`CrucibleFormalizer.formalize` becomes: prepare the crate → run the `setup` artifact (author + +`compile`, no validate) to get the fixture → run the `component` artifact through the loop +above. Two readable `await`s over backend callouts; no state machine. + +## 4. How Crucible implements it + +- `units(input)` → one `Unit{ property: title, unit: "c_" }` per invariant (the current + `_unique_slugs` mapping, moved into the backend). `kind="setup"` ⇒ `[]`. +- `author_prompt(input, failure)` → `kind="setup"` ⇒ the fixture prompt; `kind="component"` ⇒ + the all-invariants prompt (listing the `units`' fn names); `failure` ⇒ append revise context, + dispatched on `failure.kind`: a `Compile` failure appends the prior draft + compiler errors + (`revise_suffix`), a `Judge` rejection appends the prior draft + review feedback framed as + *not* a build error (`judge_revise_suffix`). +- `judge_prompt` → overridden for `kind="component"` (skipped for the fixture): a reviewer turn + modeled on Foundry's feedback judge, retargeted to fuzzing — the load-bearing question is + reachability (can the fuzzer drive a state where the invariant could fail?). Emits the + `{accept, feedback}` JSON the host's `_parse_judge` reads. Whenever a wheel supplies a judge for an + input, the host runs it as a `request_review` **tool inside the author session** — the author + self-revises and can only finalize an accepted draft (see `docs/crucible-judge-in-loop.md`) — so + there is no separate post-authoring judge turn. A wheel with no judge (`judge_prompt → None`) gets + the plain single-shot author. +- `compile(input, spec, workdir, sandbox)` → `run_confined(sandbox, "crucible", ["run", program, + probe, "--release", "--dry-run"], files={"fuzz//src/main.rs": fixture+spec}, workdir)`; + `Failed{errors: tail}` if `is_build_error(out)` or nonzero exit, else `Ok`. +- `validate(input, spec, target, workdir, sandbox)` → `run_confined(sandbox, "crucible", ["run", + program, target, "--release", "--mode", "explore", "--timeout", n], files={main.rs: fixture+spec}, + workdir)`. `target` is the fuzz feature (Crucible's single `c_invariants`, shared by every + invariant — see `docs/crucible-unit-granularity.md`). The run is classified once (`[FUZZ_FINDING]` + → BAD, exit 0 → GOOD, build markers → `BuildFailed`) and **attributed by the backend** to a + `Verdict` per report unit the target covers (`ValidateOutcome::Verdicts`): a counterexample is + pinned to the property whose title the finding names (assertions are tagged `[]`), the rest + held GOOD. The host records these verbatim — it owns no verdict logic and never parses a finding. +- `finalize` → unchanged. + +Every piece is the body of a current `resume` arm turned into a pure/blocking function — +directly unit-testable in Rust (feed a spec + a fake `crucible` on `PATH`, assert the command +or the verdict). + +## 5. Why there is still no async runtime in the wheel + +`compile`/`validate` block on a child process. To keep them off the event loop **without** +tokio or a Python-await bridge: + +- the `#[pyfunction]` wraps its subprocess work in `Python::allow_threads(|| …)`, releasing the + GIL for the (minutes-long) build/fuzz; +- Python calls it with `await asyncio.to_thread(mod.compile, …)`. + +So the wheel stays **synchronous** — no tokio, no `future_into_py`/`into_future`, no +GIL-across-await marshaling, no contextvar-through-a-bridge risk (the earlier +async-into-Rust concern in [rust-ioc-loop.md]). The backend just spawns and waits; Python +just moves the wait to a thread. This is the whole reason to spawn `run-confined` *directly* +rather than await a Python runner: the sandbox is already a standalone binary, so the backend +needs nothing from Python at run time except the policy data. + +Because `validate` is **per-unit**, the host *can* fan the units out (`asyncio.gather` of +`to_thread` calls) for free at the Python layer — but actually running Crucible fuzz builds +concurrently against the **shared crate** hits the binary-name collision from +[crucible-unit-granularity.md §7](./crucible-unit-granularity.md) (every feature builds the +same `invariant_test`), so real parallelism still needs the `--binary-in` build/fuzz split. +The per-unit signature is the right shape regardless (the host owns enumeration/scheduling); +it runs **serial today** and becomes parallel when §7 is done — no API change either way. + +### Scope: self-contained vs run-service backends + +This shape fits a backend whose checker is a **local tool** (Crucible: `cargo`/`crucible`; +a future soroban twin). A backend whose "validate" is a **remote/Python service** (the Certora +prover) can't spawn it under `run-confined`; that path would keep a Python-side effect for the +service call (or the real prover stays the Python `ProverBackend` it already is, not a Rust +wheel). The echoprover demo's prover step is the one place this matters — it either keeps a +thin Python `run_prover` hook or drops the prover step; Crucible, the actual target, is fully +self-contained. + +## 6. What changes, concretely + +**Delete** (SDK + host): `Command`/`Observation`, `FormalizeSession`/`resume`, the +`RustSession` pyclass, `drive_session`, the `Emitter` shim, the loop's scratch cache, and the +`RealEffects` `run_command` routing (the backend now spawns `run-confined` itself). +**Keep**: `descriptor`/`validate_preconditions`/`finalize`, the sandbox *policy* layer +(`SandboxPolicy`/`SandboxConfig.build_policy`), `run-confined` itself, the existing RAG +mechanism (`crucible_kb`, imported via `composer.scripts.rag_import`, surfaced as `env.rag_tools`), `_llm_agent` +(now called directly by the pipeline), and `run_prover`/`run_feedback` only for the run-service +exception. +**Add**: the `Backend` callouts above (pure `descriptor`/`validate_preconditions`/`units`/ +`author_prompt`/`judge_prompt`/`finalize` + the two GIL-releasing blocking `compile`/`validate`), the shared +`autoprover_sdk::run_confined` helper (the `SandboxPolicy` → `run-confined` argv assembly moves +here, out of the Python `launcher` provider), and one generic `formalize` in the core pipeline. + +Net: the FFI goes from "a coroutine hand-compiled into a state machine + an 8-variant effect +protocol" to "pure prompt/unit callouts + `compile`/`validate` that run the toolchain via one +shared launcher helper." The control flow lives in one Python function that reads like the +procedure it is. + +## 7. Security invariant (unchanged, and clearer) + +The rule ([command-sandbox.md](./command-sandbox.md) §2/§7): the **command line** (`program` + +`args`) is authored by trusted compiled code; only file **contents** may derive from the LLM. +Here `compile`/`validate` construct `program`/`args` in Rust and place them after `--`; the +`Sandbox` policy (the `run-confined` flags before `--`) is authored by Python. The LLM's spec +is written into the `--rw` workdir and confined by Landlock. Two audit points, both trivial: +the pure command-building in the backend, and `SandboxConfig.build_policy` in Python. A test +asserts the launched argv is `[run-confined, …policy…, "--", "crucible"/"cargo", …]` and never +contains LLM text. + +## 8. Decisions and deferrals + +Resolved: + +- **Shared launcher helper — yes.** `autoprover_sdk::run_confined` owns the `Sandbox` → + `run-confined` argv assembly; every backend calls it (§2). +- **`compile` and `validate` stay separate.** No fusing the dry-run into the first fuzz — one + whole-spec compile, then per-unit validation (§2). +- **`validate` is per-unit.** The host enumerates units (`units(input)`) and calls `validate` + once per unit, so the backend never discovers units and the host owns scheduling; serial + today, parallel once the §7 build-collision split lands (§5). +- **Judge — API present, default no-op.** `judge_prompt` stays in the trait but defaults to + `None` (skip), so the loop has a judge step wired in and a backend can opt in later without + a reshape. No backend overrides it today. +- **Wheel-owned RAG — deferred.** RAG stays the external `crucible_kb` mechanism; a + `knowledge_base()` callout the host indexes is an additive later step, out of scope here. + +Still open: + +- Timeout mechanics in the SDK helper (std `Command` has no built-in timeout — a wait-thread + vs a small `wait-timeout`-style dependency). +- Where `units`' slug-uniqueness lives if two backends want to share it (SDK helper vs + per-backend), and whether `author_prompt` should take the `units` list explicitly rather + than re-deriving it internally. diff --git a/docs/rust-formalization-backends.md b/docs/rust-formalization-backends.md new file mode 100644 index 00000000..dc765f1f --- /dev/null +++ b/docs/rust-formalization-backends.md @@ -0,0 +1,552 @@ +# Design Doc — Rust Formalization Backends via PyO3 + +> How to implement an AutoProver formalization backend in Rust and plug it into the +> generic Python pipeline through PyO3, what the boundary looks like, the additional +> work required to let Rust call *back* into the async Python services, and a +> hypothetical sketch of the CVL prover backend rewritten in Rust. +> +> Companion to [formalization-abstraction.md](./formalization-abstraction.md), which +> defines the backend seam this leans on, and +> [application-abstraction.md](./application-abstraction.md), which covers how a backend +> is wired into a runnable application. Read the formalization doc first — this document +> assumes its vocabulary (`FormT`, `Formalizer`, `PreparedSystem`, the phase chain). For +> the *rest* of the vertical around a Rust backend — phase enum, entry point, frontend, +> `main()` — see [rust-applications.md](./rust-applications.md). + +--- + +## 1. Problem & motivation + +The formalization seam ([formalization-abstraction.md §3](./formalization-abstraction.md)) +is deliberately narrow: a backend is any object that structurally satisfies the +`PipelineBackend` protocol and hands the generic driver three immutable phase objects +(`PipelineBackend → PreparedSystem → Formalizer`). The driver in +[composer/pipeline/core.py](../composer/pipeline/core.py) never imports a concrete +backend — it moves opaque `FormT` values around and never reads a field. + +We want to author backends (or performance-critical parts of them) in **Rust**: a native +verification engine, a fast artifact transformer, a solver driver, or a +whole-backend reimplementation that only borrows the shared analysis/extraction/report +machinery. **PyO3** is the bridge — it lets a Rust crate expose functions and classes +that Python can call as if they were native. + +The seam being structural (a `Protocol`, not a base class you must subclass) is what makes +this tractable: nothing in the driver needs to know a backend is "really" Rust. The +question is entirely about the **boundary** — what crosses it, in which direction, and +synchronously or not. + +### Design goals + +1. **Confine the PyO3 surface.** The FFI boundary should be as small, synchronous, and + serde-friendly as the backend allows. Every awaitable, pydantic model, or deep object + graph that crosses the boundary is a cost. +2. **Reuse the driver unchanged.** Caching, the artifact store, the report, and the + concurrency structure are driver-owned; a Rust backend inherits them for free + ([formalization-abstraction.md §7](./formalization-abstraction.md)). +3. **Keep the main tree pure-Python.** The project builds with `setuptools` today + ([pyproject.toml](../pyproject.toml)); adding Rust should not force a build-system + rewrite of `ai-composer` itself. + +--- + +## 2. The boundary, and what crosses it + +The whole interface a backend must implement is five async methods plus a handful of +properties and one sync mapper ([formalization-abstraction.md §3](./formalization-abstraction.md)): + +| Member | Direction | Kind | +|---|---|---| +| `prepare_system` | driver → backend | `async` | +| `PreparedSystem.prepare_formalization` | driver → backend | `async` | +| `Formalizer.formalize` | driver → backend | `async` | +| `Formalizer.fetch_verdicts` | driver → backend | `async` | +| `Formalizer.finalize` | driver → backend | `async` (optional hook) | +| `to_artifact_id`, `extra_report_inputs`, the four properties | driver → backend | sync | + +Four properties of this boundary drive the entire design. + +### 2.1 It is thoroughly `async` + +Every real method is a coroutine, driven by `asyncio.create_task` / +`asyncio.gather(..., return_exceptions=True)` in the driver +([core.py](../composer/pipeline/core.py)). PyO3 does not make Rust `async fn` visible to +Python for free — see [§4](#4-the-async-problem-three-tiers). + +### 2.2 The result type must stay cacheable + +The driver keys the cache on `formalizer.formalized_type` and calls +`cache_put(result)` / `cache_get(type)` ([core.py](../composer/pipeline/core.py)). Both +existing results — `GeneratedCVL` ([cvl_generation.py](../composer/spec/cvl_generation.py)) +and `GeneratedFoundryTest` ([foundry/author.py](../composer/foundry/author.py)) — are +pydantic v2 `BaseModel`s that serialize cleanly. A raw `#[pyclass]` result would have to +satisfy *both* structural protocols (`FormalResult` + `ReportableResult`) **and** +round-trip through the cache's (de)serialization. + +> **Decision:** keep `FormT` a Python pydantic model. Rust does the work and returns +> plain data; the pydantic result is constructed on the Python side (or PyO3 instantiates +> the pydantic class). Caching, the artifact store, and the report all keep working +> unchanged. + +### 2.3 The inputs are a deep object graph + +`formalize` receives a `ContractComponentInstance` (a dataclass wrapping a pydantic +`SourceApplication` / `HarnessedApplication` graph — [system_model.py](../composer/spec/system_model.py)), +a `list[PropertyFormulation]`, a `WorkflowContext`, and a `PipelineRun`. Reading these +from Rust via GIL-bound attribute access is verbose and brittle. + +> **Decision:** marshal at the boundary. The thin Python adapter serializes the *slice* +> the Rust backend needs (`model_dump()` / JSON) and passes that in; Rust deserializes +> into its own `serde` structs and never touches Python objects directly. +> `PropertyFormulation` and the result models are trivially JSON-able. + +### 2.4 It is a `Protocol`, not a base class + +Neither `ProverBackend` nor `FoundryBackend` inherits from `PipelineBackend` — they match +by shape. So the "backend" the driver sees can be a **thin Python adapter** that +implements the protocol and delegates to the Rust extension. That adapter is where all the +async-wrapping, marshalling, and pydantic-construction live. + +--- + +## 3. Recommended architecture + +``` +┌─────────────────────────────────────────────────────────────────┐ +│ composer/pipeline/core.py (generic driver — UNCHANGED) │ +└───────────────┬─────────────────────────────────────────────────┘ + │ holds an opaque PipelineBackend[...] (structural) + ▼ +┌─────────────────────────────────────────────────────────────────┐ +│ composer/rustbackend/adapter.py (thin PYTHON adapter) │ +│ • implements the async protocol methods │ +│ • async def formalize(...): │ +│ payload = _marshal(feat, props) # pydantic → JSON │ +│ raw = await asyncio.to_thread(_rs.formalize, payload) │ +│ return GeneratedRustResult.model_validate(raw) # → pydantic│ +│ • keeps FormT a pydantic BaseModel │ +└───────────────┬─────────────────────────────────────────────────┘ + │ sync, serde-friendly FFI calls + ▼ +┌─────────────────────────────────────────────────────────────────┐ +│ _rustbackend (PyO3 / maturin extension wheel) │ +│ #[pyfunction] fn formalize(payload: &str) -> PyResult<String> │ +│ • serde_json::from_str → own structs │ +│ • py.allow_threads(|| heavy_rust_work()) │ +│ • returns serde_json::to_string(&result) │ +└─────────────────────────────────────────────────────────────────┘ +``` + +This confines the entire PyO3 surface to **synchronous, `&str`-in / `String`-out +functions**, sidestepping three problems at once: no Tokio↔asyncio bridge, no +pydantic-in-Rust, no cache-serialization of a `#[pyclass]`. + +### 3.1 Packaging + +Two options; the first is strongly preferred. + +- **(a) Separate wheel — recommended.** The Rust crate is its own maturin project + producing a `_rustbackend` extension wheel. `ai-composer` gains one dependency; its + `[build-system]` stays on setuptools. Least invasive, independently versioned/CI'd. +- **(b) Mixed maturin build for `ai-composer` itself.** Rewrites `[build-system]` and + `[tool.setuptools.*]`. Only worth it if the Rust and Python are co-developed in + lockstep. + +`requires-python = ">=3.12"` is a hard floor (the seam uses PEP 695 generics), so build +abi3 wheels for `cp312+`. The project is managed with `uv`; add the crate as a `uv` +source (path dep during development, published wheel in CI). + +### 3.2 Two small wiring changes (not PyO3-specific) + +These are the same steps any new backend takes ([application-abstraction.md](./application-abstraction.md)): + +1. Add a wrapper `run_<rust>_pipeline` that constructs the adapter backend and calls the + generic `run_pipeline` — mirror [foundry/pipeline.py](../composer/foundry/pipeline.py), + the simplest reference backend, plus a `[project.scripts]` CLI entry. +2. Widen the `ReportBackend` literal — currently a closed `Literal["prover","foundry"]` + at [report/schema.py](../composer/spec/source/report/collect.py) — to include the new + backend tag. + +### 3.3 GIL and errors + +- Release the GIL (`py.allow_threads(|| ...)`) around heavy Rust work, so the + `asyncio.to_thread` offload in the adapter yields real concurrency across the + semaphore-bounded per-component fan-out. +- Map Rust `Err`/panics to Python exceptions (`PyResult`, `catch_unwind` at the FFI edge). +- For the *declined* outcome, return the existing `GaveUp(BaseModel)` (`{reason: str}`) + from the adapter — the driver treats it as a normal, reportable result, **not** a crash + ([formalization-abstraction.md §8](./formalization-abstraction.md)). Reserve raised + exceptions for genuine failures the driver should capture via `return_exceptions=True`. + +--- + +## 4. The async problem, three tiers + +Extensions **will** need LLM authoring loops — a backend that authors an artifact, runs a +verifier, reads the feedback, and revises, turn after turn, is the whole point. That work +is inherently a dance with async Python services (the LLM, the prover tool, the feedback +judge, the cache). The naive way to give Rust that capability is to let Rust `await` Python +coroutines directly — the full async FFI bridge. **We can avoid that**, because of one fact +about how this codebase is already built. + +### 4.0 The enabling fact: the loop already separates *decide* from *do* + +The authoring loop is a langgraph `StateGraph` (`initial → tools ⇄ tool_result → __end__`), +but underneath, every node is a **pure generator** ([graphcore/graph.py](../graphcore/graphcore/graph.py)): + +```python +type PureFunctionGenerator[ResT] = Generator[list[AnyMessage], BaseMessage, ResT] +``` + +A node *yields* the messages it wants sent to the LLM and *receives* the reply via +`.send()`. A tiny adapter, `_stitch_async_impl`, performs the one awaited effect +(`res = await llm_impl(d)`) between the yield and the send. **The "decide next action" logic +is already pure and synchronous; only the effect in the middle is async.** The routing +predicates that end the loop — `should_end`, `ai_message_router`, `check_completion` — +are ordinary side-effect-free functions of the state dict +([cvl_generation.py](../composer/spec/cvl_generation.py)). + +That split is exactly what we relocate across the FFI boundary. Rust owns the pure decider; +Python keeps owning the async effects. No bridge required. + +### 4.1 Tier 1 — self-contained Rust (`asyncio.to_thread`) + +If the Rust work does its own thing (spawns a solver, shells out, computes an artifact) and +needs no Python callbacks, the adapter wraps a **synchronous** Rust call in a thread: + +```python +async def formalize(self, label, feat, props, ctx, run) -> FormT | GaveUp: + payload = _marshal(feat, props, self._config) + raw = await asyncio.to_thread(_rs.formalize, payload) # sync Rust, off the loop + obj = json.loads(raw) + if obj["kind"] == "gave_up": + return GaveUp(reason=obj["reason"]) + return GeneratedRustResult.model_validate(obj["result"]) +``` + +This is exactly the pattern the CVL backend already uses for its off-thread prover query +([formalization-abstraction.md §4.5](./formalization-abstraction.md)). No new +infrastructure. Good for pure computation, useless for an LLM loop. + +### 4.2 Tier 2 — inversion of control: Rust decides, Python does the I/O ⭐ + +**This is the recommended way to give an extension an LLM authoring loop.** It requires +**no async bridge at all** — the FFI stays 100% synchronous. + +Python owns the async event loop and *every* effect (LLM call, prover run, feedback judge, +cache). Rust is a **pure, synchronous state machine**. Python calls one sync FFI function, +`resume(handle, observation) -> Command`; Rust decides the next action and returns a +`Command`; Python performs that async effect and calls `resume` again with the outcome. +This is the classic **sans-I/O** pattern, and it mirrors the `PureFunctionGenerator` split +above one-to-one — the generator's `yield` becomes a returned `Command`, its `.send()` +becomes the next `resume` argument. + +**The command vocabulary** (a closed enum, JSON across the wire) is read straight off the +real loop's effects: + +```rust +enum Command { // Rust → Python: "please perform this effect" + CallLlm { messages: Json }, // an LLM turn (initial / tool_result node) + RunProver { spec: String, config: Json, rules: Option<Vec<String>> }, + RunFeedback { spec: String, skipped: Json, rebuttals: Json }, // nested judge agent + CacheGet { key: String }, + CachePut { key: String, value: Json }, + Summarize { messages: Json }, // history-compaction LLM turn + Publish { result: Json }, // ⇒ FormT ; loop ends + GiveUp { reason: String }, // ⇒ GaveUp ; loop ends +} + +enum Observation { // Python → Rust: "here is the result" + LlmReply(Json), ProverResult(Json), FeedbackResult(Json), + Cached(Option<Json>), Ack, Start, +} +``` + +The Python driver is a plain `while` loop with **no bridge, no Tokio, no `pyo3-async-runtimes`**: + +```python +async def formalize(self, label, feat, props, ctx, run) -> FormT | GaveUp: + handle = _rs.new_session(_marshal(feat, props, self._config)) # sync: build Rust state + obs = _rs.START + while True: + cmd = json.loads(_rs.resume(handle, obs)) # sync FFI: Rust decides + match cmd["kind"]: + case "call_llm": obs = await self._llm(cmd["messages"]) # async, in PYTHON + case "run_prover": obs = await self._verify_spec(cmd) # async, in PYTHON + case "run_feedback": obs = await self._feedback_judge(cmd) # async, in PYTHON + case "cache_get": obs = await ctx.cache_get_raw(cmd["key"]) # async, in PYTHON + case "cache_put": await ctx.cache_put_raw(cmd["key"], cmd["value"]); obs = _rs.ACK + case "publish": return GeneratedRustResult.model_validate(cmd["result"]) + case "give_up": return GaveUp(reason=cmd["reason"]) +``` + +Because Python owns the loop, the extension inherits everything for free: `run.runner(...)` +task/telemetry wrapping, the existing `verify_spec` prover tool, the +`property_feedback_judge` sub-agent, the hierarchical cache, cancellation +(`asyncio.CancelledError` just unwinds the Python loop), and structured-concurrency +isolation under `gather(return_exceptions=True)`. Rust contributes only the *policy*: prompt +construction, response interpretation, the validation-gate check, and the publish/give-up +decision. + +**Ergonomics for the Rust author — two flavors:** + +- **Explicit state machine.** Author writes `fn resume(&mut self, obs) -> Command` over an + explicit state enum. Simple, zero dependencies, but verbose for a rich loop. +- **Self-driven coroutine (recommended).** Author writes *linear, idiomatic* `async fn` + Rust, but `await`s custom `HostCall` futures whose leaf `poll` suspends a **Rust-owned, + single-threaded** executor and hands a `Command` out through `resume`. Python resumes by + feeding the `Observation` back in. The executor never leaves Rust and never touches + asyncio, so there is still **no cross-language async bridge** — it's a coroutine whose + "syscalls" happen to be Python async operations. This gives the write-it-linearly feel of + the full bridge at the cost of a ~200-line effect runtime, entirely in Rust. + +**What Tier 2 must reproduce that langgraph gives for free:** + +- **State threading.** The loop state is a flat, reducer-merged dict — `curr_spec`, + `skipped`, `property_rules`, `validations: dict[str,str]`, `required_validations`, + `rule_skips`, `config`, `prover_link`, `messages` ([author.py](../composer/spec/source/author.py)). + Rust holds this as its session struct; each `Command`'s observation merges in exactly as + the langgraph reducers do today. +- **The validation gates.** Publication is gated by `check_completion`: a content digest of + `(spec, skipped)` must match a stored digest for each of `required_validations` + (`["feedback","prover"]`), and **any spec edit invalidates both** by changing the digest. + Rust owns this predicate — it is pure — and only emits `Publish` when it passes; otherwise + it emits another `CallLlm` with the rejection reason, exactly as `PublishResultTool` does + today. +- **A turn budget.** langgraph's `recursion_limit` bounds the loop; Rust owns the counter and + emits `GiveUp` when it's exhausted. +- **Injection.** Today tools reach state/identity via langgraph `InjectedState` / + `InjectedToolCallId` and a `contextvars` runtime. Across FFI there is no ambient context — + the adapter passes what the effect needs explicitly in each `Command`. + +The one genuine constraint: **effects must be coarse-grained** — one `resume` per LLM turn +or per tool call, not per token. That is already the loop's natural granularity, so it costs +nothing here. Treat `RunFeedback` as a single opaque effect (Python runs the whole nested +judge agent) rather than recursing the state machine. + +### 4.3 Tier 3 — the full async bridge (only if Rust must *drive*) + +If Rust must genuinely `await` Python coroutines from inside deeply nested Rust `async` +code — e.g. it spawns its own concurrent Tokio tasks that each need to call Python +mid-flight — then and only then do you need `pyo3-async-runtimes` (Python awaitable ↔ Rust +`Future` via `into_future` / `future_into_py`), a pinned asyncio-loop ↔ Tokio-runtime +pairing, two-way cancellation translation, and panic-isolation so a Tokio task abort +doesn't kill the process. This roughly triples the effort and the test surface. + +> **Recommendation.** Deliver LLM authoring loops at **Tier 2** — inversion of control. +> It gives extensions the full author→verify→revise loop with none of the bridge's cost, +> and it fits the codebase's existing decide/do seam exactly. Reach for Tier 3 only if a +> concrete backend must drive concurrent async Python from within Rust — a need none of the +> current backends have. + +--- + +## 5. Hypothetical: the CVL prover backend in Rust + +To make the tiers concrete, here is how the CVL backend +([formalization-abstraction.md §4](./formalization-abstraction.md)) *might* be structured +in Rust. This is illustrative, not a proposal to rewrite it. + +### 5.1 What maps cleanly (Tier 1 candidates) + +The CVL backend's heavy, self-contained steps are natural Rust: + +- **`fetch_verdicts`** — resolves each spec's prover run and rolls per-rule outcomes into + `Verdict`s ([formalization-abstraction.md §4.5](./formalization-abstraction.md)). Pure + data-in/data-out over the prover output; already runs off-thread today. In Rust: + + ```rust + #[derive(Deserialize)] + struct ReportInput { name: String, final_link: Option<String>, /* ... */ } + #[derive(Serialize)] + struct RuleVerdict { rule: String, outcome: String, line: Option<u32>, + duration_seconds: Option<f64>, unit_file: Option<String> } + + #[pyfunction] + fn fetch_verdicts(payload: String) -> PyResult<String> { + let inp: ReportInput = serde_json::from_str(&payload)?; + let verdicts: Vec<RuleVerdict> = query_prover_output(&inp)?; // native HTTP/parse + Ok(serde_json::to_string(&verdicts)?) + } + ``` + + The adapter turns the returned list back into `dict[RuleName, Verdict]`. + +- **`finalize`** — builds the `{spec → prover-run link}` map and writes + `components_to_prover_runs.json` ([formalization-abstraction.md §4.6](./formalization-abstraction.md)). + Trivial serde + file write. + +- **The artifact bundle** — emitting the `.spec` + rendering the `.conf` (base config + + fixed run overlay) is string/JSON assembly, a good fit for a Rust `ArtifactStore` + helper, though the `ArtifactStore` object itself can stay Python and call a Rust + formatter. + +- **`GeneratedCVL` as `FormT`** stays a Python pydantic model + ([cvl_generation.py](../composer/spec/cvl_generation.py)); Rust returns its fields as + JSON and the adapter does `GeneratedCVL.model_validate(...)`. Its protocol methods + (`property_units()`, `artifact_text`, `output_link`) remain Python one-liners so the + cache/report keep working. + +### 5.2 The authoring loop — Tier 2, inversion of control + +`formalize` for CVL is the interesting case: it is *not* self-contained. `batch_cvl_generation` +([author.py](../composer/spec/source/author.py)) runs an **LLM agent graph to a fixpoint**, +interleaving: + +- LLM authoring turns, +- the `verify_spec` prover tool (an async service call), +- the `property_feedback_judge` agent ([feedback.py](../composer/spec/feedback.py)), +- two hard validation gates (`PROVER_VALIDATION_KEY`, `FEEDBACK_VALIDATION_KEY`) before the + agent may publish. + +Under [§4.2](#42-tier-2--inversion-of-control-rust-decides-python-does-the-io-), Rust owns +this loop as a **synchronous decider** while Python performs each async effect. The Rust +side is a plain state machine over the same state the langgraph loop threads today: + +```rust +// Rust: a pure step function. No async, no PyO3 awaitables — just decide the next effect. +fn resume(session: &mut Author, obs: Observation) -> Command { + match obs { + Observation::Start => Command::CallLlm { messages: session.opening_prompt() }, + Observation::LlmReply(msg) => session.interpret(msg), // draft edit? verify? publish? + Observation::ProverResult(r) => { session.record_prover(r); session.next() } + Observation::FeedbackResult(f) => { session.record_feedback(f); session.next() } + Observation::Cached(hit) => session.after_cache(hit), + Observation::Ack => session.next(), + } +} + +// session.next() enforces the SAME publish gate check_completion does today: +fn next(&mut self) -> Command { + if self.turns_left == 0 { return Command::GiveUp { reason: "turn budget exhausted".into() }; } + match self.gate() { // digest(spec, skipped) vs required keys + Gate::NeedProver => Command::RunProver { spec: self.spec(), config: self.conf(), rules: None }, + Gate::NeedFeedback => Command::RunFeedback { spec: self.spec(), skipped: self.skipped(), + rebuttals: self.rebuttals() }, + Gate::Ready(res) => Command::Publish { result: res }, // both digests fresh ⇒ publish + Gate::KeepAuthoring(reason) => Command::CallLlm { messages: self.reprompt(reason) }, + } +} +``` + +The Python adapter's driver loop ([§4.2](#42-tier-2--inversion-of-control-rust-decides-python-does-the-io-)) +maps each `Command` onto the *existing* async services — `self._llm`, the real `verify_spec` +tool, `property_feedback_judge`, `ctx.cache_*` — so Rust reuses all of them without ever +awaiting. The validation gates stay in Rust because `check_completion` is already pure: a +content digest of `(spec, skipped)` that must match a stored digest per `required_validation`, +with any spec edit invalidating both. **No `pyo3-async-runtimes`, no Tokio, no bridge.** + +### 5.3 `prepare_formalization` — orchestration stays in Python + +CVL's `prepare_formalization` ([formalization-abstraction.md §4.2](./formalization-abstraction.md)) +runs AutoSetup ∥ summaries ∥ structural-invariant formulation concurrently, then generates +`invariants.spec` once (with a cache short-circuit) and folds it into the resource set. This +is async orchestration of Python services — easiest left in the Python adapter. The +invariant *formulation logic* (once it has its inputs) and the invariant CVL authoring can +each be a Tier-1 helper or reuse the Tier-2 loop, but the `gather`/cache dance stays Python. + +### 5.4 The pragmatic split + +A realistic Rust CVL backend would be **hybrid**: + +| Method | Tier | Where it lives | +| --- | --- | --- | +| `prepare_system` (harness lift, prover-tool build) | — | Python adapter | +| `prepare_formalization` (concurrency + cache orchestration) | — | Python adapter | +| `formalize` **decider** (prompt policy, gate check, publish/give-up) | 2 | **Rust** state machine | +| `formalize` **effects** (LLM, `verify_spec`, feedback judge, cache) | 2 | Python adapter loop | +| `fetch_verdicts` (prover-output parse → verdicts) | 1 | **Rust** | +| `finalize` (run-link map) | 1 | **Rust** | +| artifact formatting (`.spec`/`.conf`) | 1 | **Rust** helper, Python `ArtifactStore` shell | +| `GeneratedCVL` (`FormT`) | — | Python pydantic | + +Every native-speed win (verdict parsing, artifact assembly, and now the authoring *policy* +itself) lands in Rust, and the LLM authoring loop works end-to-end — all without the Tier-3 +async bridge. + +--- + +## 6. Work breakdown + +### Phase 0 — spike (Tier 1, throwaway) +- Stand up a maturin crate producing a `cp312` abi3 wheel; import it from `ai-composer`. +- Prove the round-trip: JSON payload → Rust → `serde` structs → JSON result → + `pydantic.model_validate`. +- Confirm `py.allow_threads` + `asyncio.to_thread` gives real concurrency under the driver. + +### Phase 1 — a self-contained Rust backend (Tier 1) +- Thin Python adapter implementing the async protocol; all callbacks stay in Python. +- Rust owns whichever methods are self-contained (e.g. a native `fetch_verdicts`/`finalize`, + or a whole self-contained verifier). +- Marshalling helpers (`_marshal` / result validation); keep `FormT` pydantic. +- Wiring: `run_<rust>_pipeline`, CLI entry, widen `ReportBackend`. +- Tests: cache hit/miss round-trips, `GaveUp` path, exception → `ComponentOutcome`. + +### Phase 2 — the LLM authoring loop via inversion of control (Tier 2) + +This is the milestone that unlocks the capability extensions actually need. **No async +bridge.** + +- Define the `Command` / `Observation` JSON enums (the effect vocabulary of one turn). +- Rust: the `new_session` / `resume` sync FFI pair — the pure decider, holding the + loop state (`curr_spec`, `skipped`, `validations`, `rule_skips`, `config`, turn counter), + with the `check_completion` digest gate reproduced in Rust. +- Python: the adapter driver loop mapping each `Command` onto the *existing* async services + (`self._llm`, `verify_spec`, `property_feedback_judge`, `ctx.cache_*`). +- Decide the Rust-author ergonomics: ship the explicit-state-machine API first; add the + self-driven-coroutine effect runtime if authors want linear code. +- Tests: a full author→verify→feedback→publish trace; gate staleness on spec edit; + turn-budget give-up; `CancelledError` unwinds cleanly. + +### Phase 3 — full async bridge (Tier 3, only if a backend must *drive*) +- Adopt `pyo3-async-runtimes`; pin the asyncio-loop/Tokio-runtime pairing. +- Two-way cancellation translation; panic isolation so a Tokio abort ≠ process abort. +- Verify structured-concurrency parity (per-component isolation under + `gather(return_exceptions=True)`). + +### Cross-cutting + +- Build/CI: cross-platform abi3 wheels, `uv` source wiring, reproducible Rust toolchain. +- Docs: fold the final `Command`/`Observation` ABI and the marshalling schemas into + [formalization-abstraction.md §9](./formalization-abstraction.md)'s "new backend" checklist. + +--- + +## 7. Open questions + +1. **Do we ever need Tier 3?** Tier 2 (inversion of control) delivers the LLM authoring + loop with no bridge. Tier 3 only pays off if a concrete backend must drive concurrent + async Python from within Rust — decide per backend, not up front. +2. **Rust-author ergonomics for Tier 2** — ship the explicit `resume` state machine, or + invest in the self-driven-coroutine effect runtime so authors write linear `async` Rust? + Start with the former; graduate to the latter only if the loops get complex enough to + warrant it. +3. **Wheel vs mixed build** — does any consumer need `ai-composer` itself to remain a pure + sdist-installable package? If so, the separate-wheel path is mandatory. +4. **ABI stability** — the `Command`/`Observation` enums and the marshalling JSON schemas + become a versioned contract between the wheel and `ai-composer`. Where does that schema + live, and how is it version-checked at load time? +5. **Effect granularity** — the IoC loop assumes one `resume` per turn/tool-call. Is any + effect an extension might want finer-grained than that (e.g. streaming LLM tokens)? If so + it must still be batched to a turn boundary, or it forces Tier 3. +6. **Observability** — the driver's `run.runner(TaskInfo(...))` wrapping gives per-task + telemetry/UI rows. A Tier-1 Rust method invoked via `asyncio.to_thread` still sits inside + one `runner` task (good); a Tier-2 loop's per-turn effects should each be wrapped by the + Python adapter in `run.runner(...)` so they stay visible in the TUI. + +--- + +## 8. Key files + +| Concern | File | +|---|---| +| Driver + abstraction definitions | [composer/pipeline/core.py](../composer/pipeline/core.py) | +| Result protocols (`FormalResult`, `ArtifactIdentifier`) | [composer/spec/types.py](../composer/spec/types.py) | +| `ReportableResult`, `Verdict`, `collect` | [composer/spec/source/report/collect.py](../composer/spec/source/report/collect.py) | +| CVL backend (reference for the sketch) | [composer/spec/source/pipeline.py](../composer/spec/source/pipeline.py) | +| CVL authoring loop + completion tools (`batch_cvl_generation`, `PublishResultTool`) | [composer/spec/source/author.py](../composer/spec/source/author.py) | +| `PureFunctionGenerator` decide/do seam + graph topology (the IoC precedent) | [graphcore/graphcore/graph.py](../graphcore/graphcore/graph.py) | +| `check_completion` validation-gate predicate + loop state (`FormT`, gate digest) | [composer/spec/cvl_generation.py](../composer/spec/cvl_generation.py) | +| Foundry backend (simplest backend to copy) | [composer/foundry/pipeline.py](../composer/foundry/pipeline.py) | +| `ReportBackend` literal to widen | [composer/spec/source/report/collect.py](../composer/spec/source/report/collect.py) | +| Packaging | [pyproject.toml](../pyproject.toml) | +| The seam this builds on | [formalization-abstraction.md](./formalization-abstraction.md) | diff --git a/docs/rust-ioc-loop.md b/docs/rust-ioc-loop.md new file mode 100644 index 00000000..f5f7a15f --- /dev/null +++ b/docs/rust-ioc-loop.md @@ -0,0 +1,241 @@ +# The Rust IoC loop: what it does, why it exists, and how to remove it + +Design note. Describes the inversion-of-control (IoC) effect loop that connects the +Python host (`composer/rustapp`) to a Rust backend wheel (`autoprover-sdk` + +`crucible-app`/`example-app`), why it is shaped the way it is, and a concrete option for +**eliminating it** in favour of a small set of *pure* Rust callouts driven directly by the +Python core pipeline. No code change yet — this is to decide the shape. + +## 1. What the loop is (mechanics) + +A Rust backend does not run its own control flow. It is a **pure synchronous decider**: +a state machine that, given the result of the last effect, returns the next *command* to +perform. Python owns the actual work (LLM turns, subprocesses, caching, events) and the +async event loop. + +The FFI surface a wheel exports (`export_app!`, [autoprover-sdk/src/lib.rs](../rust/autoprover-sdk/src/lib.rs)): + +| Function | Kind | Role | +|---|---|---| +| `descriptor() -> json` | **pure** | phases, event kinds, args, artifact layout, ecosystem, backend tag | +| `validate_preconditions(args) -> str?` | **pure** | e.g. "is there a `Cargo.toml`?" | +| `new_session(input) -> RustSession` | **stateful** | build a formalization state machine | +| `new_setup_session(input) -> RustSession?` | **stateful** | build the program-wide setup state machine (Crucible fixture) | +| `RustSession.resume(obs_json) -> cmd_json` | **stateful** | **the loop step** — one decision per call | +| `fetch_verdicts(input) -> json` | **pure** | per-unit report verdicts | +| `finalize(outcomes) -> json?` | **pure** | run-level artifact files | + +Everything except the **session** (`new_session`/`new_setup_session` + `resume`) is +already a pure, stateless callout. The loop is the one stateful exception. + +### The step protocol + +The Rust session speaks two closed enums ([lib.rs](../rust/autoprover-sdk/src/lib.rs)): + +- **`Command`** (Rust → Python): `CallLlm`, `RunProver`, `RunCommand`, `RunFeedback`, + `CacheGet`, `CachePut`, `Emit`, and the terminals `Publish` / `GiveUp`. +- **`Observation`** (Python → Rust): `Start`, `LlmReply`, `ProverResult`, + `FeedbackResult`, `Cached`, `CommandResult`, `Ack`. + +The Python driver `drive_session` ([composer/rustapp/loop.py](../composer/rustapp/loop.py)) +is the whole loop: + +```python +observation = {"kind": "start"} +for _ in range(max_steps): + command = json.loads(session.resume(json.dumps(observation))) # sync FFI hop + match command["kind"]: + case "call_llm": text = await effects.call_llm(command["messages"]); observation = {"kind": "llm_reply", "text": text} + case "run_command": r = await effects.run_command(...); observation = {"kind": "command_result", ...} + case "emit": await effects.emit(...); observation = {"kind": "ack"} + case "publish": return RustFormalized(command["result"]) # terminal + case "give_up": return GaveUp(command["reason"]) # terminal + ... +``` + +`Effects` is a protocol (`call_llm`, `run_command`, `run_prover`, `run_feedback`, +`cache_get/put`, `emit`), so a fake drives the decider in tests and `RealEffects` +([adapter.py](../composer/rustapp/adapter.py)) drives it in production. + +### A session in practice + +Crucible's `BatchSession` ([crucible-app/src/lib.rs](../rust/crucible-app/src/lib.rs)) is a +`{Start, AwaitDraft, AwaitFuzz, Done}` state machine holding `test_src`, `attempts`, `cur`, +`verdicts`. One conceptual procedure — + +> author all tests → for each feature: build+fuzz → interpret → (retry the *whole* harness +> on a build error) → publish the verdicts + +— is spread across `resume` arms, one per effect boundary, with the state threaded by hand +between calls. What is logically a `for` loop with a retry is expressed as a resumable +coroutine hand-compiled into an enum. + +## 2. Why it exists (the rationale — all still real) + +1. **Decide/do split across the FFI, with no async bridge.** All async I/O (LLM, subprocess, + Postgres, event streaming) lives in Python; Rust stays pure and synchronous. Every + `resume` is a fast blocking call, so there is **no `pyo3-async`/tokio bridge** and no Rust + async runtime. This mirrors the `PureFunctionGenerator` decide/do split the CVL author + already uses in Python — relocated across the language boundary. +2. **Testability.** Because effects are a protocol, the Rust decider's logic runs against a + fake with canned command results — no LLM, no toolchain + ([test_crucible_events.py](../tests/test_crucible_events.py)). +3. **The command-line security invariant.** `RunCommand` carries a **decider-authored** + `program` + `args`; only file *contents* may be LLM-derived. The IoC boundary makes this + explicit and enforceable: the LLM never chooses what runs (see the `RunCommand` doc + comment and [command-sandbox.md](./command-sandbox.md)). **Any replacement must preserve + this.** +4. **A backend-agnostic host.** The generic Python host drives *any* wheel through the same + loop + descriptor, so "a new backend is just a wheel." The `Command`/`Observation` protocol + is the uniform contract. + +## 3. What it costs + +- **State-machine bookkeeping.** Each backend re-expresses a straight-line author-gate + procedure as a resumable enum: explicit stages, hand-threaded fields, an `emit`-queue + shim (`Emitter`) to fire events *between* real commands. `BatchSession` is ~150 lines of + plumbing for a loop. +- **Two orchestration layers.** The Python core pipeline (`composer/pipeline/core.py`) + already orchestrates phases (analysis → extraction → formalize → report) with real async + and a result cache. The IoC loop is a *second*, nested orchestration inside `formalize`, + with its own mini-cache and event channel. Two models to understand. +- **The control flow is invisible where you look for it.** The interesting logic — author, + gate with a CLI, retry on failure, publish — is scattered across `resume` arms in Rust and + a dispatch `match` in Python, joined only by JSON round-trips. Neither side reads as the + procedure it is. +- **Ceremony per step.** Every decision is a JSON serialize → FFI hop → deserialize, plus a + `Command`/`Observation` variant to define and thread on both sides. + +## 4. The key observation + +**The decider carries no state that Python couldn't hold, and needs no control flow Python +can't express.** All real work is already in Python; the Rust "decisions" are pure functions +of the accumulated state (the draft, the attempt count, the command outputs). Resumability +is not intrinsic — it is an artifact of expressing an *author→gate→retry→publish* loop as a +coroutine that happens to suspend at each effect. Move the loop to Python and Rust needs only +a handful of **pure functions** at the decision points. + +And the shape is uniform across today's backends: + +| Backend / session | author | gate command(s) | interpret | publish | +|---|---|---|---|---| +| Crucible setup | fixture | `crucible run … --dry-run` | build error? → revise | fixture source | +| Crucible batch | all invariant tests | `crucible run <program> <feat>` per feature | `[FUZZ_FINDING]`→BAD else GOOD; build error→revise all | verdicts + property map | +| echoprover (demo) | spec (or cache hit) | `RunProver` | verified? | rules | + +All three are the same template: **author (with bounded revise-on-failure) → run +decider-authored gating command(s) → interpret results → publish or give up** — exactly the +author-gate loop the CVL/foundry Python backends already run. + +## 5. Proposal: replace the state machine with pure callouts + a Python driver + +Keep the four already-pure callouts (`descriptor`, `validate_preconditions`, +`fetch_verdicts`, `finalize`). **Delete** `new_session`/`new_setup_session`/`resume`, the +`RustSession` pyclass, the `Command`/`Observation` enums, and `drive_session`. **Add** a +small pure-callout contract per session kind, and drive it from one generic Python loop that +lives in the core pipeline. + +### The callouts (pure `json → json`, no session object, no state) + +1. `prompt(input, attempt, draft?, error?) -> {system?, instruction}` + The author prompt for attempt *N* — initial when `attempt == 0`, else the revise prompt + built from the prior `draft` + `error`. (Absorbs `author_prompt` + `revise_suffix` + + cheat sheets.) +2. `gate(input, draft) -> [Effect]` + The **decider-authored** gating commands for a draft, where + `Effect = Shell{program, args, files} | Prover{spec, config, rules} | Feedback{…}`. + *This is where the command-line security invariant lives* — `program`/`args` come from + compiled Rust, never the LLM. (Absorbs `fuzz_command` / `RunProver`.) +3. `interpret(input, draft, results) -> {status: ok | retry | give_up, error?, reason?}` + Classify the gate results: pass, retry (with the error text to feed the next `prompt`), + or give up. (Absorbs `is_build_error`, `[FUZZ_FINDING]` detection, the attempt policy is + Python's.) +4. `publish(input, draft, results) -> Formalized` + Assemble `artifact_text` + `property_units` + `verdicts` from the winning draft and its + results. (Absorbs `publish`.) + +### The generic Python driver (in the core pipeline) + +```python +async def author_gate(mod, session_input, effects, *, max_attempts, emit) -> Formalized | GaveUp: + draft, error = None, None + for attempt in range(max_attempts): + draft = await effects.call_llm(mod.prompt(session_input, attempt, draft, error)) # async: Python + results = [await run_effect(effects, e) for e in mod.gate(session_input, draft)] # subprocess+sandbox: Python + verdict = mod.interpret(session_input, draft, results) # pure: Rust + if verdict.status == "ok": return mod.publish(session_input, draft, results) # pure: Rust + if verdict.status == "give_up": return GaveUp(verdict.reason) + error = verdict.error # retry + return GaveUp(f"did not converge in {max_attempts} attempts") +``` + +`run_effect` dispatches a `Shell`/`Prover`/`Feedback` effect to the existing +`RealEffects.run_command`/`run_prover`/`run_feedback` — unchanged, still exec-not-shell, +still sandboxed. Progress **events and caching move to Python**: the driver already knows the +phase and each command's result, so it emits the `verdict`/`build`/`fuzz` notices itself +(the wheel no longer needs an `Emit` command or the `Emitter` shim), and the pipeline's +existing result cache subsumes the loop's scratch `CacheGet`/`CachePut`. + +`CrucibleFormalizer.formalize` then becomes: prepare the crate, then `author_gate(...)` — the +control flow is one readable Python function instead of a Rust enum plus a JSON dispatcher. + +### What each side ends up owning + +- **Rust:** prompts, the command line (security), result interpretation, result assembly — + all pure functions, unit-testable in Rust with no Python. +- **Python:** the author-gate loop, retries/budget, all async effects, sandboxing, caching, + events — one orchestration model, shared with the CVL/foundry backends. + +## 6. Trade-offs and risks + +- **Loss of arbitrary per-backend control flow.** The IoC loop is Turing-complete; the + callout model fixes one template (author → gate → interpret → publish, with bounded + revise). All *current* sessions fit it, but a future backend wanting a genuinely different + flow (multi-phase, branch-and-minimize a counterexample, adaptive strategies) would need a + new template callout, not a free-form state machine. **This is the real decision:** is the + author-gate template enough for every backend we foresee? If yes, the loop is + over-general; if we expect exotic flows, the loop earns its keep. +- **Incremental emission is slightly coarser.** Today Crucible emits a live verdict as *each* + feature finishes fuzzing. With `gate` returning all commands and `interpret` classifying + them together, Python would emit verdicts after the batch — unless we keep a tiny + `classify(one_result) -> outcome` callout so the driver can emit per-command as it goes + (a fifth pure function; cheap). +- **Migration touches both languages.** Delete the enums/loop/pyclass, add the callouts, + rewrite `RealEffects` into a small `run_effect` dispatcher, and re-express both Crucible + sessions and the echoprover demo as callouts. Bounded, but not a one-liner. +- **`example-app` cache short-circuit.** echoprover's `CacheGet → maybe skip LLM` becomes a + Python-side "check the pipeline cache before authoring" — arguably clearer, and removes the + redundant second cache. +- **The security invariant must be re-audited** at its new home (`gate`), with a test that + the LLM draft cannot influence `program`/`args`. + +## 7. Recommendation + +The four pure callouts already outnumber the one stateful surface, and every current session +is an author-gate loop wearing a state-machine costume. Moving the loop into the Python core +(where the sibling CVL/foundry backends already orchestrate the identical shape) removes a +whole protocol, both `Emitter`/scratch-cache shims, and the hand-compiled enums — at the cost +of committing to the author-gate template. Recommended **if** we accept that template as the +backend contract. + +Suggested staging (each shippable): + +1. **Add the callouts alongside the loop.** Implement `prompt`/`gate`/`interpret`/`publish` + for Crucible's `BatchSession` and `SetupSession` as pure functions *next to* the existing + `resume` (they can share code). No behaviour change yet. +2. **Add `author_gate` to the core pipeline** and switch `CrucibleFormalizer` to it behind a + flag; verify against the e2e gate (identical verdicts). +3. **Port echoprover**, then **delete** `resume`/`RustSession`/`Command`/`Observation`/ + `drive_session` and trim `Effects` to `call_llm` + `run_effect`. + +## 8. Open questions + +- Is the author-gate template sufficient for the backends we actually plan (Crucible, + soroban, a CVL-in-Rust prover)? Any that need branching multi-phase flow? +- Keep the per-result `classify` callout for live per-unit emission, or accept batch-end + emission? +- Should `gate` return **all** commands up front (enables Python to parallelize them — + ties into the `--binary-in` fuzz-parallelism lever in + [crucible-unit-granularity.md §7](./crucible-unit-granularity.md)) or one at a time? +- Does any backend need Rust to hold state *between* gate commands that isn't recoverable + from `(input, draft, results)`? (None does today.) diff --git a/docs/rust-pure-app.md b/docs/rust-pure-app.md new file mode 100644 index 00000000..265f0b4c --- /dev/null +++ b/docs/rust-pure-app.md @@ -0,0 +1,400 @@ +# Design Doc — Defining an AutoProver application *purely* in Rust + +> Today a Rust *backend* is a passive wheel ([rust-backend-api.md](./rust-backend-api.md)) and +> a generic Python host ([rust-applications.md](./rust-applications.md)) turns any wheel into a +> runnable application — phase enum, argparse, entry point, frontend, store, `main()`. That +> works end-to-end for a *simple* app: `echoprover` ships as `console_main("echoprover")` with +> **zero bespoke Python**. +> +> Crucible does not. It carries a whole `composer/crucible/` package that *forks* the generic +> host in ~10 specific places (a crate-shaped deliverable, a shared setup fixture, dependency +> warming, an `.so` pre-build, a RAG env, sandbox grants, a verdict summary). This doc +> inventories every one of those forks and proposes a Rust-side seam for each, so that Crucible +> becomes what echoprover already is: a wheel + a descriptor, launched by the generic host with +> no application-specific Python. +> +> Companion to [rust-applications.md](./rust-applications.md) (§4.5 already anticipated "Python +> shell, Rust formatter" for the store — this doc discharges that and the rest) and +> [rust-backend-api.md](./rust-backend-api.md) (the callout surface we extend). + +--- + +## 1. Goal and non-goal + +**Goal.** An application is *defined* entirely by its Rust wheel (`rust/<app>-app`) and the +`AppDescriptor` it exports. Standing up a new verifier — even one as involved as Crucible — +requires **no** new Python package. `console-<app>` / `tui-<app>` are two-line shims over the +generic `composer.rustapp.cli`. + +**Non-goal — the ecosystem stays shared Python.** "Pure Rust *app*" does not mean "pure Rust +*everything*". The pipeline's **front half** (system analysis + property extraction) is +parametric over an *ecosystem* ([ecosystem.py](../composer/pipeline/ecosystem.py)), and the +`solana` ecosystem — its `SolanaApplication` model, j2 prompts, `locate_main`, global-extraction +strategy — is **chain-specific, not app-specific**. It is legitimately shared by any Solana +backend (Crucible today, a future Solana app tomorrow) and stays Python. The wheel *selects* an +ecosystem by tag (`descriptor.ecosystem = "solana"`); it does not reimplement it. + +So the line this doc draws is: + +> Everything downstream of "which ecosystem" that is specific to **this verifier** moves into +> the wheel. Everything that is shared **service lifecycle** (Postgres, the TUI event loop, +> `composer.bind`) or shared **chain** logic (the ecosystem) stays Python. + +**Security invariant (unchanged, load-bearing).** The LLM never controls a command line; only +file *contents* may be LLM-derived. Today the *trusted wheel* assembles every `crucible …` +argv and Python authors the `Sandbox` *policy* ([command-sandbox.md](./command-sandbox.md) §2). +Every new seam below preserves this exactly: new toolchain steps are still wheel-authored argv +run under a Python-authored policy. See §7. + +--- + +## 2. The gap inventory — where Crucible forks the generic host + +Every item below is Python that exists *only* because the generic host can't yet express what +Crucible needs. Each maps to a seam in §3–§5. + +| # | Crucible-specific Python | Location | What it does | Seam | +|---|---|---|---|---| +| 1 | `CrucibleArtifactStore`, `CrucibleHarness`, `CrucibleDep` | [store.py](../composer/crucible/store.py), [harness.py](../composer/crucible/harness.py) | Assemble **one Cargo crate** (deps + shared fixture + one feature-gated test section per property) instead of one file per component | §3.1 deliverables callout | +| 2 | Setup-fixture authoring | [backend.py](../composer/crucible/backend.py) `CruciblePreparedSystem.prepare_formalization` | Author + compile-gate a shared `kind="setup"` artifact once, before per-component formalization | §3.2 declared setup step | +| 3 | Context injection | [backend.py](../composer/crucible/backend.py) `CrucibleFormalizer._context` | Thread the fixture + `fuzz_timeout` into each component's `AuthorInput.context` | §3.3 generic context | +| 4 | Crate scaffolding | `_before_formalize` → `store.prepare_component` | Pre-place `Cargo.toml` with cumulative feature declarations before each unit builds | §4 (subsumed by prep + per-run manifest) | +| 5 | Toolchain serialization | `CrucibleFormalizer(command_sem=Semaphore(1))` | Serialize compile/validate (one shared crate / target dir) | §3.4 descriptor flag | +| 6 | Dependency warming | [store.py](../composer/crucible/store.py) `warm_dependencies` + `write_setup_manifest` | Network `cargo fetch` **outside** the sandbox into the private `CARGO_HOME`, so the confined build runs offline | §4 workspace_prep | +| 7 | Program `.so` pre-build | [pipeline.py](../composer/crucible/pipeline.py) `run_crucible_pipeline` → `build_program` | `cargo-build-sbf` / `anchor build` before the pipeline; the harness loads the `.so` | §4 workspace_prep | +| 8 | RAG env | [pipeline.py](../composer/crucible/pipeline.py) `build_crucible_env` | Wire the `crucible_kb` RAG search tools onto the author env | §5.1 descriptor-driven env | +| 9 | Repo resolution + sandbox grants + default provider | [pipeline.py](../composer/crucible/pipeline.py) `resolve_crucible_repo`, `crucible_sandbox` | Resolve `$CRUCIBLE_REPO`; grant it + the `crucible` binary as sandbox `extra_ro`; default to the `launcher` provider | §5.2 grants callout + descriptor flag | +| 10 | CLI entry points + verdict summary | [cli.py](../composer/crucible/cli.py), [results.py](../composer/crucible/results.py) | `console-crucible` / `tui-crucible`; print a per-invariant verdict tally | §5.3 generic summary | + +The current generic path (what echoprover uses) is +[`composer/rustapp/cli.py`](../composer/rustapp/cli.py) `console_main` / `tui_main` → +[`entry.py`](../composer/rustapp/entry.py) `rust_entry_point` → +[`host.py`](../composer/rustapp/host.py) `run_application` → +[`adapter.py`](../composer/rustapp/adapter.py) `RustFormalizer`. Crucible's package is a fork of +exactly this chain. The seams below let Crucible re-join it. + +--- + +## 3. Formalization seams (the loop) + +### 3.1 Deliverable assembly → a `finalize`-shaped callout + +**Today.** The base [`ArtifactStore`](../composer/spec/artifacts.py) writes the *shared* +metadata every backend produces — `properties.json`, `commentary.md`, the property→units map, +`token_usage.json` — and materializes the artifact bytes as `{prefix}_{slug}.{ext}`, one file +per component. `CrucibleArtifactStore` overrides `write_artifact` to instead fold each +component's section into a `CrucibleHarness` and re-render a whole crate (`Cargo.toml` + +`src/main.rs`). `CrucibleHarness`/`CrucibleDep` duplicate, in Python, crate rendering the wheel +*also* does in Rust (`one_file`, and the dep list the harness pins). + +**Proposal.** Split the store's job cleanly: + +- **Metadata stays generic.** `properties.json` / `commentary.md` / the property map / + token usage are not app-specific — keep them in the base store, unchanged, for every app. +- **Source deliverable becomes a callout.** Add a descriptor field + `deliverable_mode: "per_component" | "callout"` (default `per_component` = today's + echoprover behavior). In `callout` mode the base store writes **no** per-component source + file; instead the wheel renders the whole deliverable from the full result set. + +The natural callout is the one that already exists: `finalize`. It already receives the outcome +set and returns `{relpath: contents}` ([lib.rs](../rust/autoprover-sdk/src/lib.rs) `ffi_finalize` +→ the host writes each file, [adapter.py](../composer/rustapp/adapter.py) `RustFormalizer.finalize`). +We enrich its input so the wheel has everything the crate needs: + +```jsonc +// finalize input (per outcome), extended: +{ "name": "...", "delivered": true, "unit_file": "...", "run_link": "...", + "artifact_text": "<the authored test section>", // NEW + "property_units": [["<title>", ["c_slug"]]], // NEW + "setup": "<the shared fixture source>" } // NEW (run-level, see §3.2) +``` + +`crucible_app::finalize` then renders `fuzz/<program>/Cargo.toml` + `src/main.rs` from the +fixture + the per-property sections — the `CrucibleHarness` logic, but in Rust, as the **single +source of truth** for crate layout (it already half-lives there as `one_file`). `CrucibleDep`'s +pinned dependency stack moves into the wheel too; it reads `$CRUCIBLE_REPO` directly (§5.2). + +**Tradeoff.** The crate lands on disk only at finalize, not incrementally per component. That's +acceptable: the crate is only *runnable* once complete, and `validate` already materializes a +transient copy for each fuzz run via `run_confined`'s `files` map. We note this in the doc so a +future "stream partial deliverables" need is a known, deliberate follow-up. + +**Deletes:** `harness.py` entirely; `CrucibleArtifactStore` (the generic `RustArtifactStore` +in `callout` mode suffices — see §4 for why even the manifest pre-placement goes away). + +### 3.2 Shared setup artifact → a declared step + +**Today.** `CruciblePreparedSystem.prepare_formalization` authors a `kind="setup"` artifact +(the fixture) via `author_and_compile`, once, before per-component formalization, and stashes +it on the store. The wheel *already* handles `kind=="setup"` in `units` (→ empty), `author_prompt` +(→ fixture prompt), and `compile` (→ probe dry-run). Only the *orchestration* is Python. + +**Proposal.** Make the setup step declarative on the descriptor: + +```rust +setup: Option<SetupSpec> // { phase_key: "build_harness", label: "Build Harness", + // context_key: "fixture" } +``` + +When present, the generic `RustPreparedSystem.prepare_formalization` (in +[adapter.py](../composer/rustapp/adapter.py)) runs the existing `author_and_compile` for a +`kind="setup"` input under `setup.phase_key`, and stashes the compiled spec on the formalizer. +This lifts `CruciblePreparedSystem` into the host verbatim — no new logic, just a descriptor +gate. Apps with no setup (echoprover) omit the field and skip the step. + +### 3.3 Context injection → generic + +**Today.** `_context` returns `{program, fixture, fuzz_timeout}`; the base returns `{program}`. + +**Proposal.** The host always injects into every component's `AuthorInput.context`: +(a) the setup result under `setup.context_key` (if a setup step ran), and (b) each **declared +CLI arg** value (so `fuzz_timeout` — already a descriptor `ArgSpec` — is present). The wheel +reads them exactly as it does now (`ctx_str(input, "fixture")`, `ctx_u64(input, "fuzz_timeout")`). +`_context` and the `CrucibleFormalizer` override disappear. + +### 3.4 Toolchain serialization → a descriptor flag + +**Today.** `CrucibleFormalizer` passes `command_sem=asyncio.Semaphore(1)` because all +compile/validate runs share one crate / target dir. + +**Proposal.** Descriptor flag `serialize_toolchain: bool` (default `false`). When `true`, the +generic formalizer constructs the `Semaphore(1)` itself and threads it into `_run_blocking` +(the plumbing already exists — `RustFormalizer.__init__(command_sem=...)`). Crucible sets it +`true`; echoprover leaves it `false` (its `validate` is a no-op with no shared state). + +--- + +## 4. Workspace preparation — a pure plan the host executes + +Items **4, 6, 7** (Cargo manifest placement, dependency warming, the `.so` pre-build) are all +the same shape: a **toolchain step that must run before formalization**, part of it needing +**network** (the dependency fetches). Today they're three ad-hoc Python steps +(`write_setup_manifest`, `warm_dependencies`, `build_program`). + +**Design constraint discovered in the code.** The command sandbox *never* gives a confined +process network access — `rust_build_policy` hardcodes `network=False`, and the only network +step, `warm_cargo_cache`, runs **unconfined** ([command-sandbox.md](./command-sandbox.md) §5). +So a "prep sandbox with `network: true`" (an earlier draft of this section) would be a brand-new +security capability — a confined process with a socket — which the codebase deliberately avoids. +The right seam therefore does **not** hand the wheel a confined-with-network policy. + +**Proposal.** One new **pure** callout that returns a *plan*, executed by the **host** with the +existing shared helpers: + +```rust +fn workspace_prep(&self, input: &AuthorInput) -> WorkspacePrep { + // { files: {relpath: contents}, // e.g. the harness Cargo.toml (deps only the wheel knows) + // warm_dirs: [String], // dirs to `cargo fetch` (unconfined, network) + // build_program: Option<String> // build this program to its platform binary } +} +``` + +- The host writes `files` (path-confined via `confined_join`), runs `warm_cargo_cache` on each + `warm_dirs` (**unconfined, network** — a fetch runs no untrusted code), and, if + `build_program` is set, calls the shared `build_program` capability (which itself warms the + *program* crate then builds it **confined + offline**). +- **Posture unchanged and Python-owned end to end**: fetches unconfined, code-executing builds + confined+offline. The wheel touches no command line — it contributes only file *contents* and + declarative intent (which dirs, which program). Strictly within the existing trust model; no + new capability. +- `build_program` is the shared Solana build capability + ([solana/build.py](../composer/spec/solana/build.py)); the generic host invokes it lazily when + the plan requests it (shared ecosystem capability, not app-specific Python). + +**Bonus simplification — the cumulative-feature manifest race disappears (item 4).** The reason +`prepare_component` reserves features *cumulatively* on a shared on-disk `Cargo.toml` is that +concurrent per-component sessions each rewrite it and could drop each other's feature. But with +`serialize_toolchain: true` (§3.4), runs are serialized, and the wheel can materialize +`Cargo.toml` (deps + exactly the one feature this build needs — features are inert `f = []` +entries that don't affect dep resolution) in the **`files` map of each `compile`/`validate` run**. +No shared-manifest mutation across runs ⇒ no race ⇒ no `reserve_features` / `_reserved` / +`prepare_component` machinery at all. The `workspace_prep` plan places only the deps-only manifest that +warming needs. This is the last thing keeping the `CrucibleArtifactStore` alive; with it gone, +§3.1's generic `callout` store fully suffices. + +**Deletes:** `write_setup_manifest`, `warm_dependencies`, the `build_program` pre-step call, and +the whole feature-reservation path in `harness.py`. + +--- + +## 5. Entry-point seams (services & UI) + +These stay Python (service lifecycle / event loop — the "shell" of +[rust-applications.md](./rust-applications.md) §1), but are made **descriptor-driven** so no +Crucible fork is needed. + +### 5.1 RAG env from the descriptor + +**Today.** `build_crucible_env` builds the `crucible_kb` RAG tools and is passed as +`env_builder=` everywhere. The generic `build_neutral_env` builds *no* RAG even though the +descriptor already declares `rag_db_default: "crucible_kb"`. + +**Proposal.** Fold `build_crucible_env`'s logic into the generic env builder, gated on +`descriptor.rag_db_default`: when set, open that RAG DB and add its search tools (falling back +to no-RAG on failure, exactly as `build_crucible_env` does today); when `None`, the neutral env. +The `env_builder=` override parameter can stay for exotic cases but Crucible stops needing it. + +### 5.2 Sandbox grants + default confinement + +**Today.** `crucible_sandbox` resolves `$CRUCIBLE_REPO`, grants it + `which("crucible")` as +`extra_ro`, and defaults the provider to `launcher` (fail-closed). `resolve_crucible_repo` +validates the checkout. + +**Proposal.** +- **Grants → a pure callout** `sandbox_grants(args) -> { extra_ro: [String], extra_env: [String] }`. + The wheel resolves `$CRUCIBLE_REPO` and scans `$PATH` for `crucible` in Rust (it already scans + `$PATH` in `on_path`). The host unions the returned grants into its `SandboxConfig`. +- **Default confinement → a descriptor flag** `confine_by_default: bool` (true for any wheel + with real toolchain callouts). The generic entry builds the `launcher` `SandboxConfig` when + set (still overridable by `COMPOSER_SANDBOX_PROVIDER=none`), replacing the hardcoded default in + `crucible_sandbox`. +- **Repo validation → `validate_preconditions`.** The wheel already validates the workspace + there (`Cargo.toml` present, required binaries on `$PATH`); add the `$CRUCIBLE_REPO` / + `crates/crucible-fuzzer` check. `resolve_crucible_repo` disappears; the `--crucible-repo` flag + becomes a descriptor `ArgSpec` the wheel reads from `args`/env. + +### 5.3 Verdict summary → generic + +**Today.** `results.py` (`summarize_verdicts`, `format_verdict_lines`) turns +`RustFormalResult.verdicts` into a console tally; `crucible/cli.py` prints it. But `verdicts` is +**already a generic field** on the generic result type. + +**Proposal.** Move `results.py` into `composer/rustapp/` and have the generic `console_main` / +`tui_main` print the verdict tally whenever the results carry verdicts (empty ⇒ prints nothing, +exactly as today for echoprover). Parametrize the outcome wording by `descriptor.backend_tag` +(the report's `outcome_label(tag, …)` already takes the tag). One nicety: make the component +noun a descriptor field `component_noun: "instruction"` (default `"component"`) so Crucible's +"Instructions:" line and echoprover's "Components:" line come from the same code. + +--- + +## 6. What Crucible collapses to + +After §3–§5: + +- **Deleted:** `composer/crucible/` in its entirety — `backend.py`, `store.py`, `harness.py`, + `pipeline.py`, `results.py`, `cli.py`. +- **`pyproject.toml`:** + ```toml + console-crucible = "composer.crucible_launch:console_crucible" # 2-line shim, or: + console-crucible = "composer.rustapp.cli:console_main" # via a --module arg + tui-crucible = "composer.rustapp.cli:tui_main" + ``` + (A thin `crucible_launch.py` = `def console_crucible(): return console_main("crucible_app")` + keeps the bare `console-crucible` command with no positional module arg. This is the *only* + Python left, and it's shared-shaped — echoprover has the identical shim.) +- **The wheel (`rust/crucible-app`)** grows: `workspace_prep`, `sandbox_grants`, a richer + `finalize` (crate rendering, absorbing `CrucibleHarness`/`CrucibleDep`), the repo precondition, + and a descriptor carrying `setup`, `deliverable_mode: "callout"`, `serialize_toolchain: true`, + `confine_by_default: true`, `component_noun: "instruction"`. It reads `$CRUCIBLE_REPO` itself. +- **Everything downstream of "ecosystem = solana"** is the shared front half — unchanged. + +The proof obligation: `console-crucible <project> <program> <doc> --fuzz-timeout N` produces a +byte-identical deliverable + report to today, at parity runtime (the e2e Vault gate — 16 GOOD, +~41 min baseline). + +--- + +## 7. Security invariant — preserved, and audited per seam + +> The LLM controls file *contents* only; the trusted wheel controls every argv; Python authors +> every sandbox *policy*. + +| Seam | New capability | Who controls argv | Who authors policy | Net | +|---|---|---|---|---| +| §3.1 finalize deliverable | writes files under project root | — (host writes, path-confined via `confined_join`) | n/a | same as today's store | +| §4 workspace_prep | warm dirs + build a program | — (host runs the shared `warm_cargo_cache` / `build_program`; wheel supplies only file *contents* + which dirs/program) | **Python** `SandboxConfig` | **identical** to today (fetch unconfined, build confined+offline) | +| §5.2 sandbox_grants | adds `extra_ro`/`extra_env` | n/a (data) | Python unions into its policy | same grants, now wheel-declared | + +No seam gives the *LLM* argv control, and no seam lets the *wheel* invent a sandbox policy. §4 is +a *pure declaration* — the host runs the same shared warm/build helpers it does today, so the +network posture (fetch unconfined, code-executing build confined + offline) is byte-for-byte the +current behavior. Nothing here weakens or "tightens" confinement; it only moves *who declares the +plan* into the wheel. + +--- + +## 8. New surface, summarized + +**Descriptor (`AppDescriptor`, mirrored in [descriptor.py](../composer/rustapp/descriptor.py)):** + +```rust +setup: Option<SetupSpec>, // { phase_key, label, context_key } (§3.2) +deliverable_mode: DeliverableMode, // PerComponent | Callout, default PerComponent (§3.1) +serialize_toolchain: bool, // default false (§3.4) +confine_by_default: bool, // default false (§5.2) +component_noun: Option<String>, // default "component" (§5.3) +``` + +**New callouts on the `Backend` trait (both pure):** + +```rust +fn workspace_prep(&self, input) -> WorkspacePrep { default } // { files, warm_dirs, build_program } (§4) +fn sandbox_grants(&self, args: &serde_json::Value) -> SandboxGrants { default } // { extra_ro, extra_env } (§5.2) +// finalize's input gains artifact_text / property_units / setup (§3.1) — signature unchanged. +``` + +All are **defaulted**, so existing wheels (echoprover) keep working untouched — this is a +backward-compatible extension of the passive-backend API, not a new protocol. + +--- + +## 9. Work breakdown + +1. **Descriptor + defaults** — add the five fields to `AppDescriptor` (Rust + pydantic mirror), + all defaulted; `workspace_prep` / `sandbox_grants` pure trait methods with no-op defaults. + *No behavior change; echoprover + Crucible-via-Python still run.* +2. **Generic host honors them** — setup step, context injection, `serialize_toolchain`, + `callout` deliverable via enriched `finalize`, `workspace_prep` execution (write files → warm + → build), RAG-from-descriptor, `confine_by_default` + grants union, generic verdict summary. + Land behind the descriptor gates so echoprover is unaffected. +3. **Port `crucible_app`** — move `CrucibleHarness`/`CrucibleDep` rendering into `finalize`; + implement `workspace_prep` (harness `Cargo.toml` + warm dirs + program build) and + `sandbox_grants`; add the repo precondition; set the descriptor flags; materialize `Cargo.toml` + per confined run. +4. **Delete `composer/crucible/`** and repoint the console scripts. Update the gate tests + (`test_crucible_*`) to drive the wheel through the generic host. +5. **e2e parity** — run the Vault gate; confirm 16 GOOD at ~parity runtime and byte-identical + deliverable + report. (Posture is unchanged, so no new confinement risk to validate.) + +Steps 1–2 are the reusable investment (they benefit *every* future Rust app); 3–5 are the +Crucible port that proves the seam. + +--- + +## 10. Open questions / decisions to confirm + +1. **`finalize` vs. a dedicated `render_deliverables` callout.** Reusing `finalize` keeps the + surface minimal but overloads one method with "side-effect artifacts" and "the primary + deliverable." A separate `render_deliverables(results) -> {relpath: contents}` is clearer at + the cost of one more callout. *Leaning: reuse `finalize`, revisit if a second app wants both.* +2. **Where the Solana build capability lives.** `workspace_prep`'s `build_program` routes to the + shared `composer.spec.solana.build.build_program` — so the generic host gains one lazy import + of an ecosystem-specific helper. Acceptable (it's shared, not app-specific, and gated on the + plan), but the cleaner long-term home is a `build` capability on the `Ecosystem` itself. + *Leaning: lazy import now; promote to the ecosystem seam if a second ecosystem needs it.* +3. **Per-run `Cargo.toml` materialization vs. a stable on-disk crate.** Materializing the + manifest per confined run (§4) is what removes the feature-race, but it means the on-disk + crate between runs is whatever the last run wrote. Since the *authoritative* crate is produced + at `finalize`, this is fine — but if any tooling expects a stable mid-run crate dir, we'd keep + a single `workspace_prep`-placed manifest with all features (reintroducing a mild coupling). +4. **`--module`-style generic command vs. per-app shims.** Whether `console-crucible` is a 2-line + shim (`console_main("crucible_app")`) or the generic `console_main` takes the module as its + first positional. Shims keep the familiar command names; the generic form is one entry for all + wheels. *Leaning: shims, matching echoprover.* + +--- + +## 11. Key files + +- Wheel: [rust/crucible-app/src/lib.rs](../rust/crucible-app/src/lib.rs), + [rust/autoprover-sdk/src/lib.rs](../rust/autoprover-sdk/src/lib.rs) (trait + descriptor + `run_confined`). +- Generic host (the reuse target): [composer/rustapp/](../composer/rustapp/) — + `descriptor.py`, `adapter.py`, `host.py`, `entry.py`, `cli.py`, `frontend.py`, `store.py`. +- To be deleted: [composer/crucible/](../composer/crucible/) — + `backend.py`, `store.py`, `harness.py`, `pipeline.py`, `results.py`, `cli.py`. +- Shared, staying: [composer/pipeline/ecosystem.py](../composer/pipeline/ecosystem.py) (the + `solana` ecosystem), [composer/spec/solana/](../composer/spec/solana/) (the model), + [composer/sandbox/](../composer/sandbox/) (`SandboxConfig` policy authoring). +</content> +</invoke> diff --git a/pyproject.toml b/pyproject.toml index 92a4eb5e..0a819374 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -88,6 +88,23 @@ ragbuild = [ "markdown-it-py>=3.0", "mdit-py-plugins>=0.4", ] +dev = [ + "maturin>=1.14.1", +] +# The Rust/PyO3 application wheels (built via maturin from the `rust/` workspace), +# declared as editable path deps below so `uv sync --group apps` builds AND +# preserves them — otherwise a bare `uv sync` prunes any hand-built wheel as +# extraneous. Kept OUT of [project.dependencies] and out of the default groups so +# the container image (final stage has no Rust toolchain; syncs with UV_NO_DEV=1 +# --group ragbuild) never tries to compile them. `maturin-import-hook` transparently +# recompiles a changed crate on import, so editing Rust needs no manual `maturin` +# step either — activate it once per venv with: +# python -m maturin_import_hook site install +apps = [ + "crucible_app", + "echoprover", + "maturin-import-hook>=0.3.0", +] [tool.uv] conflicts = [ @@ -101,6 +118,9 @@ conflicts = [ [tool.uv.sources] graphcore = { path = "./graphcore", editable = true } +# Rust/PyO3 wheels built from the local workspace (see the `apps` group above). +crucible_app = { path = "rust/crucible-app", editable = true } +echoprover = { path = "rust/example-app", editable = true } torch = [ { index = "pytorch-cpu", extra = "cpu" }, { index = "pytorch-cu128", extra = "cuda" }, @@ -131,6 +151,8 @@ console-autoprove = "composer.cli.console_autoprove:main" tui-autoprove = "composer.cli.tui_autoprove:main" console-foundry = "composer.cli.console_foundry:main" tui-foundry = "composer.cli.tui_foundry:main" +console-crucible = "composer.crucible_launch:console_crucible" +tui-crucible = "composer.crucible_launch:tui_crucible" autoprove-report-render = "composer.spec.source.report.render:main" tui-natspec = "composer.cli.tui_pipeline:main" cache-natspec = "composer.cli.cache_natspec:main" diff --git a/rust/.gitignore b/rust/.gitignore new file mode 100644 index 00000000..ef6eb6fc --- /dev/null +++ b/rust/.gitignore @@ -0,0 +1,6 @@ +/target +**/*.rs.bk +Cargo.lock +# maturin / local build venvs +.venv/ +*.whl diff --git a/rust/Cargo.toml b/rust/Cargo.toml new file mode 100644 index 00000000..d0f65e5e --- /dev/null +++ b/rust/Cargo.toml @@ -0,0 +1,28 @@ +# Cargo workspace for AutoProver's Rust extension framework. +# +# * `autoprover-sdk` — the library new Rust-based applications import. It defines +# the ABI (serde types), the `Application` / `FormalizeSession` traits, the FFI +# helpers, and the `export_app!` macro that emits the PyO3 module. +# * `example-app` — a self-contained demonstration application built into a +# Python wheel via maturin, used by the framework's round-trip test. +# +# A new application is its own crate (cdylib) that depends on `autoprover-sdk` +# and invokes `autoprover_sdk::export_app!`. It lives outside this workspace in +# real use; `example-app` is kept here so the framework has something to test. +[workspace] +resolver = "2" +members = ["autoprover-sdk", "example-app", "crucible-app", "run-confined"] + +[workspace.package] +edition = "2021" +version = "0.1.0" +license = "MIT" + +[workspace.dependencies] +serde = { version = "1", features = ["derive"] } +serde_json = "1" +pyo3 = "0.24" +# Compile-time Jinja-style templates (the `.j2` convention used by composer/templates/*.j2, +# here for the Rust prompt/crate-file templates). Templates are checked at build time and +# embedded in the wheel, so no runtime template loading or packaging. +askama = "0.13" diff --git a/rust/README.md b/rust/README.md new file mode 100644 index 00000000..0c1ea876 --- /dev/null +++ b/rust/README.md @@ -0,0 +1,96 @@ +# AutoProver Rust framework + +Build AutoProver formalization backends / applications in Rust and run them +through the generic Python pipeline via PyO3. Design rationale: +[docs/rust-formalization-backends.md](../docs/rust-formalization-backends.md) and +[docs/rust-applications.md](../docs/rust-applications.md). + +## Layout + +| Crate | Role | +| --- | --- | +| [`autoprover-sdk`](autoprover-sdk) | The library a Rust application imports: the ABI (serde types), the `Application` / `FormalizeSession` traits, the FFI helpers, and the `export_app!` macro. | +| [`example-app`](example-app) | The `echoprover` demo — a complete, self-contained application built into a wheel and exercised by `tests/test_rustapp.py`. | + +The Python side is [`composer/rustapp`](../composer/rustapp): it loads a wheel, +synthesizes the pipeline's phase enum from the descriptor, and drives the Rust +decider through the inversion-of-control loop (Python owns every async effect — +LLM, prover, cache, event streaming — Rust only decides the next one). No +`pyo3-async` bridge is involved. + +## The FFI surface + +A wheel exports exactly (all synchronous, JSON strings across the boundary): + +```text +descriptor() -> str # the AppDescriptor +validate_preconditions(args_json) -> str|None +new_session(input_json) -> RustSession # .resume(observation_json) -> command_json +fetch_verdicts(input_json) -> str +finalize(outcomes_json) -> str|None +``` + +`export_app!` generates all of these. + +## Writing a new application + +1. New crate: `cdylib`, depending on `autoprover-sdk` and `pyo3` + (`features = ["extension-module", "abi3-py312"]`). See + [example-app/Cargo.toml](example-app/Cargo.toml). + +2. Implement `Application` (descriptor + `new_session` + `fetch_verdicts`) and a + `FormalizeSession` — a **pure synchronous decider** whose `resume(Observation)` + returns the next `Command` (`CallLlm` / `RunProver` / `CacheGet` / `Emit` / … + / `Publish` / `GiveUp`). See [example-app/src/lib.rs](example-app/src/lib.rs). + +3. Export the module (ident must match the wheel/module name): + + ```rust + autoprover_sdk::export_app!(my_app, MyApp); + ``` + +4. Add a maturin `pyproject.toml` (`module-name = "my_app"`), then build: + + ```sh + cd my-app && maturin develop # or: maturin build --out dist + ``` + +5. Ship a CLI. The generic entry point + frontend are synthesized from the + descriptor — a runnable app is a two-line `main()`: + + ```python + # my_app_cli.py + from composer.rustapp.cli import tui_main, console_main + def main() -> int: return tui_main("my_app") # Textual TUI + def main_console() -> int: return console_main("my_app") # stdout + ``` + + Register them in `pyproject.toml` under `[project.scripts]`. No bespoke + argparse, entry point, frontend, or `main()` to write — the descriptor drives + all of it (CLI flags, precondition validation, phase labels, event rendering, + artifact layout). + + For programmatic / headless use, the pipeline wrapper is also exposed directly: + + ```python + from composer.rustapp import run_rust_pipeline + result = await run_rust_pipeline("my_app", source_input, ctx, handler_factory, env) + ``` + +## Building & testing the demo + +```sh +cd rust/example-app && maturin develop # builds & installs the `echoprover` wheel +cd ../.. && python -m pytest tests/test_rustapp.py +``` + +## Notes + +* `FormalizeSession` is `Send + Sync` because PyO3 wraps it in a `#[pyclass]`; a + state machine over plain owned data satisfies this without effort. +* Keep effects coarse-grained — one `resume` per turn / tool-call, never per + token. +* A self-contained (Tier-1) backend that does verification inside Rust simply + never emits `RunProver`/`RunFeedback`; a run-service-backed one surfaces those + as effects and the deployment supplies the `prover=` / `feedback=` hooks to + `RustFormalizer` (see `composer/rustapp/adapter.py`). diff --git a/rust/autoprover-sdk/Cargo.toml b/rust/autoprover-sdk/Cargo.toml new file mode 100644 index 00000000..9153028b --- /dev/null +++ b/rust/autoprover-sdk/Cargo.toml @@ -0,0 +1,18 @@ +[package] +name = "autoprover-sdk" +edition.workspace = true +version.workspace = true +license.workspace = true +description = "SDK for building AutoProver formalization backends / applications in Rust, callable from the Python pipeline via PyO3." + +[lib] +name = "autoprover_sdk" + +[dependencies] +serde = { workspace = true } +serde_json = { workspace = true } +# Re-exported (see `pub use pyo3`) so the `export_app!` macro can reference +# `$crate::pyo3::…` and the app crate needs no direct dependency edge for the +# macro body. The app crate still lists pyo3 to enable `extension-module` / +# `abi3-py312`; cargo feature-unifies the two. +pyo3 = { workspace = true } diff --git a/rust/autoprover-sdk/src/lib.rs b/rust/autoprover-sdk/src/lib.rs new file mode 100644 index 00000000..f21dbdd1 --- /dev/null +++ b/rust/autoprover-sdk/src/lib.rs @@ -0,0 +1,879 @@ +//! # autoprover-sdk +//! +//! The library a Rust-based AutoProver application imports. It defines the seam +//! between a Rust backend and the generic Python pipeline +//! (`composer/pipeline/core.py`), realized over a **synchronous, JSON** FFI +//! boundary — the service-shaped design in `docs/rust-backend-api.md`. +//! +//! The backend is a **passive service**, not a driver: the Python pipeline owns the +//! author→compile→judge→validate loop and every LLM turn, and calls the backend's +//! callouts. Most are pure ([`Backend::descriptor`], [`Backend::units`], +//! [`Backend::author_prompt`], [`Backend::judge_prompt`], [`Backend::finalize`]). The +//! two gating callouts ([`Backend::compile`], [`Backend::validate`]) run the toolchain +//! directly — each spawns the `run-confined` launcher via [`run_confined`] — and BLOCK; +//! the host calls them off the event loop (`asyncio.to_thread`) while the wheel releases +//! the GIL. There is no `async`/`pyo3-async` bridge and no `Command`/`Observation` resume +//! protocol on the Rust side. +//! +//! An application implements [`Backend`] and calls [`export_app!`] to emit the PyO3 +//! module the Python host loads. + +use serde::{Deserialize, Serialize}; +use std::collections::BTreeMap; + +/// Re-exported so [`export_app!`] can reference `$crate::pyo3::…`; an app crate +/// still depends on pyo3 directly to enable `extension-module` / `abi3-py312`. +pub use pyo3; + +// =========================================================================== +// Descriptor — the declarative spine the Python host consumes to synthesize the +// phase enum, argparse, frontend and artifact store (see rust-applications.md). +// =========================================================================== + +/// Which of the four driver-tagged core phases a declared phase fills. A phase +/// with no core slot is a UI-only phase (cf. autoprove's harness/autosetup). +#[derive(Debug, Clone, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum CoreSlot { + Analysis, + Extraction, + Formalization, + Report, +} + +/// One task-grouping phase. `key` becomes the synthesized `enum.Enum` member +/// name; `label`/`order` drive UI grouping. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct PhaseSpec { + pub key: String, + pub label: String, + pub order: u32, + #[serde(skip_serializing_if = "Option::is_none")] + pub core_slot: Option<CoreSlot>, +} + +/// Default value for a declared CLI argument. +#[derive(Debug, Clone, Serialize, Deserialize)] +#[serde(tag = "kind", rename_all = "snake_case")] +pub enum ArgDefault { + Str { value: Option<String> }, + Int { value: Option<i64> }, + Bool { value: bool }, +} + +/// A CLI flag the generic entry point adds beyond the three positional inputs +/// (`project_root`, `main_contract`, `system_doc`). +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct ArgSpec { + pub flag: String, + pub help: String, + pub default: ArgDefault, + #[serde(default)] + pub required: bool, +} + +/// A domain event kind the frontend should render (see `Command::Emit`). +/// +/// A `notice` kind is surfaced as a persistent, always-visible callout (plus a toast) +/// rather than a line in the collapsible per-task events log — for one-shot important +/// results such as a per-unit verdict. Ordinary kinds stream into the log. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct EventKind { + pub kind: String, + pub label: String, + #[serde(default)] + pub notice: bool, +} + +impl EventKind { + /// A streaming event kind — rendered as a line in the collapsible events log. + pub fn log(kind: impl Into<String>, label: impl Into<String>) -> Self { + Self { kind: kind.into(), label: label.into(), notice: false } + } + + /// A notice event kind — surfaced as a persistent callout + toast. + pub fn notice(kind: impl Into<String>, label: impl Into<String>) -> Self { + Self { kind: kind.into(), label: label.into(), notice: true } + } +} + +/// On-disk deliverable layout. All paths are project-root-relative. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct ArtifactLayout { + pub deliverable_dir: String, + pub internal_dir: String, + pub report_dir: String, + /// Where the verification artifacts themselves are written. + pub artifact_dir: String, + /// Filename prefix for a per-component artifact (e.g. `autospec` → `autospec_<slug>.spec`). + pub artifact_prefix: String, + /// Artifact file extension, no dot (e.g. `spec`, `t.sol`). + pub artifact_extension: String, + /// The store's term for the property→units map file suffix (`property_rules`, `property_tests`). + pub property_suffix: String, + /// Under `DeliverableMode::Callout`, the project-relative path of the primary deliverable + /// file, `{program}`-templated (Crucible: `fuzz/{program}/src/main.rs`). Used only as each + /// component's report link — the actual files come from `finalize`. Ignored `PerComponent`. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub deliverable_primary: Option<String>, +} + +/// A shared "setup" artifact authored once, before per-component formalization (Crucible's +/// shared fixture). When a descriptor carries one, the host runs the author→compile loop for a +/// `kind="setup"` [`AuthorInput`] under `phase_key`, then threads the compiled spec into every +/// component's `AuthorInput.context` under `context_key`. Absent → no setup step. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct SetupSpec { + /// The descriptor phase key the setup task is grouped under (a UI-only phase). + pub phase_key: String, + /// The task label shown in the frontend. + pub label: String, + /// The `AuthorInput.context` key the compiled setup spec is injected under for components. + pub context_key: String, +} + +/// How the source deliverable is written to disk. `PerComponent` (the default): the generic +/// store writes one `{prefix}_{slug}.{ext}` file per component from its `artifact_text`. +/// `Callout`: the store writes no per-component source; the wheel's `finalize` renders the whole +/// deliverable (e.g. Crucible's one shared crate assembled from all sections + the fixture). +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum DeliverableMode { + #[default] + PerComponent, + Callout, +} + +/// The complete declaration the Python host reads once at load time. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct AppDescriptor { + pub name: String, + pub header_text: String, + /// The ecosystem (chain) tag: "evm" | "solana" | "soroban". Selects the shared front + /// half's system model + prompts; the Python host resolves it against its ecosystem + /// registry. Defaults to "evm" so a descriptor built before this field existed still loads. + #[serde(default = "default_ecosystem")] + pub ecosystem: String, + /// The report's backend tag (`AutoProverReport.backend`). + pub backend_tag: String, + /// Prose injected into the property-extraction prompt (verification-surface guidance). + pub backend_guidance: String, + /// The system-analysis cache key (`SystemAnalysisSpec.analysis_key`). + pub analysis_key: String, + pub phases: Vec<PhaseSpec>, + #[serde(default)] + pub args: Vec<ArgSpec>, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub rag_db_default: Option<String>, + #[serde(default)] + pub event_kinds: Vec<EventKind>, + pub artifact_layout: ArtifactLayout, + /// Optional shared-setup step run before per-component formalization (see [`SetupSpec`]). + #[serde(default, skip_serializing_if = "Option::is_none")] + pub setup: Option<SetupSpec>, + /// How the source deliverable is written (see [`DeliverableMode`]). + #[serde(default)] + pub deliverable_mode: DeliverableMode, + /// Serialize the blocking toolchain callouts (`prepare_workspace`/`compile`/`validate`) on + /// one semaphore — set when the app shares a single build dir / target across units. + #[serde(default)] + pub serialize_toolchain: bool, + /// Default to the fail-closed `launcher` sandbox provider (still overridable by + /// `COMPOSER_SANDBOX_PROVIDER`). Set by any wheel that runs untrusted native toolchains. + #[serde(default)] + pub confine_by_default: bool, + /// Human noun for one formalized unit in the console/TUI summary ("instruction" for + /// Crucible). Defaults to "component". + #[serde(default, skip_serializing_if = "Option::is_none")] + pub component_noun: Option<String>, +} + +fn default_ecosystem() -> String { + "evm".to_string() +} + +// =========================================================================== +// The service API — the data crossing the FFI. The backend is PASSIVE: the Python +// pipeline drives the author→compile→judge→validate loop and calls these callouts; +// nothing here holds state across calls (see docs/rust-backend-api.md). +// =========================================================================== + +/// One property to formalize (mirrors `composer.spec.types.PropertyFormulation`), plus a +/// host-assigned unique `slug` used to name its unit/artifact (Crucible: `c_<slug>`). +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct Property { + pub title: String, + /// One of "attack_vector" | "safety_property" | "invariant". + pub sort: String, + pub description: String, + #[serde(default)] + pub slug: String, +} + +/// The input to the authoring/gating callouts for one artifact. `kind` selects what is being +/// authored ("setup" fixture vs "component" tests); `context` carries backend dependencies +/// (e.g. the shared fixture source a component builds on). `component`/`context` are opaque +/// JSON the backend interprets. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct AuthorInput { + pub kind: String, + pub program: String, + #[serde(default)] + pub component: serde_json::Value, + #[serde(default)] + pub props: Vec<Property>, + #[serde(default)] + pub context: serde_json::Value, +} + +/// An authoring instruction (+ optional backend-defined system prompt) for one LLM turn. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct Prompt { + #[serde(default, skip_serializing_if = "Option::is_none")] + pub system: Option<String>, + pub instruction: String, +} + +/// Whether a draft was rejected by the toolchain or by the optional LLM judge — so a +/// re-author can frame the retry correctly (a judge rejection is NOT a build failure: the +/// draft compiled). Defaults to `Compile` for backends/hosts that predate the field. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Default, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum FailureKind { + #[default] + Compile, + Judge, +} + +/// Why a draft was rejected — the failing `draft` plus the compiler errors / judge feedback +/// — fed into the next `author_prompt` as revise context. `draft` is carried because each +/// authoring turn is fresh (no LLM-side memory of the prior attempt). `kind` says which gate +/// rejected it so the revise prompt can distinguish compiler errors from review feedback. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct Failure { + #[serde(default)] + pub draft: String, + pub errors: String, + #[serde(default)] + pub kind: FailureKind, +} + +/// The outcome of `compile`. +#[derive(Debug, Clone, Serialize, Deserialize)] +#[serde(tag = "status", rename_all = "snake_case")] +pub enum CompileResult { + Ok, + Failed { errors: String }, +} + +/// One report row: a property title and its backend-specific unit name (the rule the report keys +/// by). `target` is the *validation target the host runs* — several report units may share one +/// target (e.g. Crucible puts every invariant in one `c_invariants` target), so the host runs the +/// target once and the backend attributes the outcome back to each unit. `None` ⇒ the unit is its +/// own target (one run per unit, the default). +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct Unit { + pub property: String, + pub unit: String, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub target: Option<String>, +} + +impl Unit { + /// The validation target this unit is checked by — its own `unit` name unless it shares a + /// target with others. + pub fn target_or_unit(&self) -> &str { + self.target.as_deref().unwrap_or(&self.unit) + } +} + +/// A pure plan for preparing the workspace before formalization (Crucible: place the harness +/// `Cargo.toml`, warm its deps, build the program `.so`). The wheel *declares* the plan; the +/// **host executes it with the shared toolchain helpers**, so the standard network posture holds +/// without the wheel touching a command line: dependency fetches run *unconfined* (network, no +/// untrusted code), and the code-executing build runs *confined + offline* +/// (`docs/command-sandbox.md` §5). This keeps warming out of confinement — the codebase never +/// gives a confined process network — while still letting a pure-Rust app own its layout. +#[derive(Debug, Clone, Default, Serialize, Deserialize)] +pub struct WorkspacePrep { + /// Files to write under the workdir (path-confined) before warming — e.g. the harness + /// `Cargo.toml` (whose deps only the wheel knows). Contents only; no command line. + #[serde(default)] + pub files: BTreeMap<String, String>, + /// Project-relative manifest dirs to `cargo fetch` (unconfined, network) so a later + /// confined + offline build finds every dep warm in the private `CARGO_HOME`. + #[serde(default)] + pub warm_dirs: Vec<String>, + /// If set, build this program to its platform binary before formalization, via the host's + /// shared build capability (Solana: `cargo-build-sbf` → `target/deploy/<program>.so`). + #[serde(default, skip_serializing_if = "Option::is_none")] + pub build_program: Option<String>, +} + +/// Extra sandbox grants a wheel needs unioned into the host-authored policy (Crucible: the +/// crucible checkout + the `crucible` binary dir as read-only). Pure data — the wheel declares +/// grants, Python decides the policy; the wheel never invents confinement. +#[derive(Debug, Clone, Default, Serialize, Deserialize)] +pub struct SandboxGrants { + /// Extra read-only paths. + #[serde(default)] + pub extra_ro: Vec<String>, + /// Extra `NAME=VALUE` env entries to pass through confinement. + #[serde(default)] + pub extra_env: Vec<String>, +} + +/// The result of `validate` — the fused build+check for one validation **target**. Either the +/// build failed (so the whole spec must be re-authored — the build is shared), or it built and +/// produced a `Verdict` **per report unit the target covers** (`(unit, verdict)`). A target may +/// cover several units (e.g. Crucible runs every invariant in one target), and the backend — which +/// owns its own result/failure format — attributes the run to those units; the host records the +/// verdicts verbatim (it does no verdict logic). Fusing the build gate into `validate` (rather than +/// a separate `compile` dry-run) is the component path's efficiency win (docs/rust-backend-api.md). +#[derive(Debug, Clone, Serialize, Deserialize)] +#[serde(tag = "kind", rename_all = "snake_case")] +pub enum ValidateOutcome { + BuildFailed { errors: String }, + Verdicts { verdicts: Vec<(String, Verdict)> }, +} + +/// A per-unit outcome (mirrors `composer…report.collect.Verdict`). `outcome` is one of +/// GOOD | BAD | ERROR | TIMEOUT | UNKNOWN. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct Verdict { + pub outcome: String, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub line: Option<u32>, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub duration_seconds: Option<f64>, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub unit_file: Option<String>, + /// Human-readable explanation of a non-GOOD outcome — the failure detail (a counterexample / + /// assertion message) for a `BAD`, or the error text for an `ERROR`. Surfaced live and persisted + /// to the report so a verdict is self-explaining (otherwise a bare `BAD` gives no clue why). + #[serde(default, skip_serializing_if = "Option::is_none")] + pub detail: Option<String>, +} + +impl Verdict { + pub fn with_outcome(outcome: impl Into<String>) -> Self { + Verdict { + outcome: outcome.into(), line: None, duration_seconds: None, unit_file: None, + detail: None, + } + } +} + +// =========================================================================== +// Sandbox — the confinement policy (Python-authored) + the shared launcher helper. +// =========================================================================== + +/// Resource caps (rlimits); `None` = unset. +#[derive(Debug, Clone, Default, Serialize, Deserialize)] +pub struct Rlimits { + #[serde(default)] + pub mem_bytes: Option<u64>, + #[serde(default)] + pub cpu_seconds: Option<u64>, + #[serde(default)] + pub nproc: Option<u64>, + #[serde(default)] + pub fsize_bytes: Option<u64>, +} + +/// The confinement policy for a command, authored by Python (`SandboxConfig`/`SandboxPolicy`) +/// and passed to `compile`/`validate`. `run_confined = None` runs the command directly (the +/// trusted / `none` path). The backend never invents policy — it only assembles this into a +/// `run-confined` argv (see [`run_confined`]); the mapping mirrors +/// `composer/sandbox/launcher.py::LauncherProvider.wrap`. +#[derive(Debug, Clone, Default, Serialize, Deserialize)] +pub struct Sandbox { + #[serde(default)] + pub run_confined: Option<String>, + #[serde(default)] + pub rw: Vec<String>, + #[serde(default)] + pub ro: Vec<String>, + #[serde(default)] + pub allow_env: Vec<String>, // "NAME=VALUE" + #[serde(default)] + pub network: bool, + #[serde(default)] + pub rlimits: Rlimits, + #[serde(default = "default_timeout")] + pub timeout_s: u64, +} + +fn default_timeout() -> u64 { + 600 +} + +/// The captured result of a (confined) command. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct CommandOutput { + pub exit_code: i32, + pub stdout: String, + pub stderr: String, +} + +/// Exit code synthesized when the program isn't found (mirrors shells' 127). +const NOT_FOUND_EXIT: i32 = 127; + +/// Reject absolute paths / `..` traversal (mirrors `composer.sandbox.command._confined_target`). +fn confined_join(workdir: &std::path::Path, rel: &str) -> Result<std::path::PathBuf, String> { + use std::path::{Component, Path}; + let p = Path::new(rel); + if p.is_absolute() || p.components().any(|c| matches!(c, Component::ParentDir)) { + return Err(format!("unsafe file path {rel:?}: absolute or traverses outside the workdir")); + } + Ok(workdir.join(p)) +} + +/// Materialize `files` into `workdir` (path-confined), then run `program args` there confined +/// by `run-confined` per `sandbox` (or directly, when `sandbox.run_confined` is `None`). Blocks +/// on the child; **call from within `Python::allow_threads`**. Enforces `sandbox.timeout_s`. +/// +/// The **command line (`program`/`args`) is authored by the trusted backend**; only file +/// *contents* may derive from the LLM (`docs/command-sandbox.md` §2). `run-confined` confines +/// *itself* (Landlock+seccomp+rlimits+env scrub) and `execve`s the tool. +pub fn run_confined( + sandbox: &Sandbox, + program: &str, + args: &[String], + files: &BTreeMap<String, String>, + workdir: &std::path::Path, +) -> Result<CommandOutput, String> { + use std::io::Read; + use std::process::{Command, Stdio}; + use std::time::{Duration, Instant}; + + // 1. Materialize untrusted files (contents may be LLM-derived; the command line is not). + std::fs::create_dir_all(workdir).map_err(|e| e.to_string())?; + for (rel, contents) in files { + let target = confined_join(workdir, rel)?; + if let Some(parent) = target.parent() { + std::fs::create_dir_all(parent).map_err(|e| e.to_string())?; + } + std::fs::write(&target, contents).map_err(|e| e.to_string())?; + } + + // 2. Build argv: `run-confined <policy flags> -- program args`, or `program args` direct. + let mut cmd = match &sandbox.run_confined { + Some(bin) => { + let mut c = Command::new(bin); + for p in &sandbox.ro { + c.arg("--ro").arg(p); + } + for p in &sandbox.rw { + c.arg("--rw").arg(p); + } + for e in &sandbox.allow_env { + c.arg("--allow-env").arg(e); + } + if sandbox.network { + c.arg("--allow-network"); + } + if let Some(v) = sandbox.rlimits.mem_bytes { + c.arg("--rlimit-as").arg(v.to_string()); + } + if let Some(v) = sandbox.rlimits.cpu_seconds { + c.arg("--rlimit-cpu").arg(v.to_string()); + } + if let Some(v) = sandbox.rlimits.nproc { + c.arg("--rlimit-nproc").arg(v.to_string()); + } + if let Some(v) = sandbox.rlimits.fsize_bytes { + c.arg("--rlimit-fsize").arg(v.to_string()); + } + c.arg("--").arg(program).args(args); + c + } + None => { + let mut c = Command::new(program); + c.args(args); + c + } + }; + cmd.current_dir(workdir) + .stdin(Stdio::null()) + .stdout(Stdio::piped()) + .stderr(Stdio::piped()); + + // 3. Spawn + capture with a timeout. Reader threads avoid a pipe-buffer deadlock. + let mut child = match cmd.spawn() { + Ok(c) => c, + Err(e) if e.kind() == std::io::ErrorKind::NotFound => { + return Ok(CommandOutput { + exit_code: NOT_FOUND_EXIT, + stdout: String::new(), + stderr: format!("{}: not found", sandbox.run_confined.as_deref().unwrap_or(program)), + }) + } + Err(e) => return Err(e.to_string()), + }; + let mut out = child.stdout.take().expect("piped stdout"); + let mut err = child.stderr.take().expect("piped stderr"); + let t_out = std::thread::spawn(move || { + let mut s = Vec::new(); + let _ = out.read_to_end(&mut s); + s + }); + let t_err = std::thread::spawn(move || { + let mut s = Vec::new(); + let _ = err.read_to_end(&mut s); + s + }); + + let deadline = Instant::now() + Duration::from_secs(sandbox.timeout_s.max(1)); + let mut timed_out = false; + let status = loop { + match child.try_wait().map_err(|e| e.to_string())? { + Some(st) => break Some(st), + None => { + if Instant::now() >= deadline { + let _ = child.kill(); + let _ = child.wait(); + timed_out = true; + break None; + } + std::thread::sleep(Duration::from_millis(50)); + } + } + }; + let stdout = String::from_utf8_lossy(&t_out.join().unwrap_or_default()).into_owned(); + let stderr = String::from_utf8_lossy(&t_err.join().unwrap_or_default()).into_owned(); + if timed_out { + return Ok(CommandOutput { + exit_code: -1, + stdout, + stderr: format!("{stderr}\ncommand timed out after {}s", sandbox.timeout_s), + }); + } + Ok(CommandOutput { + exit_code: status.and_then(|s| s.code()).unwrap_or(-1), + stdout, + stderr, + }) +} + +// =========================================================================== +// The trait an application implements. +// =========================================================================== + +/// A Rust AutoProver backend — a **passive service** the Python pipeline drives. One instance +/// per wheel; construct it in [`export_app!`]. Metadata/authoring callouts are pure; `compile` +/// and `validate` run the toolchain (via [`run_confined`]) and BLOCK — the host calls them off +/// the event loop (`asyncio.to_thread`) while the wheel releases the GIL. +pub trait Backend: Send + Sync + 'static { + /// The declaration the Python host reads at load time. + fn descriptor(&self) -> AppDescriptor; + + /// Validate application-specific preconditions before any service opens. `Err(msg)` aborts. + fn validate_preconditions(&self, _args: &serde_json::Value) -> Result<(), String> { + Ok(()) + } + + /// The units this input formalizes — one per property — each a property title and its + /// unit name (Crucible: `c_<slug>`). Pure and pre-authoring: the prompt requires exactly + /// these fn names, the host validates each, and it is the report's property→unit map. + fn units(&self, input: &AuthorInput) -> Vec<Unit>; + + /// The instruction (+ optional system prompt) to author `input.kind`'s spec, covering all + /// its units. `failure = Some(..)` on a re-author after a compile failure / judge rejection. + fn author_prompt(&self, input: &AuthorInput, failure: Option<&Failure>) -> Prompt; + + /// Optional LLM review of a compiled spec, before validation. `None` (the default) skips + /// judging — the compiler + checker are the judges. + fn judge_prompt(&self, _input: &AuthorInput, _spec: &str) -> Option<Prompt> { + None + } + + /// Compile/typecheck the whole spec once (all units share one build). BLOCKING. + fn compile( + &self, + input: &AuthorInput, + spec: &str, + workdir: &std::path::Path, + sandbox: &Sandbox, + ) -> CompileResult; + + /// Build + check ONE unit against the spec (the fused build gate — no separate compile for + /// components). Returns [`ValidateOutcome::BuildFailed`] to trigger a re-author of the whole + /// spec (the build is shared across units), or a per-unit [`Verdict`]. Per-unit so the host + /// owns enumeration/scheduling. BLOCKING. + fn validate( + &self, + input: &AuthorInput, + spec: &str, + unit: &str, + workdir: &std::path::Path, + sandbox: &Sandbox, + ) -> ValidateOutcome; + + /// Extra sandbox grants to union into the host's policy (see [`SandboxGrants`]). Pure; called + /// once before any confined step. Default: no extra grants. + fn sandbox_grants(&self, _args: &serde_json::Value) -> SandboxGrants { + SandboxGrants::default() + } + + /// Declare the pre-formalization workspace prep (see [`WorkspacePrep`]). Pure — the host + /// executes the returned plan with the shared warm/build helpers, so the network posture + /// stays Python-owned. Default: nothing to prepare. + fn workspace_prep(&self, _input: &AuthorInput) -> WorkspacePrep { + WorkspacePrep::default() + } + + /// Optional run-level artifacts from the full outcome set, as `{relpath: contents}`. + /// + /// Under [`DeliverableMode::Callout`] this renders the whole source deliverable (Crucible's + /// one crate); the host enriches the outcome set with each component's `artifact_text` / + /// `property_units` and the `setup` result so the wheel has everything the deliverable needs. + fn finalize(&self, _outcomes: &serde_json::Value) -> BTreeMap<String, String> { + BTreeMap::new() + } +} + +// =========================================================================== +// FFI helpers — the sync, JSON-string boundary. `export_app!` wraps these in +// #[pyfunction]s (compile/validate release the GIL); also unit-testable without Python. +// =========================================================================== + +fn parse<T: serde::de::DeserializeOwned>(json: &str, what: &str) -> Result<T, String> { + serde_json::from_str(json).map_err(|e| format!("invalid {what} JSON: {e}")) +} + +/// `descriptor() -> str` (JSON). +pub fn ffi_descriptor(b: &dyn Backend) -> String { + serde_json::to_string(&b.descriptor()) + .unwrap_or_else(|e| format!("{{\"error\":\"descriptor serialize: {e}\"}}")) +} + +/// `validate_preconditions(args_json) -> str | None` (None = ok). +pub fn ffi_validate_preconditions(b: &dyn Backend, args_json: &str) -> Option<String> { + let args: serde_json::Value = serde_json::from_str(args_json).unwrap_or(serde_json::Value::Null); + b.validate_preconditions(&args).err() +} + +/// `units(input_json) -> str` (JSON `[Unit]`). +pub fn ffi_units(b: &dyn Backend, input_json: &str) -> String { + match parse::<AuthorInput>(input_json, "AuthorInput") { + Ok(input) => serde_json::to_string(&b.units(&input)).unwrap_or_else(|_| "[]".into()), + Err(_) => "[]".into(), + } +} + +/// `author_prompt(input_json, failure_json | None) -> str` (JSON `Prompt`). +pub fn ffi_author_prompt(b: &dyn Backend, input_json: &str, failure_json: Option<&str>) -> String { + let input: AuthorInput = match parse(input_json, "AuthorInput") { + Ok(v) => v, + Err(e) => { + return serde_json::to_string(&Prompt { system: None, instruction: format!("ERROR: {e}") }) + .unwrap_or_default() + } + }; + let failure: Option<Failure> = failure_json.and_then(|s| serde_json::from_str(s).ok()); + let prompt = b.author_prompt(&input, failure.as_ref()); + serde_json::to_string(&prompt).unwrap_or_default() +} + +/// `judge_prompt(input_json, spec) -> str | None` (None = skip judging). +pub fn ffi_judge_prompt(b: &dyn Backend, input_json: &str, spec: &str) -> Option<String> { + let input: AuthorInput = parse(input_json, "AuthorInput").ok()?; + b.judge_prompt(&input, spec) + .map(|p| serde_json::to_string(&p).unwrap_or_default()) +} + +/// `compile(input_json, spec, workdir, sandbox_json) -> str` (JSON `CompileResult`). BLOCKING. +pub fn ffi_compile( + b: &dyn Backend, + input_json: &str, + spec: &str, + workdir: &str, + sandbox_json: &str, +) -> String { + let input: AuthorInput = match parse(input_json, "AuthorInput") { + Ok(v) => v, + Err(e) => return serde_json::to_string(&CompileResult::Failed { errors: e }).unwrap_or_default(), + }; + let sandbox: Sandbox = parse(sandbox_json, "Sandbox").unwrap_or_default(); + let r = b.compile(&input, spec, std::path::Path::new(workdir), &sandbox); + serde_json::to_string(&r).unwrap_or_else(|e| { + serde_json::to_string(&CompileResult::Failed { errors: e.to_string() }).unwrap_or_default() + }) +} + +/// `validate(input_json, spec, unit, workdir, sandbox_json) -> str` (JSON `ValidateOutcome`). BLOCKING. +pub fn ffi_validate( + b: &dyn Backend, + input_json: &str, + spec: &str, + unit: &str, + workdir: &str, + sandbox_json: &str, +) -> String { + let sandbox: Sandbox = parse(sandbox_json, "Sandbox").unwrap_or_default(); + let outcome = match parse::<AuthorInput>(input_json, "AuthorInput") { + Ok(input) => b.validate(&input, spec, unit, std::path::Path::new(workdir), &sandbox), + Err(e) => { + let mut v = Verdict::with_outcome("ERROR"); + v.detail = Some(e); + ValidateOutcome::Verdicts { verdicts: vec![(unit.to_string(), v)] } + } + }; + serde_json::to_string(&outcome).unwrap_or_default() +} + +/// `sandbox_grants(args_json) -> str` (JSON `SandboxGrants`). +pub fn ffi_sandbox_grants(b: &dyn Backend, args_json: &str) -> String { + let args: serde_json::Value = serde_json::from_str(args_json).unwrap_or(serde_json::Value::Null); + serde_json::to_string(&b.sandbox_grants(&args)).unwrap_or_else(|_| "{}".into()) +} + +/// `workspace_prep(input_json) -> str` (JSON `WorkspacePrep`). Pure. +pub fn ffi_workspace_prep(b: &dyn Backend, input_json: &str) -> String { + match parse::<AuthorInput>(input_json, "AuthorInput") { + Ok(input) => serde_json::to_string(&b.workspace_prep(&input)).unwrap_or_else(|_| "{}".into()), + Err(_) => "{}".into(), + } +} + +/// `finalize(outcomes_json) -> str | None` (JSON `{relpath: contents}`, or None). +pub fn ffi_finalize(b: &dyn Backend, outcomes_json: &str) -> Option<String> { + let outcomes: serde_json::Value = serde_json::from_str(outcomes_json).ok()?; + let files = b.finalize(&outcomes); + if files.is_empty() { + None + } else { + serde_json::to_string(&files).ok() + } +} + +// =========================================================================== +// The export macro. +// =========================================================================== + +/// Emit the PyO3 module the Python host loads. Invoke it once in an application crate +/// (a `cdylib` depending on `autoprover-sdk` and `pyo3`): +/// +/// ```ignore +/// autoprover_sdk::export_app!(my_app, MyApp::new()); +/// ``` +/// +/// `module_ident` MUST match the wheel's module name. The expansion defines the pure callouts +/// (`descriptor`/`validate_preconditions`/`units`/`author_prompt`/`judge_prompt`/`finalize`) and +/// the two BLOCKING ones (`compile`/`validate`, which release the GIL while `run-confined` runs), +/// all delegating to the `ffi_*` helpers. +#[macro_export] +macro_rules! export_app { + ($module:ident, $ctor:expr) => { + fn __autoprover_app() -> &'static dyn $crate::Backend { + static APP: ::std::sync::OnceLock<::std::boxed::Box<dyn $crate::Backend>> = + ::std::sync::OnceLock::new(); + &**APP.get_or_init(|| ::std::boxed::Box::new($ctor)) + } + + #[$crate::pyo3::pyfunction] + fn descriptor() -> ::std::string::String { + $crate::ffi_descriptor(__autoprover_app()) + } + + #[$crate::pyo3::pyfunction] + fn validate_preconditions( + args_json: ::std::string::String, + ) -> ::std::option::Option<::std::string::String> { + $crate::ffi_validate_preconditions(__autoprover_app(), &args_json) + } + + #[$crate::pyo3::pyfunction] + fn units(input_json: ::std::string::String) -> ::std::string::String { + $crate::ffi_units(__autoprover_app(), &input_json) + } + + #[$crate::pyo3::pyfunction] + #[pyo3(signature = (input_json, failure_json=None))] + fn author_prompt( + input_json: ::std::string::String, + failure_json: ::std::option::Option<::std::string::String>, + ) -> ::std::string::String { + $crate::ffi_author_prompt(__autoprover_app(), &input_json, failure_json.as_deref()) + } + + #[$crate::pyo3::pyfunction] + fn judge_prompt( + input_json: ::std::string::String, + spec: ::std::string::String, + ) -> ::std::option::Option<::std::string::String> { + $crate::ffi_judge_prompt(__autoprover_app(), &input_json, &spec) + } + + #[$crate::pyo3::pyfunction] + fn compile( + py: $crate::pyo3::Python<'_>, + input_json: ::std::string::String, + spec: ::std::string::String, + workdir: ::std::string::String, + sandbox_json: ::std::string::String, + ) -> ::std::string::String { + // Release the GIL for the (minutes-long) build — no async runtime needed. + py.allow_threads(move || { + $crate::ffi_compile(__autoprover_app(), &input_json, &spec, &workdir, &sandbox_json) + }) + } + + #[$crate::pyo3::pyfunction] + fn validate( + py: $crate::pyo3::Python<'_>, + input_json: ::std::string::String, + spec: ::std::string::String, + unit: ::std::string::String, + workdir: ::std::string::String, + sandbox_json: ::std::string::String, + ) -> ::std::string::String { + py.allow_threads(move || { + $crate::ffi_validate( + __autoprover_app(), + &input_json, + &spec, + &unit, + &workdir, + &sandbox_json, + ) + }) + } + + #[$crate::pyo3::pyfunction] + fn sandbox_grants(args_json: ::std::string::String) -> ::std::string::String { + $crate::ffi_sandbox_grants(__autoprover_app(), &args_json) + } + + #[$crate::pyo3::pyfunction] + fn workspace_prep(input_json: ::std::string::String) -> ::std::string::String { + $crate::ffi_workspace_prep(__autoprover_app(), &input_json) + } + + #[$crate::pyo3::pyfunction] + fn finalize( + outcomes_json: ::std::string::String, + ) -> ::std::option::Option<::std::string::String> { + $crate::ffi_finalize(__autoprover_app(), &outcomes_json) + } + + #[$crate::pyo3::pymodule] + fn $module( + m: &$crate::pyo3::Bound<'_, $crate::pyo3::types::PyModule>, + ) -> $crate::pyo3::PyResult<()> { + use $crate::pyo3::types::PyModuleMethods as _; + m.add_function($crate::pyo3::wrap_pyfunction!(descriptor, m)?)?; + m.add_function($crate::pyo3::wrap_pyfunction!(validate_preconditions, m)?)?; + m.add_function($crate::pyo3::wrap_pyfunction!(units, m)?)?; + m.add_function($crate::pyo3::wrap_pyfunction!(author_prompt, m)?)?; + m.add_function($crate::pyo3::wrap_pyfunction!(judge_prompt, m)?)?; + m.add_function($crate::pyo3::wrap_pyfunction!(compile, m)?)?; + m.add_function($crate::pyo3::wrap_pyfunction!(validate, m)?)?; + m.add_function($crate::pyo3::wrap_pyfunction!(sandbox_grants, m)?)?; + m.add_function($crate::pyo3::wrap_pyfunction!(workspace_prep, m)?)?; + m.add_function($crate::pyo3::wrap_pyfunction!(finalize, m)?)?; + ::std::result::Result::Ok(()) + } + }; +} diff --git a/rust/crucible-app/Cargo.toml b/rust/crucible-app/Cargo.toml new file mode 100644 index 00000000..02377465 --- /dev/null +++ b/rust/crucible-app/Cargo.toml @@ -0,0 +1,19 @@ +[package] +name = "crucible-app" +edition.workspace = true +version.workspace = true +license.workspace = true +description = "AutoProver Solana verification application (Crucible fuzzing backend), built into a Python wheel." + +[lib] +# MUST match the `export_app!` module ident and the maturin module name. +name = "crucible_app" +crate-type = ["cdylib"] + +[dependencies] +autoprover-sdk = { path = "../autoprover-sdk" } +askama = { workspace = true } +serde = { workspace = true } +serde_json = { workspace = true } +# `extension-module` (don't link libpython) + `abi3-py312` (one wheel for cp312+). +pyo3 = { workspace = true, features = ["extension-module", "abi3-py312"] } diff --git a/rust/crucible-app/askama.toml b/rust/crucible-app/askama.toml new file mode 100644 index 00000000..731899f5 --- /dev/null +++ b/rust/crucible-app/askama.toml @@ -0,0 +1,4 @@ +# Preserve template whitespace verbatim — these templates render prompts and Rust/TOML +# source where newlines are significant, so we don't want askama trimming around tags. +[general] +whitespace = "preserve" diff --git a/rust/crucible-app/crucible_kb.rag.json b/rust/crucible-app/crucible_kb.rag.json new file mode 100644 index 00000000..6dc53e12 --- /dev/null +++ b/rust/crucible-app/crucible_kb.rag.json @@ -0,0 +1,2072 @@ +{ + "version": 1, + "knowledge_base": "crucible_kb", + "source": "crucible docs (docs/*.md + README.md) @ 35ec899", + "sections": [ + { + "headers": [ + "TestContext API Reference", + "Index" + ], + "blocks": [ + { + "kind": "text", + "body": "- [Program Loading](#program-loading) — `add_program`\n- [Account Creation](#account-creation) — `create_account`, `create_mint`, `create_token_account`\n- [Program Calls](#program-calls) — `program(...).call(...).accounts(...).signers(...).send()`\n- [Transaction Results (`TxOutcome`)](#transaction-results-txoutcome)\n- [Raw Instruction Calls](#raw-instruction-calls) — `raw_call` for non-Anchor / custom instructions\n- [Transaction Batching](#transaction-batching) — `add_transaction`, `send_batch`\n- [Account Reading/Writing](#account-readingwriting) — `read_anchor_account`, `update_account`, zero-copy helpers\n- [RPC Account Cloning](#rpc-account-cloning) — `clone_from_rpc`, `clone_account(s)`\n- [Time Control](#time-control) — `slot`, `warp_to_slot`, `advance_slots`\n- [Mock Pyth Oracles](#mock-pyth-oracles)\n- [Fixture Hooks](#fixture-hooks) — `setup`, `after_action` (see also [Writing Tests](writing-tests.md))" + } + ] + }, + { + "headers": [ + "TestContext API Reference", + "Program Loading" + ], + "blocks": [ + { + "kind": "code", + "body": "ctx.add_program(&program_id, \"path/to/program.so\")?;" + } + ] + }, + { + "headers": [ + "TestContext API Reference", + "Account Creation" + ], + "blocks": [ + { + "kind": "code", + "body": "// Generic account\nctx.create_account()\n .pubkey(address)\n .lamports(1_000_000_000)\n .owner(system_program::id())\n .size(128) // optional\n .create()?;\n\n// Mint account\nctx.create_mint()\n .pubkey(mint_address)\n .mint_authority(authority)\n .decimals(9)\n .create()?;\n\n// Token account\nctx.create_token_account()\n .pubkey(token_address)\n .mint(mint_address)\n .token_owner(owner)\n .amount(1000)\n .create()?;" + } + ] + }, + { + "headers": [ + "TestContext API Reference", + "Program Calls" + ], + "blocks": [ + { + "kind": "code", + "body": "ctx.program(program_id)\n .call(instruction::DoSomething { amount })\n .accounts(accounts::DoSomething {\n user: user.pubkey(),\n pool: pool_pda,\n system_program: system_program::id(),\n })\n .signers(&[&*user])\n .send()?;" + } + ] + }, + { + "headers": [ + "TestContext API Reference", + "Transaction Results (`TxOutcome`)" + ], + "blocks": [ + { + "kind": "text", + "body": "The `.send()` method returns `Result<TxOutcome>`, a parsed transaction result:" + }, + { + "kind": "code", + "body": "use crucible_test_context::TxOutcome;\n\nlet result = ctx.program(program_id)\n .call(instruction::DoSomething { amount })\n .accounts(accounts::DoSomething { /* ... */ })\n .signers(&[&*user])\n .send()?;\n\nmatch result {\n TxOutcome::Success { compute_units, logs } => {\n println!(\"Used {} CU\", compute_units);\n }\n TxOutcome::ProgramError { error, error_code, logs, .. } => {\n if let Some(code) = error_code {\n println!(\"Failed with error code: {}\", code);\n }\n }\n}" + }, + { + "kind": "text", + "body": "**Helper methods:**\n- `is_success()` / `is_error()` - Check outcome type\n- `error_code()` - Extract `Custom(N)` error codes\n- `logs()` - Get program logs\n- `unwrap()` / `expect(msg)` - Panic on error with detailed message\n- `into_result()` - Convert to `Result<(), TxError>` for `?` operator\n\n---" + } + ] + }, + { + "headers": [ + "TestContext API Reference", + "Raw Instruction Calls" + ], + "blocks": [ + { + "kind": "text", + "body": "For non-Anchor programs or custom instructions:" + }, + { + "kind": "code", + "body": "use solana_instruction::{AccountMeta, Instruction};\n\nlet instruction = Instruction {\n program_id: my_program_id,\n accounts: vec![\n AccountMeta::new(user.pubkey(), true),\n AccountMeta::new(pool_pda, false),\n AccountMeta::new_readonly(config, false),\n ],\n data: my_instruction_data.try_to_vec()?,\n};\n\nctx.raw_call(instruction)\n .signers(&[&*user])\n .send()?;" + } + ] + }, + { + "headers": [ + "TestContext API Reference", + "Transaction Batching" + ], + "blocks": [ + { + "kind": "text", + "body": "Combine multiple instructions into a single atomic transaction:" + }, + { + "kind": "code", + "body": "ctx.program(program_id)\n .call(instruction::Initialize {})\n .accounts(accounts::Initialize { /* ... */ })\n .signers(&[&*payer])\n .add_transaction()?;\n\nctx.program(program_id)\n .call(instruction::Deposit { amount: 1000 })\n .accounts(accounts::Deposit { /* ... */ })\n .signers(&[&*user])\n .add_transaction()?;\n\n// Send all queued instructions as ONE atomic transaction\nctx.send_batch()?;" + } + ] + }, + { + "headers": [ + "TestContext API Reference", + "Account Reading/Writing" + ], + "blocks": [ + { + "kind": "code", + "body": "let account: MyAccount = ctx.read_anchor_account(&address)?;\nlet account: Account = ctx.get_account(&address)?;\nctx.write_anchor_account(&address, &my_data)?;" + }, + { + "kind": "text", + "body": "Read-modify-write a single account atomically:" + }, + { + "kind": "code", + "body": "ctx.update_account(&address, |account| {\n account.lamports += 1_000_000;\n Ok(())\n})?;" + }, + { + "kind": "text", + "body": "Check whether an account exists with at least `min_size` bytes of data (useful in invariants that need to skip uninitialized accounts):" + }, + { + "kind": "code", + "body": "if ctx.account_has_data(&pubkey, 8) {\n let data: MyAccount = ctx.read_anchor_account(&pubkey)?;\n // ...\n}" + } + ] + }, + { + "headers": [ + "TestContext API Reference", + "Account Reading/Writing", + "Zero-Copy Accounts" + ], + "blocks": [ + { + "kind": "text", + "body": "For accounts using `bytemuck`/zero-copy layouts. Default discriminator length is 8 bytes; use the `_with_discriminator` variants for non-Anchor zero-copy formats." + }, + { + "kind": "code", + "body": "let pool: Pool = ctx.read_zero_copy_account(&pool_address)?;\nctx.write_zero_copy_account(&pool_address, &updated_pool)?;\n\n// Custom discriminator length (e.g., 0 for raw layouts):\nlet raw: RawState = ctx.read_zero_copy_account_with_discriminator(&address, 0)?;\nctx.write_zero_copy_account_with_discriminator(&address, &raw, 0)?;" + } + ] + }, + { + "headers": [ + "TestContext API Reference", + "Account Reading/Writing", + "SPL Token Balance" + ], + "blocks": [ + { + "kind": "code", + "body": "let amount = ctx.token_balance(&token_account_address);" + } + ] + }, + { + "headers": [ + "TestContext API Reference", + "RPC Account Cloning" + ], + "blocks": [ + { + "kind": "text", + "body": "Clone accounts directly from mainnet/devnet into your test environment. Accounts are cached to disk in `.fuzz-cache/accounts/` - subsequent runs load from cache without RPC calls.\n\nRequires the `rpc-clone` feature in your fuzz harness Cargo.toml:" + }, + { + "kind": "code", + "body": "crucible-test-context = { ..., features = [\"rpc-clone\"] }" + } + ] + }, + { + "headers": [ + "TestContext API Reference", + "RPC Account Cloning", + "Basic Usage" + ], + "blocks": [ + { + "kind": "code", + "body": "fn setup() -> Self {\n let mut ctx = TestContext::new();\n let mut cloner = ctx.clone_from_rpc(\"https://api.mainnet-beta.solana.com\");\n\n // Clone a program (auto-handles BPF Upgradeable Loader)\n cloner.clone_account(&program_id)?;\n\n // Batch clone accounts (PDAs, token accounts, oracles, etc)\n cloner.clone_accounts(&[reserve_a, reserve_b, oracle_a])?;\n\n // ...\n}" + } + ] + }, + { + "headers": [ + "TestContext API Reference", + "RPC Account Cloning", + "Cache Management" + ], + "blocks": [ + { + "kind": "code", + "body": "// Force re-fetch from RPC (ignore disk cache)\nlet mut cloner = ctx.clone_from_rpc(rpc_url).force_refresh();\ncloner.clone_account(&stale_account)?;\n\n// Custom cache directory\nlet mut cloner = ctx.clone_from_rpc(rpc_url).cache_dir(\"./my-cache\");\n\n// Invalidate specific account or entire cache\ncloner.invalidate(&pubkey)?;\ncloner.clear_cache()?;" + } + ] + }, + { + "headers": [ + "TestContext API Reference", + "RPC Account Cloning", + "Program Account Fetching" + ], + "blocks": [ + { + "kind": "code", + "body": "use solana_rpc_client_api::filter::{RpcFilterType, Memcmp};\n\n// Fetch all accounts owned by a program (requires filters)\n// WARNING: Unfiltered getProgramAccounts can return millions of results\nlet filters = vec![RpcFilterType::DataSize(200)];\nlet pubkeys = cloner.clone_program_accounts(&program_id, &filters)?;" + } + ] + }, + { + "headers": [ + "TestContext API Reference", + "RPC Account Cloning", + "How Caching Works" + ], + "blocks": [ + { + "kind": "text", + "body": "- First run: fetches from RPC and saves to `.fuzz-cache/accounts/<pubkey>.json` + `.bin`\n- Subsequent runs: loads from disk (no network calls)\n- Use `.force_refresh()` to re-fetch stale accounts\n- Add `.fuzz-cache/` to `.gitignore`" + } + ] + }, + { + "headers": [ + "TestContext API Reference", + "Time Control" + ], + "blocks": [ + { + "kind": "code", + "body": "let current = ctx.slot();\nctx.warp_to_slot(current + 1000);\nctx.advance_slots(100);" + } + ] + }, + { + "headers": [ + "TestContext API Reference", + "Mock Pyth Oracles" + ], + "blocks": [ + { + "kind": "code", + "body": "let oracle = ctx.create_mock_pyth_oracle()\n .price(100_00000000) // $100 with 8 decimals\n .exponent(-8)\n .confidence(100_000)\n .build()?;\n\nctx.update_pyth_price(&oracle, 95_00000000, -8)?;\nctx.refresh_pyth_oracle(&oracle)?;" + } + ] + }, + { + "headers": [ + "TestContext API Reference", + "Fixture Hooks" + ], + "blocks": [ + { + "kind": "text", + "body": "`#[fuzz_fixture]` recognises a few special method names on the fixture impl:\n\n- `setup() -> Self` — required. Builds the initial state once per iteration (stateless) or once per state-pool entry (stateful).\n- `action_*` — fuzz-discovered actions, see [Writing Tests › Action Naming Convention](writing-tests.md#action-naming-convention).\n- `after_action(&self)` — optional. Called after every action dispatch. Useful for diagnostic logging or state stats. See [Writing Tests › After-Action Callback](writing-tests.md#after-action-callback)." + } + ] + }, + { + "headers": [ + "CLI Reference", + "`crucible init`" + ], + "blocks": [ + { + "kind": "code", + "body": "crucible init <program_name> [-C <HARNESS_DIR>]" + }, + { + "kind": "text", + "body": "Creates a standalone fuzz workspace in `fuzz/<program_name>/` by default, or directly in `-C/--harness-dir`." + } + ] + }, + { + "headers": [ + "CLI Reference", + "`crucible run`" + ], + "blocks": [ + { + "kind": "code", + "body": "crucible run <program_name> <test_name> [OPTIONS]" + }, + { + "kind": "text", + "body": "| Flag | Description |\n|------|-------------|\n| `--binary-in <PATH>` | Run a prebuilt harness binary directly |\n| `--release` | Build in release mode (recommended) |\n| `--coverage` | Enable coverage reporting |\n| `--timeout <SECS>` | Stop after N seconds |\n| `--cores N` / `-j N` | Run N parallel fuzzer workers |\n| `--corpus-in <DIR>` | Load seed corpus from directory |\n| `--corpus-out <DIR>` | Write corpus to directory |\n| `--crashes-out <DIR>` | Custom crash output directory |\n| `--replay <FILE>` | Replay a single input file |\n| `--dry-run` | Validate setup without fuzzing |\n| `--seed <N>` | Random seed for reproducible fuzzing |\n| `--symbols <PATH>` | Path to debug binary with DWARF symbols (for source-level coverage) |\n| `--no-tracing` | Disable SVM register tracing (~2x faster, no coverage) |\n| `--stop-on-crash` | Stop fuzzing on first crash |\n| `--max-actions <N>` | Max actions per iteration (default: 8 stateless, 100 stateful) |\n| `--stateful` | Stateful fuzzing: single action per iteration with state pool |\n| `--max-depth <N>` | Maximum state depth (action chain length) in stateful mode (default: 15) |\n| `--pool-size <N>` | State pool capacity in stateful mode (default: 256000) |\n| `--program-so <PATH>` | Override the program `.so` loaded by the harness |\n| `--mode <MODE>` | Remote fuzzing operational mode (see [Remote Fuzzing Integration](remote-fuzzing.md)) |\n| `--lcov-out <PATH>` | Custom LCOV coverage output file path |\n\n**Examples:**" + }, + { + "kind": "code", + "body": "# Basic fuzzing\ncrucible run myproject invariant_test --release --timeout 60\n\n# Run a prebuilt harness directly\ncrucible run myproject invariant_test --binary-in ./fuzz/myproject/target/release/invariant_test\n\n# Multi-core fuzzing (4 workers)\ncrucible run myproject invariant_test --release -j 4\n\n# Coverage report\ncrucible run myproject invariant_test --release --coverage --timeout 120\n\n# Coverage with custom output path\ncrucible run myproject invariant_test --release --coverage --lcov-out ./output/coverage.lcov\n\n# Custom harness directory\ncrucible run myproject invariant_test -C ./custom-harness --release\n\n# Dry-run validation\ncrucible run myproject invariant_test --dry-run\n\n# Replay a crash\ncrucible run myproject invariant_test --replay ./crashes/invariant_test/abc123\n\n# Reproducible fuzzing\ncrucible run myproject invariant_test --release --seed 12345\n\n# Stateful mode\ncrucible run myproject invariant_test --release --stateful\n\n# Stateful with custom depth and multi-core\ncrucible run myproject invariant_test --release --stateful --max-depth 20 -j 4" + }, + { + "kind": "text", + "body": "**Environment variables:**\n\n| Variable | Description |\n|----------|-------------|\n| `FUZZ_VERBOSE=1` | Enable verbose harness output |\n| `FUZZ_STATS_CSV=<path>` | Write per-second stats to CSV file for benchmarking |" + } + ] + }, + { + "headers": [ + "CLI Reference", + "`crucible list`" + ], + "blocks": [ + { + "kind": "code", + "body": "crucible list <program_name> # List tests for a program\ncrucible list -C ./fuzz/myproject # List tests from a harness dir\ncrucible list # List all fuzz harnesses" + } + ] + }, + { + "headers": [ + "CLI Reference", + "`crucible show`" + ], + "blocks": [ + { + "kind": "code", + "body": "crucible show <program> # List all crashes\ncrucible show <program> <crash_file> # View metadata\ncrucible show <program> <crash_file> --replay # Replay crash\ncrucible show <program> --crashes-dir <DIR> # List crashes from custom dir\ncrucible show <program> <crash_file> --crashes-dir <DIR> # View metadata from custom dir\ncrucible show . <crash_file> # Auto-detect from current dir" + }, + { + "kind": "text", + "body": "Use `\".\"` as the program name to auto-detect the fuzz harness when running from within the fuzz directory.\n\n`-C/--harness-dir` is a global option for locating a custom harness directory on `init`, `run`, `list`, `show`, `tmin`, and `cmin`.\n\n| Flag | Description |\n|------|-------------|\n| `--replay` | Actually replay the crash by running the binary |\n| `--regen` | Regenerate crash metadata (requires `--replay`) |\n| `--crashes-dir <DIR>` | Custom crashes directory to read from (supports flat and nested layouts) |" + } + ] + }, + { + "headers": [ + "CLI Reference", + "`crucible cmin`" + ], + "blocks": [ + { + "kind": "text", + "body": "Minimize corpus to smallest set preserving all coverage." + }, + { + "kind": "code", + "body": "crucible cmin <program> <test> <corpus_dir> --release\ncrucible cmin <program> <test> <corpus_dir> --corpus-out ./corpus_min --release\ncrucible cmin <program> <test> --corpus-in <DIR> --release" + }, + { + "kind": "text", + "body": "| Flag | Description |\n|------|-------------|\n| `--release` | Build in release mode |\n| `--corpus-in <DIR>` | Input corpus directory (alternative to positional arg) |\n| `--corpus-out <DIR>` | Output directory (default: overwrite input) |" + } + ] + }, + { + "headers": [ + "CLI Reference", + "`crucible tmin`" + ], + "blocks": [ + { + "kind": "text", + "body": "Minimize a crash to the smallest action sequence that still reproduces the violation." + }, + { + "kind": "code", + "body": "crucible tmin <program> <test> <crash_file> [OPTIONS]\ncrucible tmin <program> <test> --all [OPTIONS]" + }, + { + "kind": "text", + "body": "| Flag | Description |\n|------|-------------|\n| `--release` | Build in release mode |\n| `--all` | Minimize all crashes for this test |\n\nOnly works with structured/invariant tests (action sequences). Overwrites crash files in place." + }, + { + "kind": "code", + "body": "# Minimize a single crash\ncrucible tmin myproject invariant_test crash_abc123 --release\n\n# Minimize all crashes\ncrucible tmin myproject invariant_test --all --release" + }, + { + "kind": "text", + "body": "---" + } + ] + }, + { + "headers": [ + "CLI Reference", + "Execution Modes" + ], + "blocks": [ + { + "kind": "text", + "body": "1. **Normal Fuzzing** (default) - Continuous input generation and mutation\n2. **Dry-Run** (`--dry-run`) - Single iteration to validate harness setup\n3. **Input Replay** (`--replay`) - Execute one specific input file\n4. **Coverage-Only** (`--coverage --corpus-in`) - Run corpus once for coverage report\n5. **Seeded Fuzzing** (`--corpus-in`) - Start from pre-existing corpus\n6. **Multi-Core** (`--cores N`) - Parallel fuzzer workers with shared coverage\n7. **Stateful** (`--stateful`) - Single action per iteration with state pool\n8. **Corpus Minimization** (`crucible cmin`) - Reduce corpus to minimal set preserving coverage\n9. **Crash Minimization** (`crucible tmin`) - Reduce crash to minimal reproducing action sequence" + } + ] + }, + { + "headers": [ + "CLI Reference", + "Execution Modes", + "Stateful Mode" + ], + "blocks": [ + { + "kind": "text", + "body": "Stateful mode (`--stateful`) uses a state-pool approach where each fuzzer iteration executes a **single action** on a state selected from a pool, rather than replaying an entire action sequence from scratch.\n\n- States form a tree: each state has a parent and a depth (action chain length)\n- New states are created by applying an action to an existing state\n- The state pool is bounded; states are evicted based on coverage novelty\n- `--max-depth <N>` controls maximum chain length (default: 100)\n- Works with both single-core and multi-core (`-j N`)\n- Crashes record the full action chain from root to violation" + }, + { + "kind": "code", + "body": "# Basic stateful fuzzing\ncrucible run myproject invariant_test --release --stateful\n\n# With custom depth limit\ncrucible run myproject invariant_test --release --stateful --max-depth 50\n\n# Multi-core stateful\ncrucible run myproject invariant_test --release --stateful -j 4" + }, + { + "kind": "text", + "body": "---" + } + ] + }, + { + "headers": [ + "CLI Reference", + "Remote Fuzzing Integration" + ], + "blocks": [ + { + "kind": "text", + "body": "For running Crucible as a managed engine on a remote fuzzing platform, see the dedicated **[Remote Fuzzing Integration Guide](remote-fuzzing.md)** covering:\n\n- All five operational modes (`dry_run`, `explore`, `reproduce`, `coverage`, `corpus_merge`)\n- Directory conventions (`./corpus`, `./input`, `./output`)\n- Structured output protocol (`[FUZZ_PULSE]`, `[FUZZ_FINDING]`, `[FUZZ_ERROR]`)\n- Bundle layout and manifest configuration\n- Exit code semantics per mode" + } + ] + }, + { + "headers": [ + "Coverage Reports" + ], + "blocks": [ + { + "kind": "text", + "body": "Crucible can generate LCOV coverage reports showing which lines of your Solana program were exercised during fuzzing. There are two modes: **bytecode-level** (default, no extra setup) and **source-level** (requires a debug binary, produces highlighted Rust source)." + } + ] + }, + { + "headers": [ + "Coverage Reports", + "Bytecode-level coverage" + ], + "blocks": [ + { + "kind": "text", + "body": "Just add `--coverage` to your run command. This writes `./coverage/coverage.lcov` by default and uses SBF program counter addresses as line numbers:" + }, + { + "kind": "code", + "body": "crucible run myproject invariant_test --release --coverage --timeout 60" + }, + { + "kind": "text", + "body": "This is useful for tracking coverage growth over time, but the LCOV output uses fake line numbers (PC addresses) rather than real source locations." + } + ] + }, + { + "headers": [ + "Coverage Reports", + "Source-level coverage" + ], + "blocks": [ + { + "kind": "text", + "body": "For proper source-level reports that map to your Rust code, you need a **debug binary** — the unstripped ELF with DWARF debug sections." + } + ] + }, + { + "headers": [ + "Coverage Reports", + "Source-level coverage", + "Step 1: Build the debug binary" + ], + "blocks": [ + { + "kind": "text", + "body": "Add these settings to your program's workspace `Cargo.toml` under `[profile.release]`:" + }, + { + "kind": "code", + "body": "[profile.release]\nopt-level = 1 # Better DWARF accuracy (opt-level 3 causes gaps due to inlining)\ndebug = 2 # Full DWARF debug info\nstrip = false # Keep debug sections in the binary" + }, + { + "kind": "text", + "body": "Then build with `--debug` and **platform-tools v1.51+** (earlier versions have a linker bug that corrupts DWARF address relocations):" + }, + { + "kind": "code", + "body": "cargo build-sbf --debug --tools-version v1.51 \\\n --manifest-path programs/<your_program>/Cargo.toml" + }, + { + "kind": "text", + "body": "> **Important:** Platform-tools versions before v1.51 produce corrupted DWARF debug info due to a bug in the SBF linker (LLD) where `R_SBF_64_64` relocations are incorrectly applied to debug sections instead of `R_SBF_64_ABS64`. This causes `addr2line` to resolve 0 PCs. See [anza-xyz/llvm-project#159](https://github.com/anza-xyz/llvm-project/pull/159) for details.\n\nThis produces two binaries:\n- `target/deploy/<program>.so` — stripped, for SVM execution (~1.5 MB)\n- `target/sbpf-solana-solana/release/<program>.so` — unstripped with DWARF (~30-40 MB)\n\n> **Important:** The execution binary and debug binary must come from the **same build** so that PC addresses match. If you rebuild the debug binary, also copy the deploy binary to your fuzz harness." + } + ] + }, + { + "headers": [ + "Coverage Reports", + "Source-level coverage", + "Step 2: Copy the deploy binary" + ], + "blocks": [ + { + "kind": "text", + "body": "Copy the freshly built deploy binary to your fuzz harness directory (replacing the existing one):" + }, + { + "kind": "code", + "body": "cp target/deploy/<program>.so path/to/fuzz/harness/<program>.so" + } + ] + }, + { + "headers": [ + "Coverage Reports", + "Source-level coverage", + "Step 3: Run with `--symbols`" + ], + "blocks": [ + { + "kind": "text", + "body": "Point `--symbols` at the unstripped debug binary:" + }, + { + "kind": "code", + "body": "crucible run myproject invariant_test --release --coverage --timeout 60 \\\n --symbols /path/to/target/sbpf-solana-solana/release/<program>.so" + }, + { + "kind": "text", + "body": "To write the LCOV file to a custom path (e.g., for remote fuzzing integration), use `--lcov-out`:" + }, + { + "kind": "code", + "body": "crucible run myproject invariant_test --release --coverage --corpus-in ./corpus \\\n --lcov-out ./output/coverage.lcov" + }, + { + "kind": "text", + "body": "You should see output like:" + }, + { + "kind": "code", + "body": "[COVERAGE] DWARF source map loaded: 143160 PCs resolved\n[LCOV] Source-level coverage: 133 source files\n[LCOV] Coverage written to coverage.lcov (2 programs, 23994 lines, 2166 branches)" + } + ] + }, + { + "headers": [ + "Coverage Reports", + "Source-level coverage", + "Step 4: Generate HTML report" + ], + "blocks": [ + { + "kind": "text", + "body": "By default the LCOV file is written to `./coverage/coverage.lcov` (or the path passed via `--lcov-out`). Use `lcov` and `genhtml` to produce a browsable HTML report.\n\nFirst, extract only your program's source files (filtering out stdlib and third-party crate paths):" + }, + { + "kind": "code", + "body": "lcov --extract coverage.lcov '*/programs/<your_program>/*' -o program_coverage.lcov" + }, + { + "kind": "text", + "body": "Then generate HTML:" + }, + { + "kind": "code", + "body": "genhtml program_coverage.lcov -o coverage_html --legend\nopen coverage_html/index.html" + }, + { + "kind": "text", + "body": "> **Tip:** Run `genhtml` from your program's workspace root so that relative source paths resolve correctly." + } + ] + }, + { + "headers": [ + "Coverage Reports", + "Why opt-level 1?" + ], + "blocks": [ + { + "kind": "text", + "body": "At `opt-level = 3` (the default for release), the compiler aggressively inlines and reorders code. This produces lossy DWARF debug info — many source lines have no corresponding instructions, causing gaps in the coverage report. `opt-level = 1` preserves the source-to-instruction mapping much more faithfully:\n\n| Setting | Executable lines | Lines hit | Coverage |\n|---------|-----------------|-----------|----------|\n| opt-level 3 | 2,708 | 657 | 24.3% |\n| opt-level 1 | 1,988 | 856 | 43.1% |\n\nThe opt-level 1 binary is larger and slower at runtime. The recommended workflow is to fuzz normally (without `--coverage`) to build up a corpus, then generate the coverage report separately:" + }, + { + "kind": "code", + "body": "# 1. Fuzz normally to build a corpus (fast, no coverage overhead)\ncrucible run myproject invariant_test --release --timeout 300 --corpus-out ./corpus\n\n# 2. Replay the corpus with --coverage to generate the report (quick single pass)\ncrucible run myproject invariant_test --release --coverage --corpus-in ./corpus \\\n --symbols /path/to/debug/binary.so" + }, + { + "kind": "text", + "body": "This way your main fuzzing runs at full speed, and coverage reports are generated on demand from the saved corpus." + } + ] + }, + { + "headers": [ + "Coverage Reports", + "Different optimization levels (3-binary workflow)" + ], + "blocks": [ + { + "kind": "text", + "body": "When you fuzz with an optimized binary (default `cargo build-sbf`) but need source-level coverage, the PCs from the optimized binary won't match the DWARF debug info (which requires `opt-level = 1`). Use `--program-so` to load a debug-stripped binary whose instruction layout matches the DWARF symbols binary." + } + ] + }, + { + "headers": [ + "Coverage Reports", + "Different optimization levels (3-binary workflow)", + "Build the binaries" + ], + "blocks": [ + { + "kind": "code", + "body": "cd programs/my_program\n\n# Build 1: optimized binary for fuzzing\ncargo build-sbf\ncp target/deploy/my_program.so /path/to/fuzz/optimized.so\n\n# Build 2: debug binary for coverage (edit Cargo.toml first, or append)\ncat >> Cargo.toml <<'EOF'\n[profile.release]\nopt-level = 1\ndebug = 2\nstrip = false\nEOF\n\ncargo build-sbf\ncp target/deploy/my_program.so /path/to/fuzz/debug.so # stripped, for execution\ncp target/sbpf-solana-solana/release/my_program.so /path/to/fuzz/symbols.so # unstripped, for DWARF" + }, + { + "kind": "text", + "body": "This produces 3 binaries:\n\n| Binary | Optimization | DWARF | Purpose |\n|--------|-------------|-------|---------|\n| `optimized.so` | default (fast) | no | Fuzzing |\n| `debug.so` | opt-level=1, stripped | no | Coverage execution (PCs match symbols) |\n| `symbols.so` | opt-level=1, unstripped | yes | DWARF source mapping |" + } + ] + }, + { + "headers": [ + "Coverage Reports", + "Different optimization levels (3-binary workflow)", + "Run coverage with --program-so" + ], + "blocks": [ + { + "kind": "code", + "body": "# Fuzz with the optimized binary (fast)\ncrucible run myproject invariant_test --release --timeout 300 --corpus-out ./corpus\n\n# Generate coverage: load debug.so for execution, symbols.so for DWARF\ncrucible run myproject invariant_test --release --coverage --corpus-in ./corpus \\\n --program-so ./debug.so --symbols ./symbols.so" + }, + { + "kind": "text", + "body": "The `--program-so` flag overrides the `.so` that the harness loads into litesvm at runtime, without requiring any harness code changes." + } + ] + }, + { + "headers": [ + "Coverage Reports", + "LCOV format reference" + ], + "blocks": [ + { + "kind": "text", + "body": "The generated `coverage.lcov` file uses standard LCOV format. This section documents the format and how to programmatically consume it for automated coverage analysis (e.g., in `solana-rag`'s `coverage_analysis` / `coverage_loop` tools)." + } + ] + }, + { + "headers": [ + "Coverage Reports", + "LCOV format reference", + "File structure" + ], + "blocks": [ + { + "kind": "text", + "body": "Each source file gets a record block:" + }, + { + "kind": "code", + "body": "TN:fuzzer\nSF:/absolute/path/to/source_file.rs\nFN:<line>,<function_name>\nFNDA:<hit_count>,<function_name>\nFNF:<functions_found>\nFNH:<functions_hit>\nDA:<line_number>,<execution_count>\nDA:<line_number>,<execution_count>\n...\nLF:<lines_found>\nLH:<lines_hit>\nBRDA:<line>,<block>,<branch>,<taken_count>\nBRF:<branches_found>\nBRH:<branches_hit>\nend_of_record" + }, + { + "kind": "text", + "body": "**Key record types:**\n- `SF:` — absolute path to the source file (only user source files, stdlib/deps filtered out)\n- `FN:` / `FNDA:` — function declarations and hit counts\n- `DA:<line>,<count>` — line hit data. `count=0` means executable but not reached\n- `BRDA:<line>,<block>,<branch>,<taken>` — branch data. `taken=-` means not executed\n- `LF:` / `LH:` — total lines found vs hit (for computing line coverage %)" + } + ] + }, + { + "headers": [ + "Coverage Reports", + "LCOV format reference", + "Source-level vs bytecode-level" + ], + "blocks": [ + { + "kind": "text", + "body": "| | Source-level (with `--symbols`) | Bytecode-level (without `--symbols`) |\n|---|---|---|\n| `SF:` paths | Real source files (`programs/stake/src/processor.rs`) | Synthetic (`program_2a54379117d8a106.bpf`) |\n| `DA:` line numbers | Actual source line numbers (1-based) | SBF program counter addresses |\n| `FN:` names | Demangled Rust function names | `fn_0`, `fn_25`, etc. |\n| Useful for | Gap analysis, genhtml, CI reporting | Tracking coverage growth over time |" + } + ] + }, + { + "headers": [ + "Coverage Reports", + "LCOV format reference", + "Generating source-level LCOV for programmatic analysis" + ], + "blocks": [ + { + "kind": "text", + "body": "The recommended flow for tools that want to analyze coverage gaps:" + }, + { + "kind": "code", + "body": "# 1. Fuzz to build corpus (fast, multi-core, no coverage overhead)\ncrucible run prog test --release -j 4 --timeout 300 --corpus-out ./corpus\n\n# 2. Generate source-level LCOV (single pass over corpus, no mutation)\n# Requires: debug.so (same opt-level as symbols.so) + symbols.so (DWARF)\ncrucible run prog test --release --coverage --corpus-in ./corpus \\\n --program-so ./debug.so \\\n --symbols ./symbols.so \\\n --lcov-out ./output/coverage.lcov" + }, + { + "kind": "text", + "body": "**What happens in step 2:**\n1. `FUZZ_COVERAGE_ONLY=1` is set automatically (coverage + corpus-in + no timeout)\n2. Each corpus input is replayed once with SVM register tracing enabled\n3. PCs from execution are mapped to source locations via DWARF from `--symbols`\n4. LCOV is written with real file paths and line numbers" + } + ] + }, + { + "headers": [ + "Coverage Reports", + "LCOV format reference", + "Parsing LCOV programmatically" + ], + "blocks": [ + { + "kind": "text", + "body": "Python example (matches `solana-rag`'s `coverage.parse_lcov()`):" + }, + { + "kind": "code", + "body": "def parse_lcov(path: str) -> list[dict]:\n \"\"\"Returns per-file coverage: source_file, lines_hit, lines_total, hit_pct, uncovered_lines.\"\"\"\n results = []\n current_file = \"\"\n lines_hit = lines_total = 0\n uncovered = []\n\n for line in open(path):\n line = line.strip()\n if line.startswith(\"SF:\"):\n current_file = line[3:]\n lines_hit = lines_total = 0\n uncovered = []\n elif line.startswith(\"DA:\"):\n parts = line[3:].split(\",\")\n line_no, count = int(parts[0]), int(parts[1])\n lines_total += 1\n if count > 0:\n lines_hit += 1\n else:\n uncovered.append(line_no)\n elif line == \"end_of_record\" and current_file:\n hit_pct = (lines_hit / lines_total * 100) if lines_total else 0\n results.append({\n \"source_file\": current_file,\n \"lines_hit\": lines_hit,\n \"lines_total\": lines_total,\n \"hit_pct\": round(hit_pct, 1),\n \"uncovered_lines\": sorted(uncovered),\n })\n current_file = \"\"\n return results" + } + ] + }, + { + "headers": [ + "Coverage Reports", + "LCOV format reference", + "Mapping uncovered lines to instructions/functions" + ], + "blocks": [ + { + "kind": "text", + "body": "Once you have `uncovered_lines` per file, map them to program instructions using the source tree:\n\n1. For each uncovered line range, find which function it belongs to (using `line_start`/`line_end` from the indexed source tree)\n2. Group uncovered ranges by function name\n3. Use these gaps to decide which actions to add or improve in the fuzz harness\n\n**Example gap output:**" + }, + { + "kind": "code", + "body": "processor.rs: StakeInstruction::Merge (lines 412-445) — 0% covered\n → Need action_merge in harness\n\nprocessor.rs: StakeInstruction::AuthorizeChecked (lines 220-235) — 0% covered\n → Admin-only, intentionally excluded\n\nhelpers/delegate.rs: validate_delegated_amount (lines 89-102) — partial\n → Edge case: delegation with lockup not tested" + } + ] + }, + { + "headers": [ + "Coverage Reports", + "LCOV format reference", + "CLI flags summary for coverage" + ], + "blocks": [ + { + "kind": "text", + "body": "| Flag | Env var | Description |\n|------|---------|-------------|\n| `--coverage` | `FUZZ_COVERAGE` | Enable LCOV output (sets `COVERAGE_ENABLED`) |\n| `--symbols <path>` | `FUZZ_SYMBOLS` | Path to unstripped `.so` with DWARF debug info |\n| `--program-so <path>` | `FUZZ_PROGRAM_SO` | Override which `.so` litesvm loads (for coverage with different opt-level) |\n| `--lcov-out <path>` | `FUZZ_COVERAGE_OUT` | Custom LCOV output path (default: `coverage.lcov`) |\n| `--corpus-in <dir>` | `FUZZ_CORPUS_IN` | Load corpus for replay (with `--coverage`, triggers coverage-only mode) |\n\n**Coverage-only mode** is activated automatically when `--coverage` + `--corpus-in` are both set without `--timeout`. It replays each corpus input once and writes LCOV, then exits." + } + ] + }, + { + "headers": [ + "Coverage Reports", + "LCOV format reference", + "Build requirements" + ], + "blocks": [ + { + "kind": "text", + "body": "| Requirement | Details |\n|-------------|---------|\n| **platform-tools >= v1.51** | Required for correct DWARF. Use `cargo build-sbf --tools-version v1.51`. See [anza-xyz/llvm-project#159](https://github.com/anza-xyz/llvm-project/pull/159). |\n| **`--debug` flag** | Pass to `cargo build-sbf` to set `DW_AT_comp_dir` for source path resolution |\n| **`[profile.release]`** | `opt-level = 1, debug = 2, strip = false` in program's `Cargo.toml` |\n| **Same build for debug.so + symbols.so** | Both must come from the same `cargo build-sbf` invocation so PCs match |" + } + ] + }, + { + "headers": [ + "Coverage Reports", + "Installing lcov and genhtml" + ], + "blocks": [ + { + "kind": "code", + "body": "# macOS\nbrew install lcov\n\n# Ubuntu/Debian\nsudo apt install lcov" + } + ] + }, + { + "headers": [ + "Crash Analysis & Replay", + "List all crashes" + ], + "blocks": [ + { + "kind": "code", + "body": "crucible show <project>" + } + ] + }, + { + "headers": [ + "Crash Analysis & Replay", + "View crash metadata" + ], + "blocks": [ + { + "kind": "code", + "body": "crucible show <project> <crash_file>" + } + ] + }, + { + "headers": [ + "Crash Analysis & Replay", + "Replay crash" + ], + "blocks": [ + { + "kind": "code", + "body": "crucible show <project> <crash_file> --replay" + } + ] + }, + { + "headers": [ + "Crash Analysis & Replay", + "Minimize a crash" + ], + "blocks": [ + { + "kind": "text", + "body": "Crashes often contain unnecessary actions — setup noise that doesn't contribute to the violation. Use `crucible tmin` to reduce a crash to the smallest action sequence that still triggers the same invariant violation:" + }, + { + "kind": "code", + "body": "# Minimize a single crash\ncrucible tmin <project> <test_name> <crash_file> --release\n\n# Minimize all crashes for a test\ncrucible tmin <project> <test_name> --all --release" + }, + { + "kind": "text", + "body": "The minimized crash overwrites the original file in place (same filename, updated binary + metadata). Use `crucible show` to view the minimized sequence.\n\n**Algorithm:** Multi-pass forward removal. First, actions after the violation are truncated. Then each remaining action is tried for removal — if the crash still reproduces without it, it's discarded. Passes repeat until no more actions can be removed (convergence), handling cases where removing later actions makes earlier ones removable.\n\n**Example output:**" + }, + { + "kind": "code", + "body": "[TMIN] Original: 10 actions\n[TMIN] Crash reproduces. Minimizing...\n[TMIN] Truncated 2 post-violation actions (violation at index 7)\n[TMIN] [0] send_batch — REMOVED\n[TMIN] [0] borrow — REMOVED\n[TMIN] [0] deposit — KEPT\n[TMIN] [1] flashloan_start — KEPT\n[TMIN] [2] borrow — KEPT\n[TMIN] [3] transfer_account — KEPT\n[TMIN] [4] flashloan_end — KEPT\n[TMIN] Pass 2...\n[TMIN] [0] deposit — KEPT\n[TMIN] [1] flashloan_start — KEPT\n[TMIN] [2] borrow — KEPT\n[TMIN] [3] transfer_account — KEPT\n[TMIN] [4] flashloan_end — KEPT\n[TMIN] Result: 10 → 5 actions (5 removed)" + } + ] + }, + { + "headers": [ + "Getting Started", + "Setup & Running", + "Initialize a fuzz project" + ], + "blocks": [ + { + "kind": "code", + "body": "crucible init <project_name>" + }, + { + "kind": "text", + "body": "This creates `fuzz/<project_name>/` by default. Use `-C <dir>` to place the harness elsewhere." + } + ] + }, + { + "headers": [ + "Getting Started", + "Setup & Running", + "Run a fuzz test" + ], + "blocks": [ + { + "kind": "code", + "body": "crucible run <project_name> <test_name>\n\ncrucible run <project_name> <test_name> --release # Optimized\n\ncrucible run <project_name> <test_name> --timeout 60 # Stop after 60 seconds\n\ncrucible run <project_name> <test_name> --release --cores 8 # 8 parallel fuzzer workers (-j 8 also works)\n\ncrucible run <project_name> <test_name> --release --stateful # Stateful mode (single action per iter with state pool)" + } + ] + }, + { + "headers": [ + "Getting Started", + "Setup & Running", + "Feature flags" + ], + "blocks": [ + { + "kind": "text", + "body": "Every fuzz test must be added as a feature in `Cargo.toml`:" + }, + { + "kind": "code", + "body": "[features]\nfuzz_single = []\ninvariant_fuzz = []" + }, + { + "kind": "text", + "body": "The test name must match the feature name exactly.\n\n---" + } + ] + }, + { + "headers": [ + "Getting Started", + "Project Structure" + ], + "blocks": [ + { + "kind": "text", + "body": "After `crucible init myproject`:" + }, + { + "kind": "code", + "body": "myproject/\n├── Cargo.toml\n├── programs/\n│ └── myproject/ # Your Solana program\n├── fuzz/\n│ └── myproject/\n│ ├── Cargo.toml # Standalone workspace + features\n│ ├── idls/\n│ │ └── myproject.json # Program IDL\n│ ├── src/\n│ │ └── main.rs # Fixtures, actions, and tests\n│ └── crashes/ # Crash artifacts saved here\n│ └── <test_name>/\n│ ├── abc123 # Raw crash input\n│ └── abc123.meta.json # Crash metadata" + } + ] + }, + { + "headers": [ + "Writing Solana/Anchor Fuzz Harnesses" + ], + "blocks": [ + { + "kind": "text", + "body": "This guide covers critical practices for writing effective fuzz harnesses for Solana programs built with Anchor. **First read the [API Reference](api-reference.md) and [Writing Tests](writing-tests.md)** for the framework API and examples.\n\n---" + } + ] + }, + { + "headers": [ + "Writing Solana/Anchor Fuzz Harnesses", + "Core Workflow: Iterative Feedback Loop" + ], + "blocks": [ + { + "kind": "text", + "body": "**The key to writing effective harnesses is running the fuzzer frequently and using feedback to guide development.**" + } + ] + }, + { + "headers": [ + "Writing Solana/Anchor Fuzz Harnesses", + "Core Workflow: Iterative Feedback Loop", + "Run Early and Often" + ], + "blocks": [ + { + "kind": "code", + "body": "# Run with short timeouts during development (3-5 seconds)\ncrucible run <program> <test> --release --timeout 5" + }, + { + "kind": "text", + "body": "**During development:**\n- Run for **3-5 seconds** if there's verbose debug output\n- Run for **10-30 seconds** if output is clean to see coverage trends\n- Watch for: error patterns, coverage plateaus, action failures" + } + ] + }, + { + "headers": [ + "Writing Solana/Anchor Fuzz Harnesses", + "Core Workflow: Iterative Feedback Loop", + "What to Look For" + ], + "blocks": [ + { + "kind": "text", + "body": "1. **Coverage growth** - Is `edges` count increasing over time?\n2. **Error patterns** - Are the same errors repeating? (indicates a blocker)\n3. **Execution speed** - Should be 200+ exec/sec once setup completes\n4. **Corpus growth** - Is the fuzzer finding new interesting inputs?" + }, + { + "kind": "code", + "body": "# Good output - coverage growing, exec speed normal:\n[FUZZ_PULSE] run time: 3s, corpus: 50, exec/sec: 1234, edges: 1491/65536 (2%)\n[FUZZ_PULSE] run time: 6s, corpus: 85, exec/sec: 1180, edges: 1572/65536 (2%) <- growth!\n\n# Bad output - coverage stuck, repeated errors:\n[DEBUG] Deposit failed: Custom(6090)\n[DEBUG] Deposit failed: Custom(6090)\n[DEBUG] Deposit failed: Custom(6090) <- same error = blocker" + } + ] + }, + { + "headers": [ + "Writing Solana/Anchor Fuzz Harnesses", + "Core Workflow: Iterative Feedback Loop", + "Debug Output Management" + ], + "blocks": [ + { + "kind": "text", + "body": "Use a global DEBUG flag to control verbose output:" + }, + { + "kind": "code", + "body": "const DEBUG: bool = true; // Set to false for production runs\n\nif DEBUG {\n eprintln!(\"[DEBUG] Action result: {:?}\", result);\n}" + }, + { + "kind": "text", + "body": "---" + } + ] + }, + { + "headers": [ + "Writing Solana/Anchor Fuzz Harnesses", + "1. Result Handling" + ], + "blocks": [ + { + "kind": "code", + "body": "use crucible_test_context::TxOutcome;\n\nmatch result? {\n TxOutcome::Success { compute_units, logs } => {\n // Transaction succeeded\n }\n TxOutcome::ProgramError { error, error_code, logs, .. } => {\n // Transaction failed\n if let Some(code) = error_code {\n println!(\"Failed with error code: {}\", code);\n }\n }\n}" + } + ] + }, + { + "headers": [ + "Writing Solana/Anchor Fuzz Harnesses", + "2. Admin/Authority Whitelists" + ], + "blocks": [ + { + "kind": "text", + "body": "Many Solana programs have **hardcoded authority whitelists**. Look for:\n- `ADMINS`, `AUTHORITIES`, `ALLOWED_SIGNERS` arrays\n- `is_admin()`, `is_authority()` check functions\n- Constraints like `#[account(constraint = is_admin(&signer.key()))]`\n\n**Always check the program source** for these patterns. If found:\n1. Locate the predefined keypair files (often in `localnet/`, `test-keys/`, or similar)\n2. Load and use those specific keypairs instead of generating random ones" + } + ] + }, + { + "headers": [ + "Writing Solana/Anchor Fuzz Harnesses", + "3. PDA Seed Encoding" + ], + "blocks": [ + { + "kind": "text", + "body": "Anchor PDAs can encode arguments as either **binary bytes** or **string bytes**. Check the actual program source:" + }, + { + "kind": "code", + "body": "// Binary encoding:\nseeds = [b\"account\", &id.to_le_bytes()]\n\n// String encoding:\nseeds = [b\"tick_array\", whirlpool.as_ref(), start_tick_index.to_string().as_bytes()]" + }, + { + "kind": "text", + "body": "**The IDL won't tell you which encoding is used** - you must read the Rust source." + } + ] + }, + { + "headers": [ + "Writing Solana/Anchor Fuzz Harnesses", + "4. Setup Must Be Loud and Fail-Fast" + ], + "blocks": [ + { + "kind": "text", + "body": "During initialization, **always panic on failure**:" + }, + { + "kind": "code", + "body": "match result? {\n TxOutcome::Success { .. } => eprintln!(\"[SETUP] InitializeX SUCCESS\"),\n TxOutcome::ProgramError { error, logs, .. } => {\n eprintln!(\"[SETUP] InitializeX FAILED: {:?}\", error);\n for log in &logs { eprintln!(\" {}\", log); }\n panic!(\"Setup failed: InitializeX\");\n }\n}" + } + ] + }, + { + "headers": [ + "Writing Solana/Anchor Fuzz Harnesses", + "5. Initialization Order Dependencies" + ], + "blocks": [ + { + "kind": "text", + "body": "Solana programs have strict initialization order. Common pattern:\n1. **Global config** (WhirlpoolsConfig, Protocol, etc.)\n2. **Fee/tier structures** (FeeTier, PriceFeed, etc.)\n3. **Token mints** (often must be ordered by pubkey: `mint_a < mint_b`)\n4. **Main accounts** (Pool, Vault, Market)\n5. **Supporting structures** (TickArrays, OrderBooks, etc.)\n6. **User accounts** (with token accounts funded)\n7. **Positions/state** with initial values (liquidity, deposits, etc.)" + } + ] + }, + { + "headers": [ + "Writing Solana/Anchor Fuzz Harnesses", + "6. Reading Anchor Error Codes" + ], + "blocks": [ + { + "kind": "text", + "body": "When you see errors like `Custom(2003)` or `Custom(6038)`:\n- Anchor errors 2000-2999 are **constraint errors** (seeds, signer, owner, etc.)\n- Anchor errors 6000+ are **program-specific errors** defined in the program\n\nLook up the error in:\n1. The program's `error.rs` or error enum\n2. Anchor's standard errors (ConstraintSeeds=2006, ConstraintRaw=2003, etc.)" + } + ] + }, + { + "headers": [ + "Writing Solana/Anchor Fuzz Harnesses", + "7. Token Account Requirements" + ], + "blocks": [ + { + "kind": "text", + "body": "For DeFi programs:\n- Create token accounts with correct mint and owner\n- Fund accounts with sufficient tokens (check decimal places!)\n- Mint authority must match what the program expects\n- Order mints correctly if program requires (e.g., `token_mint_a < token_mint_b`)" + } + ] + }, + { + "headers": [ + "Writing Solana/Anchor Fuzz Harnesses", + "8. Sysvar and Program IDs" + ], + "blocks": [ + { + "kind": "text", + "body": "For Solana v3:" + }, + { + "kind": "code", + "body": "use anchor_lang::system_program;\nsystem_program::ID // not system_program::id()" + } + ] + }, + { + "headers": [ + "Writing Solana/Anchor Fuzz Harnesses", + "9. Debugging Checklist" + ], + "blocks": [ + { + "kind": "text", + "body": "When a harness isn't working:\n1. Enable **loud setup** (eprintln + panic on all failures)\n2. Check **admin/authority requirements** in program source\n3. Verify **PDA seed encoding** matches program source\n4. Confirm **initialization order** is correct\n5. Check **account constraints** in the Accounts struct\n6. Read **program logs** from failed transactions\n7. Verify **token balances** are sufficient" + } + ] + }, + { + "headers": [ + "Writing Solana/Anchor Fuzz Harnesses", + "10. Coverage Expectations" + ], + "blocks": [ + { + "kind": "text", + "body": "Different program types have different coverage profiles:\n- **AMM/DEX** (Whirlpool, Raydium): ~2000-3000 edges\n- **Lending** (Marginfi, Kamino): ~2000-4000 edges\n\n---" + } + ] + }, + { + "headers": [ + "Writing Solana/Anchor Fuzz Harnesses" + ], + "blocks": [ + { + "kind": "text", + "body": "When coverage plateaus or actions consistently fail, you have a **blocker**." + } + ] + }, + { + "headers": [ + "Writing Solana/Anchor Fuzz Harnesses", + "11. Identifying Blockers from Error Messages" + ], + "blocks": [ + { + "kind": "text", + "body": "**Common error patterns and fixes:**\n\n| Error | Likely Cause | Fix |\n|-------|--------------|-----|\n| `Custom(6090)` DepositLimitExceeded | Config limit too low | Patch account data or use UpdateConfig |\n| `Custom(6007)` MathOverflow | last_update.slot is 0 | Set to current SVM slot |\n| `Custom(3002)` AccountDiscriminatorMismatch | Instruction format wrong | IDL outdated vs binary |\n| `Custom(3008)` InvalidProgramId | Wrong token_program | Check SPL Token vs Token-2022 |" + } + ] + }, + { + "headers": [ + "Writing Solana/Anchor Fuzz Harnesses", + "12. Adjusting Initial State (Account Patching)" + ], + "blocks": [ + { + "kind": "text", + "body": "When UpdateConfig instructions fail, patch account data directly:" + }, + { + "kind": "code", + "body": "fn configure_reserve_manually(ctx: &mut TestContext, reserve: &Pubkey, program_id: &Pubkey) {\n let account = ctx.get_account(reserve).unwrap();\n let mut data = account.data.clone();\n\n let config_offset = 5000;\n data[config_offset] = 0; // status = Active\n data[config_offset + 16..config_offset + 24]\n .copy_from_slice(&u64::MAX.to_le_bytes()); // deposit_limit = MAX\n\n ctx.create_account()\n .pubkey(*reserve)\n .lamports(account.lamports)\n .owner(*program_id)\n .data(&data)\n .create()\n .unwrap();\n}" + } + ] + }, + { + "headers": [ + "Writing Solana/Anchor Fuzz Harnesses", + "13. IDL vs Binary Layout Mismatches" + ], + "blocks": [ + { + "kind": "text", + "body": "**The IDL may not match the actual binary layout.** Write a known value and see what the program reads:" + }, + { + "kind": "code", + "body": "data[config_offset + 16..config_offset + 18].copy_from_slice(&1000u16.to_le_bytes());\n// Run the program - if it shows \"limit: 1000\", deposit_limit is at offset 16" + } + ] + }, + { + "headers": [ + "Writing Solana/Anchor Fuzz Harnesses", + "14. Action Dependencies and Multi-Step Flows" + ], + "blocks": [ + { + "kind": "text", + "body": "Some actions depend on previous actions completing successfully:" + }, + { + "kind": "code", + "body": "deposit_reserve_liquidity -> get cTokens\ndeposit_obligation_collateral -> cTokens become collateral\nborrow_obligation_liquidity -> borrow against collateral" + } + ] + }, + { + "headers": [ + "Writing Solana/Anchor Fuzz Harnesses", + "15. Early Return Checks" + ], + "blocks": [ + { + "kind": "text", + "body": "Check prerequisites before attempting actions that depend on prior state:" + }, + { + "kind": "code", + "body": "pub fn action_borrow(&mut self, user_idx: usize, amount: u64) {\n let has_collateral = /* check on-chain state */;\n if !has_collateral {\n return; // Don't waste cycles on guaranteed failure\n }\n // Proceed with borrow...\n}" + }, + { + "kind": "text", + "body": "---" + } + ] + }, + { + "headers": [ + "Writing Solana/Anchor Fuzz Harnesses", + "16. Deserialize On-Chain State" + ], + "blocks": [ + { + "kind": "text", + "body": "**Don't just track local variables** - read actual on-chain state to verify:" + }, + { + "kind": "code", + "body": "fn invariant_check(&mut self) {\n for position in &self.positions {\n let on_chain_data: PositionAccount = self.ctx\n .read_anchor_account(&position.pubkey)\n .expect(\"Position account must exist\");\n\n assert_eq!(\n on_chain_data.liquidity,\n position.expected_liquidity,\n \"Position liquidity mismatch\"\n );\n }\n}" + } + ] + }, + { + "headers": [ + "Writing Solana/Anchor Fuzz Harnesses", + "17. Designing Useful Invariants" + ], + "blocks": [ + { + "kind": "text", + "body": "**Conservation invariants** (tokens in = tokens out):" + }, + { + "kind": "code", + "body": "let total_in_vaults = self.ctx.token_balance(&vault_a) + self.ctx.token_balance(&vault_b);\nlet expected = self.total_deposits - self.total_withdrawals + self.accrued_fees;\nfuzz_assert_ge!(total_in_vaults, expected, \"Token conservation violated\");" + }, + { + "kind": "text", + "body": "**Solvency invariants** (protocol can meet obligations):" + }, + { + "kind": "code", + "body": "let total_liabilities = self.positions.iter().map(|p| p.owed_amount).sum();\nlet total_assets = self.ctx.token_balance(&vault);\nfuzz_assert_ge!(total_assets, total_liabilities, \"Protocol insolvent\");" + }, + { + "kind": "text", + "body": "**State consistency invariants** (data structures are valid):" + }, + { + "kind": "code", + "body": "for pos in &self.positions {\n fuzz_assert_lt!(pos.tick_lower, pos.tick_upper, \"Invalid tick range\");\n}" + }, + { + "kind": "text", + "body": "---" + } + ] + }, + { + "headers": [ + "Writing Solana/Anchor Fuzz Harnesses" + ], + "blocks": [ + { + "kind": "text", + "body": "When manual byte offset patching becomes unwieldy, use **bytemuck** with `#[repr(C)]` structs for type-safe account access." + }, + { + "kind": "code", + "body": "use bytemuck::{Pod, Zeroable};\n\n#[repr(C)]\n#[derive(Clone, Copy, Pod, Zeroable)]\npub struct ReserveConfig {\n pub status: u8,\n pub _padding1: [u8; 7],\n pub loan_to_value_pct: u8,\n pub liquidation_threshold_pct: u8,\n pub deposit_limit: u64,\n pub borrow_limit: u64,\n // ...\n}" + }, + { + "kind": "text", + "body": "Reading and writing:" + }, + { + "kind": "code", + "body": "let reserve: &Reserve = bytemuck::from_bytes(&account.data[8..8 + RESERVE_SIZE]);\nlet reserve: &mut Reserve = bytemuck::from_bytes_mut(&mut data[8..8 + RESERVE_SIZE]);\nreserve.config.deposit_limit = u64::MAX;" + }, + { + "kind": "text", + "body": "---" + } + ] + }, + { + "headers": [ + "Writing Solana/Anchor Fuzz Harnesses" + ], + "blocks": [ + { + "kind": "text", + "body": "Track per-action success/failure rates to identify blockers:" + }, + { + "kind": "code", + "body": "use std::sync::atomic::{AtomicU32, Ordering};\n\nstruct ActionStats {\n attempts: AtomicU32,\n success: AtomicU32,\n early_return: AtomicU32,\n program_error: AtomicU32,\n}" + }, + { + "kind": "text", + "body": "---" + } + ] + }, + { + "headers": [ + "Writing Solana/Anchor Fuzz Harnesses" + ], + "blocks": [ + { + "kind": "text", + "body": "| Code | Name | Cause | Fix |\n|------|------|-------|-----|\n| 6006 | InvalidAccountInput | Wrong remaining_accounts | Read state to discover required accounts |\n| 6007 | MathOverflow | last_update.slot = 0 | Set to current slot |\n| 6020 | ObligationDepositsEmpty | Borrow without collateral | Early return check for deposits |\n| 6051 | ReserveStale | Reserve not refreshed | Call RefreshReserve or patch last_update |\n| 6087 | CollateralNonLiquidatable | LTV = 0% | Set loan_to_value_pct > 0 |\n| 6090 | DepositLimitExceeded | deposit_limit too low | Set deposit_limit = u64::MAX |\n| 6091 | BorrowLimitExceeded | borrow_limit too low | Set borrow_limit = u64::MAX |\n\n---" + } + ] + }, + { + "headers": [ + "Writing Solana/Anchor Fuzz Harnesses" + ], + "blocks": [ + { + "kind": "text", + "body": "- [ ] Create `types.rs` with bytemuck structs matching program accounts\n- [ ] Set up mock oracles for any price feeds\n- [ ] Configure reserves/pools with reasonable limits (u64::MAX)\n- [ ] Patch freshness/staleness fields (last_update.slot)\n- [ ] Add remaining_accounts logic for instructions that need them\n- [ ] Add early return checks for state prerequisites\n- [ ] Add diagnostic logging to all actions\n- [ ] Run initial fuzzing and analyze error patterns\n- [ ] Iterate on fixes until major actions succeed\n\n---" + } + ] + }, + { + "headers": [ + "Writing Solana/Anchor Fuzz Harnesses" + ], + "blocks": [ + { + "kind": "code", + "body": "1. WRITE/MODIFY -> 2. BUILD -> 3. RUN (3-5s) -> 4. ANALYZE\n ^ |\n +----------------------------------------------------+" + }, + { + "kind": "text", + "body": "| Symptom | Likely Cause | Section |\n|---------|--------------|---------|\n| Setup panics | Initialization order wrong | 5 |\n| Same error every time | Blocker - config/state issue | 11-15 |\n| Coverage stuck at low % | Actions failing silently | 14 |\n| Program panics with enum error | Patching corrupted enum | 12 |\n| Later actions can't find accounts | State not persisted | 14, 16 |" + } + ] + }, + { + "headers": [ + "Using crucible-idl-gen (Standalone Harnesses)" + ], + "blocks": [ + { + "kind": "text", + "body": "For programs using different Solana versions than the fuzzer, `crucible-idl-gen` generates types from IDL without a crate dependency." + } + ] + }, + { + "headers": [ + "Using crucible-idl-gen (Standalone Harnesses)", + "Setup" + ], + "blocks": [ + { + "kind": "text", + "body": "1. **Convert your IDL to JSON format:**" + }, + { + "kind": "code", + "body": " anchor idl convert target/idl/my_program.json -o fuzz/my_fuzz/idls/my_program.json" + }, + { + "kind": "text", + "body": "2. **Add crucible-idl-gen to your fuzz Cargo.toml:**" + }, + { + "kind": "code", + "body": " [dependencies]\n crucible-idl-gen = { git = \"https://github.com/asymmetric-research/crucible\", branch = \"main\" }" + }, + { + "kind": "text", + "body": "3. **Generate types in main.rs:**" + }, + { + "kind": "code", + "body": " crucible_idl_gen::declare_fuzz_program!(\"idls/my_program.json\");\n\n // Or with explicit module name\n crucible_idl_gen::declare_fuzz_program!(my_program = \"idls/my_program.json\");\n\n // Now use generated types\n use my_program::{instruction, accounts, ID};" + } + ] + }, + { + "headers": [ + "Using crucible-idl-gen (Standalone Harnesses)", + "Generated Code" + ], + "blocks": [ + { + "kind": "text", + "body": "The macro generates:\n- `instruction::*` - Instruction structs with `InstructionData` impl\n- `accounts::*` - Account context structs with `ToAccountMetas` impl\n- `state::*` - Account state types for deserialization\n- `types::*` - Custom type definitions\n- `ID` - Program ID constant\n- `register_schemas()` - Register account schemas for semantic field-level diffs in crash output" + } + ] + }, + { + "headers": [ + "Using crucible-idl-gen (Standalone Harnesses)", + "Example Usage" + ], + "blocks": [ + { + "kind": "code", + "body": "crucible_idl_gen::declare_fuzz_program!(\"idls/lending.json\");\n\n#[fuzz_fixture]\nimpl MyFixture {\n pub fn setup() -> Self {\n lending::register_schemas(); // Enable semantic field diffs in crash output\n\n let mut ctx = TestContext::new();\n // ... setup accounts, load program ...\n Self { ctx }\n }\n\n pub fn action_deposit(&mut self, amount: u64) {\n self.ctx.program(lending::ID)\n .call(lending::instruction::Deposit { amount })\n .accounts(lending::accounts::Deposit {\n user: user_pubkey,\n reserve: reserve_pda,\n // ...\n })\n .signers(&[&user])\n .send().ok();\n }\n}" + } + ] + }, + { + "headers": [ + "Remote Fuzzing Integration" + ], + "blocks": [ + { + "kind": "text", + "body": "Crucible integrates with the standard driver protocol used by remote fuzzing platforms, allowing it to run as a managed fuzzing engine. The integration is implemented entirely through the `--mode` CLI flag, which translates the driver's operational modes into equivalent Crucible flags and directory conventions.\n\n---" + } + ] + }, + { + "headers": [ + "Remote Fuzzing Integration", + "Quick Start" + ], + "blocks": [ + { + "kind": "text", + "body": "A minimal remote fuzzing bundle for a Crucible harness needs:\n\n1. The `crucible` binary\n2. The compiled fuzz harness (under `fuzz/<program>/`)\n3. The program `.so` binary\n4. A manifest pointing the standard driver at `crucible run`" + }, + { + "kind": "code", + "body": "{\n \"executable_path_in_bundle\": \"bin/crucible\",\n \"executable_sub_command\": [\"run\", \"myproject\", \"invariant_test\", \"--release\"],\n \"supported_tasks\": [\"explore\", \"reproduce\", \"lineage_cover\", \"corpus_merge\"],\n \"coverage\": {\n \"kind\": \"lcov\",\n \"lcov_config\": {\n \"SourcesPathInBundle\": \"src/\",\n \"SourcesOriginalPath\": \"/build/src/\"\n }\n }\n}" + }, + { + "kind": "text", + "body": "The standard driver appends `--mode <mode>` and `--cores <N>` to the sub-command. Crucible accepts both.\n\n---" + } + ] + }, + { + "headers": [ + "Remote Fuzzing Integration", + "How Invocation Works" + ], + "blocks": [ + { + "kind": "text", + "body": "The driver invokes:" + }, + { + "kind": "code", + "body": "<bundle>/bin/crucible run <program> <test> --release --mode <mode> --cores <N>" + }, + { + "kind": "text", + "body": "Crucible's `--mode` flag translates each mode into native flags before the underlying harness binary is launched. All standard Crucible flags (`--corpus-in`, `--crashes-out`, etc.) can still be passed alongside `--mode` to override defaults.\n\n---" + } + ] + }, + { + "headers": [ + "Remote Fuzzing Integration", + "Operational Modes", + "`dry_run`" + ], + "blocks": [ + { + "kind": "text", + "body": "Validates that the harness can compile, set up, and execute one iteration.\n\n| Aspect | Behavior |\n|--------|----------|\n| Maps to | `--dry-run` |\n| Corpus | None required |\n| Output | None |\n| Exit code | 0 = OK, non-zero = error |" + }, + { + "kind": "code", + "body": "crucible run myproject invariant_test --release --mode dry_run\n# Equivalent to:\ncrucible run myproject invariant_test --release --dry-run" + } + ] + }, + { + "headers": [ + "Remote Fuzzing Integration", + "Operational Modes", + "`explore`" + ], + "blocks": [ + { + "kind": "text", + "body": "Primary fuzzing mode. The harness continuously generates and mutates inputs, looking for invariant violations.\n\n| Aspect | Behavior |\n|--------|----------|\n| Corpus in | `./corpus` (if the directory exists) |\n| Corpus out | `./output` |\n| Crashes out | `./output` |\n| Stop-on-crash | Enabled (exits after first finding) |\n| Structured output | `[FUZZ_PULSE]`, `[FUZZ_FINDING]` on stdout; verbose details on stderr |" + }, + { + "kind": "code", + "body": "crucible run myproject invariant_test --release --mode explore --cores 4\n# Equivalent to:\ncrucible run myproject invariant_test --release \\\n --corpus-in ./corpus --corpus-out ./output --crashes-out ./output \\\n --stop-on-crash -j 4" + }, + { + "kind": "text", + "body": "**Override defaults** by passing flags explicitly:" + }, + { + "kind": "code", + "body": "# Use a custom corpus directory\ncrucible run myproject invariant_test --release --mode explore --corpus-in ./seeds\n\n# Custom crashes directory (overrides ./output for crashes)\ncrucible run myproject invariant_test --release --mode explore --crashes-out ./findings" + }, + { + "kind": "text", + "body": "When a crash is found:\n1. The crash file (binary input) is written to `./output` (or `--crashes-out`)\n2. A `.meta.json` file with the action sequence is written alongside it\n3. `[FUZZ_FINDING] reproduces:true summary:<msg>` is emitted\n4. The process exits" + } + ] + }, + { + "headers": [ + "Remote Fuzzing Integration", + "Operational Modes", + "`reproduce`" + ], + "blocks": [ + { + "kind": "text", + "body": "Attempts to replay a known crash to verify it still triggers.\n\n| Aspect | Behavior |\n|--------|----------|\n| Input | First file found in `./input/` directory |\n| Non-reproduction signal | `did not reproduce` printed to stdout |\n| Account diffs | Auto-enabled for rich replay output |" + }, + { + "kind": "code", + "body": "crucible run myproject invariant_test --release --mode reproduce\n# Equivalent to:\ncrucible run myproject invariant_test --release --replay ./input/<first_file>" + }, + { + "kind": "text", + "body": "**Output protocol:**\n\n- If the crash **reproduces**: `[FUZZ_FINDING] reproduces:true summary:<msg>` on stdout, exit code 1\n- If the crash **does not reproduce**: `[FUZZ_FINDING] reproduces:false summary:did not reproduce` on stdout, plus the literal string `did not reproduce` on stdout. Exit code 0.\n\nThe driver uses the presence of `did not reproduce` (case-sensitive) on stdout to determine reproduction status. The exit code is **not** used for this determination." + } + ] + }, + { + "headers": [ + "Remote Fuzzing Integration", + "Operational Modes", + "`coverage`" + ], + "blocks": [ + { + "kind": "text", + "body": "Replays the corpus once and generates an LCOV coverage report.\n\n| Aspect | Behavior |\n|--------|----------|\n| Maps to | `--coverage --corpus-in ./corpus` |\n| Corpus in | `./corpus` |\n| LCOV output | `./output/coverage.lcov` |\n| Exit code | 0 = success |" + }, + { + "kind": "code", + "body": "crucible run myproject invariant_test --release --mode coverage\n# Equivalent to:\ncrucible run myproject invariant_test --release \\\n --coverage --corpus-in ./corpus --lcov-out ./output/coverage.lcov" + }, + { + "kind": "text", + "body": "The LCOV file is in standard LCOV format. For source-level coverage (real file paths and line numbers rather than bytecode PCs), pass `--symbols <path>` pointing to the unstripped debug binary. See [Coverage Reports](coverage.md) for details." + } + ] + }, + { + "headers": [ + "Remote Fuzzing Integration", + "Operational Modes", + "`corpus_merge`" + ], + "blocks": [ + { + "kind": "text", + "body": "Merges and deduplicates the corpus, writing the effective (minimized) subset to the output directory.\n\n| Aspect | Behavior |\n|--------|----------|\n| Corpus in | `./corpus` |\n| Output | `./output` (merged corpus files) |\n| Method | Greedy set-cover: replays all inputs, keeps minimum set preserving all coverage |\n| Exit code | 0 = success |" + }, + { + "kind": "code", + "body": "crucible run myproject invariant_test --release --mode corpus_merge\n# Equivalent to:\ncrucible run myproject invariant_test --release \\\n --corpus-in ./corpus --corpus-out ./output\n# (internally runs the cmin greedy set-cover algorithm)" + }, + { + "kind": "text", + "body": "The driver checks the number of files in `./output` after execution. Fewer files than in `./corpus` indicates effective merging.\n\n---" + } + ] + }, + { + "headers": [ + "Remote Fuzzing Integration", + "Directory Layout" + ], + "blocks": [ + { + "kind": "text", + "body": "When running under a remote driver, the working directory is structured by the driver:" + }, + { + "kind": "code", + "body": "<workdir>/\n├── corpus/ # Input corpus (explore, coverage, corpus_merge)\n├── input/ # Single crash file (reproduce mode)\n└── output/ # All output: corpus, crashes, coverage.lcov" + }, + { + "kind": "text", + "body": "Crucible maps these directories to its native `--corpus-in`, `--corpus-out`, `--crashes-out`, and `--lcov-out` flags.\n\n---" + } + ] + }, + { + "headers": [ + "Remote Fuzzing Integration", + "Structured Output Protocol" + ], + "blocks": [ + { + "kind": "text", + "body": "Crucible emits structured output that the remote driver parses for status tracking and finding detection." + } + ] + }, + { + "headers": [ + "Remote Fuzzing Integration", + "Structured Output Protocol", + "`[FUZZ_PULSE]` — Progress Updates" + ], + "blocks": [ + { + "kind": "text", + "body": "Emitted periodically during `explore` mode (approximately once per second)." + }, + { + "kind": "code", + "body": "[FUZZ_PULSE] execs/s:<uint64> corpus_count:<uint64> coverage:<uint64> memory_kib:<uint64>" + }, + { + "kind": "text", + "body": "All fields are optional per the protocol. Crucible typically emits a superset:" + }, + { + "kind": "code", + "body": "[FUZZ_PULSE] run time: 30s, exec/sec: 1234, corpus: 567, crashes: 0, edges: 1500/65536 (2.3%), memory_kib: 204800" + }, + { + "kind": "text", + "body": "The driver extracts `execs/s`, `corpus_count`, `coverage`, and `memory_kib` from any `[FUZZ_PULSE]` line using key-value parsing.\n\nCrucible emits two pulse lines per update:\n- **stdout**: spec-compliant `[FUZZ_PULSE] execs/s:N corpus_count:N coverage:N memory_kib:N` (for the driver)\n- **stderr**: verbose human-readable line with edges, branches, action stats, etc. (captured as artifact)" + } + ] + }, + { + "headers": [ + "Remote Fuzzing Integration", + "Structured Output Protocol", + "`[FUZZ_FINDING]` — Crash/Violation Discovered" + ], + "blocks": [ + { + "kind": "text", + "body": "Emitted when an invariant violation is detected." + }, + { + "kind": "code", + "body": "[FUZZ_FINDING] reproduces:true summary:<string>" + }, + { + "kind": "text", + "body": "In `explore` mode, this is followed by the crash file being written to the output directory. The process then exits (stop-on-crash is enforced).\n\nIn `reproduce` mode, findings go to stdout:" + }, + { + "kind": "code", + "body": "[FUZZ_FINDING] reproduces:true summary:Invariant violated: total_supply exceeded cap\n[FUZZ_FINDING] reproduces:false summary:did not reproduce" + } + ] + }, + { + "headers": [ + "Remote Fuzzing Integration", + "Structured Output Protocol", + "`[FUZZ_ERROR]` — Fatal Error" + ], + "blocks": [ + { + "kind": "text", + "body": "Emitted when the harness cannot continue (e.g., missing input file, configuration error)." + }, + { + "kind": "code", + "body": "[FUZZ_ERROR] <error_message>" + }, + { + "kind": "text", + "body": "The driver treats any `[FUZZ_ERROR]` as a fatal failure." + } + ] + }, + { + "headers": [ + "Remote Fuzzing Integration", + "Structured Output Protocol", + "`did not reproduce`" + ], + "blocks": [ + { + "kind": "text", + "body": "In `reproduce` mode, if the crash input does not trigger a violation, the literal string `did not reproduce` (case-sensitive) is printed to stdout. This is the driver's detection mechanism — the exit code is not used.\n\n---" + } + ] + }, + { + "headers": [ + "Remote Fuzzing Integration", + "Manifest Configuration Reference" + ], + "blocks": [ + { + "kind": "text", + "body": "The standard driver manifest is a JSON object:" + }, + { + "kind": "code", + "body": "{\n \"executable_path_in_bundle\": \"bin/crucible\",\n \"executable_sub_command\": [\"run\", \"myproject\", \"invariant_test\", \"--release\"],\n \"supported_tasks\": [\"explore\", \"reproduce\", \"lineage_cover\", \"corpus_merge\"],\n \"extra\": {\n \"FUZZ_VERBOSE\": \"1\",\n \"RUST_LOG\": \"info\"\n },\n \"coverage\": {\n \"kind\": \"lcov\",\n \"lcov_config\": {\n \"SourcesPathInBundle\": \"programs/myproject/src/\",\n \"SourcesOriginalPath\": \"/home/builder/programs/myproject/src/\"\n }\n }\n}" + } + ] + }, + { + "headers": [ + "Remote Fuzzing Integration", + "Manifest Configuration Reference", + "Fields" + ], + "blocks": [ + { + "kind": "text", + "body": "| Field | Required | Description |\n|-------|----------|-------------|\n| `executable_path_in_bundle` | Yes | Relative path to the `crucible` binary in the bundle |\n| `executable_sub_command` | Yes | Arguments passed before `--mode` and `--cores` |\n| `supported_tasks` | Yes | Which modes the harness supports |\n| `extra` | No | Environment variables set before invocation |\n| `coverage.kind` | If coverage | `\"lcov\"` for Crucible |\n| `coverage.lcov_config.SourcesPathInBundle` | If coverage | Path to sources in the bundle (for LCOV path rewriting) |\n| `coverage.lcov_config.SourcesOriginalPath` | If coverage | Original source path on the build machine |" + } + ] + }, + { + "headers": [ + "Remote Fuzzing Integration", + "Manifest Configuration Reference", + "Supported Tasks" + ], + "blocks": [ + { + "kind": "text", + "body": "| Task | Driver Name | Crucible Mode |\n|------|---------------|---------------|\n| Explore | `explore` | `--mode explore` |\n| Reproduce | `reproduce` | `--mode reproduce` |\n| Coverage | `lineage_cover` | `--mode coverage` |\n| Corpus merge | `corpus_merge` | `--mode corpus_merge` |\n\nNote: `lineage_cover` in `supported_tasks` maps to `--mode coverage`. The task name differs from the mode name." + } + ] + }, + { + "headers": [ + "Remote Fuzzing Integration", + "Manifest Configuration Reference", + "Environment Variables via `extra`" + ], + "blocks": [ + { + "kind": "text", + "body": "Any key-value pairs in `extra` are set as environment variables. Useful Crucible variables:\n\n| Variable | Description |\n|----------|-------------|\n| `FUZZ_VERBOSE=1` | Verbose harness output (action stats, state snapshots) |\n| `FUZZ_STATS_CSV=stats.csv` | Write per-second fuzzing stats to CSV |\n\n---" + } + ] + }, + { + "headers": [ + "Remote Fuzzing Integration", + "Building a Remote Fuzzing Bundle" + ], + "blocks": [ + { + "kind": "text", + "body": "A typical bundle layout:" + }, + { + "kind": "code", + "body": "bundle/\n├── bin/\n│ └── crucible # CLI binary\n├── fuzz/\n│ └── myproject/\n│ ├── Cargo.toml\n│ ├── src/main.rs # Harness code\n│ ├── idls/myproject.json # Program IDL\n│ └── target/release/invariant_test # Compiled harness binary\n├── target/\n│ └── deploy/myproject.so # Program binary (loaded by harness)\n├── programs/\n│ └── myproject/src/ # Source files (for LCOV path mapping)\n└── manifest.json # Standard driver config" + } + ] + }, + { + "headers": [ + "Remote Fuzzing Integration", + "Building a Remote Fuzzing Bundle", + "Build Steps" + ], + "blocks": [ + { + "kind": "code", + "body": "# 1. Build the program binary\ncd /path/to/program\ncargo build-sbf\n\n# 2. Build the fuzz harness\ncd fuzz/myproject\ncargo build --release --features invariant_test\n\n# 3. Build the crucible CLI\ncd /path/to/crucible\ncargo build --release -p crucible-fuzz-cli\n\n# 4. Assemble the bundle\nmkdir -p bundle/bin bundle/fuzz/myproject bundle/target/deploy bundle/programs/myproject/src\ncp target/release/crucible bundle/bin/\ncp -r fuzz/myproject/ bundle/fuzz/myproject/\ncp target/deploy/myproject.so bundle/target/deploy/\ncp -r programs/myproject/src/ bundle/programs/myproject/src/" + }, + { + "kind": "text", + "body": "---" + } + ] + }, + { + "headers": [ + "Remote Fuzzing Integration", + "CLI Flags Accepted from Driver" + ], + "blocks": [ + { + "kind": "text", + "body": "The standard driver appends these flags to every invocation:\n\n| Flag | Source | How Crucible Handles It |\n|------|--------|------------------------|\n| `--mode <mode>` | Driver | Translated into native Crucible flags (see mode table above) |\n| `--cores <N>` | Driver | Passed to `-j N` for multi-core fuzzing |\n\nBoth are defined as clap arguments on `crucible run` and will not cause parse errors.\n\n---" + } + ] + }, + { + "headers": [ + "Remote Fuzzing Integration", + "Exit Codes" + ], + "blocks": [ + { + "kind": "text", + "body": "| Mode | Exit 0 | Non-zero |\n|------|--------|----------|\n| `dry_run` | Setup OK | Setup failed |\n| `explore` | Crash found and written | Build/runtime error |\n| `reproduce` | Did not reproduce (check stdout for `did not reproduce`) | Crash reproduced, or input read error |\n| `coverage` | LCOV written | Corpus missing or build error |\n| `corpus_merge` | Merged corpus written | Corpus missing or build error |\n\nNote: In `reproduce` mode, the exit code is **not** used to determine reproduction status. The driver checks for the string `did not reproduce` on stdout. Exit 0 = did not reproduce, exit 1 = crash reproduced (the inner harness returns 1 on reproduction, and the CLI passes it through without treating it as an error).\n\n---" + } + ] + }, + { + "headers": [ + "Remote Fuzzing Integration", + "Differences from Standalone Usage" + ], + "blocks": [ + { + "kind": "text", + "body": "When running under a remote driver (`--mode` set), Crucible behaves slightly differently:\n\n| Behavior | Standalone | Remote |\n|----------|-----------|----------|\n| Corpus directory | User-specified or `fuzz/<prog>/corpus/` | `./corpus` (CWD-relative) |\n| Crash directory | `fuzz/<prog>/crashes/<test>/` | `./output` |\n| Coverage output | `./coverage/coverage.lcov` | `./output/coverage.lcov` |\n| Stop-on-crash | Off by default | Always on in `explore` |\n| Account diffs on replay | Auto-enabled | Auto-enabled |\n\nExplicit flags always override mode defaults (e.g., `--mode explore --crashes-out ./custom` uses `./custom` instead of `./output`)." + } + ] + }, + { + "headers": [ + "Writing Tests", + "Fixture Requirements" + ], + "blocks": [ + { + "kind": "code", + "body": "#[derive(Clone)] // Required - fixture is cloned each iteration\nstruct MyFixture {\n ctx: TestContext,\n program_id: Pubkey,\n user: Rc<Keypair>, // Keypairs wrapped in Rc<>\n pool_pda: Pubkey,\n}\n\n#[fuzz_fixture]\nimpl MyFixture {\n pub fn setup() -> Self {\n let mut ctx = TestContext::new();\n // ... initialization ...\n Self { ctx, /* fields */ }\n }\n}" + }, + { + "kind": "text", + "body": "---" + } + ] + }, + { + "headers": [ + "Writing Tests", + "Action Naming Convention" + ], + "blocks": [ + { + "kind": "text", + "body": "Actions must be prefixed with `action_` for auto-discovery:" + }, + { + "kind": "code", + "body": "#[fuzz_fixture]\nimpl MyFixture {\n pub fn setup() -> Self { /* ... */ }\n\n pub fn action_stake(&mut self, amount: u64) { }\n pub fn action_withdraw(&mut self, #[range(0..3)] user_idx: usize) { }\n\n pub fn helper_method(&self) { } // Not discovered\n}" + } + ] + }, + { + "headers": [ + "Writing Tests", + "Action Naming Convention", + "Action Return Types" + ], + "blocks": [ + { + "kind": "code", + "body": "// Implicit success\npub fn action_advance_time(&mut self, slots: u64) {\n self.ctx.advance_slots(slots);\n}\n\n// Explicit success/failure tracking\npub fn action_deposit(&mut self, amount: u64) -> Result<()> {\n self.ctx.program(self.program_id)\n .call(instruction::Deposit { amount })\n .accounts(accounts::Deposit { /* ... */ })\n .signers(&[&*self.user])\n .send()?\n .into_result()\n}" + }, + { + "kind": "text", + "body": "For non-Anchor programs or hand-rolled instructions, use `ctx.raw_call(instruction)` instead of the typed `ctx.program(..).call(..)` builder. See [API Reference › Raw Instruction Calls](api-reference.md#raw-instruction-calls)." + } + ] + }, + { + "headers": [ + "Writing Tests", + "Action Naming Convention", + "After-Action Callback" + ], + "blocks": [ + { + "kind": "code", + "body": "#[fuzz_fixture]\nimpl MyFixture {\n pub fn setup() -> Self { /* ... */ }\n pub fn action_deposit(&mut self, amount: u64) { /* ... */ }\n\n // Optional: called after EVERY action dispatch\n pub fn after_action(&self) {\n let pool = self.ctx.read_anchor_account::<Pool>(&self.pool_pda).ok();\n if let Some(pool) = pool {\n eprintln!(\"[STATS] total_deposited: {}\", pool.total_deposited);\n }\n }\n}" + }, + { + "kind": "text", + "body": "---" + } + ] + }, + { + "headers": [ + "Writing Tests", + "Range Constraints" + ], + "blocks": [ + { + "kind": "text", + "body": "Bound fuzz inputs using `#[range(start..end)]`:" + }, + { + "kind": "code", + "body": "pub fn action_stake(\n &mut self,\n #[range(0..3)] user_idx: usize, // 0, 1, or 2\n #[range(0..1_000_000)] amount: u64,\n) { }" + }, + { + "kind": "text", + "body": "---" + } + ] + }, + { + "headers": [ + "Writing Tests", + "Simple Fuzzing (`#[crucible_fuzz]`)" + ], + "blocks": [ + { + "kind": "text", + "body": "For testing individual operations with random inputs:" + }, + { + "kind": "code", + "body": "#[crucible_fuzz]\nfn fuzz_stake(fixture: &mut StakingFixture, #[range(0..100_000)] amount: u64) {\n fixture.action_stake(0, amount);\n let user = fixture.ctx.read_anchor_account::<User>(&fixture.user_pda).unwrap();\n fuzz_assert_le!(user.staked, INITIAL_BALANCE);\n}" + }, + { + "kind": "text", + "body": "---" + } + ] + }, + { + "headers": [ + "Writing Tests", + "Invariant Fuzzing (`#[invariant_test]` + `#[fuzz_fixture]`)" + ], + "blocks": [ + { + "kind": "text", + "body": "For testing complex interactions with random action sequences:" + }, + { + "kind": "code", + "body": "#[fuzz_fixture]\nimpl StakingFixture {\n pub fn setup() -> Self { /* ... */ }\n pub fn action_stake(&mut self, #[range(0..3)] user: usize, amount: u64) { }\n pub fn action_unstake(&mut self, #[range(0..3)] user: usize, amount: u64) { }\n pub fn action_claim(&mut self, #[range(0..3)] user: usize) { }\n pub fn action_advance_time(&mut self, #[range(0..10000)] slots: u64) {\n self.ctx.warp_to_slot(self.ctx.slot() + slots);\n }\n}\n\n#[invariant_test]\nfn invariant_fuzz(fixture: &mut StakingFixture) {\n // Runs AFTER EACH action in the sequence\n let pool = fixture.ctx.read_anchor_account::<Pool>(&fixture.pool_pda).unwrap();\n fuzz_assert_le!(pool.total_staked, fixture.total_deposited);\n}" + }, + { + "kind": "text", + "body": "---" + } + ] + }, + { + "headers": [ + "Writing Tests", + "Assertion Macros" + ], + "blocks": [ + { + "kind": "text", + "body": "Use `fuzz_assert_*` macros instead of `assert!` in invariant checks:\n\n| Macro | Description |\n|-------|-------------|\n| `fuzz_assert!(cond)` | Assert condition is true |\n| `fuzz_assert_eq!(a, b)` | Assert `a == b` |\n| `fuzz_assert_ne!(a, b)` | Assert `a != b` |\n| `fuzz_assert_lt!(a, b)` | Assert `a < b` |\n| `fuzz_assert_le!(a, b)` | Assert `a <= b` |\n| `fuzz_assert_gt!(a, b)` | Assert `a > b` |\n| `fuzz_assert_ge!(a, b)` | Assert `a >= b` |\n| `fuzz_assert_approx_eq!(a, b, delta)` | Assert `\\|a - b\\| <= delta` |\n\nStandard `assert!` panics crash the entire fuzzer process. The `fuzz_assert_*` macros record violations and report them as crashes to LibAFL without killing the process." + } + ] + }, + { + "headers": [ + "Readme" + ], + "blocks": [ + { + "kind": "text", + "body": "<p align=\"center\">\n <img src=\"assets/crucible_logo.png\" alt=\"Crucible\" width=\"400\">\n</p>\n\n<p align=\"center\">\n <strong>Coverage-guided fuzzing framework for Solana smart contracts</strong>\n</p>\n\n<p align=\"center\">\n Built on <a href=\"https://github.com/AFLplusplus/LibAFL\">LibAFL</a> and <a href=\"https://github.com/LiteSVM/litesvm\">LiteSVM</a> for fast, local transaction simulation with edge-level coverage tracking.\n</p>\n\n---\n\nCrucible enables property-based testing and stateful invariant checking for Solana programs through randomly generated action sequences. Define your program's actions, write invariants, and let the fuzzer find violations." + } + ] + }, + { + "headers": [ + "Readme", + "Performance" + ], + "blocks": [ + { + "kind": "text", + "body": "Upperbound throughput on small/medium programs with MacBook Pro M3:\n\n| Mode | 1 core | 12 cores | 12 cores, no tracing |\n|-----------|-----------|------------|----------------------|\n| Stateless | 1,200/s | 9,600/s | 22,000/s |\n| Stateful | 8,200/s | 67,000/s | 130,000/s |" + } + ] + }, + { + "headers": [ + "Readme", + "Features" + ], + "blocks": [ + { + "kind": "text", + "body": "- **Works on any compiled Solana program.** Coverage comes from sBPF edge tracing on the LiteSVM execution trace, so no source-side instrumentation or program rebuild is required. Standard Anchor/Codama/Shank IDL is sufficient to generate typed call bindings at compile time. See [IDL Code Generation](docs/idl-gen.md). When no IDL available, calling can still be done with `raw_call()`\n- **Concise harness API.** TestContext collapses account construction, signing, instruction encoding, and result parsing into one builder chain: `ctx.program(...).call(...).accounts(...).signers(...).send()`. Helpers cover time control (`advance_epoch`, `warp_to_slot`), account creation, and mint setup. Reduces testing boilerplate by 50-70%. See [API Reference](docs/api-reference.md). \n- **Typed-action mutator.** Mutations rewrite typed `Vec<Action>` entries directly. Sequence operations (insert, delete, swap, splice) and parameter rewrites (numeric values biased toward boundaries and declared `#[range(..)]` endpoints) keep every generated input structurally valid. Roughly a 5x improvement in bug discovery rate over `arbitrary`. \n- **Stateless mode.** Each iteration clones the post-setup snapshot and executes a full mutated sequence against it. Coverage feedback is per-sequence: sequences that reach a new edge survive into the corpus. \n- **Stateful mode.** Keeps a coverage-indexed pool of live program states and applies one mutated action at a time, picking up where prior iterations left off. Roughly an order-of-magnitude throughput gain over stateless, at the cost of higher memory use as the pool grows with coverage.\n- **Crash triage built in.** Findings are minimized with `crucible tmin` and replayed with `crucible show`. Programs compiled with debug symbols produce LCOV reports viewable with `genhtml`. See [Crash Analysis](docs/crash-analysis.md) and [Coverage Reports](docs/coverage.md).\n\n---" + } + ] + }, + { + "headers": [ + "Readme", + "Quick Start", + "Install" + ], + "blocks": [ + { + "kind": "text", + "body": "From source:" + }, + { + "kind": "code", + "body": "git clone https://github.com/asymmetric-research/crucible\ncd crucible\ncargo install --path crates/crucible-fuzz-cli" + } + ] + }, + { + "headers": [ + "Readme", + "Quick Start", + "Initialize a fuzz harness" + ], + "blocks": [ + { + "kind": "code", + "body": "crucible init <program_name>" + }, + { + "kind": "text", + "body": "Defaults to `fuzz/<program_name>/`" + } + ] + }, + { + "headers": [ + "Readme", + "Quick Start", + "Run a fuzz test" + ], + "blocks": [ + { + "kind": "code", + "body": "crucible run <program_name> <test_name> --release" + }, + { + "kind": "text", + "body": "---" + } + ] + }, + { + "headers": [ + "Readme", + "Documentation" + ], + "blocks": [ + { + "kind": "text", + "body": "| Topic | Description |\n|-------|-------------|\n| [Getting Started](docs/getting-started.md) | Setup, running, project structure, feature flags |\n| [IDL Code Generation](docs/idl-gen.md) | Using `crucible-idl-gen` for standalone harnesses |\n| [API Reference](docs/api-reference.md) | TestContext API — program loading, accounts, transactions, RPC cloning, time, oracles |\n| [Writing Tests](docs/writing-tests.md) | Fixtures, actions, range constraints, simple & invariant fuzzing, assertion macros |\n| [CLI Reference](docs/cli-reference.md) | All `crucible` commands and execution modes |\n| [Crash Analysis](docs/crash-analysis.md) | Listing, viewing, replaying, and minimizing crashes |\n| [Coverage Reports](docs/coverage.md) | Bytecode & source-level coverage, LCOV, genhtml |\n| [Harness Guide](docs/harness-guide.md) | In-depth guide to writing effective fuzz harnesses |\n\n---" + } + ] + }, + { + "headers": [ + "Readme", + "License" + ], + "blocks": [ + { + "kind": "text", + "body": "Licensed under the [MIT License](LICENSE)." + } + ] + } + ] +} \ No newline at end of file diff --git a/rust/crucible-app/pyproject.toml b/rust/crucible-app/pyproject.toml new file mode 100644 index 00000000..4b89eaed --- /dev/null +++ b/rust/crucible-app/pyproject.toml @@ -0,0 +1,16 @@ +[build-system] +requires = ["maturin>=1.5,<2.0"] +build-backend = "maturin" + +[project] +name = "crucible_app" +version = "0.1.0" +description = "AutoProver Solana verification application (Crucible fuzzing backend)." +requires-python = ">=3.12" +classifiers = ["Programming Language :: Rust"] + +[tool.maturin] +# The compiled module is imported as `crucible_app` (matches `export_app!(crucible_app, …)`). +module-name = "crucible_app" +# abi3 wheel — one artifact for cp312+. +features = ["pyo3/extension-module"] diff --git a/rust/crucible-app/src/lib.rs b/rust/crucible-app/src/lib.rs new file mode 100644 index 00000000..4dcc3c97 --- /dev/null +++ b/rust/crucible-app/src/lib.rs @@ -0,0 +1,1096 @@ +//! The **Crucible** application — AutoProver's Solana verification backend, which +//! authors [Crucible](https://github.com/asymmetric-research/crucible) fuzz harnesses +//! and gates them with the local `crucible` CLI. Pairs with the shared `solana` +//! ecosystem front half (see `docs/crucible-application.md`). +//! +//! A passive [`Backend`] (`docs/rust-backend-api.md`): it supplies the descriptor, +//! toolchain precondition checks, the per-invariant `units`, the authoring prompts +//! (fixture + tests), and the two gating callouts — `compile` (a `crucible … --dry-run` +//! build) and `validate` (one `crucible … --mode explore` fuzz run per unit) — which run +//! the toolchain through the shared `run_confined` launcher. Python owns the loop. + +use std::collections::BTreeMap; +use std::path::{Path, PathBuf}; + +use autoprover_sdk::{ + run_confined, AppDescriptor, ArgDefault, ArgSpec, ArtifactLayout, AuthorInput, Backend, + CommandOutput, CompileResult, CoreSlot, DeliverableMode, EventKind, Failure, FailureKind, + PhaseSpec, Prompt, Sandbox, SandboxGrants, SetupSpec, Unit, ValidateOutcome, Verdict, + WorkspacePrep, +}; + +use askama::Template; + +// The crucible/solana/anchor stack a harness pins (docs/crucible-application.md §6.1). Hardcoded +// for now to the combination the installed toolchain matches (was Python's `CrucibleHarness`). +const ANCHOR_VERSION: &str = "1.0.1"; +const SOLANA_VERSION: &str = "3.0"; +const LIBAFL_VERSION: &str = "0.15.1"; + +/// The single `#[invariant_test]` fn that holds ALL of a program's invariants — one harness, +/// one build, one fuzz run (the Crucible macro self-gates `main()` by fn name == feature, so this +/// is also the feature/target selector). See docs/crucible-unit-granularity.md §3. +const SINGLE_HARNESS_FN: &str = "c_invariants"; + +// --- askama templates --------------------------------------------------------------------- +// Each struct binds a `.j2` file under `templates/` (the same convention as composer/templates/ +// *.j2, here for the Rust side). They replace the former inline string consts and `format!` +// literals: `render()` fills the holes. `escape = "none"` because these render prompts and +// Rust/TOML source — NOT HTML — so no entity escaping. Whitespace is preserved (see askama.toml). + +/// Backend-guidance prose injected into the property-extraction prompt. Crucible is a fuzzer, +/// so — like Foundry — refutations are valuable but universals can't be *proven*. +#[derive(Template)] +#[template(path = "backend_guidance.j2", escape = "none")] +struct BackendGuidance; + +/// Concise Crucible harness API reference for the fixture-authoring prompt (§7.5 cheat-sheet). +#[derive(Template)] +#[template(path = "harness_cheat_sheet.j2", escape = "none")] +struct HarnessCheatSheet<'a> { + program: &'a str, +} + +/// A complete, compiling worked example (a different `escrow` program) to pattern-match. +#[derive(Template)] +#[template(path = "example_fixture.j2", escape = "none")] +struct ExampleFixture; + +/// Cheat-sheet for authoring the single `#[invariant_test]` fn holding all invariants. +#[derive(Template)] +#[template(path = "test_cheat_sheet.j2", escape = "none")] +struct TestCheatSheet; + +/// Reviewer persona for the `judge_prompt` turn (peer of Foundry's judge system prompt). +#[derive(Template)] +#[template(path = "judge_system.j2", escape = "none")] +struct JudgeSystem; + +/// A `#[invariant_test]` probe appended by the host to validate the fixture via `--dry-run`. +#[derive(Template)] +#[template(path = "probe_fn.j2", escape = "none")] +struct ProbeFn; + +/// The pinned `[dependencies]` block for the harness crate. +#[derive(Template)] +#[template(path = "cargo_deps.j2", escape = "none")] +struct CargoDeps<'a> { + cf: &'a str, + ctc: &'a str, + program: &'a str, + anchor_version: &'a str, + libafl_version: &'a str, + solana_version: &'a str, +} + +/// The harness `Cargo.toml` skeleton (`deps` + `feats` are pre-rendered strings). +#[derive(Template)] +#[template(path = "cargo_toml.j2", escape = "none")] +struct CargoToml<'a> { + program: &'a str, + deps: &'a str, + feats: &'a str, +} + +/// Re-author suffix after a failed build / dry-run (leads with extracted compiler errors). +#[derive(Template)] +#[template(path = "revise_compile.j2", escape = "none")] +struct ReviseCompile<'a> { + errors: &'a str, + prev_src: &'a str, + tail: &'a str, +} + +/// Re-author suffix after a security reviewer rejected a compiling suite. +#[derive(Template)] +#[template(path = "revise_judge.j2", escape = "none")] +struct ReviseJudge<'a> { + feedback: &'a str, + prev_src: &'a str, +} + +/// The fixture-authoring prompt (the `setup` phase). +#[derive(Template)] +#[template(path = "author_setup.j2", escape = "none")] +struct AuthorSetup<'a> { + program: &'a str, + cheat: &'a str, + example: &'a str, + facts: &'a str, + model: &'a str, + revise: &'a str, +} + +/// The invariant-suite authoring prompt (per component). +#[derive(Template)] +#[template(path = "author_component.j2", escape = "none")] +struct AuthorComponent<'a> { + harness_fn: &'a str, + program: &'a str, + n: usize, + first: &'a str, + listed: &'a str, + component: &'a str, + cheat: &'a str, + fixture: &'a str, + revise: &'a str, +} + +/// The judge instruction (embeds `judge_guidance.j2` via `{% include %}`). +#[derive(Template)] +#[template(path = "judge_instruction.j2", escape = "none")] +struct JudgeInstruction<'a> { + program: &'a str, + harness_fn: &'a str, + listed: &'a str, + component: &'a str, + fixture: &'a str, + spec: &'a str, +} + +/// One instruction's mined Anchor facts (a row in `api_facts.j2`). +struct IxFact { + name: String, + pascal: String, + args: Vec<String>, + accounts: Vec<String>, +} + +/// The "API facts" block mined from the analyzed model (crate id, ids, types, instructions). +#[derive(Template)] +#[template(path = "api_facts.j2", escape = "none")] +struct ApiFacts<'a> { + crate_id: &'a str, + analysis_id: Option<&'a str>, + program_id: String, + account_types: Vec<String>, + instructions: Vec<IxFact>, +} + +/// The crucible checkout that resolves the harness crate's path deps (`$CRUCIBLE_REPO`). Read +/// here so crate rendering is fully wheel-owned; `validate_preconditions` guarantees it is set. +fn crucible_repo() -> Option<PathBuf> { + std::env::var("CRUCIBLE_REPO").ok().map(PathBuf::from) +} + +/// The `[dependencies]` block for the harness crate — the pinned crucible/solana/anchor stack +/// plus the program-under-test as a path dep (was `CrucibleDep::render_deps`). +fn crucible_deps(program: &str, repo: &Path) -> String { + let crates = repo.join("crates"); + let cf = crates.join("crucible-fuzzer").display().to_string(); + let ctc = crates.join("crucible-test-context").display().to_string(); + CargoDeps { + cf: &cf, + ctc: &ctc, + program, + anchor_version: ANCHOR_VERSION, + libafl_version: LIBAFL_VERSION, + solana_version: SOLANA_VERSION, + } + .render() + .expect("render cargo_deps") +} + +/// The harness `Cargo.toml`: one `[[bin]]` (`invariant_test`) selected by a per-component Cargo +/// feature. `features` are inert (`f = []`) — Crucible's macro self-gates `main()` by fn name == +/// feature — so a build only needs the feature it selects declared (was `CrucibleHarness`). +fn render_cargo_toml(program: &str, repo: &Path, features: &[String]) -> String { + let feats = if features.is_empty() { + "# (no components yet)".to_string() + } else { + features.iter().map(|f| format!("{f} = []")).collect::<Vec<_>>().join("\n") + }; + let deps = crucible_deps(program, repo); + CargoToml { program, deps: &deps, feats: &feats } + .render() + .expect("render cargo_toml") +} + +/// The crate's on-disk files for a confined build: `src/main.rs` plus a `Cargo.toml` declaring +/// exactly `features` (materialized per run — with `serialize_toolchain` there is no concurrent +/// writer, so no shared-manifest race and no cumulative feature reservation). +fn crate_files(program: &str, main_rs: String, features: &[String]) -> BTreeMap<String, String> { + let mut files = BTreeMap::new(); + files.insert(format!("fuzz/{program}/src/main.rs"), main_rs); + if let Some(repo) = crucible_repo() { + files.insert( + format!("fuzz/{program}/Cargo.toml"), + render_cargo_toml(program, &repo, features), + ); + } + files +} + +/// The directory of `bin` on `$PATH` (for a read-only sandbox grant), if found. +fn which_dir(bin: &str) -> Option<String> { + let path = std::env::var("PATH").ok()?; + std::env::split_paths(&path) + .find(|dir| dir.join(bin).is_file()) + .map(|dir| dir.display().to_string()) +} + +/// The compiled binaries a Crucible run needs on `PATH`. Checked up-front so a run +/// fails fast with an actionable message rather than deep in the build phase. +const REQUIRED_BINARIES: &[&str] = &["crucible", "cargo-build-sbf", "anchor"]; + +/// Is `bin` an executable file reachable via `$PATH`? A pure filesystem scan — we do +/// not *run* anything here (validate_preconditions must stay a cheap, sync check). +fn on_path(bin: &str) -> bool { + let Ok(path) = std::env::var("PATH") else { + return false; + }; + std::env::split_paths(&path).any(|dir| dir.join(bin).is_file()) +} + + +/// Extract just the rustc error diagnostics from a (possibly long) cargo build log so +/// the revise prompt leads with the actual errors instead of pages of "Compiling …". +/// Keeps each `error[..]`/`error:` block with its `-->`/`|`/`=` context; drops warnings +/// and progress. Returns "" if there are no error lines. +fn compiler_diagnostics(out: &str) -> String { + let mut kept: Vec<&str> = Vec::new(); + let mut in_err = false; + for line in out.lines() { + let t = line.trim_start(); + if t.starts_with("error[") || t.starts_with("error:") { + in_err = true; + kept.push(line); + } else if in_err { + if line.is_empty() + || line.starts_with(' ') + || t.starts_with("-->") + || t.starts_with('|') + || t.starts_with('=') + { + kept.push(line); + } else { + in_err = false; + } + } + } + while kept.last().is_some_and(|l| l.trim().is_empty()) { + kept.pop(); + } + let joined = kept.join("\n"); + // Cap so a pathological error count can't blow up the prompt. + joined[..joined.len().min(4000)].to_string() +} + +/// snake_case → PascalCase — Anchor's `instruction`/`accounts` struct naming. +fn to_pascal(snake: &str) -> String { + snake + .split('_') + .filter(|s| !s.is_empty()) + .map(|w| { + let mut c = w.chars(); + match c.next() { + Some(f) => f.to_uppercase().collect::<String>() + c.as_str(), + None => String::new(), + } + }) + .collect() +} + +/// A concise, high-signal "API facts" block mined from the analyzed model so the author +/// need not dig through the full JSON (or rediscover Anchor names by exploring): the crate +/// id, declare_id, state types, and each instruction's snake→Pascal name + args + accounts. +/// Returns "" if the model shape isn't recognized. +fn api_facts(analyzed: &serde_json::Value, program: &str) -> String { + let components = match analyzed.get("components").and_then(|c| c.as_array()) { + Some(c) => c, + None => return String::new(), + }; + let is_prog = |c: &&serde_json::Value| c.get("instructions").is_some_and(|i| i.is_array()); + let prog = components + .iter() + .find(|c| { + is_prog(c) + && (c.get("program_identifier").and_then(|v| v.as_str()) == Some(program) + || c.get("name").and_then(|v| v.as_str()) == Some(program)) + }) + .or_else(|| components.iter().find(is_prog)); + let prog = match prog { + Some(p) => p, + None => return String::new(), + }; + + let str_of = |v: Option<&serde_json::Value>| v.and_then(|x| x.as_str()).unwrap_or("?").to_string(); + // The crate id is the harness's actual Cargo dependency name — the program name + // (== CrucibleDep's crate), NOT the analysis's `program_identifier`, which may be the + // `#[program] pub mod` name and would mis-resolve `use <id>::*`. Surface the module name as + // a note only when it differs (the template renders it iff `analysis_id` is `Some`). + let analysis_raw = str_of(prog.get("program_identifier")); + let analysis_id: Option<String> = + (analysis_raw != program && analysis_raw != "?").then_some(analysis_raw); + let program_id = prog + .get("program_id") + .and_then(|v| v.as_str()) + .unwrap_or("(not declared)") + .to_string(); + let account_types: Vec<String> = prog + .get("account_types") + .and_then(|v| v.as_array()) + .map(|types| types.iter().filter_map(|t| t.as_str().map(String::from)).collect()) + .unwrap_or_default(); + let instructions: Vec<IxFact> = prog + .get("instructions") + .and_then(|v| v.as_array()) + .map(|ixs| { + ixs.iter() + .map(|ix| { + let name = str_of(ix.get("name")); + let pascal = to_pascal(&name); + let args = ix + .get("args") + .and_then(|v| v.as_array()) + .map(|a| a.iter().filter_map(|x| x.as_str().map(String::from)).collect()) + .unwrap_or_default(); + let accounts = ix + .get("accounts") + .and_then(|v| v.as_array()) + .map(|a| { + a.iter() + .filter_map(|x| x.get("name").and_then(|n| n.as_str()).map(String::from)) + .collect() + }) + .unwrap_or_default(); + IxFact { name, pascal, args, accounts } + }) + .collect() + }) + .unwrap_or_default(); + ApiFacts { + crate_id: program, + analysis_id: analysis_id.as_deref(), + program_id, + account_types, + instructions, + } + .render() + .expect("render api_facts") +} + +/// The "previous attempt failed, fix it" suffix shared by both authoring loops: lead with +/// the *extracted* compiler errors, then the prior source, then a trimmed raw-log tail. +fn revise_suffix(prev_src: &str, raw: &str) -> String { + let errors = compiler_diagnostics(raw); + let tail = &raw[raw.len().saturating_sub(2500)..]; + ReviseCompile { errors: &errors, prev_src, tail } + .render() + .expect("render revise_compile") +} + +/// The "previous attempt was rejected by the reviewer" suffix. Unlike `revise_suffix`, the +/// draft *compiled* — so frame it as review feedback to address, not compiler errors to fix +/// (otherwise the author thrashes hunting for build errors that do not exist). +fn judge_revise_suffix(prev_src: &str, feedback: &str) -> String { + ReviseJudge { feedback, prev_src } + .render() + .expect("render revise_judge") +} + +/// Dispatch the re-author suffix on which gate rejected the prior attempt. +fn revise_for(f: &Failure) -> String { + match f.kind { + FailureKind::Judge => judge_revise_suffix(&f.draft, &f.errors), + FailureKind::Compile => revise_suffix(&f.draft, &f.errors), + } +} + + + +/// Did the build fail (as opposed to the harness building and fuzzing)? +fn is_build_error(out: &str) -> bool { + out.contains("could not compile") || out.contains("error[") || out.contains("Build failed") +} + +/// Pull the human-readable finding out of a `crucible run` log so a `BAD` verdict explains +/// itself. Crucible prints `[FUZZ_FINDING] crash:<id> reproduces:<bool> summary:<msg>`, where +/// `<msg>` is the failed `fuzz_assert_*` message (with the actual vs expected values). Returns +/// `crash <id>: <msg>`, or the raw marker line if the summary can't be parsed, or None. +fn finding_detail(out: &str) -> Option<String> { + let line = out.lines().find(|l| l.contains("[FUZZ_FINDING]"))?.trim(); + match line.split_once("summary:") { + Some((head, summary)) if !summary.trim().is_empty() => { + let crash = + head.split_whitespace().find_map(|t| t.strip_prefix("crash:")).unwrap_or("?"); + Some(format!("crash {crash}: {}", summary.trim())) + } + _ => Some(line.to_string()), + } +} + +/// Attribute a shared-target counterexample across the covered report units. Crucible tags each +/// assertion message with its property title (`[<title>]`), so the finding names the invariant it +/// refutes: that unit gets `BAD` (carrying the finding); the rest held over the explored space, so +/// `GOOD`. If nothing can be attributed (no tagged title matched), mark them all `BAD` rather than +/// silently pass a real counterexample. This is the backend's own attribution — the host never +/// parses the finding. +fn attribute_finding(covered: &[Unit], detail: Option<String>) -> ValidateOutcome { + let d = detail.clone().unwrap_or_default(); + let named: std::collections::HashSet<&str> = covered + .iter() + .filter(|u| !u.property.is_empty() && d.contains(&u.property)) + .map(|u| u.unit.as_str()) + .collect(); + let all_bad = named.is_empty(); + ValidateOutcome::Verdicts { + verdicts: covered + .iter() + .map(|u| { + if all_bad || named.contains(u.unit.as_str()) { + let mut v = Verdict::with_outcome("BAD"); + v.detail = detail.clone(); + (u.unit.clone(), v) + } else { + (u.unit.clone(), Verdict::with_outcome("GOOD")) + } + }) + .collect(), + } +} + +// =========================================================================== +// Backend glue: small pure helpers shared by the callouts. +// =========================================================================== + +/// A string field from the input's `context` blob (e.g. the shared fixture source). +fn ctx_str(input: &AuthorInput, key: &str) -> String { + input.context.get(key).and_then(|v| v.as_str()).unwrap_or_default().to_string() +} + +/// A u64 field from the input's `context` blob, with a default. +fn ctx_u64(input: &AuthorInput, key: &str, default: u64) -> u64 { + input.context.get(key).and_then(|v| v.as_u64()).unwrap_or(default) +} + +/// The compiler errors to hand back to the model — extracted diagnostics, else a raw tail. +fn build_errors(out: &CommandOutput) -> String { + let combined = format!("{}\n{}", out.stdout, out.stderr); + let d = compiler_diagnostics(&combined); + if d.is_empty() { + combined[combined.len().saturating_sub(2000)..].to_string() + } else { + d + } +} + +// =========================================================================== +// The backend. +// =========================================================================== + +struct CrucibleApp; + +impl Backend for CrucibleApp { + fn descriptor(&self) -> AppDescriptor { + AppDescriptor { + name: "crucible".to_string(), + header_text: "Crucible — Solana fuzzing backend | AutoProver".to_string(), + // Selects the shared `solana` ecosystem front half (system model + prompts). + ecosystem: "solana".to_string(), + backend_tag: "crucible".to_string(), + backend_guidance: BackendGuidance.render().expect("render backend_guidance"), + analysis_key: "crucible-solana-analysis".to_string(), + phases: vec![ + // UI-only phase: discover the design doc when one isn't supplied (§host). + PhaseSpec { key: "discover_design_doc".into(), label: "Design Doc Discovery".into(), order: 0, core_slot: None }, + PhaseSpec { key: "analysis".into(), label: "System Analysis".into(), order: 1, core_slot: Some(CoreSlot::Analysis) }, + PhaseSpec { key: "extraction".into(), label: "Property Extraction".into(), order: 2, core_slot: Some(CoreSlot::Extraction) }, + // UI-only phase: build the program `.so` + IDL + shared fixture (§5.1). + PhaseSpec { key: "build_harness".into(), label: "Build Harness".into(), order: 3, core_slot: None }, + PhaseSpec { key: "formalization".into(), label: "Harness Authoring".into(), order: 4, core_slot: Some(CoreSlot::Formalization) }, + PhaseSpec { key: "report".into(), label: "Report".into(), order: 5, core_slot: Some(CoreSlot::Report) }, + ], + // Only `--fuzz-timeout` is wired through to `crucible run`. Other tuning knobs + // (parallel cores, stateful mode, a version pin) are deliberately omitted until + // they're actually threaded to the fuzz command — an inert flag is worse than none. + args: vec![ + ArgSpec { + flag: "--fuzz-timeout".to_string(), + help: "Per-test fuzzing budget in seconds (`crucible run --timeout`).".to_string(), + default: ArgDefault::Int { value: Some(60) }, + required: false, + }, + ], + rag_db_default: Some("crucible_kb".to_string()), + event_kinds: vec![ + EventKind::log("fuzz_pulse", "Fuzzing"), + EventKind::log("fuzz_finding", "Finding"), + EventKind::log("build_output", "Build"), + // The reviewer (judge) turn's accept/reject on a compiled test suite. + EventKind::log("judge", "Review"), + // The per-invariant verdict — surfaced as a persistent callout + toast. + EventKind::notice("verdict", "Verdict"), + ], + // Metadata (properties.json / commentary / property→tests map) lands under + // `certora/crucible/` — the split Foundry uses — while the crate deliverable is the + // one file under `fuzz/<program>/` (deliverable_primary + the finalize render). + artifact_layout: ArtifactLayout { + deliverable_dir: "certora/crucible".into(), + internal_dir: ".certora_internal/crucible".into(), + report_dir: "certora/crucible/reports".into(), + artifact_dir: "certora/crucible/harnesses".into(), + artifact_prefix: "harness".into(), + artifact_extension: "rs".into(), + property_suffix: "property_tests".into(), + deliverable_primary: Some("fuzz/{program}/src/main.rs".into()), + }, + // A shared fixture authored once (the setup step), one crate assembled by finalize + // (callout), all toolchain runs serialized on the one crate/target, confined by + // default (untrusted native builds), and "instruction" as the unit noun. + setup: Some(SetupSpec { + phase_key: "build_harness".into(), + label: "Build Harness".into(), + context_key: "fixture".into(), + }), + deliverable_mode: DeliverableMode::Callout, + serialize_toolchain: true, + confine_by_default: true, + component_noun: Some("instruction".into()), + } + } + + fn validate_preconditions(&self, args: &serde_json::Value) -> Result<(), String> { + let mut problems: Vec<String> = Vec::new(); + + let missing: Vec<&str> = REQUIRED_BINARIES + .iter() + .copied() + .filter(|b| !on_path(b)) + .collect(); + if !missing.is_empty() { + problems.push(format!( + "required tool(s) not found on PATH: {}. Install the Solana toolchain \ + (solana-cli / cargo-build-sbf), Anchor, and the crucible CLI \ + (`cargo install --path crates/crucible-fuzz-cli`).", + missing.join(", ") + )); + } + + // The target must be a buildable Cargo/Anchor workspace (cf. foundry's + // foundry.toml precondition). We only check structure here; the actual build + // happens in the build phase. + if let Some(root) = args.get("project_root").and_then(|v| v.as_str()) { + if !Path::new(root).join("Cargo.toml").is_file() { + problems.push(format!( + "{root}/Cargo.toml not found — Crucible needs a buildable Cargo/Anchor \ + workspace with the program under programs/<name>/." + )); + } + } else { + problems.push("no project_root in args".to_string()); + } + + // The crucible checkout resolves the harness crate's path deps (§6.1). Was + // `resolve_crucible_repo` in Python; now the wheel owns it (it renders the deps). + match std::env::var("CRUCIBLE_REPO") { + Ok(repo) if Path::new(&repo).join("crates/crucible-fuzzer").is_dir() => {} + Ok(repo) => problems.push(format!( + "$CRUCIBLE_REPO={repo} has no crates/crucible-fuzzer — set it to a local crucible \ + clone (the harness deps resolve against it)." + )), + Err(_) => problems.push( + "$CRUCIBLE_REPO is not set — point it at a local crucible clone (must contain \ + crates/crucible-fuzzer); the harness crate's path deps resolve against it." + .to_string(), + ), + } + + if problems.is_empty() { + Ok(()) + } else { + Err(problems.join("\n")) + } + } + + fn units(&self, input: &AuthorInput) -> Vec<Unit> { + // The setup fixture has no report units. A component's invariants all live in ONE harness + // fn (`SINGLE_HARNESS_FN`) — a single build + fuzz run covering every property + // (docs/crucible-unit-granularity.md §3) — but each property is still its own report row, + // mapping to that shared fuzz target. The host runs the shared target once and attributes + // a counterexample to the offending property via the finding message. + if input.kind == "setup" { + return Vec::new(); + } + input + .props + .iter() + .enumerate() + .map(|(i, p)| { + let slug = if p.slug.is_empty() { format!("inv{i}") } else { p.slug.clone() }; + // Report row = c_<slug> (one per property); fuzz target = the shared c_invariants. + Unit { + property: p.title.clone(), + unit: format!("c_{slug}"), + target: Some(SINGLE_HARNESS_FN.to_string()), + } + }) + .collect() + } + + fn author_prompt(&self, input: &AuthorInput, failure: Option<&Failure>) -> Prompt { + let program = &input.program; + let revise = failure.map(revise_for).unwrap_or_default(); + let instruction = if input.kind == "setup" { + // Author the shared fixture from the analyzed model (carried in `component`). + let analyzed = &input.component; + let model = + serde_json::to_string_pretty(analyzed).unwrap_or_else(|_| analyzed.to_string()); + let cheat = HarnessCheatSheet { program }.render().expect("render harness_cheat_sheet"); + let example = ExampleFixture.render().expect("render example_fixture"); + let facts = api_facts(analyzed, program); + AuthorSetup { + program, + cheat: &cheat, + example: &example, + facts: &facts, + model: &model, + revise: &revise, + } + .render() + .expect("render author_setup") + } else { + // Author ONE #[invariant_test] fn holding ALL invariants (single build + run). + let listed = input + .props + .iter() + .map(|p| format!("- [{}] {}: {}", p.sort, p.title, p.description)) + .collect::<Vec<_>>() + .join("\n"); + let component = serde_json::to_string_pretty(&input.component) + .unwrap_or_else(|_| input.component.to_string()); + let fixture = ctx_str(input, "fixture"); + let cheat = TestCheatSheet.render().expect("render test_cheat_sheet"); + AuthorComponent { + harness_fn: SINGLE_HARNESS_FN, + program, + n: input.props.len(), + first: input.props.first().map(|p| p.title.as_str()).unwrap_or("property"), + listed: &listed, + component: &component, + cheat: &cheat, + fixture: &fixture, + revise: &revise, + } + .render() + .expect("render author_component") + }; + Prompt { system: None, instruction } + } + + fn judge_prompt(&self, input: &AuthorInput, spec: &str) -> Option<Prompt> { + // The shared fixture is scaffolding, not test evidence — the compile/dry-run gate + // already vets it, and there is no property to judge it against. Judge only the + // per-component test suites (the peer of Foundry's feedback judge). + if input.kind == "setup" { + return None; + } + let program = &input.program; + let listed = input + .props + .iter() + .map(|p| format!("- [{}] {}: {}", p.sort, p.title, p.description)) + .collect::<Vec<_>>() + .join("\n"); + let component = serde_json::to_string_pretty(&input.component) + .unwrap_or_else(|_| input.component.to_string()); + let fixture = ctx_str(input, "fixture"); + let instruction = JudgeInstruction { + program, + harness_fn: SINGLE_HARNESS_FN, + listed: &listed, + component: &component, + fixture: &fixture, + spec, + } + .render() + .expect("render judge_instruction"); + let system = JudgeSystem.render().expect("render judge_system"); + Some(Prompt { system: Some(system), instruction }) + } + + fn compile( + &self, + input: &AuthorInput, + spec: &str, + workdir: &Path, + sandbox: &Sandbox, + ) -> CompileResult { + let program = &input.program; + // Setup: dry-run the fixture behind a probe test. Component: fixture + the authored tests, + // dry-run behind the shared harness feature `c_invariants` (which gates `main`). + let (main_rs, feature) = if input.kind == "setup" { + let probe = ProbeFn.render().expect("render probe_fn"); + (format!("{spec}{probe}"), "c_probe".to_string()) + } else { + let fixture = ctx_str(input, "fixture"); + (format!("{fixture}\n\n{spec}"), SINGLE_HARNESS_FN.to_string()) + }; + let files = crate_files(program, main_rs, std::slice::from_ref(&feature)); + let args = vec![ + "run".to_string(), + program.clone(), + feature, + "--release".to_string(), + "--dry-run".to_string(), + ]; + match run_confined(sandbox, "crucible", &args, &files, workdir) { + Ok(out) + if out.exit_code == 0 + && !is_build_error(&format!("{}\n{}", out.stdout, out.stderr)) => + { + CompileResult::Ok + } + Ok(out) => CompileResult::Failed { errors: build_errors(&out) }, + Err(e) => CompileResult::Failed { errors: e }, + } + } + + fn validate( + &self, + input: &AuthorInput, + spec: &str, + unit: &str, + workdir: &Path, + sandbox: &Sandbox, + ) -> ValidateOutcome { + let program = &input.program; + let fixture = ctx_str(input, "fixture"); + let timeout = ctx_u64(input, "fuzz_timeout", 30); + let files = crate_files(program, format!("{fixture}\n\n{spec}"), std::slice::from_ref(&unit.to_string())); + let args = vec![ + "run".to_string(), + program.clone(), + unit.to_string(), + "--release".to_string(), + "--mode".to_string(), + "explore".to_string(), + "--timeout".to_string(), + timeout.to_string(), + ]; + // The report units this fuzz target covers (Crucible: every invariant shares `c_invariants`). + // The backend owns attribution — it maps ONE run to a verdict per covered unit. + let covered: Vec<Unit> = + self.units(input).into_iter().filter(|u| u.target_or_unit() == unit).collect(); + let all = |o: &str, detail: Option<String>| -> ValidateOutcome { + ValidateOutcome::Verdicts { + verdicts: covered + .iter() + .map(|u| { + let mut v = Verdict::with_outcome(o); + v.detail = detail.clone(); + (u.unit.clone(), v) + }) + .collect(), + } + }; + match run_confined(sandbox, "crucible", &args, &files, workdir) { + Ok(out) => { + let combined = format!("{}\n{}", out.stdout, out.stderr); + // Order matters: a fuzz finding and a clean run both mean the harness BUILT, so + // classify those first — only a *non-zero* exit with build markers is a real + // build failure. This keeps `error[...]`-looking runtime/log text in a clean + // (exit 0) fuzz run from being misread as a build failure. + if combined.contains("[FUZZ_FINDING]") { + // A crash refutes ONE invariant — pin BAD to the property the finding names + // (each assertion is tagged `[<title>]`), holding the rest GOOD over the + // explored space. If it can't be attributed, mark all BAD (never hide it). + attribute_finding(&covered, finding_detail(&combined)) + } else if out.exit_code == 0 { + all("GOOD", None) // ran to the budget with no violation = every invariant held + } else if is_build_error(&combined) { + // Shared build; re-author the whole spec (docs/rust-backend-api.md). + ValidateOutcome::BuildFailed { errors: build_errors(&out) } + } else { + // Non-zero exit with no build markers and no finding — capture the tail. + all("ERROR", Some(build_errors(&out))) + } + } + Err(e) => all("ERROR", Some(e)), + } + } + + fn sandbox_grants(&self, _args: &serde_json::Value) -> SandboxGrants { + // Read-only grants beyond the launcher's discovered Rust toolchain: the crucible checkout + // (path deps) and the `crucible` binary's dir. Was Python's `crucible_sandbox` extra_ro. + let mut extra_ro = Vec::new(); + if let Ok(repo) = std::env::var("CRUCIBLE_REPO") { + extra_ro.push(repo); + } + if let Some(dir) = which_dir("crucible") { + extra_ro.push(dir); + } + SandboxGrants { extra_ro, extra_env: Vec::new() } + } + + fn workspace_prep(&self, input: &AuthorInput) -> WorkspacePrep { + // Place a deps-only harness manifest (probe feature) so warming has a manifest and the + // setup dry-run can select a feature; per-run builds overwrite it with their own feature. + // Then warm the harness crate's deps and build the program `.so` (the host runs both with + // the shared helpers — fetch unconfined, build confined+offline). + let program = &input.program; + let mut files = BTreeMap::new(); + if let Some(repo) = crucible_repo() { + files.insert( + format!("fuzz/{program}/Cargo.toml"), + render_cargo_toml(program, &repo, std::slice::from_ref(&"c_probe".to_string())), + ); + } + WorkspacePrep { + files, + warm_dirs: vec![format!("fuzz/{program}")], + build_program: Some(program.clone()), + } + } + + fn finalize(&self, outcomes: &serde_json::Value) -> BTreeMap<String, String> { + // Assemble the one deliverable crate: the shared fixture + every delivered invariant's + // test section, keyed by its feature (was Python's `CrucibleHarness`/`CrucibleArtifactStore`). + let program = outcomes.get("program").and_then(|v| v.as_str()).unwrap_or_default(); + let fixture = outcomes.get("setup").and_then(|v| v.as_str()).unwrap_or_default(); + + // feature -> test section (BTreeMap keeps a stable, sorted crate — matches the old store). + let mut sections: BTreeMap<String, String> = BTreeMap::new(); + if let Some(comps) = outcomes.get("components").and_then(|v| v.as_array()) { + for c in comps { + if !c.get("delivered").and_then(|v| v.as_bool()).unwrap_or(false) { + continue; + } + let text = c.get("artifact_text").and_then(|v| v.as_str()).unwrap_or_default(); + // property_units: [[title, [feature, ...]], ...] + if let Some(pu) = c.get("property_units").and_then(|v| v.as_array()) { + for entry in pu { + if let Some(units) = entry.get(1).and_then(|v| v.as_array()) { + for u in units.iter().filter_map(|v| v.as_str()) { + sections.insert(u.to_string(), text.trim().to_string()); + } + } + } + } + } + } + if program.is_empty() || sections.is_empty() { + return BTreeMap::new(); + } + + let features: Vec<String> = sections.keys().cloned().collect(); + let body = features.iter().map(|f| sections[f].clone()).collect::<Vec<_>>().join("\n\n"); + let main_rs = format!( + "{}\n\n{}{}", + fixture.trim_end(), + body, + if body.is_empty() { "" } else { "\n" } + ); + let mut files = BTreeMap::new(); + files.insert(format!("fuzz/{program}/src/main.rs"), main_rs); + if let Some(repo) = crucible_repo() { + files.insert( + format!("fuzz/{program}/Cargo.toml"), + render_cargo_toml(program, &repo, &features), + ); + } + files + } +} + +autoprover_sdk::export_app!(crucible_app, CrucibleApp); + +#[cfg(test)] +mod template_parity { + //! Guards the askama migration: the build-critical crate files must render byte-identically + //! to the former `format!` output (else the harness crate won't compile), and the static + //! prose templates must preserve their bytes. Prompts are checked for template residue only. + use super::*; + + /// The OLD `crucible_deps` format! body — kept here verbatim as the parity oracle. + fn expected_deps(program: &str, repo: &Path) -> String { + let crates = repo.join("crates"); + format!( + "crucible-fuzzer = {{ path = \"{cf}\" }}\n\ + crucible-test-context = {{ path = \"{ctc}\" }}\n\ + anchor-lang = \"{ANCHOR_VERSION}\"\n\ + arbitrary = {{ version = \"1\", features = [\"derive\"] }}\n\ + ctrlc = \"3.4\"\n\ + libafl = {{ version = \"{LIBAFL_VERSION}\", features = [\"std\", \"cli\", \"prelude\"] }}\n\ + libafl_bolts = {{ version = \"{LIBAFL_VERSION}\", features = [\"std\"] }}\n\ + {program} = {{ path = \"../../programs/{program}\", features = [\"no-entrypoint\"] }}\n\ + solana-keypair = \"{SOLANA_VERSION}\"\n\ + solana-pubkey = \"{SOLANA_VERSION}\"\n\ + solana-signer = \"{SOLANA_VERSION}\"", + cf = crates.join("crucible-fuzzer").display(), + ctc = crates.join("crucible-test-context").display(), + ) + } + + /// The OLD `render_cargo_toml` format! body — the parity oracle. + fn expected_cargo_toml(program: &str, repo: &Path, features: &[String]) -> String { + let feats = if features.is_empty() { + "# (no components yet)".to_string() + } else { + features.iter().map(|f| format!("{f} = []")).collect::<Vec<_>>().join("\n") + }; + format!( + "[package]\n\ + name = \"{program}_fuzz\"\n\ + version = \"0.1.0\"\n\ + edition = \"2021\"\n\ + \n\ + [workspace]\n\ + \n\ + [dependencies]\n\ + {deps}\n\ + \n\ + [[bin]]\n\ + name = \"invariant_test\"\n\ + path = \"src/main.rs\"\n\ + \n\ + [features]\n\ + {feats}\n", + deps = expected_deps(program, repo), + ) + } + + #[test] + fn crate_files_are_byte_identical_to_the_old_format() { + let repo = Path::new("/home/user/crucible"); + assert_eq!(crucible_deps("vault", repo), expected_deps("vault", repo)); + // empty features + assert_eq!( + render_cargo_toml("vault", repo, &[]), + expected_cargo_toml("vault", repo, &[]), + ); + // one and several features + for feats in [vec!["c_invariants".to_string()], vec!["c_probe".into(), "c_invariants".into()]] { + assert_eq!( + render_cargo_toml("vault", repo, &feats), + expected_cargo_toml("vault", repo, &feats), + "cargo_toml mismatch for features {feats:?}", + ); + } + } + + #[test] + fn static_templates_preserve_their_bytes() { + // askama drops exactly one trailing newline from a template file, so every `.j2` carries + // one extra (see the trailing blank line in each). The content is otherwise preserved + // verbatim, i.e. `render() + "\n" == file`. Asserting that here pins both facts: the + // static prose is byte-for-byte what shipped, and the one-newline convention holds. + let eq = |rendered: String, file: &str| assert_eq!(format!("{rendered}\n"), file); + eq(BackendGuidance.render().unwrap(), include_str!("../templates/backend_guidance.j2")); + eq(ExampleFixture.render().unwrap(), include_str!("../templates/example_fixture.j2")); + eq(TestCheatSheet.render().unwrap(), include_str!("../templates/test_cheat_sheet.j2")); + eq(JudgeSystem.render().unwrap(), include_str!("../templates/judge_system.j2")); + eq(ProbeFn.render().unwrap(), include_str!("../templates/probe_fn.j2")); + } + + #[test] + fn harness_cheat_sheet_substitutes_program_and_has_no_placeholder() { + let out = HarnessCheatSheet { program: "vault" }.render().unwrap(); + assert!(out.contains("use vault::*;"), "program not substituted:\n{out}"); + assert!(!out.contains("<program>"), "leftover <program> placeholder"); + assert!(!out.contains("{{"), "leftover askama expression"); + } + + #[test] + fn api_facts_renders_from_the_analyzed_model() { + let model = serde_json::json!({ + "components": [{ + "name": "vault", + "program_identifier": "vault_program", + "program_id": "Vau1t111", + "account_types": ["VaultState"], + "instructions": [ + {"name": "deposit", "args": ["amount"], + "accounts": [{"name": "vault"}, {"name": "depositor"}]}, + ], + }], + }); + let out = api_facts(&model, "vault"); + for needle in [ + "crate id (for `use <id>::*`): vault", + "pub mod vault_program", // module-name note (differs from crate id) + "declare_id / program id: Vau1t111", + "state/account types: VaultState", + "- deposit → instruction::Deposit, accounts::Deposit; args: [amount]; accounts: [vault, depositor]", + ] { + assert!(out.contains(needle), "api_facts missing {needle:?} in:\n{out}"); + } + assert!(!out.contains("{{") && !out.contains("{%"), "template residue in api_facts"); + // Unrecognized model shape → empty (unchanged contract). + assert_eq!(api_facts(&serde_json::json!({}), "vault"), ""); + } + + fn assert_no_residue(s: &str) { + for t in ["{{", "{%", "{#"] { + assert!(!s.contains(t), "template residue {t:?} in:\n{s}"); + } + } + + #[test] + fn prompt_templates_render_end_to_end() { + use autoprover_sdk::Property; + + let app = CrucibleApp; + let component = serde_json::json!({ "instructions": [{ "name": "deposit" }] }); + let prop = Property { + title: "no overflow".into(), + sort: "invariant".into(), + description: "balance never overflows".into(), + slug: "no_overflow".into(), + }; + + // setup branch + a compile failure (exercises author_setup.j2 + revise_compile.j2). + let setup = AuthorInput { + kind: "setup".into(), + program: "vault".into(), + component: component.clone(), + props: vec![], + context: serde_json::Value::Null, + }; + let compile_fail = Failure { + draft: "prior fixture src".into(), + errors: "error[E0433]: failed to resolve".into(), + kind: FailureKind::Compile, + }; + // Prose templates are wrapped to 120, so a phrase can span a newline — compare with + // whitespace collapsed so the checks are wrap-insensitive. + let norm = |s: &str| s.split_whitespace().collect::<Vec<_>>().join(" "); + let has = |hay: &str, needle: &str| assert!( + norm(hay).contains(&norm(needle)), "missing {needle:?} in:\n{hay}" + ); + + let p = app.author_prompt(&setup, Some(&compile_fail)); + assert_no_residue(&p.instruction); + has(&p.instruction, "FIXTURE (only) for the Solana program `vault`"); + has(&p.instruction, "The previous attempt FAILED"); + has(&p.instruction, "error[E0433]"); + + // component branch + a judge failure (exercises author_component.j2 + revise_judge.j2). + let comp = AuthorInput { + kind: "component".into(), + program: "vault".into(), + component: component.clone(), + props: vec![prop], + context: serde_json::json!({ "fixture": "struct Fixture { ctx: TestContext }" }), + }; + let judge_fail = Failure { + draft: "prior tests".into(), + errors: "Criterion 1: vacuous assertion".into(), + kind: FailureKind::Judge, + }; + let p = app.author_prompt(&comp, Some(&judge_fail)); + assert_no_residue(&p.instruction); + has(&p.instruction, "named EXACTLY `c_invariants`"); + has(&p.instruction, "`\"[no overflow] ...\"`"); + has(&p.instruction, "reviewer REJECTED"); + + // judge prompt (exercises judge_instruction.j2 + the judge_guidance.j2 include + system). + let jp = app.judge_prompt(&comp, "fn c_invariants(f: &mut Fixture) {}").expect("component judge"); + assert_no_residue(&jp.instruction); + has(&jp.instruction, "Evaluate the Crucible fuzz-test suite"); + has(&jp.instruction, "Criterion 1"); + has(jp.system.as_deref().unwrap(), "senior Solana security engineer"); + // setup has no judge turn. + assert!(app.judge_prompt(&setup, "x").is_none()); + } +} diff --git a/rust/crucible-app/templates/api_facts.j2 b/rust/crucible-app/templates/api_facts.j2 new file mode 100644 index 00000000..b31b6527 --- /dev/null +++ b/rust/crucible-app/templates/api_facts.j2 @@ -0,0 +1,8 @@ +PROGRAM API FACTS (use these EXACT names — do not guess): + crate id (for `use <id>::*`): {{ crate_id }} +{% if let Some(aid) = analysis_id %} (note: `#[program] pub mod {{ aid }}` is the module name — it does NOT change the crate-root paths `{{ crate_id }}::instruction::*` / `{{ crate_id }}::accounts::*`) +{% endif %} declare_id / program id: {{ program_id }} +{% if !account_types.is_empty() %} state/account types: {{ account_types|join("; ") }} +{% endif %} instructions (snake handler → Anchor Pascal structs): +{% for ix in instructions %} - {{ ix.name }} → instruction::{{ ix.pascal }}, accounts::{{ ix.pascal }}; args: [{{ ix.args|join(", ") }}]; accounts: [{{ ix.accounts|join(", ") }}] +{% endfor %} diff --git a/rust/crucible-app/templates/author_component.j2 b/rust/crucible-app/templates/author_component.j2 new file mode 100644 index 00000000..8cc40754 --- /dev/null +++ b/rust/crucible-app/templates/author_component.j2 @@ -0,0 +1,18 @@ +Author a SINGLE Crucible `#[invariant_test]` function named EXACTLY `{{ harness_fn }}` for the `{{ program }}` program +that checks ALL {{ n }} properties below — one fuzz run covers them all. It must hold after ANY sequence of actions the +fuzzer drives; read on-chain state and assert each property in the one fn. + +Properties to check (assert each; PREFIX every assertion message with its property title in brackets, e.g. +`"[{{ first }}] ..."`, so a counterexample names the property it refutes): +{{ listed }} + +Program API (drive instructions via the fixture's `action_*` methods): +{{ component }} + +{{ cheat }} + +The shared fixture is ALREADY defined (do not redefine it); use `Fixture` and its `action_*` methods. Fixture source for +reference: +```rust +{{ fixture }} +```{{ revise }} diff --git a/rust/crucible-app/templates/author_setup.j2 b/rust/crucible-app/templates/author_setup.j2 new file mode 100644 index 00000000..af8fbbfb --- /dev/null +++ b/rust/crucible-app/templates/author_setup.j2 @@ -0,0 +1,11 @@ +Author a Crucible fuzz-harness FIXTURE (only) for the Solana program `{{ program }}`. +{{ cheat }} + +{{ example }} + +{{ facts }} +Full analyzed system model (accounts, PDAs, authorities, requirements): +{{ model }} + +Use the source-exploration tools to read the program's Rust source for exact signatures. Return the complete fixture +module source as your final answer.{{ revise }} diff --git a/rust/crucible-app/templates/backend_guidance.j2 b/rust/crucible-app/templates/backend_guidance.j2 new file mode 100644 index 00000000..a702a0a5 --- /dev/null +++ b/rust/crucible-app/templates/backend_guidance.j2 @@ -0,0 +1,9 @@ +These properties will be checked with Crucible, a coverage-guided fuzzer for Solana programs. As a fuzzer it cannot +*prove* universally quantified properties or invariants, but it approximates them well and any *refutation* (a fuzzing +counterexample / crash) is extremely valuable. So state universal safety properties and invariants freely — do not +restrict yourself because a fuzzer cannot definitively prove them. + +A few categories are a poor fit and should be skipped: properties about off-chain events (key compromise, social +engineering, oracle manipulation outside the modeled accounts), and pure hash-collision resistance ("no two inputs ever +collide"), which sampling cannot refute. Arithmetic-overflow and type-level facts are worth stating: Rust overflow +panics and Anchor constraint failures surface as crashes the fuzzer can find. diff --git a/rust/crucible-app/templates/cargo_deps.j2 b/rust/crucible-app/templates/cargo_deps.j2 new file mode 100644 index 00000000..952e082e --- /dev/null +++ b/rust/crucible-app/templates/cargo_deps.j2 @@ -0,0 +1,11 @@ +crucible-fuzzer = { path = "{{ cf }}" } +crucible-test-context = { path = "{{ ctc }}" } +anchor-lang = "{{ anchor_version }}" +arbitrary = { version = "1", features = ["derive"] } +ctrlc = "3.4" +libafl = { version = "{{ libafl_version }}", features = ["std", "cli", "prelude"] } +libafl_bolts = { version = "{{ libafl_version }}", features = ["std"] } +{{ program }} = { path = "../../programs/{{ program }}", features = ["no-entrypoint"] } +solana-keypair = "{{ solana_version }}" +solana-pubkey = "{{ solana_version }}" +solana-signer = "{{ solana_version }}" diff --git a/rust/crucible-app/templates/cargo_toml.j2 b/rust/crucible-app/templates/cargo_toml.j2 new file mode 100644 index 00000000..bab0b485 --- /dev/null +++ b/rust/crucible-app/templates/cargo_toml.j2 @@ -0,0 +1,17 @@ +[package] +name = "{{ program }}_fuzz" +version = "0.1.0" +edition = "2021" + +[workspace] + +[dependencies] +{{ deps }} + +[[bin]] +name = "invariant_test" +path = "src/main.rs" + +[features] +{{ feats }} + diff --git a/rust/crucible-app/templates/example_fixture.j2 b/rust/crucible-app/templates/example_fixture.j2 new file mode 100644 index 00000000..9bb53288 --- /dev/null +++ b/rust/crucible-app/templates/example_fixture.j2 @@ -0,0 +1,62 @@ + +EXAMPLE — a full, compiling fixture for a *different* program (an `escrow`). Study the +shape, then write the equivalent for THIS program: + +```rust +use crucible_fuzzer::anchor_lang::system_program; +use crucible_fuzzer::*; +use solana_keypair::Keypair; +use solana_pubkey::Pubkey; +use solana_signer::Signer; +use std::rc::Rc; +use escrow::*; // the crate id — NOT the `#[program] mod` name + +#[derive(Clone)] +struct Fixture { // MUST be named `Fixture` + ctx: TestContext, + program_id: Pubkey, + depositor: Rc<Keypair>, + vault_pda: Pubkey, +} + +#[fuzz_fixture] +impl Fixture { + pub fn setup() -> Self { + let mut ctx = TestContext::new(); + let program_id = Pubkey::new_from_array(ID.to_bytes()); + ctx.add_program(&program_id, "../../target/deploy/escrow.so").unwrap(); + + let depositor = Rc::new(Keypair::new()); + ctx.create_account().pubkey(depositor.pubkey()) + .lamports(10_000_000_000).owner(system_program::ID).create().unwrap(); + + let (vault_pda, _) = + Pubkey::find_program_address(&[b"vault", depositor.pubkey().as_ref()], &program_id); + + ctx.program(program_id) + .call(instruction::Initialize {}) // args struct; `{}` when the ix has no args + .accounts(accounts::Initialize { + vault: vault_pda, + depositor: depositor.pubkey(), + system_program: system_program::ID, + }) + .signers(&[&*depositor]) + .send().unwrap(); // panic in setup() if init fails + + Self { ctx, program_id, depositor, vault_pda } + } + + pub fn action_deposit(&mut self, #[range(1..1_000_000)] amount: u64) -> bool { + self.ctx.program(self.program_id) + .call(instruction::Deposit { amount }) + .accounts(accounts::Deposit { + vault: self.vault_pda, + depositor: self.depositor.pubkey(), + system_program: system_program::ID, + }) + .signers(&[&*self.depositor]) + .send().map(|o| o.is_success()).unwrap_or(false) + } +} +``` + diff --git a/rust/crucible-app/templates/harness_cheat_sheet.j2 b/rust/crucible-app/templates/harness_cheat_sheet.j2 new file mode 100644 index 00000000..bd7dda36 --- /dev/null +++ b/rust/crucible-app/templates/harness_cheat_sheet.j2 @@ -0,0 +1,46 @@ + +Crucible harness API (author a FIXTURE only — no test fns): + +- Imports: + use crucible_fuzzer::*; // TestContext, macros, fuzz_assert_* + use crucible_fuzzer::anchor_lang::system_program; + use {{ program }}::*; // instruction, accounts, ID, state types + use solana_keypair::Keypair; use solana_pubkey::Pubkey; use solana_signer::Signer; + use std::rc::Rc; + +- The fixture struct MUST be named `Fixture` and derive Clone; keypairs go in `Rc`: + #[derive(Clone)] + struct Fixture { ctx: TestContext, program_id: Pubkey, /* pdas, users (Rc<Keypair>) */ } + +- #[fuzz_fixture] impl Fixture { ... } with: + pub fn setup() -> Self { + let mut ctx = TestContext::new(); + let program_id = Pubkey::new_from_array(ID.to_bytes()); + ctx.add_program(&program_id, "../../target/deploy/{{ program }}.so").unwrap(); + // create funded accounts: ctx.create_account().pubkey(kp.pubkey()) + // .lamports(N).owner(system_program::ID).create().unwrap(); + // derive PDAs: Pubkey::find_program_address(&[b"seed", key.as_ref()], &program_id) + // run any init instruction (see calling convention). Panic on setup failure. + Self { ctx, program_id, /* ... */ } + } + // one `action_<name>` per instruction; fuzzable args get #[range(lo..hi)]: + pub fn action_<name>(&mut self, #[range(0..1_000_000)] amount: u64) -> bool { + self.ctx.program(self.program_id) + .call(instruction::<Name> { amount }) + .accounts(accounts::<Name> { /* fields */ }) + .signers(&[&*self.some_keypair]) + .send().map(|o| o.is_success()).unwrap_or(false) + } + +- Anchor path conventions (use the API FACTS below — do NOT guess these): + * `use {{ program }}::*;` brings the crate's generated items into scope. `{{ program }}` is + the crate id in the facts, NOT the `#[program] pub mod <name>` module name — the + module name does NOT change these crate-root paths. + * Instruction args struct: `instruction::<PascalName>` — snake_case `foo_bar` → `instruction::FooBar`. + * Accounts struct: `accounts::<PascalName>` — same PascalCase as the instruction. + * Program id: the `ID` constant; make a Pubkey via `Pubkey::new_from_array(ID.to_bytes())`. +- Read the program source (via the tools) to confirm exact field names, PDA seeds + (binary vs string), and signer requirements — the API facts + model below are a summary. +- Output ONLY the fixture module source (imports + struct + #[fuzz_fixture] impl). + Do NOT write `fn main`, `#[invariant_test]`, or `#[crucible_fuzz]` — those are added later. + diff --git a/rust/crucible-app/templates/judge_guidance.j2 b/rust/crucible-app/templates/judge_guidance.j2 new file mode 100644 index 00000000..8fab3891 --- /dev/null +++ b/rust/crucible-app/templates/judge_guidance.j2 @@ -0,0 +1,115 @@ +Read the test functions AND the program source (via the +tools); several criteria require comparing the tests' claims against what the program actually +does. Then evaluate against the criteria below. The examples show the SHAPE of each defect; they +are not exhaustive — a defect matching a criterion is in scope even if it resembles no example. + +## Criterion 1 — Vacuous / tautological assertions +Flag assertions that hold regardless of the program's behavior: +- bounds guaranteed by the type or by arithmetic (`fuzz_assert_ge!(x, 0)` on a `u64`; a bound a +checked subtraction already guarantees); +- asserting a value the test/fixture itself just wrote, via a path that bypasses the logic under +test (seed an account field, then read the same field back); +- degenerate operands: solvency (`assets >= liabilities`) where liabilities are zero all campaign; +"unauthorized caller is rejected" where no one was ever authorized; a preservation invariant +that only ever runs in the initial state (holds by init, not by preservation). +An assertion is tautological IFF it holds under every program implementation. + +## Criterion 2 — Claim / assertion gap +For each test, compare (a) the property it claims (name, comments) against (b) the strongest fact +its assertions actually establish. Flag any test where (b) is materially weaker. Typical shapes: +- asserting an upstream precursor while the claimed consequence lives only in comments (assert a +field was stored, when the property is that an unauthorized withdraw is *rejected*; assert an +authority was rotated, when the property is that the old authority's calls now fail); +- comments narrate an attack the executable test never drives; +- the property specifies a sequence/interleaving (a stale read, a specific instruction order) but +the test never constructs it through the `action_*` sequence. +Prose proves nothing; a test's evidentiary value is exactly its assertions on on-chain state. + +## Criterion 3 — Reachability: can the fuzzer reach a state where the property could FAIL? +This is the load-bearing criterion for a fuzzing backend — an invariant is only as strong as the +states the campaign can drive it into. Audit the fixture + the `action_*` set the property leans on: +- **Missing actions**: no `action_*` exists for the instruction(s) the property is about, so the +campaign never drives the relevant transition and the invariant is only ever checked over states +that trivially satisfy it. +- **Collapsed input domains**: `#[range(lo..hi)]` narrowed to near-constants, a range that excludes +the violating region, or a fuzzed arg that influences no assertion. +- **Actions that never succeed**: an `action_*` whose `.send()` fails on most inputs (wrong +accounts / signers / funding) returns `false` and leaves state near-initial — a green invariant +over states never reached is evidence of nothing. +- **Degenerate fixtures**: a single actor/account for a multi-party property; zero balances or +empty state; identity-element params (a fee of 0, a rate of 1) that cannot distinguish a correct +program from a plausible wrong one. +Precondition seeding is fine; substituting for the enforced logic is not (see C4). + +## Criterion 4 — Real execution vs bypassed logic +Crucible runs the real program `.so` in LiteSVM, so fidelity is usually high — but check the +fixture/test does not bypass the logic under test: +- seeding an account's state directly (`create_account` / raw data) instead of driving the +program's instruction, when the property is about that instruction *computing or enforcing* that +state (injecting state the subject must compute assumes the conclusion; injecting state it merely +consumes is legitimate setup); +- reads must go through `read_anchor_account::<State>(&pda)` against the PDA the program actually +writes — not a local Rust mirror the test maintains alongside the chain. + +## Criterion 5 — Oracle independence +Flag tests whose expected value is the program's own logic fed back to itself (the same formula +transcribed into the test; the same derivation/hash the program uses). Prefer anchors knowable +without the implementation: conservation identities (sum of balances constant), boundary values, +round-trips, monotonicity between concrete states, input/output pairs computed offline. + +## Criterion 6 — Pass/fail directionality (especially attack-vector properties) +For each test ask: if the property regressed tomorrow, would this test FAIL? +- Attack-vector properties ("the exploit cannot occur"): a campaign that stays GREEN while the +attack succeeds has inverted semantics. A green suite must never certify a live vulnerability. If +the author genuinely found the attack possible, the correct artifact is an explicit finding plus +a test marked as a known-vulnerability demonstration — never a silently-passing test. +- "Must be rejected" checks: confirm the action fails FOR THE RIGHT REASON. On Solana an +instruction can fail for reasons unrelated to the property (missing signer, unfunded account, +wrong PDA, arithmetic overflow in setup). Assert the specific failure — a custom program error, +or that the guarded on-chain state is unchanged — and confirm a CONTROL action SUCCEEDS when the +guarded condition is absent. `action_*` returning `false` is not, by itself, evidence the +program's own check rejected the call. + +## Criterion 7 — Fuzz / invariant mechanics +- Right macro for the property: `#[invariant_test]` runs after EACH action (preservation +invariants — conservation, solvency, monotonic state); `#[crucible_fuzz]` runs one random op +(a per-instruction property). Flag a single-shot `#[crucible_fuzz]` used for what is really a +preservation invariant, or vice-versa. +- Assertions must read committed on-chain state via `read_anchor_account`; PDAs/addresses must +match what the program writes. +- Real-execution costs (a false-oracle trap): under LiteSVM the transaction fee payer — the +FIRST signer of an `action_*`'s `.send()` — is debited the tx fee (~5000 lamports/signature) +on top of any lamports the instruction moves, and accounts owe rent-exemption. An EXACT +lamport-delta assertion on a signer/fee-payer that ignores the fee (e.g. +`depositor_before - amount == depositor_after`) is a false oracle: it FAILS on a correct +program. Flag it — exact-delta assertions belong on non-signer accounts (the receiving PDA), +or must subtract the fee / assert a bound. +- Ties back to C3: confirm the reachable state space actually includes the property's danger +region under the available `action_*` sequences. + +## Criterion 8 — Coverage and redundancy +- Every listed property must have a correctly-named test fn. A property "addressed" only by tests +failing C1–C3 is NOT covered — say so explicitly and reject. +- Evaluate any declared skip critically: Crucible can drive real instructions, arbitrary account +state, multiple signers, and fuzz — few properties of on-chain logic are genuinely untestable. +Sketch the test you believe possible before accepting a skip. +- Flag redundant tests (different names, same fact about the same state); note any property left +uncovered as a result. + +Discard low-value feedback: style/naming/organization; compute-unit or performance quibbles; +"brittleness" (tests are tied to the implementation they verify); the exact bounds/magnitudes +chosen unless they make a test degenerate under C3; demands for tests beyond the listed properties. +The goal is to rule OUT low-quality tests, not to demonstrate thoroughness by volume — a short list +of load-bearing defects, each tied to a criterion and a concrete fix, beats an exhaustive +enumeration. + +Do NOT assert Crucible / LiteSVM / Anchor semantics you have not verified (from memory or the +docs). Before claiming a test is wrong about the program's behavior, read the relevant source and +cite what it does. Never contradict prior-round feedback unless you are ~95% certain it was in error. + +Output contract: use tools and reason as needed, but your FINAL message MUST be a single JSON +object and NOTHING else: +{"accept": true, "feedback": ""} +{"accept": false, "feedback": "<load-bearing defects — each tied to a criterion and a concrete fix>"} +Reject (set accept:false) if any listed property is uncovered, or is covered only by tests failing +Criteria 1–3. diff --git a/rust/crucible-app/templates/judge_instruction.j2 b/rust/crucible-app/templates/judge_instruction.j2 new file mode 100644 index 00000000..18fb5721 --- /dev/null +++ b/rust/crucible-app/templates/judge_instruction.j2 @@ -0,0 +1,21 @@ +Evaluate the Crucible fuzz-test suite below for the Solana program `{{ program }}`. Decide, per property, whether a +PASSING fuzz campaign is real evidence — not merely that the tests compile (the build gate already ensures that). + +Properties this suite must demonstrate (ALL checked inside the single `{{ harness_fn }}` invariant fn): +{{ listed }} + +Program API (instructions / accounts / state — driven via the fixture's `action_*` methods): +{{ component }} + +The shared fixture the tests build on (already compiled — do not re-review it, but use it to judge what states the +`action_*` methods can reach): +```rust +{{ fixture }} +``` + +Test suite under review: +```rust +{{ spec }} +``` + +{% include "judge_guidance.j2" %} diff --git a/rust/crucible-app/templates/judge_system.j2 b/rust/crucible-app/templates/judge_system.j2 new file mode 100644 index 00000000..00c28f53 --- /dev/null +++ b/rust/crucible-app/templates/judge_system.j2 @@ -0,0 +1,6 @@ +You are a senior Solana security engineer reviewing a colleague's Crucible fuzz-test suite, written to demonstrate a set +of security properties of a Solana program. A Crucible test is an experiment the fuzzer drives: it calls the fixture's +`action_*` methods in arbitrary sequences and, after each, runs the `#[invariant_test]`/`#[crucible_fuzz]` assertions. +Judge whether a GREEN campaign is real evidence for each property — i.e. under which program implementations this suite +would actually FAIL. Use the source-exploration tools to read the program's Rust source and the fixture before asserting +anything about behavior; record durable findings with the memory tool. diff --git a/rust/crucible-app/templates/probe_fn.j2 b/rust/crucible-app/templates/probe_fn.j2 new file mode 100644 index 00000000..1f7e27c7 --- /dev/null +++ b/rust/crucible-app/templates/probe_fn.j2 @@ -0,0 +1,7 @@ + + +#[invariant_test] +fn c_probe(fixture: &mut Fixture) { + let _ = fixture; +} + diff --git a/rust/crucible-app/templates/revise_compile.j2 b/rust/crucible-app/templates/revise_compile.j2 new file mode 100644 index 00000000..e8d62010 --- /dev/null +++ b/rust/crucible-app/templates/revise_compile.j2 @@ -0,0 +1,13 @@ + + +The previous attempt FAILED to build / dry-run. Fix it. +{% if !errors.is_empty() %}COMPILER ERRORS to fix (extracted): +{{ errors }} + +{% endif %}Prior source: +```rust +{{ prev_src }} +``` + +Raw build/dry-run output (tail): +{{ tail }} diff --git a/rust/crucible-app/templates/revise_judge.j2 b/rust/crucible-app/templates/revise_judge.j2 new file mode 100644 index 00000000..0fd9627d --- /dev/null +++ b/rust/crucible-app/templates/revise_judge.j2 @@ -0,0 +1,10 @@ + + +Your previous suite COMPILED but a security reviewer REJECTED it — this is NOT a build error. Revise the tests to +address the review feedback below (each point names a criterion and a concrete fix): +{{ feedback }} + +Prior source: +```rust +{{ prev_src }} +``` diff --git a/rust/crucible-app/templates/test_cheat_sheet.j2 b/rust/crucible-app/templates/test_cheat_sheet.j2 new file mode 100644 index 00000000..4ab9720a --- /dev/null +++ b/rust/crucible-app/templates/test_cheat_sheet.j2 @@ -0,0 +1,27 @@ + +Write ONE Crucible `#[invariant_test]` function holding ALL the program's invariants (no +fixture — it already exists as `Fixture`): + +- `#[invariant_test] fn c_invariants(fixture: &mut Fixture) { ... }` runs AFTER EACH fuzzed + action — put EVERY property's checks in this one fn (conservation, solvency, bounds, access + control, …). One fn = one build = one fuzz run covering all invariants. +- Assert against ON-CHAIN state, not local mirrors: + let acct = fixture.ctx.read_anchor_account::<SomeState>(&fixture.some_pda).unwrap(); + fuzz_assert_le!(acct.balance, cap, "message"); // fuzz_assert_{eq,ne,lt,le,gt,ge} +- Read RAW lamports / data / owner (bypassing Anchor deserialization) via + `read_account` / `get_account`, which return `Result<Account>` — unwrap/`?` FIRST, + THEN the field (there is NO `get_account_lamports`, and `Result` has no `.lamports`): + let lamports = fixture.ctx.read_account(&fixture.some_pda).unwrap().lamports; + Existence check: `fixture.ctx.account_exists(&fixture.some_pda)` (returns bool). +- TRANSACTION FEE — do NOT assert an EXACT lamport delta on a signer. The fee payer (the + FIRST signer of the `action_*`'s `.send()`, e.g. the `depositor`/`authority`) is debited the + tx fee (~5000 lamports/signature) ON TOP OF whatever lamports the instruction moves, so + `depositor_before - amount == depositor_after` is WRONG (off by the fee). Assert exact deltas + on NON-signer accounts (e.g. the vault PDA, which only receives the transfer); for a signer, + subtract the fee or assert a bound (`<=`) instead of `==`. +- Drive state via the fixture's existing `action_*` methods; do not re-`send()` + instructions yourself unless necessary. +- PREFIX each assertion's message with its property title in brackets (`"[<title>] ..."`) so a + counterexample's message names which property it refutes. +- Return ONLY the one annotated `fn c_invariants`. + diff --git a/rust/example-app/Cargo.toml b/rust/example-app/Cargo.toml new file mode 100644 index 00000000..f217c196 --- /dev/null +++ b/rust/example-app/Cargo.toml @@ -0,0 +1,18 @@ +[package] +name = "example-app" +edition.workspace = true +version.workspace = true +license.workspace = true +description = "Demonstration AutoProver application in Rust (the 'echo prover'), built into a Python wheel." + +[lib] +# MUST match the `export_app!` module ident and the maturin module name. +name = "echoprover" +crate-type = ["cdylib"] + +[dependencies] +autoprover-sdk = { path = "../autoprover-sdk" } +serde = { workspace = true } +serde_json = { workspace = true } +# `extension-module` (don't link libpython) + `abi3-py312` (one wheel for cp312+). +pyo3 = { workspace = true, features = ["extension-module", "abi3-py312"] } diff --git a/rust/example-app/pyproject.toml b/rust/example-app/pyproject.toml new file mode 100644 index 00000000..301f03c4 --- /dev/null +++ b/rust/example-app/pyproject.toml @@ -0,0 +1,16 @@ +[build-system] +requires = ["maturin>=1.5,<2.0"] +build-backend = "maturin" + +[project] +name = "echoprover" +version = "0.1.0" +description = "Demonstration AutoProver application implemented in Rust." +requires-python = ">=3.12" +classifiers = ["Programming Language :: Rust"] + +[tool.maturin] +# The compiled module is imported as `echoprover` (matches `export_app!(echoprover, …)`). +module-name = "echoprover" +# abi3 wheel — one artifact for cp312+. +features = ["pyo3/extension-module"] diff --git a/rust/example-app/src/lib.rs b/rust/example-app/src/lib.rs new file mode 100644 index 00000000..5483f567 --- /dev/null +++ b/rust/example-app/src/lib.rs @@ -0,0 +1,113 @@ +//! The "echo prover" — a minimal, self-contained demonstration of a Rust-based +//! AutoProver [`Backend`] on `autoprover-sdk`. It is intentionally not a real +//! verifier: it authors a "spec" from an LLM turn, treats compilation as a no-op, +//! and validates every unit as GOOD — enough to exercise the Python host + FFI +//! round-trip (descriptor, units, author_prompt, compile, validate) without any real +//! toolchain. A production backend keeps this exact shape and swaps the callouts for +//! real ones (see `docs/rust-backend-api.md`). + +use std::path::Path; + +use autoprover_sdk::{ + AppDescriptor, ArtifactLayout, AuthorInput, Backend, CompileResult, CoreSlot, EventKind, + Failure, PhaseSpec, Prompt, Sandbox, Unit, ValidateOutcome, Verdict, +}; + +struct EchoApp; + +impl Backend for EchoApp { + fn descriptor(&self) -> AppDescriptor { + AppDescriptor { + name: "echoprover".to_string(), + header_text: "Echo Prover (Rust demo) | AutoProver".to_string(), + ecosystem: "evm".to_string(), + backend_tag: "echoprover".to_string(), + backend_guidance: "These properties are checked by the echo backend, a demo that \ + accepts any well-formed spec. Feel free to state universal properties." + .to_string(), + analysis_key: "echoprover-analysis".to_string(), + phases: vec![ + PhaseSpec { key: "analysis".into(), label: "System Analysis".into(), order: 0, core_slot: Some(CoreSlot::Analysis) }, + PhaseSpec { key: "extraction".into(), label: "Property Extraction".into(), order: 1, core_slot: Some(CoreSlot::Extraction) }, + // A UI-only phase with no core slot (cf. autoprove's harness/autosetup). + PhaseSpec { key: "solving".into(), label: "Solving".into(), order: 2, core_slot: None }, + PhaseSpec { key: "formalization".into(), label: "Formalization".into(), order: 3, core_slot: Some(CoreSlot::Formalization) }, + PhaseSpec { key: "report".into(), label: "Report".into(), order: 4, core_slot: Some(CoreSlot::Report) }, + ], + args: vec![autoprover_sdk::ArgSpec { + flag: "--echo-tag".to_string(), + help: "An arbitrary tag stamped into the echo spec.".to_string(), + default: autoprover_sdk::ArgDefault::Str { value: Some("demo".to_string()) }, + required: false, + }], + rag_db_default: None, + event_kinds: vec![EventKind::log("solver_line", "Solver")], + artifact_layout: ArtifactLayout { + deliverable_dir: "certora/echo".into(), + internal_dir: ".certora_internal/echo".into(), + report_dir: "certora/echo/reports".into(), + artifact_dir: "certora/echo/specs".into(), + artifact_prefix: "echospec".into(), + artifact_extension: "espec".into(), + property_suffix: "property_rules".into(), + deliverable_primary: None, + }, + // A simple per-component wheel: no shared setup, one file per component, no toolchain + // confinement/serialization (its compile/validate are pure). All defaults. + setup: None, + deliverable_mode: autoprover_sdk::DeliverableMode::PerComponent, + serialize_toolchain: false, + confine_by_default: false, + component_noun: None, + } + } + + fn units(&self, input: &AuthorInput) -> Vec<Unit> { + input + .props + .iter() + .enumerate() + .map(|(i, p)| { + let slug = if p.slug.is_empty() { format!("p{i}") } else { p.slug.clone() }; + Unit { property: p.title.clone(), unit: format!("rule_{slug}"), target: None } + }) + .collect() + } + + fn author_prompt(&self, input: &AuthorInput, failure: Option<&Failure>) -> Prompt { + let titles: Vec<&str> = input.props.iter().map(|p| p.title.as_str()).collect(); + let mut instruction = format!( + "Author a spec with a rule per property: {}. Return the spec source only.", + titles.join(", ") + ); + if let Some(f) = failure { + instruction.push_str(&format!("\n\nThe previous attempt was rejected: {}", f.errors)); + } + Prompt { system: None, instruction } + } + + fn compile( + &self, + _input: &AuthorInput, + _spec: &str, + _workdir: &Path, + _sandbox: &Sandbox, + ) -> CompileResult { + // The demo accepts any well-formed spec — no build gate. + CompileResult::Ok + } + + fn validate( + &self, + _input: &AuthorInput, + _spec: &str, + unit: &str, + _workdir: &Path, + _sandbox: &Sandbox, + ) -> ValidateOutcome { + // Self-contained: any well-formed spec builds; the (own-target) unit "passes". + ValidateOutcome::Verdicts { verdicts: vec![(unit.to_string(), Verdict::with_outcome("GOOD"))] } + } +} + +autoprover_sdk::export_app!(echoprover, EchoApp); diff --git a/rust/run-confined/Cargo.toml b/rust/run-confined/Cargo.toml new file mode 100644 index 00000000..b02f9e46 --- /dev/null +++ b/rust/run-confined/Cargo.toml @@ -0,0 +1,19 @@ +[package] +name = "run-confined" +edition.workspace = true +version.workspace = true +license.workspace = true +description = "Trusted launcher that confines a command (Landlock filesystem + seccomp network/ptrace + rlimits + scrubbed env), then execs it. The first SandboxProvider for the RunCommand effect — see docs/command-sandbox.md." + +[[bin]] +name = "run-confined" +path = "src/main.rs" + +[dependencies] +# Filesystem confinement (ABI negotiation + full access-right set). +landlock = "0.4" +# seccomp-BPF compiler from AWS Firecracker — we hand it a small allow/deny +# policy, not raw bytecode. +seccompiler = "0.5" +# setrlimit / prctl / syscall numbers / AF_* constants. +libc = "0.2" diff --git a/rust/run-confined/src/main.rs b/rust/run-confined/src/main.rs new file mode 100644 index 00000000..bb296137 --- /dev/null +++ b/rust/run-confined/src/main.rs @@ -0,0 +1,288 @@ +//! `run-confined` — the trusted launcher for the `RunCommand` sandbox. +//! +//! It applies four unprivileged, in-kernel confinements to *itself*, then `execve`s +//! the requested command (which inherits all of them across the exec): +//! +//! 1. **Landlock** — a filesystem ruleset: default-deny, then grant `--rw` paths +//! full access and `--ro` paths read+execute. Confines reads *and* writes and, +//! by not granting `/proc`, closes the same-uid `/proc/<parent>/environ` leak. +//! 2. **seccomp** — deny inet-domain `socket()` (blocks TCP, UDP/DNS, and the EC2 +//! metadata endpoint) and `ptrace`/`process_vm_readv`/`process_vm_writev`. +//! 3. **env allowlist** — `execve` with only `--allow-env` variables (a scrubbed +//! environment). +//! 4. **rlimits** — `--rlimit-*` caps on address space / CPU-seconds / pids / file size. +//! +//! This is trusted code: its argv is authored by the Python side (never the LLM, +//! which controls only file *contents*). It is **fail-closed** — any setup failure, +//! or a kernel without Landlock, exits nonzero *without* execing the command, so +//! untrusted input never runs unconfined. +//! +//! See `docs/command-sandbox.md` (§6) for the design and the validation matrix. + +use std::collections::BTreeMap; +use std::os::unix::process::CommandExt; +use std::path::PathBuf; +use std::process::Command; + +use landlock::{ + Access, AccessFs, CompatLevel, Compatible, PathBeneath, PathFd, Ruleset, RulesetAttr, + RulesetCreatedAttr, RulesetStatus, ABI, +}; +use seccompiler::{ + apply_filter, BpfProgram, SeccompAction, SeccompCmpArgLen, SeccompCmpOp, SeccompCondition, + SeccompFilter, SeccompRule, TargetArch, +}; + +/// Bad command line (a programming error on the trusted caller's side). +const EXIT_USAGE: i32 = 2; +/// The sandbox could not be established — fail-closed, command NOT run. +const EXIT_SANDBOX_UNAVAILABLE: i32 = 3; +/// The confined `execve` itself failed (e.g. program not found on PATH). +const EXIT_EXEC_FAILED: i32 = 127; + +/// `landlock_create_ruleset(NULL, 0, LANDLOCK_CREATE_RULESET_VERSION)` returns the +/// kernel's supported ABI version, or fails if Landlock is unavailable. +const LANDLOCK_CREATE_RULESET_VERSION: libc::c_ulong = 1; + +#[derive(Default)] +struct Config { + rw_paths: Vec<PathBuf>, + ro_paths: Vec<PathBuf>, + env: Vec<(String, String)>, + allow_network: bool, + rlimit_as: Option<u64>, + rlimit_cpu: Option<u64>, + rlimit_nproc: Option<u64>, + rlimit_fsize: Option<u64>, + program: String, + args: Vec<String>, +} + +fn die(code: i32, msg: &str) -> ! { + eprintln!("run-confined: {msg}"); + std::process::exit(code); +} + +fn main() { + let argv: Vec<String> = std::env::args().skip(1).collect(); + + if argv.first().map(String::as_str) == Some("--probe") { + probe(); + } + + let cfg = parse(&argv).unwrap_or_else(|e| die(EXIT_USAGE, &e)); + + // Order matters: rlimits + env are harmless early; apply Landlock, then seccomp + // LAST so our own setup syscalls aren't caught by the filter; then exec. + set_rlimits(&cfg); + set_no_new_privs(); + if let Err(e) = apply_landlock(&cfg) { + die(EXIT_SANDBOX_UNAVAILABLE, &format!("Landlock setup failed: {e}")); + } + if let Err(e) = apply_seccomp(&cfg) { + die(EXIT_SANDBOX_UNAVAILABLE, &format!("seccomp setup failed: {e}")); + } + + let mut cmd = Command::new(&cfg.program); + cmd.args(&cfg.args).env_clear().envs(cfg.env.iter().cloned()); + // `exec` replaces this process image; it only returns on failure. + let err = cmd.exec(); + die(EXIT_EXEC_FAILED, &format!("exec {:?} failed: {err}", cfg.program)); +} + +/// `--probe`: report whether the kernel supports Landlock. Exit 0 + print the ABI +/// if so; exit `EXIT_SANDBOX_UNAVAILABLE` otherwise. Drives Python's fail-closed +/// `available()` check without restricting this (throwaway) process. +fn probe() -> ! { + let abi = unsafe { + libc::syscall( + libc::SYS_landlock_create_ruleset, + std::ptr::null::<libc::c_void>(), + 0usize, + LANDLOCK_CREATE_RULESET_VERSION, + ) + }; + if abi > 0 { + println!("landlock abi {abi}"); + std::process::exit(0); + } + die( + EXIT_SANDBOX_UNAVAILABLE, + "kernel does not support Landlock (need Linux >= 5.13); refusing to run unconfined", + ); +} + +fn parse(argv: &[String]) -> Result<Config, String> { + let mut cfg = Config::default(); + let mut i = 0; + + let take = |i: &mut usize, flag: &str| -> Result<String, String> { + *i += 1; + argv.get(*i) + .cloned() + .ok_or_else(|| format!("{flag} requires a value")) + }; + let parse_u64 = |s: &str, flag: &str| -> Result<u64, String> { + s.parse::<u64>().map_err(|_| format!("{flag} expects an integer, got {s:?}")) + }; + + while i < argv.len() { + match argv[i].as_str() { + "--rw" => cfg.rw_paths.push(PathBuf::from(take(&mut i, "--rw")?)), + "--ro" => cfg.ro_paths.push(PathBuf::from(take(&mut i, "--ro")?)), + "--allow-network" => cfg.allow_network = true, + "--rlimit-as" => cfg.rlimit_as = Some(parse_u64(&take(&mut i, "--rlimit-as")?, "--rlimit-as")?), + "--rlimit-cpu" => cfg.rlimit_cpu = Some(parse_u64(&take(&mut i, "--rlimit-cpu")?, "--rlimit-cpu")?), + "--rlimit-nproc" => cfg.rlimit_nproc = Some(parse_u64(&take(&mut i, "--rlimit-nproc")?, "--rlimit-nproc")?), + "--rlimit-fsize" => cfg.rlimit_fsize = Some(parse_u64(&take(&mut i, "--rlimit-fsize")?, "--rlimit-fsize")?), + "--allow-env" => { + let spec = take(&mut i, "--allow-env")?; + if let Some((name, value)) = spec.split_once('=') { + cfg.env.push((name.to_string(), value.to_string())); + } else if let Ok(value) = std::env::var(&spec) { + // NAME with no '=': pass through from the current environment if set. + cfg.env.push((spec, value)); + } + // NAME not present in the environment: silently skip (nothing to pass). + } + "--" => { + i += 1; + if i >= argv.len() { + return Err("no program given after `--`".to_string()); + } + cfg.program = argv[i].clone(); + cfg.args = argv[i + 1..].to_vec(); + return Ok(cfg); + } + other => return Err(format!("unknown flag {other:?} (did you forget `--` before the command?)")), + } + i += 1; + } + Err("missing `--` and command to run".to_string()) +} + +fn set_rlimits(cfg: &Config) { + let set = |resource: libc::__rlimit_resource_t, value: u64| { + let lim = libc::rlimit { rlim_cur: value, rlim_max: value }; + // Best-effort: a failure to *lower* a limit is not worth aborting the run over. + unsafe { libc::setrlimit(resource, &lim) }; + }; + if let Some(v) = cfg.rlimit_as { + set(libc::RLIMIT_AS, v); + } + if let Some(v) = cfg.rlimit_cpu { + set(libc::RLIMIT_CPU, v); + } + if let Some(v) = cfg.rlimit_nproc { + set(libc::RLIMIT_NPROC, v); + } + if let Some(v) = cfg.rlimit_fsize { + set(libc::RLIMIT_FSIZE, v); + } +} + +fn set_no_new_privs() { + // Required before loading a seccomp filter (and by Landlock) for an unprivileged + // process; ensures no exec can regain privileges. + unsafe { libc::prctl(libc::PR_SET_NO_NEW_PRIVS, 1, 0, 0, 0) }; +} + +fn apply_landlock(cfg: &Config) -> Result<(), String> { + // Handle the full access-right set the crate knows; BestEffort tolerates a kernel + // that lacks the newest rights, but we still require Landlock to be *enforcing* + // at all (checked below) — otherwise we would silently run unconfined. + let abi = ABI::V5; + + let mut created = Ruleset::default() + .set_compatibility(CompatLevel::BestEffort) + .handle_access(AccessFs::from_all(abi)) + .map_err(|e| e.to_string())? + .create() + .map_err(|e| e.to_string())?; + + for p in &cfg.ro_paths { + match PathFd::new(p) { + Ok(fd) => { + created = created + .add_rule(PathBeneath::new(fd, AccessFs::from_read(abi))) + .map_err(|e| e.to_string())?; + } + Err(e) => eprintln!("run-confined: skipping missing --ro path {p:?}: {e}"), + } + } + for p in &cfg.rw_paths { + match PathFd::new(p) { + Ok(fd) => { + created = created + .add_rule(PathBeneath::new(fd, AccessFs::from_all(abi))) + .map_err(|e| e.to_string())?; + } + Err(e) => return Err(format!("required --rw path {p:?} is unopenable: {e}")), + } + } + + let status = created.restrict_self().map_err(|e| e.to_string())?; + if matches!(status.ruleset, RulesetStatus::NotEnforced) { + return Err("kernel did not enforce Landlock (need Linux >= 5.13)".to_string()); + } + Ok(()) +} + +fn apply_seccomp(cfg: &Config) -> Result<(), String> { + let mut rules: BTreeMap<i64, Vec<SeccompRule>> = BTreeMap::new(); + + if !cfg.allow_network { + // Deny socket() for AF_INET / AF_INET6 (arg 0) — blocks TCP, UDP (incl. DNS), + // and the IMDS endpoint. AF_UNIX and other local families still work. + let cond = |domain: i32| -> Result<SeccompRule, String> { + SeccompRule::new(vec![SeccompCondition::new( + 0, + SeccompCmpArgLen::Dword, + SeccompCmpOp::Eq, + domain as u64, + ) + .map_err(|e| e.to_string())?]) + .map_err(|e| e.to_string()) + }; + rules.insert( + libc::SYS_socket as i64, + vec![cond(libc::AF_INET)?, cond(libc::AF_INET6)?], + ); + } + + // Deny cross-process memory/ptrace (belt-and-suspenders to Landlock's own + // out-of-domain ptrace restriction). An empty rule vec = match unconditionally. + for nr in [ + libc::SYS_ptrace, + libc::SYS_process_vm_readv, + libc::SYS_process_vm_writev, + ] { + rules.insert(nr as i64, Vec::new()); + } + + let filter = SeccompFilter::new( + rules, + SeccompAction::Allow, // default: allow syscalls we didn't name + SeccompAction::Errno(libc::EPERM as u32), // named + matched: deny with EPERM + target_arch(), + ) + .map_err(|e| e.to_string())?; + + let program: BpfProgram = filter.try_into().map_err(|e: seccompiler::BackendError| e.to_string())?; + apply_filter(&program).map_err(|e| e.to_string()) +} + +fn target_arch() -> TargetArch { + #[cfg(target_arch = "x86_64")] + { + TargetArch::x86_64 + } + #[cfg(target_arch = "aarch64")] + { + TargetArch::aarch64 + } + #[cfg(not(any(target_arch = "x86_64", target_arch = "aarch64")))] + { + compile_error!("run-confined supports only x86_64 and aarch64") + } +} diff --git a/scripts/Dockerfile b/scripts/Dockerfile index bf94e041..cf195393 100644 --- a/scripts/Dockerfile +++ b/scripts/Dockerfile @@ -59,6 +59,16 @@ RUN set -eux; \ ( cd "$target" && /venv/bin/sphinx-build -M singlehtml . tmp ); \ cp "$target/tmp/singlehtml/index.html" /out/cvl.html +# ---- Stage 1b: build the `run-confined` command-sandbox launcher ---- +# Crucible confines every RunCommand via this small Rust binary (Landlock + seccomp; +# docs/command-sandbox.md). It must be on PATH in the runtime image or Crucible +# fail-closes. Built in a throwaway Rust stage; only the binary is copied out. +FROM --platform=linux/amd64 rust:1-slim AS sandbox-builder +WORKDIR /build +COPY rust/ ./rust/ +# Only run-confined (+ its landlock/seccompiler/libc deps) — not the pyo3 members. +RUN cd rust && cargo build -p run-confined --release + # ---- Stage 2: final runtime image ---- FROM --platform=linux/amd64 python:3.12-slim AS final @@ -85,6 +95,8 @@ RUN mkdir -p $HOME/.cache/huggingface \ # postgresql-client provides `psql` for the entrypoint's setup-db; git is # occasionally shelled out to by tooling. No openssh-client / SSH — the # Python install pulls from PyPI + the vendored graphcore submodule. +# (The `RunCommand` sandbox uses in-kernel Landlock + seccomp — no package +# needed; see docs/command-sandbox.md §6.) RUN apt-get update \ && apt-get install -y --no-install-recommends \ git ca-certificates \ @@ -149,6 +161,15 @@ RUN --mount=type=cache,id=uv-cache,target=/root/.cache/uv \ # Pre-built CVL HTML — the entrypoint's setup-db reads this to populate rag_db. COPY --from=docs-builder /out/cvl.html $AUTOPROVE_HOME/prover-docs/cvl.html +# Pre-built crucible_kb RAG manifest, committed in the crucible app dir — the entrypoint's +# setup-db imports it (composer.scripts.rag_import) to populate the crucible_rag schema. No +# crucible checkout is needed at build or run time; the corpus travels as this committed JSON. +COPY rust/crucible-app/crucible_kb.rag.json $AUTOPROVE_HOME/crucible_kb.rag.json + +# The command-sandbox launcher, on PATH (/usr/local/bin is in PATH above). Crucible's +# default provider is `launcher`, so this must be present or Crucible fail-closes. +COPY --from=sandbox-builder /build/rust/target/release/run-confined /usr/local/bin/run-confined + # Pre-download the sentence-transformers embedding model so first runs are # offline-fast. nomic-embed-text-v1.5 ships custom modeling code, hence # trust_remote_code. Then make the cache world-writable so the HF library can diff --git a/scripts/autoprove-entrypoint.sh b/scripts/autoprove-entrypoint.sh index ac8b8a3d..9596d298 100755 --- a/scripts/autoprove-entrypoint.sh +++ b/scripts/autoprove-entrypoint.sh @@ -29,6 +29,7 @@ export USER=autoprove LOGNAME=autoprove PGHOST="${CERTORA_AI_COMPOSER_PGHOST:-postgres}" PGPORT="${CERTORA_AI_COMPOSER_PGPORT:-5432}" RAG_CONN="postgresql://rag_user:rag_password@${PGHOST}:${PGPORT}/rag_db" +CRUCIBLE_RAG_CONN="postgresql://crucible_rag_user:rag_password@${PGHOST}:${PGPORT}/rag_db" if [[ "${1:-}" == "setup-db" ]]; then shift @@ -53,6 +54,10 @@ if [[ "${1:-}" == "setup-db" ]]; then python -m composer.scripts.ragbuild \ --output "$RAG_CONN" \ "$AUTOPROVE_HOME/prover-docs/cvl.html" + echo "[autoprove] populating crucible_kb RAG at ${CRUCIBLE_RAG_CONN} ..." + python -m composer.scripts.rag_import \ + --output "$CRUCIBLE_RAG_CONN" \ + "$AUTOPROVE_HOME/crucible_kb.rag.json" echo "[autoprove] populating LangGraph knowledge base ..." python -m composer.scripts.kb_populate echo "[autoprove] setup-db done." diff --git a/test_scenarios/solana_vault/.gitignore b/test_scenarios/solana_vault/.gitignore new file mode 100644 index 00000000..682a37fc --- /dev/null +++ b/test_scenarios/solana_vault/.gitignore @@ -0,0 +1,19 @@ +# Pipeline-generated deliverables/diagnostics (written by a run; not source). +/certora/ +/.certora_internal/ + +# Build outputs (cargo-build-sbf / crucible). +/target/ +Cargo.lock +crashes/ +# Fuzzing artifacts (crucible run: corpus + explore-mode ./corpus, ./output). +/corpus/ +/output/ +# The fuzz harness is generated by the gate/backend (its Cargo.toml pins a +# machine-resolved crucible checkout path); not committed. +/fuzz/ +# Ephemeral per-run sandbox scratch created in the workdir by the command sandbox +# (docs/command-sandbox.md §3): a private CARGO_HOME + linker TMPDIR. +/.sandbox_cargo/ +/.sandbox_tmp/ + diff --git a/test_scenarios/solana_vault/Cargo.toml b/test_scenarios/solana_vault/Cargo.toml new file mode 100644 index 00000000..ee3db645 --- /dev/null +++ b/test_scenarios/solana_vault/Cargo.toml @@ -0,0 +1,16 @@ +# Workspace for the lamports-vault example program. Built to sBPF with +# `cargo build-sbf` (→ target/deploy/vault.so), which the Crucible fuzz harness +# under fuzz/vault/ loads. Modeled on crucible's examples/escrow. +[workspace] +members = ["programs/*"] +resolver = "2" + +[profile.release] +overflow-checks = true +lto = "fat" +codegen-units = 1 + +[profile.release.build-override] +opt-level = 3 +incremental = false +codegen-units = 1 diff --git a/test_scenarios/solana_vault/programs/vault/Cargo.toml b/test_scenarios/solana_vault/programs/vault/Cargo.toml new file mode 100644 index 00000000..0fc8d084 --- /dev/null +++ b/test_scenarios/solana_vault/programs/vault/Cargo.toml @@ -0,0 +1,21 @@ +[package] +name = "vault" +version = "0.1.0" +description = "Lamports vault example program (AutoProver Crucible gate)" +edition = "2021" + +[lib] +crate-type = ["cdylib", "lib"] +name = "vault" + +[features] +default = [] +cpi = ["no-entrypoint"] +no-entrypoint = [] +no-idl = [] +no-log-ix-name = [] +idl-build = ["anchor-lang/idl-build"] +test-sbf = [] + +[dependencies] +anchor-lang = "1.0.1" diff --git a/test_scenarios/solana_vault/programs/vault/src/lib.rs b/test_scenarios/solana_vault/programs/vault/src/lib.rs new file mode 100644 index 00000000..503cd7c5 --- /dev/null +++ b/test_scenarios/solana_vault/programs/vault/src/lib.rs @@ -0,0 +1,113 @@ +//! A minimal Anchor "lamports vault" program. +//! +//! A user creates a vault (a PDA) with themselves as the authority, deposits lamports into it, +//! and later withdraws. Only the vault's authority may withdraw. Illustrative — not audited. + +#![allow(unexpected_cfgs)] +use anchor_lang::prelude::*; +use anchor_lang::prelude::program::invoke; +use anchor_lang::solana_program::system_instruction; + +declare_id!("BdmwBcVB95UpLzXFwqRnbeJBsrMDLKB4sgJb123oxUoj"); + +#[program] +pub mod vault_program { + use super::*; + + /// Create a vault PDA owned by `authority`. Fails if the vault already exists. + pub fn initialize(ctx: Context<Initialize>) -> Result<()> { + let vault = &mut ctx.accounts.vault; + vault.authority = ctx.accounts.authority.key(); + vault.balance = 0; + vault.bump = ctx.bumps.vault; + Ok(()) + } + + /// Deposit `amount` lamports from the depositor into the vault PDA. + pub fn deposit(ctx: Context<Deposit>, amount: u64) -> Result<()> { + invoke( + &system_instruction::transfer( + &ctx.accounts.depositor.key(), + &ctx.accounts.vault.key(), + amount, + ), + &[ + ctx.accounts.depositor.to_account_info(), + ctx.accounts.vault.to_account_info(), + ctx.accounts.system_program.to_account_info(), + ], + )?; + let vault = &mut ctx.accounts.vault; + vault.balance = vault.balance.checked_add(amount).ok_or(VaultError::Overflow)?; + Ok(()) + } + + /// Withdraw `amount` lamports from the vault to the authority. Only the authority may call. + pub fn withdraw(ctx: Context<Withdraw>, amount: u64) -> Result<()> { + let vault = &mut ctx.accounts.vault; + require!(amount <= vault.balance, VaultError::InsufficientFunds); + + **vault.to_account_info().try_borrow_mut_lamports()? -= amount; + **ctx.accounts.authority.to_account_info().try_borrow_mut_lamports()? += amount; + vault.balance = vault.balance.checked_sub(amount).ok_or(VaultError::Overflow)?; + Ok(()) + } +} + +#[account] +pub struct VaultState { + /// The only key allowed to withdraw. + pub authority: Pubkey, + /// Lamports recorded as deposited (mirrors the PDA's spendable lamports). + pub balance: u64, + pub bump: u8, +} + +impl VaultState { + pub const SIZE: usize = 32 + 8 + 1; +} + +#[derive(Accounts)] +pub struct Initialize<'info> { + #[account( + init, + payer = authority, + space = 8 + VaultState::SIZE, + seeds = [b"vault", authority.key().as_ref()], + bump, + )] + pub vault: Account<'info, VaultState>, + #[account(mut)] + pub authority: Signer<'info>, + pub system_program: Program<'info, System>, +} + +#[derive(Accounts)] +pub struct Deposit<'info> { + #[account(mut, seeds = [b"vault", vault.authority.as_ref()], bump = vault.bump)] + pub vault: Account<'info, VaultState>, + #[account(mut)] + pub depositor: Signer<'info>, + pub system_program: Program<'info, System>, +} + +#[derive(Accounts)] +pub struct Withdraw<'info> { + #[account( + mut, + seeds = [b"vault", authority.key().as_ref()], + bump = vault.bump, + has_one = authority, + )] + pub vault: Account<'info, VaultState>, + #[account(mut)] + pub authority: Signer<'info>, +} + +#[error_code] +pub enum VaultError { + #[msg("arithmetic overflow")] + Overflow, + #[msg("insufficient funds in vault")] + InsufficientFunds, +} diff --git a/test_scenarios/solana_vault/rust-toolchain.toml b/test_scenarios/solana_vault/rust-toolchain.toml new file mode 100644 index 00000000..edc4de49 --- /dev/null +++ b/test_scenarios/solana_vault/rust-toolchain.toml @@ -0,0 +1,4 @@ +[toolchain] +channel = "1.89.0" +components = ["rustfmt", "clippy"] +profile = "minimal" diff --git a/test_scenarios/solana_vault/system.md b/test_scenarios/solana_vault/system.md new file mode 100644 index 00000000..ef18f54d --- /dev/null +++ b/test_scenarios/solana_vault/system.md @@ -0,0 +1,31 @@ +# Lamports Vault + +A minimal Solana program (Anchor) that lets a user custody SOL in a program-derived vault. + +## Overview + +Each user owns a **vault**, a PDA derived from `["vault", authority]`. The vault records its +`authority` (the owner) and a `balance`. There is one program, `vault`. + +## Instructions + +- **initialize** — creates the caller's vault PDA, sets `authority` to the caller, and `balance` + to 0. A given authority has exactly one vault; initializing twice must fail. +- **deposit(amount)** — transfers `amount` lamports from the depositor into the vault (via a + System Program transfer) and increases `balance` by `amount`. Anyone may deposit into any vault. +- **withdraw(amount)** — transfers `amount` lamports from the vault back to the authority and + decreases `balance`. Only the vault's `authority`, who must sign, may withdraw, and only up to + the recorded `balance`. + +## Requirements + +- Only the account recorded in `vault.authority` (and it must sign) can withdraw from that vault. +- A withdrawal never removes more than the vault's recorded `balance`. +- The vault's recorded `balance` tracks the net of deposits minus withdrawals and never + underflows or overflows. +- The vault PDA is always the canonical PDA of `["vault", authority]` for its recorded authority. + +## Actors + +- **Vault authority** — a user keypair; the owner of a vault and the only withdrawer. +- **System Program** — the standard Solana program used to move lamports on deposit. diff --git a/tests/test_agent_index.py b/tests/test_agent_index.py index b53c96c0..1745eb8b 100644 --- a/tests/test_agent_index.py +++ b/tests/test_agent_index.py @@ -8,8 +8,6 @@ semantic similarity. """ -from __future__ import annotations - from typing import Callable import pytest diff --git a/tests/test_autoprove_integration.py b/tests/test_autoprove_integration.py index 20d6bb7f..6b1de3d7 100644 --- a/tests/test_autoprove_integration.py +++ b/tests/test_autoprove_integration.py @@ -56,6 +56,8 @@ "solc": "solc", "verify": "Counter:certora/specs/sanity-Counter.spec", "wait_for_results": "none", + "server": "production", + "prover_version": "master" } # AutoSetup's summaries spec, relative to certora/ (the SetupSuccess contract). _SUMMARIES_REL = "specs/summaries/Counter_base_summaries.spec" diff --git a/tests/test_autoprove_report.py b/tests/test_autoprove_report.py index 9849699a..4b4ca5a2 100644 --- a/tests/test_autoprove_report.py +++ b/tests/test_autoprove_report.py @@ -378,6 +378,24 @@ def _mini_report() -> AutoProverReport: skipped=skipped, coverage=cov) +def test_render_html_shows_finding_message(): + # A backend diagnostic (the Crucible fuzzer's counterexample / failed-assertion message) is + # surfaced on the rule row, so a non-GOOD verdict explains itself in the report. + p1 = _fp("C", "p_dep", [("c.spec", "c_deposit")], desc="deposit increases balance") + rules = [RuleVerdict(name="c_deposit", spec_file="c.spec", outcome=Outcome.BAD, + message="crash abc: deposit(5) - expected 105 got 100")] + groups = [PropertyGroup(slug="deposits", title="Deposits", description="d", + status=GroupStatus.BAD, members=[("C", "p_dep")])] + cov = CoverageReport(total_properties=1, total_rules=1, total_groups=1, + properties_per_group_min=1, properties_per_group_max=1, + property_coverage_complete=True) + h = render_html(AutoProverReport(contract_name="Vault", backend="crucible", + properties=[p1], rules=rules, groups=groups, + skipped=[], coverage=cov)) + assert 'class="finding"' in h + assert "crash abc: deposit(5) - expected 105 got 100" in h + + def test_render_html_group_rows_and_edge_labels(): h = render_html(_mini_report()) assert "deposit-openness" in h and "Deposit is open" in h diff --git a/tests/test_crucible_e2e_gate.py b/tests/test_crucible_e2e_gate.py new file mode 100644 index 00000000..ee8bd4da --- /dev/null +++ b/tests/test_crucible_e2e_gate.py @@ -0,0 +1,161 @@ +"""Phase-5 gate: the WHOLE Crucible vertical, with a REAL model. + +Runs the whole vertical via the generic host (`build_application` + `run_application`) on +the `solana_vault` scenario — the SOLANA ecosystem front half (analysis → property +extraction) → the crucible_app wheel (shared fixture via the setup step, then per-component +test authoring + fuzzing) → report — exactly as `console-crucible` would. Pass = it analyzes the program +into instructions, extracts properties, and produces per-component fuzz verdicts +with no human edits. + +The heaviest, most expensive gate: real LLM across every phase + a fuzz campaign per +instruction. Same prerequisites as the other crucible gates (toolchain, `crucible`, +`CRUCIBLE_REPO`). Run in the background. + + CRUCIBLE_REPO=/path/to/crucible \ + .venv/bin/python -m pytest tests/test_crucible_e2e_gate.py -m expensive -q -s +""" + +import asyncio +import json +import os +import shutil +from pathlib import Path +from types import SimpleNamespace +from typing import Any, cast, TYPE_CHECKING + +import psycopg +import pytest +from psycopg.sql import SQL, Identifier, Literal + +import composer.workflow.services as services +from composer.kb.knowledge_base import DefaultEmbedder +from composer.pipeline.core import Delivered +from composer.pipeline.ecosystem import RUST_FORBIDDEN_READ +from composer.rustapp.frontend import GenericRustConsoleHandler +from composer.rustapp.host import build_application, run_application +from composer.sandbox.config import SandboxConfig +from composer.spec.context import SourceCode, WorkflowContext +from composer.spec.service_host import ModelProvider, PureServiceHost +from composer.llm.registry import get_provider_for +from composer.spec.source.source_env import build_basic_source_tools, build_source_tools +from composer.spec.system_model import SolidityIdentifier +from composer.ui.tool_display import async_tool_context +from composer.workflow.services import llm_factory, standard_connections + +from tests.conftest import MockSentenceTransformer, needs_postgres +from tests.test_autoprove_integration import _MEMORIES_DDL, _RAG_DB, _VECTOR_DBS, _db_url + +if TYPE_CHECKING: + from testcontainers.postgres import PostgresContainer + +pytestmark = [pytest.mark.expensive, needs_postgres, pytest.mark.asyncio] + +_SCENARIO = Path(__file__).parent.parent / "test_scenarios" / "solana_vault" +_PROGRAM = "vault" + + +def _model_args() -> object: + return SimpleNamespace( + heavy_model="claude-opus-4-6", lite_model="claude-sonnet-4-6", + tokens=128_000, thinking_tokens=2048, memory_tool=False, interleaved_thinking=False, + ) + + +def _crucible_repo() -> Path | None: + raw = os.environ.get("CRUCIBLE_REPO") + if not raw: + return None + repo = Path(raw) + return repo if (repo / "crates" / "crucible-fuzzer").is_dir() else None + + +def _require(cond: bool, why: str) -> None: + if not cond: + pytest.skip(why) + + +async def test_crucible_full_vertical(pg_container: "PostgresContainer", monkeypatch): + _require(_SCENARIO.is_dir(), f"scenario missing: {_SCENARIO}") + _require(shutil.which("cargo-build-sbf") is not None, "cargo-build-sbf not on PATH") + _require(shutil.which("crucible") is not None, "crucible CLI not on PATH") + _require(_crucible_repo() is not None, "set CRUCIBLE_REPO to a local crucible checkout") + + admin_url = pg_container.get_connection_url(driver=None) + with psycopg.connect(admin_url, autocommit=True) as admin: + for cfg in services._DATABASE_CONFIGS.values(): + admin.execute(SQL("CREATE ROLE {} LOGIN PASSWORD {}").format( + Identifier(cfg["user"]), Literal(cfg["password"]))) + admin.execute(SQL("CREATE DATABASE {} OWNER {}").format( + Identifier(cfg["database"]), Identifier(cfg["user"]))) + admin.execute(SQL("CREATE DATABASE {}").format(Identifier(_RAG_DB))) + for db in _VECTOR_DBS: + with psycopg.connect(_db_url(pg_container, db), autocommit=True) as conn: + conn.execute("CREATE EXTENSION IF NOT EXISTS vector") + mem = services._DATABASE_CONFIGS["memory"] + mem_url = ( + f"postgresql://{mem['user']}:{mem['password']}" + f"@{pg_container.get_container_host_ip()}:{pg_container.get_exposed_port(5432)}/{mem['database']}" + ) + with psycopg.connect(mem_url, autocommit=True) as conn: + conn.execute(_MEMORIES_DDL) + + monkeypatch.setenv("CERTORA_AI_COMPOSER_PGHOST", pg_container.get_container_host_ip()) + monkeypatch.setenv("CERTORA_AI_COMPOSER_PGPORT", str(pg_container.get_exposed_port(5432))) + + args = _model_args() + async with ( + standard_connections(provider="anthropic", embedder=DefaultEmbedder(MockSentenceTransformer())) as conns, + async_tool_context(), + ): + content = await conns.uploader.get_document(_SCENARIO / "system.md") + assert content is not None + source = SourceCode( + content=content, project_root=str(_SCENARIO), + contract_name=SolidityIdentifier(_PROGRAM), + relative_path=f"programs/{_PROGRAM}/src/lib.rs", forbidden_read=RUST_FORBIDDEN_READ, + ) + _tiered = get_provider_for(tiered=cast(Any, args)) + model_provider = ModelProvider( + heavy_model=_tiered.heavy, lite_model=_tiered.lite, checkpointer=conns.checkpointer, + ) + basic = build_basic_source_tools(root=str(_SCENARIO), forbidden_read=RUST_FORBIDDEN_READ) + full = build_source_tools(basic, model_provider, conns.indexed_store, ("crucible_e2e", "src"), recursion_limit=100) + env = PureServiceHost(models=model_provider, rag_tools=(), sort="existing").bind_source_tools(full) + ctx = WorkflowContext.create( + services=conns.memory, + thread_id="crucible_e2e", store=conns.store, recursion_limit=100, + cache_namespace=None, memory_namespace=None, + ) + + handler = GenericRustConsoleHandler({"fuzz_pulse", "fuzz_finding", "build_output"}) + + # The whole vertical via the generic host — `console-crucible` drives exactly this (the + # wheel's workspace_prep builds the .so + warms deps; setup authors the fixture; per-unit + # validate fuzzes). Mirror what the entry point does: thread the fuzz budget in as a + # declared arg and build the launcher policy from the wheel's sandbox grants. + app = build_application("crucible_app", command_timeout_s=1800) + app.options.declared_args = {"fuzz_timeout": 12} + grants = json.loads(app.module.sandbox_grants("{}")) + app.options.sandbox = SandboxConfig( + provider=os.environ.get("COMPOSER_SANDBOX_PROVIDER", "launcher"), + extra_ro=tuple(Path(p) for p in grants.get("extra_ro", [])), + ) + result = await run_application( + app, source, ctx, handler.make_handler, env, + max_concurrent=2, max_bug_rounds=1, interactive=False, + ) + + print(f"\nCrucible E2E: {result.n_components} invariant(s), {result.n_properties} properties") + for o in result.outcomes: + verdicts = o.result.result.verdicts if isinstance(o.result, Delivered) else "(not delivered)" + print(f" == {o.feat.display_name} == delivered={isinstance(o.result, Delivered)} verdicts={verdicts}") + if result.failures: + print("failures:", result.failures) + + # The front half ran (instructions analyzed, properties extracted) ... + assert result.n_components > 0, "no invariants extracted" + assert result.n_properties > 0, "no properties extracted" + # ... and at least one component was formalized into a fuzz-verdicted test. + delivered = [o for o in result.outcomes if isinstance(o.result, Delivered)] + assert delivered, f"no component was delivered; failures={result.failures}" + assert any(o.result.result.verdicts for o in delivered), "no fuzz verdicts produced" diff --git a/tests/test_crucible_events.py b/tests/test_crucible_events.py new file mode 100644 index 00000000..06275e03 --- /dev/null +++ b/tests/test_crucible_events.py @@ -0,0 +1,239 @@ +"""Unit tests for the Crucible backend's pure callouts + the event routing (no toolchain / LLM). + +The Rust wheel is now a passive service (docs/rust-backend-api.md): these exercise the pure +callouts (`units` / `author_prompt` / `judge_prompt`) directly, and — separately — the +out-of-graph `push_custom_update` routing the Python loop's `emit` relies on. +""" + +import json + +import pytest + +crucible_app = pytest.importorskip( + "crucible_app", + reason="crucible_app wheel not built (uv run maturin develop -m rust/crucible-app/Cargo.toml)", +) + + +def _component_input(*slugs: str) -> str: + return json.dumps( + { + "kind": "component", + "program": "vault", + "component": {"name": "vault", "program": "vault"}, + "props": [ + {"title": f"p {s}", "sort": "invariant", "description": "d", "slug": s} + for s in slugs + ], + "context": {"fixture": "struct Fixture {}", "fuzz_timeout": 5}, + } + ) + + +def _setup_input() -> str: + return json.dumps( + {"kind": "setup", "program": "vault", "component": {"programs": []}, "props": [], "context": {}} + ) + + +def test_descriptor_declares_design_doc_discovery_phase(): + from composer.rustapp.entry import _discovery_phase + from composer.rustapp.host import build_application + + app = build_application("crucible_app") + assert app.section_order[0] == "Design Doc Discovery" + assert _discovery_phase(app) is app.phase["discover_design_doc"] + + +def test_each_property_is_its_own_row_sharing_one_fuzz_target(): + # Collapse (docs/crucible-unit-granularity.md §3): every property is its own report row + # (`c_<slug>`) but they all share ONE fuzz `target` (`c_invariants`), so the host runs a + # single build + fuzz and attributes the outcome per property. + units = json.loads(crucible_app.units(_component_input("solvency", "conservation"))) + assert units == [ + {"property": "p solvency", "unit": "c_solvency", "target": "c_invariants"}, + {"property": "p conservation", "unit": "c_conservation", "target": "c_invariants"}, + ] + + +def test_setup_has_no_units(): + assert json.loads(crucible_app.units(_setup_input())) == [] + + +def test_component_author_prompt_asks_for_one_invariant_fn_covering_all_props(): + prompt = json.loads(crucible_app.author_prompt(_component_input("solvency", "conservation"), None)) + assert prompt.get("system") is None + ins = prompt["instruction"] + # One c_invariants fn asserting ALL listed properties. + assert "c_invariants" in ins + assert "p solvency" in ins and "p conservation" in ins + + +def test_setup_author_prompt_asks_for_a_fixture(): + prompt = json.loads(crucible_app.author_prompt(_setup_input(), None)) + assert "FIXTURE" in prompt["instruction"] + + +def test_author_prompt_failure_appends_revise_context(): + failure = json.dumps({"draft": "fn c_x() {}", "errors": "error[E0425]: cannot find value"}) + prompt = json.loads(crucible_app.author_prompt(_component_input("x"), failure)) + assert "FAILED" in prompt["instruction"] and "E0425" in prompt["instruction"] + + +def test_component_judge_prompt_reviews_the_suite(): + spec = "#[invariant_test]\nfn c_invariants(fixture: &mut Fixture) {}" + raw = crucible_app.judge_prompt(_component_input("solvency"), spec) + assert raw is not None + prompt = json.loads(raw) + # A reviewer persona + the criteria-based task, listing the properties under review and the + # accept/reject JSON contract the host's _parse_judge consumes. + assert "Solana security engineer" in prompt["system"] + ins = prompt["instruction"] + assert "p solvency" in ins and "c_invariants" in ins + assert "Criterion 3 — Reachability" in ins + assert '{"accept": false' in ins + assert spec in ins + + +def test_setup_has_no_judge_prompt(): + # The shared fixture is scaffolding, not test evidence — nothing to judge. + assert crucible_app.judge_prompt(_setup_input(), "spec") is None + + +def test_review_gate_blocks_until_the_submitted_draft_was_accepted(): + from composer.rustapp import adapter + + # Accept only when the judge accepted THIS exact draft. + assert adapter._review_gate({"review_ok": True, "reviewed_text": "spec-A"}, "spec-A") is None + # A draft that was never reviewed (or a different one) is blocked. + assert adapter._review_gate({"review_ok": True, "reviewed_text": "spec-A"}, "spec-B") is not None + # A reviewed-but-rejected draft is blocked. + assert adapter._review_gate({"review_ok": False, "reviewed_text": "spec-A"}, "spec-A") is not None + # No review yet → blocked. + assert adapter._review_gate({}, "spec-A") is not None + + +def test_fetch_verdicts_threads_finding_detail_into_message(): + # A BAD verdict's `detail` (the fuzzer's counterexample / assertion message, captured by + # `validate`) must reach the report `Verdict.message` so a bare BAD explains itself. + import asyncio + from pathlib import Path + + from composer.pipeline.core import Delivered + from composer.rustapp.adapter import RustFormalizer + from composer.rustapp.descriptor import AppDescriptor + from composer.rustapp.result import RustFormalResult + from composer.spec.source.report.collect import ReportComponentInput + from composer.spec.source.report.schema import Outcome + + desc = AppDescriptor.model_validate_json(crucible_app.descriptor()) + fz = RustFormalizer(crucible_app, desc) + res = RustFormalResult( + verdicts={ + "c_x": {"outcome": "BAD", "detail": "crash abc: deposit(5) — expected 105 got 100"}, + "c_y": {"outcome": "GOOD"}, + } + ) + inp = ReportComponentInput( + name="vault", props=[], formalized=Delivered(res, Path("fuzz/vault/src/main.rs")) + ) + verdicts = asyncio.run(fz.fetch_verdicts(inp)) + assert verdicts["c_x"].outcome == Outcome.BAD + assert verdicts["c_x"].message == "crash abc: deposit(5) — expected 105 got 100" + # GOOD verdict with no detail carries no message. + assert verdicts["c_y"].message is None + + +def test_validate_returns_per_unit_verdicts_and_the_host_records_them(): + # The backend owns attribution: `validate` returns a verdict PER report unit the target covers + # (the host does no verdict logic). Here we exercise the FFI contract shape + fetch_verdicts. + import asyncio + from pathlib import Path + + from composer.pipeline.core import Delivered + from composer.rustapp.adapter import RustFormalizer + from composer.rustapp.descriptor import AppDescriptor + from composer.rustapp.result import RustFormalResult + from composer.spec.source.report.collect import ReportComponentInput + from composer.spec.source.report.schema import Outcome + + # A parse error still yields the per-unit `verdicts` shape the host consumes. + out = json.loads(crucible_app.validate("not json", "spec", "c_invariants", "/tmp", "{}")) + assert out["kind"] == "verdicts" + assert out["verdicts"][0][0] == "c_invariants" + assert out["verdicts"][0][1]["outcome"] == "ERROR" + + # And the host records whatever per-unit verdicts the backend produced (one BAD, one GOOD). + fz = RustFormalizer(crucible_app, AppDescriptor.model_validate_json(crucible_app.descriptor())) + res = RustFormalResult( + units=[("solvency", ["c_solvency"]), ("conservation", ["c_conservation"])], + verdicts={ + "c_conservation": {"outcome": "BAD", "detail": "crash abc: [conservation] drift"}, + "c_solvency": {"outcome": "GOOD"}, + }, + ) + inp = ReportComponentInput( + name="vault", props=[], formalized=Delivered(res, Path("fuzz/vault/src/main.rs")) + ) + verdicts = asyncio.run(fz.fetch_verdicts(inp)) + assert verdicts["c_conservation"].outcome == Outcome.BAD + assert "conservation" in verdicts["c_conservation"].message + assert verdicts["c_solvency"].outcome == Outcome.GOOD + + +def test_author_prompt_judge_failure_uses_review_framing(): + # A judge rejection is NOT a build failure (the draft compiled): the revise prompt must + # frame it as review feedback, not compiler errors to fix. + failure = json.dumps( + {"draft": "fn c_x() {}", "errors": "REJECTED: c_x fails Criterion 3", "kind": "judge"} + ) + ins = json.loads(crucible_app.author_prompt(_component_input("x"), failure))["instruction"] + assert "reviewer REJECTED" in ins + assert "FAILED to build" not in ins + assert "Criterion 3" in ins + + +# --------------------------------------------------------------------------- +# The out-of-graph emit routing the Python loop's `emit` relies on. +# --------------------------------------------------------------------------- + + +class _RecordingEventHandler: + def __init__(self) -> None: + self.events: list[tuple[dict, list[str]]] = [] + + async def handle_event(self, payload: dict, path: list[str], checkpoint_id: str) -> None: + self.events.append((payload, path)) + + async def handle_progress_event(self, payload: dict) -> None: + pass + + +class _NullIO: + async def log_checkpoint_id(self, *, path, checkpoint_id): ... + async def log_state_update(self, path, st): ... + async def log_start(self, *, path, description, tool_id): ... + async def log_end(self, path): ... + async def human_interaction(self, ty, debug_thunk): return "" + + +@pytest.mark.asyncio +async def test_push_custom_update_reaches_handle_event_outside_a_graph(): + from composer.io.context import push_custom_update, with_handler + + rec = _RecordingEventHandler() + async with with_handler(_NullIO(), rec): + delivered = push_custom_update( + {"type": "verdict", "outcome": "GOOD", "name": "solvency"}, thread_id="formalize-0" + ) + assert delivered is True + assert rec.events, "custom update never reached handle_event" + payload, path = rec.events[0] + assert payload["type"] == "verdict" and payload["outcome"] == "GOOD" + assert path == ["formalize-0"] + + +def test_push_custom_update_without_scope_is_dropped_not_raised(): + from composer.io.context import push_custom_update + + assert push_custom_update({"type": "x"}, thread_id="t") is False diff --git a/tests/test_crucible_formalize_gate.py b/tests/test_crucible_formalize_gate.py new file mode 100644 index 00000000..f5d88720 --- /dev/null +++ b/tests/test_crucible_formalize_gate.py @@ -0,0 +1,271 @@ +"""Phase-4 gate: per-component test authoring + fuzzing + verdict, with a REAL model. + +Drives the crucible wheel's `new_session` (per-component) decider on one vault +instruction: the agent authors a `#[invariant_test]`/`#[crucible_fuzz]` fn against a +fixed known-good `Fixture`, the loop runs `crucible run … --mode explore --timeout`, +and bakes a verdict (clean run to budget = GOOD; a `[FUZZ_FINDING]` = BAD). Pass = +the session publishes a compiling test with a GOOD verdict on the clean vault. + +Uses a fixed fixture (Phase 3 already gates fixture *authoring*) to isolate the +per-component loop. Heavy + paid; same prerequisites as the other crucible gates. + + CRUCIBLE_REPO=/path/to/crucible \ + .venv/bin/python -m pytest tests/test_crucible_formalize_gate.py -m expensive -q -s +""" + +import asyncio +import json +import os +import shutil +from pathlib import Path +from types import SimpleNamespace +from typing import Any, cast, TYPE_CHECKING + +import psycopg +import pytest +from psycopg.sql import SQL, Identifier, Literal + +import composer.workflow.services as services +from composer.io.multi_job import TaskInfo +from composer.kb.knowledge_base import DefaultEmbedder +from composer.pipeline.core import GaveUp, PipelineRun +from composer.pipeline.ecosystem import RUST_FORBIDDEN_READ +from composer.rustapp.adapter import author_and_compile, make_emitter +from composer.rustapp.frontend import GenericRustConsoleHandler +from composer.rustapp.host import build_phase_enum, load_descriptor, load_module +from composer.spec.context import SourceCode, WorkflowContext +from composer.spec.service_host import ModelProvider, PureServiceHost +from composer.llm.registry import get_provider_for +from composer.spec.solana.build import build_program +from composer.spec.source.source_env import build_basic_source_tools, build_source_tools +from composer.spec.system_model import SolidityIdentifier +from composer.ui.tool_display import async_tool_context +from composer.workflow.services import llm_factory, standard_connections + +from tests.conftest import MockSentenceTransformer, needs_postgres +from tests.test_autoprove_integration import _MEMORIES_DDL, _RAG_DB, _VECTOR_DBS, _db_url + +if TYPE_CHECKING: + from testcontainers.postgres import PostgresContainer + +pytestmark = [pytest.mark.expensive, needs_postgres, pytest.mark.asyncio] + +_SCENARIO = Path(__file__).parent.parent / "test_scenarios" / "solana_vault" +_PROGRAM = "vault" +_SLUG = "deposit" +_FEATURE = f"c_{_SLUG}" + +# A fixed, known-good shared fixture (`struct Fixture`) — the per-component decider +# authors a test *against* it. Mirrors the Phase-1 harness, renamed to `Fixture`. +_FIXTURE = """\ +use crucible_fuzzer::*; +use crucible_fuzzer::anchor_lang::system_program; +use vault::*; +use solana_keypair::Keypair; +use solana_pubkey::Pubkey; +use solana_signer::Signer; +use std::rc::Rc; + +const INITIAL_BALANCE: u64 = 10_000_000_000; + +#[derive(Clone)] +struct Fixture { + ctx: TestContext, + program_id: Pubkey, + authority: Rc<Keypair>, + vault_pda: Pubkey, +} + +#[fuzz_fixture] +impl Fixture { + pub fn setup() -> Self { + let mut ctx = TestContext::new(); + let program_id = Pubkey::new_from_array(ID.to_bytes()); + ctx.add_program(&program_id, "../../target/deploy/vault.so").unwrap(); + + let authority = Rc::new(Keypair::new()); + ctx.create_account() + .pubkey(authority.pubkey()) + .lamports(INITIAL_BALANCE) + .owner(system_program::ID) + .create() + .unwrap(); + + let (vault_pda, _) = + Pubkey::find_program_address(&[b"vault", authority.pubkey().as_ref()], &program_id); + + ctx.program(program_id) + .call(instruction::Initialize {}) + .accounts(accounts::Initialize { + vault: vault_pda, + authority: authority.pubkey(), + system_program: system_program::ID, + }) + .signers(&[&*authority]) + .send() + .expect("initialize must succeed during setup"); + + Self { ctx, program_id, authority, vault_pda } + } + + pub fn action_deposit(&mut self, #[range(1..1_000_000)] amount: u64) -> bool { + self.ctx.program(self.program_id) + .call(instruction::Deposit { amount }) + .accounts(accounts::Deposit { + vault: self.vault_pda, + depositor: self.authority.pubkey(), + system_program: system_program::ID, + }) + .signers(&[&*self.authority]) + .send().map(|o| o.is_success()).unwrap_or(false) + } + + pub fn action_withdraw(&mut self, #[range(1..1_000_000)] amount: u64) -> bool { + self.ctx.program(self.program_id) + .call(instruction::Withdraw { amount }) + .accounts(accounts::Withdraw { + vault: self.vault_pda, + authority: self.authority.pubkey(), + }) + .signers(&[&*self.authority]) + .send().map(|o| o.is_success()).unwrap_or(false) + } +} +""" + +_COMPONENT = { + "program": "vault", + "instruction": { + "name": "deposit", + "args": ["amount: u64"], + "description": "Transfer `amount` lamports into the vault PDA and increase the recorded balance.", + }, +} +_PROPS = [ + { + "title": "recorded balance never exceeds deposited funds", + "sort": "invariant", + "description": "The vault's recorded `balance` never exceeds the authority's initial funds " + "(a conservation bound: you cannot have more recorded than could have been deposited).", + }, +] + + +def _model_args() -> object: + return SimpleNamespace( + heavy_model="claude-opus-4-6", lite_model="claude-sonnet-4-6", + tokens=128_000, thinking_tokens=2048, memory_tool=False, interleaved_thinking=False, + ) + + +def _crucible_repo() -> Path | None: + raw = os.environ.get("CRUCIBLE_REPO") + if not raw: + return None + repo = Path(raw) + return repo if (repo / "crates" / "crucible-fuzzer").is_dir() else None + + +def _require(cond: bool, why: str) -> None: + if not cond: + pytest.skip(why) + + +async def test_crucible_per_component_formalize(pg_container: "PostgresContainer", monkeypatch): + _require(_SCENARIO.is_dir(), f"scenario missing: {_SCENARIO}") + _require(shutil.which("cargo-build-sbf") is not None, "cargo-build-sbf not on PATH") + _require(shutil.which("crucible") is not None, "crucible CLI not on PATH") + crucible_repo = _crucible_repo() + _require(crucible_repo is not None, "set CRUCIBLE_REPO to a local crucible checkout") + assert crucible_repo is not None + + admin_url = pg_container.get_connection_url(driver=None) + with psycopg.connect(admin_url, autocommit=True) as admin: + for cfg in services._DATABASE_CONFIGS.values(): + admin.execute(SQL("CREATE ROLE {} LOGIN PASSWORD {}").format( + Identifier(cfg["user"]), Literal(cfg["password"]))) + admin.execute(SQL("CREATE DATABASE {} OWNER {}").format( + Identifier(cfg["database"]), Identifier(cfg["user"]))) + admin.execute(SQL("CREATE DATABASE {}").format(Identifier(_RAG_DB))) + for db in _VECTOR_DBS: + with psycopg.connect(_db_url(pg_container, db), autocommit=True) as conn: + conn.execute("CREATE EXTENSION IF NOT EXISTS vector") + mem = services._DATABASE_CONFIGS["memory"] + mem_url = ( + f"postgresql://{mem['user']}:{mem['password']}" + f"@{pg_container.get_container_host_ip()}:{pg_container.get_exposed_port(5432)}/{mem['database']}" + ) + with psycopg.connect(mem_url, autocommit=True) as conn: + conn.execute(_MEMORIES_DDL) + + monkeypatch.setenv("CERTORA_AI_COMPOSER_PGHOST", pg_container.get_container_host_ip()) + monkeypatch.setenv("CERTORA_AI_COMPOSER_PGPORT", str(pg_container.get_exposed_port(5432))) + + args = _model_args() + built = await build_program(_SCENARIO, _PROGRAM, timeout_s=590) + assert built.so_path.is_file(), built.so_path + + async with ( + standard_connections(provider="anthropic", embedder=DefaultEmbedder(MockSentenceTransformer())) as conns, + async_tool_context(), + ): + content = await conns.uploader.get_document(_SCENARIO / "system.md") + assert content is not None + source = SourceCode( + content=content, project_root=str(_SCENARIO), + contract_name=SolidityIdentifier(_PROGRAM), + relative_path=f"programs/{_PROGRAM}/src/lib.rs", forbidden_read=RUST_FORBIDDEN_READ, + ) + _tiered = get_provider_for(tiered=cast(Any, args)) + model_provider = ModelProvider( + heavy_model=_tiered.heavy, lite_model=_tiered.lite, checkpointer=conns.checkpointer, + ) + basic = build_basic_source_tools(root=str(_SCENARIO), forbidden_read=RUST_FORBIDDEN_READ) + full = build_source_tools(basic, model_provider, conns.indexed_store, ("crucible_fmz", "src"), recursion_limit=100) + env = PureServiceHost(models=model_provider, rag_tools=(), sort="existing").bind_source_tools(full) + ctx = WorkflowContext.create( + services=conns.memory, + thread_id="crucible_fmz", store=conns.store, recursion_limit=100, + cache_namespace=None, memory_namespace=None, + ) + + # No manifest pre-placement needed: `compile`/`validate` materialize the harness crate + # (Cargo.toml + main.rs) themselves per run (docs/rust-pure-app.md §4). + module = load_module("crucible_app") + phase = build_phase_enum(load_descriptor(module)) + + # The component artifact: author the test(s), `compile` (dry-run), then `validate` + # the unit (fuzz). Unsandboxed here (trusted inputs), so run_confined=None. + input_dict = { + "kind": "component", "program": _PROGRAM, "component": _COMPONENT, "props": _PROPS, + "context": {"fixture": _FIXTURE, "fuzz_timeout": 15}, + } + input_json = json.dumps(input_dict) + sandbox_dict = {"run_confined": None, "timeout_s": 1200} + sandbox_json = json.dumps(sandbox_dict) + + run = PipelineRun(ctx=ctx, source=source, _handler_factory=GenericRustConsoleHandler(set()).make_handler, _semaphore=asyncio.Semaphore(2), env=env) + test_src = await run.runner( + TaskInfo("crucible_fmz", "Harness Authoring", phase["formalization"]), + lambda: author_and_compile( + module, input_dict, env=env, sandbox_dict=sandbox_dict, + workdir=Path(_SCENARIO), recursion_limit=100, backend_name="crucible", + emit=make_emitter(), + ), + ) + if isinstance(test_src, GaveUp): + pytest.fail(f"per-component authoring gave up: {test_src.reason}") + + units = json.loads(module.units(input_json)) + # validate is the fused build+fuzz; a clean vault run → a GOOD verdict (not build_failed). + res = json.loads( + await asyncio.to_thread(module.validate, input_json, test_src, _FEATURE, str(_SCENARIO), sandbox_json) + ) + + print("\n===== authored test =====\n" + test_src) + print("units:", units, "validate:", res) + assert "#[invariant_test]" in test_src or "#[crucible_fuzz]" in test_src + assert res["kind"] == "verdict", res + assert res["verdict"]["outcome"] == "GOOD", res + # property → this test's unit is recorded for the report. + assert units == [{"property": _PROPS[0]["title"], "unit": _FEATURE}] diff --git a/tests/test_crucible_gate.py b/tests/test_crucible_gate.py new file mode 100644 index 00000000..29355730 --- /dev/null +++ b/tests/test_crucible_gate.py @@ -0,0 +1,320 @@ +"""Phase-1 gate for the Crucible backend: the build + dry-run *infrastructure*. + +No LLM, no property authoring (those are later phases). This proves the plumbing +that phase 1 delivers, on the ``solana_vault`` Anchor scenario: + +1. the descriptor loads and its ecosystem resolves to ``solana``; +2. ``validate_preconditions`` accepts the buildable workspace; +3. the shared Solana build step compiles the program to ``target/deploy/vault.so``; +4. a trivial hand-written Crucible harness passes ``crucible run … --dry-run`` + (compiles against the built ``.so`` + the program crate, and ``setup()`` runs + one iteration). + +Marked ``expensive``: it needs the Solana/Anchor toolchain + the ``crucible`` CLI, +and a local Crucible checkout for the harness's crate deps (``CRUCIBLE_REPO``). +It builds real sBPF + a fuzz harness, so it is slow. +Skips cleanly when any prerequisite is missing. Run with:: + + CRUCIBLE_REPO=/path/to/crucible \ + .venv/bin/python -m pytest tests/test_crucible_gate.py -m expensive -q -s +""" + +import json +import os +import shutil +from pathlib import Path + +import pytest + +from composer.sandbox.command import run_local_command +from composer.rustapp.descriptor import DeliverableMode +from composer.rustapp.host import load_descriptor, load_module +from composer.rustapp.result import RustArtifact, RustFormalResult +from composer.rustapp.store import RustArtifactStore +from composer.spec.solana.build import build_program + +pytestmark = [pytest.mark.expensive, pytest.mark.asyncio] + +_SCENARIO = Path(__file__).parent.parent / "test_scenarios" / "solana_vault" +_PROGRAM = "vault" +_TEST = "invariant_vault" # crucible test name == the feature gating it + + +def _crucible_repo() -> Path | None: + raw = os.environ.get("CRUCIBLE_REPO") + if not raw: + return None + repo = Path(raw) + return repo if (repo / "crates" / "crucible-fuzzer").is_dir() else None + + +def _fuzz_cargo_toml(crucible_repo: Path) -> str: + crates = crucible_repo / "crates" + return f"""\ +[package] +name = "vault_fuzz" +version = "0.1.0" +edition = "2021" + +[workspace] + +[dependencies] +crucible-fuzzer = {{ path = "{crates / 'crucible-fuzzer'}" }} +crucible-test-context = {{ path = "{crates / 'crucible-test-context'}" }} +anchor-lang = "1.0.1" +arbitrary = {{ version = "1", features = ["derive"] }} +ctrlc = "3.4" +libafl = {{ version = "0.15.1", features = ["std", "cli", "prelude"] }} +libafl_bolts = {{ version = "0.15.1", features = ["std"] }} +vault = {{ path = "../../programs/vault", features = ["no-entrypoint"] }} +solana-keypair = "3.0" +solana-pubkey = "3.0" +solana-signer = "3.0" + +[[bin]] +name = "invariant_test" +path = "src/main.rs" + +[features] +{_TEST} = [] +""" + + +# A minimal hand-written harness (no LLM): load the built .so, initialize a vault, +# expose deposit/withdraw actions, and a trivial balance invariant. Enough for +# --dry-run (compile + one setup iteration). +_FUZZ_MAIN_RS = """\ +use crucible_fuzzer::anchor_lang::system_program; +use crucible_fuzzer::*; +use solana_keypair::Keypair; +use solana_pubkey::Pubkey; +use solana_signer::Signer; +use std::rc::Rc; +use vault::*; + +const INITIAL_BALANCE: u64 = 10_000_000_000; + +#[derive(Clone)] +struct VaultFixture { + ctx: TestContext, + program_id: Pubkey, + authority: Rc<Keypair>, + vault_pda: Pubkey, +} + +#[fuzz_fixture] +impl VaultFixture { + pub fn setup() -> Self { + let mut ctx = TestContext::new(); + let program_id = Pubkey::new_from_array(ID.to_bytes()); + ctx.add_program(&program_id, "../../target/deploy/vault.so").unwrap(); + + let authority = Rc::new(Keypair::new()); + ctx.create_account() + .pubkey(authority.pubkey()) + .lamports(INITIAL_BALANCE) + .owner(system_program::ID) + .create() + .unwrap(); + + let (vault_pda, _) = + Pubkey::find_program_address(&[b"vault", authority.pubkey().as_ref()], &program_id); + + ctx.program(program_id) + .call(instruction::Initialize {}) + .accounts(accounts::Initialize { + vault: vault_pda, + authority: authority.pubkey(), + system_program: system_program::ID, + }) + .signers(&[&*authority]) + .send() + .unwrap(); + + Self { ctx, program_id, authority, vault_pda } + } + + pub fn action_deposit(&mut self, #[range(1..1_000_000)] amount: u64) -> bool { + self.ctx + .program(self.program_id) + .call(instruction::Deposit { amount }) + .accounts(accounts::Deposit { + vault: self.vault_pda, + depositor: self.authority.pubkey(), + system_program: system_program::ID, + }) + .signers(&[&*self.authority]) + .send() + .map(|o| o.is_success()) + .unwrap_or(false) + } + + pub fn action_withdraw(&mut self, #[range(1..1_000_000)] amount: u64) -> bool { + self.ctx + .program(self.program_id) + .call(instruction::Withdraw { amount }) + .accounts(accounts::Withdraw { + vault: self.vault_pda, + authority: self.authority.pubkey(), + }) + .signers(&[&*self.authority]) + .send() + .map(|o| o.is_success()) + .unwrap_or(false) + } +} + +#[invariant_test] +fn invariant_vault(fixture: &mut VaultFixture) { + if let Ok(vault) = fixture.ctx.read_anchor_account::<VaultState>(&fixture.vault_pda) { + fuzz_assert_le!(vault.balance, INITIAL_BALANCE, "recorded balance exceeds initial funds"); + } +} +""" + + +def _require(cond: bool, why: str) -> None: + if not cond: + pytest.skip(why) + + +async def test_crucible_phase1_build_and_dry_run(): + _require(_SCENARIO.is_dir(), f"scenario missing: {_SCENARIO}") + _require(shutil.which("cargo-build-sbf") is not None, "cargo-build-sbf not on PATH") + _require(shutil.which("crucible") is not None, "crucible CLI not on PATH") + crucible_repo = _crucible_repo() + _require(crucible_repo is not None, "set CRUCIBLE_REPO to a local crucible checkout") + assert crucible_repo is not None # for the type checker + + # 0. Descriptor + ecosystem + preconditions (cheap, no build). + import composer.bind as _ # noqa: F401 (DI bootstrap) + from composer.rustapp.host import load_descriptor, load_module, resolve_ecosystem + + module = load_module("crucible_app") + descriptor = load_descriptor(module) + assert descriptor.ecosystem == "solana" + assert resolve_ecosystem(descriptor).name == "solana" + err = module.validate_preconditions(json.dumps({"project_root": str(_SCENARIO)})) + assert err is None, f"validate_preconditions rejected the scenario: {err}" + + # 1. Build the program to sBPF. + built = await build_program(_SCENARIO, _PROGRAM, timeout_s=590) + assert built.so_path.is_file(), built.so_path + + # 2. Materialize the trivial fuzz harness (crate deps resolved to the checkout). + fuzz_dir = _SCENARIO / "fuzz" / _PROGRAM + (fuzz_dir / "src").mkdir(parents=True, exist_ok=True) + (fuzz_dir / "Cargo.toml").write_text(_fuzz_cargo_toml(crucible_repo)) + (fuzz_dir / "src" / "main.rs").write_text(_FUZZ_MAIN_RS) + + # 3. Dry-run: compiles the harness + runs setup() one iteration. Run from the + # project root (crucible resolves fuzz/<program>/ relative to cwd). + res = await run_local_command( + "crucible", + ["run", _PROGRAM, _TEST, "--release", "--dry-run"], + {}, + workdir=_SCENARIO, + timeout_s=590, + ) + assert res.exit_code == 0, ( + f"crucible --dry-run failed (exit {res.exit_code})\n" + f"STDOUT:\n{res.stdout[-3000:]}\n\nSTDERR:\n{res.stderr[-3000:]}" + ) + + +# --- Phase 2: the deliverable model (wheel finalize assembles the crate) --- + +# The shared fixture is the phase-1 harness minus its test fn (the store composes +# fixture + per-component test sections). +_FIXTURE_SRC = _FUZZ_MAIN_RS.partition("#[invariant_test]")[0] + +# One component's test section. Its fn name MUST equal its feature (c_deposit) — +# Crucible's #[invariant_test] macro gates main() by #[cfg(feature = "<fn name>")]. +_DEPOSIT_SECTION = """\ +#[invariant_test] +fn c_deposit(fixture: &mut VaultFixture) { + if let Ok(vault) = fixture.ctx.read_anchor_account::<VaultState>(&fixture.vault_pda) { + fuzz_assert_le!(vault.balance, INITIAL_BALANCE, "recorded balance exceeds initial funds"); + } +} +""" + + +async def test_crucible_phase2_store_assembles_crate(): + _require(_SCENARIO.is_dir(), f"scenario missing: {_SCENARIO}") + _require(shutil.which("cargo-build-sbf") is not None, "cargo-build-sbf not on PATH") + _require(shutil.which("crucible") is not None, "crucible CLI not on PATH") + crucible_repo = _crucible_repo() + _require(crucible_repo is not None, "set CRUCIBLE_REPO to a local crucible checkout") + assert crucible_repo is not None + + built = await build_program(_SCENARIO, _PROGRAM, timeout_s=590) + assert built.so_path.is_file(), built.so_path + + # A co-located EVM (autoprove) deliverable, to prove the layouts don't collide. + evm_sentinel = _SCENARIO / "certora" / "specs" / "autospec_sentinel.spec" + evm_sentinel.parent.mkdir(parents=True, exist_ok=True) + evm_sentinel.write_text("// pretend EVM output\n") + + # The deliverable is now split (docs/rust-pure-app.md): the generic callout-mode store + # writes the per-component metadata + returns the crate report link; the wheel's `finalize` + # renders the one crate (fixture + sections) from the full result set. Drive both, exactly + # as the pipeline does. + module = load_module("crucible_app") + layout = load_descriptor(module).artifact_layout + store = RustArtifactStore( + str(_SCENARIO), layout, deliverable_mode=DeliverableMode.CALLOUT, program=_PROGRAM + ) + comp = RustArtifact("deposit", "harness", "rs") + result = RustFormalResult( + commentary="The recorded vault balance never exceeds the authority's initial funds.", + artifact_text=_DEPOSIT_SECTION, + units=[("balance bounded by initial funds", ["c_deposit"])], + ) + store.write_properties(comp, []) + main_rel = store.write_artifact(comp, result) # metadata + the crate report link + assert str(main_rel) == f"fuzz/{_PROGRAM}/src/main.rs" + + # finalize renders the crate (fixture + the delivered section, features from property_units). + payload = json.dumps( + { + "program": _PROGRAM, + "setup": _FIXTURE_SRC, + "components": [ + { + "name": "deposit", + "delivered": True, + "artifact_text": _DEPOSIT_SECTION, + "property_units": [["balance bounded by initial funds", ["c_deposit"]]], + } + ], + } + ) + for rel, contents in json.loads(module.finalize(payload)).items(): + target = _SCENARIO / rel + target.parent.mkdir(parents=True, exist_ok=True) + target.write_text(contents) + + # Deliverable crate under fuzz/<program>/ ... + assert (_SCENARIO / main_rel).is_file() + assert (_SCENARIO / "fuzz" / _PROGRAM / "Cargo.toml").is_file() + assert 'c_deposit = []' in (_SCENARIO / "fuzz" / _PROGRAM / "Cargo.toml").read_text() + # ... metadata under certora/crucible/ (not under fuzz/, not under certora/specs/) ... + props = _SCENARIO / "certora" / "crucible" / "properties" + assert (props / "harness_deposit.commentary.md").is_file() + assert (props / "harness_deposit.property_tests.json").is_file() + # ... and the EVM output is untouched (coexistence). + assert evm_sentinel.read_text() == "// pretend EVM output\n" + + # The assembled crate compiles and dry-runs (feature == the component's). + res = await run_local_command( + "crucible", + ["run", _PROGRAM, "c_deposit", "--release", "--dry-run"], + {}, + workdir=_SCENARIO, + timeout_s=590, + ) + assert res.exit_code == 0, ( + f"assembled-crate --dry-run failed (exit {res.exit_code})\n" + f"STDOUT:\n{res.stdout[-3000:]}\n\nSTDERR:\n{res.stderr[-3000:]}" + ) diff --git a/tests/test_crucible_granularity.py b/tests/test_crucible_granularity.py new file mode 100644 index 00000000..05ab244c --- /dev/null +++ b/tests/test_crucible_granularity.py @@ -0,0 +1,189 @@ +"""Whole-program (global) extraction + per-invariant units — docs/crucible-unit-granularity.md. + +Covers the three pieces of the granularity change without the toolchain/LLM: +- the new Solana units (whole-program extraction context + per-invariant unit), +- the SOLANA ecosystem's global-extraction wiring, +- the driver branch that turns one whole-program extraction into one batch per invariant. +""" + +import types + +import pytest + +from composer.spec.solana.model import ( + SolanaApplication, + SolanaInvariantUnit, + SolanaProgramInstance, +) +from composer.spec.system_model import FeatureUnit +from composer.spec.types import PropertyFormulation + + +def _app() -> SolanaApplication: + return SolanaApplication.model_validate( + { + "application_type": "defi", + "description": "a vault program", + "components": [ + { + "name": "vault", + "description": "the vault program", + "program_identifier": "vault", + "account_types": ["Vault"], + "instructions": [ + {"name": "deposit", "description": "d", "requirements": []}, + {"name": "withdraw", "description": "w", "requirements": []}, + ], + } + ], + } + ) + + +def _inv(title: str) -> PropertyFormulation: + return PropertyFormulation(title=title, sort="invariant", description="desc") + + +def test_program_instance_is_a_feature_unit_with_whole_program_api(): + main = SolanaProgramInstance(0, _app()) + assert isinstance(main, FeatureUnit) # runtime_checkable + assert main.display_name == "vault" + # feature_json carries the whole-program instruction API (for the test author). + names = [i["name"] for i in main.feature_json()["instructions"]] + assert names == ["deposit", "withdraw"] + + +def test_invariant_unit_maps_title_to_slug_and_carries_program_api(): + main = SolanaProgramInstance(0, _app()) + u = SolanaInvariantUnit(2, main, _inv("total shares == balance")) + assert isinstance(u, FeatureUnit) + assert u.display_name == "total shares == balance" + assert u.slug == "total_shares_balance" or "total" in u.slug # slugified + assert u.unit_index == 2 + assert [i["name"] for i in u.feature_json()["instructions"]] == ["deposit", "withdraw"] + # Distinct invariants → distinct cache material (so formalize caches per invariant). + assert u.cache_material() != SolanaInvariantUnit(3, main, _inv("other")).cache_material() + + +def test_solana_ecosystem_uses_global_extraction(): + from composer.pipeline.ecosystem import SOLANA + + main = SolanaProgramInstance(0, _app()) + assert SOLANA.global_extraction is True + assert SOLANA.collapse_units is True # one whole-program batch (single harness + run) + assert SOLANA.extraction_unit is not None and SOLANA.property_unit is not None + # extraction context is the whole program; property_unit fans an invariant into a unit. + assert SOLANA.extraction_unit(main) is main + u = SOLANA.property_unit(main, _inv("x"), 0) + assert isinstance(u, SolanaInvariantUnit) and u.display_name == "x" + + +# --- driver branch: one extraction → one batch per invariant ------------------------ + + +class _FakeUnit: + def __init__(self, name: str, i: int): + self._n, self._i = name, i + + @property + def display_name(self) -> str: + return self._n + + @property + def slug(self) -> str: + return self._n + + @property + def unit_index(self) -> int: + return self._i + + def cache_material(self) -> str: + return self._n + + def context_tag(self) -> dict: + return {} + + def feature_json(self) -> dict: + return {} + + +class _ChildCtx: + async def child(self, key, tag=None): + return object() + + +class _Ctx: + recursion_limit = 100 + + def child(self, key): + return _ChildCtx() + + +class _Run: + ctx = _Ctx() + env = object() + + async def runner(self, info, job): + return await job(None) # the extraction job takes a refinement-conv arg + + +@pytest.mark.asyncio +async def test_global_extraction_fans_out_one_batch_per_invariant(monkeypatch): + from composer.pipeline import core + + invs = [_inv("inv0"), _inv("inv1"), _inv("inv2")] + + async def fake_rpi(*a, **k): + return invs + + monkeypatch.setattr(core, "run_property_inference", fake_rpi) + + eco = types.SimpleNamespace( + global_extraction=True, + collapse_units=False, # per-invariant fan-out path + property_prompts=types.SimpleNamespace(system="s.j2", initial="i.j2"), + extraction_unit=lambda main: _FakeUnit("program", 0), + property_unit=lambda main, prop, i: _FakeUnit(prop.title, i), + units=lambda main: [], + ) + + batches = await core._extract_all( + main=object(), backend_guidance="", run=_Run(), phase=None, + interactive=False, threat_model=None, max_rounds=1, ecosystem=eco, + ) + + assert [b.feat.display_name for b in batches] == ["inv0", "inv1", "inv2"] + # each batch carries exactly its own single invariant + assert [[p.title for p in b.props] for b in batches] == [["inv0"], ["inv1"], ["inv2"]] + + +@pytest.mark.asyncio +async def test_collapse_units_makes_one_whole_program_batch(monkeypatch): + # docs/crucible-unit-granularity.md §3: with collapse_units, global extraction keeps ALL + # invariants in ONE batch on the whole-program unit (single harness + run). + from composer.pipeline import core + + invs = [_inv("inv0"), _inv("inv1"), _inv("inv2")] + + async def fake_rpi(*a, **k): + return invs + + monkeypatch.setattr(core, "run_property_inference", fake_rpi) + + eco = types.SimpleNamespace( + global_extraction=True, + collapse_units=True, # the collapse: one batch, all props + property_prompts=types.SimpleNamespace(system="s.j2", initial="i.j2"), + extraction_unit=lambda main: _FakeUnit("program", 0), + property_unit=lambda main, prop, i: _FakeUnit(prop.title, i), + units=lambda main: [], + ) + + batches = await core._extract_all( + main=object(), backend_guidance="", run=_Run(), phase=None, + interactive=False, threat_model=None, max_rounds=1, ecosystem=eco, + ) + + assert len(batches) == 1 + assert batches[0].feat.display_name == "program" + assert [p.title for p in batches[0].props] == ["inv0", "inv1", "inv2"] diff --git a/tests/test_crucible_harness.py b/tests/test_crucible_harness.py new file mode 100644 index 00000000..ac4ce58b --- /dev/null +++ b/tests/test_crucible_harness.py @@ -0,0 +1,78 @@ +"""Unit tests for the crucible wheel's crate rendering (was Python's `CrucibleHarness`). + +Pure/fast (no build, no LLM): the wheel now owns crate assembly (`docs/rust-pure-app.md`). +`workspace_prep` places a deps-only manifest for warming, and `finalize` renders the whole crate +(shared fixture + one feature-gated test section per delivered invariant) from the outcome set. +These pin that rendering — including that the manifest declares each invariant's feature, which +replaces the old cumulative-feature-reservation dance (per-run manifests remove the race +entirely, so there is no shared manifest to clobber). +""" + +import json + +import pytest + +crucible_app = pytest.importorskip( + "crucible_app", + reason="crucible_app wheel not built (maturin build -m rust/crucible-app/Cargo.toml)", +) + + +@pytest.fixture(autouse=True) +def _crucible_repo(monkeypatch): + # Crate rendering only needs the checkout path as a *string* for the path-deps; a real dir + # isn't required to exercise the manifest/main.rs assembly. + monkeypatch.setenv("CRUCIBLE_REPO", "/nonexistent/crucible") + + +def _finalize(*sections: tuple[str, str]) -> dict[str, str]: + """Render the crate for delivered invariants, each ``(feature, test_src)``.""" + payload = { + "program": "vault", + "setup": "// FIXTURE\nstruct Fixture {}", + "components": [ + { + "name": feat, + "delivered": True, + "artifact_text": src, + "property_units": [[f"p {feat}", [feat]]], + } + for feat, src in sections + ], + } + return json.loads(crucible_app.finalize(json.dumps(payload))) + + +def test_workspace_prep_places_deps_only_manifest_and_warm_plan(): + plan = json.loads( + crucible_app.workspace_prep( + json.dumps({"kind": "setup", "program": "vault", "component": {}, "props": [], "context": {}}) + ) + ) + assert plan["warm_dirs"] == ["fuzz/vault"] + assert plan["build_program"] == "vault" + cargo = plan["files"]["fuzz/vault/Cargo.toml"] + assert 'name = "vault_fuzz"' in cargo + assert "c_probe = []" in cargo # a feature to select for the setup dry-run + # The pinned crucible/solana stack + the program path dep (was CrucibleDep). + assert 'vault = { path = "../../programs/vault", features = ["no-entrypoint"] }' in cargo + + +def test_finalize_renders_fixture_plus_each_section_and_features(): + files = _finalize( + ("c_deposit", "#[invariant_test]\nfn c_deposit(f: &mut Fixture) {}"), + ("c_withdraw", "#[invariant_test]\nfn c_withdraw(f: &mut Fixture) {}"), + ) + main_rs = files["fuzz/vault/src/main.rs"] + # Fixture first, then every section (verbatim; the macro self-gates by fn name). + assert main_rs.startswith("// FIXTURE") + assert "fn c_deposit" in main_rs and "fn c_withdraw" in main_rs + # Both features are declared in the manifest (sorted, stable). + cargo = files["fuzz/vault/Cargo.toml"] + assert "c_deposit = []" in cargo and "c_withdraw = []" in cargo + + +def test_finalize_skips_undelivered_and_is_empty_without_sections(): + # No delivered components → nothing to assemble; finalize returns None (the host skips it). + raw = crucible_app.finalize(json.dumps({"program": "vault", "setup": "// F", "components": []})) + assert raw is None diff --git a/tests/test_crucible_rag.py b/tests/test_crucible_rag.py new file mode 100644 index 00000000..21c7b3f1 --- /dev/null +++ b/tests/test_crucible_rag.py @@ -0,0 +1,65 @@ +"""The Crucible docs-search tools must degrade gracefully — a backend failure (DB +down, or the embedding model choking on a query, as seen with nomic-embed on certain +inputs) returns "no results", never crashing the authoring turn / failing a component. +""" + +import asyncio + +from composer.tools.crucible_rag import ( + CrucibleKeywordSearch, + CrucibleSectionGet, + CrucibleVectorSearch, + _UNAVAILABLE, +) + + +class _BoomDB: + """A ComposerRAGDB stand-in whose every call raises (e.g. the embedding model + RuntimeError observed on some queries).""" + + async def find_refs(self, *a, **k): + raise RuntimeError("size of tensor a (21) must match tensor b (18)") + + async def search_manual_keywords(self, *a, **k): + raise RuntimeError("db connection reset") + + async def get_manual_section(self, *a, **k): + raise RuntimeError("db connection reset") + + +class _EmptyDB: + async def find_refs(self, *a, **k): + return [] + + async def search_manual_keywords(self, *a, **k): + return [] + + +def _run(tool_cls, db, **fields) -> str: + tool_cls._dep_ctx.set(db) # what `tool_deps()` reads; normally set by binding + return asyncio.run(tool_cls(**fields).run()) + + +def test_vector_search_degrades_when_embedding_crashes(): + out = _run(CrucibleVectorSearch, _BoomDB(), query="how do I write an invariant?") + assert out == _UNAVAILABLE + + +def test_keyword_search_degrades_on_db_error(): + out = _run(CrucibleKeywordSearch, _BoomDB(), query="fuzz_fixture") + assert out == _UNAVAILABLE + + +def test_section_get_degrades_on_db_error(): + out = _run(CrucibleSectionGet, _BoomDB(), section_names=["A", "B"]) + assert out == _UNAVAILABLE + + +def test_vector_search_reports_no_results_without_crashing(): + out = _run(CrucibleVectorSearch, _EmptyDB(), query="anything") + assert out == "(No results found)" + + +def test_keyword_search_reports_no_results(): + out = _run(CrucibleKeywordSearch, _EmptyDB(), query="anything") + assert out == "No results found" diff --git a/tests/test_crucible_results.py b/tests/test_crucible_results.py new file mode 100644 index 00000000..bb548d07 --- /dev/null +++ b/tests/test_crucible_results.py @@ -0,0 +1,84 @@ +"""The generic console/TUI verdict rollup (``composer.rustapp.results``) — no wheel / LLM. + +A completed Rust-backend run bakes a per-unit ``Outcome`` into each delivered result +(``RustFormalResult.verdicts``). These tests pin how that rollup renders in the console counts +block for the ``crucible`` backend tag: a tally line in the report's crucible vocabulary ("No +counterexample" / "Counterexample") plus a per-unit listing, give-ups excluded (they live in +``failures``), and nothing at all when no unit was delivered. +""" + +from pathlib import Path +from typing import Any, cast + +from composer.rustapp.results import ( + format_verdict_lines, + summarize_verdicts, +) +from composer.pipeline.core import ComponentOutcome, CorePipelineResult, Delivered, GaveUp +from composer.rustapp.result import RustFormalResult +from composer.spec.source.report.schema import Outcome + + +class _Feat: + """Minimal duck-typed unit — the rollup only reads ``display_name``.""" + + def __init__(self, name: str): + self.display_name = name + + +def _feat(name: str) -> Any: + # The rollup only reads ``display_name``; a real ContractComponentInstance is overkill. + return cast(Any, _Feat(name)) + + +def _delivered(name: str, outcome: str) -> ComponentOutcome: + res = RustFormalResult(verdicts={name: {"outcome": outcome}}) + return ComponentOutcome(_feat(name), [], Delivered(res, Path(f"fuzz/{name}.rs"))) + + +def _gave_up(name: str) -> ComponentOutcome: + return ComponentOutcome(_feat(name), [], GaveUp(reason="did not compile")) + + +def _result(*outcomes: ComponentOutcome) -> CorePipelineResult: + return CorePipelineResult(len(outcomes), len(outcomes), list(outcomes), []) + + +def test_tally_uses_crucible_labels_and_counts(): + result = _result( + _delivered("conservation", "GOOD"), + _delivered("solvency", "BAD"), + _delivered("bounds", "GOOD"), + ) + summary = summarize_verdicts(result, "crucible") + assert summary.counts == {Outcome.GOOD: 2, Outcome.BAD: 1} + # GOOD is listed before BAD (display order), with the crucible wording. + assert summary.tally == "2 No counterexample, 1 Counterexample" + + +def test_lines_have_tally_then_per_invariant_listing(): + result = _result(_delivered("solvency", "BAD"), _delivered("bounds", "GOOD")) + lines = format_verdict_lines(summarize_verdicts(result, "crucible")) + # The tally is outcome-ordered (GOOD before BAD); the listing keeps pipeline order. + assert lines[0] == " Verdicts: 1 No counterexample, 1 Counterexample" + assert lines[1] == " ✗ solvency — Counterexample" + assert lines[2] == " ✓ bounds — No counterexample" + + +def test_gave_up_invariants_are_excluded(): + # Give-ups are surfaced in `failures`, not as verdicts. + result = _result(_delivered("bounds", "GOOD"), _gave_up("solvency")) + summary = summarize_verdicts(result, "crucible") + assert [v.name for v in summary.verdicts] == ["bounds"] + + +def test_no_delivered_verdicts_renders_nothing(): + assert format_verdict_lines(summarize_verdicts(_result(_gave_up("x")), "crucible")) == [] + assert format_verdict_lines(summarize_verdicts(_result(), "crucible")) == [] + + +def test_delivered_without_baked_verdict_is_unknown(): + res = RustFormalResult(verdicts={}) + outcome = ComponentOutcome(_feat("x"), [], Delivered(res, Path("fuzz/x.rs"))) + summary = summarize_verdicts(_result(outcome), "crucible") + assert summary.counts == {Outcome.UNKNOWN: 1} diff --git a/tests/test_crucible_sandbox_gate.py b/tests/test_crucible_sandbox_gate.py new file mode 100644 index 00000000..65ab84f5 --- /dev/null +++ b/tests/test_crucible_sandbox_gate.py @@ -0,0 +1,64 @@ +"""Phase-6 gate — Part B: the legitimate path works UNDER the launcher sandbox. + +Part A (the escape suite proving confinement holds — every exfil vector denied) is +[tests/test_sandbox_escape.py], runnable without the Solana toolchain. This file is +Part B: the *real* toolchain runs confined + offline. + +`test_solana_vault_builds_under_launcher` builds the real `solana_vault` program with +`cargo-build-sbf` inside the `run-confined` launcher (network off, `CARGO_NET_OFFLINE`) +and asserts the `.so` is produced — proving the policy grants exactly the toolchain a +real sBPF build needs, and that the offline build resolves against the warm cache. No +LLM. + +The FULL vertical (shared fixture + per-instruction harness build + fuzz, all confined) +is the existing e2e gate run with the launcher enabled — the generic host builds the +launcher policy from the wheel's `confine_by_default` + `sandbox_grants` and honors +`$COMPOSER_SANDBOX_PROVIDER`, so no separate test is needed: + + CRUCIBLE_REPO=/path/to/crucible COMPOSER_SANDBOX_PROVIDER=launcher \ + .venv/bin/python -m pytest tests/test_crucible_e2e_gate.py -m expensive -q -s + +Prereqs: `cargo-build-sbf` + `crucible` on PATH, a built `run-confined`, `CRUCIBLE_REPO`. +""" + +import os +import shutil +from pathlib import Path + +import pytest + +from composer.sandbox.config import SandboxConfig +from composer.sandbox.launcher import LauncherProvider +from composer.spec.solana.build import build_program + +pytestmark = [pytest.mark.expensive, pytest.mark.asyncio] + +_SCENARIO = Path(__file__).parent.parent / "test_scenarios" / "solana_vault" + + +def _require(cond: bool, why: str) -> None: + if not cond: + pytest.skip(why) + + +async def test_solana_vault_builds_under_launcher(): + _require(_SCENARIO.is_dir(), f"scenario missing: {_SCENARIO}") + _require(shutil.which("cargo-build-sbf") is not None, "cargo-build-sbf not on PATH") + _require(LauncherProvider().available().ok, "run-confined unbuilt or kernel lacks Landlock") + + # Warm the dep cache with an ordinary (unsandboxed) build first, so the confined + # offline build has everything it needs — mirrors the §5 fetch-outside / build-inside + # split on a fresh machine. + await build_program(_SCENARIO, "vault", timeout_s=480) + + cargo_home = Path(os.environ.get("CARGO_HOME", Path.home() / ".cargo")) + extra_ro: list[Path] = [Path.home() / ".cargo" / "bin"] + if (crucible := os.environ.get("CRUCIBLE_REPO")): + extra_ro.append(Path(crucible)) + cfg = SandboxConfig( + provider="launcher", + extra_ro=tuple(extra_ro), + extra_rw=(cargo_home,), + ) + built = await build_program(_SCENARIO, "vault", sandbox=cfg, timeout_s=480) + assert built.so_path.is_file(), "cargo-build-sbf under the launcher did not produce the .so" diff --git a/tests/test_crucible_setup_gate.py b/tests/test_crucible_setup_gate.py new file mode 100644 index 00000000..a4c6649c --- /dev/null +++ b/tests/test_crucible_setup_gate.py @@ -0,0 +1,199 @@ +"""Phase-3 gate: the Crucible fixture-authoring loop, with a REAL model. + +Drives the crucible wheel's `new_setup_session` decider through the IoC loop +(`drive_session` + `RealEffects`) on the `solana_vault` scenario: the agent reads +the program source (tool-enabled `call_llm`), authors a `Fixture`, and the loop +validates it with `crucible run … --dry-run`, revising on failure. Pass = the +session *publishes* a fixture (i.e. a dry-run went green) with no human edits. + +Heavy + paid: real LLM + Postgres (testcontainers) + the Solana/Anchor toolchain + +the `crucible` CLI + a local crucible checkout (`CRUCIBLE_REPO`). The first +harness build compiles litesvm/libafl (minutes); run +it in the background. Skips cleanly if a prerequisite is missing. + + CRUCIBLE_REPO=/path/to/crucible \ + .venv/bin/python -m pytest tests/test_crucible_setup_gate.py -m expensive -q -s +""" + +import asyncio +import json +import os +import shutil +from pathlib import Path +from types import SimpleNamespace +from typing import Any, cast, TYPE_CHECKING + +import psycopg +import pytest +from psycopg.sql import SQL, Identifier, Literal + +import composer.workflow.services as services +from composer.io.multi_job import TaskInfo +from composer.kb.knowledge_base import DefaultEmbedder +from composer.pipeline.core import GaveUp, PipelineRun +from composer.pipeline.ecosystem import RUST_FORBIDDEN_READ +from composer.rustapp.adapter import author_and_compile, make_emitter +from composer.rustapp.frontend import GenericRustConsoleHandler +from composer.rustapp.host import build_phase_enum, load_descriptor, load_module +from composer.spec.context import SourceCode, WorkflowContext +from composer.spec.service_host import ModelProvider, PureServiceHost +from composer.llm.registry import get_provider_for +from composer.spec.solana.build import build_program +from composer.spec.source.source_env import build_basic_source_tools, build_source_tools +from composer.spec.system_model import SolidityIdentifier +from composer.ui.tool_display import async_tool_context +from composer.workflow.services import llm_factory, standard_connections + +from tests.conftest import MockSentenceTransformer, needs_postgres +from tests.test_autoprove_integration import _MEMORIES_DDL, _RAG_DB, _VECTOR_DBS, _db_url + +if TYPE_CHECKING: + from testcontainers.postgres import PostgresContainer + +pytestmark = [pytest.mark.expensive, needs_postgres, pytest.mark.asyncio] + +_SCENARIO = Path(__file__).parent.parent / "test_scenarios" / "solana_vault" +_PROGRAM = "vault" + +# Minimal analyzed model — enough context for the prompt; the agent reads the real +# source for exact signatures. (The full front-half analysis produces this in prod.) +_ANALYZED: dict = { + "programs": [ + { + "name": "vault", + "program_identifier": "vault", + "description": "A lamports vault: each user owns a PDA vault (seeds [b\"vault\", authority]) " + "holding SOL, recording an authority and a balance.", + "instructions": [ + {"name": "initialize", "description": "Create the caller's vault PDA; set authority=caller, balance=0."}, + {"name": "deposit", "args": ["amount: u64"], "description": "Transfer `amount` lamports from the depositor into the vault PDA (System Program transfer); increase balance."}, + {"name": "withdraw", "args": ["amount: u64"], "description": "Transfer `amount` lamports from the vault to the authority (only the authority may sign); decrease balance."}, + ], + } + ], +} + + +def _model_args() -> object: + return SimpleNamespace( + heavy_model="claude-opus-4-6", + lite_model="claude-sonnet-4-6", + tokens=128_000, + thinking_tokens=2048, + memory_tool=False, + interleaved_thinking=False, + ) + + +def _crucible_repo() -> Path | None: + raw = os.environ.get("CRUCIBLE_REPO") + if not raw: + return None + repo = Path(raw) + return repo if (repo / "crates" / "crucible-fuzzer").is_dir() else None + + +def _require(cond: bool, why: str) -> None: + if not cond: + pytest.skip(why) + + +async def test_crucible_fixture_authoring(pg_container: "PostgresContainer", monkeypatch): + _require(_SCENARIO.is_dir(), f"scenario missing: {_SCENARIO}") + _require(shutil.which("cargo-build-sbf") is not None, "cargo-build-sbf not on PATH") + _require(shutil.which("crucible") is not None, "crucible CLI not on PATH") + crucible_repo = _crucible_repo() + _require(crucible_repo is not None, "set CRUCIBLE_REPO to a local crucible checkout") + assert crucible_repo is not None + + # --- Postgres roles + databases (matches services._DATABASE_CONFIGS) --- + admin_url = pg_container.get_connection_url(driver=None) + with psycopg.connect(admin_url, autocommit=True) as admin: + for cfg in services._DATABASE_CONFIGS.values(): + admin.execute(SQL("CREATE ROLE {} LOGIN PASSWORD {}").format( + Identifier(cfg["user"]), Literal(cfg["password"]))) + admin.execute(SQL("CREATE DATABASE {} OWNER {}").format( + Identifier(cfg["database"]), Identifier(cfg["user"]))) + admin.execute(SQL("CREATE DATABASE {}").format(Identifier(_RAG_DB))) + for db in _VECTOR_DBS: + with psycopg.connect(_db_url(pg_container, db), autocommit=True) as conn: + conn.execute("CREATE EXTENSION IF NOT EXISTS vector") + mem = services._DATABASE_CONFIGS["memory"] + mem_url = ( + f"postgresql://{mem['user']}:{mem['password']}" + f"@{pg_container.get_container_host_ip()}:{pg_container.get_exposed_port(5432)}/{mem['database']}" + ) + with psycopg.connect(mem_url, autocommit=True) as conn: + conn.execute(_MEMORIES_DDL) + + monkeypatch.setenv("CERTORA_AI_COMPOSER_PGHOST", pg_container.get_container_host_ip()) + monkeypatch.setenv("CERTORA_AI_COMPOSER_PGPORT", str(pg_container.get_exposed_port(5432))) + + args = _model_args() + embedder = MockSentenceTransformer() + + # --- Build the program up front (target/deploy/vault.so) --- + built = await build_program(_SCENARIO, _PROGRAM, timeout_s=590) + assert built.so_path.is_file(), built.so_path + + async with ( + standard_connections(provider="anthropic", embedder=DefaultEmbedder(embedder)) as conns, + async_tool_context(), + ): + content = await conns.uploader.get_document(_SCENARIO / "system.md") + assert content is not None + source = SourceCode( + content=content, + project_root=str(_SCENARIO), + contract_name=SolidityIdentifier(_PROGRAM), + relative_path=f"programs/{_PROGRAM}/src/lib.rs", + forbidden_read=RUST_FORBIDDEN_READ, + ) + _tiered = get_provider_for(tiered=cast(Any, args)) + model_provider = ModelProvider( + heavy_model=_tiered.heavy, lite_model=_tiered.lite, checkpointer=conns.checkpointer, + ) + basic = build_basic_source_tools(root=str(_SCENARIO), forbidden_read=RUST_FORBIDDEN_READ) + full = build_source_tools(basic, model_provider, conns.indexed_store, ("crucible_setup", "src"), recursion_limit=100) + env = PureServiceHost(models=model_provider, rag_tools=(), sort="existing").bind_source_tools(full) + + ctx = WorkflowContext.create( + services=conns.memory, + thread_id="crucible_setup", store=conns.store, recursion_limit=100, + cache_namespace=None, memory_namespace=None, + ) + + # No manifest pre-placement needed: `compile` materializes the harness crate + # (Cargo.toml + main.rs) itself per run (docs/rust-pure-app.md §4). + module = load_module("crucible_app") + descriptor = load_descriptor(module) + phase = build_phase_enum(descriptor) + + # The setup artifact: author the fixture, then `compile` (crucible --dry-run) it. + # Unsandboxed here (the gate trusts its inputs), so run_confined=None. + setup_input = { + "kind": "setup", "program": _PROGRAM, + "component": _ANALYZED, "props": [], "context": {}, + } + sandbox_dict = {"run_confined": None, "timeout_s": 1200} + run = PipelineRun(ctx=ctx, source=source, _handler_factory=GenericRustConsoleHandler(set()).make_handler, _semaphore=asyncio.Semaphore(2), env=env) + + result = await run.runner( + TaskInfo("crucible_setup", "Build Harness", phase["build_harness"]), + lambda: author_and_compile( + module, setup_input, env=env, sandbox_dict=sandbox_dict, + workdir=Path(_SCENARIO), recursion_limit=100, backend_name="crucible", + emit=make_emitter(), + ), + ) + + if isinstance(result, GaveUp): + pytest.fail(f"fixture authoring gave up: {result.reason}") + fixture = result + print("\n===== authored fixture =====\n" + fixture) + # It published, so a --dry-run went green. Sanity-check the shape. + assert "struct Fixture" in fixture, "fixture must define `struct Fixture`" + assert "fuzz_fixture" in fixture, "fixture must use #[fuzz_fixture]" + # And it must NOT smuggle in a test fn (those are per-component, later). Check for + # the attributes, not bare substrings — `use crucible_fuzzer::*` is expected. + assert "#[invariant_test]" not in fixture and "#[crucible_fuzz]" not in fixture diff --git a/tests/test_rust_frontend.py b/tests/test_rust_frontend.py new file mode 100644 index 00000000..5b4e25e7 --- /dev/null +++ b/tests/test_rust_frontend.py @@ -0,0 +1,70 @@ +"""Generic Rust frontend event routing (no wheel / no running TUI). + +A declared ``notice`` event kind (e.g. Crucible's per-invariant ``verdict``) must be +surfaced as a persistent callout via ``post_notice`` — not buried in the collapsible +events log — while ordinary kinds still stream to the log and undeclared kinds are +ignored. The notice headline carries an outcome glyph (✓/✗) when the payload has one. +""" + +import asyncio +from typing import Any, cast + +from composer.rustapp.frontend import GenericRustTaskHandler, _notice_headline +from composer.ui.tool_display import ToolDisplayConfig + + +def test_notice_headline_prefixes_outcome_glyph(): + assert _notice_headline({"outcome": "GOOD", "line": "held"}) == "✓ held" + assert _notice_headline({"outcome": "BAD", "line": "refuted"}) == "✗ refuted" + + +def test_notice_headline_without_outcome_is_plain_line(): + assert _notice_headline({"line": "building…"}) == "building…" + + +class _RecordingHandler(GenericRustTaskHandler): + """Records where each event is routed, bypassing real textual mounting.""" + + def __init__(self, event_kinds: set[str], notice_kinds: set[str]): + super().__init__( + "t", "Label", cast(Any, None), cast(Any, None), ToolDisplayConfig(), + event_kinds, notice_kinds, + ) + self.notices: list[str] = [] + self.logged: list[str] = [] + + async def post_notice(self, headline, detail=None, *, toast=True): # type: ignore[override] + self.notices.append(headline if isinstance(headline, str) else headline.plain) + + async def _ensure_event_log(self): # type: ignore[override] + handler = self + + class _Log: + def write(self, line: str) -> None: + handler.logged.append(line) + + return _Log() + + +def _handle(handler: _RecordingHandler, payload: dict) -> None: + asyncio.run(handler.handle_event(payload, ["t"], "cp")) + + +def test_notice_kind_routes_to_post_notice_not_log(): + h = _RecordingHandler(event_kinds={"fuzz_pulse", "verdict"}, notice_kinds={"verdict"}) + _handle(h, {"type": "verdict", "outcome": "BAD", "line": "counterexample found"}) + assert h.notices == ["✗ counterexample found"] + assert h.logged == [] + + +def test_streaming_kind_routes_to_log_not_notice(): + h = _RecordingHandler(event_kinds={"fuzz_pulse", "verdict"}, notice_kinds={"verdict"}) + _handle(h, {"type": "fuzz_pulse", "line": "fuzzing…"}) + assert h.logged == ["[fuzz_pulse] fuzzing…"] + assert h.notices == [] + + +def test_undeclared_kind_is_ignored(): + h = _RecordingHandler(event_kinds={"verdict"}, notice_kinds={"verdict"}) + _handle(h, {"type": "mystery", "line": "nope"}) + assert h.notices == [] and h.logged == [] diff --git a/tests/test_rust_llm_agent.py b/tests/test_rust_llm_agent.py new file mode 100644 index 00000000..d719d51e --- /dev/null +++ b/tests/test_rust_llm_agent.py @@ -0,0 +1,31 @@ +"""The tool-enabled authoring turn's prompt handling (no wheel / LLM needed). + +The backend owns the prompt: its author/judge prompt payload carries the ``instruction`` +and may define its own ``system`` prompt; otherwise a neutral, backend-agnostic default +applies (no language/domain leak — the trigger for this was the old prompt hardcoding +"Rust-based"). Lives in ``composer.rustapp.adapter`` (merged from the former ``_llm_agent``). +""" + +from composer.rustapp.adapter import _DEFAULT_SYS_PROMPT, _split_prompt + + +def test_bare_string_is_the_instruction_with_default_system(): + assert _split_prompt("do the thing") == (None, "do the thing") + + +def test_dict_instruction_extracted_cleanly(): + # Not JSON-wrapped (the old behavior dumped the whole dict as the prompt). + assert _split_prompt({"instruction": "author X"}) == (None, "author X") + + +def test_backend_may_define_its_own_system_prompt(): + assert _split_prompt({"system": "you are a fuzz author", "instruction": "author X"}) == ( + "you are a fuzz author", + "author X", + ) + + +def test_default_system_prompt_is_backend_agnostic(): + # No language/domain leak; still conveys the result-tool contract. + assert "Rust" not in _DEFAULT_SYS_PROMPT + assert "result" in _DEFAULT_SYS_PROMPT diff --git a/tests/test_rustapp.py b/tests/test_rustapp.py new file mode 100644 index 00000000..84965890 --- /dev/null +++ b/tests/test_rustapp.py @@ -0,0 +1,228 @@ +"""End-to-end tests for the Rust application/backend framework (composer.rustapp). + +These drive the ``echoprover`` demo wheel (built from ``rust/example-app``) as a +:class:`~autoprover_sdk.Backend`: the pure callouts (``descriptor`` / ``units`` / +``author_prompt`` / ``compile`` / ``validate``) plus the descriptor synthesis and the +host wiring. They need the ``echoprover`` wheel importable (``maturin develop`` in +``rust/example-app``); tests skip cleanly otherwise. + +No Postgres / LLM is required — the callouts are pure (echoprover's ``compile`` is a +no-op and ``validate`` returns GOOD), which is the point of the passive-service design: +the loop lives in Python and the wheel just answers questions. +""" + +import json + +import pytest + +echoprover = pytest.importorskip( + "echoprover", + reason="build the demo wheel first: (cd rust/example-app && maturin develop)", +) + +from composer.rustapp.descriptor import AppDescriptor, CoreSlot +from composer.rustapp.result import RustFormalResult + + +def _component_input(*titles: str) -> str: + return json.dumps( + { + "kind": "component", + "program": "Counter", + "component": {"name": "Counter"}, + "props": [ + {"title": t, "sort": "invariant", "description": "x", "slug": t.replace(" ", "_")} + for t in titles + ], + "context": {}, + } + ) + + +def test_descriptor_parses_and_maps_core_phases(): + desc = AppDescriptor.model_validate_json(echoprover.descriptor()) + assert desc.name == "echoprover" + assert desc.backend_tag == "echoprover" + # All four core slots are mapped, plus a UI-only "solving" phase. + slots = desc.core_slot_map() + assert set(slots) == set(CoreSlot) + keys = [p.key for p in desc.ordered_phases()] + assert keys == ["analysis", "extraction", "solving", "formalization", "report"] + + +def test_units_are_one_per_property(): + units = json.loads(echoprover.units(_component_input("increment_increases", "never_overflows"))) + assert units == [ + {"property": "increment_increases", "unit": "rule_increment_increases"}, + {"property": "never_overflows", "unit": "rule_never_overflows"}, + ] + + +def test_author_prompt_lists_the_properties(): + prompt = json.loads(echoprover.author_prompt(_component_input("increment_increases"), None)) + assert "increment_increases" in prompt["instruction"] + assert prompt.get("system") is None + + +def test_compile_is_a_noop_ok(): + # The demo accepts any well-formed spec — compile is a no-op gate. + r = json.loads(echoprover.compile(_component_input("p"), "spec", "/tmp", json.dumps({"run_confined": None}))) + assert r == {"status": "ok"} + + +def test_validate_returns_a_good_verdict(): + res = json.loads( + echoprover.validate(_component_input("p"), "spec", "rule_p", "/tmp", json.dumps({"run_confined": None})) + ) + # ValidateOutcome: the demo always builds, so a verdict (not build_failed). + assert res == {"kind": "verdict", "verdict": {"outcome": "GOOD"}} + + +def test_result_round_trips_through_cache_serialization(): + # The driver caches by model_dump/validate; ensure that survives. + res = RustFormalResult.from_formalized( + { + "commentary": "c", + "artifact_text": "spec", + "property_units": [("p", ["rule_p"])], + "skipped": [{"property_title": "q", "reason": "n/a"}], + "output_link": "local://x", + } + ) + reloaded = RustFormalResult.model_validate_json(res.model_dump_json()) + assert reloaded.property_units() == [("p", ["rule_p"])] + assert reloaded.artifact_text == "spec" + assert reloaded.skipped[0].property_title == "q" + + +# --------------------------------------------------------------------------- +# Generic host: entry point (argparse), shared-enum identity, frontend. +# These import the heavier host (needs the full composer stack). If it can't +# import (e.g. running against a slim env), skip rather than error. +# --------------------------------------------------------------------------- + +host = pytest.importorskip( + "composer.rustapp.host", reason="needs the full composer stack installed" +) + + +def test_entry_argparser_has_positionals_and_declared_flags(): + from composer.rustapp.entry import build_arg_parser + + app = host.build_application("echoprover") + parser = build_arg_parser(app) + + # Declared flag default (from the descriptor's ArgSpec) is applied. + args = parser.parse_args(["/proj", "src/C.sol:C", "doc.md"]) + assert args.project_root == "/proj" + assert args.main_contract == "src/C.sol:C" + assert args.system_doc == "doc.md" + assert args.max_concurrent == 4 + assert args.echo_tag == "demo" + + # …and is overridable. + args2 = parser.parse_args(["/proj", "src/C.sol:C", "doc.md", "--echo-tag", "hi"]) + assert args2.echo_tag == "hi" + + +def test_system_doc_is_optional_with_discovery_phase_fallback(): + from composer.rustapp.entry import _discovery_phase, build_arg_parser + + app = host.build_application("echoprover") + parser = build_arg_parser(app) + + # system_doc may be omitted (→ discovery); still parses. + ns = parser.parse_args(["/proj", "src/C.sol:C"]) + assert ns.system_doc is None + assert parser.parse_args(["/proj", "src/C.sol:C", "doc.md"]).system_doc == "doc.md" + + # A wheel that declares no discover_design_doc phase falls back to its first phase. + first_key = app.descriptor.ordered_phases()[0].key + assert _discovery_phase(app) is app.phase[first_key] + + +def test_frontend_labels_and_backend_phases_share_one_enum(): + # The correctness invariant: the phases the driver stamps on TaskInfo (from + # the backend's core_phases) must be the SAME enum members the frontend's + # phase_labels are keyed by, or label lookup silently misses. + from composer.input.files import InMemoryTextFile + from composer.spec.context import SourceCode + from composer.spec.system_model import SolidityIdentifier + + app = host.build_application("echoprover") + source = SourceCode( + content=InMemoryTextFile(basename="doc.md", string_contents="doc", provider="test"), + project_root="/tmp/echo-proj", + contract_name=SolidityIdentifier("C"), + relative_path="src/C.sol", + forbidden_read="", + ) + backend = app.make_backend(source) + for slot, member in backend.core_phases.items(): + assert member in app.phase_labels, (slot, member) + # Section order lists every declared phase's label. + assert set(app.section_order) == set(app.phase_labels.values()) + + +def test_generic_console_handler_renders_declared_events(capsys): + import asyncio + + from composer.rustapp.frontend import GenericRustConsoleHandler, _render_event + + assert _render_event({"type": "solver_line", "line": "hello"}) == "hello" + assert _render_event({"type": "x", "a": 1}) == '{"a": 1}' + + handler = GenericRustConsoleHandler({"solver_line"}) + asyncio.run(handler.handle_event({"type": "solver_line", "line": "L1"}, ["t"], "cp")) + # An undeclared kind is ignored. + asyncio.run(handler.handle_event({"type": "other", "line": "nope"}, ["t"], "cp")) + out = capsys.readouterr().out + assert "solver_line: L1" in out + assert "nope" not in out + + +def test_generic_tui_app_constructs(): + from composer.rustapp.frontend import GenericRustApp + + app = host.build_application("echoprover") + tui = GenericRustApp( + phase_labels=app.phase_labels, + section_order=app.section_order, + header_text=app.header_text, + event_kinds={e.kind for e in app.descriptor.event_kinds}, + ) + assert tui is not None + + +def test_descriptor_carries_ecosystem_and_resolves(): + from composer.pipeline.ecosystem import EVM + from composer.rustapp.host import resolve_ecosystem + + desc = AppDescriptor.model_validate_json(echoprover.descriptor()) + assert desc.ecosystem == "evm" + assert resolve_ecosystem(desc) is EVM + + +def test_build_application_carries_resolved_ecosystem(): + from composer.pipeline.ecosystem import EVM + + app = host.build_application("echoprover") + assert app.ecosystem is EVM + + +def test_resolve_ecosystem_rejects_unregistered_chain(): + from composer.rustapp.host import resolve_ecosystem + + desc = AppDescriptor.model_validate_json(echoprover.descriptor()) + # soroban is a valid ChainTag but not registered until a later phase. + unregistered = desc.model_copy(update={"ecosystem": "soroban"}) + with pytest.raises(ValueError, match="not registered"): + resolve_ecosystem(unregistered) + + +def test_descriptor_ecosystem_defaults_to_evm_when_absent(): + # Wheels built before the field existed omit it; the mirror defaults to evm. + raw = json.loads(echoprover.descriptor()) + del raw["ecosystem"] + desc = AppDescriptor.model_validate(raw) + assert desc.ecosystem == "evm" diff --git a/tests/test_sandbox.py b/tests/test_sandbox.py new file mode 100644 index 00000000..f3618fa0 --- /dev/null +++ b/tests/test_sandbox.py @@ -0,0 +1,127 @@ +"""Unit tests for the command-sandbox provider seam (step 1). + +Pure and fast — no Rust wheel, no subprocess, no Postgres/LLM. They pin the +tool-agnostic contract (``docs/command-sandbox.md`` §4/§7) that every provider +must honor: the ``none`` passthrough is byte-for-byte today's behavior, the +registry resolves/rejects names, and the fail-closed check fires only when a +provider reports itself unavailable. +""" + +from pathlib import Path + +import pytest + +from composer.sandbox.policy import ( + Availability, + LaunchSpec, + NoneProvider, + SandboxPolicy, + SandboxProvider, + SandboxUnavailable, + ensure_available, + get_provider, + register_provider, +) + + +def test_policy_defaults_are_locked_down(): + """A default policy denies everything: no paths, empty env, network off, no caps.""" + p = SandboxPolicy() + assert p.rw_paths == () + assert p.ro_paths == () + assert dict(p.env_allowlist) == {} + assert p.network is False + assert p.mem_bytes is None and p.cpu_seconds is None + assert p.nproc is None and p.fsize_bytes is None + + +def test_policy_is_frozen(): + p = SandboxPolicy(rw_paths=(Path("/work"),)) + with pytest.raises((AttributeError, TypeError)): + p.network = True # type: ignore[misc] + + +def test_none_provider_is_a_passthrough(): + """``none`` execs the command verbatim and inherits the env (env is None).""" + spec = NoneProvider().wrap(SandboxPolicy(), "cargo", ["build", "--offline"]) + assert spec == LaunchSpec(argv=("cargo", "build", "--offline"), env=None) + + +def test_none_provider_ignores_policy(): + """Passthrough grants no isolation, so a rich policy must not alter its argv.""" + rich = SandboxPolicy( + rw_paths=(Path("/work"),), + ro_paths=(Path("/usr"),), + env_allowlist={"PATH": "/usr/bin"}, + network=True, + mem_bytes=1 << 32, + ) + spec = NoneProvider().wrap(rich, "echo", ["hi"]) + assert spec.argv == ("echo", "hi") + assert spec.env is None + + +def test_none_provider_available(): + assert NoneProvider().available() == Availability(ok=True) + + +def test_none_provider_satisfies_protocol(): + # runtime_checkable structural check: the concrete class implements the seam. + assert isinstance(NoneProvider(), SandboxProvider) + + +def test_get_provider_known(): + prov = get_provider("none") + assert isinstance(prov, NoneProvider) + assert prov.name == "none" + + +def test_get_provider_unknown_is_value_error(): + with pytest.raises(ValueError, match="unknown sandbox provider 'bogus'"): + get_provider("bogus") + + +def test_register_provider_roundtrip(): + """A newly registered factory becomes resolvable by name (how step 2 adds the + launcher without this module importing it).""" + + class _Fake: + name = "fake" + + def available(self) -> Availability: + return Availability(ok=True) + + def wrap(self, policy: SandboxPolicy, program: str, args: list[str]) -> LaunchSpec: + return LaunchSpec(argv=("fake", program, *args)) + + register_provider("fake", _Fake) + try: + assert isinstance(get_provider("fake"), _Fake) + finally: + # keep the module-level registry clean for other tests + from composer.sandbox import policy as _s + + _s._PROVIDERS.pop("fake", None) + + +def test_ensure_available_passes_for_ok_provider(): + ensure_available(NoneProvider()) # must not raise + + +def test_ensure_available_fails_closed(): + """An unavailable provider raises rather than letting the command run unconfined.""" + + class _Unavailable: + name = "landlock-missing" + + def available(self) -> Availability: + return Availability(ok=False, reason="kernel lacks Landlock (need Linux >= 5.13)") + + def wrap(self, policy: SandboxPolicy, program: str, args: list[str]) -> LaunchSpec: + raise AssertionError("wrap must not be reached when unavailable") + + with pytest.raises(SandboxUnavailable) as ei: + ensure_available(_Unavailable()) + assert ei.value.provider == "landlock-missing" + assert "Landlock" in ei.value.reason + assert "unavailable" in str(ei.value) diff --git a/tests/test_sandbox_command.py b/tests/test_sandbox_command.py new file mode 100644 index 00000000..62a6453f --- /dev/null +++ b/tests/test_sandbox_command.py @@ -0,0 +1,64 @@ +"""Unit tests for the shared local-command runner (``run_local_command``). + +Needs neither a Rust wheel nor Postgres/LLM: ``run_local_command`` shells out to +trivial system binaries. Covers file materialization, path confinement, and the +error/timeout paths. (It still backs the trusted Python build steps — e.g. the sBPF +build; the Rust backend's own toolchain runs now go through ``run-confined`` in the +wheel, see ``docs/rust-backend-api.md``.) +""" + +import pytest + +from composer.sandbox.command import ( + NOT_FOUND_EXIT, + UnsafePath, + run_local_command, +) + + +@pytest.mark.asyncio +async def test_run_local_command_materializes_files_and_captures_output(tmp_path): + res = await run_local_command( + "printf", ["%s", "hello"], {"note.txt": "hi", "sub/deep.txt": "deep"}, workdir=tmp_path + ) + assert res.exit_code == 0 + assert res.stdout == "hello" + # files (incl. a nested path) were materialized into the workdir. + assert (tmp_path / "note.txt").read_text() == "hi" + assert (tmp_path / "sub" / "deep.txt").read_text() == "deep" + + +@pytest.mark.asyncio +async def test_run_local_command_missing_binary(tmp_path): + res = await run_local_command("autoprover-no-such-binary-xyz", [], {}, workdir=tmp_path) + assert res.exit_code == NOT_FOUND_EXIT + assert "not found" in res.stderr + + +@pytest.mark.asyncio +async def test_run_local_command_nonzero_exit(tmp_path): + res = await run_local_command("false", [], {}, workdir=tmp_path) + assert res.exit_code != 0 + + +@pytest.mark.asyncio +@pytest.mark.parametrize("bad", ["../evil.txt", "/etc/evil", "a/../../evil"]) +async def test_run_local_command_rejects_path_escape(tmp_path, bad): + with pytest.raises(UnsafePath): + await run_local_command("true", [], {bad: "x"}, workdir=tmp_path) + + +@pytest.mark.asyncio +async def test_run_local_command_no_shell_injection(tmp_path): + # Args are argv, never a shell string: a shell metacharacter is inert. `printf` + # emits it literally rather than a subshell running `id`. + res = await run_local_command("printf", ["%s", "$(id)"], {}, workdir=tmp_path) + assert res.exit_code == 0 + assert res.stdout == "$(id)" + + +@pytest.mark.asyncio +async def test_run_local_command_timeout(tmp_path): + res = await run_local_command("sleep", ["5"], {}, workdir=tmp_path, timeout_s=1) + assert res.exit_code == -1 + assert "timed out" in res.stderr diff --git a/tests/test_sandbox_config.py b/tests/test_sandbox_config.py new file mode 100644 index 00000000..da7e0c0c --- /dev/null +++ b/tests/test_sandbox_config.py @@ -0,0 +1,102 @@ +"""Unit tests for the sandbox config + the Rust-build policy recipe (step 3). + +Pure: no subprocess, no Rust binary. They pin provider selection (default ``none``, +``$COMPOSER_SANDBOX_PROVIDER`` override) and that the recipe grants the workdir +read-write, discoverable toolchain dirs read-only, and a scrubbed env with the +network off. +""" + +from pathlib import Path + +from composer.sandbox.config import SandboxConfig +from composer.sandbox.launcher import LauncherProvider +from composer.sandbox.policy import NoneProvider, SandboxPolicy +from composer.sandbox.recipes import rust_build_policy + + +def test_config_default_is_none_and_disabled(): + cfg = SandboxConfig() + assert cfg.provider == "none" + assert cfg.enabled is False + assert isinstance(cfg.resolve_provider(), NoneProvider) + + +def test_config_from_env_default(monkeypatch): + monkeypatch.delenv("COMPOSER_SANDBOX_PROVIDER", raising=False) + assert SandboxConfig.from_env().provider == "none" + + +def test_config_from_env_launcher(monkeypatch): + monkeypatch.setenv("COMPOSER_SANDBOX_PROVIDER", "launcher") + cfg = SandboxConfig.from_env(extra_ro=(Path("/usr"),)) + assert cfg.provider == "launcher" + assert cfg.enabled is True + assert isinstance(cfg.resolve_provider(), LauncherProvider) + assert cfg.extra_ro == (Path("/usr"),) + + +def test_config_none_build_policy_is_empty(): + """The none provider ignores the policy, so build_policy returns a bare one.""" + pol = SandboxConfig().build_policy("/work") + assert pol == SandboxPolicy() + + +def test_config_enabled_build_policy_grants_workdir(tmp_path): + cfg = SandboxConfig(provider="launcher", mem_bytes=1 << 30) + pol = cfg.build_policy(tmp_path) + assert tmp_path in pol.rw_paths + assert pol.network is False + assert pol.mem_bytes == (1 << 30) + + +def test_rust_build_policy_shape(tmp_path, monkeypatch): + monkeypatch.setenv("PATH", "/usr/bin:/bin") + monkeypatch.setenv("MY_SECRET", "do-not-pass") + extra_ro_dir = tmp_path / "toolchain" + extra_ro_dir.mkdir() + extra_rw_dir = tmp_path / "scratch" + extra_rw_dir.mkdir() + + pol = rust_build_policy( + tmp_path, + extra_ro=(extra_ro_dir, tmp_path / "does-not-exist"), + extra_rw=(extra_rw_dir,), + cpu_seconds=900, + ) + + # workdir + existing extra_rw are writable + assert tmp_path in pol.rw_paths + assert extra_rw_dir in pol.rw_paths + # existing extra_ro granted; non-existent dropped + assert extra_ro_dir in pol.ro_paths + assert (tmp_path / "does-not-exist") not in pol.ro_paths + # env: only allowlisted names pass through; secrets do not + assert pol.env_allowlist.get("PATH") == "/usr/bin:/bin" + assert "MY_SECRET" not in pol.env_allowlist + # network off, caps threaded + assert pol.network is False + assert pol.cpu_seconds == 900 + + +def test_rust_build_policy_offline_sets_cargo_net_offline(tmp_path): + """Default (offline) forces every cargo — incl. the one `crucible run` spawns — + offline via CARGO_NET_OFFLINE; opting out drops it.""" + on = rust_build_policy(tmp_path) + assert on.env_allowlist.get("CARGO_NET_OFFLINE") == "1" + off = rust_build_policy(tmp_path, offline=False) + assert "CARGO_NET_OFFLINE" not in off.env_allowlist + + +def test_config_enabled_policy_is_offline_by_default(tmp_path): + pol = SandboxConfig(provider="launcher").build_policy(tmp_path) + assert pol.env_allowlist.get("CARGO_NET_OFFLINE") == "1" + pol_net = SandboxConfig(provider="launcher", offline=False).build_policy(tmp_path) + assert "CARGO_NET_OFFLINE" not in pol_net.env_allowlist + + +def test_rust_build_policy_includes_system_and_dev_when_present(): + pol = rust_build_policy("/tmp") + if Path("/usr").exists(): + assert Path("/usr") in pol.ro_paths + if Path("/dev/null").exists(): + assert Path("/dev/null") in pol.rw_paths diff --git a/tests/test_sandbox_escape.py b/tests/test_sandbox_escape.py new file mode 100644 index 00000000..93752b4f --- /dev/null +++ b/tests/test_sandbox_escape.py @@ -0,0 +1,143 @@ +"""The escape suite — Part A of the Phase-6 gate (docs/command-sandbox.md §10). + +A *malicious* program (standing in for a harness `setup()` / a program's `build.rs`) +is compiled with `rustc`, then run through the **real** `run-confined` launcher via +`run_local_command` under a Crucible-representative policy (`rust_build_policy`). It +attempts every escape and writes each result into the workdir (allowed); the test +reads them back and asserts *denied* for all. A no-sandbox control runs the same +binary unconfined and confirms the leaks would otherwise happen — proving it is the +sandbox doing the blocking. + +Runnable without the full Crucible stack (std-only program, no crates, no network +needed to compile). Skipped unless `rustc` and a working launcher are present. The +*legitimate* half (a real `solana_vault` build+fuzz under the launcher) is the +expensive Part B in `tests/test_crucible_sandbox_gate.py`. +""" + +import asyncio +import os +import shutil +import subprocess +from pathlib import Path + +import pytest + +from composer.sandbox.command import run_local_command +from composer.sandbox.launcher import LauncherProvider +from composer.sandbox.recipes import rust_build_policy + +pytestmark = pytest.mark.asyncio + +_PROVIDER = LauncherProvider() +_needs = pytest.mark.skipif( + shutil.which("rustc") is None or not _PROVIDER.available().ok, + reason="needs rustc + a working run-confined launcher (Linux/Landlock)", +) + +_ENV_CANARY = "ENVCANARY-a1b2c3" +_HOSTFILE_CANARY = "HOSTFILECANARY-d4e5f6" + +# Standing in for hostile code in setup()/build.rs. std-only so it compiles offline. +_MALICIOUS_RS = """ +use std::fs; +use std::net::{SocketAddr, TcpStream}; +use std::time::Duration; + +fn probe(name: &str, result: &str) { + let _ = fs::write(format!("probe_{}.txt", name), result); +} + +fn net(addr: &str) -> String { + let sa: SocketAddr = addr.parse().unwrap(); + match TcpStream::connect_timeout(&sa, Duration::from_secs(2)) { + Ok(_) => "LEAK:connected".to_string(), + Err(_) => "denied".to_string(), + } +} + +fn main() { + let args: Vec<String> = std::env::args().collect(); + let outside = args.get(1).cloned().unwrap_or_default(); + let parent_pid = args.get(2).cloned().unwrap_or_default(); + + probe("env", &match std::env::var("ANTHROPIC_API_KEY") { + Ok(v) => format!("LEAK:{}", v), + Err(_) => "denied".to_string(), + }); + + probe("procenv", &match fs::read_to_string(format!("/proc/{}/environ", parent_pid)) { + Ok(s) if s.contains("ENVCANARY") => "LEAK:found-canary".to_string(), + Ok(_) => "LEAK:proc-readable".to_string(), + Err(_) => "denied".to_string(), + }); + + probe("hostfile", &match fs::read_to_string(&outside) { + Ok(s) => format!("LEAK:{}", s.trim()), + Err(_) => "denied".to_string(), + }); + + probe("net_ext", &net("1.1.1.1:80")); + probe("imds", &net("169.254.169.254:80")); +} +""" + + +def _compile(tmp_path: Path, workdir: Path) -> None: + src = tmp_path / "malicious.rs" + src.write_text(_MALICIOUS_RS) + # Compiled UNSANDBOXED (we're testing runtime confinement, not the build here). + subprocess.run( + ["rustc", "-O", str(src), "-o", str(workdir / "malicious")], + check=True, + capture_output=True, + ) + + +@pytest.fixture +def scenario(tmp_path, monkeypatch): + workdir = tmp_path / "work" + workdir.mkdir() + _compile(tmp_path, workdir) + outside = tmp_path / "host_secret.txt" # OUTSIDE the granted workdir + outside.write_text(_HOSTFILE_CANARY) + # Plant the secret in *this* process's env; run-confined must scrub it, and the + # /proc/<ppid>/environ read (ppid = this pytest process) must be denied. + monkeypatch.setenv("ANTHROPIC_API_KEY", _ENV_CANARY) + return workdir, outside + + +@_needs +async def test_all_escapes_denied(scenario): + workdir, outside = scenario + policy = rust_build_policy(workdir) # grants workdir + toolchains; NOT /proc, NOT `outside` + res = await run_local_command( + "./malicious", [str(outside), str(os.getpid())], {}, + workdir=workdir, provider=_PROVIDER, policy=policy, + ) + assert res.exit_code == 0, res.stderr + + def probe(name: str) -> str: + return (workdir / f"probe_{name}.txt").read_text().strip() + + # every vector denied — and specifically no canary leaked + assert probe("env") == "denied" + assert probe("procenv") == "denied" + assert probe("hostfile") == "denied" + assert probe("net_ext") == "denied" + assert probe("imds") == "denied" + for name in ("env", "procenv", "hostfile", "net_ext", "imds"): + assert "LEAK" not in probe(name) + + +@_needs +async def test_control_unconfined_would_leak(scenario): + """Without the sandbox the same binary reads the secret env + the host file — + confirming the assertions above are enforced by the sandbox, not by accident.""" + workdir, outside = scenario + res = await run_local_command( + "./malicious", [str(outside), str(os.getpid())], {}, + workdir=workdir, # provider=None → unconfined passthrough + ) + assert res.exit_code == 0, res.stderr + assert (workdir / "probe_env.txt").read_text().strip() == f"LEAK:{_ENV_CANARY}" + assert _HOSTFILE_CANARY in (workdir / "probe_hostfile.txt").read_text() diff --git a/tests/test_sandbox_launcher.py b/tests/test_sandbox_launcher.py new file mode 100644 index 00000000..a1e36418 --- /dev/null +++ b/tests/test_sandbox_launcher.py @@ -0,0 +1,106 @@ +"""Tests for the ``run-confined`` launcher provider (Phase 6 step 2). + +The ``wrap`` tests are pure argv construction (no binary, no subprocess) and pin +the exact flag mapping. The ``available`` / ``--probe`` tests exercise the real +binary when it has been built (``cargo build -p run-confined --release``) and skip +otherwise, so the suite stays green on a machine without the Rust build. +""" + +from pathlib import Path + +import pytest + +from composer.sandbox.launcher import LauncherProvider, _resolve_binary +from composer.sandbox.policy import Availability, LaunchSpec, SandboxPolicy, get_provider + +_FAKE_BIN = "/opt/run-confined" + + +def _provider() -> LauncherProvider: + return LauncherProvider(binary=_FAKE_BIN) + + +def test_wrap_minimal_policy(): + """A workdir-only policy maps to the workdir grant + the command after `--`.""" + policy = SandboxPolicy(rw_paths=(Path("/work"),)) + spec = _provider().wrap(policy, "cargo", ["build", "--offline"]) + assert spec == LaunchSpec( + argv=(_FAKE_BIN, "--rw", "/work", "--", "cargo", "build", "--offline"), + env=None, + ) + + +def test_wrap_full_policy_flag_order(): + """ro before rw, then env, network, then rlimits, then `-- program args`.""" + policy = SandboxPolicy( + rw_paths=(Path("/work"), Path("/dev")), + ro_paths=(Path("/usr"), Path("/lib")), + env_allowlist={"PATH": "/usr/bin", "HOME": "/work"}, + network=False, + mem_bytes=4 << 30, + cpu_seconds=900, + nproc=512, + fsize_bytes=1 << 30, + ) + spec = _provider().wrap(policy, "crucible", ["run", "vault", "c_deposit"]) + assert spec.argv == ( + _FAKE_BIN, + "--ro", "/usr", + "--ro", "/lib", + "--rw", "/work", + "--rw", "/dev", + "--allow-env", "PATH=/usr/bin", + "--allow-env", "HOME=/work", + "--rlimit-as", str(4 << 30), + "--rlimit-cpu", "900", + "--rlimit-nproc", "512", + "--rlimit-fsize", str(1 << 30), + "--", "crucible", "run", "vault", "c_deposit", + ) + assert spec.env is None + + +def test_wrap_network_flag(): + policy = SandboxPolicy(rw_paths=(Path("/work"),), network=True) + spec = _provider().wrap(policy, "echo", []) + assert "--allow-network" in spec.argv + # no rlimit flags when caps are unset + assert not any(a.startswith("--rlimit") for a in spec.argv) + + +def test_wrap_uses_binary_name_when_unresolved(): + """With no binary resolved, argv[0] falls back to the bare name (kept runnable + if it is later placed on PATH); wrap never crashes on a missing binary.""" + prov = LauncherProvider(binary=None) + prov._binary = None # force the unresolved case regardless of the dev tree + spec = prov.wrap(SandboxPolicy(rw_paths=(Path("/w"),)), "true", []) + assert spec.argv[0] == "run-confined" + + +def test_available_reports_missing_binary(): + prov = LauncherProvider(binary=None) + prov._binary = None + avail = prov.available() + assert avail.ok is False + assert "run-confined" in avail.reason + + +def test_launcher_registered_in_seam(): + """Importing this module registered the provider under its name.""" + prov = get_provider("launcher") + assert isinstance(prov, LauncherProvider) + assert prov.name == "launcher" + + +# --- tests that need the actual built binary (skip if unbuilt) --- + +_REAL_BIN = _resolve_binary() +_needs_bin = pytest.mark.skipif(_REAL_BIN is None, reason="run-confined not built") + + +@_needs_bin +def test_probe_reports_available_on_this_host(): + """On a Landlock-capable host the real binary's --probe → available().""" + avail = LauncherProvider().available() + assert isinstance(avail, Availability) + assert avail.ok is True, avail.reason diff --git a/tests/test_sandbox_run_confined.py b/tests/test_sandbox_run_confined.py new file mode 100644 index 00000000..f9d1ba05 --- /dev/null +++ b/tests/test_sandbox_run_confined.py @@ -0,0 +1,105 @@ +"""Integration test: `run_local_command` actually confines via the launcher provider. + +This proves the *wiring* (step 3) end-to-end — the runner routes a command through +a real `SandboxProvider` and the confinement takes effect — as opposed to the pure +argv/golden tests elsewhere. Skipped unless the `run-confined` binary is built and +the kernel supports Landlock (so CI without the Rust build stays green); the full +escape gate on the real Crucible build is step 5. +""" + +import os +from pathlib import Path + +import pytest + +from composer.sandbox.command import run_local_command +from composer.sandbox.launcher import LauncherProvider +from composer.sandbox.policy import SandboxPolicy, SandboxUnavailable + +pytestmark = pytest.mark.asyncio + +_PROVIDER = LauncherProvider() +_needs_sandbox = pytest.mark.skipif( + not _PROVIDER.available().ok, reason="run-confined unbuilt or kernel lacks Landlock" +) + + +def _system_policy(workdir: Path) -> SandboxPolicy: + """A minimal policy: workdir + the dev nodes rw, the system dirs ro. Deliberately + does NOT grant /etc, so reading a host file outside the workdir is denied.""" + ro = tuple(p for p in (Path("/usr"), Path("/lib"), Path("/lib64"), Path("/bin")) if p.exists()) + rw = (workdir, *(Path(d) for d in ("/dev/null", "/dev/urandom") if Path(d).exists())) + return SandboxPolicy(rw_paths=rw, ro_paths=ro, env_allowlist={"PATH": os.environ.get("PATH", "/usr/bin:/bin")}) + + +@_needs_sandbox +async def test_confined_command_can_write_workdir(tmp_path): + res = await run_local_command( + "bash", ["-c", "echo hi > w.txt"], {}, workdir=tmp_path, + provider=_PROVIDER, policy=_system_policy(tmp_path), + ) + assert res.exit_code == 0, res.stderr + assert (tmp_path / "w.txt").read_text().strip() == "hi" + + +@_needs_sandbox +async def test_confined_command_cannot_read_outside_workdir(tmp_path): + outside = tmp_path.parent / f"secret-{tmp_path.name}.txt" + outside.write_text("TOPSECRET") + try: + res = await run_local_command( + "bash", ["-c", f"cat {outside} && echo LEAK || echo denied"], {}, workdir=tmp_path, + provider=_PROVIDER, policy=_system_policy(tmp_path), + ) + finally: + outside.unlink(missing_ok=True) + assert "TOPSECRET" not in res.stdout + assert "LEAK" not in res.stdout + assert "denied" in res.stdout + + +@_needs_sandbox +async def test_confined_command_has_no_network(tmp_path): + res = await run_local_command( + "python3", + ["-c", "import socket; socket.socket(socket.AF_INET, socket.SOCK_STREAM); print('LEAK')"], + {}, workdir=tmp_path, provider=_PROVIDER, policy=_system_policy(tmp_path), + ) + assert res.exit_code != 0 + assert "LEAK" not in res.stdout + + +@_needs_sandbox +async def test_none_provider_is_not_confined(tmp_path): + """Control: without a provider the same outside-read succeeds — proving it is the + sandbox, not something else, doing the blocking above.""" + outside = tmp_path.parent / f"plain-{tmp_path.name}.txt" + outside.write_text("readable") + try: + res = await run_local_command( + "bash", ["-c", f"cat {outside}"], {}, workdir=tmp_path, # provider=None (passthrough) + ) + finally: + outside.unlink(missing_ok=True) + assert res.exit_code == 0 + assert "readable" in res.stdout + + +async def test_unavailable_provider_fails_closed(tmp_path): + """A provider that reports unavailable must raise, never run unconfined.""" + + class _Unavailable: + name = "x" + + def available(self): + from composer.sandbox.policy import Availability + + return Availability(ok=False, reason="nope") + + def wrap(self, policy, program, args): # pragma: no cover - must not be called + raise AssertionError("wrap reached despite unavailable") + + with pytest.raises(SandboxUnavailable): + await run_local_command( + "true", [], {}, workdir=tmp_path, provider=_Unavailable(), policy=SandboxPolicy() + ) diff --git a/tests/test_solana_build.py b/tests/test_solana_build.py new file mode 100644 index 00000000..9b935d48 --- /dev/null +++ b/tests/test_solana_build.py @@ -0,0 +1,80 @@ +"""Tests for the Solana build step's offline warming (Phase 6 step 4). + +Pin the §5 split: the registry is warmed with network *outside* the sandbox +(`warm_cargo_cache` → `cargo fetch`, no provider) only when a sandbox is enabled, +and the actual build is then handed the provider (so it runs confined + offline). +Uses fakes — no real cargo/toolchain. +""" + +from pathlib import Path + +import pytest + +import composer.spec.solana.build as buildmod +from composer.sandbox.command import CommandResult +from composer.sandbox.config import SandboxConfig + +pytestmark = pytest.mark.asyncio + + +async def test_warm_cargo_cache_runs_unsandboxed_fetch(tmp_path, monkeypatch): + seen = {} + + async def fake_run(program, args, files, *, workdir, timeout_s=600, sem=None, + provider=None, policy=None, env_overlay=None): + seen.update(program=program, args=args, workdir=Path(workdir), + provider=provider, env_overlay=env_overlay) + return CommandResult(0, "", "") + + monkeypatch.setattr(buildmod, "run_local_command", fake_run) + res = await buildmod.warm_cargo_cache(tmp_path, cargo_home=tmp_path / ".sandbox_cargo") + assert res.exit_code == 0 + assert seen["program"] == "cargo" and seen["args"] == ["fetch"] + assert seen["workdir"] == tmp_path + assert seen["provider"] is None # warming must NOT be sandboxed (it needs network) + # fetch targets the private per-run CARGO_HOME (same one the offline build reads) + assert seen["env_overlay"] == {"CARGO_HOME": str(tmp_path / ".sandbox_cargo")} + + +async def _fake_build_run_factory(calls): + async def fake_run(program, args, files, *, workdir, timeout_s=600, sem=None, + provider=None, policy=None, env_overlay=None): + calls.append((program, provider is not None)) + so = Path(workdir) / "target" / "deploy" / "vault.so" + so.parent.mkdir(parents=True, exist_ok=True) + so.write_text("") # simulate a produced .so so build_program succeeds + return CommandResult(0, "", "") + + return fake_run + + +async def test_build_program_skips_warm_when_unsandboxed(tmp_path, monkeypatch): + calls: list = [] + warm_count = {"n": 0} + + async def fake_warm(*a, **k): + warm_count["n"] += 1 + return CommandResult(0, "", "") + + monkeypatch.setattr(buildmod, "warm_cargo_cache", fake_warm) + monkeypatch.setattr(buildmod, "run_local_command", await _fake_build_run_factory(calls)) + + await buildmod.build_program(tmp_path, "vault") # no sandbox → no warm, no provider + assert warm_count["n"] == 0 + assert calls and calls[0][1] is False + + +async def test_build_program_warms_and_sandboxes_when_enabled(tmp_path, monkeypatch): + calls: list = [] + warm_count = {"n": 0} + + async def fake_warm(*a, **k): + warm_count["n"] += 1 + return CommandResult(0, "", "") + + monkeypatch.setattr(buildmod, "warm_cargo_cache", fake_warm) + monkeypatch.setattr(buildmod, "run_local_command", await _fake_build_run_factory(calls)) + + await buildmod.build_program(tmp_path, "vault", sandbox=SandboxConfig(provider="launcher")) + assert warm_count["n"] == 1 # warmed once, outside the sandbox + assert calls[0][1] is True # the build command received the provider diff --git a/tests/test_solana_gate.py b/tests/test_solana_gate.py new file mode 100644 index 00000000..cfab5eb4 --- /dev/null +++ b/tests/test_solana_gate.py @@ -0,0 +1,123 @@ +"""Phase-4 gate: the Solana front half (analysis + property extraction) end-to-end. + +Runs the shared driver over the SOLANA ecosystem + a null backend on a sample Anchor vault, +with REAL models. Everything else (Postgres, source tools) runs for real; there is no prover or +AutoSetup (the null backend just records properties). Pass = the pipeline runs to completion and +extracts a non-empty set of properties; the printed properties let a human judge "sane". + +Marked ``expensive`` (real LLM + containers). Run with: + env -u CERTORA .venv/bin/python -m pytest tests/test_solana_gate.py -m expensive -q -s +""" +from pathlib import Path +from types import SimpleNamespace +from typing import Any, cast, TYPE_CHECKING + +import psycopg +import pytest +from psycopg.sql import SQL, Identifier, Literal + +import composer.workflow.services as services +from composer.pipeline.core import PipelineRun, run_pipeline +from composer.pipeline.ecosystem import SOLANA, RUST_FORBIDDEN_READ +from composer.spec.context import SourceCode, WorkflowContext +from composer.spec.service_host import ModelProvider, PureServiceHost +from composer.llm.registry import get_provider_for +from composer.spec.solana.null_backend import NullSolanaArtifactStore, NullSolanaBackend +from composer.spec.source.source_env import build_basic_source_tools, build_source_tools +from composer.spec.system_model import SolidityIdentifier +from composer.ui.tool_display import async_tool_context +from composer.rustapp.frontend import GenericRustConsoleHandler +from composer.workflow.services import standard_connections, llm_factory +from composer.kb.knowledge_base import DefaultEmbedder + +from tests.conftest import needs_postgres, MockSentenceTransformer +from tests.test_autoprove_integration import _RAG_DB, _VECTOR_DBS, _MEMORIES_DDL, _db_url + +if TYPE_CHECKING: + from testcontainers.postgres import PostgresContainer + +pytestmark = [pytest.mark.expensive, needs_postgres, pytest.mark.asyncio] + +_SCENARIO = Path(__file__).parent.parent / "test_scenarios" / "solana_vault" + + +def _model_args() -> object: + return SimpleNamespace( + heavy_model="claude-opus-4-6", + lite_model="claude-sonnet-4-6", + tokens=128_000, + thinking_tokens=2048, + memory_tool=False, + interleaved_thinking=False, + ) + + +async def test_solana_vault_front_half(pg_container: "PostgresContainer", monkeypatch): + assert _SCENARIO.is_dir(), _SCENARIO + + # 1. Roles + databases the pipeline expects (matches services._DATABASE_CONFIGS). + admin_url = pg_container.get_connection_url(driver=None) + with psycopg.connect(admin_url, autocommit=True) as admin: + for cfg in services._DATABASE_CONFIGS.values(): + admin.execute(SQL("CREATE ROLE {} LOGIN PASSWORD {}").format( + Identifier(cfg["user"]), Literal(cfg["password"]))) + admin.execute(SQL("CREATE DATABASE {} OWNER {}").format( + Identifier(cfg["database"]), Identifier(cfg["user"]))) + admin.execute(SQL("CREATE DATABASE {}").format(Identifier(_RAG_DB))) + for db in _VECTOR_DBS: + with psycopg.connect(_db_url(pg_container, db), autocommit=True) as conn: + conn.execute("CREATE EXTENSION IF NOT EXISTS vector") + mem = services._DATABASE_CONFIGS["memory"] + mem_url = ( + f"postgresql://{mem['user']}:{mem['password']}" + f"@{pg_container.get_container_host_ip()}:{pg_container.get_exposed_port(5432)}/{mem['database']}" + ) + with psycopg.connect(mem_url, autocommit=True) as conn: + conn.execute(_MEMORIES_DDL) + + monkeypatch.setenv("CERTORA_AI_COMPOSER_PGHOST", pg_container.get_container_host_ip()) + monkeypatch.setenv("CERTORA_AI_COMPOSER_PGPORT", str(pg_container.get_exposed_port(5432))) + + model = MockSentenceTransformer() # deterministic embedder; nothing here needs real embeddings + args = _model_args() + + async with ( + standard_connections(provider="anthropic", embedder=DefaultEmbedder(model)) as conns, + async_tool_context(), + ): + content = await conns.uploader.get_document(_SCENARIO / "system.md") + assert content is not None + source = SourceCode( + content=content, + project_root=str(_SCENARIO), + contract_name=SolidityIdentifier("vault"), # the target program's identifier + relative_path="programs/vault/src/lib.rs", + forbidden_read=RUST_FORBIDDEN_READ, + ) + _tiered = get_provider_for(tiered=cast(Any, args)) + model_provider = ModelProvider( + heavy_model=_tiered.heavy, lite_model=_tiered.lite, checkpointer=conns.checkpointer, + ) + basic = build_basic_source_tools(root=str(_SCENARIO), forbidden_read=RUST_FORBIDDEN_READ) + full = build_source_tools(basic, model_provider, conns.indexed_store, ("solana_gate", "src"), recursion_limit=100) + env = PureServiceHost(models=model_provider, rag_tools=(), sort="existing").bind_source_tools(full) + + ctx = WorkflowContext.create( + services=conns.memory, + thread_id="solana_gate", store=conns.store, recursion_limit=100, + cache_namespace=None, memory_namespace=None, + ) + + import asyncio + backend = NullSolanaBackend(NullSolanaArtifactStore(str(_SCENARIO))) + run = PipelineRun(ctx=ctx, source=source, _handler_factory=GenericRustConsoleHandler(set()).make_handler, _semaphore=asyncio.Semaphore(4), env=env) + result = await run_pipeline(backend, run, ecosystem=SOLANA, interactive=False, threat_model=None, max_bug_rounds=1) + + # Pass: front half ran and extracted properties. + print(f"\nSolana gate: {result.n_components} invariant(s), {result.n_properties} properties") + for o in result.outcomes: + print(f"\n== {o.feat.display_name} ==") + for p in o.props: + print(f" [{p.sort}] {p.title}: {p.description}") + assert result.n_components > 0, "no invariants extracted" + assert result.n_properties > 0, "no properties extracted" diff --git a/uv.lock b/uv.lock index a099d25b..d8509aae 100644 --- a/uv.lock +++ b/uv.lock @@ -1,5 +1,5 @@ version = 1 -revision = 2 +revision = 3 requires-python = ">=3.12" resolution-markers = [ "python_full_version >= '3.14' and extra != 'extra-11-ai-composer-certora-cli' and extra != 'extra-11-ai-composer-certora-cli-beta' and extra == 'extra-11-ai-composer-certora-cli-beta-mirror' and extra != 'extra-11-ai-composer-cpu' and extra == 'extra-11-ai-composer-cuda'", @@ -124,9 +124,17 @@ s3 = [ ] [package.dev-dependencies] +apps = [ + { name = "crucible-app" }, + { name = "echoprover" }, + { name = "maturin-import-hook" }, +] ci = [ { name = "pyright" }, ] +dev = [ + { name = "maturin" }, +] ragbuild = [ { name = "beautifulsoup4" }, { name = "einops" }, @@ -186,7 +194,13 @@ requires-dist = [ provides-extras = ["ml", "s3", "certora-cli", "certora-cli-beta", "certora-cli-beta-mirror", "prover", "cpu", "cuda"] [package.metadata.requires-dev] +apps = [ + { name = "crucible-app", editable = "rust/crucible-app" }, + { name = "echoprover", editable = "rust/example-app" }, + { name = "maturin-import-hook", specifier = ">=0.3.0" }, +] ci = [{ name = "pyright" }] +dev = [{ name = "maturin", specifier = ">=1.14.1" }] ragbuild = [ { name = "beautifulsoup4", specifier = "==4.13.5" }, { name = "einops" }, @@ -924,7 +938,7 @@ name = "click" version = "8.3.1" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "colorama", marker = "sys_platform == 'win32' or (extra == 'extra-11-ai-composer-cpu' and extra == 'extra-11-ai-composer-cuda')" }, + { name = "colorama", marker = "sys_platform == 'win32' or (extra == 'extra-11-ai-composer-certora-cli' and extra == 'extra-11-ai-composer-certora-cli-beta') or (extra == 'extra-11-ai-composer-certora-cli' and extra == 'extra-11-ai-composer-certora-cli-beta-mirror') or (extra == 'extra-11-ai-composer-certora-cli-beta' and extra == 'extra-11-ai-composer-certora-cli-beta-mirror') or (extra == 'extra-11-ai-composer-cpu' and extra == 'extra-11-ai-composer-cuda')" }, ] sdist = { url = "https://files.pythonhosted.org/packages/3d/fa/656b739db8587d7b5dfa22e22ed02566950fbfbcdc20311993483657a5c0/click-8.3.1.tar.gz", hash = "sha256:12ff4785d337a1bb490bb7e9c2b1ee5da3112e94a8622f26a6c77f5d2fc6842a", size = 295065, upload-time = "2025-11-15T20:45:42.706Z" } wheels = [ @@ -962,6 +976,11 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/0c/00/3106b1854b45bd0474ced037dfe6b73b90fe68a68968cef47c23de3d43d2/confection-0.1.5-py3-none-any.whl", hash = "sha256:e29d3c3f8eac06b3f77eb9dfb4bf2fc6bcc9622a98ca00a698e3d019c6430b14", size = 35451, upload-time = "2024-05-31T16:16:59.075Z" }, ] +[[package]] +name = "crucible-app" +version = "0.1.0" +source = { editable = "rust/crucible-app" } + [[package]] name = "cryptography" version = "49.0.0" @@ -1259,6 +1278,11 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/b0/0d/9feae160378a3553fa9a339b0e9c1a048e147a4127210e286ef18b730f03/durationpy-0.10-py3-none-any.whl", hash = "sha256:3b41e1b601234296b4fb368338fdcd3e13e0b4fb5b67345948f4f2bf9868b286", size = 3922, upload-time = "2025-05-17T13:52:36.463Z" }, ] +[[package]] +name = "echoprover" +version = "0.1.0" +source = { editable = "rust/example-app" } + [[package]] name = "einops" version = "0.8.2" @@ -2270,6 +2294,39 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/70/bc/6f1c2f612465f5fa89b95bead1f44dcb607670fd42891d8fdcd5d039f4f4/markupsafe-3.0.3-cp314-cp314t-win_arm64.whl", hash = "sha256:32001d6a8fc98c8cb5c947787c5d08b0a50663d139f1305bac5885d98d9b40fa", size = 14146, upload-time = "2025-09-27T18:37:28.327Z" }, ] +[[package]] +name = "maturin" +version = "1.14.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/e7/b3/addd877f871fb1860d46d3a4f206ecb10b946c85846805e6367631926fd3/maturin-1.14.1.tar.gz", hash = "sha256:9d6577a62cd08e0ceba7a0db06fb098e0c9b1b3429bad747a4f3a18215a1b3df", size = 369637, upload-time = "2026-06-19T05:19:49.774Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/f4/f0/97c5a5bd9c71653a066c0976a484eaaae50b9369557838a4176b7b0bdaa5/maturin-1.14.1-py3-none-linux_armv6l.whl", hash = "sha256:522292398945442cdafa9daeb2271b2340fbde57027b818f923f88eab04174f8", size = 10207496, upload-time = "2026-06-19T05:19:09.321Z" }, + { url = "https://files.pythonhosted.org/packages/fe/83/294bca639b0e052f1e2f65199b3db258780c7d4e31408b934c9c974a1379/maturin-1.14.1-py3-none-macosx_10_12_x86_64.macosx_11_0_arm64.macosx_10_12_universal2.whl", hash = "sha256:ffe5ad71f21d1e6603c4dd75f7fee34adf5ed5ebcebb692886549888ebb329ed", size = 19680113, upload-time = "2026-06-19T05:19:13.43Z" }, + { url = "https://files.pythonhosted.org/packages/43/b6/79c881410a3b1c187f7eb3d407aecae646c6a4433d630d72200359015e83/maturin-1.14.1-py3-none-macosx_10_12_x86_64.whl", hash = "sha256:f3306078070c1508fd715b9116070cbcaff5959024272a9f1e6f5cb29768b86c", size = 10169205, upload-time = "2026-06-19T05:19:16.615Z" }, + { url = "https://files.pythonhosted.org/packages/93/9d/44b6f26dcb7f7a04c5501ac2dbb6ca1490150682baa525ca5860504f9eab/maturin-1.14.1-py3-none-manylinux_2_12_i686.manylinux2010_i686.musllinux_1_1_i686.whl", hash = "sha256:cd457cd88961156e26379e1155bd287cc0ec1c8b2f1582b0660fb31b87c8842d", size = 10188098, upload-time = "2026-06-19T05:19:19.736Z" }, + { url = "https://files.pythonhosted.org/packages/1a/bd/9c0d5d6983905ce2c9edaa073a7e89355a9cf7f396988e05d32f1c37785d/maturin-1.14.1-py3-none-manylinux_2_12_x86_64.manylinux2010_x86_64.musllinux_1_1_x86_64.whl", hash = "sha256:dfc54ae32e6fcb18302193ab9a30b0b25eefffba994ae13238974805533ef75e", size = 10627576, upload-time = "2026-06-19T05:19:22.713Z" }, + { url = "https://files.pythonhosted.org/packages/e5/33/b096412bd6a7cb399652b260666f901adf88a687181a6dbd6a3f89f0a94e/maturin-1.14.1-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.musllinux_1_1_aarch64.whl", hash = "sha256:a131d912b5267e640bc96d70f4914e10590aed64082ec9abacba7cea52004224", size = 10085181, upload-time = "2026-06-19T05:19:25.69Z" }, + { url = "https://files.pythonhosted.org/packages/56/8d/08c3bf469c38a23c9e6c877e338193001eb604d010fedc08341974e38528/maturin-1.14.1-py3-none-manylinux_2_17_armv7l.manylinux2014_armv7l.musllinux_1_1_armv7l.whl", hash = "sha256:be18fc568fb76884c0205456336892a75105ec398e6b667cd777c6268bd06d69", size = 10026363, upload-time = "2026-06-19T05:19:28.904Z" }, + { url = "https://files.pythonhosted.org/packages/3a/a4/c4d1a92839f8745ab4aab988a7db884a79d6d710bd3b286fcf9316dece1a/maturin-1.14.1-py3-none-manylinux_2_17_ppc64le.manylinux2014_ppc64le.musllinux_1_1_ppc64le.whl", hash = "sha256:994a0c8ba3ad8a92b3a9ee1b02645d200d610216b15cff5102b0fe65e8e08666", size = 13321347, upload-time = "2026-06-19T05:19:32.411Z" }, + { url = "https://files.pythonhosted.org/packages/b3/fa/170f04624d03fd07d2a8b1b67de83a127af93aef9eaa425839553347297b/maturin-1.14.1-py3-none-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:be80866363e605d137991b491a741a84cde9ae350183c4c85f49690ca9aaaa65", size = 10877609, upload-time = "2026-06-19T05:19:35.448Z" }, + { url = "https://files.pythonhosted.org/packages/61/ad/1ae2e1d0ded282bf2c55ac13f0811d87deb425e200ae64a15785675dede9/maturin-1.14.1-py3-none-manylinux_2_31_riscv64.musllinux_1_1_riscv64.whl", hash = "sha256:5282dffd4b539d2be245f4e5b1a5ab6bc1033b58f4a4872f5833f9d43c954aa4", size = 10417316, upload-time = "2026-06-19T05:19:38.28Z" }, + { url = "https://files.pythonhosted.org/packages/fb/27/bf677183920718da49cd7982d6a3ffc440aad8919329f571d189f81b7bdf/maturin-1.14.1-py3-none-win32.whl", hash = "sha256:1a04de0a20188f95c721b5702eed18140bdcccb28c386797093eca3f62f4d4e0", size = 8931293, upload-time = "2026-06-19T05:19:41.183Z" }, + { url = "https://files.pythonhosted.org/packages/63/4b/585adeb9167b08d3cdff0032a938b0e72655c92003df4f52c3f696a1bcc2/maturin-1.14.1-py3-none-win_amd64.whl", hash = "sha256:3c9f94640ecc4895e94abaf834a0684430032c865b2748a36c12461fd9252fdd", size = 10314067, upload-time = "2026-06-19T05:19:44.389Z" }, + { url = "https://files.pythonhosted.org/packages/51/d4/dac8c0720ae246be1700afb6fbdbbea20fe35b13f6570b2f70faa005df77/maturin-1.14.1-py3-none-win_arm64.whl", hash = "sha256:15cea8fcb3ba47dd636f50092bb34baea8b04ac777392f23e6bf8a9a61efb894", size = 9718943, upload-time = "2026-06-19T05:19:47.49Z" }, +] + +[[package]] +name = "maturin-import-hook" +version = "0.3.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "filelock" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/75/a1/3b007a4132265d5da19096d6835cd4c7e9908afb9156afb96d19ff190f39/maturin_import_hook-0.3.0.tar.gz", hash = "sha256:cdd51ce267663c437c4aea1bfd5f7ee5884b54a9d866ea924227d68d758bbba2", size = 30785, upload-time = "2025-06-08T15:43:57.801Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/9f/55/0c17a1fccec90dffbb1d904d7ddb7971e4e95f994c6264637e8fe66a5b86/maturin_import_hook-0.3.0-py3-none-any.whl", hash = "sha256:f5d20b4bc3056d84170dc08fe655160421b28d2b5128b48f55312315cdbb2cfe", size = 30848, upload-time = "2025-06-08T15:43:56.298Z" }, +] + [[package]] name = "mcp" version = "1.28.0" @@ -5028,7 +5085,7 @@ name = "tqdm" version = "4.67.3" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "colorama", marker = "sys_platform == 'win32' or (extra == 'extra-11-ai-composer-cpu' and extra == 'extra-11-ai-composer-cuda')" }, + { name = "colorama", marker = "sys_platform == 'win32' or (extra == 'extra-11-ai-composer-certora-cli' and extra == 'extra-11-ai-composer-certora-cli-beta') or (extra == 'extra-11-ai-composer-certora-cli' and extra == 'extra-11-ai-composer-certora-cli-beta-mirror') or (extra == 'extra-11-ai-composer-certora-cli-beta' and extra == 'extra-11-ai-composer-certora-cli-beta-mirror') or (extra == 'extra-11-ai-composer-cpu' and extra == 'extra-11-ai-composer-cuda')" }, ] sdist = { url = "https://files.pythonhosted.org/packages/09/a9/6ba95a270c6f1fbcd8dac228323f2777d886cb206987444e4bce66338dd4/tqdm-4.67.3.tar.gz", hash = "sha256:7d825f03f89244ef73f1d4ce193cb1774a8179fd96f31d7e1dcde62092b960bb", size = 169598, upload-time = "2026-02-03T17:35:53.048Z" } wheels = [