Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions composer/templates/cvl_additions.j2
Original file line number Diff line number Diff line change
Expand Up @@ -25,4 +25,16 @@ where necessary.
<advice>
Reverting behavior is reasoned about by using the `@withrevert` annotation and examining the value of the `lastReverted` flag.
</advice>
<advice>
Multi-call properties can compare executions from a common starting state using storage snapshots: `storage init = lastStorage;`
snapshots the current blockchain state (and non-persistent ghosts), and `f(e, args) at init;` runs `f` starting from that snapshot
instead of the current state. The standard additivity idiom: run `g(x); g(y);` sequentially from a snapshot, then run `g(x + y) at init;`
from the *same* snapshot, and compare the resulting end states (e.g., final balances or shares).
</advice>
<advice>
For implication-shaped assertions (`assert A => B;`), consider a companion witness rule containing `satisfy A;`. A `satisfy`
statement checks that at least one execution reaches it with the condition true — a single witness execution suffices. This
guards against vacuously-true implications whose antecedent `A` can never actually hold. List the witness rule in the
`property_rules` mapping under the same property as the implication rule it guards, so it is retained in the final report.
</advice>
</cvl_advice>
12 changes: 10 additions & 2 deletions composer/templates/cvl_guidelines.j2
Original file line number Diff line number Diff line change
Expand Up @@ -35,11 +35,19 @@
19. CVL automatically promotes operands of comparison/arithmetic operations to `mathint`. Explicit `to_mathint` casts are
rarely need.
20. `persistent` ghosts should be used sparingly, and should *never* be used on ghosts which are intended to model/mirror
contract state.
contract state. However, ghosts tracking information never stored in contract state (e.g., call counters, cumulative transfer
amounts) are a legitimate use of `persistent`, and of expression summaries on calls to external dependencies.
21. When referring to meaningful numerical constants in a specification (e.g., max fee percentage, asset expiry period, etc.) prefer using the `definition` feature of CVL
to provide meaningful names to the constants.
22. Instead of `forall $type i` in a invariant, invariant parameters should be used instead. That is, instead of `invariant foo() forall uint i.logical_predicate(i)` simply write
`invariant foo(uint i) logical_predicate(i)`. The parameters of an invariant are implicitly universally quantified.
23. Direct storage access should be used instead of mirroring contract state in ghosts via hooks. The only reason to use hooks and ghosts to reason about storage state is
if the ghost state is used for anything other than simply reading the values.
if the ghost state is used for anything other than simply reading the values. Hooks and ghosts are also appropriate when the information of interest is not stored in
contract state at all (e.g., counting how many times a function is called, or summing the amounts transferred across a sequence of calls).
24. A bare `require` that constrains contract state inside a rule or `preserved` block is a spec smell: the assumption may not actually hold in reachable states, silently
weakening the proof. The default discipline is to state the assumption as an `invariant`, prove it, and assume it via `requireInvariant`. Bare requires over contract
state are acceptable only when accompanied by a comment referencing a justification or a documented trust assumption. Exception: capacity/overflow-avoidance bounds on
supplies, balances, or timestamps (e.g. `require totalSupply() <= 2^128`) exist to keep the solver tractable, not to model contract behavior — they are a recognized
documented-trust-assumption class; a comment naming the bound's purpose suffices, and no invariant attempt is expected because such bounds are typically not preserved
by the contract. In `preserved` blocks the stricter standard applies — `requireInvariant` or verifiable outside evidence; a comment alone is insufficient.
Comment on lines +47 to +52

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we discussed this on zoom, but we already have guidance for both the judge and author to not use require as a "make the rule pass" magic. Repeating it a third time will not fix the problem imo. The instructions you provide here are almost verbatim in the author prompt. It would be useful if we could find the run where this happened and see what reasoning/thinking tricks the agent used to talk itself out of the rules it already knew about, rather than just repeating the rule one more time and hoping that one makes it stick.

</cvl_guidelines>
16 changes: 15 additions & 1 deletion composer/templates/property_analysis_prompt.j2
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,19 @@ Security properties fall into one of three categories:
Bad example: "The protocol could be hacked" (overly broad)
Bad example: "The underlying EVM consensus layer could be compromised allowing attackers to set arbitrary storage states" (implausible)

To achieve broad coverage, sweep the component through the following six lenses while brainstorming:
1. Unit behavior: the effect of a single function in isolation, including its *complete* revert conditions (when it must revert, and when it must NOT revert).
2. Valid state: representational invariants that hold in every reachable state.
3. Variable transition: how an individual variable may change — who may change it, when, and how (e.g., monotonicity: a counter only increases; a fee only changes via an admin setter).
4. State transition: legality of transitions between modes/phases of the contract lifecycle (e.g., paused/unpaused, uninitialized/initialized, auction open/closed) — which transitions are allowed and who may trigger them.
5. Multi-call / high-level: properties spanning a *sequence* of calls, e.g., round-trips (deposit then withdraw yields no more than deposited), no-free-profit, additivity of split vs. batched operations (two calls with x and y vs. one call with x + y).
6. Risk / attack: adversarial scenarios detrimental to the protocol.

Every property you keep must still be reported under one of the three categories above (lenses 2 and 6 map directly to
categories 1 and 3; the rest usually land in category 2). The lenses are a brainstorming discipline, NOT a quota: it is
expected and normal for several lenses to yield nothing for a given component, and the standing rule against padding
with low-value properties applies in full force.

{{ backend_guidance }}

Note: Safety properties can (and should) include properties describing the intended, normal behavior of the smart contract. In other words, the properties
Expand All @@ -95,7 +108,8 @@ Review your rough draft. For each property/invariant in your rough draft, make s
attention for properties/invariants explicitly described as being uninteresting or impossible to verify in the downstream verification tool.

## Step 4
Output the results of your analysis using the result tool.
Output the results of your analysis using the result tool. In the reasoning you record, note which of the six coverage
lenses you swept and which of them yielded nothing for this component.
</task>

<guidance>
Expand Down
3 changes: 2 additions & 1 deletion composer/templates/property_generation_prompt.j2
Original file line number Diff line number Diff line change
Expand Up @@ -119,7 +119,8 @@ The `result` tool *also* requires a `property_rules` mapping: for every property
its unique snake_case title from the batch listing above), list the name(s) of the rule(s)/invariant(s) in your
spec that verify it. Every non-skipped property must appear in this mapping with at least one rule; skipped
properties must NOT appear. The result tool will be rejected if this mapping is incomplete or references a
skipped or unknown property title.
skipped or unknown property title. Supporting invariants and companion witness rules must also be listed under
the property they support: rules not referenced by any property are silently dropped from the final report.

To clarify: for your result to be accepted, all verification results from the prover must either be
VERIFIED *or* that rule must be explicitly marked as "expected to fail" using the `expect_rule_failure` tool.
Expand Down
94 changes: 94 additions & 0 deletions tests/test_template_render.py

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove this test. does nothing

Original file line number Diff line number Diff line change
@@ -0,0 +1,94 @@
"""Render smoke tests for prompt templates.

These templates are plain-jinja prompt fragments consumed by the property analysis /
generation / judge agents. The tests render them with minimal stand-in objects (jinja
only does attribute lookups, so `SimpleNamespace` suffices) and assert the load-bearing
content is present — a missing template variable or a broken include fails fast here
instead of mid-agent-run.
"""
from types import SimpleNamespace

from composer.templates.loader import load_jinja_template


def test_cvl_guidelines_render():
out = load_jinja_template("cvl_guidelines.j2")
# Guideline 24: requireInvariant discipline over bare state requires.
assert "requireInvariant" in out
assert "spec smell" in out
# Guideline 24 carve-out: solver-capacity bounds need no invariant attempt.
assert "keep the solver tractable" in out
# Guideline 24 defers preserved blocks to the stricter (judge Criteria 4) standard.
assert "stricter standard" in out
# Guideline 20 carve-out: persistent counter ghosts are legitimate (judge-visible
# counterpart of the ghost-counter advice in cvl_additions.j2).
assert "never stored in contract state" in out
# Guideline 23 carve-out: hooks/ghosts for information never stored in contract state.
assert "not stored in" in out
assert out.strip().endswith("</cvl_guidelines>")


def test_cvl_additions_render():
out = load_jinja_template("cvl_additions.j2")
# Storage snapshot / additivity idiom.
assert "lastStorage" in out
assert "at init" in out
# satisfy-witness companion rules, mapped under their parent property.
assert "satisfy" in out
assert "property_rules" in out
# Ghost counters via expression summaries.
assert "countDeposit() expect void" in out
assert out.strip().endswith("</cvl_advice>")


def _fake_component_context() -> SimpleNamespace:
"""Minimal stand-in for ContractComponentInstance as accessed by the template."""
contract = SimpleNamespace(name="Vault", solidity_identifier="Vault")
component = SimpleNamespace(
name="Deposits",
description="Handles user deposits",
requirements=["Users receive shares proportional to deposits"],
interactions=[],
)
app = SimpleNamespace(application_type="an ERC4626 vault")
return SimpleNamespace(component=component, contract=contract, app=app,
ommer_contracts=[])


def test_property_analysis_prompt_render():
out = load_jinja_template(
"property_analysis_prompt.j2",
context=_fake_component_context(),
backend_guidance="BACKEND_GUIDANCE_SENTINEL",
# A valid Sort value (see composer/spec/service_host.py) exercising the
# non-greenfield template branch.
sort="existing",
prior_properties=[],
)
# The context and backend guidance are threaded through.
assert "Deposits" in out
assert "BACKEND_GUIDANCE_SENTINEL" in out
# 6-lens coverage checklist, framed as brainstorming discipline, not a quota.
assert "Unit behavior" in out
assert "Variable transition" in out
assert "Multi-call / high-level" in out
assert "NOT a quota" in out
# Task steps ask for a record of swept lenses.
assert "which of the six coverage" in out


def test_property_analysis_prompt_render_with_prior_rounds():
prior = [SimpleNamespace(
items=[SimpleNamespace(sort="invariant", title="solvency",
description="assets cover shares")],
reasoning="looked at deposit accounting",
)]
out = load_jinja_template(
"property_analysis_prompt.j2",
context=_fake_component_context(),
backend_guidance="",
sort="existing",
prior_properties=prior,
)
assert "solvency" in out
assert "looked at deposit accounting" in out
Loading