Get token-cost / step-reduction results from recalled guidelines on a real benchmark (e.g. AppWorld)

[vinodmut]

Problem

We claim Evolve's recalled guidelines make agent sessions cheaper and shorter, but we have no defensible numbers. #6 already calls for a benchmark on AppWorld + a ReAct agent measuring step counts and pass rate before/after tip generation; this issue is the broader version of that ask: produce results that show how much recall reduces token cost, wall-clock, and steps on a real benchmark, on at least one of the platforms Evolve supports (Claude Code, Codex, Bob, claw-code, …).

AppWorld is the obvious starting candidate (per #6), but the deliverable is results, not a specific benchmark choice. If a different public benchmark fits better — SWE-bench-verified slices, ToolBench, MultiHopRAG, BrowseComp, something else — pick that and justify briefly.

What "results" means here

For each (benchmark task or task family, platform) pair:

• Token cost — total, input vs. output, cache reads vs. creates.
• Wall-clock per session.
• Steps / turns per session.
• Task success rate.
• The specific guidelines recalled (so a reader can see what knowledge drove the savings).
• N trials per condition (with vs. without recall) so the numbers are defensible, not anecdotal.

Headline output: a table per platform, plus a short writeup explaining the gap on a representative task — including the tool calls each condition made, so the why is visible alongside the what.

Scope

• At least one public benchmark, multiple distinct task families inside it.
• At least one Evolve-supported platform end-to-end. A second platform is a strong nice-to-have but not required to close.
• Whatever harness exists (e.g. tests/e2e/experiment_token_savings.py) is a starting point, not the deliverable. The deliverable is the numbers and the writeup.

Out of scope

• Building a long-lived measurement framework (separate issue if needed).
• Wiring this into CI as a regression check (separate issue).
• Multi-platform coverage beyond a second platform.

Related

• Benchmark Tip Generation #6 — Benchmark Tip Generation (AppWorld + ReAct angle — the most direct prior)
• Explore Cuga demo apps for a memory-focused Evolve demo #142 — Cuga demo apps for a memory-focused Evolve demo
• RFC(memory): transform Evolve to .md-first procedural memory system (moat = trajectory-mined guidelines) #257 — RFC: .md-first procedural memory (moat thesis this would help validate)

Acceptance

• A writeup (issue comment, doc, or PR) with the comparison tables described above, on at least one benchmark and at least one platform.
• N ≥ 5 trials per condition.
• Concrete examples of recalled guidelines and the tool-call gap they removed.

[vinodmut]

Pilot data point: hostile-sandbox EXIF demo (Claude Code)

To gut-check the size of the effect before building anything against AppWorld, I ran a one-off measurement using the existing evolve-lite Claude Code plugin in the claude-sandbox Docker image. Not the deliverable this issue is asking for, but useful as a sanity check that the savings are large enough to be worth chasing.

Setup

• Seed utterance: where was the photo @sample.jpg taken. use exif metadata
• Measure utterance: what focal length was used to take the photo @sample.jpg. use exif metadata
• Sandbox: claude-sandbox — no exiftool, no ImageMagick, no PIL preinstalled. Hostile env, so the seed utterance has dead-ends to learn from.
• Seeding mode: shared — utterance 1 ran once to produce a fixed set of recallable guidelines; utterance 2 was measured 5 times against the same seeded workspace.
• Baseline: 5 fresh demo/workspace copies with no .evolve/. Plugin still loaded; recall just had nothing to inject.
• Harness: tests/e2e/experiment_token_savings.py (added on the procedural branch).

Results (n=5 per condition)

metric	without_guidelines (n=5)	with_guidelines (n=5)	savings
total tokens	223,427 (range 191,872–256,791)	146,040 (range 139,084–169,837)	+34.6%
output	1,672 (range 1,059–2,085)	739 (range 643–837)	+55.8%
cache_read	217,244	141,146	+35.0%
duration_ms	157,949	87,529	+44.6%
num_turns	8.0 (range 7–9)	6.2 (range 6–7)	−22.5%

The with-guidelines side is tightly clustered (4 of 5 runs at exactly 6 turns; one at 7). The baseline is more variable (7–9 turns).

What the model actually did

With guidelines (representative run, 6 turns):

1. Read .evolve/entities/guideline/in-this-workspace-environment-exiftool-and-imagemagick.md
2. Read .evolve/entities/guideline/use-workspace-scripts-extract-exif-gps-py-to-read-exif-and.md
3. Bash pip install Pillow -q && python3 -c "from PIL import Image; ... print focal length" ← works on first try
4. Skill evolve-lite:learn

Without guidelines (representative run, 8 turns):

1. Bash exiftool ... | grep focal ← not installed
2. Bash exiftool ... || identify -verbose ... | grep focal ← imagemagick also missing
3. Bash python3 -c "from PIL import Image..." ← PIL not preinstalled
4. Bash raw byte poke: python3 -c "import struct...; data.find(b'Exif')"
5. Bash pip install Pillow -q && python3 -c "..." ← finally works
6. Skill evolve-lite:learn

The savings are real turn savings — about 2 trial-and-error rounds eliminated because the guidelines told the model up front that exiftool/ImageMagick aren't available and to install Pillow.

Recalled guidelines

Three guideline files were learnt from utterance 1 and recalled into every utterance-2 run. The first two were the ones the model Read in the sample run.

• in-this-workspace-environment-exiftool-and-imagemagick.md — "exiftool and ImageMagick are not installed; prefer Pillow."
• use-workspace-scripts-extract-exif-gps-py-to-read-exif-and.md — "use the workspace script for EXIF/GPS; pip install Pillow if needed."
• when-system-exif-tools-exiftool-imagemagick-identify-are.md — "install Pillow with pip install Pillow -q and use PIL.Image._getexif(); do not regex/binary-parse EXIF."

Caveats

• n=5, single task, single platform. Means are directional, not statistical.
• Sandbox is artificially hostile — the savings here represent the cost of rediscovering a workaround that the seed utterance already paid for. A realistic env would show smaller absolute savings.
• The same 3 guidelines were recalled in every run, so this measures the value of applying a fixed guideline set, not the value of generating them. An earlier per-run-seed batch (n=3) showed the seed step itself adds variance.

Artifacts

On the procedural branch (commit 1c392d9):

• Script: tests/e2e/experiment_token_savings.py
• Results: tests/e2e/experiment_results/token_savings_20260529T140605Z/ — summary.md, report.md, raw.json, plus preserved per-run workspaces and full session transcripts.

What this doesn't answer

This is a single hand-picked task in a hand-picked hostile environment on one platform. The issue's deliverable still needs:

• A real public benchmark with multiple task families (AppWorld or alternative).
• Task success rate, not just cost (every run here succeeded; we didn't measure whether recall changes pass rate).
• At least one second platform.
• Larger N within a benchmark suite.

Posting this so the order of magnitude is on record before that work starts.

[vinodmut]

Second pilot: procedural memory (synthesized skills) — same EXIF setup

Following up on the first comment. The earlier data point measured token savings from declarative recall — learn writing free-text guidelines that the next agent reads. This second experiment measures procedural memory — synthesize-skill (a new skill, separate PR) reading a saved trajectory and emitting a real Claude Code skill (SKILL.md + supporting scripts) the next agent can invoke. The earlier guideline path is "what to do" in prose; this one is "the call you can make."

Same hostile sandbox (claude-sandbox, no exiftool/PIL/ImageMagick), same demo workspace, same --output-format json accounting. Three conditions per measure run, fresh demo/workspace each time:

• no_recall — nothing.
• guidelines — seed workspace's .evolve/entities/.
• skill — seed workspace's .claude/skills/ and .evolve/skills/.

Run A (n=5, single-utterance seed: gps, measured on gps/focal_length/lens)

utterance	no_recall	guidelines	skill
focal_length total tokens	235,998	191,014	142,422 (−40%)
focal_length num_turns	8 (6–13)	8 (5–9)	6 (4–7)
gps (= seed task) total	174,307	188,126	176,604 (≈tie)
lens total	145,855	175,440	154,564 (skill loses)

Skill invocation rate (i.e. did the model actually call Skill: <synthesized-name>?): focal_length 3/5, gps 4/5, lens 2/5. The lens regression tracked the invocation rate — when the synthesized skill's name was GPS-flavored (4/5 trials produced names like extract-gps-from-jpeg), the model didn't recognize its applicability to a lens question.

Run B (n=5, two-utterance seed: gps + focal_length, measured on lens only)

metric	no_recall	guidelines	skill
total	212,129	209,734	157,362 (−26%)
output	1,796	1,803	958 (−47%)
duration_ms	219,694	195,647	151,409 (−31%)
num_turns	8	8	6

Skill invocation rate on lens: 4/5 (vs 2/5 in run A). The skill's per-invocation behavior is roughly the same — what changed is the trigger description the synthesizer wrote. With two seed utterances, 3/5 synthesized skills are named exactly extract-exif-metadata (covers any EXIF field) instead of run A's extract-gps-* flavoring.

Cost picture (per trial, NOT included in the tables above)

	run A	run B
seed cost	$0.37	$0.73 (≈2×)
synth cost	$0.42	$0.47 (≈same)
setup total	$0.79	$1.20
amortization break-even	varies (was a wash on lens)	~2–3 future EXIF queries

Procedural vs declarative head-to-head

Same lens measure, run B (the cleanest comparison since the seed produced both):

	guidelines	skill
total tokens	209,734	157,362 (−25% vs guidelines)
num_turns	8	6

In run B, on every metric, every utterance, the procedural skill beats both the declarative guidelines and the no-memory baseline. Run A had one regression (lens) that two-utterance seeding fixed. Within the conditions tested, whenever a synthesized skill exists for the task, it strictly dominates the guideline path.

Headline takeaway

Procedural memory (a synthesized skill) is directly invokable rather than read-and-paraphrased, which translates to fewer turns, less output, less cache re-reading, and a per-query token reduction of 25–40% on tasks the skill matches. The dominant variable is trigger generality — the model only saves tokens if it actually recognizes the skill's applicability. Two-utterance seeding doubled the skill invocation rate on a never-seen sibling task (2/5 → 4/5).

Same caveats as before:

• n=5, single hostile environment, single domain (EXIF). Means are directional, not statistical.
• Hostile sandbox amplifies absolute savings — the no_recall baseline burns cycles on missing tools. A real environment with exiftool would show much smaller gaps.
• Skill content varies trial to trial. Quality is a confounder.
• Cross-condition contamination: post-synthesis learn produced skill-pointer guidelines that leaked into the guidelines condition (the model wasted turns reading them). A snapshot-before-synthesize fix is a follow-up.
• Synthesis cost is real ($0.42–$0.47 per call). Promotion to a Stop hook would shift this cost into the agent's hot path; right now it's manual.

Artifacts

On procedural branch:

• Skill: platform-integrations/claude/plugins/evolve-lite/skills/evolve-lite/synthesize-skill/ — SKILL.md + scripts/synthesize.py
• Driver: experiments/skill_from_trajectory.py (supports --seed-utterances for the two-utterance variant)
• Run A: experiments/results/skill_from_trajectory_20260601T070227Z/{report,summary}.md + raw.json + synthesized_skills/
• Run B: experiments/results/skill_from_trajectory_20260601T115630Z/{report,summary}.md + raw.json + synthesized_skills/
• Side-by-side: experiments/results/skill_from_trajectory_comparison.md

A separate PR will land the skill + the experiment driver. Results stay on procedural.

What this still doesn't answer

The acceptance criteria on this issue still hold:

• Real public benchmark (AppWorld or similar) with multiple task families.
• Task success rate, not just cost — every run here succeeded; we didn't measure whether procedural memory changes pass rate.
• Second platform (Codex / Bob / claw-code).
• Larger N within a benchmark suite.

Posting both pilot data points so the shape of the effect is on record before the benchmark work starts.