Prove your agent evals catch costly business failures.
Test outcomes, business invariants, tool use, and multi-agent coordination before production does.
Quick start · Harnesses · Golden set · Multi-agent · How it works · Run a pilot
Your agent tests can all pass while an invoice is paid twice, access is granted without authority, or a required tool action silently disappears. Mendmark tests the tests themselves: it plants one controlled fault, runs the same evaluators again, and translates every survivor into a business-readable risk and a precise engineering finding.
Killed means the eval noticed the planted fault. Survived means the damaged case still passed. A critical survivor can fail the release gate.
Mendmark requires Python 3.10 or newer.
See the outcome-first value with no service credentials, model calls, or fixture setup:
pip install mendmark-evals
mendmark demoThe command compares a conventional state-only evaluator with complete outcome
assurance across eight customer, finance, identity, operations, and fulfillment
workflows. It writes a reviewable JSON suite and both privacy-safe reports to
mendmark-enterprise-demo/.
Audit a reviewed outcome suite directly—without an evaluator framework or subprocess:
mendmark audit-outcomes mendmark-enterprise-demo/suite.jsonFor an existing DeepEval suite:
pip install 'mendmark-evals[deepeval]'
git clone https://github.com/danielgaskins/mendmark.git
cd mendmark
mendmark audit examples/order_agent_suite.py \
--output mendmark-report.json \
--write-baselineMendmark agent-eval audit
Business assurance: PROTECTED
Configured business risks were detected by the eval suite.
Cases: 1
Mutations: 19 Killed: 19 Survived: 0 Errors: 0
Mutation kill rate: 100.0%
New tools: lookup_order, refund_order
Gate: PASS
Run the same command in CI without --write-baseline. Mendmark compares the
current tool schemas and mutation results with the last accepted baseline.
For a native parallel multi-agent audit:
mendmark audit-json examples/multi_agent_suite.json \
--evaluator-command "python3 examples/multi_agent_evaluator.py"Mendmark has dependency-light adapters for LangChain/LangGraph, CrewAI, and the OpenAI Agents SDK. In an existing agent repository:
python -m pip install 'mendmark-evals==0.7.1'
mendmark equip --framework auto --agent autoThe command detects bounded dependency files and creates a reviewed capture
guide, offline evaluator, and inactive CI template under .mendmark/. It does
not edit application code, upload a trace, overwrite existing work, enable CI,
or accept a baseline.
Want Codex or Claude Code to perform the integration? Install its native,
repo-scoped skill (use all to install both):
mendmark equip --framework auto --agent codex # invoke with $mendmark
mendmark equip --framework auto --agent claude-code # invoke with /mendmarkFor any other repository-capable agent, print a portable self-equip prompt:
mendmark equip --agent generic --print-agent-promptSee the agent harness integration guide for the direct Python APIs, explicit trace-approval boundary, framework compatibility, and multi-agent guidance.
▶ Watch the narrated weak-eval demonstration (original v1 fault inventory)
A refund-agent test checks only the final sentence. Mendmark finds that 15 of 19 faults escape—including a wrong refund amount and a duplicated refund. A complete evaluator checks the calls, arguments, results, and response, killing all 19.
An outcome contract records the durable business state that must exist, the invariants that must never be violated, and optional cost and latency limits. Mendmark then tests whether the evaluator detects missing or corrupted state, broken safeguards, and exceeded operating limits. Reports lead with plain-language headlines such as “An invoice can be paid incorrectly,” followed by stable case and operator identifiers engineers can act on.
Use --mutation-profile outcome-first when route variation is harmless and the
business result is the release criterion. Use the default full profile when the
route itself can change authorization, side effects, recovery, cost, or
coordination safety. See Outcome assurance for the
contract model and decision rule.
The Enterprise Outcome Golden Set targets the business
surface directly: eight common workflows, 16 system boundaries, 16 reviewed
invariants, and 64 high-importance mutations. It covers support escalation,
invoice approval, onboarding, refunds, offboarding, vendor bank changes,
incident remediation, and shipment exceptions. Its state-only profile detects
32/64; the complete outcome-contract profile detects 64/64. Run it instantly
with mendmark demo.
The Mendmark Agent Eval Golden Set is the canonical, versioned benchmark for the mutation engine.
| 24 reviewable cases |
13 tool contracts |
39 tool calls |
263 pinned mutations |
10 domains |
| Evaluator profile | Killed | Survived | Kill rate |
|---|---|---|---|
| Complete trace and outcome | 263 | 0 | 100.000% |
| Trace only | 215 | 48 | 81.749% |
| Response only | 87 | 176 | 33.080% |
The response-only profile leaves 162 critical tool-behavior mutations undetected. The complete profile catches every mutation in the golden set.
python benchmarks/benchmark_golden_set.pyReview the manifest, case suite, evaluator profiles, methodology, and reference performance directly. The benchmark is deterministic, offline, and makes no model calls.
Native coordination behavior has its own reviewable Multi-Agent Golden Set v2: 6 workflows, 17 agent declarations, 41 causal events, 30 operators, and 294/294 mutations killed by the complete reference evaluator.
python benchmarks/benchmark_multi_agent_golden_set_v2.pyFor contrast, the v2 output-only evaluator detects just 23/294 and leaves 271 survivors, 181 critical. Mendmark groups those blind spots by category and pinpoints the affected agent, event, and tool using privacy-safe identifiers. See the multi-agent guide for both commands.
| 🔎 Expose evaluator blind spots Wrong arguments, corrupted results, reordered calls, duplicate side effects, false recovery, and damaged responses. |
🧰 Fit the existing stack DeepEval, Rubric, or any local evaluator through a validated JSON subprocess protocol. |
| 🚦 Gate tool rollouts Per-tool coverage, schema-change detection, accepted baselines, mutation budgets, JUnit, and SARIF. |
🔐 Keep case content local Reports omit prompts, answers, tool arguments, and tool outputs; artifacts can be signed with Cosign. |
Single-agent suites use a simple ordered tool trace. Multi-agent suites add a
causal event graph with agent identities, delegation targets, explicit tool
permissions, returned results, shared-state events, and dependencies. Existing
1.0 suites remain unchanged; native graphs use the validated 2.0 JSON
contract.
Mendmark mutates both layers. It can break an individual tool call, route work to the wrong specialist, omit handoff context, drop or misattribute a result, violate an agent's tool permissions, remove a causal dependency, or insert a delegation loop. Independent parallel branches are not forced into an arbitrary wall-clock order.
The included three-agent reference suite has 9 events across parallel billing and risk branches. Its complete evaluator kills all 64 currently applicable mutations. The broader v2 golden set covers six topologies and kills 294/294. See the multi-agent guide and reviewable JSON suite.
The repository also includes a deliberately weak evaluator. It checks whether the final sentence is correct and ignores the tool trace.
mendmark audit examples/order_agent_weak_suite.py \
--output /tmp/mendmark-weak-report.jsonThe original refund case passes. Mendmark then changes the refund amount, removes required calls, and duplicates the side effect. Many of those faults survive because the final sentence never changed. The command exits with a failed gate and names each blind spot.
Run the complete order_agent_suite.py next. Its tool-trace evaluator checks
the ordered calls, arguments, and results, so the same planted faults are
caught. This before-and-after pair is the shortest demonstration of what
Mendmark measures.
A suite is a trusted local Python file. It exports three things:
from deepeval.metrics import BaseMetric
from deepeval.test_case import LLMTestCase, ToolCall
TOOLS = [
{
"name": "refund_order",
"input_schema": {
"type": "object",
"properties": {
"order_id": {"type": "string"},
"amount": {"type": "number"},
},
"required": ["order_id", "amount"],
},
"side_effecting": True,
}
]
MENDMARK_POLICY = {
"minimum_kill_rate": 0.9,
"fail_on_critical_survivor": True,
"fail_on_untested_tools": True,
"fail_on_tool_contract_issues": True,
"fail_on_regression": True,
}
def get_metrics():
# Return your DeepEval metrics here. The complete example includes a
# deterministic tool-trace metric that runs without an API key.
return [MyToolMetric()]
def get_cases():
refund = ToolCall(
name="refund_order",
input_parameters={"order_id": "104", "amount": 29.99},
output={"status": "accepted"},
)
return [
LLMTestCase(
name="refund-order",
input="Refund order 104 in full.",
actual_output="The refund was accepted.",
expected_output="The refund was accepted.",
tools_called=[refund],
expected_tools=[refund],
)
]get_metrics() must return fresh metric instances on every call. Metric names
must be unique. Mendmark reruns those metrics against the original case and each
mutated copy.
See the complete example and the mutation audit guide.
Teams can export cases and traces as JSON and connect any language or eval framework through a local stdin/stdout command:
mendmark audit-json examples/order_agent_suite.json \
--evaluator-command "python3 examples/json_evaluator.py" \
--junit /tmp/mendmark.xml \
--sarif /tmp/mendmark.sarifThe command runs locally and receives the original and mutated cases in one batch. Mendmark strictly validates its metric results. See the JSON adapter and protocol.
Custom domain faults can be loaded from a suite, trusted Python file, installed entry point, or module attribute. See custom mutation plugins.
The repository also includes a tested Rubric integration that runs Rubric metrics through the same JSON protocol.
Reports automatically record the Mendmark version, adapter, canonical policy
digest, and supported GitHub/GitLab CI metadata. Explicit versions can be added
with --source-commit, --source-ref, --suite-version, and
--policy-version. Use --maximum-mutants to stop before evaluator work when a
suite exceeds its approved cost ceiling.
Mendmark delegates signatures to Sigstore Cosign:
mendmark sign mendmark-report.json --bundle mendmark-report.sigstore.json
mendmark verify-signature mendmark-report.json \
--bundle mendmark-report.sigstore.json \
--certificate-identity "EXPECTED_OIDC_IDENTITY" \
--certificate-oidc-issuer "EXPECTED_OIDC_ISSUER"See artifact signing, the compatibility policy, the engine benchmark, the user assurance contracts, and the packaged report and baseline JSON Schemas.
Mendmark hashes each declared tool's name, schema, description, and side-effect flag. The baseline lets it answer four concrete questions in a pull request:
- Was a tool added?
- Did its contract change?
- Does at least one eval exercise it?
- Do those evals catch faults in its calls and results?
Mendmark also checks required arguments and basic JSON Schema types in the actual and expected traces. Reports identify the case, tool, field, and problem without storing the argument value.
This makes a tool launch visible before it reaches production. It does not prove the tool is safe. It shows whether the team's current evals can recognize the failures Mendmark introduced.
The suite file is executable Python. Only run suites from code you trust.
Mendmark's JSON report stores case IDs, operator names, severities, metric names, statuses, and tool schema digests. It does not store prompts, expected answers, tool arguments, or tool outputs. Teams can run the engine inside their own CI boundary and publish only the report.
See SECURITY.md for private vulnerability reporting, the threat model and deployment checklist for the complete trusted-code boundary, and SUPPORT.md for version and support expectations.
Teams with a real tool-using agent can follow the pilot guide and open a privacy-safe Mendmark pilot request. Do not include prompts, traces, payloads, credentials, or customer data in a public issue.
Completed pilots use the machine-validated design-partner evidence contract to record time-to-value, mutation realism, equivalent faults, discovered/remediated blind spots, runtime, cost, and CI retention without storing customer content. Until that external utility gate passes, golden-set results are engine evidence—not a claim of universal agent safety or customer validation.
Mendmark started as a benchmark for coding agents that repair ML pipelines. That
work remains available through mendmark prepare, mendmark grade, and the
MendmarkIntegrityMetric DeepEval adapter. It checks failures such as label
leakage, train-serve skew, invalid metric aggregation, and broken
reproducibility.
The ML pack is now one specialized use of the broader idea. An evaluator should be tested against known bad outcomes before its score is trusted.
See the DeepEval guide and the ML evaluation card.
Version 0.7 is a local, open-source engine. It does not yet provide a hosted dashboard, team accounts, remote trace ingestion, or a secrets service. The planned control plane is described in the product design.
Release history is maintained in CHANGELOG.md.