Desktop 1.4.3 · DeepAgent Core V4.0.4_r8
+Desktop 1.4.4 · DeepAgent Core V4.0.5
--- @@ -83,7 +83,9 @@ Composable domain packs add language, framework, platform, hardware, business, a ### Specialist agents and Expert Panel -DeepAgent can partition independent work across bounded, isolated workers. Write-capable subagents receive dedicated worktrees, return compact summaries and artifact references, and leave their full transcripts available for inspection. +DeepAgent can partition independent work across bounded, isolated workers. Delegated runs persist their identity, generation, owner, lease, phase, terminal state, result, and parent delivery, so an exact retry resumes the same work instead of silently starting another worker. Write-capable subagents receive dedicated worktrees, return compact summaries and artifact references, and leave their full transcripts available for inspection. + +Automatic write collaboration follows a durable Git/PR path. Workers commit only their scoped changes; one Reviewer session checks each exact worker SHA, the coordinator performs serial `--no-ff` merges on the Session branch, and one Senior Reviewer examines the merged batch. Resume, timeout, cancellation, takeover, review feedback, and cleanup are generation-fenced so a stale worker cannot settle or overwrite newer work. For high-risk decisions, convene an **Expert Panel**. Correctness, security, performance, architecture, and reproducibility lenses review the same frozen question, debate anonymously for up to three rounds, and feed a deterministic arbiter that preserves minority opinions and fails closed to human review. @@ -91,18 +93,18 @@ For high-risk decisions, convene an **Expert Panel**. Correctness, security, per Project IM brings people and agents into the same thread. Mention an agent to start a scoped run with project context, stream its progress, inspect its artifacts, and keep the answer attached to the conversation that requested it. -## DeepAgent Core V4.0.4_r8 +## DeepAgent Core V4.0.5 -Desktop 1.4.3 ships the eighth reliability revision of the V4.0.4 contract. This release hardens the boundaries where long-running and delegated work previously risked duplicate execution, ambiguous completion, or renderer-wide failure: +Desktop 1.4.4 ships the durable context and collaboration release. It connects Session V2, federated context, delegated execution, Git/PR review, and human supervision without moving durable authority back into process-local agent loops: -- **Durable subagent execution:** TaskRun admission, generation, ownership, leases, settlement, and parent delivery are persisted. Exact retries reconcile to the original run, terminal state uses transactional compare-and-set, and a leased outbox makes completion delivery recoverable without re-running provider work. -- **Two-stage structured finalization:** research and structured output are separate phases. The finalizer is a bounded single turn with thinking disabled when supported, only the `StructuredOutput` tool visible, no historical task or compaction payload, and no empty-success path. -- **Fail-closed model and tool boundaries:** provider capability decisions no longer silently relax required tool choice, invalid tool input is rejected before execution, and provider, schema, permission, interruption, timeout, and doom-loop failures keep distinct recoverable terminal reasons. -- **Semantic no-progress protection:** shell activity uses semantic fingerprints, while the no-progress budget compares bounded results, workspace state, and plan progress so cosmetic command changes cannot evade loop detection. -- **Contained supervision UI:** subagent controls use valid sibling interactions, display real terminal state, and run behind a local ErrorBoundary with retry, close, persisted-mode validation, same-build quarantine, and recovery after build changes. -- **Release-grade verification:** the affected Core, runtime, app, and desktop paths are covered by exact-retry, failure-injection, production Chromium, Electron cold-start, and source-map smoke tests. +- **Durable context and execution:** prompt admission, Context Epoch selection, AgentExecution claims, leases, generations, resource locks, token debits, terminal metadata, and handoffs are persisted before scheduling provider work. +- **Complete multi-agent Git/PR collaboration:** write workers use canonical isolated worktrees, commit only scoped changes, and enqueue SHA-bound PRs. A normal Reviewer handles each item, merges are serialized with `--no-ff`, and a Senior Reviewer closes the batch. +- **Generation-fenced supervision:** resume, cancellation, timeout, takeover, review feedback, and cleanup reuse durable Session identity while preventing stale workers from settling newer work. +- **One private filesystem root:** Core, CLI, Desktop, WSL, databases, logs, caches, and temporary files stay under `~/.deepagent/code/`; production writes cannot be redirected by ordinary environment variables. +- **Cache-safe real-model verification:** stable prompt prefixes are separated from volatile tails, finalizer and compaction baselines are isolated, and the DeepSeek matrix covers sandboxed tools, cache retention, parallel workers, PR closure, recovery, strength inheritance, and EVAL scoring. +- **Release-grade verification:** deterministic state-machine tests, cross-process ownership tests, real-model suites, packaged sidecar checks, Desktop UI coverage, and source-map smoke tests exercise the production entry points. -V4.0.4_r8 retains the durable documents, event delivery, governed learning, isolated worktrees, and secure credential boundaries introduced across the earlier V4.0.4 revisions. +V4.0.5 retains the durable documents, event delivery, governed learning, secure credential boundaries, and fail-closed provider/tool contracts established in the V4.0 line. ## Installation @@ -183,6 +185,8 @@ Official-provider keys added via the app/CLI are stored separately in [providers guide](https://deepagent-code.ai/docs/providers/) for the full reference (base URL overrides, headers, per-model config, gateways). +All DeepAgent Code private filesystem data lives under `~/.deepagent/code/`, including configuration, credential references, databases, Desktop state, logs, caches, and temporary files. Native secret values remain in the operating system's credential store. Tests use explicit isolated roots and cannot redirect production storage through ordinary environment variables. + ## Quick Example Start the agent and give it a task: @@ -284,6 +288,7 @@ bun run --cwd packages/deepagent-code dev import-history --from codex --dry-run - [Providers & Models](https://deepagent-code.ai/docs/providers/) - [Architecture & Design](design/README.md) +- [Real-LLM Testing Guide](design/real-llm-testing.md) - [Security Policy](SECURITY.md) - [Privacy Policy](PRIVACY.md) - [Contributing](CONTRIBUTING.md) diff --git a/README.zh.md b/README.zh.md index 4b1b3c38..394c86af 100644 --- a/README.zh.md +++ b/README.zh.md @@ -14,7 +14,7 @@ Enterprise 版本 -桌面版 1.4.3 · DeepAgent Core V4.0.4_r8
+桌面版 1.4.4 · DeepAgent Core V4.0.5
--- @@ -83,7 +83,9 @@ Session V2 运行器在持久 Context Epoch 下从明确的 Context Source 装 ### 专业子智能体与 Expert Panel -DeepAgent 可以把独立工作拆分给数量有界、相互隔离的 Worker。具备写权限的子智能体获得独立 worktree，只向父会话返回紧凑摘要和工件引用，完整执行记录仍可随时查看。 +DeepAgent 可以把独立工作拆分给数量有界、相互隔离的 Worker。委派运行会持久化自己的身份、generation、owner、lease、阶段、终态、结果和父会话投递，因此 exact retry 会恢复同一项工作，而不是静默启动另一个 Worker。具备写权限的子智能体获得独立 worktree，只向父会话返回紧凑摘要和工件引用，完整执行记录仍可随时查看。 + +自动写协作走一条持久 Git/PR 路径。Worker 只提交其作用域内的改动；同一个普通 Reviewer Session 按精确 Worker SHA 逐项审阅，协调器在 Session 分支上串行执行 `--no-ff` merge，再由同一个 Senior Reviewer Session 审阅合并后的完整批次。恢复、超时、取消、接管、审阅反馈和清理都受 generation fence 保护，过期 Worker 无法结算或覆盖较新的工作。 高风险决策可以召集 **Expert Panel**。正确性、安全、性能、架构与可复现性等专家视角审阅同一个冻结问题，进行最多三轮匿名辩论，再由确定性仲裁器生成裁定。少数派意见会被保留，无法安全达成一致时会失败关闭并交给人类。 @@ -91,18 +93,18 @@ DeepAgent 可以把独立工作拆分给数量有界、相互隔离的 Worker。 项目 IM 把团队成员和智能体放进同一条讨论。@ 某个智能体即可启动有明确作用域的运行，使用项目上下文、流式展示进度、关联执行工件，并把答案留在发起任务的对话里。 -## DeepAgent Core V4.0.4_r8 +## DeepAgent Core V4.0.5 -桌面版 1.4.3 搭载 V4.0.4 合同的第八次可靠性修订。本次发布重点加固长任务和委派任务的关键边界，避免重复执行、模糊终态或局部界面故障演变为整页崩溃： +桌面版 1.4.4 搭载持久上下文与协作版本。它把 Session V2、联邦上下文、委派执行、Git/PR 审阅和人类监督连接起来，同时不把持久权威退回到进程内 Agent loop： -- **持久化子智能体执行：** TaskRun 的准入、generation、owner、lease、结算与父会话投递均持久化。exact retry 会归并到原始运行，终态通过事务 CAS 结算，租约式 outbox 可在不重复执行 provider work 的前提下恢复完成通知。 -- **两阶段结构化终结：** 研究和结构化输出分为独立阶段。finalizer 是次数有界的单轮执行；在 provider 支持时关闭 thinking，只暴露 `StructuredOutput`，不混入历史 task 或 compaction 内容，也不存在空结果成功路径。 -- **失败关闭的模型与工具边界：** provider capability 决策不再静默放宽 required tool choice；无效工具输入在执行前被拒绝；provider、schema、permission、interruption、timeout 与 doom-loop 保留不同且可恢复的终态原因。 -- **语义级无进展保护：** shell 使用语义 fingerprint；无进展预算同时比较有界结果、工作区状态与计划进度，不能再靠改写命令描述绕过循环检测。 -- **故障隔离的监督面板：** 子智能体控件使用合法的 sibling interaction，展示真实终态，并由局部 ErrorBoundary、重试/关闭、持久 mode 校验、同 build quarantine 与 build 变化后的恢复机制保护。 -- **发布级验证：** 受影响的 Core、运行时、App 与 Desktop 路径已覆盖 exact retry、故障注入、production Chromium、Electron 冷启动和 source-map smoke。 +- **持久上下文与执行：** prompt 准入、Context Epoch 选择、AgentExecution claim、lease、generation、资源锁、token debit、终态 metadata 和 handoff 都在调度 provider work 前持久化。 +- **完整多 Agent Git/PR 协作：** 写 Worker 使用 canonical 隔离 worktree，只提交作用域内改动，并进入绑定精确 SHA 的 PR queue。普通 Reviewer 逐项审阅，协调器串行执行 `--no-ff` merge，再由 Senior Reviewer 完成批次复审。 +- **受 generation fence 保护的监督：** 恢复、取消、超时、接管、审阅反馈和清理复用持久 Session 身份，同时阻止过期 Worker 结算较新的工作。 +- **唯一私有文件数据根：** Core、CLI、Desktop、WSL、数据库、日志、缓存和临时文件统一位于 `~/.deepagent/code/`；普通环境变量不能重定向生产写入。 +- **缓存安全的真实模型验证：** 稳定 prompt prefix 与易变 tail 分离，finalizer 和 compaction 使用隔离基线；DeepSeek 矩阵覆盖 sandbox 工具、缓存保持、并行 Worker、PR 闭环、恢复、强度继承与 EVAL 评分。 +- **发布级验证：** 确定性状态机测试、跨进程 ownership 测试、真实模型 suite、packaged sidecar 检查、Desktop UI 覆盖和 source-map smoke 共同验证生产入口。 -V4.0.4_r8 同时保留此前 V4.0.4 各修订建立的持久文档、可靠事件投递、知识治理、worktree 隔离与安全凭据边界。 +V4.0.5 保留 V4.0 系列建立的持久文档、可靠事件投递、知识治理、安全凭据边界以及失败关闭的 provider/tool 合同。 ## 安装 @@ -178,6 +180,8 @@ deepagent auth list 完整参考（Base URL 覆盖、请求头、逐模型配置、网关）见 [供应商文档](https://deepagent-code.ai/docs/providers/)。 +DeepAgent Code 的所有私有文件数据都位于 `~/.deepagent/code/`，包括配置、凭据引用、数据库、桌面状态、日志、缓存和临时文件；原生 secret 值仍由操作系统凭据存储保管。测试使用显式隔离的数据根，普通环境变量不能重定向生产存储。 + ## 快速示例 启动智能体并交给它一个任务： @@ -279,6 +283,7 @@ bun run --cwd packages/deepagent-code dev import-history --from codex --dry-run - [供应商与模型](https://deepagent-code.ai/docs/providers/) - [架构与设计](design/README.md) +- [真实 LLM 测试指南](design/real-llm-testing.md) - [安全策略](SECURITY.md) - [隐私策略](PRIVACY.md) - [贡献指南](CONTRIBUTING.md) diff --git a/bun.lock b/bun.lock index eec42051..02af02fa 100644 --- a/bun.lock +++ b/bun.lock @@ -29,7 +29,7 @@ }, "packages/app": { "name": "@deepagent-code/app", - "version": "1.4.3", + "version": "1.4.4", "dependencies": { "@codemirror/autocomplete": "6", "@codemirror/commands": "6", @@ -129,7 +129,7 @@ }, "packages/core": { "name": "@deepagent-code/core", - "version": "4.0.4-r8", + "version": "4.0.5", "bin": { "deepagent-code": "./bin/deepagent-code", }, @@ -295,8 +295,10 @@ "google-auth-library": "10.5.0", "gray-matter": "4.0.3", "htmlparser2": "8.0.2", + "ignore": "7.0.5", "immer": "11.1.4", "jsonc-parser": "3.3.1", + "marked": "catalog:", "mime-types": "3.0.2", "minimatch": "10.2.3", "npm-package-arg": "13.0.2", @@ -345,8 +347,9 @@ }, "packages/desktop": { "name": "@deepagent-code/desktop", - "version": "1.4.3", + "version": "1.4.4", "dependencies": { + "@deepagent-code/core": "workspace:*", "@zip.js/zip.js": "2.7.62", "effect": "catalog:", "electron-context-menu": "4.1.2", @@ -3029,7 +3032,7 @@ "get-tsconfig": ["get-tsconfig@4.14.0", "", { "dependencies": { "resolve-pkg-maps": "^1.0.0" } }, "sha512-yTb+8DXzDREzgvYmh6s9vHsSVCHeC0G3PI5bEXNBHtmshPnO+S5O7qgLEOn0I5QvMy6kpZN8K1NKGyilLb93wA=="], - "ghostty-web": ["ghostty-web@github:anomalyco/ghostty-web#513463a", {}, "anomalyco-ghostty-web-513463a", "sha512-GZR8LSmgGzViWnBJrqRI8MpAZRCJxhcr1Hi9Tyeh7YRooHZQjK9J97FQRD3tbBaM2wjq05gzGY2UEsG+JtZeBw=="], + "ghostty-web": ["ghostty-web@github:anomalyco/ghostty-web#83c0a07", {}, "anomalyco-ghostty-web-83c0a07", "sha512-Lf2v1agHkVUpMpHBWWuCZrhOEmcwwin5/Hboc9rZwQ7/CKkIh5rU1r1CvfLlhkMoFv+ed8z52RZ8hkzGZZj3MQ=="], "giget": ["giget@3.3.0", "", { "bin": { "giget": "dist/cli.mjs" } }, "sha512-gzi2D96p+AMfDcmJHGDj3KJ9NRiwvlFAU5yfa3ROwWZmFUjX4P43x3BcyRaOMMLto1vUo7C+86+MFhYTl6Ryiw=="], diff --git a/design/README.md b/design/README.md index 07baa4b9..2b75c7d7 100644 --- a/design/README.md +++ b/design/README.md @@ -1,18 +1,22 @@ # DeepAgent Code Architecture & Design -> **Public design overview for DeepAgent Core V4.0.4_r8 / Desktop 1.4.3.** Internal implementation details and roadmap documents live in the private `docs/` tree and are intentionally not version-controlled. +> **Public design overview for DeepAgent Core V4.0.5 / Desktop 1.4.4.** Internal implementation details and roadmap documents live in the private `docs/` tree and are intentionally not version-controlled. DeepAgent Code is a document-centered, event-driven AI coding system. It combines a coding-agent runtime with a durable control plane that owns context, planning, learning, collaboration, safety, and human oversight. The architecture is designed around one requirement: a long-running agent must remain correct and governable after many model turns, tool calls, user interventions, process restarts, and worker handoffs. +## User Guides + +- [Running Real-LLM Tests](./real-llm-testing.md) + ## Design Principles - **Durable document and project memory** — atomic, recoverable storage with retrieval gates, provenance, governance, and conflict detection - **Connected context** — selective, evidence-backed assembly across code, knowledge, project memory, and execution documents - **Plans and long-running goals** — structured plans, stale-state detection, validation evidence, bounded retries, and human control - **Event-driven coordination** — durable delivery, priority routing, offline catch-up, idempotent goal ticks, and observable queue state -- **Isolated agent collaboration** — bounded subagents, worktree isolation for write-capable workers, and conflict-aware change return +- **Durable agent collaboration** — generation-fenced delegated runs, worktree isolation, resumable supervision, and Reviewer/Senior Reviewer PR closure - **AI IDE microservice** — LSP-backed semantic code navigation via `code_intel` - **Secure MCP catalog** — curated integrations, derived safety tiers, environment references, and native OS secret storage @@ -22,7 +26,7 @@ Sessions, inputs, plans, documents, goals, events, approvals, and learning decis A user instruction is durably admitted before execution is scheduled. A successful API response therefore means the instruction is recorded, not merely present in a process-local queue. -DeepAgent is built **on top of** the opencode agent/runtime/session/tool/MCP stack. V4.0.4_r8 strengthens the control plane without replacing the current turn engine, tool system, or provider layer. +DeepAgent is built **on top of** the opencode agent/runtime/session/tool/MCP stack. V4.0.5 strengthens the durable control plane without replacing the current turn engine, tool system, or provider layer. ### Durable delegated execution @@ -30,7 +34,7 @@ A delegated task is admitted before provider work begins and is identified indep Structured work uses two explicit stages. The research stage retains the child Agent's normal tools and transcript. The finalizer is a bounded, single provider turn with a narrow ephemeral tool registry, no task or compaction history, no ordinary steering, and a per-turn reasoning/tool-choice decision derived from provider capability. Completion is valid only after schema-validated structured output is persisted; provider, schema, permission, timeout, interruption, and no-progress failures remain distinct recoverable states. -### 2. One durable authority per concern +### One durable authority per concern Documents, plans, event delivery state, knowledge promotion, and goal progress each have one authoritative durable store. In-memory state may accelerate delivery, but it cannot become a second source of truth. @@ -42,10 +46,12 @@ External events, credentials, tools, paths, autonomy, worker placement, and outb ### Humans can always intervene -### 5. Keep execution boundaries explicit - Write-capable subagents run in isolated worktrees by default. Event consumers claim durable work with idempotency and retry boundaries. Users retain explicit paths to approve, steer, pause, resume, take over, or roll back long-running work. +### Keep execution boundaries explicit + +Interactive `task` collaboration and the event-driven V4 DAG share durable Session, permission, worktree, and PR primitives, but keep their scheduling contracts explicit. A task worker is supervised through a durable TaskRun generation. A V4 node is claimed through durable AgentExecution state with lease, resource, token, and dependency constraints. Both paths preserve author commits before cleanup and fail closed when they cannot prove ownership or continuation. + --- ## System Map @@ -104,6 +110,8 @@ Steering never aborts an in-flight tool or stream. The input is persisted first, Same-Session resumes join one coordinator; advisory wakes coalesce; different Sessions can run concurrently. Every provider turn performs one explicit `llm.stream(request)` call and reloads projected history before durable continuation. +All production-private filesystem data resolves from one root: `~/.deepagent/code/`. Core databases and configuration, CLI state, Desktop state/logs/caches, credential references, and runtime temporary files remain under that directory; native secret values remain in the operating system's credential store. `DEEPAGENT_CODE_HOME` is honored only together with an explicit test-home boundary, so ordinary production environments cannot redirect private writes. + **Credentials** are declared by key name in the catalog template (`CredentialSpec`). Configuration stores environment references or `secret://` handles instead of plaintext values. Handles resolve at connection time through macOS Keychain, Linux Secret Service (`libsecret`), or Windows DPAPI-backed credential storage. - Agent instructions, stable policy, and System Context baseline stay in the prefix. @@ -120,8 +128,8 @@ Same-Session resumes join one coordinator; advisory wakes coalesce; different Se | Dangerous writes: approval gate (`ctx.ask`) | Available | | Read-only DB: restricted mode enforced at server | Available | | Credential indirection (`${VAR}` / `secret://`) | Available | -| Native secret storage (macOS / Linux / Windows) | Available in V4.0.4 | -| Subagent write isolation and conflict-aware return | Available in V4.0.4 | +| Native secret storage (macOS / Linux / Windows) | Available | +| Subagent write isolation and conflict-aware return | Available | - a stable ID and monotonic version; - type, scope, status, domain, tags, and description; @@ -200,6 +208,20 @@ The Worker Pool owns bounded concurrency, placement, claims, leases, renewal, re Write-capable Agents use isolated worktrees by default. Parent Agents receive bounded summaries, status, artifact/session references, and necessary diffs; complete child transcripts remain in their own Sessions. +`AgentExecution` persists claim, lease, generation, resource locks, token debits, terminal metadata, and handoff state in SQLite. Independent DAG nodes may execute concurrently under `WorkspaceConcurrency`; dependent nodes start only after their complete upstream wave settles. Cleanup must preserve a command-scoped Git continuation ref before an author worktree is removed. + +## Multi-Agent Git and PR Collaboration + +The model-facing `task` and `pr_finalize` tools form one production collaboration path: + +1. write workers are admitted as durable TaskRuns and receive canonical isolated worktrees; +2. each worker commits only its declared scope and enqueues a PR bound to the exact author Session and SHA; +3. one ordinary Reviewer Session reviews each queued SHA, with `request_changes` resuming the same author Session and worktree; +4. accepted commits merge serially with `--no-ff` into the Session branch; +5. one Senior Reviewer Session reviews the merged batch before cleanup and terminal delivery. + +Queue state, reviewer identities, verdict SHA, batch review, merge state, and cleanup requirements are durable. Retry reuses the recorded Reviewer or Senior Reviewer Session; stale generations cannot settle current work. Default branches are never automatic merge targets, and a dirty parent checkout fails before worker or provider execution. + ## Human Collaboration and Oversight ### Repo & Wiki diff --git a/design/real-llm-testing.md b/design/real-llm-testing.md new file mode 100644 index 00000000..bef2084b --- /dev/null +++ b/design/real-llm-testing.md @@ -0,0 +1,493 @@ +# DeepAgent Code 真实 LLM 测试标准与运行指南 + +本文是仓库中真实 LLM 测试设计、覆盖、运行、资格认证和审阅标准的唯一权威文件。历史 `docs/llmrealtest*.md`、LLM 测试审阅记录和覆盖矩阵只保留迁移指针，不再定义合同；任何冲突以本文和当前源码注册表为准。 + +本文面向需要设计、审阅或运行 DeepAgent Code 真实模型测试的开发者，说明凭证、安全边界、Oracle、路由、资格和结果解释。真实 LLM 测试会访问 DeepSeek 官方 API，可能产生调用费用；先运行 `--dry-run`，再从无 Desktop、无 EVAL 的小矩阵开始。 + +## 0. 权威状态与硬合同 + +截至 2026-07-31，聚合 runner 注册 55 条命令，其中 48 条调用真实模型；headless 矩阵为 48 条命令、43 条真实模型 suite，进一步跳过 EVAL 和安装后为 46 条命令、42 条真实模型 suite。数字由 `script/run-live-llm-all.ts` 动态注册表决定，文档数字发生漂移时以注册表和 `validateSuiteManifest()` 为准。 + +`qualifiedLiveRuns` 当前为空。注册、单次通过和 EXT 可达都不代表 pre-push 资格；LIVE suite 只有完成本节资格合同后才能进入该集合。 + +测试清单有四层权威，职责不能混用： + +1. 各 package 的 `test:llm-*` scripts 定义可直接执行的 suite 入口； +2. `script/run-live-llm-all.ts` 定义完整聚合矩阵，并用 `validateSuiteManifest()` 保证所有真实 suite 恰好注册一次； +3. `packages/deepagent-code/script/live-llm/routes.ts` 把生产源码路径映射到确定性检查和受影响的模型运行； +4. `packages/deepagent-code/script/live-llm/dispatcher.ts` 把路由运行映射到实际命令，并单独维护已完成资格认证的 `qualifiedLiveRuns`。 + +`routes.ts`/`dispatcher.ts` 服务变更影响分析和 pre-push，聚合 runner 服务显式全矩阵验证。新增或重命名 suite 时必须同时更新 package script、聚合矩阵、相关源码路由和 dispatcher；`test:llm-routes` 会拒绝缺失、陈旧、重复或无法分类的注册。 + +| 模式 | 用途 | 真实模型 | 普通 pre-push | +| ------- | -------------------------------------- | -------- | ---------------- | +| DET | 状态机、路由、安全和 Oracle 自检 | 否 | 是 | +| LIVE | 小型、稳定、低成本生产合同 | 是 | 资格通过后可进入 | +| EXT | 多 Agent、故障恢复、长会话和高成本编排 | 是 | 否 | +| EVAL | 多 seed 能力评分 | 是 | 否，只报告 | +| RELEASE | Desktop、GUI 和 packaged sidecar | 是 | 否 | + +所有真实 suite 必须满足以下不变量： + +1. 只允许 `https://api.deepseek.com`，并记录 provider、模型、revision 和 generation fingerprint。 +2. API key 只从 `DEEPAGENT_CODE_LIVE_LLM_API_KEY_FILE` 指向的仓库外 `0600` 单行文件读取；raw key 环境变量一律拒绝。 +3. HOME、应用数据、数据库、workspace 和临时目录全部隔离；子进程只继承显式环境 allowlist。 +4. 工具权限默认 deny；Bash suite 在请求 provider 前完成 sandbox conformance，并拒绝宿主文件、symlink escape 和网络。 + Python hidden verifier 必须通过 sandbox 选择的原生 Python 运行时执行；运行时不可用或不能产生结构化 verifier marker 时，结果归类为 `sandbox-contract`，不得归因于模型。 +5. 通过条件优先使用 fresh-copy verifier、精确磁盘状态、durable Session/tool/PR 状态和 typed terminal reason；模型自述不能单独判定通过。 +6. suite 必须有 provider turn、工具调用、输出、磁盘、并发和 wall-time 上限，结束时不得残留进程、权限请求或问题请求。 +7. 失败必须归类为 `preflight`、`sandbox-contract`、`provider-contract`、`runtime-contract`、`model-behavior`、`budget` 或 `harness-bug`，并报告最早失效层。 + +LIVE 资格要求 committed harness、固定 fingerprint、至少三个独立进程连续 30/30、mutation/self-test、14 天滚动稳定性、artifact 脱敏审阅和负责人批准。复杂多 Agent、长会话、Desktop 与 EVAL 原则上保持 EXT/RELEASE；没有上述证据不得加入 `qualifiedLiveRuns`。 + +Provider prompt cache 的归因也必须 fail closed：同一 Session 的 system prefix 和工具 schema/order 若由 DeepAgent Code 改变，先判 `runtime-contract`；只有 assembled prefix identity、历史和参数稳定而 provider cache read 仍崩溃时，才允许判 `provider-contract`。round、plan、World State、schema 和 finalizer 等逐轮信息只能进入 ephemeral runtime tail，不能污染稳定 system prefix。 + +## 1. 前置条件 + +开始前请确认： + +- 已安装仓库声明的 Bun 版本； +- 已执行 `bun install --frozen-lockfile`； +- 网络可以访问 `https://api.deepseek.com`； +- DeepSeek API key 有权调用配置中的模型； +- 运行 Desktop 测试时，本机具有可用的图形环境和 Desktop 构建依赖。 + +在仓库根目录安装依赖： + +```sh +bun install --frozen-lockfile +``` + +## 2. 准备 API key 文件 + +不要把 API key 写入仓库文件、命令行参数或普通环境变量。测试使用 `DEEPAGENT_CODE_LIVE_LLM_API_KEY_FILE` 指向一个仓库外的 key 文件。 + +该文件只能包含一行原始 API key，不是 JSON 或 `KEY=value`。macOS 和 Linux 示例： + +```sh +mkdir -p "$HOME/.deepagent/code/tmp" +touch "$HOME/.deepagent/code/tmp/live-llm-deepseek.key" +chmod 600 "$HOME/.deepagent/code/tmp/live-llm-deepseek.key" +${EDITOR:-vi} "$HOME/.deepagent/code/tmp/live-llm-deepseek.key" +``` + +检查文件没有多余行，并确认权限： + +```sh +awk 'END { print NR }' "$HOME/.deepagent/code/tmp/live-llm-deepseek.key" +ls -l "$HOME/.deepagent/code/tmp/live-llm-deepseek.key" +``` + +测试不接受以下明文 key 环境变量： + +- `DEEPAGENT_CODE_LIVE_LLM_API_KEY` +- `DEEPSEEK_API_KEY` + +如果当前 shell 已设置它们，请先清除： + +```sh +unset DEEPAGENT_CODE_LIVE_LLM_API_KEY +unset DEEPSEEK_API_KEY +``` + +## 3. 配置聚合测试 + +在仓库根目录创建本地配置： + +```sh +cp script/live-llm.config.example.json script/live-llm.config.local.json +${EDITOR:-vi} script/live-llm.config.local.json +``` + +`script/live-llm.config.local.json` 已被 Git 忽略。配置示例： + +```json +{ + "baseURL": "https://api.deepseek.com", + "apiKeyFile": "~/.deepagent/code/tmp/live-llm-deepseek.key", + "model": "deepseek-v4-flash", + "modelRevision": "", + "requestTimeoutMs": 180000, + "suiteTimeoutMs": 1200000, + "evalRuns": 5, + "installDependencies": true +} +``` + +配置字段说明： + +| 字段 | 含义 | +| --------------------- | -------------------------------------------------------------------- | +| `baseURL` | Provider endpoint。当前真实测试要求使用 `https://api.deepseek.com`。 | +| `apiKeyFile` | key 文件路径。支持绝对路径、相对配置文件的路径和 `~/...`。 | +| `model` | 本次测试使用的模型 ID。 | +| `modelRevision` | 可选的模型修订标识；Provider 未提供明确修订时可留空。 | +| `requestTimeoutMs` | 单次 Provider 请求超时，单位毫秒。 | +| `suiteTimeoutMs` | 单个测试 suite 的总超时，单位毫秒。 | +| `evalRuns` | EVAL 重复次数，范围为 1–20。次数越高，费用和耗时越高。 | +| `installDependencies` | 聚合测试是否先运行依赖安装步骤。命令行 `--skip-install` 可以覆盖它。 | + +## 4. 启动聚合测试 + +所有聚合命令都从仓库根目录运行。 + +### 4.1 先检查配置和测试清单 + +```sh +bun run test:llm:all -- --dry-run +``` + +`--dry-run` 会验证配置结构和 suite 注册，并打印将要执行的命令；它不会读取 key 文件，也不会请求 Provider。 + +聚合 runner 只读取 `--config` 指定的 JSON（默认 `script/live-llm.config.local.json`），再为真实 suite 构造最小环境 allowlist。单 suite 不读取该 JSON，只读取第 5 节列出的 `DEEPAGENT_CODE_LIVE_LLM_*` 环境变量。不要同时假定两种配置入口会互相回填。 + +### 4.2 推荐的首次真实运行 + +首次运行建议跳过 Desktop、EVAL 和重复安装： + +```sh +bun run test:llm:all -- --headless --skip-eval --skip-install --stop-on-failure +``` + +这个组合适合先确认 Provider、Core 和 CLI 的真实模型测试可以在当前机器上运行。 + +### 4.3 运行完整矩阵 + +```sh +bun run test:llm:all +``` + +完整矩阵包含确定性前置检查、真实模型 suites、EVAL 和 Desktop suites，耗时与 API 用量都会明显增加。 + +### 4.4 常用参数 + +| 参数 | 效果 | +| ------------------- | ------------------------------------------------------------------ | +| `--config