feat: agent secret scope

[mlikasam-askui]

Lets users define secrets the agent can use (e.g. type a password) without ever exposing the value to the LLM.

The model only sees the placeholder <|secret|>NAME<|secret|>; the real value (held as a pydantic SecretStr) is substituted into tool inputs only at execution time. Literal values are redacted from the LLM conversation history and tool outputs, and the available placeholders are advertised to the model via an <AVAILABLE_SECRETS> system-prompt section.

Usage

from askui import ComputerAgent, Secret

with ComputerAgent(secrets=[Secret(name="password", value="...")]) as agent:
    agent.act("Log in as admin using the password")

What's included

• New Secret / SecretVault (exported from askui); value accepts a plain str or SecretStr.
• secrets= on all agent types (ComputerAgent, AndroidAgent, WebAgent, WebTestingAgent, MultiDeviceAgent) + per-call override via act(..., secrets=[...]). Excluded from telemetry.
• Generic substitution in ToolCollection (works for type, custom, and MCP tools) + tool-output redaction.
• type tool descriptions document the placeholder; example in examples/secrets.py; unit + integration tests.

Known limitation

A secret typed into a visible field can still appear in screenshots sent to the model — on-screen secrets aren't masked.

Jira: SOLENG-295

[philipph-askui]

Please update the docs to reflect the new concept

• I assume you tested this?

[philipph-askui]

what if the name of the secret is by concidence an important information. Could happen if the secret is e.g. adminstrator or similar. Than this would be misidentified as secret and redacted, no?

[philipph-askui]

why this format and not standard xml format or just a bullet point list?

[philipph-askui]

PR review from Claude below

PR Review: feat: agent secret scope

Overall Assessment

Well-designed feature with a clear trust model. The placeholder-based approach (<|secret|>NAME<|secret|>) is a solid choice — it keeps secrets out of the LLM context while remaining readable. The code is well-documented, follows the project's conventions, and the unit tests are thorough. A few issues to address below.

---

Critical

1. _redact_block() mutates its argument in-place (secrets.py:228-235)

redact_message() does message.model_copy(deep=True) which is correct, but _redact_block() mutates the block objects directly (block.text = ..., block.content = ..., block.input = ...). Since model_copy(deep=True) creates deep copies this happens to work, but it's fragile — if _redact_block is ever called outside of redact_message (e.g. via redact_content, which IS used independently in tools.py), it will mutate the original content blocks.

In ToolCollection._run_regular_tool() and _run_mcp_tool(), redact_content() is called on freshly-constructed _convert_to_content() results, so it's safe today — but this is a latent foot-gun. Consider making _redact_block() return a copy consistently, or add a comment documenting the mutation contract.

2. Integration tests were deleted in commit 2

The integration tests (tests/integration/test_secrets.py, 212 lines) were created in commit 1 and then fully removed in commit 2. The final branch state has no integration tests for the secrets feature. Was this intentional? These tests (LLM never sees secret, reporter never sees secret, per-call override) would be valuable to keep.

---

Medium

3. __add__ on ToolCollection uses or for vault merging (tools.py)

secret_vault=other._secret_vault or self._secret_vault,

This picks other's vault if it's truthy (non-empty), otherwise falls back to self's. But if both vaults have secrets, secrets from self are silently dropped instead of being merged. Should this be self._secret_vault.merge(other._secret_vault) to be consistent with how act() merges vaults?

4. Secret value logged in _redact_str warning message — double-check

The warning in _redact_str() correctly logs secret.name and secret.placeholder (never the value). Good. But the log message says "before it left the trusted boundary" — this warning fires every time redaction happens (every _add_message call, every tool output). This could be extremely noisy in practice if the model echoes back typed text. Consider downgrading to logger.debug after the first occurrence, or using a set to warn only once per secret per act() call.

5. _substitute_str tolerates whitespace but _redact_str doesn't

_substitute_str strips whitespace from placeholder names (name = match.group(1).strip()), which is nice. But _redact_str does exact substring matching of the literal value. This asymmetry seems intentional (redaction is defense-in-depth), but worth a brief doc comment to avoid confusion.

---

Minor / Nits

6. WebTestingAgent missing secrets in telemetry exclude

WebAgent.__init__ correctly adds "secrets" to the telemetry exclude set, but WebTestingAgent doesn't appear to — worth checking that secret values don't leak into telemetry there.

7. Stale docstring arg in act() (agent_base.py:154)

            act_model (ActModel | None, optional): Model to use for this act
                execution.

This act_model parameter doesn't exist in the signature anymore (looks like it was removed in a previous PR but the docstring wasn't updated). Pre-existing issue, but since you're touching this method it's a good time to clean it up.

8. Type annotation for _redact_block could be tighter

_redact_block(self, block: Any) -> Any — since it handles TextBlockParam | ToolResultBlockParam | ToolUseBlockParam, a ContentBlockParam union type would be more precise.

9. Example file uses hardcoded secret value

examples/secrets.py includes "hunter2-demo-only" as a hardcoded value. The docstring warns this is for demo only, which is fine, but consider whether this is the right pattern to show in an example file that users will copy-paste. The env-var approach is already demonstrated — the hardcoded variant could be removed or at minimum moved to the end and de-emphasized.

10. Missing __init__.py for test directory?

Convention says "Create __init__.py in each folder" — tests/unit/test_secrets.py is a new file. If there isn't already an __init__.py in tests/unit/, one should be added. (Likely already exists, just flagging.)

---

What looks good

• Clean separation of concerns: SecretVault owns all substitution/redaction logic, agents just call it.
• Defense-in-depth: Redaction at multiple layers (goal, initial messages, _add_message, tool outputs) is well-thought-out.
• Immutability: substitute() and redact() return new objects. redact_message() deep-copies. Good.
• Deep-copy of ActSettings: The model_copy(deep=True) in _setup_control_loop fixes what would have been a nasty bug with accumulating system prompt sections across calls.
• Pydantic SecretStr: Proper masking in repr/dump/json.
• Comprehensive unit tests: 26 tests covering substitution, redaction, merging, system prompt generation, tool output redaction, and edge cases.
• Consistent propagation: All agent types (Computer, Android, Web, WebTesting, MultiDevice) forward secrets correctly.
• Placeholder design: Using <|secret|> delimiters is unlikely to collide with natural text.

[programminx-askui]

Did you not instroduced a SecretVault? I want to have the possibilitiy to plugin different stores.

[programminx-askui]

Should this not moved to a docs section?

[programminx-askui]

I think the <|secret|> pre- & suffix is to annyoing.

[programminx-askui]

Why a deep copy?