Skip to content

Put shared state where every replica can see it, and bound what grows - #90

Merged
davidmckayv merged 7 commits into
mainfrom
fix/issue-88-shared-state
Aug 21, 2026
Merged

Put shared state where every replica can see it, and bound what grows#90
davidmckayv merged 7 commits into
mainfrom
fix/issue-88-shared-state

Conversation

@davidmckayv

Copy link
Copy Markdown
Contributor

Closes #88, except for item 8, which #30 already fixes properly and which I have reviewed and
approved rather than duplicated.

The theme is one thing rather than seven: state a deployment has to share was kept per process, and
data that grows with use had nothing bounding it. Both hold up on a laptop and stop holding somewhere
between one server and a company.

1. A boundary change reaches every server

The policy is read from memory on every action, which is right. Memory was only ever filled at boot,
so an administrator's new rule applied on the one server that received it and nowhere else, while the
admin screen reported success because the row really was saved and the audit trail agreed because it
records the boundary each process started with. Both honest, both describing something other than
what the fleet was enforcing. A deny rule that stops roughly one action in N is worse than no deny
rule, because it looks like it works.

A write now announces on Postgres and every server re-reads, the same mechanism channel activity
already uses. Reset travels too, which is the direction easy to forget: otherwise the server that
served the reset forgets the rule and the rest keep refusing work an administrator believes they have
just allowed again.

Driven with two real replicas on one database. Added the rule through the Boundaries screen on
replica 1; replica 2, which never served that request, went from deny: [] to enforcing it. Removed
it and replica 2 went back. On main replica 2 stays empty forever.

2. A computer stops accumulating browsers

A Chromium context was started the first time each Bot was used and kept. The only things that
dropped one were an explicit stop, a browser that had already died, and shutdown, so a deployment
where every employee has a Bot trends toward one resident browser per employee in a single container
at a few hundred MB each, until it is killed for memory and relaunches its way back to the same
state. There is a cap and an idle timeout now. A cap as well as a timeout, because the failure is
concurrent breadth rather than age: fifty people using their Bots inside one minute are fifty live
browsers and none of them are idle. Closing one costs a relaunch and nothing else, since the profile
is on disk. The per-Bot session map had the same shape and is bounded the same way.

3. The audit trail can be read quickly, and kept for a stated time

The screen filters by event type, by who did it and by what it was done to. There was one index, on
created_at, so every one of those was a sequential scan over the biggest table in the deployment
plus a sort. Each filter leads its own index now and carries the sort.

Nothing ever removed a row, which is a disk problem and separately a control an enterprise buyer asks
to see. AUDIT_RETENTION_DAYS expresses one, unset by default because deleting somebody's trail
because a default said so is the worse of the two failures.

The append-only guarantee mattered more than the feature. It is a trigger, deliberately, because
the application is not the only thing that can reach the table. Rather than work around it, the
trigger now permits a delete only when the transaction declares a retention window and only for rows
already outside it, so "we removed the rows about the incident" is still impossible and an UPDATE
still is under every condition. Three of the eight tests exist to hold that line rather than to test
the feature.

One server sweeps, chosen by an advisory lock on a connection the sweep owns. It cannot go through
the pool: a session lock belongs to one connection, so the pool would take it on one and release it
on another and the lock would leak. Found by the test that puts a real second connection in the way.

4. Rotating a key retires the old one

Editing a Bot's key wrote a new vault row and repointed the Bot at it. The previous credential stayed
decryptable and still valid with nothing listing it, so rotation did not do the one thing rotation is
for. Deleting a Bot left its key live too, which was the last chance anybody had to retire it. Both
revoke now, after the replacement is stored so a failure leaves a key too many rather than none.

5. Changing a Bot leaves a trace

Ten mutating routes wrote one audit row between them, and there was no event type for any of the
other nine, so it was a missing vocabulary before it was a missing call. A Bot's endpoint is where
conversation content is sent; "who pointed this Bot at that host, and when" is the first question in
an incident. Eight event types and eight rows, through one helper. What is recorded is what changed
and never a value: a replaced key is keyReplaced: true, an issued callback token is the fact it was
issued.

6 and 7. The people list and the channel list stop growing

list() took no arguments and returned everybody, joined to their roles, accounts and every session
they had ever held, on every render. find was (await list()).find(...), so reading one person ran
that aggregate over the whole deployment, twice per role change. Keyset paging with server-side
search now, and find has its own query. The cursor carries both halves of the sort, because
nulls last means a null compares the opposite way to what < says about it.

The sidebar asked for every channel, one row per channel-agent pair. The page is chosen over channels
and the agents joined to it afterwards, which is the subtlety: a limit on rows would cut a two-Bot
channel in half and its second Bot would arrive on the next page as a separate entry with the same
id. The socket that patches the list live learned about pages with it.

Driven with 120 seeded people. 50 rows, then 99, then 120 across two Show more clicks with no
duplicates and the button gone at the end. Searching person118 found somebody three pages deep,
which filtering what had arrived could never have done.

Two things found by driving, not by reading

The Boundaries screen was dead on main. #37's bot-access middleware matches /:botId/*, Hono
matches /* against zero segments, and /api/computers/policy therefore arrived as a Bot called
"policy". The entire surface for writing boundary rules answered 404 to everybody, with a message
about a Bot. I approved that PR, and every test of both features passed. Fixed here, with a test that
also holds the part that matters: the exemption is from the bot-access check and not from
requireAdmin.

The Activity pane defaulted to a stale screen. The screen belongs to the computer rather than the
conversation, so a Bot that spent a whole conversation in a terminal showed another conversation's
page captioned as its own live screen. The pane opens on whichever surface the Bot is actually using
now, and the caption claims the page only once this Bot has opened one.

aimock

mcp.ts is the one door that speaks MCP to somebody else's server and nothing tested it. What it
does with a reply is not a detail: the text lands in a model's context, the 20k cap decides how much
of a context window somebody else's server may spend, and isError decides whether a Bot is told a
tool failed or told nothing. Nine cases against a real MCP server, plus four more AG-UI cases for the
shapes a real agent turns up in rather than the happy one.

Two fixture notes found here: aimock will serve an MCP tool with no inputSchema and the official SDK
then rejects the entire listing rather than that one tool, and it will serve image content whose data
is not valid base64.

Where it runs

  • New state that outlives a request? One new setting, AUDIT_RETENTION_DAYS. No new tables.
  • What happens on the second replica? That is the point of items 1, 2, 6 and 7. Item 1 is
    driven against two real replicas; the retention sweep is single-flighted by an advisory lock.
  • Anything serialised? Two cursors, base64url, both tolerant of a stale or hand-edited value.
  • Anything fanned out to a browser? The channel list is paged and the socket patches inside a
    page rather than across pages.
  • New listener, port, or schedule? A Postgres LISTEN on its own connection, released on the
    way out with the existing one, and an interval for retention that is unrefed and off unless
    configured.

Proof

bun test: 707 pass in server, 114 in app, 123 in agent-computer, with the two pre-existing
server-side-tools.integration failures that fail identically on main. Typecheck clean,
format:check clean across 374 files, drizzle-kit check fine, and migrations verified by bringing a
database up from empty and checking the end state.

Not in scope

Nothing archives or deletes a channel. Item 7 says so and I have bounded the read rather than built
archival: accumulating conversations is inherent to a chat product, and a delete flow is a product
decision with its own surface rather than half a fix bundled into this.

…wing

Two of the eight in #88, and the same shape: state a deployment shares was held per process, and
something that grows with use had nothing bounding it.

The policy is read from memory on every action, which is right, but memory was only ever filled at
boot. An administrator's new rule arrived at one server out of N and applied there and nowhere else,
while the admin screen reported success because the row really was saved and the audit trail agreed
because it records the boundary each process started with. Both honest, both describing something
other than what the fleet was enforcing. A deny rule that stops one action in N is worse than none,
because it looks like it works. A write now announces on Postgres and every server re-reads, the same
mechanism channel activity already uses. Reset travels too, which is the direction easy to forget.

The people list took no arguments and returned everybody, joined to their roles, their linked
accounts and every session they had ever held, on every render of the screen. `find` was
`(await list()).find(...)`, so reading one person ran that aggregate over the whole deployment, twice
per role change. It is a keyset page now, with server-side search, and `find` has its own query. The
cursor carries both halves of the sort because `nulls last` means a null compares the opposite way to
what `<` says about it.

Fourteen tests across two files, both against a real database, because what is being tested is the
SQL and the delivery rather than a shape a fake was written to return.
…or a stated time

Two more from #88, both about the table that becomes the largest in the deployment within weeks.

The audit screen filters by event type, by who did it and by what it was done to. There was one
index, on `created_at`, so every one of those filters was a sequential scan over the biggest table
there is, plus a sort. Each filter now leads its own index and carries the sort, and `id` is in all
of them because the keyset pages on `(created_at, id)`.

Nothing ever removed a row. That is a disk problem and, separately, a control an enterprise buyer
asks to see: "what is your retention policy" had no answer other than "everything, forever, and you
cannot express anything else". `AUDIT_RETENTION_DAYS` now expresses one. Unset by default, because
deleting somebody's trail because a default said so is the worse of the two failures.

The trail stays append-only, and that mattered more than the feature. It is enforced by a trigger,
deliberately, because the application is not the only thing that can reach the table. Rather than
work around it, the trigger now permits a delete only when the transaction declares a retention
window and only for rows already outside it, so "we removed the rows about the incident" is still
impossible and an UPDATE still is under every condition. Three of the eight tests exist to hold that
line rather than to test the feature.

One server sweeps, chosen by an advisory lock on a connection the sweep owns. It cannot go through
the pool: a session lock belongs to one connection, so the pool would take it on one and release it
on another and the lock would leak until that connection died. Found by the test that puts a real
second connection in the way.

Migration 0006 also repairs the snapshot chain, which two branches renumbering 0004 had left
disagreeing with itself. Verified by migrating a database from empty and checking the end state.
… trace

Two more from #88.

Editing a Bot's key wrote a new vault row and repointed the Bot at it. The previous credential stayed
in the vault, decryptable and still valid, with nothing listing it and no path able to revoke it, so
the honest answer to "is that leaked key still live" was yes, and rotation did not do the one thing
rotation is for. Deleting the Bot left its key live too, which was the last chance anybody had to
retire it. Both now revoke, after the replacement is stored so a failure leaves a Bot with a key too
many rather than none, and loudly rather than fatally because the edit has already succeeded.

The trail recorded every mouse movement a Bot made and nothing about the Bot itself. Ten mutating
routes wrote one audit row between them, and there was no event type for any of the other nine, so it
was a missing vocabulary before it was a missing call. A Bot's endpoint is where conversation content
is sent and its callback token is a capability handed to somebody else's infrastructure; "who pointed
this Bot at that host, and when" is the first question in an incident and could not be answered.
Eight event types and eight rows now, through one helper so the payload cannot drift between them.

What is recorded is what changed, never a value: a replaced key is `keyReplaced: true` and an issued
callback token is the fact it was issued. A trail that records credentials is a credential store with
worse access control. Two of the sixteen tests exist to hold that, and one holds the direction that
would be a catastrophe: rotation must never revoke the key it has just stored.
The last two of the eight in #88 that are mine, and a bug the Activity pane had from the start.

A Chromium context was started the first time each Bot was used and kept. The only things that
dropped one were an explicit stop, a browser that had already died, and shutdown, so a deployment
where every employee has a Bot trends toward one resident browser per employee in a single container
at a few hundred MB each, until it is killed for memory and relaunches its way back to the same
state. There is now a cap and an idle timeout. A cap as well as a timeout because the failure is
concurrent breadth rather than age: fifty people using their Bots inside the same minute are fifty
live browsers and none of them are idle. Closing one costs a relaunch and nothing else, since the
profile is on disk. The per-Bot session map had the same shape and is bounded the same way.

The sidebar asked for every channel this person has on every render, one row per channel-agent pair.
It is a keyset page now. The page is chosen over channels and the agents joined to it afterwards,
which is the whole subtlety: a limit on rows would cut a two-Bot channel in half and its second Bot
would arrive on the next page as a separate entry with the same id. The socket that patches the list
live had to learn about pages with it.

The Activity pane defaulted to the screen even when the Bot had never opened a page. The screen
belongs to the computer rather than to the conversation, so a Bot that spent a whole conversation in
a terminal showed somebody else's order form from an hour ago, captioned as its own live screen. The
pane now opens on whichever surface the Bot is actually using, and the caption says whose page it is
only once this Bot has opened one.

The eviction decision is a pure function so it can be tested at all: createProfiles launches a real
Chromium, so a test that went through it would be testing Playwright.
aimock is ours and it is the org's deterministic backend for exactly this, which is the argument for
using it here: it tracks these protocols as they move, so OpenBot finds out about a drift in the same
week as everything else that depends on it rather than in a customer's integration.

`mcp.ts` is the one door in this deployment that speaks MCP to somebody else's server, and nothing
tested it. What it does with a reply is not a detail: the text lands in a model's context, the 20k cap
decides how much of a context window somebody else's server may spend, and `isError` decides whether
a Bot is told a tool failed or told nothing at all. A stubbed fetch would only prove we parse the
string we wrote, and every assertion would still pass if MCP changed underneath us or if we had
misread it in the first place. Nine cases against a real MCP server now cover the listing, the
arguments crossing the wire, visible truncation, a non-text part being named rather than dropped, a
server reporting its own failure, and a server that is not there.

Four more AG-UI cases, for the shapes a real agent turns up in rather than the happy one: a run that
starts and never finishes, an agent reporting its own error, a tool call in the stream, and nothing
listening. All four decide what somebody sees when they register a colleague's agent, which decides
whether they go looking at their service or at ours.

Two things worth knowing when writing fixtures, both found here: aimock will serve an MCP tool with no
`inputSchema`, and the official SDK then rejects the entire listing rather than that one tool, so a
server looks completely broken. And it will serve image content whose data is not valid base64, which
fails the call rather than the assertion.

Offline, no key, no spend, same answer every run, so all of it belongs in CI.
A regression I let through when I approved #37, found by driving the screen rather than by reading
the diff. Every test of both features passed.

`/api/computers/policy` is on the same router as the acting routes, which are all `/:botId/...`. The
bot-access middleware matches `/:botId/*`, and Hono matches `/*` against zero segments, so `/policy`
arrived as a Bot called "policy". `canUseBot` correctly answered that there is no such Bot, and the
whole surface for writing boundary rules returned 404 to everybody including an administrator, with a
message about a Bot.

The deployment-wide paths are named rather than inferred from the segment count, so a second one
added later has to think about this line. Four tests, three of which fail without the fix, and one
that holds the part that matters: the exemption is from the bot-access check and not from
`requireAdmin`, because letting a plain user rewrite the boundary would be a far worse bug than the
one being fixed.
CI cannot import `profiles.ts`: it pulls Playwright in at module scope, and the browsers are not
installed for that package there. This is the reason `authorisation.ts` and `bot-id.ts` already sit in
files of their own, and the eviction policy belongs in one for the same reason.

Caught by the `tests` job rather than locally, where Playwright happens to be installed.
@davidmckayv
davidmckayv force-pushed the fix/issue-88-shared-state branch from ea03447 to 5625f72 Compare August 21, 2026 17:36
@davidmckayv
davidmckayv merged commit 61611fe into main Aug 21, 2026
6 checks passed
@davidmckayv
davidmckayv deleted the fix/issue-88-shared-state branch August 21, 2026 17:54
davidmckayv added a commit that referenced this pull request Aug 21, 2026
Two improvements from Nathan Tarbert's #94, which arrived five minutes after #90 merged and was
better than it in these two places.

A NOTIFY reaches whoever is listening at the time. A replica that was restarting, or whose connection
had dropped, is not, so it missed the announcement and went on enforcing the rules it read at boot
until something restarted it. That is the original bug wearing a smaller hat and worse for being
intermittent: the fleet disagrees with itself and nothing says so. The handler is now passed as
`onlisten` as well, so the row is re-read whenever the driver establishes or re-establishes the
subscription, which is exactly when a notification could have been missed.

The announcement now goes inside the write transaction, so it is delivered on commit. A write that
rolls back announces nothing, and there is no window where the row has changed and the other servers
have not been told. Same shape as channel activity, which is what should have been copied.

Two tests. The reconnect one fails without `onlisten`.

The changelog also picks up the last day of merges, which had run ahead of it.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

State the deployment has to share is held per process, and data that grows with use has nothing bounding it

1 participant