Workbench budget management + other tweaks - #3948
Conversation
Soffi AI SummaryThis PR introduces a token-bucket budget management system for AI Workbenches, allowing operators to cap monthly AI spending (in dollars or tokens) so that new agent jobs cannot be spawned once the quota is exhausted until the bucket refills. Alongside this core feature, it lands several supporting improvements and fixes:
Commits
Updated: 2026-08-03 12:45 UTC |
Greptile SummaryThis PR adds token-bucket workbench budgets, pull-request difficulty and AI pricing schemas, persistent local cache metadata, and several deployment/runtime tweaks.
Confidence Score: 4/5The PR should not merge until budget admission is made atomic and restored cache entries recover when their backing tarballs are missing. Concurrent requests can spawn jobs from the same remaining budget, while a process restart during the cache sweep/flush window can restore metadata for deleted tarballs and make content fetches fail persistently. Files Needing Attention: lib/console/deployments/workbenches.ex, lib/console/deployments/local/server.ex, lib/console/deployments/local/persistent_ets.ex
|
| Filename | Overview |
|---|---|
| lib/console/deployments/workbenches.ex | Adds budget admission and accounting, but job admission is not atomic with persisted budget state. |
| lib/console/schema/workbench.ex | Defines the refillable budget embed and arithmetic; consumption can become negative after admitted work completes. |
| lib/console/deployments/local/persistent_ets.ex | Adds periodic ETS snapshots, but restoration does not validate external files referenced by table entries. |
| lib/console/deployments/local/server.ex | Restores local cache metadata and trusts restored hits even when their tarball files have been removed. |
| lib/console/schema/deployment_settings.ex | Adds validated embedded AI price sheets without an identified defect. |
| lib/console/schema/pull_request.ex | Adds an embedded pull-request difficulty shape with matching persistence migration. |
| priv/repo/migrations/20260801162842_increase_custom_stack_run_name.exs | Adds map columns required by the new Workbench budget and PullRequest difficulty embeds. |
| lib/console/deployments/global.ex | Adjusts global-service stream throttling without an identified correctness issue. |
| mix.exs | Bumps mermaid_validator to the panic-fix release. |
Comments Outside Diff (1)
-
lib/console/deployments/local/server.ex, line 48-50 (link)Restored cache hits can be stale
When the server crashes after a sweep deletes a tarball but before the updated ETS state is flushed, restart restores metadata for the deleted file and this fast path returns it without validation, causing subsequent content fetches to fail with
enoentinstead of regenerating the artifact.Knowledge Base Used: Deployments Engine
Reviews (1): Last reviewed commit: "Workbench budget management + other twea..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| |> allow(user, :read) | ||
| |> when_ok(:insert) | ||
| |> notify(:create, user) | ||
| with true <- budget_available?(wb) do |
There was a problem hiding this comment.
If two job-creation requests overlap while the remaining budget is just above min_free, both check the same unlocked workbench state and insert jobs before usage is deducted, causing work to run beyond the configured quota.
Knowledge Base Used: Schema and Repo: Console's Data Model
d647160 to
a15f953
Compare
This will allow for a token-bucket style budget mechanic within workbenches. If the monthly quota has exceeded, jobs cannot be spawned until the refill has happened. Also * add difficulty and token price sheet schemas * bump mermaid_validator with fix for rust panic in underlying lib * persist local server cache between restarts
a15f953 to
0458f31
Compare
|
plural deploy this |
![plural-copilot[bot]](https://avatars.githubusercontent.com/in/916369?s=60&v=4){kind=small}
There was a problem hiding this comment.
The requested deployment GitOps PR has been created: https://github.com/pluralsh/plrl-up-demos/pull/2499
It deploys console image tag sha-0458f31b.
|
plural deploy this, and wire up the cloud query image with the same sha tag |
There was a problem hiding this comment.
GitOps deployment PR: https://github.com/pluralsh/plrl-up-demos/pull/2500
Console and cloud-query are pinned to sha-f9b5e74.
f9b5e74 to
daa2a74
Compare
|
plural deploy this |
There was a problem hiding this comment.
Deployment PR created for this revision: https://github.com/pluralsh/plrl-up-demos/pull/2501 (pins Console to sha-daa2a74; Plural Service: mgmt/apps).
daa2a74 to
c614342
Compare
|
plural deploy this |
There was a problem hiding this comment.
Deployment GitOps PR: https://github.com/pluralsh/plrl-up-demos/pull/2502
Pins Console image tag sha-c6143427 and is annotated Plural Service: mgmt/apps.
b9f44ae to
7d29e52
Compare
|
plural deploy this |
There was a problem hiding this comment.
GitOps deployment PR
https://github.com/pluralsh/plrl-up-demos/pull/2503
Updates the console GitOps image tag to sha-7d29e52, targeting source revision 7d29e52b6cd1f2c5e871b77e07faa292b2245aac.
Plural Service: mgmt/apps
7d29e52 to
077cb28
Compare
|
plural deploy this |
There was a problem hiding this comment.
Deployment GitOps PR has been created: https://github.com/pluralsh/plrl-up-demos/pull/2504
6a62e3d to
310d128
Compare
|
plural deploy this |
There was a problem hiding this comment.
Deployment GitOps PR: https://github.com/pluralsh/plrl-up-demos/pull/2505
It deploys console image tag sha-310d128 from PR head 310d128aa02980e2bf3a5270e6e515b35afe53ef.
GitOps PR annotation: Plural Service: mgmt/apps.
310d128 to
e489f20
Compare
|
plural deploy this |
There was a problem hiding this comment.
Deployment PR created: https://github.com/pluralsh/plrl-up-demos/pull/2506 — updates console to sha-e489f20 (commit e489f20).
e489f20 to
604f9b3
Compare
|
plural deploy this |
There was a problem hiding this comment.
Requested deploy GitOps PR has been created: https://github.com/pluralsh/plrl-up-demos/pull/2507
2 participants
This will allow for a token-bucket style budget mechanic within workbenches. If the monthly quota has exceeded, jobs cannot be spawned until the refill has happened.
Also
Test Plan
Test environment: https://console.plrldemo.onplural.sh/cd/clusters/a1748282-ce8b-48ab-ae7e-326e74fce04e/services/f3f89a54-d1a7-4bc8-9152-daa07ede918d/components
Checklist
Plural Flow: console