Skip to content

EdgeZero full-migration umbrella design spec#839

Draft
aram356 wants to merge 36 commits into
mainfrom
worktree-edgezero-migration-spec
Draft

EdgeZero full-migration umbrella design spec#839
aram356 wants to merge 36 commits into
mainfrom
worktree-edgezero-migration-spec

Conversation

@aram356

@aram356 aram356 commented Jul 2, 2026

Copy link
Copy Markdown
Collaborator

Summary

Adds the umbrella design spec for moving Trusted Server completely onto EdgeZero primitives: config push, KV, secret store, config injection without an embedded trusted-server.toml, extractor-based handlers, and deletion of every pre-EdgeZero workaround.

This is a docs-only PR (one spec file). It defines the end-state, a verified current-state gap analysis, and an ordered set of phases — each of which will get its own implementation plan/PR.

Spec: docs/superpowers/specs/2026-07-02-edgezero-full-migration-design.md

Phase map (foundation-first)

  • Phase 0 — EdgeZero prerequisites (State<T> extractor + nested/array #[secret]). Upstream edgezero repo, tracked separately by its own PR.
  • Phase 1 — Stores onto EdgeZero StoreRegistry (delete bespoke PlatformConfigStore/PlatformSecretStore, RuntimeServices, management_api.rs, settings_data.rs chunk resolver). No upstream dependency — startable immediately.
  • Phase 2 — Finish config injection: store-load config on Cloudflare + Spin (kill both include_str!), drop from_toml_and_env + the config crate.
  • Phase 3 — Full secret externalization: #[secret] on Settings, move inline secrets to the secret store, delete Redacted<T>. Depends on Phase 0 nested #[secret].
  • Phase 4 — Handlers → #[action] extractors using the upstreamed State<T>; delete per-adapter handler shims.
  • Phase 5 — Delete the legacy Fastly path (legacy_main/compat.rs/rollout flags). Gated on 100% EdgeZero rollout (Fastly entry point switch (dual-path with flag) #495).

Key decisions

  • D1 — Keep load-once Arc<Settings> via State<Settings> rather than the per-request AppConfig<C> extractor (avoids re-parsing the whole Settings on every request).
  • D2 — Integration proxy nested router migration deferred to an optional Phase 4b.
  • D3 — Secrets resolve at startup, not per request.
  • D4 — Collapse the TrustedServerAppConfig wrapper onto Settings.

Status

Draft — for review of scope, phase ordering, and D1 before per-phase implementation plans begin.

Defines the end-state, current-state gap analysis, and an ordered set of
phases (stores, config injection, secret externalization, extractors,
legacy-path removal) for moving Trusted Server completely onto EdgeZero.
Phase 0 (State<T> extractor + nested #[secret]) is an upstream edgezero
prerequisite tracked separately.
@aram356 aram356 self-assigned this Jul 2, 2026
aram356 added 27 commits July 2, 2026 15:45
Clarify that platform/mod.rs and types.rs are edited (KV re-export and a
shrinking RuntimeServices remain), not deleted, in Phase 1.
…tore id

- P0-C: Fastly bypasses run_app (multi-value Set-Cookie, logger reinit,
  JA4/H2 capture) - add EdgeZero dispatch prereq or documented exception
- P-BOOT: specify boot-time config/secret store access for Cloudflare/Spin
  (build_state runs before request context; registry is per-request)
- D5: unify the split app-config store id (app_config vs trusted_server_config)
- Promote the secret inventory to a spec artifact; array + optional secrets
  confirmed present, so edgezero #305 must ship ArrayEach + Option<String>
- Phase 4 acceptance: per-adapter route parity (EC routes are Fastly-only)
- Phase 5: expand deletion ledger (route_tests, viceroy config, fastly.toml, runbook)
- Scope the include_str! ban to adapter/runtime app-config only
…iation (D5)

- D6: EdgeZero stores are read-only, but KeyRotationManager writes/deletes
  config+secrets at runtime for /_ts/admin/keys/*. management_api.rs cannot
  be deleted in Phase 1 unconditionally; gate on a keep/move-to-ops/upstream
  decision (R10)
- D5 expanded: reconcile ALL runtime store ids (app_config, secrets, JWKS,
  DataDome ts_secrets, S3, fixtures) with edgezero.toml or strict lookup fails
- Fastly needs explicit registry injection into its custom oneshot path
- Phase 1 plan starts with a store-capability inventory, not deletions
Task 1 is a decision gate (store-id inventory + D5/D6) per review, not
deletions. Read-path migration and Fastly custom-dispatch registry injection
are fully specified; management_api.rs deletion is gated on the D6 decision.
- Lock plan to D6-a; D6-b/c stop after Task 1 (separate plan)
- Kind-aware store-id reconciliation (kv/config/secrets), incl. ec_store as KV
- Composite read/write store bridge with a write-delegation test
- New task: migrate Fastly/Axum BOOT config read to EdgeZero before deleting impls
- Local Fastly registry builders (EdgeZero builders are pub(crate)); R11
- Concrete named tests, files, routes, fixtures (no <test_name> placeholders)
- Sync spec D5 (kind-aware), Phase 1 boot-read, R11
…e tests

- Task 3: composite holds the whole ConfigRegistry/SecretRegistry and resolves
  named(store_name) per read (multi-store); strict unknown-id error; test uses 2 ids
- Task 1/spec: correct KV ids to ec_store + consent_store (creative/counter/opid
  are fastly.toml platform stores, not Settings logical ids)
- Task 4: core-level loader test with InMemoryConfigStore + ConfigStoreHandle::new
- Task 5: add non-default config-id (JWKS) + non-default secret-id (DataDome, S3) tests
- Task 6: local EnvConfig runtime-dictionary reader (EdgeZero helper private); R12
- Task 8: tests build registries with >=2 ids and assert unknown id errors strictly
- Spec D5/Phase 1: kind-aware incl [stores.kv]; R9 mentions ec_identity_store; R12
- D7: runtime app-config is config-store-only; NO runtime env vars. ts config
  push reads env at push time and bakes resolved values into the blob. Stores
  open by logical id (name == id); no runtime EDGEZERO__STORES__*__NAME read.
- Task 6: drop local_env_config entirely (D7) - open stores by logical id;
  resolves the fastly::ConfigStore-has-no-iter and private-helper findings (R12)
- creative_store IS a Settings KV id (deprecated); include it, exclude
  counter_store/opid_store (Fastly-adapter constants)
- Task 2/spec: tighten kind-aware KV inventory (ec_store, consent_store, creative_store)
- Task 4: run the actual core test name; Task 5: one filter per cargo test invocation
Switch the six edgezero git deps from branch main to
worktree-state-nested-secrets-spec-review (stackpop/edgezero#306, stacked on
#300) to pick up the Phase 0 State<T> extractor work. cargo check-axum passes.
- DataDome IP-CIDR config store (datadome-ip-bypass) added to store inventory
- Explicit app-config decision: store id trusted_server_config, key app_config;
  rename DEFAULT_CONFIG_STORE_ID + repoint request_signing.config_store_id
- Task 3: ConfigRegistry::named returns ConfigStoreBinding -> use binding.handle.get;
  map EdgeZero Ok(None)/Err to PlatformError
- Task 6: exact builder signatures (KV Result<Option<..>,FastlyError>, config/secret
  Option<..>) + missing-store/open-failure policy
- Task 5: DataDome secret read tested on a protected NON-integration route
- StoreName reconciled to logical read id (D7); doc + call-site audit step
- Task 4: Axum boot reads .edgezero/local-config-trusted_server_config.json, no env override
Blockers:
- Hooks::stores() is not overridden on any adapter (empty StoresMetadata);
  add Task 2 Step 5 to wire stores() from edgezero.toml on all four adapters
- Axum uses routes()+AxumDevServer, not run_app; Task 5 Step 0 switches Axum
  to edgezero_adapter_axum::run_app::<TrustedServerApp>()
- request-signing reads use jwks_store/signing_keys but writes use
  config_store_id/secret_store_id; fix example/fixtures to jwks_store/signing_keys
  (NOT app_config/secrets); only the app-config store renames to trusted_server_config
- PlatformConfigStore/SecretStore mix read+write; Task 3 Step 0 splits write-only
  PlatformConfigWriter/PlatformSecretWriter so Task 8 can drop reads and compile
High/Medium:
- D7 softened: EdgeZero builders read EnvConfig but fall back to logical id; we set none
- Task 2 Step 6: declare stores in fastly.toml/wrangler.toml/spin.toml/Axum local files
- Task 2 test parameterized over example + fixture + all-store-refs config
- CI gate adds cargo check-cloudflare + check-spin (wasm surfaces)
- Fix stale spec D5/R9 wording
- Axum: exact path = keep AxumDevServer::with_config + .with_{config,kv,secret}_registry
  (dev_server::run_app drops custom PORT/axum.toml); add adapter-axum registry builders
- Task 5: extract whole registry via ctx.request().extensions().get::<ConfigRegistry>()
  (RequestContext has no whole-registry getter; per-id accessors would wire only default)
- Task 2: enumerate exact file paths; replace brace-glob git add that would hit
  non-existent adapter manifest paths
- Spec: state requirement that Fastly management id == runtime logical id for
  jwks_store/signing_keys (or supply a mapping, out of Phase 1 scope)
- Task 3: composite writer test asserts (StoreId, key, value) forwarding (D6-a risk)
- Minor: expect("should ...") convention; Task 7 local verify adds check-cloudflare/spin
- Fastly build_per_request_services still read FastlyPlatformConfigStore directly,
  so injected registries were unused; add Task 6 Step 4b to build RuntimeServices
  from the composite via registries in extensions
- App-config key == store id (trusted_server_config): default_config_key falls back
  to id and D7 forbids the __KEY env; set CONFIG_BLOB_KEY=trusted_server_config,
  retire the app_config key (no --key/env needed)
- Task 5: rename heading to 'preserve AxumDevServer::with_config + add registries';
  list adapter-axum/src/registries.rs (Step 0 already kept with_config)
- Task 2 Step 6: exact per-adapter manifest mappings (CF config=KV binding,
  secrets=flat Worker secrets via wrangler secret put; Spin config/kv=KV labels)
- Name S3 secret store id s3-auth explicitly in D5/Task 2
- Fix all-store-refs.toml include_str path (testdata/, not ../testdata/)
- Writer traits Send+Sync; expect_err 'should ...' convention
…y file)

- Add config_payload.rs (CONFIG_BLOB_KEY) to Task 2 files + git add (was missing)
- Task 2 Step 6b: rename app_config in generate-viceroy-config.rs (+ its test),
  tests/common/config.rs, tests/environments/axum.rs env var (they run in adapter
  suites and break under the store/key rename)
- Task 2 Step 6: Cloudflare/Spin secrets are FLAT (ignore store_name) -> provision
  the concrete secret KEYS the code reads (signing KIDs, DataDome key name,
  S3 access_key_id/secret_access_key/session token), not the store id
- Task 6: add app.rs to files; test passes only after Step 4b (not 3-4)
- Task 1 output table: drop EDGEZERO__STORES__*__NAME column (D7 -> platform
  resource per adapter, no env mapping)
The prior review's Critical 'APIs do not exist' findings were against a stale
cargo-cache checkout (ce6bcf7, edgezero #253 'Add store support for Spin') that
predates the registry refactor. Our actual pin is 6ebc29a5 (PR #306), which HAS
StoreRegistry/StoresMetadata/Hooks::stores()/dispatch_with_registries/
with_*_registry/[stores.*] ids+default/CF-KV-namespace/Axum-local-file. Added a
'Pinned dependency (verified)' note recording this to prevent re-litigation.

Valid fixes:
- Cloudflare app.rs settings_from_cloudflare_config_json reads literal
  value.get("app_config") from the side-channel; change to CONFIG_BLOB_KEY so
  the key rename doesn't break Cloudflare boot (Phase 2 does the store migration)
- Viceroy generator already emits a trusted_server_config store for rollout
  flags; the app_config->trusted_server_config rename must MERGE the blob into
  that table, not emit a duplicate table
User chose full convergence on the canonical app-demo wiring. Reshape the
umbrella spec:
- End-state: every adapter is one-line run_app::<App>; App from app! macro;
  handlers #[action] + State<Arc<AppState>>; no TS-local registry/dispatch wiring
- Two NEW required upstream edgezero prerequisites (verified gaps in pinned 6ebc29a):
  - P0-C: header-preserving Fastly run_app dispatch + pre-dispatch extension hook
    (Set-Cookie/JA4/logger) - permanent-exception fallback removed
  - P0-D: macro/run_app app-state injection (with_state is builder-only; app!
    router + run_app never call it) - a Hooks state hook run_app inserts per request
- D1 retained (load-once Settings) but injected via P0-D state hook, not with_state
- Phase 4 rewritten as full convergence (app! macro, run_app, extractors, catch-all
  fallback for integration/publisher dispatch)
- Sequencing note (R14): Phase 1 adapter-registry builders (Tasks 5-6) are throwaway
  under run_app; recommend landing P0-C/P0-D early to skip them
- Risks R7 (P0-C required), R13 (P0-D), R14 (sequencing)
…ate)

Implementation-ready spec to hand to the edgezero dev, grounded in pinned 6ebc29a:
- P0-C: C1 append_header for multi-value Set-Cookie (response.rs:29 set_header bug),
  C2 Hooks::owns_logging() opt-out, C3 run_app_with_request_extensions pre-dispatch
  hook for JA4/H2/client_info from the raw fastly::Request
- P0-D: Hooks::app_state() injected per request (symmetric with registries) +
  app! macro state= argument; note P0-D is avoidable via hand-built routes()+with_state
… gaps)

- BLOCKER: Task 8 made Fastly read impls write-only, but legacy_main (live until
  Phase 5) reads through FastlyPlatformConfigStore via build_runtime_services.
  Fix: only Axum/CF/Spin go write-only in Phase 1; Fastly read impls stay until
  Phase 5 deletes legacy_main (spec Phase 1/5 boundary + ledger updated)
- stores() drift: add Step 5b test asserting stores_metadata() and each adapter's
  Hooks::stores() equal edgezero.toml (registries build from stores(), not the
  toml that Step 1 validates)
- KV multi-id: CF/Spin build RuntimeServices from kv_store_default() only, so
  non-default KV ids (consent_store/ec_store) don't resolve; Step 2c decides
  (registry-backed KV selector vs documented defer + narrowed acceptance)
- cloudflare.toml still uses legacy [stores.*].name schema the pinned manifest
  rejects; Task 2 reconciles/deletes it
- Add Cloudflare/Spin non-default config/secret tests (Step 2b)
- KV multi-id DECIDED: registry-backed KV now (not defer). All four adapters
  resolve named KV via KvRegistry from extensions; remove Fastly consent-store
  special-casing; per-adapter non-default-KV test. Spec acceptance updated.
- Spec: fix stale Phase 1 'delete 4x read impls' bullet to match the corrected
  Fastly-stays-until-Phase-5 deletion note (no self-contradiction)
- Spec: remove duplicate P-BOOT paragraph (kept the full-convergence note)
- app_config rename: add tests/environments/cloudflare.rs literals + wrangler.toml
  placeholder JSON/comment to Task 2 Step 6b + git add
- KV named resolution is a CORE change, not adapter-only: RuntimeServices only
  has kv_store() single handle and consent (publisher.rs:626) drops the store id
  via .map(|_| services.kv_store()). Step 2c now adds RuntimeServices::kv_store_named
  + kv_registry field, updates consent call sites to pass consent_store, removes
  Fastly special-casing, and lists core files (types.rs, publisher.rs)
- Step 2d: test-support helper to build registry-populated RequestContexts +
  migrate existing direct-context tests (strict registries break them otherwise)
- ec_store scope narrowed: Phase 1 only validates it declares/resolves to a
  KvHandle; NOT EC identity-graph wiring on non-Fastly (separate larger effort)
- KvHandle/consent type path DECIDED: KvRegistry::named yields KvHandle (a wrapper,
  not a PlatformKvStore impl), and ConsentPipelineInput.kv_store is
  Option<&dyn PlatformKvStore>. Migrate the consent KV surface to KvHandle:
  RuntimeServices::kv_handle_named(id) -> Option<KvHandle> (mirrors existing
  kv_handle()); ConsentPipelineInput.kv_store -> Option<KvHandle>; consent
  persistence fns take &KvHandle. Files: consent/mod.rs, storage/kv_store.rs added.
- Task 5 Step 3/4 now run all four adapters + wasm checks and commit core + Fastly
  changes (task touches types.rs/publisher.rs/Fastly), per the all-four-green rule
…ral test)

- Fastly named-KV sequencing: Task 5 wired kv_registry + removed consent
  special-casing on Fastly, but Fastly injects registries only in Task 6.
  Move Fastly named-KV + build_per_request_services composite +
  runtime_services_for_consent_route removal into Task 6 Step 4b; Task 5 does
  core + Axum/CF/Spin and leaves Fastly compiling (kv_handle_named returns None)
- Behavioral consent test: prove consent_store loads/saves/deletes through the
  NON-default store and leaves the default untouched (not just handle distinctness)
- Convert existing core consent tests (consent/mod.rs kv_store: Some(&store) at
  ~1450/1481/1492) from &dyn PlatformKvStore to KvHandle
- Fix understated KV summaries: spec gap table + plan goal now say default KV is
  EdgeZero but named/consent-store KV selection is not
… tests)

- Interim Fastly regression: Task 5 flipped consent to kv_handle_named but Fastly
  has no KvRegistry until Task 6 -> consent would silently skip on Fastly. Fix:
  Task 5 migrates the consent TYPE but keeps the call site on kv_handle() (default,
  = Fastly's swapped consent store); the named-lookup FLIP + behavioral test are
  atomic in Task 6 Step 5b once all four inject a KvRegistry
- Absent-registry policy: builders stay infallible (return RuntimeServices); build
  a composite over an EMPTY registry so reads error, not silently fall back
- Task 6 Step 5c: Fastly named-KV/consent route test (guards the special-case removal)
- Task 3 Step 1b: explicit CompositeSecretStore test (named id, unknown-id error,
  create/delete StoreId-preserving delegation) - flat CF/Spin secret namespaces
  mean route tests don't prove store-id binding
- Task 5 hygiene: no longer edits/commits Fastly (moved to Task 6)
aram356 added 8 commits July 4, 2026 22:34
…+ commit/filter)

- BLOCKER: absent-registry policy said 'composite over empty StoreRegistry', but
  StoreRegistry fields are private and from_parts returns None on empty by_id.
  Fix: Composite{Config,Secret}Store hold Option<Registry> (reader = the Option
  from extensions.get().cloned()); error on None. No empty-registry construction.
- Fastly secret builder: does NOT open per id. One SecretHandle over FastlySecretStore
  (stateless; opens per get_bytes) + BoundSecretStore per id. Tightened Task 6 builders
  per kind (KV opens/Result, config handle, secret bound-no-open)
- Task 6 commit: add consent/mod.rs (behavioral test file)
- Task 3 test command: single shared-prefix filter 'composite_' (not two filters)
…k 5 scope)

- Task 3 test snippets: pass Some(reader) to Composite{Config,Secret}Store::new
  (constructor takes Option<Registry>, snippets passed bare registries)
- Fastly KV builder: FastlyKvStore::open(id) returns Result<FastlyKvStore, KvError>,
  not KvHandle; wrap KvHandle::new(Arc::new(store))
- Task 5 stale scope: publisher.rs call site is interim kv_handle() (flip is Task 6);
  remove Fastly platform.rs/app.rs from Task 5 files (Task 6 owns them)
- Task 3 Step 4 pass command: cargo test-fastly composite_ (runs both tests)
…Meta

P0 (State<T> + nested/array #[secret]) landed on the #306 branch. The nested-secret
work reshapes AppConfigMeta: const SECRET_FIELDS -> fn secret_fields() -> Vec<SecretField>.
Update TrustedServerAppConfig's impl accordingly (still empty until Phase 3 secret
externalization). Verified: cargo check green on core + all four adapters + cli.
…s, D7)

- Spec: core PlatformConfigStore/SecretStore read+write traits are NOT deleted in
  Phase 1 (composite implements them; RuntimeServices exposes them) - only removed
  in Phase 4. Phase 1 EXTRACTS the write-only sub-traits. Fixed deletion note + ledger.
- Writer ordering: Task 3 Step 0 now also impls PlatformConfigWriter/SecretWriter on
  the existing stores (forwarding to their put/create/delete), so Task 5 has writers
  before Task 8 converts anything. Explicit per-type impls (not blanket, coherence).
- Task 2: add Viceroy integration template + Spin runtime-config.toml (KV labels need
  a backend); both added to files/commit
- D7 consistency: spec Fastly registry text drops EnvConfig (open by logical id);
  Task 6 config builder default_key = logical id
- Task 3 verify: compile axum/cf/spin too (core trait split)
- Task 5 Step 2d: Fastly direct-context test migration -> Task 6 (Task 5 excludes Fastly)
- Writer impls can't live in core for adapter-owned types (orphan rule / dep cycle);
  blanket impl would coherence-conflict with Task 8. Task 3 Step 0 now defines the
  write-only traits ONLY (core); each adapter impls the writer for its own store as
  the first step of its composite wiring (Task 5 Axum/CF/Spin; Task 6 Fastly)
- fastly.toml holds only [local_server] Viceroy resources; production store links
  are an operator/provisioning step, not a manifest edit - split the Task 2 bullet
- Task 3 verify: cargo check-cloudflare/check-spin (wasm surfaces), not native only
- Task 2 Files: add runtime-config.toml + viceroy-template.toml (step/commit had them)
- Task 5 Files summary: direct-context test migration is axum/cf/spin (Fastly = Task 6)
- HIGH: consent flip must FAIL CLOSED. Fastly returns 503 today when consent_store
  is configured but unresolved (tests dispatch_auction_...returns_503 /
  edgezero_missing_consent_store_breaks_only_consent_routes). .and_then(kv_handle_named)
  silently -> None (regression). Fix: configured-but-unresolved -> TrustedServerError::KvStore
  (503 on consent routes); unconfigured -> None; integration routes unaffected. Also fix
  revocation delete (publisher.rs:885) to use the named handle, not default KV. Add
  a configured-but-unresolved-errors test.
- HIGH: pin note updated 6ebc29a5 -> d8f71a4a (re-verified registry APIs present;
  AppConfigMeta const->fn compat noted)
- Task 4 + Task 6: run all four + check-cloudflare/check-spin (core surfaces change)
- Task 5: bounded Axum smoke (timeout/bg+curl), not blocking cargo run
- Task 2 Files: add cloudflare.toml; Task 3 Files: add platform/types.rs
…le guard)

- HIGH: auction fail-closed gap. Fastly auction route (app.rs:584) got 503-on-missing
  -consent-store from runtime_services_for_consent_route, removed in Step 4b, but
  Step 5b only fixed publisher paths. handle_auction builds KvIdentityGraph over
  consent KV. Fix: shared core resolve_consent_kv (fail-closed) used by BOTH publisher
  and handle_auction (before KvIdentityGraph); the Fastly 503 tests are mandatory gates.
- Task 6 Files: add platform.rs (writer impls), consent/mod.rs, publisher.rs,
  auction/endpoints.rs, core consent/auction tests, Fastly direct-context test migration
- Lockfile guard: branch dep is mutable and upstream advanced past d8f71a4a; execute
  --locked, no cargo update unless API re-reviewed
- Task 8 commit msg: 'Retire NON-Fastly per-adapter read impls' (Fastly stays to Phase 5)
- Auction guard wording corrected: handle_auction does NOT build KvIdentityGraph over
  consent KV (consent = ec_context.consent(); KvIdentityGraph is the EC-graph input).
  resolve_consent_kv(...)? is a pure fail-closed guard, no EC-graph semantics change.
- Mixed edition: core/fastly 2021, axum 2024 (plan said global 2024). Follow each crate.
- --locked on every cargo command (mutable branch dep, upstream advanced past d8f71a4a)
- Preflight: cargo fetch/check --locked + confirm resolved rev d8f71a4a before Task 1
- Axum local KV: don't hand-reimplement EdgeZero's private .edgezero/kv-<slug>-<hash>.redb
  path (silent divergence); reuse/upstream a public constructor. Decision in Task 1.
- Spin: declare all kv/config labels + runtime-config backends; drop stale v_ comment;
  document spin up --runtime-config-file; secrets flat by key
- Operator runbook: create docs/internal/store-provisioning.md (mgmt-id==logical-id,
  provisioning commands) - added to files + commit
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant