feat(ai): add the ai resource with an OpenAI-compatible gateway

[ItamarZand88]

Summary

Adds an ai resource that gives a workload an OpenAI-compatible endpoint for model inference, in two modes. By default it's keyless: for the host cloud's own models — Bedrock on AWS, Vertex on GCP, Foundry on Azure — a loopback gateway signs each request with the workload's ambient cloud identity, so there are no model API keys to store. A workload can instead bring its own provider key (e.g. OpenAI), in which case the client talks to that provider directly with the vault-resolved key and the gateway isn't involved.

What happens on the keyless path, when a workload declares an ai binding for one of the host cloud's models:

1. A loopback AI gateway starts next to the app and its URL is placed in the environment; the app points a normal OpenAI-compatible client at it.
2. The gateway translates each OpenAI-style request into the target cloud's native model call — Bedrock InvokeModel, Vertex rawPredict, or the Foundry Anthropic endpoint — and signs it with the workload's ambient cloud credentials. ← the heart of the change
3. The response, including streamed tokens, is translated back to the OpenAI/Anthropic shape and returned to the app.

For a cloud's own models, this changes AI access from managing model API keys to using the cloud account the workload already runs in.

What I did

• Add a gateway engine (Rust) that translates OpenAI-shaped requests to each cloud's model API and signs them with ambient credentials.
• Add the ai resource to the SDK, plus the controllers and per-cloud setup that grant a workload access to that cloud's models.
• Support a bring-your-own-key mode: the same ai binding can point at an external provider with the workload's own key (vault-resolved, redacted in logs), which bypasses the gateway and calls the provider directly.
• Scope the ambient model-invocation grant to inference only: no deployment or data-plane writes.
• Ship the gateway as one standalone binary everywhere: containers run it as their entrypoint, the SDK spawns it on first use and reads back the URL it prints, and compiled single-file Workers embed the binary and extract and spawn it at runtime.
• Make getAvailableModels report real availability: the gateway probes each catalog model with a one-token request under the workload's own inference credential and lists only what the cloud actually has enabled (a 429 counts as enabled but rate-limited; 400/401/403/404 drop the model). Each entry carries provider and displayName so an app can build a model picker, and the example README documents each model's one-time enablement step per cloud.
• Type the client surface: chat.completions.create and responses.create return OpenAI's own result types (a types-only dev dependency), so callers read .choices[0].message.content with no cast.
• Add an ai-quickstart-ts example built around "list the available models, then pick one".

Files touched

• crates/alien-ai-gateway — the gateway engine (router, model catalog, availability probe, per-cloud translation + signing) and the alien-ai-gateway launcher binary.
• packages/ai-gateway — the TypeScript wrapper: resolves and spawns the gateway binary, and the ambient vs BYO-key connection resolution.
• crates/alien-core, packages/core, packages/sdk — the ai resource type, the model catalog with per-model activation metadata, and the SDK surface (plus embedding the gateway binary into compiled Workers via packages/package-layout and crates/alien-build).
• crates/alien-infra — the AI controllers and runtime wiring.
• crates/alien-terraform + crates/alien-cloudformation — the per-cloud setup emitters and their snapshots.
• crates/alien-permissions/permission-sets/ai/* — the ai/invoke grant (plus heartbeat/management/provision).
• examples/ai-quickstart-ts — a runnable example.

How I tested

• Manually: From a deployed test worker, real Claude requests went through the gateway with no API keys configured — Bedrock (AWS) and the Foundry Anthropic endpoint (Azure) both returned completions, including streaming. I also ran the gateway on my machine against real Bedrock and hit /v1/models: 36 of the 42 AWS catalog models came back listed and enriched, and the 6 dropped were Claude ids that account/region cannot serve.
• Compiled Worker: bun build --compile of a Worker that calls ai() — the embedded gateway binary is extracted and spawned from inside the single-file binary and serves a loopback URL (the package-layout compile smoke).
• Unit tests: cargo nextest across the workspace (CI-fast filter), including the availability-filter suite (mock upstream: an enabled model is kept and enriched, a 403 model is dropped, a 500 keeps the model and re-probes on the next call, a second call is served from cache); SDK vitest + biome; terraform validate on the AI setup-emitter output and the CloudFormation snapshot suite; generated-schema drift check clean. The live Bedrock test reads the shared test env and runs in the credentialed job.
• E2E (ran green on real Bedrock): the comprehensive TypeScript app lists every model getAvailableModels returns and invokes each one, failing if a listed model is not invocable. That run passed after it caught a real filter bug: Bedrock answers 400 "The provided model identifier is invalid" for a model the account cannot address, and I had been counting 400 as available, so it now drops the model (400 joins 401/403/404) and the list only ever contains invocable models.
• Anything I couldn't test: Vertex (GCP) live inference is pending a model-garden entitlement on the test project — unrelated to this code; the GCP emitter path is covered by snapshots + terraform validate.

One non-AI fix rides along

The cloud e2e for this PR overflowed its test thread's stack during deploy. It is a pre-existing issue, independent of the AI code: the deployment reconcile polls a deep async chain (executor to controllers to the cloud SDKs' tower/hyper/rustls futures) that needs about 3MB in a debug build, over tokio's 2MB default thread stack. Release binaries fit 2MB and are unaffected. RUST_MIN_STACK=8MB for dev/test threads via .cargo/config.toml fixes it, matching the headroom alien-worker-runtime already gives its runtime. It rides along here because this PR's own e2e could not run without it.

I also ran a security review on the diff. What it checked:

• Over-scoped model grant. ai/invoke grants inference only — AWS bedrock:InvokeModel* (+ mantle CreateInference scoped to project/default, not project/*); a GCP custom role of just aiplatform.endpoints.predict/explain (not roles/aiplatform.user, which carries deploy + dataset-export); Azure "Cognitive Services OpenAI User" (data-plane only). No deployment writes or data reads. crates/alien-permissions/permission-sets/ai/invoke.jsonc.
• Availability probe surface. The /v1/models probe sends a one-token inference request signed with that same inference-only grant; it adds no control-plane permission, cannot enable or subscribe anything, and its logs carry only the model id and status, never request or response bodies. crates/alien-ai-gateway/src/availability.rs.
• Credential exposure. On the keyless path the gateway signs with the workload's ambient cloud identity — no key is stored. On the BYO-key path the provider key is a vault-resolved secret: its Debug impl redacts it (crates/alien-core/src/bindings/ai.rs) and it's resolved into the worker environment, not logged or held in control-plane state. crates/alien-ai-gateway/src/config.rs.
• Off-host reachability. The gateway binds the loopback interface and hands the app its URL out of band, so it isn't reachable from other hosts. crates/alien-ai-gateway/src/lib.rs.
• BYO-key providers. An external (bring-your-own-key) provider is explicitly not served by the ambient-credential gateway path. crates/alien-ai-gateway/src/config.rs.

Nothing turned up.

Continues #178 (same commits, rebased on main). That PR closed automatically when this branch was renamed.

[ItamarZand88]

@greptile-apps

[greptile-apps]

Greptile Summary

The follow-up updates the AI gateway implementation and closes the previously reported release and request-buffering gaps.

• Adds a finite 32 MiB request-body limit with a structured 413 response.
• Builds, packages, smoke-tests, and publishes the gateway wrapper and platform binaries during immutable npm dev releases.
• Separates stable gateway qualification from publication, skips publication during dry runs, and makes GitHub release creation depend on successful gateway publication.

Confidence Score: 5/5

The PR appears safe to merge with respect to all previously reported Greptile findings.

No blocking failure remains: request buffering is bounded, dev releases include the wrapper and executable platform packages, stable qualification and publication are separated correctly, dry runs do not publish gateway packages, and final release creation waits for gateway publication.

Important Files Changed

Filename	Overview
crates/alien-ai-gateway/src/router/mod.rs	Adds and tests a finite 32 MiB body limit, resolving the previously reported unbounded request buffering.
.github/workflows/publish-npm-dev.yml	Builds, stages, packs, smoke-installs, and publishes the gateway wrapper and all supported platform packages in dependency order.
.github/workflows/release.yml	Qualifies gateway packages without npm publication, publishes only qualified artifacts in non-dry-run publish mode, and gates release creation on successful gateway publication.
packages/ai-gateway/package.json	Defines the gateway wrapper package and platform-specific optional dependency surface used by the updated publication workflows.
packages/sdk/package.json	Adds the AI gateway wrapper dependency consumed by the SDK.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  Build["Build platform gateway binaries"] --> Qualify["Pack and upload qualified gateway artifacts"]
  Qualify --> Publish["Publish platform packages, then wrapper"]
  Publish --> Release["Create GitHub release"]

Loading

_{Reviews (11): Last reviewed commit: "fix(ci): stop building the ai-gateway wr..." | Re-trigger Greptile}

[ItamarZand88]

@greptile-apps

[ItamarZand88]

@greptile-apps

[ItamarZand88]

@greptile-apps

[ItamarZand88]

@greptile-apps

[ItamarZand88]

@greptile-apps

[ItamarZand88]

@greptile-apps

[ItamarZand88]

@greptile-apps

[ItamarZand88]

@greptile-apps

[ItamarZand88]

@greptile-apps