feat: out-of-process host for existing Python CPEX plugins

.github/workflows/ci.yml

@@ -74,9 +74,55 @@ jobs:
       - uses: Swatinem/rust-cache@v2
       - name: cargo test
         # Valkey/testcontainers integration tests are `#[ignore]`d by default
-        # and run out-of-band, so this needs no Docker daemon.
+        # and run out-of-band, so this needs no Docker daemon. The Python host
+        # e2e tests are also `#[ignore]`d here — the `python-e2e` job below runs
+        # them with a real interpreter and framework checkout.
         run: cargo test --workspace
 
+  # The Python host's end-to-end tests, actually run.
+  #
+  # These are `#[ignore]`d so the workspace job reports them as ignored rather
+  # than passing bodies that never executed. This job builds the environment they
+  # need and sets CPEX_REQUIRE_PYTHON_E2E=1, under which an unmet prerequisite
+  # panics instead of skipping. So a missing interpreter or a stale worker.py
+  # fails this job loudly — it cannot go quiet and still report green.
+  python-e2e:
+    name: Python host e2e (skips fail)
+    if: github.event_name != 'pull_request' || !github.event.pull_request.draft
+    runs-on: ubuntu-latest
+    timeout-minutes: 45
+    steps:
+      - uses: actions/checkout@v7
+        with:
+          fetch-depth: 1
+      # The Python framework lives on its own branch, and PyPI's `cpex` is behind
+      # it (its worker.py predates the credential and extensions fields). The
+      # *stable* 0.1.x branch is not enough either: it carries no
+      # cpex/framework/isolated/worker.py at all. The companion branch below is
+      # the one whose worker.py both consumes `credential` and delivers
+      # `extensions=` to execute_plugin, which is what these tests gate on.
+      #
+      # This is a cross-branch dependency: when the Python side merges, retarget
+      # this ref. Until then a wrong ref fails loudly (skips are errors here)
+      # rather than silently skipping — which is the whole point of this job.
+      - name: Checkout the cpex Python framework
+        uses: actions/checkout@v7
+        with:
+          ref: "feat/python_plugin_compat_0.1.x"
+          path: cpex-python
+          fetch-depth: 1
+      - name: Install Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+      - name: Install Rust 1.96.0
+        uses: dtolnay/rust-toolchain@1.96.0
+      - uses: Swatinem/rust-cache@v2
+      - name: make test-python-e2e
+        env:
+          CPEX_PYTHON_SOURCE: ${{ github.workspace }}/cpex-python
+        run: make test-python-e2e
+
   examples:
     name: Examples build (Rust + Go FFI)
     if: github.event_name != 'pull_request' || !github.event.pull_request.draft

.gitignore

@@ -274,6 +274,13 @@ db_path/
 tmp/
 
 .continue
-plugin-catalog
 # Hugo build output
 docs/public/
+# python cpex install target(s) — the repo-root install dir the Python CLI
+# materializes venvs into. Leading slash anchors it to the root: unanchored,
+# `plugins/` matched at any depth and swallowed the tracked source trees
+# `builtins/plugins/` and `tests/unit/cpex/fixtures/plugins/` (existing files
+# stayed tracked, but any newly added one was silently unstageable).
+/plugins/
+data/
+plugin-catalog

CHANGELOG.md

@@ -13,6 +13,18 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
 > - **Fixed**: for any bug fixes.
 > - **Security**: in case of vulnerabilities.
 
+## [Unreleased]
+
+### Added
+
+- **Out-of-process host for existing Python CPEX plugins.** A new `cpex-hosts-python` crate registers `kind: isolated_venv`, running an unmodified Python CPEX plugin in its own cached virtualenv as a subprocess instead of in-process through the PyO3 bindings. Each plugin gets a venv keyed by a SHA-256 fingerprint of its requirements + manifest (rebuilt when either changes, `rmtree`d rather than upgraded in place so a removed dependency actually disappears), and the host drives the Python framework's `worker.py` over a newline-delimited JSON stdio protocol. Hook payloads, `context`, and the capability-filtered `Extensions` view cross as JSON; returns come back as a serialized `PluginResult`, with `modified_extensions` merged through the executor's existing copy-on-write tier validation — the host implements no tier logic of its own. Failure modes the executor cannot otherwise distinguish (venv build failure, worker death mid-flight, a task over `max_content_size`, per-invocation timeout) map to distinct `PluginError`s carrying a stable `code` and structured `details`, so the executor's configured `on_error` policy applies unchanged. Pure Rust plus a subprocess — no libpython link, so the crate is in `default-members` and a plain `cargo build` does not require a Python dev install. The wire contract is pinned in `docs/specs/extensions-wire-contract.md`; CMF §3 remains normative for the extension slots themselves. (#149)
+
+### Security
+
+- **Raw credential material can now reach an out-of-process worker.** `RawInboundToken.token` and `RawDelegatedToken.token` are `#[serde(skip)]`, and the "raw credentials never leave the host process" invariant has held because nothing read those fields directly. The `isolated_venv` host narrowly reverses that for two hooks — `identity_resolve` and `token_delegate`, the only ones whose Python payload models a raw token at all — by reading the in-memory field and sending the plaintext in a dedicated `credential` DTO. This is opt-in twice over and fails closed: a plugin receives nothing unless it declares `read_inbound_credentials` or `read_delegated_tokens`, and a plugin that declares one but cannot be served (no extension, empty token) causes the dispatch to error rather than silently sending no credential — an empty bearer would otherwise read as "no authentication required". The `raw_credentials` extension slot itself is never carried, so no hollow token slot invites a plugin to misread "not on this channel" as "no credential present", and the sensitive-header strip (`Authorization` / `Cookie` / `Set-Cookie` / `X-API-Key`, case-insensitive, both directions) keeps credential headers out of the `http` slot regardless. The FFI, Python-bindings, and audit paths are untouched, and the in-process hosts are unaffected.
+
+  **Operators should note the residual exposure the capability gate does not close.** Once the plaintext is resident in the worker process it is readable by every transitively-installed dependency in that plugin's venv — a materially larger and less audited trust boundary than the in-process host, which neither the gate nor the transport can constrain. Grant these two capabilities only to plugins whose venv contents you control, and treat a plugin's requirements file as credential-adjacent supply chain. (#149)
+
 ## [0.2.2] - 2026-07-15
 
 ### Added
@@ -93,7 +105,8 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
 
 - Initial release
 
-[Unreleased]: https://github.com/contextforge-org/cpex/compare/0.2.1...HEAD
+[Unreleased]: https://github.com/contextforge-org/cpex/compare/0.2.2...HEAD
+[0.2.2]: https://github.com/contextforge-org/cpex/compare/0.2.1...0.2.2
 [0.2.1]: https://github.com/contextforge-org/cpex/compare/0.2.0...0.2.1
 [0.2.0]: https://github.com/contextforge-org/cpex/compare/0.1.1...0.2.0
 [0.1.1]: https://github.com/contextforge-org/cpex/compare/0.1.0...0.1.1

Cargo.toml

@@ -10,6 +10,7 @@ resolver = "2"
 members = [
     "crates/cpex",
     "crates/cpex-core",
+    "crates/cpex-hosts-python",
     "crates/cpex-orchestration",
     "crates/cpex-sdk",
     "crates/cpex-builtins",
@@ -35,17 +36,25 @@ members = [
 ]
 
 # `default-members` controls what `cargo build` / `cargo test` (with no
-# `-p` or `--workspace` flag) picks up. `cpex-session-valkey` pulls a redis
-# client + a rustls TLS stack, which slows default builds — excluding it
-# from default-members keeps everyday iteration fast.
+# `-p` or `--workspace` flag) picks up. Two members are held out:
 #
-# To exercise the Valkey session store:
+#   * `bindings/python` — links libpython, so including it would make a plain
+#     `cargo build` require a Python dev install (KD3).
+#   * `builtins/session/valkey` — pulls a redis client + a rustls TLS stack,
+#     which slows every default build.
+#
+# Everything else, including `crates/cpex-hosts-python` (pure Rust; it only
+# *spawns* a Python subprocess, never links one), builds by default.
+#
+# To exercise a held-out member:
 #   cargo build --workspace                  # all members
-#   cargo build -p cpex-session-valkey       # just this one
+#   cargo build -p cpex-session-valkey       # just the Valkey session store
+#   cargo build -p cpex-python               # just the PyO3 bindings
 #   cargo test --workspace                   # full sweep (CI)
 default-members = [
     "crates/cpex",
     "crates/cpex-core",
+    "crates/cpex-hosts-python",
     "crates/cpex-orchestration",
     "crates/cpex-sdk",
     "crates/cpex-builtins",

Makefile

@@ -43,6 +43,8 @@ help:
 	@echo "  test              Run all workspace tests"
 	@echo "  test-ffi          Run only the cpex-ffi crate tests"
 	@echo "  test-all          Rust tests + Go tests (with -race)"
+	@echo "  test-python-e2e   Python host e2e (#[ignore]d); needs CPEX_PYTHON_SOURCE."
+	@echo "                    Skips FAIL here — this lane must really run."
 	@echo ""
 	@echo "Supply chain & coverage:"
 	@echo "  audit             cargo deny check (advisories, licenses, bans, sources)"
@@ -143,6 +145,38 @@ test:
 test-ffi:
 	@$(CARGO) test -p cpex-ffi --lib
 
+# The Python host's end-to-end tests. They are `#[ignore]`d because they need a
+# python3 and a checkout of the cpex Python side, so `make test` reports them as
+# ignored rather than passing a body that never ran.
+#
+# CPEX_REQUIRE_PYTHON_E2E=1 turns every in-test skip into a panic: this target
+# is the lane that is supposed to have the environment, so a skip here is a
+# broken lane, not an absent dependency. That is what stops the suite reporting
+# safety it has not verified.
+#
+# CPEX_PYTHON_SOURCE must point at a cpex Python checkout carrying
+# cpex/framework/isolated/worker.py (PyPI's is behind this branch).
+PYTHON_E2E_TESTS = credential_e2e isolated_venv_e2e extensions_merge_e2e
+
+.PHONY: test-python-e2e
+test-python-e2e:
+	@command -v python3 >/dev/null 2>&1 || { \
+		echo "❌ python3 not found — the Python host e2e tests need an interpreter"; exit 1; }
+	@test -n "$(CPEX_PYTHON_SOURCE)" || { \
+		echo "❌ CPEX_PYTHON_SOURCE is unset. Point it at a cpex Python checkout containing"; \
+		echo "   cpex/framework/isolated/worker.py, e.g.:"; \
+		echo "   make test-python-e2e CPEX_PYTHON_SOURCE=../cpex-python"; exit 1; }
+	@test -f "$(CPEX_PYTHON_SOURCE)/cpex/framework/isolated/worker.py" || { \
+		echo "❌ $(CPEX_PYTHON_SOURCE) has no cpex/framework/isolated/worker.py"; exit 1; }
+	@echo "🐍 Python host e2e (skips fail here) ..."
+	@for t in $(PYTHON_E2E_TESTS); do \
+		echo "→ $$t"; \
+		CPEX_REQUIRE_PYTHON_E2E=1 CPEX_PYTHON_SOURCE="$(CPEX_PYTHON_SOURCE)" \
+			$(CARGO) test -p cpex-hosts-python --test $$t \
+			-- --ignored --nocapture || exit 1; \
+	done
+	@echo "✅  Python host e2e passed (no skips)"
+
 # Rust workspace tests + Go tests under the race detector.
 .PHONY: test-all
 test-all: test go-test-race

crates/README.md

@@ -8,6 +8,9 @@ Phase 1a — core runtime functional, no language bindings yet.
 
 - `cpex-core`: Plugin trait, typed hooks, 5-phase executor, plugin manager
 - `cpex-sdk`: Lean re-exports for plugin authors
+- `cpex-hosts-python`: Runs existing Python plugins out-of-process, each in its
+  own cached virtualenv, under the `isolated_venv` kind
+  ([README](cpex-hosts-python/README.md))
 
 ## Prerequisites
 

crates/cpex-core/src/executor.rs

@@ -450,38 +450,54 @@ impl Executor {
                             if let Some(mp) = erased.modified_payload {
                                 *payload = mp;
                             }
-                            if let Some(owned) = erased.modified_extensions {
-                                let valid = extensions.validate_immutable(&owned);
-                                if !valid {
+                            if let Some(mut owned) = erased.modified_extensions {
+                                if extensions.validate_immutable(&owned) {
+                                    // `merge_owned` enforces the tiers per
+                                    // *field*, gated on the write tokens that
+                                    // `owned` carries. It is not a slot swap: a
+                                    // field with no token keeps its canonical
+                                    // value, so an ungated edit is dropped
+                                    // rather than merged. Previously this arm
+                                    // was reached by a bare `else` that merged
+                                    // the plugin's whole capability-filtered
+                                    // view over canonical state — a plugin with
+                                    // no security capability could wipe the
+                                    // pipeline's labels by returning `custom`.
+                                    //
+                                    // The monotonic label check that used to
+                                    // live here moved into `merge_security`,
+                                    // where it applies unconditionally instead
+                                    // of only when `read_labels` was held.
+                                    //
+                                    // Authority is re-derived from *this*
+                                    // plugin's declared capabilities rather than
+                                    // read off the returned value. A handler is
+                                    // free to build its `OwnedExtensions` any
+                                    // way it likes — apl-cpex's synthetic route
+                                    // handler returns `cow_copy()` of a freshly
+                                    // accumulated `Extensions`, whose tokens
+                                    // `Clone` deliberately drops — so tokens
+                                    // surviving the round trip is a statement
+                                    // about plumbing, not about permission. The
+                                    // capability set is the real grant, and it
+                                    // cannot be widened by the return value.
+                                    owned.http_write_token = capabilities
+                                        .contains("write_headers")
+                                        .then(WriteToken::new);
+                                    owned.labels_write_token = capabilities
+                                        .contains("append_labels")
+                                        .then(WriteToken::new);
+                                    owned.delegation_write_token = capabilities
+                                        .contains("append_delegation")
+                                        .then(WriteToken::new);
+                                    extensions.merge_owned(owned);
+                                } else {
                                     warn!(
                                         "{} plugin '{}' violated immutable tier — \
                                          modified an immutable extension slot. \
                                          Extension changes rejected.",
                                         phase_label, plugin_name
                                     );
-                                } else if capabilities.contains("read_labels") {
-                                    // Only enforce monotonic labels if the plugin
-                                    // could see them. A plugin without read_labels
-                                    // has empty labels in its filtered view — that's
-                                    // not a removal.
-                                    if let (Some(ref orig_sec), Some(ref new_sec)) =
-                                        (&extensions.security, &owned.security)
-                                    {
-                                        if !new_sec.labels.is_superset(&orig_sec.labels) {
-                                            warn!(
-                                                "{} plugin '{}' violated monotonic tier — \
-                                                 removed a security label. \
-                                                 Extension changes rejected.",
-                                                phase_label, plugin_name
-                                            );
-                                        } else {
-                                            extensions.merge_owned(owned);
-                                        }
-                                    } else {
-                                        extensions.merge_owned(owned);
-                                    }
-                                } else {
-                                    extensions.merge_owned(owned);
                                 }
                             }
                         }