fix: stale HookRef in TaskProcessor, and cmf vs. bare compatibility

CHANGELOG.md

@@ -15,6 +15,49 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
 
 ## [Unreleased]
 
+### Added
+
+- Auto-conversion of "bare-FQN" plugins: a plugin whose manifest `kind` is a Python class
+  path (e.g. `package.module.ClassName`) instead of a known kind is now converted to an
+  `isolated_venv` plugin at install time (the FQN is moved into `default_config.class_name`),
+  so it runs out-of-process in a per-plugin virtual environment ([#113](https://github.com/contextforge-org/cpex/pull/113)).
+- `--no-convert` flag on `cpex plugin install` to opt out of the conversion above and keep
+  the plugin's declared FQN `kind` (loaded in-process). `--no-convert` also softens an
+  unknown/unsupported `kind` from a hard error to a warning. Applies to pypi/test-pypi/git/local
+  installs ([#113](https://github.com/contextforge-org/cpex/pull/113)).
+
+### Changed
+
+- **Runtime model of existing FQN-declared Python plugins.** On 0.1.x, declaring a plugin
+  `kind` as a Python class path was how in-process Python plugins were declared. Because
+  conversion is now **on by default**, upgrading changes such plugins from in-process to the
+  out-of-process `isolated_venv` model unless installed with `--no-convert`. Conversion also
+  runs during `cpex plugin catalog update` and persists the converted form to
+  `plugin-manifest.yaml` / `plugins/config.yaml` ([#113](https://github.com/contextforge-org/cpex/pull/113)).
+
+### Fixed
+
+- **Isolated worker: error responses could carry a stale `request_id`.** The worker reused a
+  `main()`-local `request_id` across loop iterations, so an error raised before the next task
+  parsed could be tagged with the previous request's id — misdelivering the error to the wrong
+  caller's queue or hanging the real caller until timeout. The id is now reset per iteration
+  ([#113](https://github.com/contextforge-org/cpex/pull/113)).
+- **`cpex plugin install pkg@<constraint>` could wrongly skip.** The repeat-install check
+  dropped the version constraint and, for pypi/test-pypi, compared against a possibly stale
+  catalog entry. An explicit version constraint now always proceeds with the install
+  ([#113](https://github.com/contextforge-org/cpex/pull/113)).
+- **Upgrade no longer force-rebuilds every existing `isolated_venv` venv.** The venv cache now
+  treats a *missing* manifest version/hash signal (metadata written by an earlier CLI) as "no
+  signal" rather than a mismatch, so pre-existing venvs are not wiped and rebuilt on the first
+  run after upgrade ([#113](https://github.com/contextforge-org/cpex/pull/113)).
+- **Multi-plugin packages no longer thrash the venv cache.** The persisted plugin manifest is
+  now keyed on the plugin's full class name instead of the shared package root, so installing
+  one plugin in a package no longer invalidates a sibling plugin's cache hash and triggers a
+  rebuild loop ([#113](https://github.com/contextforge-org/cpex/pull/113)).
+- **test-pypi isolated installs resolve transitive dependencies.** Installing a plugin from
+  test.pypi into a fresh isolated venv now also passes `--extra-index-url https://pypi.org/simple/`,
+  so transitive dependencies (including `cpex` itself) resolve from real PyPI instead of failing
+  when they are absent from test.pypi ([#113](https://github.com/contextforge-org/cpex/pull/113)).
 ## [0.1.3] - 2026-08-06
 
 ### Changed

cpex/framework/constants.py

@@ -12,8 +12,26 @@
 
 # Model constants.
 # Specialized plugin types.
+NATIVE_PLUGIN_TYPE = "native"
+BUILTIN_PLUGIN_TYPE = "builtin"
+WASM_PLUGIN_TYPE = "wasm"
 EXTERNAL_PLUGIN_TYPE = "external"
 ISOLATED_VENV_PLUGIN_TYPE = "isolated_venv"
+PDP_PLUGIN_TYPE = "PDP"
+
+# The set of manifest `kind` values the installer recognizes as-is. Any `kind`
+# outside this set is a candidate for FQN auto-conversion to isolated_venv
+# (see cpex.tools.catalog.classify_plugin_kind).
+KNOWN_PLUGIN_KINDS = frozenset(
+    {
+        BUILTIN_PLUGIN_TYPE,
+        NATIVE_PLUGIN_TYPE,
+        WASM_PLUGIN_TYPE,
+        EXTERNAL_PLUGIN_TYPE,
+        ISOLATED_VENV_PLUGIN_TYPE,
+        PDP_PLUGIN_TYPE,
+    }
+)
 
 
 # MCP related constants.

cpex/framework/isolated/client.py

@@ -27,7 +27,7 @@
 from cpex.framework.hooks.registry import get_hook_registry
 from cpex.framework.isolated.venv_comm import VenvProcessCommunicator
 from cpex.framework.models import PluginConfig, PluginContext, PluginErrorModel, PluginPayload, PluginResult
-from cpex.framework.utils import find_package_path
+from cpex.framework.utils import find_package_path, manifest_filename_for_class
 
 logger = logging.getLogger(__name__)
 
@@ -51,27 +51,61 @@ def __init__(self, config: PluginConfig, plugin_dirs) -> None:
         self.cache_dir: Path = cache_root / ".cpex" / "venv_cache"
         self.cache_dir.mkdir(parents=True, exist_ok=True)
 
-    def _compute_requirements_hash(self, requirements_file: str) -> str:
+    def _compute_requirements_hash(self, requirements_file: Optional[str]) -> str:
         """Compute SHA256 hash of requirements file content.
 
         Args:
-            requirements_file: Path to the requirements file
+            requirements_file: Path to the requirements file, or None when the
+                plugin has no requirements file.
 
         Returns:
-            Hexadecimal hash string
+            Hexadecimal hash string. An absent or non-existent file hashes to
+            the empty-content digest.
         """
         hasher = hashlib.sha256()
-        req_path = Path(requirements_file)
 
-        if req_path.exists():
-            with open(req_path, "rb") as f:
-                hasher.update(f.read())
+        if requirements_file is not None:
+            req_path = Path(requirements_file)
+            if req_path.exists():
+                with open(req_path, "rb") as f:
+                    hasher.update(f.read())
+            else:
+                # If no requirements file, use empty hash
+                hasher.update(b"")
         else:
-            # If no requirements file, use empty hash
             hasher.update(b"")
 
         return hasher.hexdigest()
 
+    def _manifest_path(self) -> Path:
+        """Path to the plugin's persisted manifest under its plugin directory.
+
+        The filename is keyed on the full class name so plugins that share a
+        package (and therefore this directory and its venv) do not collide on a
+        single manifest file — which would make each install invalidate the
+        others' cache hash. Must match the write side in
+        PluginCatalog._persist_manifest_to_plugin_dir.
+        """
+        class_name = self.config.config.get("class_name")
+        return self.plugin_path / manifest_filename_for_class(class_name)
+
+    def _compute_manifest_hash(self) -> str:
+        """Compute SHA256 hash of the persisted plugin-manifest.yaml.
+
+        Returns the empty-content digest when no manifest file is present. This
+        gives plugins without a self-referencing requirements file a stable
+        change signal for cache invalidation (U5): a version bump or any edit to
+        the persisted manifest changes the hash and forces a venv reinstall.
+        """
+        hasher = hashlib.sha256()
+        manifest_path = self._manifest_path()
+        if manifest_path.exists():
+            with open(manifest_path, "rb") as f:
+                hasher.update(f.read())
+        else:
+            hasher.update(b"")
+        return hasher.hexdigest()
+
     def _get_cache_metadata_path(self, venv_path: str) -> Path:
         """Get the path to the cache metadata file.
 
@@ -84,12 +118,14 @@ def _get_cache_metadata_path(self, venv_path: str) -> Path:
         venv_name = Path(venv_path).name
         return self.cache_dir / f"{venv_name}_metadata.json"
 
-    def _is_venv_cache_valid(self, venv_path: str, requirements_file: str) -> bool:
-        """Check if cached venv is valid by comparing requirements hash.
+    def _is_venv_cache_valid(self, venv_path: str, requirements_file: Optional[str]) -> bool:
+        """Check if cached venv is valid by comparing requirements + manifest signals.
 
         Args:
             venv_path: Path to the virtual environment
-            requirements_file: Path to the requirements file
+            requirements_file: Path to the requirements file, or None when the
+                plugin has no requirements file (manifest version+hash is then
+                the sole change signal).
 
         Returns:
             True if cache is valid, False otherwise
@@ -115,33 +151,68 @@ def _is_venv_cache_valid(self, venv_path: str, requirements_file: str) -> bool:
             # Compute current requirements hash
             current_hash = self._compute_requirements_hash(requirements_file)
 
-            # Compare hashes
+            # Compare requirements hash
             cached_hash = metadata.get("requirements_hash")
             if cached_hash != current_hash:
                 logger.info("Requirements changed. Cached hash: %s, Current hash: %s", cached_hash, current_hash)
                 return False
 
+            # Compare manifest version + hash (U5). Either changing invalidates
+            # the cache — this is the sole change signal for plugins with no
+            # requirements file (whose requirements hash is a constant).
+            #
+            # A *missing* key means the metadata predates these signals (written
+            # by an earlier CLI): treat it as "no signal" and skip the check,
+            # rather than reading .get() -> None as a mismatch. Otherwise every
+            # existing isolated_venv install would be wiped and rebuilt on the
+            # first run after upgrade, contradicting plan R5. Only invalidate
+            # when the key is present and differs.
+            if "manifest_version" in metadata:
+                current_version = self.config.version
+                cached_version = metadata.get("manifest_version")
+                if cached_version != current_version:
+                    logger.info("Manifest version changed. Cached: %s, Current: %s", cached_version, current_version)
+                    return False
+
+            if "manifest_hash" in metadata:
+                current_manifest_hash = self._compute_manifest_hash()
+                cached_manifest_hash = metadata.get("manifest_hash")
+                if cached_manifest_hash != current_manifest_hash:
+                    logger.info(
+                        "Manifest content changed. Cached hash: %s, Current hash: %s",
+                        cached_manifest_hash,
+                        current_manifest_hash,
+                    )
+                    return False
+
             logger.info("Valid venv cache found for %s", venv_path)
             return True
 
         except (json.JSONDecodeError, KeyError) as e:
             logger.warning("Error reading cache metadata: %s", str(e))
             return False
 
-    def _save_cache_metadata(self, venv_path: str, requirements_file: str) -> None:
+    def _save_cache_metadata(self, venv_path: str, requirements_file: Optional[str]) -> None:
         """Save cache metadata for the venv.
 
         Args:
             venv_path: Path to the virtual environment
-            requirements_file: Path to the requirements file
+            requirements_file: Path to the requirements file, or None when the
+                plugin has no requirements file.
         """
         metadata_path = self._get_cache_metadata_path(venv_path)
         requirements_hash = self._compute_requirements_hash(requirements_file)
 
+        resolved_requirements = None
+        if requirements_file is not None and Path(requirements_file).exists():
+            resolved_requirements = str(Path(requirements_file).resolve())
+
         metadata = {
             "venv_path": str(Path(venv_path).resolve()),
-            "requirements_file": str(Path(requirements_file).resolve()) if Path(requirements_file).exists() else None,
+            "requirements_file": resolved_requirements,
             "requirements_hash": requirements_hash,
+            "manifest_version": self.config.version,
+            "manifest_hash": self._compute_manifest_hash(),
             "python_version": f"{sys.version_info.major}.{sys.version_info.minor}.{sys.version_info.micro}",
         }
 
@@ -162,8 +233,10 @@ async def create_venv(
         """
         venv_path_obj = Path(venv_path)
 
-        # Check if we can use cached venv
-        if use_cache and requirements_file and self._is_venv_cache_valid(venv_path, requirements_file):
+        # Check if we can use cached venv. A None requirements_file is valid:
+        # for converted FQN plugins the manifest version+hash is the cache signal
+        # (U5), so cache validity must be evaluated even without a requirements file.
+        if use_cache and self._is_venv_cache_valid(venv_path, requirements_file):
             logger.info("✓ Using cached virtual environment at: %s", venv_path_obj.resolve())
             return False
 
@@ -209,38 +282,38 @@ async def initialize(self) -> None:
 
         venv_path = self.plugin_path / ".venv"
 
-        # Prevent directory traversal: ensure requirements_file stays within plugin_path
-        requirements_file_input = self.config.config["requirements_file"]
-
-        # Handle both relative and absolute paths
-        if isinstance(requirements_file_input, Path):
-            requirements_file = requirements_file_input
-        else:
-            requirements_file = Path(requirements_file_input)
-
-        # Try to find the package location where plugin-manifest.yaml resides
-        # Fall back to self.plugin_path if package is not installed (e.g., in tests)
-        try:
-            package_path = find_package_path(self.config.name)
-            logger.debug("Found installed package %s at %s", self.config.name, package_path)
-        except RuntimeError:
-            # Package not installed (e.g., in test environment), use plugin_path
-            package_path = self.plugin_path
-            logger.debug("Package %s not installed, using plugin_path: %s", self.config.name, package_path)
-
-        requirements_file = package_path / requirements_file_input
-
-        # Create venv with caching support
+        # requirements_file is optional: converted FQN plugins get their package
+        # into the venv via the installer's channel source, not a requirements file.
+        requirements_file_input = self.config.config.get("requirements_file")
+
+        requirements_file: Optional[Path] = None
+        if requirements_file_input:
+            # Try to find the package location where plugin-manifest.yaml resides.
+            # Fall back to self.plugin_path if package is not installed (e.g., in tests).
+            try:
+                package_path = find_package_path(self.config.name)
+                logger.debug("Found installed package %s at %s", self.config.name, package_path)
+            except RuntimeError:
+                # Package not installed (e.g., in test environment), use plugin_path
+                package_path = self.plugin_path
+                logger.debug("Package %s not installed, using plugin_path: %s", self.config.name, package_path)
+
+            requirements_file = package_path / requirements_file_input
+
+        # Create venv with caching support. create_venv tolerates a None/missing
+        # requirements file (empty-hash path).
         new_venv = await self.create_venv(venv_path=venv_path, requirements_file=requirements_file, use_cache=True)
 
         self.comm = VenvProcessCommunicator(venv_path)
 
-        # Only install requirements if venv was newly created or cache was invalid
-        # Check if we need to install requirements
+        # Only (re)install when the venv was newly created or the cache was invalid.
         if new_venv:
-            logger.info("Installing requirements in venv")
-            self.comm.install_requirements(requirements_file)
-            # Save metadata after successful installation
+            if requirements_file is not None and Path(requirements_file).exists():
+                logger.info("Installing requirements in venv")
+                self.comm.install_requirements(requirements_file)
+            else:
+                logger.info("No requirements file for %s; skipping requirements install", self.config.name)
+            # Save metadata after successful creation (records requirements hash for cache validity).
             self._save_cache_metadata(venv_path, requirements_file)
         else:
             logger.info("Using cached venv, skipping requirements installation")