Convert every remaining raw-node crossing to shadow copies (#356)

Roll the shadow pattern out beyond templates so no binding hands an lxml
node to xmlsec anymore (fast path unchanged on matched libxml2):

- template.c: all remaining functions, incl. the create shape
  (BeginNewDoc/EndNewDoc builds the detached template in a private doc)
  and the status-int C14N helper (FindFresh locates the created node).
- tree.c: finders map results back by path (EndFind, None on not-found;
  find_parent shadows the whole tree); add_ids records id-attribute
  specs instead of writing lxml's ID hash with our libxml2.
- ds.c: sign/verify run on a whole-document copy (BeginDoc) with the
  recorded IDs replayed so #id references resolve; sign reflects all
  mutation sites (DigestValue/SignatureValue/KeyInfo) via ReflectAll,
  verify just discards the copy.
- enc.c: encrypt_binary/encrypt_uri reflect the mutated template;
  encrypt_xml/decrypt re-parse everything into one copy and reflect the
  replacement through lxml (element, content, or returned bytes).
  Replacing the document root cannot be expressed through lxml's API
  and raises a clear error on the shadow path.

ReflectAll is two-phase (prefetch payloads from the re-parsed copy in
its final state, then apply to the live tree in document order) because
each graft moves a node out of the copy and would invalidate the
indices later sites resolve through.

Two template tests now attach the created template before asserting
liveness: a shadow-created template lives in its own document until
grafted, as lxml cannot express the raw path's "detached node inside an
existing document".

Validated under a real 2.14<->2.15 mismatch (full suite, 10k-iteration
sign/verify/encrypt/decrypt loop, flat RSS, byte-identical output) and
on a matched static wheel with and without PYXMLSEC_FORCE_SHADOW.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Author: mxamin

converting-functions.md

@@ -19,13 +19,18 @@ grep -n '_c_node\|_c_doc' src/template.c
 Read the `xmlSecTmpl*` function it wraps (xmlsec1's `src/templates.c`) and
 match it to a row:
 
-| Shape | Examples | Covered? |
+| Shape | Examples | How |
 |---|---|---|
 | Adds a subtree under the element (any position, possibly with intermediate nodes) | `add_key_name`, `add_key_value`, `add_x509_data`, `x509_data_add_*`, `add_encrypted_key` | ✅ just follow step 3 |
-| Find-or-create, may set attributes on the existing node | `encrypted_data_ensure_key_info`, `encrypted_data_ensure_cipher_value` | ✅ just follow step 3 |
-| Returns a status `int`, mutates one child | `transform_add_c14n_inclusive_namespaces` | ✅ with a twist: after the call, locate the affected child on the copy (here: the `<InclusiveNamespaces>` under `shadow.root`) and pass *that* to `End`; discard the returned element and return `None` |
-| Returns a **detached** node — the element argument only supplies the document | `create`, `encrypted_data_create` | ❌ `End` would raise "unexpected result node"; needs its own reflect (no graft — parse the standalone result with lxml and return it) |
-| Reads or mutates the **whole document** | `ds.c` sign/verify, `enc.c` encrypt/decrypt, `tree.c` | ❌ needs a whole-document reflect strategy |
+| Find-or-create, may set attributes on the existing node | `encrypted_data_ensure_key_info`, `encrypted_data_ensure_cipher_value` | ✅ just follow step 3 (a call that can also *rename the prefix* of the existing node uses `EndReplace` instead of `End`) |
+| Returns a status `int`, mutates one child | `transform_add_c14n_inclusive_namespaces` | ✅ pass `PyXmlSec_LxmlShadowFindFresh(&shadow)` to `End`; discard the returned element and return `None` |
+| Returns a **detached** node — the element argument only supplies the document | `create`, `encrypted_data_create` | ✅ `BeginNewDoc`/`EndNewDoc`: the call runs against a private document and the result comes back as a new detached lxml element |
+| Read-only search | `tree.c` find_child/find_node (subtree), find_parent (whole doc) | ✅ `Begin` (or `BeginDoc` when the search leaves the subtree) + `EndFind`, which returns `None` on not-found |
+| Reads or mutates the **whole document**, possibly at several places | `ds.c` sign (whole-doc + multi-site), verify (read-only) | ✅ `BeginDoc` + `ReplayIds`, then `ReflectAll` (sign) or `Discard` (verify) |
+| **Replaces** nodes | `enc.c` encrypt_xml/decrypt (`encrypt_binary`/`encrypt_uri` are template-shaped: `Begin` + `ReflectAll`) | ✅ `BeginDoc`, remove the consumed live node/content first, then `ReflectAll` grafts the replacement (`_setroot` when the document root itself was replaced) |
+
+All shapes are converted; `developer.md`'s "Beyond templates" section explains
+each reflect. This guide stays as the recipe should new bindings appear.
 
 `res` does not have to be the topmost node the call created — `End` walks up
 to the topmost new ancestor itself. Any node inside the new subtree works.

developer.md

@@ -56,8 +56,54 @@ Child indices count exactly the node types lxml exposes as children (elements,
 comments, PIs, entity refs), so paths recorded on the raw copy resolve
 identically through lxml's `__getitem__`/`insert`. `End` assumes the xmlsec
 call mutates at most one place in the tree — true for all `xmlSecTmpl*`
-functions. Whole-document operations (`sign`, `encrypt`) mutate many places
-and will need a different reflect step on top of the same Begin machinery.
+functions.
+
+## Beyond templates: the whole-code rollout
+
+Every binding that used to hand a raw lxml node to xmlsec now goes through a
+shadow. The extra shapes (all in [src/lxml.c](src/lxml.c), contracts in
+[src/lxml.h](src/lxml.h)):
+
+- **Create** (`template.create`, `encrypted_data_create`):
+  `BeginNewDoc`/`EndNewDoc`. The call builds a *detached* subtree and only
+  needs a document to allocate in — a private one on the shadow path. The
+  result comes back as a new detached lxml element (in its own document; lxml
+  moves it when the caller grafts it), instead of the raw path's "detached
+  node inside the source document", which lxml's API cannot express.
+- **Finders** (`tree.find_child`/`find_node`/`find_parent`): `EndFind` maps
+  the found copy node back by path and returns `None` on not-found.
+  `find_parent` walks upward, so it uses the whole-document Begin.
+- **Whole-document** (`sign`, `verify`, `decrypt`, `find_parent`):
+  `BeginDoc` serializes `element.getroottree()` (comments/PIs outside the
+  root and the internal DTD subset survive), records the element's position
+  through lxml's API, and hands back the copy's counterpart node.
+- **Multi-site reflect** (`sign`, `encrypt_binary`, `encrypt_uri`):
+  `ReflectAll` scans the copy for *every* topmost untagged node and grafts
+  each back — new subtrees via `insert`, new/changed text (DigestValue,
+  SignatureValue) via the text slots. Two-phase: payloads are fetched from
+  the re-parsed copy while it is still in its final state, then applied to
+  the live tree in document order (a graft moves a node out of the re-parsed
+  copy, which would invalidate later fetches).
+- **Replacement reflect** (`encrypt_xml`, `decrypt`): encryption/decryption
+  *replace* nodes, so the live target (or its content, or the document root
+  via `_setroot`) is removed first and `ReflectAll` grafts what took its
+  place. `encrypt_xml` re-serializes the template into the same shadow doc
+  (`ImportElement`); a template attached inside the target's own tree is
+  therefore copied, not moved. `verify` needs no reflect at all — `Discard`
+  just frees the copy.
+- **ID registration** (`tree.add_ids`, `SignatureContext.register_id`): these
+  used to write lxml's ID hash with our libxml2. Under the shadow they record
+  the id-attribute specs in a registry keyed by document identity
+  (`RecordId`), and every `BeginDoc` replays them onto the copy
+  (`ReplayIds`) so `#id` references resolve. The replay scans the whole copy
+  for the recorded attribute names — a superset of the raw registration. The
+  registry holds no strong references to documents (lxml objects refuse weak
+  references, so entries are validated by a stored `_c_doc` address and
+  capped in size).
+- **Prefix rename** (`encrypted_data_ensure_key_info(ns=...)` on an existing
+  KeyInfo): `EndReplace` swaps the live element for the copy's version, since
+  lxml cannot rename a prefix in place; the returned element is then a new
+  object rather than the original proxy.
 
 ## The fast path: shadows only when needed
 
@@ -114,11 +160,30 @@ on matched libraries — safe everywhere, so the whole suite must pass).
 
 ## Status
 
-- ✅ `template.add_reference`, `template.add_transform`,
-  `template.ensure_key_info` — one per reflect shape. Validated under a real
-  2.14 ↔ 2.15 mismatch: full suite green, 10k-iteration loop with no crash,
-  no leak, byte-identical output.
-- ⬜ Rest of `src/template.c` — mechanical: follow the step-by-step guide in
-  [converting-functions.md](converting-functions.md).
-- ⬜ `src/ds.c` (sign/verify), `src/enc.c` (encrypt/decrypt), `src/tree.c` —
-  need a whole-document reflect strategy.
+- ✅ All of `src/template.c` (create, references, transforms, key info, x509,
+  encrypted data, C14N namespaces).
+- ✅ `src/tree.c` (find_child/find_node/find_parent, add_ids).
+- ✅ `src/ds.c` (register_id, sign, verify; the binary operations never
+  touched nodes).
+- ✅ `src/enc.c` (encrypt_binary, encrypt_uri, encrypt_xml, decrypt).
+- Validated under a real 2.14 ↔ 2.15 mismatch: full suite green, 10k-iteration
+  sign/verify/encrypt/decrypt loop with no crash, no leak, byte-identical
+  output; and on a matched static build: full suite green on both the fast
+  path and `PYXMLSEC_FORCE_SHADOW=1`.
+- ⬜ Endgame: turn the import-time guard into a mode switch (mismatch sets
+  the shadow flag instead of refusing to import) and retire
+  `PYXMLSEC_SKIP_VERSION_CHECK`. That is the actual user-facing resolution
+  of #356, kept as its own change.
+
+Known, deliberate divergences on the shadow path (all invisible to the
+documented API): created templates live in their own document until grafted;
+`encrypted_data_ensure_key_info(ns=...)` on an existing KeyInfo returns a new
+element object; `register_id` skips the live duplicate-id check (it runs per
+copy instead); `encrypt_xml` copies rather than moves a template that is
+attached inside the target tree; signature/encryption contexts keep no live
+result nodes after the call (they never usefully did). One hard limitation:
+operations that would replace the **document root** (encrypting the root
+element, decrypting a root `EncryptedData`) raise `xmlsec.Error` — lxml's API
+cannot swap a document's root, and morphing it in place would rewrite
+namespace prefixes, breaking signatures over the content. Re-parse the
+document or work on a subelement instead.

src/ds.c

@@ -146,6 +146,39 @@ static PyObject* PyXmlSec_SignatureContextRegisterId(PyObject* self, PyObject* a
         goto ON_FAIL;
     }
 
+    // Shadow mode: never touch lxml's document (its ID hash) with our libxml2
+    // (issue #356). Validate through lxml's API and record the spec; the
+    // whole-document shadows (sign/verify/decrypt) replay it onto their
+    // private copies. The duplicate-id check runs there, per copy.
+    if (PyXmlSec_LxmlShadowIsActive()) {
+        PyObject* key;
+        PyObject* value;
+        if (id_ns != NULL) {
+            key = PyUnicode_FromFormat("{%s}%s", id_ns, id_attr);
+        } else {
+            key = PyUnicode_FromString(id_attr);
+        }
+        if (key == NULL) {
+            goto ON_FAIL;
+        }
+        value = PyObject_CallMethod((PyObject*)node, "get", "O", key);
+        Py_DECREF(key);
+        if (value == NULL) {
+            goto ON_FAIL;
+        }
+        if (value == Py_None) {
+            Py_DECREF(value);
+            PyErr_SetString(PyXmlSec_Error, "missing attribute.");
+            goto ON_FAIL;
+        }
+        Py_DECREF(value);
+        if (PyXmlSec_LxmlShadowRecordId(node, id_attr, id_ns) < 0) {
+            goto ON_FAIL;
+        }
+        PYXMLSEC_DEBUGF("%p: register id - ok", self);
+        Py_RETURN_NONE;
+    }
+
     if (id_ns != NULL) {
         attr = xmlHasNsProp(node->_c_node, XSTR(id_attr), XSTR(id_ns));
     } else {
@@ -189,21 +222,39 @@ static PyObject* PyXmlSec_SignatureContextSign(PyObject* self, PyObject* args, P
 
     PyXmlSec_SignatureContext* ctx = (PyXmlSec_SignatureContext*)self;
     PyXmlSec_LxmlElementPtr node = NULL;
+    xmlNodePtr target;
+    PyXmlSec_LxmlShadow shadow;
     int rv;
 
     PYXMLSEC_DEBUGF("%p: sign - start", self);
     if (!PyArg_ParseTupleAndKeywords(args, kwargs, "O&:sign", kwlist, PyXmlSec_LxmlElementConverter, &node)) {
         goto ON_FAIL;
     }
 
+    // References (URI="", "#id") reach anywhere in the document, so the
+    // shadow covers the whole tree (issue #356); registered IDs are replayed
+    // onto the copy so they resolve. Signing fills several places inside
+    // <Signature> (DigestValue, SignatureValue, KeyInfo), all reflected by
+    // the multi-site reflection.
+    if (PyXmlSec_LxmlShadowBeginDoc(&shadow, node, &target) < 0) {
+        goto ON_FAIL;
+    }
+    if (PyXmlSec_LxmlShadowReplayIds(&shadow) < 0) {
+        PyXmlSec_LxmlShadowDiscard(&shadow);
+        goto ON_FAIL;
+    }
     Py_BEGIN_ALLOW_THREADS;
-    rv = xmlSecDSigCtxSign(ctx->handle, node->_c_node);
+    rv = xmlSecDSigCtxSign(ctx->handle, target);
     PYXMLSEC_DUMP(xmlSecDSigCtxDebugDump, ctx->handle);
     Py_END_ALLOW_THREADS;
     if (rv < 0) {
+        PyXmlSec_LxmlShadowDiscard(&shadow);
         PyXmlSec_SetLastError("failed to sign");
         goto ON_FAIL;
     }
+    if (PyXmlSec_LxmlShadowReflectAll(&shadow) < 0) {
+        goto ON_FAIL;
+    }
     PYXMLSEC_DEBUGF("%p: sign - ok", self);
     Py_RETURN_NONE;
 
@@ -224,17 +275,29 @@ static PyObject* PyXmlSec_SignatureContextVerify(PyObject* self, PyObject* args,
 
     PyXmlSec_SignatureContext* ctx = (PyXmlSec_SignatureContext*)self;
     PyXmlSec_LxmlElementPtr node = NULL;
+    xmlNodePtr target;
+    PyXmlSec_LxmlShadow shadow;
     int rv;
 
     PYXMLSEC_DEBUGF("%p: verify - start", self);
     if (!PyArg_ParseTupleAndKeywords(args, kwargs, "O&:verify", kwlist, PyXmlSec_LxmlElementConverter, &node)) {
         goto ON_FAIL;
     }
 
+    // Verification is read-only: whole-document shadow, replayed IDs, and no
+    // reflection at all — the copy is simply discarded.
+    if (PyXmlSec_LxmlShadowBeginDoc(&shadow, node, &target) < 0) {
+        goto ON_FAIL;
+    }
+    if (PyXmlSec_LxmlShadowReplayIds(&shadow) < 0) {
+        PyXmlSec_LxmlShadowDiscard(&shadow);
+        goto ON_FAIL;
+    }
     Py_BEGIN_ALLOW_THREADS;
-    rv = xmlSecDSigCtxVerify(ctx->handle, node->_c_node);
+    rv = xmlSecDSigCtxVerify(ctx->handle, target);
     PYXMLSEC_DUMP(xmlSecDSigCtxDebugDump, ctx->handle);
     Py_END_ALLOW_THREADS;
+    PyXmlSec_LxmlShadowDiscard(&shadow);
 
     if (rv < 0) {
         PyXmlSec_SetLastError("failed to verify");