Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/language.yml
Original file line number Diff line number Diff line change
Expand Up @@ -78,4 +78,4 @@ jobs:
tool: taplo

- name: Run repository checks
run: cargo make check
run: cargo make checks
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -638,7 +638,7 @@ comparability gates and no universal leaderboard claim.

```sh
cargo make fmt
cargo make check
cargo make checks
cargo make test-rust
```

Expand Down
9 changes: 9 additions & 0 deletions docs/log.md
Original file line number Diff line number Diff line change
Expand Up @@ -161,3 +161,12 @@ logs.
rather than generic Knowledge OS scope.
- Updated docs routing so the retained spec path now resolves by the source-backed
project memory product name and explicit Context Pack/recall boundary language.
- Updated Source Library and Memory Authority lifecycle contracts for XY-1152 so
captured sources carry active/current lifecycle metadata, deleted sources retain
tombstone metadata, memory corrections carry supersede/delete/restore lifecycle
state, and stale vector candidates are revalidated against current authority.
- Renamed the full repository gate task to `cargo make checks` for XY-1152 so the
Makefile command surface matches the registered Decodex verify command.
- Preserved historical benchmark evidence rows that recorded the former singular
check command, and constrained docs task validation to allow only those exact
legacy evidence references.
2 changes: 1 addition & 1 deletion docs/runbook/getting_started.md
Original file line number Diff line number Diff line change
Expand Up @@ -177,7 +177,7 @@ Use `cargo make` tasks from repository root.

```sh
cargo make fmt
cargo make check
cargo make checks
cargo make test-rust
cargo make test-rust-integration
cargo make test-e2e
Expand Down
2 changes: 1 addition & 1 deletion docs/spec/agent_memory_knowledge_system_v1.md
Original file line number Diff line number Diff line change
Expand Up @@ -321,7 +321,7 @@ Repository-native validation is authoritative.
docs are validation-ready.
- Before a PR handoff or any push that refreshes a PR head, run the registered
Decodex workflow gate: `cargo make fmt`, `cargo make lint-fix`, then
`cargo make check`.
`cargo make checks`.
- If a phase changes commands, schemas, config, runtime behavior, status semantics,
or benchmark claims, update the owning docs and include drift evidence as required
by `docs/policy.md`.
Expand Down
4 changes: 2 additions & 2 deletions docs/spec/production_corpus_manifest_v1.md
Original file line number Diff line number Diff line change
Expand Up @@ -100,7 +100,7 @@ evidence ID. It must not silently fall back to the checked-in synthetic corpus.
"evidence_id": "issue-xy123-resume",
"category": "issue",
"title": "XY-123 Resume State",
"text": "XY-123 resumes on branch y/example with command `cargo make check`."
"text": "XY-123 resumes on branch y/example with command `cargo make checks`."
}
],
"queries": [
Expand All @@ -110,7 +110,7 @@ evidence ID. It must not silently fall back to the checked-in synthetic corpus.
"query": "How do I resume XY-123?",
"expected_evidence_ids": ["issue-xy123-resume"],
"allowed_alternate_evidence_ids": [],
"expected_terms": ["XY-123", "cargo make check"]
"expected_terms": ["XY-123", "cargo make checks"]
}
]
}
Expand Down
10 changes: 9 additions & 1 deletion docs/spec/system_doc_source_ref_v1.md
Original file line number Diff line number Diff line change
Expand Up @@ -200,6 +200,9 @@ Normalized capture output:

- `docs_put` MUST return `source_capture.schema = "doc_source_capture/v1"`.
- `source_capture.source_record_id` MUST equal the stored `doc_documents.doc_id`.
- `source_capture.lifecycle` MUST use `elf.source_lifecycle/v1` with
`status = "active"`, `freshness = "current"`, the capture actor, transition
timestamp, and `reason_code = "SOURCE_CAPTURED"`.
- `source_capture.origin` MUST be the canonical source origin used for operator
inspection and deduplication. Source Library `canonical_uri` takes precedence
over legacy URL, URI, thread, search, or repo-derived origins.
Expand Down Expand Up @@ -254,7 +257,7 @@ Persisted normalized `source_ref`:

- The stored `doc_documents.source_ref` MUST retain the caller-provided
`doc_source_ref/v1` fields and add normalized capture fields:
`source_record_id`, `origin`, `captured_at`, `content_hash`,
`source_record_id`, `lifecycle`, `origin`, `captured_at`, `content_hash`,
`visibility_scope`, `source_type`, and `source_spans`.
- When policy spans exist, stored `doc_documents.source_ref` MUST include
`policy_spans`.
Expand All @@ -270,6 +273,11 @@ Delete, export, and private-span boundary:
refs non-recallable. Derived pages may retain stored stale text until rebuild, but
page search MUST suppress snippets whose source refs no longer resolve to active
readable document or chunk rows.
- Source Library delete MUST preserve a tombstone lifecycle in
`doc_documents.source_ref.lifecycle` with `status = "deleted"`,
`freshness = "tombstoned"`, `reason_code = "SOURCE_LIBRARY_DELETE"`,
`deleted_at`, actor metadata, and a `tombstone_ref`. The tombstone is audit
evidence only and MUST NOT make deleted source spans recallable.
- `doc_source_span/v1` entries with `status = "excluded"` or `status = "redacted"`
are audit evidence for write-policy handling. They MUST NOT be treated as captured
source evidence for derived page search, memory promotion, graph facts, or export
Expand Down
13 changes: 10 additions & 3 deletions docs/spec/system_elf_memory_service_v2.md
Original file line number Diff line number Diff line change
Expand Up @@ -1160,14 +1160,15 @@ Body:
Behavior:
- `supersede` sets the note status to `deprecated`, writes a `DEPRECATE`
`memory_note_versions` row, stores `elf.memory_correction/v1` source-ref evidence,
and enqueues an indexing `DELETE` so normal recall suppresses the note.
includes `elf.memory_authority_lifecycle/v1` superseded metadata, and enqueues an
indexing `DELETE` so normal recall suppresses the note.
- `delete` sets the note status to `deleted`, writes a `DELETE`
`memory_note_versions` row, stores `elf.memory_correction/v1` source-ref evidence,
and enqueues an indexing `DELETE`.
includes tombstone lifecycle metadata, and enqueues an indexing `DELETE`.
- `restore` restores the latest prior active snapshot from a `DELETE` or `DEPRECATE`
version, or the supplied `restore_version_id`, writes a `RESTORE`
`memory_note_versions` row, stores `elf.memory_correction/v1` source-ref evidence,
and enqueues an indexing `UPSERT`.
includes rollback lifecycle metadata, and enqueues an indexing `UPSERT`.
- Correction actions require a non-empty reason and non-empty JSON object
`source_ref`. They do not mutate raw source notes, docs, events, traces, graph
facts, or source pointers.
Expand Down Expand Up @@ -1845,6 +1846,9 @@ Behavior:
writes doc chunks, and enqueues doc-index `UPSERT` jobs for derived Qdrant points.
- The request may include write-policy redactions or exclusions; excluded spans are
retained as policy metadata but are not captured source spans.
- The normalized document `source_ref` carries `elf.source_lifecycle/v1` metadata
with active/current state at capture time. The lifecycle object is source audit
metadata; it does not promote the source into Memory Ledger authority.
- This endpoint must not create Memory Ledger notes, graph facts, knowledge pages,
search traces, or recall hits.

Expand Down Expand Up @@ -1874,6 +1878,9 @@ Response:
Behavior:
- Marks the Source Library document `deleted` when the caller owns the document and
the document scope is writable.
- Updates the stored document `source_ref.lifecycle` to
`elf.source_lifecycle/v1` with `status = "deleted"`, `freshness = "tombstoned"`,
a delete actor, `deleted_at`, and a tombstone reference.
- Enqueues a doc-index `DELETE` job for every persisted document chunk so the worker
removes derived Qdrant doc-vector points.
- Repeating delete on an already deleted document returns `op = NONE`.
Expand Down
4 changes: 2 additions & 2 deletions makefiles/check.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Rust workspace tasks: Check.
# Rust workspace tasks: Checks.

[tasks.check]
[tasks.checks]
clear = true
workspace = false
dependencies = [
Expand Down
6 changes: 4 additions & 2 deletions packages/elf-service/src/docs.rs
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,8 @@ pub use api::{
DocsExcerptsGetRequest, DocsGetRequest, DocsGetResponse, DocsPutRequest, DocsPutResponse,
DocsSearchL0Item, DocsSearchL0ItemHashes, DocsSearchL0ItemLocator, DocsSearchL0ItemPointer,
DocsSearchL0ItemReference, DocsSearchL0ItemState, DocsSearchL0Request, DocsSearchL0Response,
DocsSourceCaptureSummary, DocsSourceSpanRef, TextPositionSelector, TextQuoteSelector,
DocsSourceCaptureSummary, DocsSourceLifecycle, DocsSourceSpanRef, TextPositionSelector,
TextQuoteSelector,
};

use std::{
Expand Down Expand Up @@ -65,7 +66,8 @@ use search_support::{
};
use source_capture::{
build_doc_chunk_rows, build_source_capture_summary, doc_chunk_id_for,
normalize_source_ref_for_capture, source_record_id_for, source_span_id,
normalize_source_ref_for_capture, source_record_id_for, source_ref_with_deleted_lifecycle,
source_span_id,
};
use types::{
ByteChunk, DEFAULT_DOC_MAX_BYTES, DEFAULT_L0_MAX_BYTES, DEFAULT_L1_MAX_BYTES,
Expand Down
5 changes: 4 additions & 1 deletion packages/elf-service/src/docs/api.rs
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,10 @@ pub use self::{
excerpts::{
DocsExcerptLocator, DocsExcerptResponse, DocsExcerptVerification, DocsExcerptsGetRequest,
},
put::{DocsPutRequest, DocsPutResponse, DocsSourceCaptureSummary, DocsSourceSpanRef},
put::{
DocsPutRequest, DocsPutResponse, DocsSourceCaptureSummary, DocsSourceLifecycle,
DocsSourceSpanRef,
},
read::{DocsDeleteRequest, DocsDeleteResponse, DocsGetRequest, DocsGetResponse},
search_l0::{
DocsSearchL0Item, DocsSearchL0ItemHashes, DocsSearchL0ItemLocator, DocsSearchL0ItemPointer,
Expand Down
25 changes: 25 additions & 0 deletions packages/elf-service/src/docs/api/put.rs
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,8 @@ pub struct DocsSourceCaptureSummary {
pub schema: String,
/// Stable source record identifier. This is also the stored `doc_id`.
pub source_record_id: Uuid,
/// Source-record lifecycle and freshness state at capture time.
pub lifecycle: DocsSourceLifecycle,
/// Canonical source origin used for operator inspection and deduplication.
pub origin: String,
/// RFC3339 timestamp when ELF captured the source.
Expand All @@ -73,6 +75,29 @@ pub struct DocsSourceCaptureSummary {
pub policy_spans: Vec<DocsSourceSpanRef>,
}

/// Source Library lifecycle metadata persisted with normalized source refs.
#[derive(Clone, Debug, Serialize)]
pub struct DocsSourceLifecycle {
/// Schema identifier for this lifecycle object.
pub schema: String,
/// Authoritative Source Library row status.
pub status: String,
/// Freshness label used by recall and audit surfaces.
pub freshness: String,
/// Actor that created the current lifecycle transition.
pub actor_agent_id: String,
/// Transition timestamp.
pub ts: String,
/// Machine-readable lifecycle reason.
pub reason_code: String,
#[serde(skip_serializing_if = "Option::is_none")]
/// Timestamp when the source was tombstoned or deleted.
pub deleted_at: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
/// Review or delete metadata that proves the tombstone.
pub tombstone_ref: Option<Value>,
}

/// Stable reference to one captured or policy-affected source span.
#[derive(Clone, Debug, Serialize)]
pub struct DocsSourceSpanRef {
Expand Down
4 changes: 2 additions & 2 deletions packages/elf-service/src/docs/service.rs
Original file line number Diff line number Diff line change
Expand Up @@ -20,8 +20,8 @@ use crate::{
load_doc_search_rows, load_docs_excerpt_context, load_tokenizer,
normalize_source_ref_for_capture, record_result_projection_stage,
resolve_doc_chunking_profile, run_doc_fusion_query, slice, source_record_id_for,
split_tokens_by_offsets, validate_docs_excerpts_get, validate_docs_put,
validate_docs_search_l0,
source_ref_with_deleted_lifecycle, split_tokens_by_offsets, validate_docs_excerpts_get,
validate_docs_put, validate_docs_search_l0,
},
search,
};
1 change: 1 addition & 0 deletions packages/elf-service/src/docs/service/put.rs
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,7 @@ impl ElfService {
source_ref: source_ref_map,
doc_type,
scope: scope.as_str(),
actor_agent_id: agent_id.as_str(),
title: title.as_deref(),
content_hash: content_hash.as_str(),
raw_content_hash: raw_content_hash.as_str(),
Expand Down
5 changes: 4 additions & 1 deletion packages/elf-service/src/docs/service/read.rs
Original file line number Diff line number Diff line change
Expand Up @@ -176,8 +176,11 @@ FOR UPDATE",
}

let chunks = docs::list_doc_chunks(&mut *tx, row.doc_id).await?;
let tombstoned_source_ref =
service::source_ref_with_deleted_lifecycle(&row.source_ref, agent_id, now)?;

docs::mark_doc_deleted(&mut *tx, tenant_id, row.doc_id, now).await?;
docs::mark_doc_deleted(&mut *tx, tenant_id, row.doc_id, &tombstoned_source_ref, now)
.await?;

for chunk in &chunks {
doc_outbox::enqueue_doc_outbox(
Expand Down
58 changes: 56 additions & 2 deletions packages/elf-service/src/docs/source_capture.rs
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,8 @@ mod helpers;

use crate::docs::{
ByteChunk, DOC_SOURCE_CAPTURE_SCHEMA_V1, DOC_SOURCE_SPAN_SCHEMA_V1, DocChunk, DocType,
DocsSourceCaptureSummary, DocsSourceSpanRef, Error, Map, OffsetDateTime, Result,
SourceCaptureSummaryInput, Uuid, Value,
DocsSourceCaptureSummary, DocsSourceLifecycle, DocsSourceSpanRef, Error, Map, OffsetDateTime,
Result, SourceCaptureSummaryInput, Uuid, Value,
};

pub(super) fn build_doc_chunk_rows(
Expand Down Expand Up @@ -77,6 +77,7 @@ pub(super) fn build_source_capture_summary(
source_ref,
doc_type,
scope,
actor_agent_id,
title,
content_hash,
raw_content_hash,
Expand Down Expand Up @@ -109,10 +110,21 @@ pub(super) fn build_source_capture_summary(
})
.collect();
let policy_spans = helpers::source_policy_spans(raw_content_hash, write_policy_audit);
let captured_at_for_lifecycle = captured_at.clone();

Ok(DocsSourceCaptureSummary {
schema: DOC_SOURCE_CAPTURE_SCHEMA_V1.to_string(),
source_record_id: doc_id,
lifecycle: DocsSourceLifecycle {
schema: "elf.source_lifecycle/v1".to_string(),
status: "active".to_string(),
freshness: "current".to_string(),
actor_agent_id: actor_agent_id.to_string(),
ts: captured_at_for_lifecycle,
reason_code: "SOURCE_CAPTURED".to_string(),
deleted_at: None,
tombstone_ref: None,
},
origin: helpers::source_origin(source_ref, doc_type),
captured_at,
content_hash: content_hash.to_string(),
Expand All @@ -136,6 +148,12 @@ pub(super) fn normalize_source_ref_for_capture(
"source_record_id".to_string(),
Value::String(source_capture.source_record_id.to_string()),
);
source_ref.insert(
"lifecycle".to_string(),
serde_json::to_value(&source_capture.lifecycle).map_err(|_| Error::InvalidRequest {
message: "Failed to serialize source lifecycle.".to_string(),
})?,
);
source_ref.insert("origin".to_string(), Value::String(source_capture.origin.clone()));
source_ref.insert("captured_at".to_string(), Value::String(source_capture.captured_at.clone()));
source_ref
Expand Down Expand Up @@ -164,3 +182,39 @@ pub(super) fn normalize_source_ref_for_capture(

Ok(Value::Object(source_ref))
}

pub(super) fn source_ref_with_deleted_lifecycle(
source_ref: &Value,
actor_agent_id: &str,
now: OffsetDateTime,
) -> Result<Value> {
let ts = helpers::format_timestamp(now)?;
let tombstone_ref = serde_json::json!({
"schema": "elf.source_tombstone/v1",
"reason_code": "SOURCE_LIBRARY_DELETE",
"actor_agent_id": actor_agent_id,
"ts": ts,
});
let lifecycle = DocsSourceLifecycle {
schema: "elf.source_lifecycle/v1".to_string(),
status: "deleted".to_string(),
freshness: "tombstoned".to_string(),
actor_agent_id: actor_agent_id.to_string(),
ts: ts.clone(),
reason_code: "SOURCE_LIBRARY_DELETE".to_string(),
deleted_at: Some(ts),
tombstone_ref: Some(tombstone_ref),
};
let mut source_ref = source_ref.as_object().cloned().ok_or_else(|| Error::InvalidRequest {
message: "source_ref must be a JSON object.".to_string(),
})?;

source_ref.insert(
"lifecycle".to_string(),
serde_json::to_value(lifecycle).map_err(|_| Error::InvalidRequest {
message: "Failed to serialize source lifecycle.".to_string(),
})?,
);

Ok(Value::Object(source_ref))
}
Loading