Skip to content

fix(learner): eliminate false-positive proposals (structural error detection + by-design file exclusions + trigram filter) - #29

Merged
Luispitik merged 1 commit into
Luispitik:mainfrom
juanparisma:fix/session-learner-false-positives
Jul 27, 2026
Merged

fix(learner): eliminate false-positive proposals (structural error detection + by-design file exclusions + trigram filter)#29
Luispitik merged 1 commit into
Luispitik:mainfrom
juanparisma:fix/session-learner-false-positives

Conversation

@juanparisma

Copy link
Copy Markdown
Contributor

Problema

En una sesión real del 2026-07-16, el session-learner generó 190 proposals de las cuales 189 eran basura. Tres causas raíz, verificadas contra las observaciones reales:

  1. error_resolution fantasmaobserve_v3.py marcaba is_error=True si el output contenía la subcadena error/failed en cualquier parte, incluido el contenido de archivos leídos con éxito (p. ej. código fuente con throw new Error). Prueba: en las proposals falsas, err_msg era idéntico a sample_output y no contenía ningún error.
  2. user_correction falsas — contaba como corrección cualquier archivo editado 3+ veces, pero los diarios/paneles de control (hot.md, brief.md, working-memory.md, daily summaries, wikis) se re-editan por diseño en los flujos de cierre (/eod). Una bitácora con 11 ediciones ≠ 10 correcciones del usuario.
  3. workflow_chain ruido — proponía cualquier trigrama de tools repetido 2+ veces; eso lo domina la actividad genérica de programar (Bash>Bash>Bash ×380 en una sesión → 154 proposals inútiles).

Fix

  • observe_v3.py: detección de errores (a) estructural para toda tool (dict con campo error, marcador tool_use_error, output que EMPIEZA con Error/InputValidationError) + (b) marcadores duros (Permission denied, command not found, Traceback, npm ERR!…) solo para Bash, cuyo output es resultado de ejecución y no contenido arbitrario.
  • _session-learner.sh:
    • isRealError(): re-valida el flag is_error contra marcadores duros — necesario porque las observaciones ya escritas por observers viejos siguen contaminadas y el learner relee esa ventana.
    • user_correction: excluye archivos re-editados por diseño (BYDESIGN_RE).
    • workflow_chain: exige 5+ repeticiones, ≥2 tools distintas y ≥1 tool no-genérica (MCP/custom).
    • PATTERN 5 (agent errors): usa isRealError en vez de sniffing de keywords.

Verificación

  • 6 tests unitarios del observer con payloads sintéticos: los 3 casos de falso positivo (Read con throw new Error, Bash exitoso, Grep sobre logs con "error") dan False; los 3 errores reales (Permission denied, String to replace not found, File does not exist) dan True.
  • A/B sobre una ventana real de 8.000 observaciones: 134 proposals con el código viejo → 17 con el fix, y las 17 supervivientes son señales legítimas (3 fallos reales de tools, 7 archivos fuente iterados de verdad, 7 secuencias MCP repetidas 5-14×).

Corriendo en producción en mis 2 PCs desde hoy sin regresiones.

🤖 Generated with Claude Code

…etection, by-design file exclusions, workflow trigram filter

One real session produced 190 proposals of which 189 were junk. Three
root causes, all fixed:

1. observe_v3.py flagged is_error=True whenever the output text contained
   "error"/"failed" keywords — including inside the CONTENT of successful
   Reads (e.g. source code with 'throw new Error'). Proof: err_msg was
   identical to sample_output on the junk proposals. Now detection is
   (a) structural harness failure shapes for every tool (dict error field,
   tool_use_error marker, start-of-output Error/InputValidationError) and
   (b) hard failure markers (Permission denied, command not found,
   Traceback, npm ERR!, ...) only for Bash, whose output is a run result
   rather than arbitrary file content.

2. user_correction counted any file edited 3+ times, but journals and
   control panels (hot.md, brief.md, working-memory.md, MEMORY.md,
   CLAUDE.md, daily summaries, wiki dirs) are re-edited BY DESIGN during
   end-of-day flows. They are now excluded via BYDESIGN_RE.

3. workflow_chain proposed every tool trigram repeated 2+ times, which is
   dominated by generic coding activity (Bash>Bash>Bash x380 in one
   session, 154 junk proposals). Trigrams now require 5+ repeats, >= 2
   distinct tools, and at least one non-generic (MCP/custom) tool.

The learner also re-validates the legacy is_error flag against hard
markers (isRealError) because observations written by older observers
remain contaminated, and PATTERN 5 (agent errors) uses the same
validation instead of keyword sniffing.

A/B on a real 8000-line observation window: 134 proposals with the old
code -> 17 with this fix, and the 17 survivors are all legitimate signals
(3 genuine tool failures, 7 genuinely iterated source files, 7 repeated
MCP workflow sequences).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@Luispitik
Luispitik merged commit 0ec5e1b into Luispitik:main Jul 27, 2026
3 checks passed
@Luispitik

Copy link
Copy Markdown
Owner

Merged y publicado en v4.9.0. Gracias, y perdona la espera desde el 16 de julio.

Tu diagnóstico era el correcto y era el difícil de ver: el observador marcaba is_error por el contenido de ficheros leídos con éxito, así que leer código con un throw new Error dentro contaba como fallo de herramienta. Medido sobre una ventana real de 8.000 observaciones, tu fix baja de 1.074 observaciones marcadas a 22.

Encima de lo tuyo he añadido una cosa, y quiero que sepas por qué. Restringir los marcadores duros a Bash era correcto para matar los falsos positivos, pero dejaba sin detectar los fallos reales de Edit y Read (String to replace not found, File does not exist): no tienen forma de harness, así que la rama estructural tampoco los cogía. La detección para esas herramientas pasa a mirar la forma — un fallo devuelve un veredicto en texto plano, un éxito devuelve un payload que serializa empezando por { o [, y los payloads no se escanean nunca, que es justamente el bug que tu PR arregla. Cero falsos positivos sobre 7.931 observaciones no-Bash reales.

O sea que tu regla se mantiene entera; lo que se añade es el caso que quedaba al otro lado. Si al verlo crees que se me ha ido la mano en algún borde, dilo y lo miramos.

juanparisma pushed a commit to juanparisma/sinapsis that referenced this pull request Jul 31, 2026
…e diagnostics

Rounds out the two contributed PRs (Luispitik#29 juanparisma, Luispitik#22 Sergio-LPA) and clears
the debt they surfaced.

Non-Bash tool failures were flagged by nobody. Luispitik#29 correctly stopped scanning
file CONTENT for error words, but restricting hard markers to Bash left real
Edit/Read failures ("String to replace not found", "File does not exist")
undetected — they are not harness-shaped either, so the structural branch also
missed them. Detection now keys on shape: a failure returns a bare verdict
string, a success returns a payload serialising from { or [, and payloads are
never scanned — that scan is the bug Luispitik#29 exists to fix. Bounded by length as
well. Zero false positives over 7,931 real non-Bash observations.

The learner discarded its own diagnostics. The node block ended in 2>/dev/null,
swallowing lock-contention and failed-upsert messages and overriding the
SINAPSIS_DEBUG redirect, so the debug switch produced nothing for the 95% of the
script inside that block. Diagnostics append to _session-learner.log.

Secret scrubbing had never been tested. Test Group 4 imported a script that
cannot be imported, then looked for a function that does not exist; every
assertion returned LOAD_FAIL and printed as SKIP, so "11/11 passed" covered
nothing. It now feeds the hook a real secret and asserts what reached disk,
across five formats, with a control proving the scrubber is not simply
destroying all output.

The fixtures are synthetic — nothing here was ever a live credential, and two
of them are the vendors' own published examples — but each prefix is assembled
at runtime so no complete token-shaped literal lands in the tree. A scanner
cannot tell a fixture from a leak: written out whole, these block the push and
scare anyone who greps the repo. What reaches the observer is byte-identical.

New tests/test-error-detection.sh (10 tests) pins both halves of the fix.
Documented for future test authors: the observer resolves its config dir via
expanduser("~"), which on Windows reads USERPROFILE, not HOME — redirecting HOME
alone leaks synthetic observations into real learning data.

Docs: CHANGELOG v4.9.0, README header/badges/What's New, installer banners, and
FEATURES notes on the two accepted limits of the cross-OS key (posix collapses
macOS with Linux; the name key can fuse unrelated same-named projects, which is
why it now stamps and logs).

Suite: 217 tests across 16 files, all registered in CI.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants