Skip to content

Remap all v1 templates to the consolidated v3.0.0 skill names (hold for release day)#98

Merged
cafzal merged 2 commits into
mainfrom
skills-v3-runbook-remap
Jul 10, 2026
Merged

Remap all v1 templates to the consolidated v3.0.0 skill names (hold for release day)#98
cafzal merged 2 commits into
mainfrom
skills-v3-runbook-remap

Conversation

@cafzal

@cafzal cafzal commented Jul 7, 2026

Copy link
Copy Markdown
Collaborator

What

Prepares the templates repo for the RAI skills v3.0.0 release, which consolidates 15 skills into 11. Every skill invocation and prose mention under v1/ — runbook slash commands, README notes, script comments — moves to the new names.

Do not merge until the v3.0.0 skills release ships to the public repo. The old skill names keep working until then; the new ones don't exist for customers yet. This PR ships the same day as the release.

Name map

Old New
rai-querying, rai-rules-authoring, rai-pyrel-coding rai-pyrel
rai-build-starter-ontology, rai-ontology-design rai-ontology
rai-prescriptive-problem-formulation rai-prescriptive-problem
rai-prescriptive-solver-management (solver selection, pre-solve) rai-prescriptive-problem
rai-prescriptive-solver-management (solve execution, scenarios, diagnostics) rai-prescriptive-results
rai-prescriptive-results-interpretation rai-prescriptive-results

Unchanged: rai-setup, rai-discovery, rai-graph-analysis, rai-predictive-modeling, rai-predictive-training, rai-deployment, rai-health.

Beyond the rename

  • Combined invocations collapse: problem-formulation + solver-management steps become a single /rai-prescriptive-problem step; solver-management + results-interpretation scenario steps become a single /rai-prescriptive-results step.
  • In commercial_underwriting and shipment_compliance the merge left a rules step immediately followed by a query step that only read the derived flags back; each pair collapses into one /rai-pyrel step and the surrounding prose is reworded.
  • ASCII stage diagrams re-aligned where shorter names shifted columns (defect_root_cause, entity_resolution, telco_network_recovery).
  • Skill-content references verified against the consolidated tree: rai-prescriptive-problem/scenario-analysis.md (cited in supply_chain_resilience.py), rai-pyrel data-loading guidance (cited in two READMEs), TIME_LIMIT-is-signal guidance now under rai-prescriptive-results.

Validation

Applied the dev-templates-review checklist to the diff: chain ASCII stage labels carry the new skill names with columns re-aligned, merged prompts stay question-shaped with definitions inline, Responses stay single-paragraph, no step-number cross-references introduced, headline numbers untouched. Touched scripts pass py_compile and ruff.

Paste-test sweep results

Ran the full paste-test sweep once the consolidated v3.0.0 skills library locked (claude/skill-streamlining @ 88f7121). Protocol: fresh Sonnet agents, Snowflake engines, one runbook prompt delivered per turn in sequence — no read-ahead, no injected model/solver names, schemas, or expected answers; the only context beyond each prompt's own verbatim text is the runbook's own run-in-order preface. Scoped as a regression test: each template ran through its last step that touches a modified skill (rai-ontology, rai-pyrel, rai-prescriptive-problem, rai-prescriptive-results), including genuine prerequisite steps, skipping trailing steps that only exercise unchanged skills (pure graph/predictive/discovery tails with nothing downstream).

46 of 48 templates reached a full PASS, with zero hard failures anywhere in the sweep. The regression unit is the runbook step, not the template: every one of the 187 modified-skill steps that ran passed — 0 failures, 0 regressions (187/187), each objective/count cross-checked against the solver or an independent numpy/pandas oracle. Per skill: rai-ontology 49/49, rai-pyrel 63/63, rai-prescriptive-problem 39/39, rai-prescriptive-results 36/36. (A further 10 modified-skill steps in the corpus went unrun — not failed — almost all the terminal/build steps of the two GNN-blocked templates.) The remaining 2 templates (subscriber_retention, telco_network_recovery) are environment-scope-limited: their final modified-skill step genuinely requires a GNN prediction as input, this environment's relationalai build lacks the [gnn] runtime extra, and neither ships a precomputed-prediction fallback CSV — an environment gap, not a template or skill defect. (Several templates' background agents were killed twice by session interrupts mid-run and relaunched from scratch; the final PASS numbers are from completed clean runs.) Zero PASS templates required a skill-name fix — the remap itself holds up under real execution.

Runbook-prompt issues found were fixed directly in this PR (8 files): named the restricted/marginal industry codes in commercial_underwriting; made the persist step explicit in bom-reachability; quantified the flow-conservation tolerance in money_laundering_motif_detection; de-overloaded "relationships" in fraud-detection; named exact cost weights in defect_root_cause (previously unspecified weights could name different real-world root-cause factors on different runs); named the SPOF disambiguation criterion and allowed for ties in two memory_supply_allocation prompts; and stated the discount-monotonicity rule explicitly in retail_planning (matching retail_markdown's existing wording). Two other suspected issues were investigated and retracted after checking the real template scripts/data: an apparent memory_supply_allocation baseline infeasibility turned out to be a test-agent implementation slip (mixed up which Dependency-edge role feeds which property, not a runbook or skill defect), and a suspected retail_planning Article/PlanningArticle id mismatch turned out to be a potential Snowflake data-provisioning question, not a prompt-wording gap — flagging that one to the data team rather than editing the prompt.

Skill-content findings (not actionable in this repo — reported to the skills team): two correctness-class findings worth prioritizing before release — in-model aggregation over Scenario-indexed decision variables can silently return a wrong total with no error (demand_planning_temporal), and selecting two aggregate-derived properties together can silently return a different value for one of them than selecting it alone (fraud-detection). Also: a NaN in any column of a multi-column model.data(df) silently drops the whole row from entity creation even when only a non-null column is referenced (datacenter_compute_allocation, supply_chain_resilience), and a Python-level variable/label matching a concept's attribute name can silently mint a real server-side property when implicit_properties is on (datacenter_compute_allocation). Plus a handful of lower-severity friction points (.to_schema() column access, solve_for(where=...) needing an inline literal, pre-solve decision-variable queries returning 0 rows, .alias() Property-vs-Relationship behavior, model.union() branch-arity mismatch) where agents self-corrected from the skill's own reference material. Full writeup: dev_temp/v3_skills_sweep_skill_issues.md.

Skill-file coverage (from real Read calls across all agent transcripts, not estimated): rai-ontology 9/18, rai-pyrel 16/33, rai-prescriptive-problem 35/55, rai-prescriptive-results 7/20 (thinnest — untouched files cluster around diagnostics/conflict-IIS/failure-taxonomy and Pareto/multi-objective paths that no template in this set happened to exercise). The per-file breakdown of which reference/example files were exercised vs. never opened is captured on the skills PR (RelationalAI/rai-agent-evals#157) as input for a follow-on skill-streamlining pass.

Skills release v3.0.0 consolidates 15 skills to 11. Every skill
invocation and prose mention under v1/ moves to the new names:

- rai-querying, rai-rules-authoring, rai-pyrel-coding -> rai-pyrel
- rai-build-starter-ontology, rai-ontology-design -> rai-ontology
- rai-prescriptive-problem-formulation -> rai-prescriptive-problem
- rai-prescriptive-results-interpretation -> rai-prescriptive-results
- rai-prescriptive-solver-management splits by context: solver
  selection / pre-solve (with formulation) -> rai-prescriptive-problem;
  solve execution / scenarios / diagnostics -> rai-prescriptive-results

Combined invocations collapse accordingly: 'problem-formulation +
solver-management' steps become a single /rai-prescriptive-problem
step; 'solver-management + results-interpretation' scenario steps
become a single /rai-prescriptive-results step. In
commercial_underwriting and shipment_compliance the merge left a
rules step immediately followed by a query step that only read the
derived flags back, so each pair collapses into one /rai-pyrel step.
ASCII stage diagrams re-aligned where the shorter names shifted
columns.
Named restricted/marginal industry codes (commercial_underwriting), made
the persist step explicit (bom-reachability), quantified the flow-balance
tolerance (money_laundering_motif_detection), de-overloaded "relationships"
(fraud-detection), named exact cost weights (defect_root_cause), named the
SPOF criterion and allowed ties (memory_supply_allocation), and stated the
discount-monotonicity rule (retail_planning). Each was reproducible: a
different agent given the same ambiguous prompt could plausibly reach a
different, equally-defensible answer.
@cafzal cafzal marked this pull request as ready for review July 10, 2026 17:47
@cafzal cafzal merged commit ac73bac into main Jul 10, 2026
2 of 3 checks passed
@cafzal cafzal deleted the skills-v3-runbook-remap branch July 10, 2026 19:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant