Linkml conversion tooling by yarikoptic · Pull Request #387 · dandi/dandi-schema

yarikoptic · 2026-03-20T21:59:17Z

This is an extract with amends from

WiP: Branch with auto converted linkml model #381

which (branch linkml-auto-converted) would keep merging this branch into itself while reflecting on changes in the branch which could be rebased or gain merges from the master, and also can accumulate or drop "patch branches" from within its script defining what to patch with.

This way linkml-auto-converted would represent reflection of current state of conversion

TODO/PLAN

Specify Hatch-managed env for auto converting `dandischema.models` to LinkML schema and back to Pydantic models

Provide script to translate `dandischema.models` in to a LinkML schema and overly it with definition provided by an overlay file.

Provide script to translate `dandischema/models.yaml` back to Pydantic models and store them in `dandischema/models.py`

These prefixes are copied from https://github.com/dandi/schema/blob/master/releases/0.7.0/context.json

The previous BRE pattern used `\+` (GNU sed extension) which silently fails on macOS BSD sed. Switch to `-E` (extended regex) with POSIX character class `[^[:space:]]` instead of `\S` (also unsupported by BSD sed), making the normalization work on both macOS and Linux. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Expand comment for linkml-auto-converted hatch env with usage instructions Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

There is no prefix defined as `dandi_default`. The intended default prefix is `dandi`

…ed and some symbols from _orig for now we do it so it does not overlay models.py since then git is unable to track renames

…into linkml-auto-converted

… flake8 issues

we had to maintain original filename for models.py to apply patches easily

yarikoptic · 2026-03-20T22:00:56Z

+# Poor man patch queue implementation
+# Edit this list if you want to merge or drop PRs branches to be patched with.
+# Order matters
+branches_to_merge=( remove-discriminated-unions )


that is where we define branches from PRs to merge!

codecov · 2026-03-20T22:10:49Z

Codecov Report

❌ Patch coverage is 0% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 48.66%. Comparing base (510f161) to head (36393de).

Files with missing lines	Patch %	Lines
dandischema/models_importstab.py	0.00%	4 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #387      +/-   ##
==========================================
+ Coverage   47.96%   48.66%   +0.70%     
==========================================
  Files          19       20       +1     
  Lines        2427     2431       +4     
==========================================
+ Hits         1164     1183      +19     
+ Misses       1263     1248      -15

Flag	Coverage Δ
unittests	`48.66% <0.00%> (+0.70%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Provide a partial schema to be merged with the generated schema

…nator `dandischema.models` use `schemaKey` in each Pydantic as a de facto type designator in LinkML. However, director translation to LinkML based on individual model's defintion is not possible. This override provided in the merge file completes the translation

Add LinkML behavior tests for required: False -> True slot_usage refinement

Collection is supposed to be confined to `dandischema/` via the literal positional arg in `commands = pytest -v {posargs} dandischema`. Under tox 4.25 (local) that works as intended. Under tox 4.54 (CI), however, the literal `dandischema` following `{posargs}` is dropped from the invocation, leaving pytest to default to rootdir-based collection. That made it walk into `tests/linkml_behavior/required_refinement/`, whose dependencies (PyYAML, linkml, etc.) live in the dedicated `linkml-behavior-test` hatch env rather than the tox env, and fail collection with `ModuleNotFoundError: No module named 'yaml'`. Set `testpaths = dandischema` in the `[pytest]` section so pytest's default-collection target matches the intended scope of this test suite. With that in place, the literal `dandischema` in the tox command is redundant — and was the source of the tox-version-dependent behavior — so drop it. A path explicitly supplied via `{posargs}` (e.g. `tox -e py -- tests/foo.py`) still overrides `testpaths` and is now honored cleanly instead of being combined with `dandischema`. Co-Authored-By: Claude Code 2.1.143 / Claude Opus claude-opus-4-7 <noreply@anthropic.com>

Resolve conflit in `tox.ini` regarding pytest settings

Introduce `docs/designs/migration_to_linkml_playbook/` as a living, self-updating playbook for the ongoing migration of `dandischema` from Pydantic-defined models to a LinkML-defined schema as the source of truth. The foundation covers the problem statement and success criteria, the current wiring of `./tools/linkml_conversion` (including the `2linkml` / `2pydantic` / `2json` / `pydantic2json` Hatch scripts and the role of each file under `tools/linkml_conversion_tools/`), a repeatable procedure with parity checks against the dandi-archive frontend, an inventory of the patch queue applied during translation (`master`, `remove-discriminated-unions`), and conventions for the `log.md` / `findings.md` / `tools/` / `context/` subfiles. More content will be added in subsequent commits. Co-Authored-By: Claude Code 2.1.154 / Claude Opus 4.7 <noreply@anthropic.com>

Introduce `context/roles/` in the playbook, with a mandatory `senior-developer.md` baseline (behavioral habits inherited by every agent and subagent) plus topical stubs for Vue, Django, and LinkML slices of the migration. Role files are loaded into a working agent's context when a session touches the corresponding slice; the same files also serve as spawn-prompt material when a subagent is invoked. Each topical stub names its scope, its explicit not-in-scope handoffs to sibling roles, and curated references for filling in its content (including first-party LinkML AGENTS.md / SKILL.md material in `linkml/linkml`, the LinkML specification at w3id.org, and community subagent collections vetted as starting material). OVERVIEW.md gains a pointer to `context/roles/` from the "How to use this directory" section. Co-Authored-By: Claude Code 2.1.154 / Claude Opus 4.7 <noreply@anthropic.com>

Fill in the "What this role needs to know" sections of `vue.md`, `django.md`, and `linkml.md` from primary sources: the local `dandi-archive` (frontend `package.json`, backend `pyproject.toml`, dev docs) and `pydantic2linkml` (README, CLAUDE.md, source). Each role names its stack landscape, the seam where dandischema crosses into that slice, and a curated lift from upstream community subagent definitions with caveats about version mismatches. Shared findings across roles surfaced: - @koumoul/vjsf is the form generator that consumes the JSON Schema in the dandi-archive frontend — the concrete consumer for criterion 3. - dandi-archive pins dandischema==0.12.1 exact; dandi-cli pins dandischema ~= 0.12.0 — the generated Pydantic must satisfy both. - `pydantic2linkml -M` is implemented via deepmerge.always_merger; the README's "values from the file win on conflict" oversimplifies. The actual per-type rule: dicts deep-merge, lists append, sets union, type-mismatches and scalars override. OVERVIEW.md's wiring and procedure step 7 are corrected to match, and a list-replacement escape hatch is documented (use `-O` for top-level lists, or fix on the Pydantic side). Co-Authored-By: Claude Code 2.1.154 / Claude Opus 4.7 <noreply@anthropic.com>

Add a self-contained, runnable exhibit under the playbook's tools/ directory demonstrating that LinkML's `designates_type: true` makes the generated Pydantic models and JSON Schema resolve a superclass-typed slot value to its concrete subtype, and pins each class's `schemaKey` to its class name consistently across both representations. The exhibit is organized into: - schemas/ — two source schemas (with/without the type designator) plus their generated Pydantic + JSON Schema snapshots - subtype_resolution/ — demos asserting a Project survives in a BareAsset.wasGeneratedBy list only with the designator on - schemakey_validation/ — demos asserting schemaKey validation (valid/absent/null/wrong-class) agrees between Pydantic and JSON Schema Record the finding in findings.md. Demos run in the linkml-auto-converted pipeline env (linkml 1.10.0). Exclude each exhibit's schemas/ folder from pre-commit, since the generated snapshots are raw gen-pydantic / gen-json-schema output that the linters should not touch. Co-Authored-By: Claude Code 2.1.159 / Claude Opus 4.8 <noreply@anthropic.com>

…tch env

The linkml-auto-converted pipeline env now pins linkml==1.11.1 (was 1.10.0). Regenerate the demo's Pydantic/JSON Schema snapshots under 1.11.1 and update the exhibit README + findings.md provenance accordingly. The finding is unchanged: the designates_type behavior is identical across both versions, and all four demos still pass. Co-Authored-By: Claude Code 2.1.159 / Claude Opus 4.8 <noreply@anthropic.com>

LinkML's gen-json-schema hardcodes draft 2019-09; Pydantic and the dandi-archive frontend's Ajv are both on 2020-12. The frontend deletes $schema before validating, so the gap is mostly cosmetic except for tuple arrays. Promote this into findings.md and partly answer the matching open question in OVERVIEW.md. Co-Authored-By: Claude Code 2.1.159 / Claude Opus 4.8 claude-opus-4-8 <noreply@anthropic.com>

…forces them Verified against the linkml-auto-converted pipeline env (linkml 1.11.1): a LinkML class rule (precondition value_presence PRESENT -> postcondition required/minimum_value) is translated to an if/then block by gen-json-schema and enforced, but gen-pydantic stores it only as inert linkml_meta metadata and the generated model accepts violating instances. Also notes the expressiveness limit: nested-slot and dict-key checks are not cleanly expressible via rule slot_conditions. Relevant to preserving dandischema's conditional (publish-only) validators across the migration without losing Pydantic-side enforcement. Co-Authored-By: Claude Code 2.1.161 / Claude Opus 4.8 claude-opus-4-8 <noreply@anthropic.com>

`ifabsent: null` had no effect: the `-M` deep merge inserts it as `ifabsent: None`, then `canonicalize_schema_yml` drops it as unset, so `models.yaml` is identical either way. The slot needs no `ifabsent` anyway, since `designates_type: true` already supplies the `schemaKey` default. Co-Authored-By: Claude Code 2.1.218 / Opus 4.8 (claude-opus-4-8) <noreply@anthropic.com>

The range of this property can't be translated by pydantic2linkml systematically because it is not a listed an allowed monotonic refinement defined within pydantic2linkml. This commit specifies the range of `Dandiset.wasGeneratedBy` through the `models_merge.yaml`

The range of this property can't be translated by pydantic2linkml systematically because it is not a listed an allowed monotonic refinement defined within pydantic2linkml. This commit specifies the range of `BareAsset.wasGeneratedBy` through the `models_merge.yaml`

`remove_notes_by_pattern.py` matched its patterns against every `notes` entry, so a pattern could not target one location without suppressing identical notes elsewhere. Replace the flat `PATTERNS` list with `REMOVALS`, pairing each regex with an optional tuple of paths. A path ending in `"notes"` designates that node alone; any other path is a subtree root. A rule without paths keeps the previous document-wide behavior, so the existing patterns are unaffected and the output is unchanged. Co-Authored-By: Claude Code 2.1.220 / Claude Opus 5 (claude-opus-5) <noreply@anthropic.com>

…ages The `range` of `BareAsset.slot_usage.wasGeneratedBy` and `Dandiset.slot_usage.wasGeneratedBy` is now supplied through `dandischema/models_merge.yaml`, so the note `pydantic2linkml` emits about being unable to express it is obsolete. Scope the removal to those two locations. Co-Authored-By: Claude Code 2.1.220 / Claude Opus 5 (claude-opus-5) <noreply@anthropic.com>

Adding a second topic directory under `tests/linkml_behavior/` collides with this one. Both pytest's default "prepend" import mode and `mypy` resolve the identically named modules in sibling topics (`_cases`, `test_validate`, ...) to the same top-level module name, and refuse to collect the second one. Add an `__init__.py` so those modules are namespaced by their topic, and import the shared case lists relatively to match. Co-Authored-By: Claude Code 2.1.220 / Claude Opus 5 (claude-opus-5) <noreply@anthropic.com>

…rides `dandischema/models_merge.yaml` respecifies the range of the inherited multivalued `wasGeneratedBy` slot through `slot_usage`, narrowing it to a subclass for `Dandiset` and widening it to `Any` plus an `any_of` for `BareAsset`. Add a `range_refinement` topic exercising both forms of respecification against all three validators the pipeline uses. Also record in `findings.md` that the `BareAsset` override is redundant, since `designates_type: true` already expands `range: Activity` over its descendants, along with the intent behind keeping it. Co-Authored-By: Claude Code 2.1.220 / Claude Opus 5 (claude-opus-5) <noreply@anthropic.com>

…ftests Both LinkML-behavior topics generated their JSON schemas, JSON instances and Pydantic models with byte-identical fixture bodies, differing only in the schema, class and instance names they fed in. Move that generation into `tests/linkml_behavior/_generation.py`, leaving each `conftest.py` as its own data plus thin fixtures delegating to it. The fixtures stay topic-local rather than moving to a parent `conftest.py`, which would share one `FixtureDef` across topics and, at session scope, hand the first topic's artifacts to every later one. Co-Authored-By: Claude Code 2.1.220 / Claude Opus 5 (claude-opus-5) <noreply@anthropic.com>

…modules The `test_*.py` modules were identical across topics apart from their docstrings, each repeating the same subprocess invocation and assertion message. Move those into `tests/linkml_behavior/_validators.py`, one `assert_*` helper per validator, leaving each test module its docstring, case lists and `@pytest.mark.parametrize` decorators. Co-Authored-By: Claude Code 2.1.220 / Claude Opus 5 (claude-opus-5) <noreply@anthropic.com>

The playbook described these as JSON Schemas from the original Pydantic models, kept as the parity baseline. They are not: `linkml_conversion` runs `pydantic2json` after the rename dance, so `publish_model_schemata` resolves `dandischema.models` to the import stub and serializes the generated models instead. The script itself is neutral about which models it serializes; the ordering decides. Correct the three affected claims, point the parity baseline at the schemata published in `dandi/schema`, and record the correction in `log.md`. Co-Authored-By: Claude Code 2.1.220 / Claude Opus 5 (claude-opus-5) <noreply@anthropic.com>

The notes are no longer necessary since #419. Also handle in `find_schemakey_mismatches.py` the re-exposure of a class under an alias. Addresses the `MANUAL_NOTE` sub-task of #389. Co-Authored-By: Claude Code 2.1.224 / Claude Opus 5 <noreply@anthropic.com>

`models.yaml` no longer defines them since #419 consolidated them into `Dandiset` and `Asset`, so `gen-json-schema` emitted rootless schemas that accept any object. The filename `sed` existed only for those two targets and goes with them. Co-Authored-By: Claude Code 2.1.224 / Claude Opus 5 <noreply@anthropic.com>

`publish_model_schemata` resolves `SCHEMA_MAP` keys with `getattr`, so the stub needs the aliases #419 kept in `dandischema/models.py`. Temporary: remove them after the #419 follow-up that drops those aliases. Co-Authored-By: Claude Code 2.1.224 / Claude Opus 5 <noreply@anthropic.com>

`2json` no longer targets them, and the log entry in 09730f8 missed that the phantom classes were load-bearing for `2json` and `publish_model_schemata`. Co-Authored-By: Claude Code 2.1.224 / Claude Opus 5 <noreply@anthropic.com>

The finding attributed both halves of the slot usage to `dandischema/models_merge.yaml`. That file supplies only `range: Any`; `pydantic2linkml` emits the `any_of` from the Pydantic annotation. The finding's conclusion is unaffected. Co-Authored-By: Claude Code 2.1.223 / claude-opus-5 <noreply@anthropic.com>

candleindark and others added 17 commits March 13, 2026 17:01

build(hatch-env): specify env for conversion to LinkML and back

cffee39

Specify Hatch-managed env for auto converting `dandischema.models` to LinkML schema and back to Pydantic models

build(hatch-env): provide script to translate dandischema.models

b5644c1

Provide script to translate `dandischema.models` in to a LinkML schema and overly it with definition provided by an overlay file.

build(hatch-env): provide script to translate dandischema/models.yaml

cfe3bea

Provide script to translate `dandischema/models.yaml` back to Pydantic models and store them in `dandischema/models.py`

Now we will sanitize the converted model addresses

73c46b6

more sanitization

c79b1e2

feat: provide prefixes definition in the overlay file

0b2b51d

These prefixes are copied from https://github.com/dandi/schema/blob/master/releases/0.7.0/context.json

Added linkml: prefix and sorted all the entries

6cb2f82

Provide default prefix to be custom dandi_default

76526a9

fix: correct default_prefix to dandi

fad6879

There is no prefix defined as `dandi_default`. The intended default prefix is `dandi`

Adding models_importstab.py so we could use it to import from convert…

5f78b87

…ed and some symbols from _orig for now we do it so it does not overlay models.py since then git is unable to track renames

Adding the tool to assist in establishing "merges" with regeneration …

0208c9c

…into linkml-auto-converted

We are doomed to use --no-verify for now as converted python code has…

488e2e9

… flake8 issues

BF the commit version specification

0ba5ad7

Implement poor man patch queue

2faf932

reflect that I rewritten how we handle patching

c177479

we had to maintain original filename for models.py to apply patches easily

Inform on what we are doing

54cf276

yarikoptic commented Mar 20, 2026

View reviewed changes

Now do patch diff of master in as well for good measure

5eb4a91

Removing reporting branches - did not work

76a9d77

yarikoptic mentioned this pull request Mar 20, 2026

WiP: Branch with auto converted linkml model #381

Draft

yarikoptic and others added 4 commits March 20, 2026 18:25

ENH: patch converted to pydantic model to strip dciteCOLO

c6172c4

do pre-commit

e2d124c

feat: provide partial schema

d8b1bea

Provide a partial schema to be merged with the generated schema

feat: enable 2linkml to merge partial schema in models_merge.yaml

c0fbd02

candleindark force-pushed the linkml-conversion branch 2 times, most recently from 59b0587 to c0fbd02 Compare March 31, 2026 00:59

yarikoptic commented Mar 31, 2026

View reviewed changes

Comment thread pyproject.toml Outdated

candleindark and others added 3 commits May 15, 2026 11:31

Merge pull request #408 from dandi/gh-405-linkml-behavior-test

f62f2e9

Add LinkML behavior tests for required: False -> True slot_usage refinement

merge master branch

c931794

Resolve conflit in `tox.ini` regarding pytest settings

candleindark force-pushed the linkml-conversion branch from 93b6b29 to c931794 Compare May 28, 2026 00:56

candleindark and others added 26 commits May 28, 2026 15:11

Merge branch 'master' into linkml-conversion

30b02b0

build(hatch env): update the LinkML pin in linkml-auto-converted ha…

041a280

…tch env

Merge branch 'master' into linkml-conversion

c3b2e1d

Merge branch 'master' into linkml-conversion

35f049a

Navigation Menu

Uh oh!

Linkml conversion tooling - #387

Linkml conversion tooling#387
yarikoptic wants to merge 96 commits into
masterfrom
linkml-conversion

yarikoptic commented Mar 20, 2026 •

edited

Loading

Uh oh!

yarikoptic Mar 20, 2026

Uh oh!

codecov Bot commented Mar 20, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

yarikoptic commented Mar 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yarikoptic Mar 20, 2026

Choose a reason for hiding this comment

Uh oh!

codecov Bot commented Mar 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

yarikoptic commented Mar 20, 2026 •

edited

Loading

codecov Bot commented Mar 20, 2026 •

edited

Loading