Add direct HPOA, OMIM, and Orphanet ingests

[cbizon]

Summary

• Add a direct HPOA parser for phenotype.hpoa disease-phenotype edges and genes_to_phenotype.txt disease-conditioned gene-phenotype edges.
• Add a direct OMIM parser for mim2gene_medgen gene-condition edges.
• Add a direct Orphanet parser for en_product6.xml assessed gene-condition edges.
• Register HPOA, OMIM, and Orphanet in ORION source metadata and the Baseline graph. RobokopKG and RoboMouseKG inherit these through Baseline; no Automat graph is added.
• Filter replaced HPOA-mediated phenotype content and Monarch-projected OMIM gene-disease content out of MonarchKG while leaving MonarchKGFull unchanged.
• Add a comparison script for old-vs-new aggregate reporting by source, predicate, knowledge source, aggregator, and subject/object prefix pair.

Parser semantics

• HPOA disease-phenotype emits disease biolink:has_phenotype phenotype with primary_knowledge_source=infores:hpo-annotations.
• HPOA disease-phenotype keeps only positive phenotype annotations: aspect == P, empty qualifier, and non-zero frequency. It skips course, inheritance, modifier, clinical modifier/history rows, NOT, 0/N, and 0% rows.
• HPOA gene-phenotype emits gene biolink:has_phenotype phenotype only when the (disease_id, hpo_id) pair survived the disease-phenotype filter. These edges preserve disease_context_qualifier so they remain disease-conditioned.
• HPOA preserves raw fields including disease ID/name, evidence, reference, onset, frequency, sex, modifier, aspect, and biocuration. Disease ID provenance is retained through supporting_data_source such as infores:omim, infores:orphanet, or infores:decipher.
• OMIM keeps type == phenotype rows with a GeneID and emits NCBIGene:<GeneID> biolink:gene_associated_with_condition OMIM:<MIM> with primary_knowledge_source=infores:omim.
• Orphanet keeps only DisorderGeneAssociationStatus == Assessed, skips Candidate gene tested in and Biomarker tested in, requires an HGNC external reference, and emits HGNC:<id> biolink:gene_associated_with_condition ORPHA:<OrphaCode> with primary_knowledge_source=infores:orphanet.

Data DAGs

Current / old path into Baseline

flowchart LR
    OMIM["OMIM\nGene-Disease assertions"]
    ORPHA["Orphadata / Orphanet\nGene-Disease assertions"]
    DECIPHER["DECIPHER\nDisease-Phenotype assertions"]

    PHENO["phenotype.hpoa\nCurated Disease-Phenotype table"]
    G2P["genes_to_phenotype.txt\nDerived disease-conditioned Gene-Phenotype table"]

    MONARCH["Monarch KG\nHPOA-derived Disease-Phenotype edges\nHPOA-derived Gene-Phenotype edges\nOMIM Gene-Disease edges"]

    BASE["ORION Baseline\nHPOA Disease-Phenotype edges\nHPOA Gene-Phenotype edges\nOMIM Gene-Disease edges"]

    DECIPHER --> PHENO
    PHENO --> G2P
    OMIM --> G2P
    ORPHA --> G2P
    OMIM --> MONARCH

    PHENO --> MONARCH
    G2P --> MONARCH
    MONARCH --> BASE

Loading

New path into Baseline

flowchart LR
    OMIM["OMIM\nGene-Disease assertions"]
    ORPHA["Orphadata / Orphanet\nGene-Disease assertions"]
    DECIPHER["DECIPHER\nDisease-Phenotype assertions"]

    PHENO["phenotype.hpoa\nCurated Disease-Phenotype table"]
    G2P["genes_to_phenotype.txt\nDerived disease-conditioned Gene-Phenotype table"]

    BASE["ORION Baseline\nHPOA Disease-Phenotype edges\nHPOA Gene-Phenotype edges\nOMIM Gene-Disease edges\nOrphanet Gene-Disease edges"]

    DECIPHER --> PHENO
    PHENO --> G2P
    OMIM --> G2P
    ORPHA --> G2P

    PHENO --> BASE
    G2P --> BASE
    OMIM --> BASE
    ORPHA --> BASE

Loading

Interpretation:

• phenotype.hpoa is the curated HPOA Disease-Phenotype table. It includes disease-phenotype rows whose disease identifiers include OMIM, Orphanet, and DECIPHER records.
• genes_to_phenotype.txt is a derived HPOA Gene-Phenotype table, conditioned on disease context.
• In the old path, ORION received HPOA Disease-Phenotype, HPOA Gene-Phenotype, and Monarch-projected OMIM Gene-Disease edge families through Monarch KG.
• In the new path, ORION receives the HPOA edge families directly from HPOA, receives OMIM Gene-Disease edges directly from mim2gene_medgen, and receives Orphanet Gene-Disease edges directly from en_product6.xml.
• The old path had no clean direct OMIM or Orphanet Gene-Disease ingest into Baseline. The old OMIM Gene-Disease edges were a Monarch projection with primary_knowledge_source=infores:monarchinitiative and provided_by=omim_gene_to_disease_edges.

Why the direct ingest adds edges relative to Monarch

This is a semantic replacement of Monarch-mediated HPOA/OMIM/Orphanet content, not a count-preserving replay of Monarch's projected edges.

• Direct HPOA preserves disease-conditioned gene-phenotype rows. The current HPOA file yields 311,865 gene-phenotype rows but only 255,797 unique gene/HP pairs. The additional 56,068 rows are the same gene/phenotype association under different disease contexts. We keep those distinct with disease_context_qualifier, and ORION's merge key treats qualifiers as part of edge identity.
• Monarch appears to be a lossy or transformed projection of the HPOA files. The inspected current graph had 187,560 HPOA-primary gene-ish has_phenotype edges; direct HPOA has 311,865 disease-conditioned gene-phenotype rows. Even after ignoring disease-context multiplicity, direct HPOA has about 68k more unique raw gene/HP pairs.
• Raw direct IDs do not compare exactly to the built graph. Direct HPOA emits OMIM/ORPHA/DECIPHER -> HP; the current built graph has many of those normalized through Monarch as MONDO -> HP or MONDO -> MONDO. Exact raw-CURIE overlap therefore overstates novelty.
• Direct OMIM mostly replaces Monarch's OMIM Gene-Disease projection. The inspected current graph has 6,550 biolink:causes edges and 601 biolink:contributes_to edges with primary=infores:monarchinitiative and provided_by=omim_gene_to_disease_edges. After mapping direct OMIM disease IDs through current graph node equivalences, 6,846 of 7,129 unique Monarch OMIM-derived pairs overlap with the new direct OMIM output.
• By predicate, the overlap is 6,352 / 6,550 for Monarch biolink:causes and 514 / 601 for Monarch biolink:contributes_to. Both predicate blocks are filtered from MonarchKG in this PR.
• OMIM and Orphanet also add direct gene-condition edges by design: 7,661 NCBIGene -> OMIM edges from mim2gene_medgen and 7,940 HGNC -> ORPHA edges from Orphadata. These replace/supersede Monarch projections where present, rather than preserving Monarch's stronger or inconsistent predicates.

Expected Baseline effect

Current graph aggregate counts for HPOA-mediated Monarch content were:

• 187,560 biolink:has_phenotype edges with primary=infores:hpo-annotations; this included 132,495 gene-phenotype-ish edges plus 54,998 NCBIGene -> MONDO and small HGNC/UMLS cases.
• 139,314 disease-phenotype-ish biolink:has_phenotype edges with primary=infores:omim and HPOA/Monarch aggregation.
• 113,944 disease-phenotype-ish biolink:has_phenotype edges with primary=infores:orphanet and HPOA/Monarch aggregation.
• 6,550 OMIM Gene-Disease biolink:causes edges with primary=infores:monarchinitiative and provided_by=omim_gene_to_disease_edges.
• 601 OMIM Gene-Disease biolink:contributes_to edges with primary=infores:monarchinitiative and provided_by=omim_gene_to_disease_edges.

Current-source smoke parses from the direct loaders produce:

• HPOA: 575,345 biolink:has_phenotype edges total.
• HPOA disease-phenotype: 263,480 edges by raw prefix pair OMIM->HP, ORPHA->HP, and DECIPHER->HP.
• HPOA disease-conditioned gene-phenotype: 311,865 NCBIGene->HP rows; 255,797 unique NCBIGene/HP pairs before preserving disease context.
• OMIM: 7,661 NCBIGene->OMIM biolink:gene_associated_with_condition edges.
• Orphanet: 7,940 HGNC->ORPHA biolink:gene_associated_with_condition edges.
• Combined direct replacement output: 590,946 edges.

Relative to the inspected current aggregate Monarch/HPOA counts, this replaces about 440,818 Monarch-mediated phenotype edges with 575,345 direct HPOA phenotype edges, replaces 7,151 Monarch-projected OMIM Gene-Disease edges with 7,661 direct OMIM Gene-Disease edges, and adds 7,940 direct Orphanet Gene-Disease edges before graph merge/normalization effects.

Validation

• uv run pytest passed: 162 passed.
• Live smoke parses matched the expected current-source counts for HPOA, OMIM, and Orphanet.
• scripts/compare_hpoa_rationalization.py was run against the smoke outputs and reports counts by source, predicate, primary source, supporting source, aggregator source, prefix pair, and edge family.

Notes

• Includes a small MONDOProperties compatibility fix for the installed pyoxigraph API so default graph-spec loading can import all registered source classes during tests.

[eKathleenCarter]

While reviewing example edges from the HPOA parser, found several properties that were redundant or dead weight:

• hpoa_aspect — always "P" on every emitted edge, since hpoa_row_is_positive_phenotype() already hard-filters to aspect == "P". Dropped.
• hpoa_qualifier — same pattern: always empty, since the same filter requires no qualifier to keep a row. Dropped.
• hpoa_database_id — verified against the full built graph that it's always identical to original_subject with zero exceptions. 100% redundant. Dropped.
• hpoa_disease_id (gene-phenotype edges) — assigned the exact same value as disease_context_qualifier in the same function; confirmed zero mismatches across all 311,724 gene-phenotype edges. Dropped.
• hpoa_onset/hpoa_frequency/hpoa_sex — biolink has purpose-built qualifiers for exactly this HPOA-shaped data (onset_qualifier, frequency_qualifier, sex_qualifier). Moved onto the standard qualifiers instead of custom properties.
• hpoa_reference — mostly duplicative of publications, but not entirely: of 258K disease-phenotype edges, 162,605 have a non-PMID reference value, and 99.5% of those are pure self-citations (the disease citing its own OMIM/ORPHA/DECIPHER id). Rewrote extraction (publications_from_reference) to split the ;-delimited field and route each part correctly:
  • PMID:... → publications as-is
  • ISBN(-10/-13)?:... → publications as isbn:... (biolink's book class is a publication subtype with registered isbn prefix)
  • bare http(s)://... (mostly NCBI Bookshelf/GeneReviews links) → publications as-is
  • self-citations (reference matches the edge's own disease id) → dropped, no information lost
  • everything else — 81 rows (0.03% of disease-phenotype edges), cite a different OMIM entry (e.g. OMIM:609757 citing OMIM:609637, OMIM:300623 citing OMIM:309550). Given the vanishingly low occurrence and no clean way to represent "this is really about a related MIM entry" as either a publication or an existing qualifier, recommend dropping these too rather than adding one-off modeling for 9 pairs.
• hpoa_gene_symbol — the majority of gene symbols in this column are duplicative with dubject/object 512 edges (0.16% ) diverge, but they are untrustworthy. 6 genes have no symbol in Babel at all (immunoglobulin/TCR gene segments, a fragile-site locus), 3 genes conflict with the current official symbol (e.g. hpoa_gene_symbol: "IFT54" vs. Babel's "TRAF3IP1" — IFT54 is an older alias), and the remaining 6 are HPO's own "-" missing-value placeholder. Dropped it.

Also reviewed and kept:

• hpoa_disease_name (diverges from the normalized node's own name on ~19% of edges, mostly where MONDO lumps several OMIM entries under one label — HPO's name is more specific)
• hpoa_hpo_name (diverges from normalized node's name on ~21% of edges)
• hpoa_evidence
• hpoa_modifier (HPO Clinical Modifier terms — severity/laterality/course/trigger — with no single existing biolink qualifier that fits the whole mix).

[eKathleenCarter]

• omim_type — always "phenotype" on every edge (7,508/7,508), since the loader already filters row["type"] != "phenotype" upstream. Dead-weight edge property and dropped.

• omim_mim_number — always exactly the numeric suffix of original_object (OMIM:100100 → "100100"), checked all edges, zero mismatches. 100% redundant, dropped.

• medgen_cui - 96.2% duplication of either the subject or object node (included in equivalent identifiers from Babel), 184 edges (2.4%) either missing or a CN-prefixed NCBI placeholder, and 111 edges (1.5%) Babel doesn't have the medgen_cui in equivalent identifiers. Suggest dropping it.

• omim_comment is NCBI's documented encoding of OMIM's qualifier symbols (?, {}, []) on the gene-phenotype relationship. Two of its values may warrant different treatment than the current uniform handling. https://ftp.ncbi.nlm.nih.gov/gene/DATA/README

   >  Comment:   
   >            optional value reporting the qualifiers OMIM provides when reporting
   >            a gene/phenotype relationship
   >            The values are based on the explanation of the
   >              symbols provided by OMIM: http://omim.org/help/faq
   >              nondisease: Brackets, "[ ]", indicate "nondiseases,"
   >                           mainly genetic variations that lead to 
   >                           apparently abnormal laboratory test values 
   >                           (e.g., dysalbuminemic euthyroidal hyperthyroxinemia).
   >              susceptibility: {} indicate mutations that contribute to 
   >                          susceptibility to multifactorial disorders
   >              QTL 1: {} and qtl
   >              QTL 2: [] and qtl
   >              somatic: somatic in the disease name
   >              question: A question mark, "?", before the disease name 
   >                          indicates an unconfirmed or possibly spurious mapping.
  

  1. nondisease (87 edges, 1.2%) OMIM's statement is that these entries "mainly [represent] genetic variations that lead to apparently abnormal laboratory test values" rather than an actual disease. biolink:gene_associated_with_condition probably is a bit too strong of a predicate for these edges.

  2. question (612 edges, 8.5%) OMIM prefixes the disease name with ? specifically to mean "unconfirmed or possibly spurious mapping". Right now we assert knowledge_level: knowledge_assertion identically for all edges, which overstates confidence. knowledge_level itself doesn't have a good enum value for "curator flagged as uncertain". Not sure how best to capture the uncertainty, but wanted to flag for follow-up review.

[eKathleenCarter]

Dropping properties:

• orphanet_association_status — always "Assessed" on every edge (7,659/7,659), since the loader already filters association_status != ASSESSED_STATUS upstream. Dead-weight. Dropped.

• orphanet_gene_symbol — only 4 of 7,659 edges (0.05%) diverge from the subject node's own normalized name. All four are stale/alias symbols (e.g. orphanet_gene_symbol: "IFT54" vs. Babel's "TRAF3IP1"). Dropped.

• orphanet_source_of_validation — duplicates of publications. Dropped.

Bug in pmids_from_validation: the regex required exact-case "PMID" , so it silently dropped real PMIDs from malformed source strings actually present in the live feed — a typo'd "...[PMId]" and a truncated "...[PMID" with no closing bracket. Fixed with case-insensitive matching and an optional closing bracket.

Also reviewed and kept:

• orphanet_association_type (8 distinct values with real biological meaning — germline/somatic, gain/loss-of-function, susceptibility, modifier, fusion-gene)

• orphanet_disorder_name/orphanet_disorder_type/orphanet_disorder_group form a 3-level taxonomy from Orphanet. Each disorder is tied to one of 9 disorder types, each of which is tied to one of 3 disorder groups.

[eKathleenCarter]

Reviewed the PR

Summary:

Confirmed: All the filtering/predicate logic described for HPOA/OMIM/Orphanet, Baseline-only registration verified. The MonarchKG filtering behavior and the structural reasoning in "why the direct ingest adds edges" also verified.

Two things of note:

1. Orphanet's predicate description says it emits ORPHA:<OrphaCode>. Fixed it to Orphanet:<OrphaCode>.
2. The "HPOA preserves raw fields including... reference... aspect..." sentence is stale. I dropped several fields that were redundant or always-constant and moved others onto biolink's own qualifiers instead of raw properties.

Exact edge/overlap counts in the PR won't reproduce against a fresh pull; new versions were introduced during review, but the structural claims hold.

Current edge counts from a fresh build against live data just now:

Source	Predicate	Edge family	Edges
HPOA	has_phenotype	disease→phenotype (OMIM)	143,558
HPOA	has_phenotype	disease→phenotype (Orphanet)	114,470
HPOA	has_phenotype	gene→phenotype (disease-conditioned)	311,724
HPOA	—	total	569,752
OMIM	gene_associated_with_condition	gene→condition	7,508
Orphanet	gene_associated_with_condition	gene→condition	7,659

Note on DECIPHER edges: DECIPHER-attributed HPOA rows aren't in this table. That prefix isn't registered in Babel/node-normalization at all, so those edges are dropped during normalization.

[eKathleenCarter]

scripts/compare_hpoa_rationalization.py — this looks like a one-off analysis tool you used while building this PR (it's also the only file in scripts/, so this PR introduces that directory). It's not referenced by any tests, docs, or CI. Was this meant to stick around, or is it fine to drop before merge? If it's useful to keep as a diagnostic for future rationalization work, might be worth a one-line docstring/README note explaining when to reach for it — otherwise happy to see it go.