Implement OTF (On-The-Fly) expression call propagation and remove Affymetrix/EST datatypes#252
Implement OTF (On-The-Fly) expression call propagation and remove Affymetrix/EST datatypes#252jwollbrett wants to merge 74 commits into
Conversation
…he hiddenSpecies attribute
…tNodeId and added seqRegionName
…Id to throw an error on empty DAOConditionFilter2
…d terms or excluded terms after OTF propagation
…ditionIds are retrieved before OTF propagation
…ed terms or child terms not requested. Also implement 1st version of post propagation sorting
…present in raw to global condition
| // return null; | ||
| // } | ||
|
|
||
| private OTFExpressionCall generateOTFExpressionCall(Gene gene, Condition2 cond, |
There was a problem hiding this comment.
Please review this method carefully. This is the core aggregation: it combines self-observed p-values (repeated nObs times) and scores with the FDR-corrected p-value from descendant calls. Two behaviours are not yet implemented and must be discussed: (1) a parent with no self-observation should inherit the best (lowest) descendant p-value rather than the mean; (2) when nObs > 1 for a single data type, the p-value should be multiplied by 2 before being added to the pool (conservative correction for multiple observations).
| return log.traceExit(match); | ||
| } | ||
|
|
||
| private Map<Gene, Set<OTFExpressionCall>> propagateCalls( |
There was a problem hiding this comment.
Entry point for OTF propagation — equivalent of the old loadData(). Walks the condition graph in topological order (child → parent), accumulating calls. Redundant ancestor conditions (same score as their best descendant) are pruned at the end. Please review the traversal logic and the redundancy-detection condition carefully.
| } | ||
|
|
||
| //right now | ||
| public Map<Gene, List<OTFExpressionCall>> loadDataOnTheFly() { |
There was a problem hiding this comment.
Top-level OTF entry point. Note the single-species assumption (FIXME). Also note that post-propagation filtering (OTFExpressionCallFilterEngine) and summary-call-type filtering (matchesRequestedSummaryCallType) happen after propagation
| dataTypes.contains(DataType.RNA_SEQ) || | ||
| dataTypes.contains(DataType.SC_RNA_SEQ) || | ||
| call.getMeanRank().compareTo(BigDecimal.valueOf(20000)) < 0)) { | ||
| //FXIME: Need to consider the SummaryQuality once it is implemented. TO be done before Bgee 16.0 release |
There was a problem hiding this comment.
FIXME: temporary hardcoding — must use proper SummaryCallType/SummaryQuality before Bgee 16.0 release
There was a problem hiding this comment.
New pipeline step that pre-inserts propagated condition mappings into the DB, enabling OTF lookups without recomputing the full graph at query time.
| CommandData commandData = this.getPartialCommandData(); | ||
| commandData.initializeCaches(sleepBetweenComputeMs); | ||
|
|
||
| // --- Add condition graph cache initialization here --- |
There was a problem hiding this comment.
Cache loading condition graphs at server startup
1 participant
Summary
This branch implements a new On-The-Fly (OTF) propagation approach for expression calls,
replacing the previous pre-computed pipeline-based propagation for the gene page and data
expression API. It also completes the removal of Affymetrix and EST legacy datatypes
from all layers of the codebase.
147 files changed — 10 541 insertions / 12 648 deletions.
What has been done
1. OTF propagation — new core logic (bgee-core)
OTFExpressionCall: model class for a call after OTF propagation (aggregates rawobserved calls across the condition graph). Carries the aggregated p-value, score, supporting
data types, propagation state, and best-direct-descendant values for redundancy pruning.
OTFExpressionCallFilterEngine: post-propagation filtering that removes conditionswith excluded terms or non-requested child terms.
ConditionGraphCacheService: on-demand caching of the condition graph (topologicalorder, direct ancestor/descendant maps). Avoids reloading and recomputing the ontology
structure on every request; the cache is keyed by species and condition parameters so it
can be safely shared across calls within the same request.
ExpressionCallLoader: newloadDataOnTheFly()method replacingloadData()for thegene page and expression data API. Core sub-methods:
propagateCalls()— topological child→parent sweep over the condition graph.generateOTFExpressionCall()— aggregates self observations and descendant calls into asingle
OTFExpressionCallfor a given gene × condition; combines p-values and scores(see "reviewer focus" below).
ConditionGraph: propagation now returns aSet; removed obsoletemaxRanksPerSpecieslookups (scores are now stored in the DB by the pipeline).
2. New DAOs and pipeline step
ObservedExpressionDAO/MySQLObservedExpressionDAO: retrieves expression rowswith actual observations for selected data types.
RawExpressionCallDAO/MySQLRawExpressionCallDAO+DAORawCallFilter/DAORawCallValues.InsertPropagatedConditions: pre-computes and inserts the propagatedcondition mapping into the DB.
DAOConditionFilter2: updated to support species-only filtering for OTF queries.3. Webapp controllers (bgee-webapp)
CommandDataandCommandGenerefactored to use the new OTF path.CommandExpressionSupport: shared expression-call logic extracted from the twocontrollers.
4. Removal of Affymetrix and EST datatypes
Complete removal across all layers (bgee-core, bgee-dao-api, bgee-dao-sql, bgee-webapp):
download-file generators.
5. Schema / model updates for Bgee 16
hiddenSpeciesfrom theSpeciesmodel.OMAParentNodeId; addedseqRegionNameto gene objects.What is missing / known limitations compared to the old
loadData()approachexpressionStateandexpressionQualityhardcoded (p-value ≤ 0.05 → "expressed", quality always "gold"). ProperSummaryCallType/SummaryQualitypropagation not yet implemented.ExpressionCallResponseTypeAdapter.javaExpressionCallLoader.javaOrderingAttributenot yet implemented; only score-based descending sort is available.OTFExpressionCall.javaincludeChildTermsfilterDAOConditionFiltersent for OTF — should be removed (child term expansion is handled post-propagation).CommandExpressionSupport.javaDAOConditionFilter2ontology dependencyDAOConditionFilter2for OTF still unnecessarily uses ontologies. It is not mandatory to solve that point for Bgee 16 to be released but a new approach not using the ontologies could drasticaly fasten the creation of the condition part of the ExpressionCallLoaderExpressionCallLoader.javamaxRanksPerSpeciesExpressionCallProcessedFilter; no longer needed since scores are pre-computed.ExpressionCallLoader.javakeepOnlyParentsMoreExpressedSummaryQuality.SILVERfiltering.ExpressionCallLoader.javaExpressionCallLoader.generateOTFExpressionCall()ExpressionCallLoader.generateOTFExpressionCall()Notes for the reviewer
loadData()code path is not yet removed — it is retained for non-OTF use cases(download files, TopAnat). The OTF path is only activated for the gene page and data
expression API calls.