Conversation
When BLAST tasks disagree for one contig, select rows with the highest bitscore and then the lowest e-value before checking taxonomic consistency. Continue to fail on equally strong conflicting assignments instead of selecting one by input order.
Assign DEACON_UNION_INDEXES and ANNOTATE_LEAST_COMMON_ANCESTORS to the existing medium process profile. These operations can exceed the low profile's memory allocation on larger indexes and BLAST result sets, so request the resource class that matches their observed requirements.
Configure Clumpify to perform exact sequence deduplication instead of treating dedupe=2 as a disabled boolean. Disable optical-name matching, substitutions, N-containing matches, and optional pair reordering so duplicate decisions depend on sequence content while interleaved mates remain adjacent.
Force BBDuk to interpret known interleaved batches as paired input and remove the complete pair when either mate is rejected. Mark single-read batches explicitly as unpaired so SRA-style spot suffixes do not control pairing heuristics.
Stream known interleaved gzip input to Deacon 0.15 through its paired stdin interface so a contaminant match on either mate removes the complete pair. Retain Deacon native direct-file filtering for single-read batches and enable pipefail so decompression failures cannot be masked.
Read Deacon input and retained counts from the existing summary, fail on malformed counts, and stop zero-retention samples before BBTools, depletion, filtering, or assembly. Preserve their raw read counts and enrichment statistics, emit an explicit empty completion event, and include that event in the workflow sample count without importing v3.3 query-routing architecture.
Route SELECT_TOP_BLAST_HITS through the existing medium resource class instead of overriding memory with a process-name selector. Production retains its 16 GB request while the test profile can scale the miniature integration workload to runner-sized resources.
Treat a schema-valid header-only BLAST result with no coverage rows as a completed negative result. Write empty contig, taxon, BioBoxes, and QC artifacts while retaining failures for non-empty assignments that conflict or lack matching coverage.
Make NVD_REPO the single repository selector for the generated launcher and runtime pipeline root so the CLI, Pixi environment, and Nextflow sources cannot silently come from different checkouts. Retain NVD_PIPELINE_ROOT as a deprecated v3 compatibility alias, reject conflicting explicit roots, and migrate recognized setup-generated launchers without rewriting setup.conf or user-owned wrappers.
Collect SPAdes contigs immediately after assembly and before contig QC so downstream BLAST queries use stable NVD qseqid values while preserving original assembler contig IDs. This adds COLLECT_CONTIGS and bin/collect_contigs.py, which rewrite FASTA headers to stable nvdContig1 IDs with constrained key-value metadata and write a per-sample SQLite contig lookup keyed by qseqid. Final BLAST enrichment now joins the contig lookup to append evidence_class, producer, and original contig_id without adding duplicate length/hash columns or introducing a separate evidence_id column. Focused tests cover contig collection, final BLAST enrichment, and Nextflow wiring. The slow e2e test still requires explicit planning before it is run.
Assign collected SPAdes contigs to short_assembly_contig or long_assembly_contig using an internal 10 kb threshold. This commit should preserve stable qseqid values and original contig_id provenance while changing only the evidence_class metadata used for downstream QC and Big Table interpretation. The 10 kb boundary follows common viral metagenomics practice and MIUViG-style caution that shorter viral fragments are less reliably interpreted by automatic methods. It does not make long_assembly_contig synonymous with genome-like; genome-like promotion remains future QC-driven work.
Rename the contig-processing subworkflow from PREPROCESS_CONTIGS to PROCESS_CONTIGS now that the seam does more than preprocess already-collected records. The subworkflow now owns collection of assembler contigs under stable query IDs before masking, length filtering, and other contig QC. Move COLLECT_CONTIGS out of NVD_MAIN and into PROCESS_CONTIGS so the top-level workflow only passes mixed assembled contigs into a single contig-processing seam. NVD_MAIN now consumes PROCESS_CONTIGS.out.contigs for downstream viral filtering and PROCESS_CONTIGS.out.contig_lookups for final BLAST enrichment. Update the Nextflow integrity tripwire and the extract-human-virus-contigs comment to match the new subworkflow name and ownership boundary. Focused verification passed before describing this commit: ruff format --check, ruff check, collect_contigs tests, finalize_blast_results tests, Nextflow integrity tests, and just config-check.
Prepare the query-selection DAG without changing which sequences reach BLAST. The current EXTRACT_HUMAN_VIRUSES seam owns contig screening, read mapback, mapped-read counting, and BAM output; that name is too broad for the class-aware query model and future unmapped-read routing. This commit preserves current behavior while reshaping the workflow around direct responsibilities. Contig FASTA files should travel with their per-sample SQLite lookup in the same channel tuple so lookups cannot drift out of sync with the sequences they describe. The contig-screening and mapback/counting seams should make the existing DEACON_FILTER_CONTIGS, MAP_READS_TO_CONTIGS, and COUNT_MAPPED_READS responsibilities visible in the DAG. No merged-pair, single-read, or SELECT_BLAST_QUERIES behavior belongs in this commit. Those come after the current contig-only path has an honest set of seams.
Introduce an explicit BLAST-query selection seam while preserving the current contig-only behavior. The normal DAG should route screened contigs through SELECT_BLAST_QUERIES before MEGABLAST, rather than treating every screened FASTA record as implicitly admitted by whatever upstream process happened to emit it. This first slice is intentionally conservative. It only admits the existing assembly-contig classes and does not add overlap-merged pairs, single reads, new QC checks, or params.experimental gating. The selection step should validate screened FASTA records against the bundled SQLite lookup and pass the same lookup through unchanged. Do not mutate the input SQLite sidecar in this commit. Future admission or exclusion annotations should be represented by a new emitted sidecar file so Nextflow lineage and resume behavior remain explicit.
Add an experimental branch that extracts reads which fail to map back to screened contigs. This is the next small slice toward routing overlap-merged pairs and single reads into BLAST query selection, but this commit should not admit any new records to BLAST. The branch should preserve read provenance where the current mapback path has enough information, emit unmapped-read FASTQ plus a small count summary, and surface the count so we can estimate how much future BLAST input might grow before deciding on admission rules. Keep the new branch behind params.experimental if the wiring stays simple. The normal contig-only BLAST path and mapped-read counting should remain unchanged.
Introduce PREPARE_BLAST_QUERIES as the workflow-level home for contig screening, read mapback, experimental unmapped-read measurement, and class-aware BLAST query selection. NVD_MAIN now calls one query-preparation seam instead of directly owning DEACON_FILTER_CONTIGS, MAP_READS_TO_CONTIGS, or SELECT_BLAST_QUERIES wiring. Remove the one-process SCREEN_CONTIGS subworkflow. Screening remains implemented by DEACON_FILTER_CONTIGS, but it is now internal to PREPARE_BLAST_QUERIES, alongside MAP_READS_TO_CONTIGS and SELECT_BLAST_QUERIES. This keeps the top-level DAG focused on major pipeline stages and gives future overlap-merged-pair and single-read handling an obvious home before BLAST. Keep the direct MAP_READS_TO_CONTIGS subworkflow name while aliasing the underlying process internally to avoid a Nextflow name collision. The experimental unmapped-read branch remains inside this seam and continues to avoid BLAST admission.
Rename the mapback subworkflow to CONTIG_READ_MAPBACK so PREPARE_BLAST_QUERIES owns a noun-like mapback seam rather than importing a process-shaped workflow name. Split the minimap2 mapback process into MAP_SINGLE_READS and MAP_PAIRED_READS. The single-read path preserves the existing one-input mapping and positional dedup behavior for native single-read and long-read inputs. The paired-read path maps current interleaved paired reads with a SAM read group that records unmerged read provenance while keeping evidence_class=single_read for future query interpretation. Keep the existing experimental EXTRACT_UNMAPPED_READS measurement branch in place for now. The full paired-read merged/unmerged FIFO path is documented and proofed separately, but this commit only introduces the renamed seam and default-off paired-read mapback split.
Implement the next paired-read mapback slice: keep paired mapback inputs as one sample-level tuple with fixed read-file slots, translate those slots to evidence_class/read-group metadata only inside MAP_PAIRED_READS, and emit real class-split paired mapback-unmapped FASTQs and counts. This commit should avoid adding a stringly typed state machine to Nextflow tuples. It should not admit read-derived sequences to BLAST, create stable query IDs for reads, add an orphan evidence class, or write raw uncompressed FASTQ artifacts.
Compute the emitted-percentage denominator with math.fsum over the taxon-level CRUMBS values, then derive every percentage from that same total before joining the values back into the taxon table. This avoids order-sensitive floating-point aggregation while preserving the output schema. Strongest-row selection is inherited from the v3.2.1 patch stack; this commit now owns only percentage normalization.
Move optional paired-read merging from a mapback-only branch to the start of read preprocessing after Deacon virus enrichment. When merge_pairs is enabled, merged overlap-consensus reads and unmerged reads become the material that downstream read QC, host depletion, assembly, profiling, mapback, and later BLAST query selection operate on. This keeps read QC as the primary quality gate for all read-derived query material and avoids introducing a redundant QC regime only for mapback-unmapped reads. The change should keep pair merging optional and default off, preserve the existing non-merge path, and avoid adding stringly state-machine labels to Nextflow tuples.
Treat BBMerge's unmerged output as independent single-read queries after pair classification. Force singleton-mode read filtering so BBTools cannot reconstruct mate coupling from /1 and /2 identifiers. Retain each record that independently passes quality and length thresholds, even when its former mate is filtered out. Permit single-read batches to contain odd record counts and make no downstream pairing guarantee. Add regression coverage in which one mate passes QC and the other fails.
Rename the contig-oriented query lookup vocabulary to the generalized query-sequence model before introducing read-derived BLAST queries. This commit keeps BLAST admission behavior contig-only, but migrates the existing assembly-derived query path from the prototype contig-specific terms to the durable evidence vocabulary: the SQLite lookup table becomes query_sequences, source-local provenance is source_id rather than contig_id, and assembly query identifiers use the nvdContigQuery prefix. Keeping this migration separate from read-derived query admission makes the next feature commit additive rather than mixing schema correction with new BLAST behavior.
Add vsearch to the Pixi environment so the read-query normalization path can use a battle-tested exact dereplication tool instead of implementing high-volume per-read deduplication in Python. This commit only changes dependency metadata. The later read-query normalization work will verify and use vsearch to collapse exact duplicate unmapped read sequences within each evidence-class batch while preserving representative IDs and abundance counts.
CI has no LabKey credentials or secret store, so both LIMS end-to-end tests
(the mock and the real) are marked 'labkey' and run only via 'just e2e-labkey'.
The 'just e2e'/'e2e-test-ci' selection ('slow and network and not labkey') is
thus the SRA test only, as it was before the LIMS work; 'just e2e-labkey' uses
'... and labkey'. The self-contained mock-server unit tests still run in the
fast suite.
Construct the final LabKey log collection only when LabKey integration is enabled. Avoid eagerly creating the LabKey upload log directory for runs that cannot produce LabKey artifacts. Preserve the existing final-log collection and storage behavior for LabKey-enabled runs.
Generate deterministic per-sample plain-text coverage histograms from filtered contig-mapback BAMs using samtools' native Unicode block rendering. Publish the terminal reporting artifacts under target query preparation without coupling them to alignment or mapped-count publication. Exercise the Nextflow process with a synthetic indexed BAM and require stable contig identifiers, UTF-8 block output, summary statistics, and the expected positional coverage pattern. Raise the declared samtools floor to cover histogram rendering and the pipeline's existing markdup options.
Generate self-contained Alignoth HTML reports for each covered target-filtered contig using the corresponding filtered mapback BAM. Group the high-cardinality outputs by sample under target query preparation while keeping the reporting branch terminal and best-effort. Add Alignoth to the locked cross-platform environment, exercise actual multi-contig HTML generation with mapped and unmapped synthetic references, and verify the release container exposes the renderer.
Add a curated TSV lookup asset under assets/ that maps ePathogen-style source labels onto human risk group classifications using exact-safe lookup names plus NCBI taxid remaps where available. The table was built in stages. First, a deterministic pass generated candidate lookup labels from the source names and alternate names, checked them against names.dmp and the NCBI Virus/WVDB sourmash lineage universe, and surfaced conflicts, misses, and grouped-name rows. From there, the long tail was curated manually by interpreting source intent rather than relying on raw token matching alone. That manual pass did three main kinds of work: - promote exact-safe aliases directly into the lookup corpus, - remap historical, common, disease-adjacent, and grouped-member labels onto the most plausible NCBI Virus taxids when the intended taxon concept was clear, - conservatively drop labels that were too broad, too disease-oriented, too stock-like, or otherwise unsafe for exact joins. Several grouped rows required intent recovery rather than one-to-one alias compilation. In particular, bundled Alphapapillomavirus labels were rewritten into explicit member-level papillomavirus rows before remapping, and influenza subtype rows were treated as subtype-level remaps where the subtype intent was clear. The remaining unresolved review residue was then conservatively excluded from the first finished asset. The final TSV contains only rows with non-empty human classification values and backfilled taxids for every retained row so it can support taxid joins against BLAST outputs as well as name joins against Sourmash/NCBI Virus results.
Resolution: Add a shared Polars annotator that maps canonical taxids to WHO risk groups. Resolve the nearest classified taxon along BLAST taxonomy and Sourmash BioBoxes paths so classifications flow from ancestors to descendants but never upward. Treat malformed or ambiguous data conservatively, and warn while selecting the highest RG1-RG4 classification for conflicts on one taxid. Integration: Annotate final BLAST results and experimental Sourmash summaries, publish the derived Sourmash report, and propagate who_risk_group into query and taxon Big Tables beside their taxid columns. Keep the existing LabKey schema isolated through an explicit projection. Verification: Update the mini viral taxonomy, Sourmash lineages, fixture generator, schema checks, and end-to-end expectations for modern Orf and monkeypox species taxids. Cover directional inheritance, overrides, malformed inputs, row preservation, exact risk groups, and output column placement.
Replace the LabKey-coupled plain-text notifier with a cacheable Block Kit completion lineage driven by explicit reporting and experimental completion channels. Summarize deduplicated final RG3/RG4 taxonomic assignments with complete paginated evidence and accessible fallbacks, while preserving best-effort stateless delivery. Add final BLAST risk-group source provenance for comparison with rapid screening, make LabKey links optional and experiment-filtered, and wait for successful LabKey registration only when integration is enabled. Cover message rendering, pagination, URL construction, provenance, and real Nextflow process contracts.
Preserve explicit read ordinals retained in SRA and ENA FASTQ representative labels before reducing them to compact source IDs. Keep the duplicate-source guard so genuinely ambiguous representative identities still fail instead of entering query lookups. Correct the integration contract so only standard skip-assembly runs expect no final BLAST output. Experimental skip-assembly runs intentionally query read-derived sequences and must satisfy the existing final-result, taxonomy, risk-group, and task assertions.
Toolchain: Pin FastQC 0.12.1 and MultiQC 1.35 as exact direct Pixi dependencies so reporting behavior changes only through deliberate upgrades. Isolate the resulting cross-platform solver expansion in one reviewable commit. Validation: Use disposable biologically neutral probes to verify native FastQC discovery, MultiQC Custom Content tables, a 90-sample heatmap and two PCoA scatter shapes, and the Nextflow collection and path-staging assumptions needed by later commits. Keep these as empirical design evidence rather than maintaining tests of documented third-party behavior. Boundaries: Add no production workflow modules, report compiler code, publication changes, fixtures, or dormant runtime scaffolding. NVD-owned aliases, configuration, channels, failure semantics, and normalized outputs begin with the durable C2 reporting spine and receive tests there. Verification: Confirm the lock is current, run the 604-test fast suite, and pass formatting, lint, schema, test, and all profile-rendering checks through the existing `just check` gate. Independent C1 reviews are waived explicitly because this commit contains only direct dependency declarations and their generated lock.
Input units: Expand each gathered sample bundle into deterministic physical-FASTQ units with collision-safe aliases while preserving sample identity, read end, input ordinal, and only the lane or chunk metadata that is actually available. Run FastQC over those raw units without inventing an expected-task ledger. Reporting spine: Add the durable terminal reporting architecture: NVD-owned status and manifest contracts, a pure compiler for the minimal roster/source-identity view, stable MultiQC configuration, native FastQC discovery, a best-effort terminal process, and publication of `multiqc_report.html` with complete `multiqc_data`. Feed every artifact through explicit channels and never read published results. Failure semantics: Retry and then ignore ancillary FastQC/compiler/renderer failures at production runtime without gating BLAST or LabKey. Keep development assertions strict about NVD-generated aliases, staged inputs, normalized data, report outputs, absence-without-inference, and poison-file exclusion. Boundaries: Install the final extensible reporting spine without placeholders or dormant future channels. Defer enrichment, depletion, FASTX, assembly, classification, CRUMBS, taxon evidence, rapid evaluation, and sample similarity sections to later vertical commits.
Split FASTX length distributions into single-read, overlap-merged-pair, and filtered-contig sections so unlike scales no longer obscure one another. Keep length and quality distributions as linear line graphs, preserve all points after bounded rebinning, and retain distinct semantic series labels through MultiQC name cleaning. Publish the standalone HTML report at the NVD root while placing its supporting multiqc_data directory with experiment summaries. Update user documentation and make end-to-end section expectations follow the enabled merge and assembly capabilities.
Generate ancillary per-sample query-batch summaries in standard and experimental runs, then carry them through an explicit query-preparation report package into the typed MultiQC compiler. Keep standalone summary publication experimental-only while deriving user-facing availability from the report plan. Render a long-form Prepared BLAST Query Batches table with plain-language class and sequence-type labels, numeric counts, configured-disabled states, hidden platform and validation details, and actionable malformed-summary guidance.
Resolve query identifiers from the named qseqid column in headered annotated BLAST TSVs rather than assuming a positional field. Add focused producer tests for empty, header-only, reordered, retained, removed, and malformed schemas without changing filtering or Nextflow behavior.
Package each observed MEGABLAST query partition by sample and query class, parse it through a dedicated typed BLAST report boundary, and render a compact MultiQC table of searched, MEGABLAST-matched, and BLASTN-forwarded query counts. Preserve configured BLAST disablement as a run-plan notice while leaving missing enabled outputs unexplained.
Add an experimental, bounded RG2–RG4 Taxon Big Table projection near the top of the MultiQC report. Preserve the producer's existing columns and salience semantics, prioritize higher WHO risk groups, and retain links to complete result tables rather than turning the static report into an exhaustive explorer. Wire the existing experimental Taxon Big Table into the typed report-package/compiler path, cover parsing and rendering at public boundaries, and keep standard-mode scientific outputs unchanged.
Add strongest- and faintest-signal Taxon Big Table projections alongside the higher-risk view, preserve the producer's established column order, and use readable finding labels within MultiQC's unique-row constraint. Accept blank optional risk annotations without invalidating a sample's table, derive the roster's displayed read structure from existing FastQC receipts, and place the findings views first while suppressing MultiQC's fixed General Statistics block.
Prepend the Pixi CLI during image construction and the locked default environment at runtime so env-based launchers resolve their matching interpreters. This fixes FastQC selecting Ubuntu Perl without mixing the system and Pixi Perl installations. Strengthen container CI around the runtime contract: assert Pixi-first PATH and Perl resolution, load FindBin, run Nextflow and FastQC, process a minimal FASTQ into validated ZIP and HTML outputs, and download an NVD checksum manifest over HTTPS with Pixi curl. Exercise every retained direct bioinformatics executable and the shadowed Pixi system tools while preserving the existing Python, NVD CLI, and environment smoke tests.
Add an explicit pipeline option for highly sharded runs where per-file raw FastQC would create excessive tasks and unusable reports. Gate only the ancillary FastQC branch; the remaining preprocessing and MultiQC report continue with FastQC outputs absent.
Extend MEGABLAST and BLASTN output with the subject accession, query and reference endpoints, reference length, and strand. Preserve those fields through reference retention, taxonomy annotation, LCA assignment, risk annotation, and final BLAST enrichment, including no-hit headers. Expose the selected best hit in the Query Big Table with its reference accession and title, alignment length, normalized 1-based inclusive query and reference intervals, reference length, and strand. Resolve fully tied HSPs and references deterministically so placement fields do not depend on input order. Keep the locked LabKey projection unchanged: the expanded final BLAST artifact carries the new fields, while LabKey preparation explicitly omits them. Cover the raw BLAST, annotation, LCA, finalization, Query Big Table, downstream taxon-table fixture, LabKey boundary, and experimental end-to-end seams.
Make query-level support notes identify the selected best-hit accession while leaving placement coordinates in structured columns. Summarize how contributing queries are positioned within their best-hit references in taxon-level support notes. Merge overlapping or exactly adjacent intervals independently per reference; report combined representative-interval coverage when all queries share one reference, and bounded region/reference counts when they do not. Avoid outer-span percentages that could make distant fragments resemble whole-genome coverage. Keep geometry independent of dominant/LCA assignment, support-tier rules, query classes, and taxonomic rank. Do not add Taxon Big Table columns or let placement geometry change support tiers.
Publish the query sequences represented in the Query Big Table, their distinct selected reference sequences, and BED6 best-hit placement intervals as a coherent evidence bundle. Keep FASTA sequences unwrapped and identifiers consistent across FASTA and BED artifacts. Publish the bundle under the dedicated 11_best_hit_sequences results directory without changing existing result paths.
Merged
nrminor
marked this pull request as ready for review
August 5, 2026 15:07
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
We gotta stop making such big releases!
v3.4.0 is another big release. It introduces a significant amount of new machinery to generate a fully custom MultiQC report and final "big table" artifacts that are the intended review entrypoint for
nvdresults.MultiQC
After introducing query classes,
nvdhad significantly increased logic around which query sequences may end up moving forward into which steps. Users will reasonably ask questions like:Etc. The multiQC report is supposed to centralize answers to these questions so users don't need to dig down into our deeply nested results hierarchy to answer them. It's also a standalone artifact that can be sent between collaborators, whereas the results directory will be too unwieldly to be conveniently shared.
The "Big Tables"
All pipelines produce big tables at some point.
nvdis no exception. Building off our work in #32 introducing a de-biased abundance metric and in #31 introducing query classes, v3.3 introduces two new reporting tables, one focused on individual queries (the "query big table") and one aggregated table focused on taxa (the "taxon big table").We put significant thought into these tables, in particular how the table guide's the user's attention toward things that might be interesting, and how the table qualifies each entry to answer questions that the user my have progressively. These qualifications include ranking each hit into strong/moderate/weak bins based on BLAST statistics, explanatory entries based on each LCA resolution, and much, much more. We expect to tweak this table as it gets more use and more review, but suffice it to say that this is a step-change from the previous regime where users were directly reviewing overwhelming volumes of BLAST results.
Our attempt at sorting pathogens by risk
Version 3.4.0 also introduces a new column that notes whether each virus is considered part of a WHO Risk Group, which is one way of enabling users to sort results so that particularly dangerous detections bubble up. It's also a building block for rapid notification systems where users could be notified only on high-risk detections.
Building toward even better detection reporting
Additionally, v3.4.0 now tracks where BLAST placed queries on their best reference placement. These positions are reported in BED format alongside a FASTA of the query sequences themselves as well as a FASTA of the selected reference sequences. Note that in some cases, the LCA-selected taxon for a given query sequence may not be the same as the taxon for a selected reference; future versions may explore some creative solutions to this mismatch.
Other goodies
This release also brings new slack reporting features, both for quick rapid screening notifications for >RG2 viruses, as well as for pipeline completion. We also now provide interactive Alignoth alignment pileup visualizations for contigs to see how read mapback against assembled contigs went. We also fixed some issues in our container build, continued cleaning up our results directory structure, and more!
This is a foundational release the opens up many new, exciting improvements we can now make in the future.