Conversation
Generate deterministic per-sample plain-text coverage histograms from filtered contig-mapback BAMs using samtools' native Unicode block rendering. Publish the terminal reporting artifacts under target query preparation without coupling them to alignment or mapped-count publication. Exercise the Nextflow process with a synthetic indexed BAM and require stable contig identifiers, UTF-8 block output, summary statistics, and the expected positional coverage pattern. Raise the declared samtools floor to cover histogram rendering and the pipeline's existing markdup options.
Generate self-contained Alignoth HTML reports for each covered target-filtered contig using the corresponding filtered mapback BAM. Group the high-cardinality outputs by sample under target query preparation while keeping the reporting branch terminal and best-effort. Add Alignoth to the locked cross-platform environment, exercise actual multi-contig HTML generation with mapped and unmapped synthetic references, and verify the release container exposes the renderer.
Add a curated TSV lookup asset under assets/ that maps ePathogen-style source labels onto human risk group classifications using exact-safe lookup names plus NCBI taxid remaps where available. The table was built in stages. First, a deterministic pass generated candidate lookup labels from the source names and alternate names, checked them against names.dmp and the NCBI Virus/WVDB sourmash lineage universe, and surfaced conflicts, misses, and grouped-name rows. From there, the long tail was curated manually by interpreting source intent rather than relying on raw token matching alone. That manual pass did three main kinds of work: - promote exact-safe aliases directly into the lookup corpus, - remap historical, common, disease-adjacent, and grouped-member labels onto the most plausible NCBI Virus taxids when the intended taxon concept was clear, - conservatively drop labels that were too broad, too disease-oriented, too stock-like, or otherwise unsafe for exact joins. Several grouped rows required intent recovery rather than one-to-one alias compilation. In particular, bundled Alphapapillomavirus labels were rewritten into explicit member-level papillomavirus rows before remapping, and influenza subtype rows were treated as subtype-level remaps where the subtype intent was clear. The remaining unresolved review residue was then conservatively excluded from the first finished asset. The final TSV contains only rows with non-empty human classification values and backfilled taxids for every retained row so it can support taxid joins against BLAST outputs as well as name joins against Sourmash/NCBI Virus results.
Resolution: Add a shared Polars annotator that maps canonical taxids to WHO risk groups. Resolve the nearest classified taxon along BLAST taxonomy and Sourmash BioBoxes paths so classifications flow from ancestors to descendants but never upward. Treat malformed or ambiguous data conservatively, and warn while selecting the highest RG1-RG4 classification for conflicts on one taxid. Integration: Annotate final BLAST results and experimental Sourmash summaries, publish the derived Sourmash report, and propagate who_risk_group into query and taxon Big Tables beside their taxid columns. Keep the existing LabKey schema isolated through an explicit projection. Verification: Update the mini viral taxonomy, Sourmash lineages, fixture generator, schema checks, and end-to-end expectations for modern Orf and monkeypox species taxids. Cover directional inheritance, overrides, malformed inputs, row preservation, exact risk groups, and output column placement.
Replace the LabKey-coupled plain-text notifier with a cacheable Block Kit completion lineage driven by explicit reporting and experimental completion channels. Summarize deduplicated final RG3/RG4 taxonomic assignments with complete paginated evidence and accessible fallbacks, while preserving best-effort stateless delivery. Add final BLAST risk-group source provenance for comparison with rapid screening, make LabKey links optional and experiment-filtered, and wait for successful LabKey registration only when integration is enabled. Cover message rendering, pagination, URL construction, provenance, and real Nextflow process contracts.
Preserve explicit read ordinals retained in SRA and ENA FASTQ representative labels before reducing them to compact source IDs. Keep the duplicate-source guard so genuinely ambiguous representative identities still fail instead of entering query lookups. Correct the integration contract so only standard skip-assembly runs expect no final BLAST output. Experimental skip-assembly runs intentionally query read-derived sequences and must satisfy the existing final-result, taxonomy, risk-group, and task assertions.
Toolchain: Pin FastQC 0.12.1 and MultiQC 1.35 as exact direct Pixi dependencies so reporting behavior changes only through deliberate upgrades. Isolate the resulting cross-platform solver expansion in one reviewable commit. Validation: Use disposable biologically neutral probes to verify native FastQC discovery, MultiQC Custom Content tables, a 90-sample heatmap and two PCoA scatter shapes, and the Nextflow collection and path-staging assumptions needed by later commits. Keep these as empirical design evidence rather than maintaining tests of documented third-party behavior. Boundaries: Add no production workflow modules, report compiler code, publication changes, fixtures, or dormant runtime scaffolding. NVD-owned aliases, configuration, channels, failure semantics, and normalized outputs begin with the durable C2 reporting spine and receive tests there. Verification: Confirm the lock is current, run the 604-test fast suite, and pass formatting, lint, schema, test, and all profile-rendering checks through the existing `just check` gate. Independent C1 reviews are waived explicitly because this commit contains only direct dependency declarations and their generated lock.
Input units: Expand each gathered sample bundle into deterministic physical-FASTQ units with collision-safe aliases while preserving sample identity, read end, input ordinal, and only the lane or chunk metadata that is actually available. Run FastQC over those raw units without inventing an expected-task ledger. Reporting spine: Add the durable terminal reporting architecture: NVD-owned status and manifest contracts, a pure compiler for the minimal roster/source-identity view, stable MultiQC configuration, native FastQC discovery, a best-effort terminal process, and publication of `multiqc_report.html` with complete `multiqc_data`. Feed every artifact through explicit channels and never read published results. Failure semantics: Retry and then ignore ancillary FastQC/compiler/renderer failures at production runtime without gating BLAST or LabKey. Keep development assertions strict about NVD-generated aliases, staged inputs, normalized data, report outputs, absence-without-inference, and poison-file exclusion. Boundaries: Install the final extensible reporting spine without placeholders or dormant future channels. Defer enrichment, depletion, FASTX, assembly, classification, CRUMBS, taxon evidence, rapid evaluation, and sample similarity sections to later vertical commits.
Split FASTX length distributions into single-read, overlap-merged-pair, and filtered-contig sections so unlike scales no longer obscure one another. Keep length and quality distributions as linear line graphs, preserve all points after bounded rebinning, and retain distinct semantic series labels through MultiQC name cleaning. Publish the standalone HTML report at the NVD root while placing its supporting multiqc_data directory with experiment summaries. Update user documentation and make end-to-end section expectations follow the enabled merge and assembly capabilities.
Generate ancillary per-sample query-batch summaries in standard and experimental runs, then carry them through an explicit query-preparation report package into the typed MultiQC compiler. Keep standalone summary publication experimental-only while deriving user-facing availability from the report plan. Render a long-form Prepared BLAST Query Batches table with plain-language class and sequence-type labels, numeric counts, configured-disabled states, hidden platform and validation details, and actionable malformed-summary guidance.
Resolve query identifiers from the named qseqid column in headered annotated BLAST TSVs rather than assuming a positional field. Add focused producer tests for empty, header-only, reordered, retained, removed, and malformed schemas without changing filtering or Nextflow behavior.
Package each observed MEGABLAST query partition by sample and query class, parse it through a dedicated typed BLAST report boundary, and render a compact MultiQC table of searched, MEGABLAST-matched, and BLASTN-forwarded query counts. Preserve configured BLAST disablement as a run-plan notice while leaving missing enabled outputs unexplained.
Add an experimental, bounded RG2–RG4 Taxon Big Table projection near the top of the MultiQC report. Preserve the producer's existing columns and salience semantics, prioritize higher WHO risk groups, and retain links to complete result tables rather than turning the static report into an exhaustive explorer. Wire the existing experimental Taxon Big Table into the typed report-package/compiler path, cover parsing and rendering at public boundaries, and keep standard-mode scientific outputs unchanged.
Add strongest- and faintest-signal Taxon Big Table projections alongside the higher-risk view, preserve the producer's established column order, and use readable finding labels within MultiQC's unique-row constraint. Accept blank optional risk annotations without invalidating a sample's table, derive the roster's displayed read structure from existing FastQC receipts, and place the findings views first while suppressing MultiQC's fixed General Statistics block.
Prepend the Pixi CLI during image construction and the locked default environment at runtime so env-based launchers resolve their matching interpreters. This fixes FastQC selecting Ubuntu Perl without mixing the system and Pixi Perl installations. Strengthen container CI around the runtime contract: assert Pixi-first PATH and Perl resolution, load FindBin, run Nextflow and FastQC, process a minimal FASTQ into validated ZIP and HTML outputs, and download an NVD checksum manifest over HTTPS with Pixi curl. Exercise every retained direct bioinformatics executable and the shadowed Pixi system tools while preserving the existing Python, NVD CLI, and environment smoke tests.
Add an explicit pipeline option for highly sharded runs where per-file raw FastQC would create excessive tasks and unusable reports. Gate only the ancillary FastQC branch; the remaining preprocessing and MultiQC report continue with FastQC outputs absent.
Extend MEGABLAST and BLASTN output with the subject accession, query and reference endpoints, reference length, and strand. Preserve those fields through reference retention, taxonomy annotation, LCA assignment, risk annotation, and final BLAST enrichment, including no-hit headers. Expose the selected best hit in the Query Big Table with its reference accession and title, alignment length, normalized 1-based inclusive query and reference intervals, reference length, and strand. Resolve fully tied HSPs and references deterministically so placement fields do not depend on input order. Keep the locked LabKey projection unchanged: the expanded final BLAST artifact carries the new fields, while LabKey preparation explicitly omits them. Cover the raw BLAST, annotation, LCA, finalization, Query Big Table, downstream taxon-table fixture, LabKey boundary, and experimental end-to-end seams.
Make query-level support notes identify the selected best-hit accession while leaving placement coordinates in structured columns. Summarize how contributing queries are positioned within their best-hit references in taxon-level support notes. Merge overlapping or exactly adjacent intervals independently per reference; report combined representative-interval coverage when all queries share one reference, and bounded region/reference counts when they do not. Avoid outer-span percentages that could make distant fragments resemble whole-genome coverage. Keep geometry independent of dominant/LCA assignment, support-tier rules, query classes, and taxonomic rank. Do not add Taxon Big Table columns or let placement geometry change support tiers.
Publish the query sequences represented in the Query Big Table, their distinct selected reference sequences, and BED6 best-hit placement intervals as a coherent evidence bundle. Keep FASTA sequences unwrapped and identifiers consistent across FASTA and BED artifacts. Publish the bundle under the dedicated 11_best_hit_sequences results directory without changing existing result paths.
Remove the post-BLAST query-group gates that required a non-phage viral hit after MEGABLAST and BLASTN. Target enrichment and depletion continue to control which sequences reach BLAST, while every annotated top-hit group now proceeds directly to merging, LCA annotation, and reporting. MEGABLAST partitioning and existing hit-retention limits remain unchanged. Delete the obsolete Nextflow processes, Python filter and summary scripts, unit tests, filter decision artifacts, and sequence-flow rows. Rename affected channels around annotated-hit semantics, publish annotated MEGABLAST and BLASTN TSVs at their result-family roots, and remove the virus-only result directories. Describe the existing enrichment parameters as target-enrichment controls in executable help without renaming the 3.x API. Normalize the sequence-flow process input to the codebase's scalar-or-list convention because removing the filter summaries allows valid runs with a single evidence file.
Use the v3.4.0 checkout in the NVD_REPO installation example after the v3.4 release line, while preserving the existing launcher and compatibility guidance.
Carry validated input and retained counts on Deacon outputs, route zero-retention samples directly to completion, and keep downstream reporting aware of samples that cannot produce contigs. Cover the empty route through workflow and MultiQC boundaries.
Apply the configured Deacon target index independently to both BBMerge output classes when target enrichment and pair merging are enabled. This removes nonmatching former mates retained by the pair-atomic initial pass while preserving genuinely interleaved, merge-disabled, and originally single-read paths. Emit one Deacon summary per post-merge query class and publish those summaries beneath the existing target-enrichment results at summaries/postmerge. Current MultiQC reporting remains unchanged while the experimental evidence is retained for later reporting decisions. Exercise the semantic premise directly against the locked Deacon CLI: a single FASTQ containing one matching and one nonmatching former mate retains only the match and reports two sequences in and one out. Channel routing remains intentionally reserved for end-to-end validation.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
After #31, this release is much smaller. Its many feature addition is direct streaming of reads from SRA into our deacon enrichment, with continued support for no-enrichment runs as well. This means entire bioprojects of accessions, for example, can be put through
nvdwithout persisting the entirety of each FASTQ onto disk first. Instead, reads are streamed directly into deacon filtering. Reads are thus only written to disk if they pass filtering with the provided index. This feature does have a downside: if you wish to download the entire FASTQs and then use nvd resumes to filter with different indexes,nvddoes not currently support that. Reads must be re-downloaded from SRA each time. This was a design choice that can be adjust in the future if it poses problems, though ultimately, we do not recommend using nvd resumes to reprocess with different parameters because each resumed run will overwrite previous runs' results!This release also introduces a
skip_rapid_screenparameter, which skips our currently sourmash-based rapid screening subworkflow.Both of these features were ultimately crucial in dholab's ability to scan our entire wastewater bioproject on SRA for Cyclospora reads during the summer 2026 outbreak.
Additional other improvements include zero-read samples more clearly being reported, development tooling moving to Mise instead of a Nix Flake+direnv, some updated pixi dependencies, and improved repository-override documentation.