v3.3.0 release candidate

[nrminor]

This is a big one. TL;DR: v3.3.0 refactors the entire pipeline around a paradigm henceforth referred to as "query classes," where each query class can be come its own parallelized shard within the nvd workflows.

The idea behind query classes

Before v3.3.0, nvd collapsed all reads into whatever contigs SPAdes gave us. These contigs were then QC'd and then sent into big BLAST searches against Core NT.

This system works well in that the relatively small number of contigs that fall out of each sample's reads put a low bound on the query search space that BLAST has to contend with. Its downside is that it introduces yet another significant risk of sensitivity loss/false negative introduction. If the assembler can't resolve contigs for a given read, it will simply be lost as a potential fragment of interest and will never be searched against Core NT.

If you're working with a clinical sample where most taxa are expected to have good whole-genome representation, this is unlikely to pose a problem. Most reads will come from those well-represented genomes, giving the assembler plenty of grist to collapse down into representative contigs.

But if your library is highly fragmented, many reads will come from sheared genomes that simply don't have the read representation they'd need to resolve into contigs. These genomes, and the taxa they stem from, will thus be lost in the old nvd paradigm. Despite its claims to be tuned for use on sheared environmental samples, this was a major weakness for nvd, and in many cases it led to visible under-sensitivity compared to other pipelines, e.g., nao-mgs.

Version 3.3.0 introduces an elegant solution to this problem and refactors the entirety of the pipeline around that solution. Rather than all queries being contigs, nvd now tracks whether a read resolved into a contig or not, and also whether a read pair was merged into a consensus read. Each of these kinds of queries (hence, "query classes") then becomes its own BLAST input. For a given sample, contigs will be searched in their own BLAST process. But merged reads that didn't resolve into a contig will also get searched in their own BLAST instance, as will single reads. Retaining reads that were not captured in any of the contigs almost entirely restores sensitivity that was lost in the previous nvd paradigm.

Query classes are still experimental and have a downside

This feature remains experimental (gated behind params.experimental == true); running in the old paradigm still works and remains exercised in our end-to-end testing. That said, this new paradigm is transformative enough that we expect to make it the only run mode supported by nvd.

The major downside to searching many more query sequences in BLAST is that you're now searching many more query sequences in BLAST, and thus nvd as a whole has potential to take 1.5 to 2 times longer. To maintain the fastest possible reporting turnaround, we've thus adjusted our LabKey LIMS integration to upload BLAST results as they become available rather than waiting until all query class searches have finished. With this adjustment, contig-based results still upload more or less as quickly as they did pre-v3.3.0. High-sensitivity read results will become available later as their respective searches complete.

Enabling future feature releases

Query classes also come with a variety of sidecar artifacts that give observability into how query sequences are routed into each of the query classes, with summaries provided at various stages of the pipeline where query classes for each sample are handled in parallel. This helps build toward upcoming changes in v3.4 that enhance the information nvd reports to users.

In addition, v3.3.0 brings a significant semantic reorganization of the results directory in hopes of empowering more intuitive exploration of results by users.

Overall, if cv3.2.0 was the "CRUMBS release", v3.3.0 is the "query classes" release. This all builds toward producing a reimagined results reporting system in #46, which will be up next!

[wkgardner]

More LIMS side refactor commits coming tomorrow. I am having to reorder some of the steps towards the end of the BLAST step like Stacking blast results after ADD_READ_COUNTS_TO_BLAST which is necessary row level info for blast uploads.

On the LIMS side it looks like we only need to update the schema in one place:
Hits list (labkey_blast_meta_hits_list): add a query_class text column.

The fasta reports for each query on the LIMS side will be limited to only contigs (both short and long) to prevent blowing up prod lists.

I am also attempting to build a just e2e-labkey test but if LK permissions are too much of a pain I may have to ditch this idea.