Alexhillsley/bootstrap replace vectorized#17
Merged
Conversation
…ntal + add-on modules The argparse subpackage (`python -m ...combination.pca_optimization`) is now the canonical entry point, so the config/baseline.yml path is removed and the remaining modules are grouped by role. Remove deprecated config/baseline.yml path (no non-test/non-scratch importers): - cli.py, config_handler.py, file_validator.py - combiners.py (the duplicate PcaOptimizationCombiner + deprecated ComprehensiveCombiner) - classifier_combiner.py / classifier_aggregator.py (dormant; never wired into the CLI) Group downstream-only analysis tools into analysis/: - embedding_overlays, compare_map_scores, compare_modalities, pca_component_to_feature, marker_norm_sweep_runner Group optional flag-gated stages into pipeline_add_ons/: - op_signal, chromosome, guide_chrom_arm_correction Update all importers (pca_optimization __init__/handlers/phase2/embeddings and models/attention/embedding/regen_umap_html) to the new paths. pca_sweep_op_signal is still re-exported through pca_optimization's namespace; test_pca_optimization_refactor passes (41/41). Add README.md (how to run the subpackage) and SCRIPT_MAP.md (core vs supplemental inventory). cell_filters.py is retained pending a port-vs-drop decision (no subpackage equivalent yet). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…tion Two additive features for the pca_optimization combination pipeline, both reusing the existing argparse Namespace + Phase 1/2 with no parallel schema. 1. --config <yaml>: run the pipeline from a config file whose keys are the CLI argument names (snake_case dest names). Config values populate argparse defaults via set_defaults; any explicit CLI flag still overrides. Adds run_from_config() (programmatic entry) and _load_and_validate_config() (rejects unknown keys + the phase_only/no_phase conflict). main() is split into main() (parse + config merge) and run(args) (the unchanged dispatch). Example at pca_optimization/example_config.yml. 2. signal_paths (a config key): combine cell-level embedding h5ads that live OUTSIDE the standard experiment layout. Maps a signal-group name -> one h5ad path or a list of paths (pooled); each h5ad uses the same schema as the discovery features_processed_*.h5ad. phase1.pca_sweep_pooled_signal gains an optional cell_paths override (explicit path instead of find_cell_h5ad_path); new handlers._handle_external builds signal groups from the manifest and reuses the pooled worker + Phase 2. Experiment discovery is skipped; output lands under <output_dir>/external/. Verified: 41/41 structural tests pass; external ingest validated end-to-end (two synthetic h5ads pooled into one signal -> per_signal guide/gene outputs). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Move src/ops_model/eval into src/ops_model/deprecated/eval (following the existing deprecated/ convention) and delete tests/eval. Also drop the now-dead run_eval console-script entry point from pyproject.toml. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace the two near-duplicate pipelines (evaluate_cp.process and evaluate_embeddings.process_embedding_csv) with a single processing_common.process_features_csv that branches on feature type: CellProfiler builds the cell AnnData and splits by reporter; embedding models (dinov3/cell_dino/subcell) build one per-channel AnnData. Shared embedding-config parsing, guide/gene aggregation, and validate_and_save now live in processing_common; cp_features and batch_process_embeddings call the single entry point and the old ones are removed. Also folds in the features/ cleanup: dead functions + unused imports removed, the good_rows_mask NaN-row bug fixed, and the broken test_evaluate_dinov3.py deleted (its module was generalized into evaluate_embeddings). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
tests/e2e_tests/ — self-contained scripts (run directly with uv run python, not pytest) covering each core ops_model feature: the cell_dino, dinov3, subcell and cell_profiler extractors, and the pca_optimization combination pipeline. Each subsets real inputs to a minimal example in a tmp dir, points an inline config at it, runs the feature normally, and verifies the outputs at each step. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two hardcoded paths that silently disabled features outside one environment: - annotated_gene_panel_July2025.csv: repoint from the now-missing /hpc/projects/intracellular_dashboard/ops/configs path to the present icd.fast.ops/configs location (data/embeddings/utils.py, funk_clusters.py, combination/analysis/embedding_overlays.py). The dead path was caught-and- skipped, silently dropping CORUM consistency scoring. - gene_supercategory_mapping.yaml: default to the in-repo copy (resolved from the repo root) instead of a personal home-dir path that was permission-denied for other users (combination/analysis/embedding_overlays.py, compare_modalities.py, models/attention/atlas/attention_atlas.py). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
_score_consistency previously ran CORUM, CHAD and EBI inside one shared try/except, so a failure in one metric (e.g. CHAD failing to parse its annotation) silently suppressed the others and dropped EBI entirely. Each metric now runs in its own try/except and returns (None, 0.0) on failure; the panel/volcano plots are best-effort. One metric failing no longer takes the rest down. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
/hpc/projects/icd.ops/configs/gene_clusters/ no longer exists; the CHAD cluster YAMLs live under icd.fast.ops. Repoint the dead references (CHAD overlay hierarchy/cluster-map in pca_optimization phase2/handlers, deprecated gene/guide eval, titration decay tools, compare_map_scores) so they load instead of being skipped with warnings. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Delete the old base_dataset.py, the data/embeddings/* helpers (cosine_similarity, embedding_metrics, funk_clusters, pca, umap_plots, utils), and move_links.py, along with their now-obsolete tests (test_basedataset.py, test_feature_metrics.py). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…and should not be aprt of the public repo
post_process/map/ was only a backward-compat shim re-exporting the mAP functions from ops_utils.analysis (map_scores, map_umap). Nothing in ops_model imports it anymore, so move it to deprecated/ (kept locally only) and drop it from the tree. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Per-guide titration subsampling now supports a true nonparametric bootstrap (draw min(count, budget) cells per guide WITH replacement) behind a new --bootstrap-replace flag, threaded through _run_titration_points, titrate_single_target_for_reporter and titrate_single_reporter. The per-guide selection is also rewritten as a single vectorized helper (_bootstrap_indices_per_group) that replaces the O(n_guides * n_cells) np.where-per-guide loop. The legacy without-replacement path is left byte-for-byte unchanged for reproducibility; the fast path runs only when replace=True. Option 1 semantics: with replacement still keeps min(count, budget) cells per guide, so n_cells / x-axis bookkeeping is identical to the legacy subsample and results stay directly comparable. Timing (synthetic, 1,000,000 cells / 1,500 guides, budget 30 cells/guide): legacy np.where-per-guide loop : 19.01 s vectorized, without replacement : 0.28 s (~67x faster, identical counts) vectorized, with replacement : 0.11 s The legacy loop is O(n_guides * n_cells); on Phase (~60M cells) it was minutes-to-hours per draw, now seconds. End-to-end sanity (lysosome_LAMP1, 30 cells/guide, 3 draws): both modes run and emit mean/std/draws; n_cells (125,727) and n_guides (4,204) identical across modes; distinctiveness mAP 0.0129 (no-replace) vs 0.0117 (with-replace, the expected shift from resampling duplicates). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Collaborator
Author
|
Per-guide titration subsampling now supports a true nonparametric bootstrap The per-guide selection is also rewritten as a single vectorized helper Option 1 semantics: with replacement still keeps min(count, budget) cells per Timing (synthetic, 1,000,000 cells / 1,500 guides, budget 30 cells/guide): End-to-end sanity (lysosome_LAMP1, 30 cells/guide, 3 draws): both modes run and |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.