Restructuring: top-level layout + workflow/papers/scratch division

[cailmdaley]

Restructuring: top-level layout + workflow / papers / scratch split

Reorganizes sp_validation around one principle — the things you run live at the top — with a clean split between analysis, paper figures, and personal scratch, plus a modular Snakemake workflow built for more than one person. Base branch: develop (untouched).

Layout

sp_validation/
├── src/sp_validation/   library code (incl. the glass_mock core)
├── cosmo_val/           shear / cosmology validation — code + config
├── cosmo_inference/     inference — code + config (cosmosis / cosmocov)
├── workflow/            ALL analysis — modular Snakemake, multi-person → results/
├── papers/              final-figure assembly only — bmodes, catalog, cosmo_val, harmonic
├── scripts/             reduction scripts (+ examples/, glass_mock/)
├── scratch/             per-person ad hoc work, tracked (cdaley/, guerrini/)
├── results/             analysis products + diagnostic plots (contents gitignored)
└── docs/  config/  + tests under src/sp_validation/tests/

Previously cosmo_val was buried inside notebooks/ while cosmo_inference/ sat at the top, so you had to hunt for where each piece lived. Now the things you actually run sit side by side, sharing src/ underneath.

Division of labor

The boundary is the inputs to a paper figure: everything up to that point is analysis, the figure itself is presentation.

• workflow/ — all analysis. Generic, reusable, modular. Produces analysis products and diagnostic plots into results/. The bulk of the work lives here.
• papers/{name}/ — final-figure assembly only. Figure PDFs, colour, layout. Tied to one paper; may never touch Snakemake.
• scratch/{person}/ — personal, ad hoc, tracked. One-off experiments and custom workflows; tracked because seeing each other's scratch is useful.

Modular workflow

Nothing here is computed once — the catalog changed ~20× in the first release, and every paper varies the data vector, covariance, and inference. So the workflow is parameterized: the rules are shared, the config changes per run. Snakemake's module directive imports the rules under your own config and an output prefix, with per-rule override:

module analysis:
    snakefile: "../../workflow/Snakefile"
    config:    config           # this run's catalog, cuts, blind
    prefix:    "results/bmodes"  # products land here — no clobbering
use rule * from analysis

Each run namespaces under results/{run}/; a --dry-run on every composition guards against silent breakage as the structure grows.

Notebooks

The analysis tree is now notebook-free — the top-level notebooks/ directory is gone, and every notebook was moved to a proper home, converted, or deleted:

• Reusable code → library / scripts. Reduction-notebook logic lives in src/sp_validation/; runnable workflows in scripts/examples/ (e.g. extract_info.py, calibrate_comprehensive_cat.py, leakage_minimal.py).
• Tutorial → docs. tutorial_UNIONS_SP_v1.0 is now the live Sphinx page "Using the weak-lensing catalogues".
• Paper-plot notebooks → papers/. The harmonic plot set and the catalog check_gaia plot moved under papers/{harmonic,catalog}/ (kept as notebooks — final-figure assembly).
• cosmo_val/ resolved. Generic helpers lifted into src/sp_validation/basic.py; the five working notebooks converted to jupytext percent-light scripts under scratch/guerrini/. cosmo_val/ now holds only code + config + README.
• Obsolete deleted. The exploratory reduction notebooks, glass_mock/validate_glass_mock, and defunct/ (quarantined since 2024) — all recoverable from develop history.
• Discipline via tooling. nbstripout strips notebook outputs on commit, plus a large-file pre-commit hook (pre-commit install; see CONTRIBUTING).

Full per-notebook ledger (every .ipynb moved or deleted vs develop)

Moved → papers/harmonic/ (from cosmo_inference/notebooks/2D_harmonic_space_cosmic_shear_plots/, preserved as notebooks):
2025_09_26_plot_contours + its variants (_NL_modelling, _blind, _cl_vs_xi, _covariance, _glass_mock, _iNKA_vs_OneCov, _leakage, _scale_cut, _small_vs_large_scales, _weak_lensing), 2025_10_28_plot_whisker, S8_whisker

Moved → papers/catalog/ (from notebooks/cosmo_val/catalog_paper_plot/):
check_gaia

Promoted → docs (from notebooks/analyse_shear_cat/):
tutorial_UNIONS_SP_v1.0 → docs/source/using_the_catalogues.md

Converted → scripts in scratch/guerrini/ (from notebooks/cosmo_val/; reusable helpers chi2_and_pte, corr_from_cov, cov_from_one_covariance lifted to src/sp_validation/basic.py):
compute_pte_cell.py, one_covariance.py, plot_comparison.py, get_prior_leakage.py, exploration.py (+ its namaster_utils.py helper)

Deleted (recoverable from develop history):

• notebooks/: main_set_up, metacal_global, metacal_local, psf_leakage, maps, maps_local, match_stats, correlation, cosmology, write_cat, frac_error_local_calib, analyse_matched_stars_UNIONS_HSC, analyse_shear_cat/m2_SP_LF_alpha
• glass_mock/: validate_glass_mock
• defunct/: TD_WL_cycle2_2021, validation_local_cal

glass_mock

The generation core folded into src/sp_validation/glass_mock.py; the runner scripts moved to scripts/glass_mock/. The top-level glass_mock/ directory is gone.

Still open

Intentionally left for follow-up passes:

• cosmo_inference/ notebooks still need the same notebook-free treatment as cosmo_val/ — the 2D cosmic-shear paper plots belong in papers/ (parallel to papers/harmonic/), the working notebooks in scratch//scripts. Deferred to a careful pass on Candide where the inference stack runs, so each migration can be executed and verified rather than transformed blind.
• scratch/ promotion candidates — some scratch material may still deserve a home in src/ or workflow/, most notably scratch/guerrini/namaster_utils.py, a NaMaster covariance toolkit that overlaps cosmo_val/harmonic_covariance_gaussian_sims.py. Consolidating the two (with tests) is a covariance-API decision best left to Sacha.

Safety net

A back-pressure guard suite holds structural invariants as files move — imports + standalone-scripts/ resolution, snakemake -n dry-runs, config-path existence, symlink integrity, and a dangling-reference / move-map guard. Sacha's develop foundation (paper plots, harmonic configs, library changes) is folded in, with the cosmology.py mnu KeyError fixed.

— Claude on behalf of Cail

[martinkilbinger]

Can we cluster scripts that are used for calibration? These would be (at the moment, this might change for v2.0)

• extract_info.py (to extract relevant metacal and other information from final shapepipe output)
• create_joint_comprehensive_cat.py (create joined comprehensive catalogue merging all patches; v2.0 no longer has patches)
• demo_apply_hsp_masks.py (add structural masks)
• calibrate_comprehensive_cat.py (performs galaxy selection and calibrates)
  In the last step, config files from config/calibration are used.

[martinkilbinger]

Please move to scratch:

• analyse_matched_stars_UNIONS_HSC.ipynb (-> as py script)
• demo_binned_mask.py
• plot_binned_quantities.py

This can be deleted:

• leakage_minimal.py
• tests_bump.py
• create_shear_mb_empty.py
• demo_add_bands.py
• demo_add_bands_to_empty.py
• des_y3_cat.py
• star_response.py

[sachaguer]

The PR looks fine to me.

Regarding the namaster_utils script being redundant with other scripts, I think I will handle it when I will upgrade cosmo_val harmonic space scripts to tomography.
One objective would be to move the namaster_utils to the source so that one can use it directly and use cosmo_val to iterate on the basic functions. I will use it to refactor the script computing the Gaussian simulations with less duplication of code. I would keep it for the next PR.

Some paper scripts from configuration space paper remain to be moved to the papers directory. The folder 2D_consistency_check can be moved there as well. It contains figures regarding the consistency between harmonic and configuration space.

I am sure that we will find bugs when we will try to run some of the scripts but, as for me, the objective of this PR to refactor the global structure of the repository has been met.

[cailmdaley]

Thanks for the comments both of you! @martinkilbinger I have started addressing your requests in #199. Will merge this now so we can keep moving and do targeted bugfix/improvement PRs directly to develop.