3274 hi goodtimes filter out gain tests

[tmplummer]

Sorry for the size of this PR. Hi has requested I get these changes in for COSPAR next week 😬.
Closes: #3274

Summary

This branch (3274-hi-goodtimes---filter-out-gain-tests) adds per-pointing gain-configuration classification and detector-voltage-based gain-test filtering to IMAP-Hi processing. Summary of changes:

Details

New capability: gain configuration classification

• New ancillary file imap_hi_{sensor}-gain-configuration_*.csv — multi-indexed by (config_id, esa_energy_step), forward-filled HV nominal/tolerance columns per {field}_v/{field}_delta_v, plus geometric_factor per row. Loaded via hi/utils.py::load_gain_configuration().
• hi_l1b.py: classify_gain_configuration() matches a pointing's first HVSCI segment against the table to determine its config_id; get_gain_configuration_lut() builds a MET-range lookup (mirroring the existing EsaEnergyStepLookupTable pattern) so mid-pointing gain-test excursions are excluded independently of ESA-voltage matching.

L1B DE processing

• de_esa_energy_step() (ESA-voltage matching) and de_gain_configuration() (gain-config classification/exclusion) are fully decoupled functions, each independently testable, run in sequence (de_ccsds_qf() → de_esa_energy_step() → de_gain_configuration()).
• All three progressively build up a single ccsds_qf quality-flag variable in place (BAD_ESA_VOLTAGE, BAD_DETECTOR_VOLTAGE, plus existing PACKET_FULL/BADSPIN) rather than juggling intermediate variables.
• New gain_configuration_id global attribute on the L1B DE CDF records the pointing's classified config (or a NO_MATCH sentinel), set directly by de_gain_configuration().
• Removed the earlier per-DE geometric_factor variable — no longer duplicated across every event.

L1C PSET processing

• pset_geometric_factor() looks up geometric factor per esa_energy_step directly from the gain-configuration ancillary file using the L1B gain_configuration_id, producing the natural (esa_energy_step,)-shaped PSET variable.

Goodtimes

• mark_bad_esa_voltage/CullCode.BAD_ESA_VOLTAGE renamed to mark_bad_voltage/CullCode.BAD_HV_VALUE (same culling l
• Removed the superseded mark_bad_detector_voltage, CullCode.BAD_DETECTOR_VOLTAGE, and the old detector-voltage-limits ancillary file/loader.

Plumbing & schema

• cli.py updated to fetch/pass the new gain-configuration ancillary file to both L1B and L1C branches.
• CDF variable attrs updated (ccsds_qf VAR_NOTES for all 4 bits, pset_geometric_factor definition); cull_flags confirmed to stay uint8 (only 8 cull codes exist, no widening needed).
• Tests updated/added across test_hi_l1b.py, test_hi_l1c.py, test_hi_goodtimes.py, test_utils.py, test_cli.py. Fu passing, ruff clean.

[Copilot]

Pull request overview

Adds gain-configuration classification and gain-test filtering to IMAP-Hi processing by introducing a gain-configuration ancillary table, using it to classify pointings / flag HV excursions in L1B DE, and propagating the classified configuration into L1C PSET geometric-factor lookup.

Changes:

• Introduces imap_hi_{sensor}-gain-configuration_*.csv and a loader (load_gain_configuration) plus a new MET→config lookup (GainConfigLookupTable).
• Updates Hi L1B DE processing to (a) build ccsds_qf incrementally and (b) filter/flag packets whose detector HV does not match the pointing’s gain configuration; adds gain_configuration_id global attr.
• Updates Hi L1C PSET generation to compute pset_geometric_factor via the gain-configuration ancillary table; updates CLI plumbing and test suite accordingly.

Reviewed changes

Copilot reviewed 14 out of 14 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
imap_processing/cli.py	Wires new gain-configuration ancillary dependency into L1B/L1C processing.
imap_processing/hi/hi_l1b.py	Adds gain-config classification + detector-HV-based filtering and new gain_configuration_id attribute.
imap_processing/hi/hi_l1c.py	Adds pset_geometric_factor lookup using gain_configuration_id and gain-config ancillary file.
imap_processing/hi/hi_goodtimes.py	Renames ESA-voltage culling to generalized HV culling and updates cull code name.
imap_processing/hi/utils.py	Adds gain-config constants, CSV loader, and GainConfigLookupTable.
imap_processing/quality_flags.py	Extends Hi L1B DE packet flags with BAD_ESA_VOLTAGE and BAD_DETECTOR_VOLTAGE bits.
imap_processing/cdf/config/imap_hi_variable_attrs.yaml	Updates ccsds_qf bit documentation and adds PSET geometric-factor variable attrs.
imap_processing/tests/test_cli.py	Updates CLI dependency expectations for new ancillary input.
imap_processing/tests/hi/conftest.py	Adds session fixture for gain-configuration ancillary path.
imap_processing/tests/hi/test_hi_l1b.py	Adds/updates tests for new L1B gain-config logic and ccsds_qf behavior.
imap_processing/tests/hi/test_hi_l1c.py	Updates tests for new L1C gain-config dependency and geometric factor lookup.
imap_processing/tests/hi/test_hi_goodtimes.py	Updates tests for renamed HV culling and cull code rename.
imap_processing/tests/hi/test_utils.py	Adds tests for load_gain_configuration behavior/validation.
imap_processing/tests/hi/data/l1/imap_hi_90sensor-gain-configuration_20240101_v001.csv	Adds test ancillary gain-configuration table used by unit tests.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.