diff --git a/.gitignore b/.gitignore index 49e7696..e9244d7 100644 --- a/.gitignore +++ b/.gitignore @@ -15,3 +15,10 @@ venv/ # ASI application logs ASI/logs/ + +# Large ASI binary (144 MB) — exceeds GitHub's 100 MB push limit. +# Distributed as ASI/tpx3dump.zip instead; keep the raw binary out of git. +ASI/tpx3dump + +# UI subsystem logs (written by LogManager) +logs/ diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index a7afe42..94e4dff 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -32,15 +32,15 @@ repos: description: Check for files that contain merge conflict strings - id: check-added-large-files description: Prevent giant files from being committed - args: ['--maxkb=1000'] + args: ['--maxkb=100000'] - # black / isort / flake8 — versions from pyproject.toml via scripts/precommit_lint.sh + # black / isort / flake8 — versions from pyproject.toml via tools/dev/precommit_lint.sh - repo: local hooks: - id: lint name: black / isort / flake8 language: system - entry: bash scripts/precommit_lint.sh + entry: bash tools/dev/precommit_lint.sh pass_filenames: false always_run: true @@ -50,7 +50,7 @@ repos: - id: pytest name: pytest language: system - entry: bash scripts/precommit_pytest.sh + entry: bash tools/dev/precommit_pytest.sh pass_filenames: false always_run: true diff --git a/ASI/live-cli b/ASI/live-cli index fa56fa4..4c7ff44 100755 Binary files a/ASI/live-cli and b/ASI/live-cli differ diff --git a/ASI/long-daq/live-cli b/ASI/long-daq/live-cli new file mode 100644 index 0000000..fa56fa4 Binary files /dev/null and b/ASI/long-daq/live-cli differ diff --git a/ASI/tpx3dump.zip b/ASI/tpx3dump.zip new file mode 100644 index 0000000..1770921 Binary files /dev/null and b/ASI/tpx3dump.zip differ diff --git a/ASI/ui-updates/live-cli b/ASI/ui-updates/live-cli new file mode 100644 index 0000000..78ad897 Binary files /dev/null and b/ASI/ui-updates/live-cli differ diff --git a/README.md b/README.md index b3a2fe2..f31cd19 100644 --- a/README.md +++ b/README.md @@ -6,6 +6,8 @@ Streaming and pre-processing time-resolved TimePix3 detector data This package provides a pipeline for streaming, pre-processing, and visualizing data from Amsterdam Scientific Instruments (ASI) TimePix3 detectors. It implements time-resolved spectroscopy binning with TDC (Time-to-Digital Converter) triggering and supports both real-time visualization and ZMQ publishing for downstream analysis. +It also ships a PySide6 **acquisition UI** (`tpx-ui`) that drives the full pipeline — beam alignment, acquisition control, live heatmaps/spectra, clustering parameter sweeps, unified logs, and a session timeline (see [Acquisition UI](#5-acquisition-ui-tpx-ui)). + ## Platform Requirements - **Python 3.9+** @@ -23,16 +25,61 @@ source .venv/bin/activate pip install -e .[dev] ``` +To use the PySide6 acquisition UI, also install the optional `ui` extra: +```bash +pip install -e ".[ui]" +``` + +### Linux: UDP receive buffers (Serval / streaming host) + +On any **Linux** machine that will **receive high-rate detector UDP traffic** (for example the Serval or acquisition PC), configure kernel UDP socket buffer limits **before your first live acquisition**. Serval expects a large receive buffer; the default kernel ceiling can silently cap it and increase the risk of packet loss under burst traffic. + +Follow the step-by-step **`sysctl` / `sysctl.d` instructions** in [UDP receive buffers (first-time Linux setup)](info/UDP_RECEIVER_BUFFER_LIMITS.md). + +### Optional: register the UI with GNOME + +On a fresh Linux/GNOME machine the UI launches fine via `tpx-ui`, but the +window will appear in the dock with a generic gear icon and a `main.py` +tooltip until the desktop integration is installed. The repository ships a +script that creates the icon, an installed `.desktop` launcher, and a +clickable shortcut on the Desktop, all derived from the current clone path: + +```bash +./scripts/install_desktop_integration.sh +``` + +Re-running is safe (every output is overwritten). To remove: + +```bash +./scripts/install_desktop_integration.sh --uninstall +``` + +What it sets up: + +| Path | Purpose | +|------|---------| +| `~/.local/share/icons/hicolor/256x256/apps/splash_timepix.png` | Themed icon for the dock and Activities. | +| `~/.local/share/applications/splash_timepix.desktop` | Launcher GNOME indexes; `StartupWMClass=splash_timepix` pairs the running window to this entry. | +| `~/Desktop/splash_timepix.desktop` | Clickable shortcut on the user's Desktop. | + +After running it once, log out and back in (or restart the dock extension) +so GNOME picks up the new launcher. On Wayland-GNOME the dock will then +show the bundled icon instead of the gear. + ### Dependencies - `numpy` - Array operations +- `h5py` - HDF5 I/O (centroider outputs) - `opencv-python` - Real-time visualization - `pyzmq` - ZMQ publishing - `msgpack` - Serialization for ZMQ - `typer` - CLI interface - `psutil` - System monitoring +- `requests` - HTTP client (Serval status) - `pydantic` - Message schema validation +UI extra (`pip install -e ".[ui]"`): `PySide6`, `pyqtgraph`. + ## Configuration - All tool and code style configurations are in `pyproject.toml` @@ -47,13 +94,14 @@ pre-commit install Additional markdown guides in the repository: - [Socket server](info/SOCKET_SERVER_README.md) — threading, ring buffer, and parser callback behavior -- [Implementation overview](info/IMPLEMENTATION_OVERVIEW.md) — start/stop ZMQ messages, schemas, and listener pattern (ArroyoXPS alignment) - [Sample start message](info/SAMPLE_START_MESSAGE.md) — example ZMQ start message (wire format and fields) -- [Testing guide](info/TESTING_GUIDE.md) — manual and automated testing (simulator, ZMQ subscribers, ArroyoXPS integration) +- [UDP receive buffers (first-time Linux setup)](info/UDP_RECEIVER_BUFFER_LIMITS.md) — raise `net.core.rmem_*` for Serval/streaming hosts (`sysctl.d` drop-in) +- [Testing guide](info/TESTING_GUIDE.md) — manual and automated testing (simulator, ZMQ subscribers, ArroyoXPS integration). +- [Centroider tutorial](info/centroider_tutorial.md) — scientist's guide to the Centroider tab (clustering parameter sweeps on `.tpx3` files). ## Architecture -The system consists of four main components: +The system consists of five main components: ### 1. Socket Server (`socket_server.py`) Multi-threaded TCP server that receives 12-byte TimePix3 packets and parses them with NumPy (`parser`): @@ -68,6 +116,8 @@ Main application implementing time-resolved binning: - Bins pixel events into 3D arrays (x, y, time) based on TDC triggers - Flushes accumulated 3D arrays to processing queue - Provides statistics display and user commands +- Publishes a heartbeat with queue depths and server state on a separate ZMQ port (default 5658) so the UI can monitor pipeline health +- Supports an `--alignment` mode: TDCs are ignored and wall-clock-gated 2D X/Y histograms are emitted at 1–30 Hz (used by the UI's Alignment tab) ### 3. Worker Threads Two alternative workers for consuming 3D arrays: @@ -91,9 +141,30 @@ Simulated TimePix3 data source for testing: - Configurable pixel count rate and TDC frequency - Interactive CLI for control +### 5. Acquisition UI (`tpx-ui`) +PySide6 desktop application (`src/splash_timepix/ui/`) that orchestrates Serval, the streaming server, and live-cli, and visualizes the ZMQ stream. Tabs: + +- **Alignment** (default) — live 2D X/Y heatmap at 1–30 Hz for beam alignment, with grayscale LUT, binarize/log/integrate display modes, target overlay, and a rolling cps strip chart. Includes a local simulator mode that needs no hardware. +- **Operator** — acquisition control (Start / Preview / Simulator / Replay), live current-flush and running-average heatmaps, ROI cursors with energy-calibrated spectra, pipeline queue monitoring, and saving of averaged data (PNG, CSV, energy/time axes, JSON metadata) under a shared per-scan slug. +- **Logs** — unified system/server/Serval log viewer; all logs are also written to `/logs/` for post-mortem debugging. +- **Timeline** — chronological overview of the session's runs and log events in one place. +- **Centroider** — clustering parameter sweeps (eps-s × eps-t) on `.tpx3` files via the bundled `tpx3dump`, with side-by-side x-histogram comparison. See the [Centroider tutorial](info/centroider_tutorial.md). + +Launch with: +```bash +tpx-ui +``` +Widget preferences (acquisition settings, alignment display options) persist across sessions. Only one UI instance can run at a time (a lock with a takeover prompt guards against double-starts). + ## Quick Start - Basic Usage Examples -The package installs **`tpx-stream`** as the console entry point for the streaming app (see `pyproject.toml`); `python -m splash_timepix.app` is equivalent. +The package installs three console entry points (see `pyproject.toml`): + +| Command | Equivalent | Purpose | +|---------|------------|---------| +| `tpx-stream` | `python -m splash_timepix.app` | Streaming server | +| `tpx-ui` | `python -m splash_timepix.ui.main` | Acquisition UI | +| `tpx-sim` | `python -m splash_timepix.simulator_cli` | Simulated data source | ### Production Mode (ZMQ Publishing) @@ -113,7 +184,7 @@ python -m splash_timepix.example_zmq_sub **Terminal 3** - Start detector: ```bash -./ASI/live-cli_alpha-1/live-cli +./ASI/live-cli ``` ### Display Real-time Data (Live Plotting) @@ -126,9 +197,9 @@ tpx-stream --plot **Terminal 2** - Start data source: ```bash # Using real detector -./ASI/live-cli_alpha-1/live-cli +./ASI/live-cli # OR replaying from file -./ASI/live-cli_alpha-1/live-cli --source-files path/to/recording.tpx3 +./ASI/live-cli --source-files path/to/recording.tpx3 # OR using the simulator python -m splash_timepix.simulator_cli ``` @@ -153,7 +224,7 @@ cps 100000 tdc 0.1 start 60 # OR (III) Using Replay From File (live-cli) -./ASI/live-cli_alpha-1/live-cli --source-files path/to/recording.tpx3 +./ASI/live-cli --source-files path/to/recording.tpx3 ``` ## Command-Line Options @@ -173,13 +244,21 @@ python -m splash_timepix.app [OPTIONS] - `--buffer-size INT`: Internal message queue size (default: 1000) - `--callback-batch-size INT`: Number of parsed packets to batch per callback (default: 10000) - `--zmq-port INT`: ZMQ publishing port (default: 5657) +- `--heartbeat-port INT`: ZMQ heartbeat/status port (default: 5658) +- `--exit-on-disconnect`: Stop the server when the client disconnects (default: keep running) **Time-Resolved Binning Options:** -- `--tdc-ch INT`: TDC channel to use - 0=both, 1=ch1, 2=ch2 (default: 1) +- `--tdc-ch INT`: TDC channel to use - 0=both, 1=ch1, 2=ch2 (default: 0) - `--tdc-edge STR`: TDC edge to trigger on - "rising" or "falling" (default: rising) -- `--tdc-frequency` FLOAT: Expected TDC trigger frequency in Hz (default: 1.0) -- `--t-delta-ns FLOAT`: Time bin width in nanoseconds (default: 10) -- `--flush-interval FLOAT`: Time between 3D x, y, t array flushes in seconds (default: 2.0) +- `--tdc-frequency FLOAT`: Expected TDC trigger frequency in Hz (default: 100.0) +- `--t-delta-ns FLOAT`: Time bin width in nanoseconds (default: -1 = auto; bin width is derived from `--n-bins`) +- `--n-bins INT`: Number of time bins per TDC cycle, used when `--t-delta-ns` is not given (default: 350) +- `--flush-interval FLOAT`: Time between 3D x, y, t array flushes in seconds (default: 1.0) +- `--collapse-y`: Sum over the Y axis and emit 2D (x, t) arrays instead of 3D + +**Alignment Mode:** +- `--alignment`: Ignore TDCs and emit wall-clock-gated 2D X/Y histograms (used by the UI's Alignment tab) +- `--alignment-rate-hz FLOAT`: Flush rate for alignment mode, 1–30 Hz (default: 30.0) **Display Options:** - `--stats-update-time INT`: Stats refresh interval in seconds (default: 1) @@ -190,6 +269,8 @@ After starting the test source, use these commands: - `cps ` - Set pixel count rate (events/second) - `tdc ` - Set TDC frequency (Hz) +- `count ` - Enable/disable packet counting statistics +- `batch ` - Wire-level burst batching interval (0 disables; sends per-packet) - `start ` - Start streaming for duration (seconds) - `stop` - Stop streaming data - `quit` - Exit @@ -203,12 +284,13 @@ Published when data acquisition begins (first data arrives): ```python { 'msg_type': 'start', - 'scan_name': 'acquisition_20250112T160536Z_8b850728', + 'scan_name': 'a1b2c3d4-e5f6-4a7b-8c9d-0e1f23456789', # full UUID4 'tdc_frequency_hz': 10.0, 'detector_size_x': 256, 'detector_size_y': 256, 'n_bins': 350, 't_delta_ns': 285714.29, + 'mode': 'timing', # 'timing' (default) or 'alignment' # ... other configuration parameters } ``` @@ -239,7 +321,7 @@ Published when data acquisition ends (client disconnects or server shuts down): ```python { 'msg_type': 'stop', - 'scan_name': 'acquisition_20250112T160536Z_8b850728', + 'scan_name': 'a1b2c3d4-e5f6-4a7b-8c9d-0e1f23456789', # same UUID as start 'total_flushes': 9, 'total_cycles': 99, 'total_packets': 50000, @@ -344,9 +426,13 @@ The application bins pixel events into 3D arrays based on TDC triggers: 4. Worker either **plots** or **publishes** the array **Key parameters:** +- `tdc_frequency`: Inverse of total time window to capture per TDC trigger (`t_cycle = 1 / tdc_frequency`) - `t_delta`: Width of each time bin (temporal resolution) -- `tdc_frequency`: Inverse of total time window to capture per TDC trigger -- `n_bins`: Automatically calculated as `ceil((1 / tdc_frequency) / t_delta)` +- `n_bins`: Number of time bins per cycle + +You specify either the bin width or the bin count; the other is derived: +- With `--t-delta-ns` set: `n_bins = ceil(t_cycle / t_delta)` +- Without it (default): `t_delta = t_cycle / n_bins` (with `--n-bins`, default 350) **Pixels outside the time window** or **before the first TDC** are discarded and counted for diagnostics. @@ -360,13 +446,13 @@ The application bins pixel events into 3D arrays based on TDC triggers: ### Live Detector (`live-cli`) Real-time streaming from TimePix3: ```bash -./ASI/live-cli_alpha-1/live-cli +./ASI/live-cli ``` ### Recorded Data (`live-cli --source-files`) Replay recorded `.tpx3` files: ```bash -./ASI/live-cli_alpha-1/live-cli --source-files path/to/file.tpx3 +./ASI/live-cli --source-files path/to/file.tpx3 ``` ## Development @@ -409,7 +495,7 @@ Data Source → Socket Server → Callback (Binning) → Processing Queue → Wo 3. Plotting/ZMQ Worker Thread (output bound) 4. Input Listener Thread (user commands) -See [info/SOCKET_SERVER_README.md](info/SOCKET_SERVER_README.md) for server details. For the start/stop design and message flow, see [info/IMPLEMENTATION_OVERVIEW.md](info/IMPLEMENTATION_OVERVIEW.md). +See [info/SOCKET_SERVER_README.md](info/SOCKET_SERVER_README.md) for server details. The start/stop wire format is documented in [info/SAMPLE_START_MESSAGE.md](info/SAMPLE_START_MESSAGE.md), and the schema definitions live in `src/splash_timepix/schemas.py`. ## Troubleshooting @@ -426,7 +512,7 @@ See [info/SOCKET_SERVER_README.md](info/SOCKET_SERVER_README.md) for server deta ### Queue overflow warnings - Increase `--buffer-size` - Reduce callback processing time -- Increase `--callback_batch_size` +- Increase `--callback-batch-size` - Check if worker thread is keeping up ### Qt warnings on shutdown diff --git a/info/IMPLEMENTATION_OVERVIEW.md b/info/IMPLEMENTATION_OVERVIEW.md deleted file mode 100644 index 50f8704..0000000 --- a/info/IMPLEMENTATION_OVERVIEW.md +++ /dev/null @@ -1,273 +0,0 @@ -# Implementation Overview: Start/Stop Messages for splash_timepix - -## Executive Summary - -We successfully implemented start/stop message functionality for the splash_timepix data acquisition system, following the same architectural pattern used in ArroyoXPS. This enables downstream consumers to track the complete lifecycle of data acquisition runs, from initialization through data collection to completion. - -## Objective - -Add start and stop control messages to the existing ZMQ data stream, allowing downstream processors to: -- Know when an acquisition run begins (start message with configuration) -- Track data flushes during acquisition (event messages) -- Know when an acquisition run ends (stop message with statistics) - -This pattern matches the architecture used in ArroyoXPS, making it easier to build unified processing pipelines. - -## Approach - -We used ArroyoXPS as a reference implementation, studying its: -- Message schema definitions (`schemas.py`) -- ZMQ listener pattern (`XPSLabviewZMQListener`) -- Operator pattern for message processing - -We then adapted this pattern for splash_timepix's TimePix3 data structures and message formats. - -## Implementation Details - -### 1. Message Schemas (`src/splash_timepix/schemas.py`) - NEW - -Created Pydantic-based schema definitions for three message types: - -**TimePixStart** -- Published when first data arrives -- Contains all acquisition configuration parameters -- Fields: scan_name, tdc_frequency, detector_size, timing parameters, etc. - -**TimePixEvent** -- Published for each data flush -- Contains array data (numpy) and flush metadata -- Fields: array, flush_number, cycles_in_flush, statistics, etc. - -**TimePixStop** -- Published when acquisition ends (client disconnect or server shutdown) -- Contains run summary statistics -- Fields: scan_name, total_flushes, total_cycles, duration, discarded pixels, etc. - -**Benefits:** -- Type safety and validation via Pydantic -- Clear documentation of message structure -- Easy serialization/deserialization - -### 2. ZMQ Worker Updates (`src/splash_timepix/workers.py`) - MODIFIED - -**Changes:** -- Added `message_queue` parameter to `zmq_worker()` function -- Added logic to publish start/stop control messages from queue -- Added `msg_type="event"` to data flush messages for identification -- Control messages (start/stop) are single-part; data messages are multi-part - -**Key Implementation:** -- Control messages have higher priority and are processed before data -- Uses non-blocking sends to avoid blocking on slow subscribers -- Handles ZMQ slow joiner problem with appropriate delays - -### 3. Main Application Updates (`src/splash_timepix/app.py`) - MODIFIED - -**Changes:** -- Added imports: `uuid`, `datetime`, `TimePixStart`, `TimePixStop` -- Created `message_queue` for control messages (only when using ZMQ worker) -- Generated unique `scan_name` for each acquisition run -- Added tracking variables: `start_message_sent`, `acquisition_start_time` - -**Start Message Logic:** -- Sent in `data_callback()` when first data packet arrives -- Contains all configuration parameters from app initialization -- Queued to `message_queue` for ZMQ worker to publish - -**Stop Message Logic:** -- Sent when client disconnects (detected via `was_client_connected` flag) -- Also sent in `finally` block on server shutdown (if not already sent) -- Contains accumulated statistics: flushes, cycles, packets, duration, discarded pixels - -**Key Features:** -- Automatic lifecycle tracking -- No manual intervention required -- Works with both `--exit-on-disconnect` and normal modes - -### 4. ZMQ Listener (`src/splash_timepix/listener.py`) - NEW - -Created `SplashTimePixZMQListener` class following the ArroyoXPS pattern: - -**Functionality:** -- Subscribes to ZMQ PUB socket from splash_timepix -- Receives and parses messages (start/stop/event) -- Converts raw messages to schema objects (TimePixStart, TimePixStop, TimePixEvent) -- Calls operator callback function for each message -- Handles multi-part messages correctly (metadata + array bytes) - -**Usage Pattern:** -```python -def my_operator(message): - if isinstance(message, TimePixStart): - # Initialize processing - elif isinstance(message, TimePixEvent): - # Process data array - elif isinstance(message, TimePixStop): - # Finalize processing - -listener = SplashTimePixZMQListener( - zmq_address="tcp://localhost:5657", - operator=my_operator -) -listener.start() -``` - -**Benefits:** -- Enables operator pattern for downstream processing -- Type-safe message handling -- Consistent with ArroyoXPS architecture - -### 5. Example Files - NEW/MODIFIED - -**`src/splash_timepix/example_listener.py`** - NEW -- Demonstrates listener + operator pattern -- Shows how to process all three message types -- Example operator class with start/event/stop handlers - -**`src/splash_timepix/example_zmq_sub.py`** - MODIFIED -- Updated to handle start/stop control messages -- Distinguishes single-part (control) vs multi-part (data) messages -- Displays formatted output for all message types - -### 6. Testing Infrastructure - NEW - -**`tests/test_start_stop_messages.py`** - NEW -- Unit tests for schema validation -- Integration tests for full message flow -- Tests start/event/stop message reception - -**`tests/test_start_stop_quick.py`** - NEW -- Quick manual test script -- Subscribes to ZMQ and displays all messages -- Useful for debugging and verification - -### 7. Documentation Updates - -**`README.md`** - MODIFIED -- Added pydantic to dependencies -- Updated ZMQ message format section with all three message types -- Added "Using the Listener Pattern" section -- Updated architecture diagram -- Added test commands - -**`pyproject.toml`** - MODIFIED -- Added pydantic to dependencies - -## Message Flow - -``` -1. Server starts → waits for client connection -2. Client connects → server ready -3. First data arrives → START message sent - - Contains: scan_name, configuration parameters -4. Data flushes → EVENT messages sent (each flush) - - Contains: array data + metadata -5. Client disconnects → STOP message sent - - Contains: scan_name, statistics -6. Server shutdown → cleanup -``` - -## Technical Challenges Solved - -### 1. ZMQ Slow Joiner Problem -**Issue:** Subscribers connecting after messages are sent miss those messages. - -**Solution:** -- Added 2-second delay after server start for subscribers to connect -- Added 2-second delay in test scripts after connecting -- Increased ZMQ worker startup delay to 1 second - -### 2. Stop Message Timing -**Issue:** Stop message only sent on server shutdown, not when client disconnects. - -**Solution:** -- Added client disconnect detection via `was_client_connected` flag -- Send stop message immediately when client disconnects -- Also send in `finally` block as fallback -- Use flag to prevent duplicate stop messages - -### 3. Message Type Identification -**Issue:** Need to distinguish control messages from data messages. - -**Solution:** -- Control messages (start/stop): single-part (metadata only) -- Data messages (event): multi-part (metadata + array bytes) -- Added `msg_type` field to all messages -- Subscribers check message type and handle accordingly - -## Testing - -### Unit Tests -- Schema validation tests (all three message types) -- Import tests -- All tests passing - -### Integration Tests -- Full message flow: start → events → stop -- Message format validation -- Listener pattern verification - -### Manual Testing -- Verified start message on first data -- Verified event messages for each flush -- Verified stop message on client disconnect -- All message types received correctly - -## Files Summary - -### Created Files (5) -1. `src/splash_timepix/schemas.py` - Message schemas -2. `src/splash_timepix/listener.py` - ZMQ listener class -3. `src/splash_timepix/example_listener.py` - Listener example -4. `tests/test_start_stop_messages.py` - pytest tests -5. `tests/test_start_stop_quick.py` - Quick manual test - -### Modified Files (4) -1. `src/splash_timepix/app.py` - Added start/stop message sending -2. `src/splash_timepix/workers.py` - Added message queue support -3. `src/splash_timepix/example_zmq_sub.py` - Updated for start/stop -4. `pyproject.toml` - Added pydantic dependency -5. `README.md` - Updated documentation - -## Benefits - -1. **Lifecycle Tracking**: Downstream processors can track complete acquisition runs -2. **Type Safety**: Pydantic schemas provide validation and type hints -3. **Consistency**: Matches ArroyoXPS pattern for unified architecture -4. **Backward Compatible**: Existing subscribers continue to work -5. **Operator Pattern**: Enables structured message processing -6. **Better Debugging**: Clear start/stop boundaries for troubleshooting - -## Architecture Alignment - -The implementation follows the same pattern as ArroyoXPS: - -``` -ArroyoXPS: splash_timepix: -LabVIEW → ZMQ PUB app.py → ZMQ PUB - ↓ ↓ -XPSLabviewZMQListener SplashTimePixZMQListener - ↓ ↓ -XPSOperator Your Operator - ↓ ↓ -Publishers Downstream Processing -``` - -This makes it easier to: -- Build unified processing pipelines -- Share code between systems -- Train users on consistent patterns -- Maintain both systems - -## Future Enhancements - -Potential improvements: -- Add message versioning for schema evolution -- Add message filtering/topics for selective subscription -- Add message replay capability -- Add metrics/telemetry in messages -- Add support for multiple concurrent acquisitions - -## Conclusion - -The start/stop message implementation successfully adds lifecycle tracking to splash_timepix while maintaining backward compatibility and following established patterns from ArroyoXPS. The system is production-ready and fully tested. diff --git a/info/SAMPLE_START_MESSAGE.md b/info/SAMPLE_START_MESSAGE.md index 2a2f5dd..f5dd464 100644 --- a/info/SAMPLE_START_MESSAGE.md +++ b/info/SAMPLE_START_MESSAGE.md @@ -15,7 +15,7 @@ The following is the logical content of the start message as a JSON object. Over ```json { "msg_type": "start", - "scan_name": "acquisition_20250128T143022Z_a1b2c3d4", + "scan_name": "a1b2c3d4-e5f6-4a7b-8c9d-0e1f23456789", "tdc_frequency_hz": 1000.0, "t_delta_ns": 10.0, "t_cycle_ns": 1000000.0, @@ -28,7 +28,8 @@ The following is the logical content of the start message as a JSON object. Over "tdc_edge": "rising", "collapse_y": false, "zmq_port": 5657, - "tcp_port": 9090 + "tcp_port": 9090, + "mode": "timing" } ``` @@ -37,7 +38,7 @@ The following is the logical content of the start message as a JSON object. Over | Field | Type | Description | |-------|------|-------------| | `msg_type` | string | Always `"start"` for this message type. | -| `scan_name` | string | Unique identifier for this acquisition run. Format: `acquisition_YYYYMMDDTHHMMSSZ_` (UTC, ISO 8601). Example: `acquisition_20250128T143022Z_a1b2c3d4`. | +| `scan_name` | string | Unique identifier for this acquisition run. A full UUID4 (36 characters), e.g. `a1b2c3d4-e5f6-4a7b-8c9d-0e1f23456789`. | | `tdc_frequency_hz` | float | TDC trigger frequency in Hz. | | `t_delta_ns` | float | Time bin width in nanoseconds. | | `t_cycle_ns` | float | Full time cycle in nanoseconds. | @@ -51,10 +52,11 @@ The following is the logical content of the start message as a JSON object. Over | `collapse_y` | bool | Whether the Y dimension is collapsed in the 3D array. | | `zmq_port` | int | ZMQ publishing port for this server. | | `tcp_port` | int | TCP socket port for the live-cli connection. | +| `mode` | string | Acquisition mode: `"timing"` (default, TDC-driven time-resolved binning) or `"alignment"` (wall-clock-gated 2D X/Y histogram used by the Alignment tab). Subscribers route on this field; older subscribers can ignore it. | -## Time in `scan_name` +## `scan_name` format -The timestamp in `scan_name` is **UTC** in **ISO 8601** form: `YYYYMMDDTHHMMSSZ` (e.g. `20250128T143022Z`). It is generated when a new client connects and the run starts, using `datetime.now(timezone.utc)`, so it is unambiguous and sortable across machines and timezones. +`scan_name` is a **full UUID4** (36 characters), generated when a new client connects and the run starts. The same UUID is used by the UI's file-naming convention: saved output files (CSV, PNG, JSON metadata) embed the first 8 characters of this UUID plus a local timestamp, so downstream files can be matched back to the run's start message. ## Usage in code @@ -66,4 +68,4 @@ The timestamp in `scan_name` is **UTC** in **ISO 8601** form: `YYYYMMDDTHHMMSSZ` - **Event message** (`msg_type: "event"`): Multi-part ZMQ message (metadata + raw array bytes); one per flush. - **Stop message** (`msg_type: "stop"`): Single-part, msgpack; sent when acquisition ends (disconnect or shutdown). -See `IMPLEMENTATION_OVERVIEW.md` and `schemas.py` for full details. +See `src/splash_timepix/schemas.py` for the authoritative field definitions (`TimePixStart`, `TimePixEvent`, `TimePixStop`). diff --git a/info/TESTING_GUIDE.md b/info/TESTING_GUIDE.md index d2127f5..3b4fcf2 100644 --- a/info/TESTING_GUIDE.md +++ b/info/TESTING_GUIDE.md @@ -24,7 +24,7 @@ Use **three terminals**. Start in this order: server first, then listener, then ```bash # Go to splash_timepix project -cd /home/gabrielgazolla/Downloads/task/splash_timepix +cd /path/to/splash_timepix # Activate your splash_timepix environment (or base if installed there) # conda activate splash_timepix # if you use one @@ -41,7 +41,7 @@ Leave this running. You should see the server listening (e.g. on port 9090 for T conda activate arroyoxps # Go to ArroyoXPS project -cd /home/gabrielgazolla/Downloads/task/ArroyoXPS +cd /path/to/ArroyoXPS # Run the TimePix ZMQ listener (DummyOperator prints start/event/stop) python -m tr_ap_xps.timepix @@ -53,7 +53,7 @@ Leave this running. It will connect to `tcp://localhost:5657` (splash_timepix’ ```bash # Go to splash_timepix project -cd /home/gabrielgazolla/Downloads/task/splash_timepix +cd /path/to/splash_timepix # Same environment as Terminal 1 python -m splash_timepix.simulator_cli @@ -73,7 +73,7 @@ This sends data for 10 seconds. Type `stop` or wait for the run to finish. - **splash_timepix server (Terminal 1):** Logs when start/stop are queued and when flushes are published. - **ArroyoXPS listener (Terminal 2):** Connects to `tcp://localhost:5657`, receives and logs: - - **Start:** e.g. `Dummy operator received START: scan_name=acquisition_YYYYMMDDTHHMMSSZ_xxxxxxxx` (UTC, ISO 8601) + - **Start:** e.g. `Dummy operator received START: scan_name=` (scan_name is a full UUID4, e.g. `a1b2c3d4-e5f6-4a7b-8c9d-0e1f23456789`) - **Events:** e.g. `Dummy operator received EVENT with image shape: (256, 256, 350)` (shape may vary) - **Stop:** e.g. `Dummy operator received STOP` - **Simulator (Terminal 3):** Sends packets to the server; `start 10` runs for 10 seconds. @@ -93,7 +93,7 @@ This usually means the listener is running in the wrong environment or from the The `arroyopy` package (and correct `ZMQListener` signature) must be available in this env. - **Run from the ArroyoXPS project directory:** - `cd /home/gabrielgazolla/Downloads/task/ArroyoXPS` + `cd /path/to/ArroyoXPS` Then run `python -m tr_ap_xps.timepix` so that `tr_ap_xps` and its config (e.g. ZMQ port) resolve correctly. - **Check ArroyoXPS code:** In `tr_ap_xps.timepix`, `XPSTimepixZMQListener` must call `super().__init__(operator, zmq_socket)` (operator first, socket second) to match `arroyopy.zmq.ZMQListener`. If you still see the error after fixing env/cd, verify that `__init__` passes arguments in that order. @@ -172,10 +172,14 @@ This starts server and simulator in subprocesses, subscribes to ZMQ, and checks ```bash cd /path/to/splash_timepix -pytest tests/test_schemas.py -v -pytest tests/test_listener.py -v +# Schema validation + start/event/stop integration coverage +pytest tests/test_start_stop_messages.py -v ``` +The schema-only assertions live inside `class TestSchemas` in that file; the +remaining tests in the module spin up the streaming server via the +`streaming_rig` fixture and assert on real ZMQ traffic. + ### Method 4: Real hardware (TimePix3) 1. Start server: `cd /path/to/splash_timepix` then `python -m splash_timepix.app --tdc-frequency 1000` diff --git a/info/UDP_RECEIVER_BUFFER_LIMITS.md b/info/UDP_RECEIVER_BUFFER_LIMITS.md new file mode 100644 index 0000000..28a87e4 --- /dev/null +++ b/info/UDP_RECEIVER_BUFFER_LIMITS.md @@ -0,0 +1,59 @@ +# UDP receive buffers (first-time Linux setup) + +When you deploy **splash_timepix** with **Serval** and the ASI detector stack, one machine (often the acquisition PC) receives high-rate detector traffic over the network. Serval is configured to use a **large UDP receive buffer** (on the order of tens of MB) so short bursts and normal scheduling jitter do not overflow a tiny kernel queue. + +On many Linux installs, the kernel’s global ceiling for socket receive buffers (`net.core.rmem_max`) is still at a small default. Applications can *ask* for a large buffer, but the kernel will **silently cap** the granted size at that ceiling. **Raising `rmem_max` and `rmem_default` once on the streaming host ensures the buffer Serval expects is actually granted**, which avoids unnecessary packet loss under burst traffic and normal scheduling jitter. + +**Do this on the host that receives the detector UDP stream** (or on any machine where you run Serval and care about stable streaming), ideally **before** your first long run or production tuning. + +--- + +## 1. Apply for this session (immediate) + +```bash +sudo sysctl -w net.core.rmem_max=26214400 +sudo sysctl -w net.core.rmem_default=26214400 +``` + +These values are in bytes (here **25 MiB**). They apply immediately; they are **not** kept across reboot. + +--- + +## 2. Make the settings persistent + +Prefer a **drop-in file** under `/etc/sysctl.d/` instead of editing `/etc/sysctl.conf`. Drop-ins are easy to copy between machines, version in config management, or remove. + +```bash +sudo tee /etc/sysctl.d/99-serval.conf > /dev/null <<'EOF' +# Serval / detector UDP receive buffer tuning (25 MiB ceiling) +net.core.rmem_max=26214400 +net.core.rmem_default=26214400 +EOF +``` + +The `99-` prefix loads after most defaults so these values win when order matters. + +--- + +## 3. Load all sysctl files + +```bash +sudo sysctl --system +``` + +This applies configuration from `/etc/sysctl.conf`, `/etc/sysctl.d/*.conf`, `/run/sysctl.d/*.conf`, and `/usr/lib/sysctl.d/*.conf`. + +--- + +## 4. Confirm + +```bash +sysctl net.core.rmem_max net.core.rmem_default +``` + +Expected: + +``` +net.core.rmem_max = 26214400 +net.core.rmem_default = 26214400 +``` diff --git a/info/centroider_tutorial.md b/info/centroider_tutorial.md new file mode 100644 index 0000000..e2cb9ae --- /dev/null +++ b/info/centroider_tutorial.md @@ -0,0 +1,328 @@ +# Centroider Tab — Scientist's Tutorial + +**Purpose:** Find the best clustering parameters for your `.tpx3` data by running +a parameter sweep and comparing the resulting x-histograms side by side. + +--- + +## Background — Why clustering parameters matter + +The Timepix3 detector records individual pixel hits. Each photon or particle that +arrives produces one or more adjacent hits in both space and time. **Clustering** +groups those related hits into a single event called a **cluster**, whose centroid +gives sub-pixel position resolution. + +Two parameters control how aggressively hits are merged: + +| Parameter | What it controls | UI field | +|---|---|---| +| **eps-s** | Maximum pixel distance between hits in the same cluster | eps-s (pixels) | +| **eps-t** | Maximum time gap between hits in the same cluster (nanoseconds) | eps-t (ns) | + +Choosing these values incorrectly leads to: +- **Too small** → clusters are split; one real event becomes many small clusters, + broadening the peak and reducing count efficiency. +- **Too large** → unrelated hits are merged; clusters absorb neighbours, shifting + centroids and smearing the peak. + +The Centroider tab runs every combination of the values you specify across the full +eps-s × eps-t grid so you can compare the resulting x-histograms and pick the +best pair by eye — without having to run anything manually. + +--- + +## Quick start + +1. Open the **Centroider** tab in the application. +2. Click **Browse** and select a representative `.tpx3` file from your experiment. +3. Type a few candidate values into **eps-s (pixels)** and **eps-t (ns)**. +4. Press **Centroid** and watch the plot fill in as each run completes. +5. Choose the combination whose histogram peak is sharpest and best-resolved. + +The whole process typically takes a few minutes for a 20 MB file with a 3 × 3 grid. + +--- + +## The UI at a glance + +``` +┌─ Inputs ──────────────────────────────────────────────────────────────┐ +│ TPX3 File [ /path/to/data.tpx3 ] [ Browse ] │ +│ eps-s (pixels) [ 1,2,3 ] │ +│ eps-t (ns) [ 5,10,20 ] │ +│ [ Centroid ] [ ⏹ Stop ] │ +└───────────────────────────────────────────────────────────────────────┘ +┌─ Progress ─────────────────────────────────────────────────────────────┐ +│ [████████████░░░░░░░░░░░░░] [3/10] s=2&t=10ns: ok | ETA: 42s │ +└───────────────────────────────────────────────────────────────────────┘ +┌─ Combinations ─────────┐ ┌─ Summary plot ────────────────────────────┐ +│ t=5ns t=10ns … │ │ [ Waterfall ] │ +│ s=1 [ ] ok [ ] ok │ │ │ +│ s=2 [ ] ok [ ] … │ │ counts vs y (dispersive axis) │ +│ s=3 [ ] … [ ] … │ │ │ +└────────────────────────┘ └───────────────────────────────────────────┘ +``` + +--- + +## Step-by-step walkthrough + +### Step 1 — Select a TPX3 file + +Click **Browse** and navigate to the `.tpx3` file you want to analyse. You can +also type or paste the path directly into the text box. + +> **Tip:** Use a short, representative file from your experiment — you do not need +> the full dataset. A 20–50 MB file gives reliable statistics and runs in a few +> minutes per combination. + +--- + +### Step 2 — Choose eps-s values (pixels) + +Type a comma-separated list of positive integers into the **eps-s (pixels)** field. + +``` +1,2,3 +``` + +Each integer is the maximum allowed pixel distance between two hits that belong to +the same cluster. Typical starting values for a Timepix3 detector are **1, 2, 3**. + +| Value | Effect | +|---|---| +| 1 | Only immediately adjacent pixels are merged. Good for high-flux, sparse hits. | +| 2 | Merges hits within a 2-pixel radius. Standard starting point. | +| 3+ | Merges hits across a wider area. Useful for large charge-sharing clouds. | + +--- + +### Step 3 — Choose eps-t values (nanoseconds) + +Type a comma-separated list of integers (no unit needed — nanoseconds are assumed) +into the **eps-t (ns)** field. + +``` +5,10,20 +``` + +Each value is the maximum allowed time gap between hits that belong to the same +cluster. + +> **Important:** Only nanosecond values are accepted. Larger time windows (µs, ms) +> cause the clustering algorithm to run dramatically slower and will merge +> unrelated hits. Stick to the **5 – 500 ns range** for practical sweeps. + +| Value | Effect | +|---|---| +| 5 ns | Tight time window; only very fast charge clouds grouped. | +| 20 ns | Typical value for most experiments. | +| 100 ns | Broad window; useful if charge collection is slow. | +| 500 ns | Very broad; likely to over-cluster at high flux. | + +--- + +### Step 4 — Press Centroid + +Pressing **Centroid** starts the sweep. The button changes to **Centroiding…** +and the inputs are locked until the sweep finishes or is stopped. + +The tool runs every (eps-s, eps-t) combination sequentially, **plus one extra +PixelHits baseline run** (clustering disabled) that serves as the "raw hits" +reference. For a 3 × 3 grid this means 10 total runs. + +**Progress bar** — fills as each run completes. The label shows the current +combination, its status, and the estimated time remaining. + +**ETA** — computed as a running average of completed run times. It displays +"estimating…" until the first run finishes. + +--- + +### Step 5 — Watch the combinations grid + +The **Combinations** panel on the left shows a table where: +- **Rows** correspond to eps-s values. +- **Columns** correspond to eps-t values. +- Each cell shows the status of that run and a **show** checkbox. + +| Status label | Meaning | +|---|---| +| `queued` | Not yet started. | +| `running…` | tpx3dump is currently processing this combination. | +| `ok (7.3s)` | Completed successfully. Wall time shown in parentheses. | +| `cached` | Output file already existed from a previous run; skipped. | +| `failed` | tpx3dump returned a non-zero exit code. | + +Once a run finishes successfully, its **show** checkbox becomes enabled and its +histogram curve appears in the plot. + +--- + +### Step 6 — Read the summary plot + +The **Summary plot** on the right overlays one x-histogram curve per +combination, plus the grey PixelHits baseline. + +- **X axis:** y pixel position (the dispersive axis of the detector — the + physically meaningful direction for your experiment). +- **Y axis:** counts (number of events at each position). +- **Grey curve:** raw PixelHits baseline (before any clustering). Use this as + your reference for what the signal looks like without merging. +- **Coloured curves:** one per (eps-s, eps-t) combination. Colours are assigned + in order of completion and stay consistent within a sweep. + +**What to look for:** + +- A good set of parameters produces a **sharp, well-separated peak** (or peaks) + with low background between them. +- If the peak is broader than the PixelHits baseline, eps-s or eps-t is too + large and unrelated hits are being merged. +- If the peak is about the same width as the baseline, clustering is not helping + (eps values may be too small). +- The PixelHits baseline typically shows a smoother, broader distribution because + it has no sub-pixel centroid precision. + +**Interacting with the plot:** +- **Scroll** to zoom in/out. +- **Right-click → View All** to reset zoom. +- **Drag** to pan. +- Uncheck a **show** checkbox in the Combinations grid to hide that curve. + +--- + +### Step 7 — Waterfall mode (optional) + +If many curves overlap and are hard to distinguish, toggle the **Waterfall** +button above the plot. + +Waterfall mode offsets each visible curve vertically by a fixed step (1/10th of +the visible range), so overlapping histograms separate into a stacked display. +The relative shape of each curve is preserved; only the vertical position changes. +Toggle it off to return to the normal overlaid view. + +--- + +### Step 8 — Choose the best parameters + +Compare the curves and identify the combination that gives the sharpest, +best-resolved histogram. Note down the eps-s and eps-t values of that combination +— these are the parameters to use for your full data reduction. + +> **Rule of thumb:** start with the smallest values that produce a +> noticeably sharper peak than the PixelHits baseline. Going larger than +> necessary costs runtime and degrades centroid accuracy at high flux. + +--- + +## Stopping a sweep mid-run + +Press **⏹ Stop** at any time to interrupt the sweep. The application will: + +1. Send a termination signal to the currently running `tpx3dump` process. +2. Wait up to 3 seconds for it to exit cleanly; if it does not, send a + force-kill signal. +3. Delete the partially-written output `.h5` file (it would be corrupted). +4. Return the already-completed curves to the plot. + +The status bar will show how many runs completed, how many were cached, and how +many were cancelled. + +You can immediately re-run with adjusted parameters after stopping — press +**Centroid** again. + +> **On app close:** if a sweep is running when you close the application, the +> active `tpx3dump` process is killed automatically before the window closes. + +--- + +## Output files + +All outputs land in a single subdirectory next to your `.tpx3` file: + +``` +/path/to/data_centroided/ + data_t5ns_s1.h5 ← clustered HDF5, one per combination + data_t5ns_s2.h5 + ... + data_PixelHits.h5 ← raw hits baseline (no clustering) + summary.json ← timing, status, and command for each run + summary.csv ← same data in CSV format + luna.meta ← full reproducibility record (parameters + software version) +``` + +**HDF5 files:** each clustered `.h5` contains a `Clusters` dataset with columns +including the centroid positions `cx` (x) and `cy` (y). The PixelHits baseline +contains a `PixelHits` dataset with the raw hit coordinates. + +**summary.json / summary.csv:** produced after each run, so even a partially +completed sweep leaves a readable record of what finished. + +**luna.meta:** records the exact `tpx3dump` version, binary path, all parameters, +and a timestamp so any run can be reproduced exactly at a later date. + +--- + +## Caching — re-running without redoing work + +If you run the sweep a second time with the same file and parameter values, any +`.h5` file that already exists on disk is **skipped automatically** and its curve +is loaded directly — marked as `cached` in the grid. This means: + +- You can safely re-open the application and re-run to reload the plot from a + previous session. +- You can add new eps-s or eps-t values to an existing sweep; only the new + combinations run, the old ones are reused. +- To force a complete re-run (e.g. after updating the software), delete the + `_centroided/` directory and press Centroid again. + +--- + +## Frequently asked questions + +**Q: How do I pick my initial parameter range?** + +Start with eps-s = `1,2,3` and eps-t = `5,10,20`. This 3 × 3 grid covers the +most common practical range and completes in a few minutes. Once you see a clear +trend in the plot, narrow or expand from there. + +**Q: The PixelHits baseline always appears last — is that normal?** + +Yes. The tool runs all clustered combinations first (Phase 1), then the PixelHits +baseline (Phase 2). The grey baseline curve appears in the plot only after the +last run finishes. + +**Q: A run shows "failed" in the grid. What happened?** + +Hover over the cell to see the status, or check the application log for the +`tpx3dump` error message. Common causes are: the input file is corrupted, the +disk is full, or tpx3dump is not installed at the expected path. + +**Q: Can I run the sweep on multiple files?** + +Not simultaneously from the UI — run one file at a time. For batch processing +of many files, use the command-line tool directly: + +```bash +python tools/centroider/sweep.py \ + -i /path/to/data.tpx3 \ + -o /path/to/output \ + -t 5ns,10ns,20ns \ + -s 1,2,3 +``` + +**Q: Where is tpx3dump?** + +The application uses the binary bundled with the software: +``` +/ASI/tpx3dump +``` +No additional installation is needed; it is found automatically. To use a +different build, set the `TPX3DUMP` environment variable to its full path +before launching (applies to both the UI and the command-line sweep tool). + +**Q: Why are ms or s not accepted as time units?** + +At those timescales the clustering algorithm merges thousands of unrelated hits, +making each run take many minutes or hang entirely. Nanoseconds are the only +physically meaningful range for Timepix3 data. diff --git a/pyproject.toml b/pyproject.toml index a15f9f1..e068413 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -5,7 +5,7 @@ build-backend = "setuptools.build_meta" [project] name = "splash_timepix" -version = "0.6.0" +version = "0.6.3" description = "Code to stream from timepix" readme = "README.md" requires-python = ">=3.9" @@ -26,10 +26,10 @@ classifiers = [ "Programming Language :: Python :: 3 :: Only", ] -dependencies = ["numpy", "typer", "psutil", "opencv-python", "pyzmq", "msgpack", "requests", "pydantic"] +dependencies = ["numpy", "h5py", "typer", "psutil", "opencv-python", "pyzmq", "msgpack", "requests", "pydantic"] [project.optional-dependencies] -ui = ["PySide6>=6.5.0",] +ui = ["PySide6>=6.5.0", "pyqtgraph>=0.13"] # Pin linters to match .pre-commit-config.yaml (avoids CI vs local version drift). dev = [ "check-manifest", @@ -54,6 +54,9 @@ tpx-sim = "splash_timepix.simulator_cli:app" [tool.setuptools] package-dir = {"" = "src"} +[tool.setuptools.package-data] +"splash_timepix.ui" = ["assets/*.png"] + # Tool configurations [tool.black] line-length = 120 @@ -79,7 +82,8 @@ addopts = [ "-v", "--tb=short", "--strict-markers", - "--color=yes" + "--color=yes", + "-m", "not slow" ] markers = [ "slow: marks tests as slow", diff --git a/src/splash_timepix/app.py b/src/splash_timepix/app.py index c269b0b..2a761ea 100644 --- a/src/splash_timepix/app.py +++ b/src/splash_timepix/app.py @@ -13,7 +13,6 @@ import threading import time import uuid -from datetime import datetime, timezone import numpy as np import psutil @@ -72,6 +71,8 @@ def main( exit_on_disconnect: bool = False, collapse_y: bool = False, heartbeat_port: int = 5658, + alignment: bool = False, + alignment_rate_hz: float = 30.0, ): """ Time-resolved TimePix3 data streaming server. @@ -94,16 +95,38 @@ def main( exit_on_disconnect: Exit when client disconnects (for orchestrated runs) collapse_y: Send x,y,t (False) or x,t data (True) heartbeat_port: Port for ZMQ heartbeat messages (default: 5658) + alignment: Run in alignment mode — TDCs ignored, wall-clock 2D (x, y, 1) + histograms emitted at ``alignment_rate_hz``. Reuses the rest of the + pipeline (parse callback, xyt_lock, xyt_queue, zmq_worker) unchanged. + alignment_rate_hz: Wall-clock flush rate for alignment mode (1–30 Hz, default 30). """ _clear_screen_safely() print("Starting TimPix3 Streaming Application") print("=" * 50) + if alignment: + print("Mode: Alignment (TDC-free, wall-clock 2D X/Y histograms)") if exit_on_disconnect: print("Mode: Exit on client disconnect (orchestrated)") else: print("Mode: Persistent (Ctrl+C to stop)") print() + # Alignment-mode parameter coercion. We reuse the entire xyt_* pipeline + # (array, lock, queue, ZMQ worker) by forcing n_bins=1 and collapse_y=False + # so the existing allocation produces shape (X, Y, 1). Flush cadence is + # driven by alignment_rate_hz (1–30 Hz). TDC fields stay at their defaults + # — alignment callbacks never read them, but downstream pydantic schemas + # and the t_cycle math require finite values. + if alignment: + if alignment_rate_hz < 1.0 or alignment_rate_hz > 30.0: + logger.warning(f"alignment_rate_hz {alignment_rate_hz} outside 1-30; clamping") + alignment_rate_hz = max(1.0, min(30.0, alignment_rate_hz)) + n_bins = 1 + collapse_y = False + flush_interval = 1.0 / alignment_rate_hz + if tdc_frequency <= 0: + tdc_frequency = 1.0 # placeholder; never read in alignment mode + # Calculate binning and display parameters from user inputs t_cycle = (1.0 / tdc_frequency) * 1e12 # seconds → picoseconds t_cycle_ticks = t_cycle / TIMESTAMP_PS_PER_TICK @@ -172,13 +195,13 @@ def _queue_stats_for_heartbeat(): heartbeat.set_queue_stats_provider(_queue_stats_for_heartbeat) - # Generate unique scan name - # Generate initial scan_name (will be regenerated for each new client connection) - # UTC, ISO 8601 format (YYYYMMDDTHHMMSSZ) for unambiguous, sortable identifiers + # UUID for the current acquisition. + # Full UUID4 (36 chars) so it matches scan_name in the UI's *_meta.json at save time. + # Single source of truth: generated exclusively in the client-connect block below. def generate_scan_name(): - return f"acquisition_{datetime.now(timezone.utc).strftime('%Y%m%dT%H%M%SZ')}_{uuid.uuid4().hex[:8]}" + return str(uuid.uuid4()) - scan_name = generate_scan_name() + scan_name: str | None = None # set by client-connect block; None until first connect # Define x, y, t accumulator array config = SimulatorConfig() @@ -191,6 +214,13 @@ def generate_scan_name(): else: xyt_array = np.zeros((detector_size_x, detector_size_y, n_bins), dtype=np.uint32) + logger.info( + "XYT array allocated: shape=%s, dtype=%s, size=%.2f MB", + xyt_array.shape, + xyt_array.dtype, + xyt_array.nbytes / 1e6, + ) + xyt_lock = threading.Lock() # Build static metadata dict (parameters that don't change during run) @@ -207,6 +237,7 @@ def generate_scan_name(): "tdc_channel": tdc_ch, "tdc_edge": tdc_edge, "collapse_y": collapse_y, + "mode": "alignment" if alignment else "timing", } # Choose worker based on plot flag @@ -225,13 +256,14 @@ def generate_scan_name(): # Calculate flush cycles from interval and frequency flush_every_n_cycles = max(1, int(flush_interval * tdc_frequency)) - # Errors and warnings - if tdc_frequency < 0.1: - logger.error(f"Low TDC frequency ({tdc_frequency} Hz) → detector may miss TDC events") - return - if flush_interval < (1.0 / tdc_frequency): - logger.warning(f"Flush interval ({flush_interval}s) < TDC period ({1.0/tdc_frequency:.2f}s)") - logger.warning(" Will flush every TDC cycle (flush_every_n_cycles = 1)") + # Errors and warnings (TDC-related — skipped in alignment mode where TDCs are unused). + if not alignment: + if tdc_frequency < 0.1: + logger.error(f"Low TDC frequency ({tdc_frequency} Hz) → detector may miss TDC events") + return + if flush_interval < (1.0 / tdc_frequency): + logger.warning(f"Flush interval ({flush_interval}s) < TDC period ({1.0/tdc_frequency:.2f}s)") + logger.warning(" Will flush every TDC cycle (flush_every_n_cycles = 1)") # Memory check array_size_gb = xyt_array.nbytes / (1024**3) @@ -255,11 +287,11 @@ def generate_scan_name(): process = psutil.Process(os.getpid()) print(f"Monitoring Python server process (PID: {os.getpid()})") - # Set up a callback to handle new data + # Per-acquisition state variables. + # These declarations satisfy Python's nonlocal scoping requirements for the nested + # functions (data_callback, handle_tdc, bin_pixels, do_final_flush). + # Authoritative reset on every client connect: see the "client_connected" branch below. event_count = 0 - - # Time-resolved binning callback - # State variables t_zero = None cycle_count = 0 flush_count = 0 @@ -267,8 +299,23 @@ def generate_scan_name(): pixels_outside_window = 0 last_tdc_warning_time = None first_pixel_time = None - start_message_sent = False # Track if start message has been sent - acquisition_start_time = None # Track when acquisition actually started + start_message_sent = False + acquisition_start_time = None + # Wall-clock flush gate state. + # last_flush_time: time.monotonic() of the most recent emit, or None + # before the first TDC of the acquisition has armed the clock. Using + # monotonic (not wall) makes the gate NTP/DST-proof; the wire-level + # flush_metadata "timestamp" published by zmq_worker still uses + # time.time() so subscribers see no change in format. + # cycles_since_last_flush: number of *closed* TDC cycles currently + # accumulated in local_accumulator. Exactly what the old gate's + # "flush_every_n_cycles" constant used to carry, except now driven + # by wall-clock boundaries rather than a modulus on cycle_count. + last_flush_time = None + cycles_since_last_flush = 0 + # Alignment-mode running counter for the wire-level pixel total in start/stop + # messages. Reset on every client connect alongside the TDC-mode counters. + pixel_count_total = 0 # Convert edge string to enum value target_edge = 0 if tdc_edge.lower() == "rising" else 1 @@ -296,8 +343,10 @@ def data_callback(result) -> None: current_time = time.time() - # Send start message when first data arrives - if not start_message_sent and message_queue is not None: + # Send start message when first data arrives. + # Guard on scan_name: the main loop generates it on client-connect; if it hasn't + # fired yet (main loop mid-sleep) we skip here and let the main-loop start take over. + if not start_message_sent and message_queue is not None and scan_name is not None: acquisition_start_time = current_time start_msg = TimePixStart( scan_name=scan_name, @@ -423,40 +472,30 @@ def bin_pixels(mask): # Helper function to handle TDC trigger (flush check + update t_zero) def handle_tdc(tdc_ts): - """Process a TDC trigger: check for flush, update t_zero.""" - nonlocal t_zero, cycle_count, flush_count - nonlocal pixels_before_trigger, pixels_outside_window - nonlocal last_tdc_warning_time - nonlocal xyt_array - - # Check if we need to flush - if cycle_count > 0 and cycle_count % flush_every_n_cycles == 0: - with xyt_lock: - xyt_array += local_accumulator - array_copy = xyt_array.copy() - xyt_array.fill(0) - - local_accumulator.fill(0) - flush_count += 1 - - flush_metadata = { - "cycles_in_flush": flush_every_n_cycles, - "total_cycles": cycle_count, - "flush_number": flush_count, - "pixels_discarded_before_trigger": int(pixels_before_trigger), - "pixels_discarded_outside_window": int(pixels_outside_window), - } - - try: - xyt_queue.put_nowait((array_copy, flush_metadata)) - logger.info(f"Flushed: #{flush_count}, cycles={flush_every_n_cycles}") - pixels_before_trigger = 0 - pixels_outside_window = 0 - except queue.Full: - logger.warning("Processing queue full, dropping array") + """Process a TDC trigger: wall-clock-gated flush, then update t_zero. + + A TDC closes the *previous* cycle. We therefore increment the + closed-cycle counter only when ``t_zero is not None``, i.e. only + when there was a previous cycle to close. Flush emission is + delegated to ``emit_flush_if_due()`` which does the gate check, + the accumulator copy, and the state reset all under + ``xyt_lock`` to avoid a TOCTOU race with the main-loop watchdog. + """ + nonlocal t_zero, cycle_count, last_tdc_warning_time + nonlocal last_flush_time, cycles_since_last_flush + + if t_zero is not None: + cycles_since_last_flush += 1 + cycle_count += 1 + + # Arm the wall-clock gate on the first TDC of the acquisition + # (or after reconnect); afterwards let the gate decide. + if last_flush_time is None: + last_flush_time = time.monotonic() + else: + emit_flush_if_due() t_zero = int(tdc_ts) - cycle_count += 1 last_tdc_warning_time = current_time # ===================================================================== @@ -486,8 +525,305 @@ def handle_tdc(tdc_ts): mask = pixel_indices > last_boundary bin_pixels(mask) - # Set the callback - server.set_data_callback(data_callback) + def emit_flush_if_due() -> bool: + """Publish one flush event if the wall-clock gate is open. + + Called from two producers: ``handle_tdc`` on the TCP callback + thread, and the main loop's silence watchdog. Both entry points + funnel through here so the gate check, the accumulator copy, + and the state reset all happen atomically under ``xyt_lock``. + That atomicity is what makes the check-and-reset race-free: a + second caller sees ``cycles_since_last_flush == 0`` and bails + without double-emitting. + + Returns True iff an event was actually put on ``xyt_queue``. + """ + nonlocal last_flush_time, cycles_since_last_flush + nonlocal flush_count, pixels_before_trigger, pixels_outside_window + nonlocal xyt_array + + with xyt_lock: + # Gate: nothing to do if no TDC has armed the clock yet, + # if the accumulator is empty, or if the flush interval + # has not elapsed since the last emit. + if last_flush_time is None: + return False + if cycles_since_last_flush == 0: + return False + if (time.monotonic() - last_flush_time) < flush_interval: + return False + + xyt_array += local_accumulator + array_copy = xyt_array.copy() + xyt_array.fill(0) + local_accumulator.fill(0) + + flush_count += 1 + cycles_in_flush = cycles_since_last_flush + total_cycles_snapshot = cycle_count + flush_number = flush_count + pbt = int(pixels_before_trigger) + pow_ = int(pixels_outside_window) + + last_flush_time = time.monotonic() + cycles_since_last_flush = 0 + pixels_before_trigger = 0 + pixels_outside_window = 0 + + # Build metadata and put on the queue OUTSIDE the lock. Queue + # has its own internal lock and we want the critical section + # above to stay as short as possible. + flush_metadata = { + "scan_name": scan_name, + "cycles_in_flush": cycles_in_flush, + "total_cycles": total_cycles_snapshot, + "flush_number": flush_number, + "pixels_discarded_before_trigger": pbt, + "pixels_discarded_outside_window": pow_, + } + try: + xyt_queue.put_nowait((array_copy, flush_metadata)) + logger.info(f"Flushed: #{flush_number}, cycles={cycles_in_flush}") + return True + except queue.Full: + logger.warning("Processing queue full, dropping array (scan=%s)", scan_name) + return False + + def _wait_for_xyt_drain(timeout: float = 30.0) -> None: + """Block until xyt_queue is empty or timeout expires. + + Must be called before putting a stop message into message_queue so that + all buffered event messages are published by the ZMQ worker before the + stop control message. The ZMQ worker checks message_queue with higher + priority, so without this wait the stop would jump ahead of queued flushes. + """ + deadline = time.time() + timeout + while not xyt_queue.empty() and time.time() < deadline: + time.sleep(0.05) + if not xyt_queue.empty(): + logger.warning( + "xyt_queue not drained after %.1fs (scan=%s); stop message may arrive before remaining flushes", + timeout, + scan_name, + ) + + def do_final_flush() -> None: + """Flush any data remaining in the accumulator before sending a stop message. + + The normal flush is triggered by an incoming TDC pulse closing the previous + cycle. When a stream ends the last N partial cycles never see a closing TDC, + so this helper is called at each stop site to publish that residual data as + one last event message before the stop control message goes out. + """ + nonlocal flush_count, xyt_array + + with xyt_lock: + xyt_array += local_accumulator + has_data = bool(np.any(xyt_array)) + if has_data: + array_copy = xyt_array.copy() + xyt_array.fill(0) + + local_accumulator.fill(0) + + if not has_data: + return + + flush_count += 1 + # Accurate residual cycle count: cycles_since_last_flush is + # decremented to 0 after each emit_flush_if_due, so what's left + # here is exactly the number of closed cycles still sitting in + # local_accumulator. Replaces the pre-fix + # `cycle_count % flush_every_n_cycles` heuristic, which was + # off-by-one (a TDC arms the next cycle without it being + # closed yet). + partial_cycles = cycles_since_last_flush + flush_metadata = { + "scan_name": scan_name, + "cycles_in_flush": partial_cycles, + "total_cycles": cycle_count, + "flush_number": flush_count, + "pixels_discarded_before_trigger": int(pixels_before_trigger), + "pixels_discarded_outside_window": int(pixels_outside_window), + } + try: + xyt_queue.put_nowait((array_copy, flush_metadata)) + logger.info( + "Final flush #%d: %d partial cycles flushed before stop (scan=%s)", + flush_count, + partial_cycles, + scan_name, + ) + except queue.Full: + logger.warning("Processing queue full, dropping final flush (scan=%s)", scan_name) + + # ========================================================================= + # Alignment-mode parallels to data_callback / emit_flush_if_due / do_final_flush. + # + # These use the same xyt_lock, xyt_queue, xyt_array, and local_accumulator as + # the timing-mode functions above. Only the *gate* differs: TDCs are ignored + # entirely and the flush fires on a pure wall-clock interval. Reusing the + # rest of the pipeline means alignment benefits from the same backpressure + # (q_ingest, xyt_queue, ZMQ SNDHWM) as timing mode — see the plan doc for the + # buffering analysis. Only one of the two callback families is wired into + # ``server.set_data_callback`` based on the ``alignment`` flag below. + # ========================================================================= + + def data_callback_alignment(result) -> None: + """Alignment-mode callback: 2D X/Y histogram, no TDC handling. + + Writes pixel hits into ``local_accumulator[:, :, 0]`` (singleton time + bin). TDC packets in the batch are silently ignored. Increments + ``pixel_count_total`` for the running stop-message summary. + """ + nonlocal first_pixel_time, start_message_sent, acquisition_start_time + nonlocal pixel_count_total, last_flush_time + + current_time = time.time() + + # Mirror the start-message logic from data_callback so the first data + # batch can publish the start message if the main-loop reset fired + # mid-sleep. Guarded by start_message_sent + scan_name like the timing path. + if not start_message_sent and message_queue is not None and scan_name is not None: + acquisition_start_time = current_time + start_msg = TimePixStart( + scan_name=scan_name, + tdc_frequency_hz=tdc_frequency, + t_delta_ns=t_delta_ns, + t_cycle_ns=t_cycle / 1e3, + n_bins=n_bins, + detector_size_x=detector_size_x, + detector_size_y=detector_size_y, + flush_interval_s=flush_interval, + cycles_per_flush=max(1, int(flush_interval * tdc_frequency)), + tdc_channel=tdc_ch, + tdc_edge=tdc_edge, + collapse_y=collapse_y, + zmq_port=zmq_port, + tcp_port=port, + mode="alignment", + ) + try: + message_queue.put_nowait(start_msg.model_dump()) + start_message_sent = True + logger.info(f"Queued alignment start message for scan: {scan_name}") + print(f"Queued alignment start message for scan: {scan_name}") + except queue.Full: + logger.warning("Message queue full, dropping start message") + + if result.n_pixels == 0: + return + + if first_pixel_time is None: + first_pixel_time = current_time + # Arm the wall-clock gate on first data so the first emit fires after a + # full window rather than immediately on partial data. + if last_flush_time is None: + last_flush_time = time.monotonic() + + # Vectorized 2D accumulate. local_accumulator is shape (X, Y, 1) in + # alignment mode (forced via n_bins=1, collapse_y=False above), so all + # pixels go into time-bin 0. + np.add.at(local_accumulator, (result.pixel_x, result.pixel_y, 0), 1) + pixel_count_total += int(result.n_pixels) + + def emit_alignment_flush_if_due() -> bool: + """Wall-clock-gated flush for alignment mode (no TDC dependency). + + Mirrors the critical-section structure of ``emit_flush_if_due``: + check the gate, swap the accumulator, push to xyt_queue — all under + ``xyt_lock`` so concurrent callers (callback thread + main loop) + cannot double-emit. Empty arrays are still emitted so the UI gets + regular "0 cps" updates during data droughts. + """ + nonlocal last_flush_time, flush_count, xyt_array + + with xyt_lock: + if last_flush_time is None: + # Not yet armed — first data batch (or main-loop tick after data) + # will set last_flush_time. Arming here keeps the silence-watchdog + # behavior: even with zero pixels we will emit on the next tick. + last_flush_time = time.monotonic() + return False + if (time.monotonic() - last_flush_time) < flush_interval: + return False + + xyt_array += local_accumulator + array_copy = xyt_array.copy() + xyt_array.fill(0) + local_accumulator.fill(0) + + flush_count += 1 + flush_number = flush_count + pixels_in_flush = int(array_copy.sum()) + last_flush_time = time.monotonic() + + flush_metadata = { + "scan_name": scan_name, + "flush_number": flush_number, + # cycles_* kept for downstream compatibility with the timing-mode + # consumers; meaningless in alignment mode. + "cycles_in_flush": 1, + "total_cycles": flush_number, + "pixels_in_flush": pixels_in_flush, + "pixels_discarded_before_trigger": 0, + "pixels_discarded_outside_window": 0, + } + try: + xyt_queue.put_nowait((array_copy, flush_metadata)) + logger.info(f"Alignment flush #{flush_number}: pixels={pixels_in_flush}") + return True + except queue.Full: + logger.warning("Processing queue full, dropping alignment flush") + return False + + def do_final_flush_alignment() -> None: + """Publish residual data in local_accumulator before stop in alignment mode.""" + nonlocal flush_count, xyt_array + + with xyt_lock: + xyt_array += local_accumulator + has_data = bool(np.any(xyt_array)) + if has_data: + array_copy = xyt_array.copy() + xyt_array.fill(0) + + local_accumulator.fill(0) + + if not has_data: + return + + flush_count += 1 + pixels_in_flush = int(array_copy.sum()) + flush_metadata = { + "scan_name": scan_name, + "flush_number": flush_count, + "cycles_in_flush": 1, + "total_cycles": flush_count, + "pixels_in_flush": pixels_in_flush, + "pixels_discarded_before_trigger": 0, + "pixels_discarded_outside_window": 0, + } + try: + xyt_queue.put_nowait((array_copy, flush_metadata)) + logger.info(f"Alignment final flush #{flush_count}: pixels={pixels_in_flush}") + except queue.Full: + logger.warning("Processing queue full, dropping final alignment flush") + + # Wire the appropriate callback + flush helpers into the server. + if alignment: + server.set_data_callback(data_callback_alignment) + _emit_flush = emit_alignment_flush_if_due + _final_flush = do_final_flush_alignment + # Sleep half a window so the wall-clock gate fires near its target rate + # (30 Hz → 16 ms tick). At 1 Hz this becomes 0.5 s, still snappier than + # the 1 s default and harmless. + main_loop_sleep_s = max(0.05, min(1.0, flush_interval / 2.0)) + else: + server.set_data_callback(data_callback) + _emit_flush = emit_flush_if_due + _final_flush = do_final_flush + main_loop_sleep_s = 1.0 try: # Start the server @@ -533,15 +869,30 @@ def handle_tdc(tdc_ts): while server.running: # Check if we should exit due to client disconnect if exit_on_disconnect and server.client_disconnected_event.is_set(): - logger.info("Client disconnected, initiating shutdown...") + logger.info("Client disconnected, initiating shutdown (scan=%s)...", scan_name) # Send stop message before shutdown if message_queue is not None and start_message_sent and not stop_message_sent_on_disconnect: + # Drain any TDC/pixel batches still sitting in + # SocketDataServer.message_queue before snapshotting + # cycle_count. Without this, trailing TDCs that + # arrived over TCP but haven't yet been routed + # through data_callback are silently dropped from + # stop.total_cycles (the dominant component of the + # ~flush_interval-of-cycles "loss" observed in + # short-disconnect experiments). + if not server.wait_for_idle(timeout=5.0): + logger.warning( + "ingest queue did not drain within 5s; " + "stop.total_cycles may understate the true cycle count" + ) + _final_flush() + _wait_for_xyt_drain() acquisition_duration = (time.time() - acquisition_start_time) if acquisition_start_time else 0.0 stop_msg = TimePixStop( scan_name=scan_name, total_flushes=flush_count, - total_cycles=cycle_count, - total_packets=event_count, + total_cycles=0 if alignment else cycle_count, + total_packets=pixel_count_total if alignment else event_count, acquisition_duration_s=acquisition_duration, pixels_discarded_before_trigger=int(pixels_before_trigger), pixels_discarded_outside_window=int(pixels_outside_window), @@ -551,7 +902,6 @@ def handle_tdc(tdc_ts): logger.info(f"Queued stop message (client disconnect) for scan: {scan_name}") print(f"Queued stop message (client disconnect) for scan: {scan_name}") stop_message_sent_on_disconnect = True - time.sleep(0.5) # Give worker time to send except queue.Full: logger.warning("Message queue full, dropping stop message") break @@ -560,26 +910,66 @@ def handle_tdc(tdc_ts): if server.client_connected and not was_client_connected: heartbeat.set_state(ServerState.STREAMING) was_client_connected = True - # Reset stop message flag when new client connects + # ── Authoritative per-acquisition reset ────────────────────── + # scan_name is generated here and nowhere else (single source of truth). + # All counters declared above are also reset here for each new client. stop_message_sent_on_disconnect = False - # Reset acquisition state for new client connection (new acquisition) start_message_sent = False acquisition_start_time = None scan_name = generate_scan_name() - # Reset counters for new acquisition cycle_count = 0 flush_count = 0 event_count = 0 + pixel_count_total = 0 pixels_before_trigger = 0 pixels_outside_window = 0 t_zero = None first_pixel_time = None + # Wall-clock gate: clear so the first TDC of the new + # acquisition re-arms the clock (instead of carrying + # over a stale t_last from the previous client). + last_flush_time = None + cycles_since_last_flush = 0 # Clear arrays for fresh start with xyt_lock: xyt_array.fill(0) local_accumulator.fill(0) logger.info(f"New client connected - resetting acquisition state. New scan: {scan_name}") print(f"New client connected - resetting acquisition state. New scan: {scan_name}") + + # Send start message immediately on client connect so that ZMQ + # subscribers receive it before any data — even when count rates + # are zero and data_callback may never fire before STOP. + # data_callback also tries to send the start message (guarded by + # start_message_sent) as an immediate fallback for the first + # data batch if the main loop is mid-sleep when data arrives. + if message_queue is not None: + acquisition_start_time = time.time() + start_msg = TimePixStart( + scan_name=scan_name, + tdc_frequency_hz=tdc_frequency, + t_delta_ns=t_delta_ns, + t_cycle_ns=t_cycle / 1e3, + n_bins=n_bins, + detector_size_x=detector_size_x, + detector_size_y=detector_size_y, + flush_interval_s=flush_interval, + cycles_per_flush=max(1, int(flush_interval * tdc_frequency)), + tdc_channel=tdc_ch, + tdc_edge=tdc_edge, + collapse_y=collapse_y, + zmq_port=zmq_port, + tcp_port=port, + mode="alignment" if alignment else "timing", + ) + try: + message_queue.put_nowait(start_msg.model_dump()) + start_message_sent = True + logger.info("Queued start message on connect for scan: %s", scan_name) + logger.info("Acquisition session started: %s", start_msg.model_dump()) + print(f"Queued start message on connect for scan: {scan_name}") + except queue.Full: + logger.warning("Message queue full, dropping start message on connect") elif not server.client_connected and was_client_connected: # Client just disconnected - send stop message heartbeat.set_state(ServerState.READY) @@ -587,12 +977,20 @@ def handle_tdc(tdc_ts): # Send stop message when client disconnects (even without --exit-on-disconnect) if message_queue is not None and start_message_sent and not stop_message_sent_on_disconnect: + # See exit_on_disconnect branch above for the rationale. + if not server.wait_for_idle(timeout=5.0): + logger.warning( + "ingest queue did not drain within 5s; " + "stop.total_cycles may understate the true cycle count" + ) + _final_flush() + _wait_for_xyt_drain() acquisition_duration = (time.time() - acquisition_start_time) if acquisition_start_time else 0.0 stop_msg = TimePixStop( scan_name=scan_name, total_flushes=flush_count, - total_cycles=cycle_count, - total_packets=event_count, + total_cycles=0 if alignment else cycle_count, + total_packets=pixel_count_total if alignment else event_count, acquisition_duration_s=acquisition_duration, pixels_discarded_before_trigger=int(pixels_before_trigger), pixels_discarded_outside_window=int(pixels_outside_window), @@ -602,13 +1000,28 @@ def handle_tdc(tdc_ts): logger.info(f"Queued stop message (client disconnected) for scan: {scan_name}") print(f"Queued stop message (client disconnected) for scan: {scan_name}") stop_message_sent_on_disconnect = True - time.sleep(0.5) # Give worker time to send except queue.Full: logger.warning("Message queue full, dropping stop message") - - time.sleep(1) + # ``do_final_flush`` does not clear the wall-clock gate. If we leave + # ``last_flush_time`` / ``cycles_since_last_flush`` armed, the main-loop + # watchdog ``_emit_flush()`` in this same iteration can enqueue another + # flush *after* the stop. The ZMQ worker prioritizes control messages, so + # subscribers see stop then a stale event (previous scan_name), which + # breaks strict per-run tests and can violate cycle conservation. + last_flush_time = None + cycles_since_last_flush = 0 + + time.sleep(main_loop_sleep_s) current_time = time.time() + # Silence watchdog: if the upstream goes quiet (no TDCs in timing + # mode, or no pixels in alignment mode), the callback cannot push + # the gate. Tick it from the main loop too so the last buffered + # window is released even when the wire is idle. Safe to call + # concurrently with the callback thread: both emit helpers are + # internally locked via xyt_lock. + _emit_flush() + # Show/ update overall stats if current_time - last_stats_time >= stats_update_time: queue_size = server.get_queue_size() @@ -727,14 +1140,25 @@ def handle_tdc(tdc_ts): print("\nShutting down server...") finally: - # Send stop message (only if we haven't already sent it on client disconnect) - if message_queue is not None and not stop_message_sent_on_disconnect: + _shutdown_start = time.time() + logger.info("Shutdown sequence initiated") + # Send stop message only if a start was sent and stop hasn't been sent yet. + # Guarding on start_message_sent prevents an orphan stop from being published + # with the server's initial UUID when the client connection was never detected + # (e.g. connect+disconnect happened within one main-loop sleep cycle). + if message_queue is not None and start_message_sent and not stop_message_sent_on_disconnect: + if not server.wait_for_idle(timeout=5.0): + logger.warning( + "ingest queue did not drain within 5s; stop.total_cycles may understate the true cycle count" + ) + _final_flush() + _wait_for_xyt_drain() acquisition_duration = (time.time() - acquisition_start_time) if acquisition_start_time else 0.0 stop_msg = TimePixStop( scan_name=scan_name, total_flushes=flush_count, - total_cycles=cycle_count, - total_packets=event_count, + total_cycles=0 if alignment else cycle_count, + total_packets=pixel_count_total if alignment else event_count, acquisition_duration_s=acquisition_duration, pixels_discarded_before_trigger=int(pixels_before_trigger), pixels_discarded_outside_window=int(pixels_outside_window), @@ -742,9 +1166,7 @@ def handle_tdc(tdc_ts): try: message_queue.put_nowait(stop_msg.model_dump()) logger.info(f"Queued stop message for scan: {scan_name}") - print(f"Queued stop message for scan: {scan_name}") # Also print to console - # Give worker time to send the stop message - time.sleep(0.5) # Increased wait time + print(f"Queued stop message for scan: {scan_name}") except queue.Full: logger.warning("Message queue full, dropping stop message") print("WARNING: Message queue full, dropping stop message") @@ -772,6 +1194,7 @@ def handle_tdc(tdc_ts): if input_thread.is_alive(): logger.warning("Input listener did not finish in time") + logger.info("Shutdown sequence complete (elapsed %.1fs)", time.time() - _shutdown_start) print("Server stopped successfully") diff --git a/src/splash_timepix/heartbeat.py b/src/splash_timepix/heartbeat.py index cc039b1..9a96579 100644 --- a/src/splash_timepix/heartbeat.py +++ b/src/splash_timepix/heartbeat.py @@ -223,6 +223,12 @@ def wait_for_ready(port: int = 5658, timeout: float = 30.0) -> bool: except zmq.Again: # Timeout on recv, check overall timeout + elapsed = time.time() - start_time + logger.debug( + "wait_for_ready: no heartbeat received after %.1fs (timeout=%.1fs); retrying", + elapsed, + timeout, + ) continue logger.warning(f"Timeout waiting for server ready after {timeout}s") diff --git a/src/splash_timepix/parser.py b/src/splash_timepix/parser.py index 58526d6..55a648e 100644 --- a/src/splash_timepix/parser.py +++ b/src/splash_timepix/parser.py @@ -12,12 +12,15 @@ result = parser.parse_batch(data_bytes) # BatchParseResult with NumPy arrays """ +import logging from dataclasses import dataclass from enum import IntEnum from typing import Union import numpy as np +logger = logging.getLogger(__name__) + # Constants for time conversions TIMESTAMP_CLOCK_MHZ = 3840 TIMESTAMP_PS_PER_TICK = 260.41666 # 1 / 3840 MHz in picoseconds @@ -218,13 +221,14 @@ def parse(self, data: bytes) -> Union[PixelPacket, TDCPacket, ControlPacket, Non elif packet_type == PacketType.CONTROL: return self._parse_control(packet_type, timestamp, packet_specific) else: + logger.debug("Unknown packet type %d in 12-byte packet, returning None", packet_type) return None def _parse_pixel(self, packet_type: int, timestamp: int, specific_data: int) -> PixelPacket: reserved = specific_data & 0x3F - # Wire format: reserved(0-5) | y(6-15) | x(16-25) | tot(26-35) - y = (specific_data >> 6) & 0x3FF - x = (specific_data >> 16) & 0x3FF + # Wire format: reserved(0-5) | x(6-15) | y(16-25) | tot(26-35) + x = (specific_data >> 6) & 0x3FF + y = (specific_data >> 16) & 0x3FF tot = (specific_data >> 26) & 0x3FF return PixelPacket( packet_type=packet_type, @@ -291,6 +295,14 @@ def parse_batch(self, data: bytes) -> BatchParseResult: return self._empty_result() # Trim to exact multiple of packet size + remainder = len(data) % self.packet_size + if remainder != 0: + logger.warning( + "parse_batch: received %d bytes, not a multiple of %d; trimming %d trailing byte(s)", + len(data), + self.packet_size, + remainder, + ) data = data[: n_packets * self.packet_size] # Convert to numpy array and reshape to (n_packets, 12) @@ -354,8 +366,8 @@ def parse_batch(self, data: bytes) -> BatchParseResult: pixel_specific = specific[pixel_mask] pixel_timestamps = timestamps[pixel_mask] - pixel_y = ((pixel_specific >> 6) & 0x3FF).astype(np.int32) - pixel_x = ((pixel_specific >> 16) & 0x3FF).astype(np.int32) + pixel_x = ((pixel_specific >> 6) & 0x3FF).astype(np.int32) + pixel_y = ((pixel_specific >> 16) & 0x3FF).astype(np.int32) pixel_tot = ((pixel_specific >> 26) & 0x3FF).astype(np.int32) # ---- EXTRACT TDC FIELDS ---- @@ -373,11 +385,19 @@ def parse_batch(self, data: bytes) -> BatchParseResult: control_subtype = ((control_specific >> 32) & 0xF).astype(np.uint8) + n_unknown = int(np.sum(unknown_mask)) + if n_unknown > 0: + logger.warning( + "parse_batch: %d unknown/invalid packet(s) in batch of %d", + n_unknown, + n_packets, + ) + return BatchParseResult( n_pixels=len(pixel_indices), n_tdc=len(tdc_indices), n_control=len(control_indices), - n_unknown=int(np.sum(unknown_mask)), + n_unknown=n_unknown, pixel_x=pixel_x, pixel_y=pixel_y, pixel_timestamp=pixel_timestamps, diff --git a/src/splash_timepix/schemas.py b/src/splash_timepix/schemas.py index d2a06c4..4e14b94 100644 --- a/src/splash_timepix/schemas.py +++ b/src/splash_timepix/schemas.py @@ -56,6 +56,11 @@ class TimePixStart(BaseModel): collapse_y: bool = Field(..., description="Whether Y dimension is collapsed") zmq_port: int = Field(..., description="ZMQ publishing port") tcp_port: int = Field(..., description="TCP socket port for live-cli connection") + # Acquisition mode. "timing" is the default time-resolved pipeline (TDC-driven binning); + # "alignment" is the wall-clock-gated 2D X/Y histogram path used by the Alignment tab. + # Subscribers route on this field; old subscribers ignoring it still get correct + # shape/dtype info from `shape` and `dtype` and behave as before. + mode: Literal["timing", "alignment"] = Field(default="timing", description="Acquisition mode") class TimePixStop(BaseModel): diff --git a/src/splash_timepix/serval_client/lib.py b/src/splash_timepix/serval_client/lib.py index fee5734..1f86b29 100644 --- a/src/splash_timepix/serval_client/lib.py +++ b/src/splash_timepix/serval_client/lib.py @@ -32,9 +32,17 @@ def _request(self, method: str, endpoint: str, *, expected_status: int = 200, ** try: response = self.session.request(method, url, timeout=self.timeout, **kwargs) except requests.RequestException as e: + logging.error("HTTP %s %s failed (connection error): %s", method.upper(), url, e) raise ServalError(f"Connection error: {e}") if response.status_code != expected_status: + logging.error( + "HTTP %s %s failed (status %d): %s", + method.upper(), + url, + response.status_code, + response.text, + ) raise ServalError(f"Unexpected status {response.status_code} for {url}: {response.text}") return response @@ -55,6 +63,7 @@ def load_configuration(self, fmt: str, file_path: Path) -> None: """ if not file_path.exists(): raise ServalError(f"Configuration file not found: {file_path}") + logging.debug("Opening detector config file: %s (format: %s)", file_path, fmt) params = {"format": fmt, "file": str(file_path)} resp = self._request("get", "/config/load", params=params) diff --git a/src/splash_timepix/simulator.py b/src/splash_timepix/simulator.py index e844d80..126478c 100644 --- a/src/splash_timepix/simulator.py +++ b/src/splash_timepix/simulator.py @@ -57,7 +57,7 @@ def build_pixel_packet(timestamp: int, tot: int, x: int, y: int, reserved: int = assert 0 <= reserved <= 63, f"Reserved out of range: {reserved}" packet_type = PacketType.PIXEL - pixel_data = reserved | (y << 6) | (x << 16) | (tot << 26) + pixel_data = reserved | (x << 6) | (y << 16) | (tot << 26) full_value = pixel_data | (timestamp << 36) | (packet_type << 92) return full_value.to_bytes(12, byteorder="big") @@ -125,6 +125,14 @@ class SimulatorConfig: detector_size_y: int = 256 # Detector height include_control_packets: bool = False # Whether to generate control packets control_packet_interval: float = 1.0 # Seconds between control sequences + # Wire-level batching: when > 0, the CLI's TCP send loop accumulates + # packets in-memory for this many seconds, then sendalls them in one + # bundle. Default 0 preserves the original one-packet-per-sendall + # behaviour (and therefore does not change any production workload). + # This knob exists to reproduce the Serval+luna-iterator symptom in + # integration tests without a real detector: it injects multi-second + # sorted boluses on the wire the way the upstream sorter does. + tcp_batch_interval_s: float = 0.0 class PacketSimulator: diff --git a/src/splash_timepix/simulator_cli.py b/src/splash_timepix/simulator_cli.py index 25e6c53..1d2a4c4 100644 --- a/src/splash_timepix/simulator_cli.py +++ b/src/splash_timepix/simulator_cli.py @@ -6,6 +6,7 @@ """ import datetime +import logging import socket import threading import time @@ -16,6 +17,8 @@ from splash_timepix.parser import PacketParser from splash_timepix.simulator import PacketSimulator, PacketType, SimulatorConfig +logger = logging.getLogger(__name__) + app = typer.Typer() @@ -38,6 +41,12 @@ def __init__(self, host: str = "localhost", port: int = 9090): self.pixel_count_rate = 2 self.tdc_frequency = 0.2 self.counting = True + # See SimulatorConfig.tcp_batch_interval_s. 0 = per-packet sendall + # (original behaviour, what real hardware drivers do); >0 buffers + # bytes for this many seconds and emits them as a single sendall + # bundle, used by tests to reproduce the Serval+luna-iterator + # upstream-batching regime on the wire. + self.tcp_batch_interval_s: float = 0.0 def set_counts_per_second(self, cps: float): """Average number of pixel events per second (cps, counts/second)""" @@ -64,6 +73,20 @@ def set_counting(self, counting: bool): else: print("Not counting sent packets (allows for higher count rates [cps])") + def set_tcp_batch_interval(self, interval_s: float) -> None: + """Set wire-level bolus batching interval. + + 0 disables (original per-packet sendall). Non-zero accumulates + packets for this many seconds, then sends as a single sendall. + Used by tests to reproduce the Serval+luna-iterator batching + regime; not intended for production use. + """ + if interval_s < 0: + raise ValueError(f"tcp_batch_interval_s must be >= 0, got {interval_s}") + self.tcp_batch_interval_s = float(interval_s) + if interval_s > 0: + print(f"TCP batching enabled: flush every {interval_s}s") + def connect(self) -> bool: """ Connect to the server. @@ -74,10 +97,10 @@ def connect(self) -> bool: try: self.socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM) self.socket.connect((self.host, self.port)) - print(f"Connected to server at {self.host}:{self.port}") + logger.info("Connected to server at %s:%d", self.host, self.port) return True except Exception as e: - print(f"Failed to connect to server: {e}") + logger.error("Failed to connect to server at %s:%d: %s", self.host, self.port, e) return False def disconnect(self) -> None: @@ -85,7 +108,7 @@ def disconnect(self) -> None: if self.socket: self.socket.close() self.socket = None - print("Disconnected from server") + logger.info("Disconnected from server") def start_auto_sending(self, duration: float) -> None: """ @@ -98,24 +121,40 @@ def start_auto_sending(self, duration: float) -> None: print("Auto-sending is already running") return + # Connect (or reconnect after a previous DAQ) when the DAQ starts so + # the server only sees a new client — and publishes the ZMQ start + # message — at this point, not when the CLI was launched. + if self.socket is None: + print("Connecting to server for new acquisition...") + if not self.connect(): + return + self.running = True self.send_thread = threading.Thread(target=self._auto_send_worker, args=(duration,), daemon=True) self.send_thread.start() - print(f"Started auto-sending messages for {duration} seconds") + logger.info( + "Auto-sending started: cps=%g, tdc_frequency=%g Hz, duration=%gs", + self.pixel_count_rate, + self.tdc_frequency, + duration, + ) dt = datetime.datetime.fromtimestamp(time.time()) formatted = dt.strftime("%Y-%m-%d %H:%M:%S.") + f"{dt.microsecond // 1000:03d}" print(f"Current time: {formatted}") def stop_auto_sending(self) -> None: """Stop automatic message sending.""" - if not self.running: - print("Auto-sending is not running") - return - - self.running = False - if self.send_thread and self.send_thread.is_alive(): - self.send_thread.join(timeout=5) - print("Stopped auto-sending messages") + if self.running: + self.running = False + if self.send_thread and self.send_thread.is_alive(): + self.send_thread.join(timeout=5) + print("Stopped auto-sending messages") + # Disconnect regardless of whether the stream was manually stopped or + # ended naturally (timer expired). This is what signals end-of-acquisition + # to the server so it publishes a ZMQ stop message. Safe to call even + # when already disconnected. A reconnect happens automatically the next + # time start_auto_sending() is called. + self.disconnect() def run_blocking(self, duration: float) -> None: """ @@ -135,10 +174,17 @@ def run_blocking(self, duration: float) -> None: self.stop_auto_sending() def _auto_send_worker(self, duration: float) -> None: - """Worker thread for sending packets using PacketSimulator.""" - # initialize simulator + """Worker thread for sending packets using PacketSimulator. + + Normally each packet is pushed to the socket as soon as it is + produced. When ``self.tcp_batch_interval_s > 0`` packet bytes + are instead accumulated in an in-memory buffer and emitted in + one ``sendall`` per ``tcp_batch_interval_s`` seconds. This + mode exists to reproduce the production Serval+luna-iterator + symptom on the wire for integration tests; it is single- + threaded so no lock is required. + """ simulator = PacketSimulator(SimulatorConfig()) - # write count rate and TDC frequency to simulator configuration simulator.config.pixel_count_rate = self.pixel_count_rate simulator.config.tdc_frequency = self.tdc_frequency if self.counting: @@ -147,34 +193,71 @@ def _auto_send_worker(self, duration: float) -> None: sent_count_tdc = 0 sent_count_ctrl = 0 + batch_interval = float(self.tcp_batch_interval_s) + batching = batch_interval > 0.0 + tx_buffer: list[bytes] = [] + last_flush_monotonic = time.monotonic() + + def _flush_tx_buffer() -> None: + """Send accumulated packet bytes as one sendall, clear buffer.""" + nonlocal last_flush_monotonic + if not tx_buffer: + return + payload = b"".join(tx_buffer) + tx_buffer.clear() + last_flush_monotonic = time.monotonic() + self.socket.sendall(payload) + packet_generator = simulator.generate_stream(duration_seconds=duration) - for packet in packet_generator: - if not self.running: - break - try: - self.socket.sendall(packet) - if self.counting: - parsed = parser.parse(packet) - if parsed and parsed.packet_type == PacketType.PIXEL: - sent_count_pixel += 1 - elif parsed and parsed.packet_type == PacketType.TDC: - sent_count_tdc += 1 - elif parsed and parsed.packet_type == PacketType.CONTROL: - sent_count_ctrl += 1 - except Exception as e: - print(f"Failed to send simulated packet: {e}") - break + try: + for packet in packet_generator: + if not self.running: + break + try: + if batching: + tx_buffer.append(packet) + if time.monotonic() - last_flush_monotonic >= batch_interval: + _flush_tx_buffer() + else: + self.socket.sendall(packet) + if self.counting: + parsed = parser.parse(packet) + if parsed and parsed.packet_type == PacketType.PIXEL: + sent_count_pixel += 1 + elif parsed and parsed.packet_type == PacketType.TDC: + sent_count_tdc += 1 + elif parsed and parsed.packet_type == PacketType.CONTROL: + sent_count_ctrl += 1 + except Exception as e: + logger.error("Failed to send simulated packet: %s", e) + break + finally: + # Drain residual bytes so the server sees the tail of the + # acquisition. If this raises we still want to fall through + # to the running=False / disconnect path below. + if batching: + try: + _flush_tx_buffer() + except Exception as e: + print(f"Failed to flush final TCP batch: {e}") self.running = False - print("Auto-sending finished.") + logger.info("Auto-sending finished.") if self.counting: - print("Sent events during last session:") - print(f" {sent_count_pixel} pixel events") - print(f" {sent_count_tdc} TDC events") - print(f" {sent_count_ctrl} control events") + logger.info( + "Sent events during last session: %d pixel, %d TDC, %d control", + sent_count_pixel, + sent_count_tdc, + sent_count_ctrl, + ) dt = datetime.datetime.fromtimestamp(time.time()) formatted = dt.strftime("%Y-%m-%d %H:%M:%S.") + f"{dt.microsecond // 1000:03d}" print(f"Current time: {formatted}") + # Disconnect so the server sees the TCP close and publishes the ZMQ stop + # message. When the user typed "stop" this is handled by stop_auto_sending(); + # when the timer expires naturally, we must do it here. disconnect() is + # idempotent (checks self.socket), so a double-call is harmless. + self.disconnect() @app.command() @@ -190,6 +273,13 @@ def main( help="Start streaming immediately without interactive mode", ), no_count: bool = typer.Option(False, "--no-count", help="Disable packet counting for higher performance"), + tcp_batch_interval: float = typer.Option( + 0.0, + "--tcp-batch-interval", + help="If >0, buffer packets for this many seconds and emit them in one sendall. " + "Used by tests to reproduce the Serval+luna-iterator batching regime on the wire. " + "0 (default) preserves real-hardware-like per-packet sending.", + ), ): """ Simulated TimePix3 data source for testing. @@ -213,9 +303,11 @@ def main( source.set_counts_per_second(cps) source.set_tdc_frequency(tdc_frequency) source.set_counting(not no_count) + source.set_tcp_batch_interval(tcp_batch_interval) - if not source.connect(): - return + # Do NOT connect here. Connection is deferred to start_auto_sending() so + # that the server only sees a new TCP client — and therefore only publishes + # the ZMQ start message — when a DAQ actually begins, not at CLI startup. try: if auto_start: @@ -229,6 +321,7 @@ def main( print(" 'cps ' - Set counts per second") print(" 'tdc ' - Set TDC frequency (Hz)") print(" 'count ' - Count sent packets (default: y)") + print(" 'batch ' - Wire-level burst batching; 0 disables (default: 0)") print(" 'start ' - Start auto-sending") print(" 'stop' - Stop auto-sending") print(" 'quit' - Exit") @@ -264,6 +357,22 @@ def main( counting = True if command[1] == "y" else False source.set_counting(counting) + elif command[0] == "batch": + if len(command) < 2: + print("Usage: batch (0 disables; non-zero enables wire-level burst batching)") + continue + try: + interval = float(command[1]) + except ValueError: + print(f"Invalid batch interval: {command[1]!r} (expected a number of seconds)") + continue + try: + source.set_tcp_batch_interval(interval) + if interval == 0.0: + print("TCP batching disabled (per-packet sendall)") + except ValueError as e: + print(f"Invalid batch interval: {e}") + elif command[0] == "start": if len(command) < 2: print("Usage: start ") diff --git a/src/splash_timepix/socket_server.py b/src/splash_timepix/socket_server.py index 7b158bf..aa679b6 100644 --- a/src/splash_timepix/socket_server.py +++ b/src/splash_timepix/socket_server.py @@ -186,6 +186,30 @@ def wait_for_client_disconnect(self, timeout: Optional[float] = None) -> bool: """Block until a client disconnects.""" return self.client_disconnected_event.wait(timeout=timeout) + def wait_for_idle(self, timeout: float = 5.0) -> bool: + """Block until every batch currently in ``message_queue`` has been processed. + + The data processor calls ``message_queue.task_done()`` after each + batch returns from ``data_callback``, so when ``unfinished_tasks`` + reaches zero we know the callback has been invoked for every batch + the socket reader put in. + + Intended to be called *after* ``_handle_client`` has exited (i.e. + the client has disconnected and no further batches will be put); + otherwise this can wait indefinitely for new producer activity. + + Returns: + True if the queue drained within ``timeout``, False on timeout. + """ + deadline = time.monotonic() + timeout + with self.message_queue.all_tasks_done: + while self.message_queue.unfinished_tasks > 0: + remaining = deadline - time.monotonic() + if remaining <= 0: + return False + self.message_queue.all_tasks_done.wait(timeout=remaining) + return True + def _socket_listener(self) -> None: """Thread that listens for connections and reads data.""" try: diff --git a/src/splash_timepix/ui/alignment_tab.py b/src/splash_timepix/ui/alignment_tab.py new file mode 100644 index 0000000..44127e4 --- /dev/null +++ b/src/splash_timepix/ui/alignment_tab.py @@ -0,0 +1,1069 @@ +"""Alignment tab — live 2D X/Y heatmap with grayscale LUT for beam alignment. + +Drives the same pipeline as the Operator preview button (streaming-server + +live-cli + ``acq.py --preview``), but with the streaming server in +``--alignment`` mode: TDCs are ignored and a wall-clock-gated 2D histogram is +emitted at 1–30 Hz. Layout matches the Operator tab: a fixed-width **left +sidebar** holds alignment-only controls and a **Statistics** group; the **main +area** is the square X/Y view with Z histogram to its right (LUT only; **Z +Auto/Manual** lives in the sidebar under frame rate). The top bar has Start, **Simulator** +(local synthetic stream), Stop, a 60 s cps strip chart, and the large cps readout. + +Auto-stop semantics: when the user switches to the Operator tab while +alignment is running, ``MainWindow`` calls ``stop_requested`` on this tab so +the two modes never compete for the streaming server. See +``MainWindow._on_tab_changed`` in ``main.py``. +""" + +from __future__ import annotations + +import logging +import time +from collections import deque +from contextlib import contextmanager +from pathlib import Path +from typing import Optional, Union + +import numpy as np +import pyqtgraph as pg +from PySide6.QtCore import Qt, QTimer, Signal, Slot +from PySide6.QtGui import QFont, QFontMetrics, QPen +from PySide6.QtWidgets import ( + QComboBox, + QFrame, + QGroupBox, + QHBoxLayout, + QLabel, + QPushButton, + QSizePolicy, + QSpinBox, + QVBoxLayout, + QWidget, +) + +from . import preferences, theme +from .workers import FlushData + +logger = logging.getLogger(__name__) + + +# Apply dark theme defaults to pyqtgraph at import time so the alignment image +# matches the rest of the UI. setConfigOptions is process-global, but operator +# / engineering tabs don't use pyqtgraph today, so this is benign. +pg.setConfigOptions(background=theme.BG_DARK, foreground=theme.TEXT_PRIMARY, antialias=False) + + +# Detector geometry. Mirrored from SimulatorConfig.detector_size_x/y; quad-chip +# detectors would change these to 512. Keep aligned with app.py. +_DETECTOR_X = 256 +_DETECTOR_Y = 256 + +# Rolling CPS strip chart in the alignment top bar (wall-clock window). +_CPS_HISTORY_WINDOW_S = 60.0 + + +def _grayscale_lut() -> np.ndarray: + """Return a 256x4 (RGBA) grayscale lookup table for ImageItem.setLookupTable.""" + ramp = np.arange(256, dtype=np.uint8) + lut = np.zeros((256, 4), dtype=np.uint8) + lut[:, 0] = ramp + lut[:, 1] = ramp + lut[:, 2] = ramp + lut[:, 3] = 255 + return lut + + +def _alignment_target_pen() -> QPen: + """Red dotted pen for the beam-alignment target overlay.""" + return pg.mkPen(color=(255, 72, 72), width=1, style=Qt.PenStyle.DotLine) + + +# Alignment overlay: outer box in detector pixel coords; inner square shares the +# same center and keeps a fixed side-length ratio vs. the outer (same proportion +# as the original 28 px inner on a 156 px outer). +_ALIGNMENT_TARGET_OUTER = (50.0, 50.0, 206.0, 206.0) # x0, y0, x1, y1 +_ALIGNMENT_INNER_SIDE_FRAC = 28.0 / 156.0 # inner_edge / outer_edge + + +def _build_alignment_target_overlay(plot: pg.PlotItem) -> list[pg.PlotDataItem]: + """Two red dotted squares: outer from constants; inner centered with proportional size.""" + pen = _alignment_target_pen() + items: list[pg.PlotDataItem] = [] + + def add_axis_aligned_square(x0: float, y0: float, x1: float, y1: float) -> None: + # Walk the perimeter once (closed). + xs = [x0, x1, x1, x0, x0] + ys = [y0, y0, y1, y1, y0] + it = pg.PlotDataItem(xs, ys, pen=pen, connect="all") + it.setZValue(10) + plot.addItem(it) + items.append(it) + + ox0, oy0, ox1, oy1 = _ALIGNMENT_TARGET_OUTER + add_axis_aligned_square(ox0, oy0, ox1, oy1) + + cx = (ox0 + ox1) * 0.5 + cy = (oy0 + oy1) * 0.5 + outer_side = float(max(ox1 - ox0, oy1 - oy0)) + inner_side = outer_side * _ALIGNMENT_INNER_SIDE_FRAC + half = 0.5 * inner_side + add_axis_aligned_square(cx - half, cy - half, cx + half, cy + half) + + return items + + +def _format_cps_body(rate: float) -> str: + """Scientific notation body only: ``d.ddE±xx`` — always exactly two digits after ``'.'``.""" + if not np.isfinite(rate) or rate <= 0: + return "0.00E+00" + t = f"{rate:.2E}" + if "e" in t and "E" not in t: + t = t.replace("e", "E") + mant, e_part = t.split("E", 1) + exp_i = int(e_part) + return f"{mant}E{exp_i:+03d}" + + +def _format_cps(rate: float) -> str: + """Fixed-width style cps readout: ``d.ddE±xx cps`` (mantissa always two decimals).""" + return f"{_format_cps_body(rate)} cps" + + +# Same QSS as OperatorTab ROI On/Off toggles (readout panel right column). +_ALIGNMENT_ONOFF_STYLE = f""" + QPushButton {{ + background-color: {theme.GREY_DARK}; + color: {theme.TEXT_MUTED}; + border: 1px solid {theme.BORDER_SUBTLE}; + border-radius: 3px; + padding: 1px 4px; + font-size: 10px; + }} + QPushButton:checked {{ + background-color: {theme.BLUE_PRIMARY}; + color: {theme.TEXT_PRIMARY}; + border-color: {theme.BLUE_LIGHT_2}; + }} + QPushButton:disabled {{ + background-color: {theme.BG_BUTTON_GROUP}; + color: {theme.TEXT_MUTED}; + border-color: {theme.BORDER_SUBTLE}; + }} +""" + + +class _AlignmentOnOffToggle(QWidget): + """Single checkable On/Off control (Operator ROI style); API matches QCheckBox for callers.""" + + toggled = Signal(bool) + + def __init__(self, *, initial: bool = False, parent: Optional[QWidget] = None) -> None: + super().__init__(parent) + self._btn = QPushButton("On" if initial else "Off") + self._btn.setCheckable(True) + self._btn.setChecked(initial) + self._btn.setFixedWidth(36) + self._btn.setStyleSheet(_ALIGNMENT_ONOFF_STYLE) + self._btn.toggled.connect(self._on_btn_toggled) + lay = QHBoxLayout(self) + lay.setContentsMargins(0, 0, 0, 0) + lay.addWidget(self._btn) + + def _on_btn_toggled(self, checked: bool) -> None: + self._btn.setText("On" if checked else "Off") + self.toggled.emit(checked) + + def isChecked(self) -> bool: + return self._btn.isChecked() + + def setChecked(self, value: bool) -> None: + self._btn.blockSignals(True) + try: + self._btn.setChecked(value) + self._btn.setText("On" if value else "Off") + finally: + self._btn.blockSignals(False) + + def set_option_tooltip(self, tip: str) -> None: + QWidget.setToolTip(self, tip) + self._btn.setToolTip(tip) + + +class AlignmentTab(QWidget): + """Live 2D X/Y heatmap for beam alignment.""" + + # Same signal shape as OperatorTab so MainWindow's start/stop dispatch is + # identical: (mode_str, params dict). mode is always "alignment" here. + start_requested = Signal(str, dict) + stop_requested = Signal() + + def __init__(self, parent=None): + super().__init__(parent) + + self._acquiring = False + # Latest 2D (X, Y) array displayed (after squeeze of the (X, Y, 1) flush). + self._last_2d: Optional[np.ndarray] = None + # Cumulative sum for "Show integrated" mode. Reset when the toggle goes + # off→on or on→off, and on Start. + self._integrated_sum: Optional[np.ndarray] = None + # Diagnostic counter so we don't spam the system log at 30 Hz; first + # few flushes are always logged, then periodic. + self._flush_count = 0 + # Local simulator (◈) timer: synthetic flushes only; bypasses Serval / live-cli / + # app.py / ZMQ. Feeds ``on_flush_received`` so the render path can be tested + # without the detector. + self._fake_timer: Optional[QTimer] = None + self._fake_rng = np.random.default_rng() + # Mirrored from OperatorTab / MainWindow process signals so Start stays + # disabled until Serval prints the chip-temperature line (HW ready). + self._serval_process_running = False + self._serval_hw_ready = False + # (time.monotonic(), cps) for the top-bar rolling chart. + self._cps_history: deque[tuple[float, float]] = deque() + # HistogramLUTItem can emit levels changes during wiring; ignore those + # until prefs are loaded so Z range stays default Auto. + self._hist_lut_user_callbacks_enabled = False + + self._setup_ui() + try: + self.load_alignment_preferences() + except Exception: + logger.exception("Failed to load alignment preferences; using widget defaults") + finally: + self._hist_lut_user_callbacks_enabled = True + self._refresh_alignment_start_state() + + # ------------------------------------------------------------------ + # UI construction + # ------------------------------------------------------------------ + + def _setup_ui(self) -> None: + layout = QVBoxLayout(self) + layout.setSpacing(8) + + layout.addWidget(self._build_top_bar()) + + content = QHBoxLayout() + content.setSpacing(10) + left_panel = self._build_left_sidebar() + content.addWidget(left_panel) + + right_col = QVBoxLayout() + right_col.setSpacing(8) + right_col.setContentsMargins(0, 0, 0, 0) + right_col.addWidget(self._build_image_area(), stretch=1) + + right_wrap = QWidget() + right_wrap.setLayout(right_col) + content.addWidget(right_wrap, stretch=1) + layout.addLayout(content, stretch=1) + + def _build_top_bar(self) -> QWidget: + """Row 1: Start/Stop (left), rolling 60 s cps chart (center), giant cps readout (right).""" + top_bar = QFrame() + top_bar.setStyleSheet(f"QFrame {{ background-color: {theme.BG_WIDGET}; border-radius: 6px; }}") + top_layout = QHBoxLayout(top_bar) + top_layout.setContentsMargins(8, 8, 8, 8) + top_layout.setSpacing(8) + + # Start/Stop group (matches OperatorTab style) + mode_group = QFrame() + mode_group.setStyleSheet( + f"QFrame {{ background-color: {theme.BG_BUTTON_GROUP}; " + f"border-radius: 6px; border: 1px solid {theme.BLUE_LIGHT_2}; }}" + ) + mode_layout = QHBoxLayout(mode_group) + mode_layout.setContentsMargins(6, 6, 6, 6) + mode_layout.setSpacing(6) + + BUTTON_WIDTH = 125 + self._start_btn = QPushButton("▶ Start") + self._start_btn.setFixedWidth(BUTTON_WIDTH) + self._start_btn.setStyleSheet(theme.button_style(theme.BUTTON_PREVIEW)) + self._start_btn.setToolTip( + "Start the alignment pipeline (no file saving). " + "Stops any running operator session is not allowed; stop that first." + ) + self._start_btn.clicked.connect(self._on_start_clicked) + mode_layout.addWidget(self._start_btn) + + self._simulator_btn = QPushButton("◈ Simulator") + self._simulator_btn.setFixedWidth(BUTTON_WIDTH) + self._simulator_btn.setCheckable(True) + self._simulator_btn.setStyleSheet(theme.button_style(theme.BUTTON_SIMULATOR)) + self._simulator_btn.setToolTip( + "Local synthetic alignment stream (random background + drifting beam spot).\n" + "No Serval, live-cli, or ZMQ — use to verify the UI. When active, Start does not\n" + "require Serval to be ready." + ) + self._simulator_btn.toggled.connect(self._on_simulator_toggled) + mode_layout.addWidget(self._simulator_btn) + + self._stop_btn = QPushButton("⏹ Stop") + self._stop_btn.setFixedWidth(BUTTON_WIDTH) + self._stop_btn.setStyleSheet(theme.button_style(theme.BUTTON_STOP)) + self._stop_btn.setEnabled(False) + self._stop_btn.clicked.connect(self._on_stop_clicked) + mode_layout.addWidget(self._stop_btn) + + top_layout.addWidget(mode_group) + + # Middle: rolling 60 s cps vs time (same metric as the giant readout). + self._cps_history_plot = pg.PlotWidget() + self._cps_history_plot.setBackground(theme.BG_DARK) + self._cps_history_plot.setSizePolicy( + QSizePolicy.Policy.Expanding, + QSizePolicy.Policy.Fixed, + ) + self._cps_history_plot.setMinimumWidth(180) + # Height chosen to match Operator top-bar button row (see top_bar min height below). + self._cps_history_plot.setFixedHeight(42) + self._cps_history_plot.setToolTip( + "Rolling last 60 s of live cps (sum of counts in each frame / flush interval).\n" + "Horizontal axis: seconds within the window (0 = oldest, 60 = now)." + ) + cps_pi = self._cps_history_plot.getPlotItem() + cps_pi.setMenuEnabled(False) + cps_pi.hideButtons() + cps_pi.showGrid(x=False, y=False) + cps_pi.setLabel("left", "cps") + # No bottom-axis text label — saves vertical space in the top bar strip chart. + # Qualitative strip chart only: no tick marks or numeric ticks. + for ax_name in ("left", "bottom"): + ax = cps_pi.getAxis(ax_name) + ax.setStyle(showValues=False, tickLength=0) + ax.setTicks([[]]) + self._cps_history_plot.getViewBox().setMouseEnabled(x=False, y=False) + self._cps_history_curve = self._cps_history_plot.plot( + [], + [], + pen=pg.mkPen(color=theme.BLUE_LIGHT_1, width=2), + ) + top_layout.addWidget( + self._cps_history_plot, + stretch=1, + alignment=Qt.AlignmentFlag.AlignVCenter, + ) + + # Giant cps readout — fixed-width scientific so the bar does not jump. + self._cps_label = QLabel(_format_cps(0.0)) + _cps_font = QFont() + _cps_font.setFamilies(["Consolas", "Monaco", "Courier New"]) + _cps_font.setPixelSize(32) + _cps_font.setBold(True) + self._cps_label.setFont(_cps_font) + self._cps_label.setStyleSheet(f"color: {theme.TEXT_PRIMARY}; padding: 0 8px;") + self._cps_label.setAlignment(Qt.AlignmentFlag.AlignRight | Qt.AlignmentFlag.AlignVCenter) + _fm = QFontMetrics(_cps_font) + self._cps_label.setMinimumWidth(_fm.horizontalAdvance("9.99E+999 cps")) + self._cps_label.setToolTip( + "Live count rate: sum of pixels in the latest displayed frame divided by\n" + "the alignment flush interval (1 / Rate Hz). Updates every flush." + ) + top_layout.addWidget(self._cps_label) + + return top_bar + + def _build_left_sidebar(self) -> QWidget: + """Left column: alignment parameters (mirrors Operator tab fixed-width sidebar).""" + left_panel = QWidget() + left_panel.setFixedWidth(280) + left_layout = QVBoxLayout(left_panel) + left_layout.setContentsMargins(0, 0, 0, 0) + left_layout.setSpacing(0) + + group = QGroupBox("Alignment") + group.setStyleSheet(theme.group_box_style()) + group.setToolTip( + "Alignment-only controls. Update rate applies on the next Start.\n" + "Display options apply immediately to the live view." + ) + g = QVBoxLayout(group) + g.setSpacing(8) + + rate_row = QHBoxLayout() + self._rate_label = QLabel("Rate (Hz)") + self._rate_label.setStyleSheet( + f"QLabel {{ color: {theme.TEXT_SECONDARY}; }} " f"QLabel:disabled {{ color: {theme.TEXT_MUTED}; }}" + ) + self._rate_label.setToolTip( + "Wall-clock flush rate driving the alignment image (1–30 Hz). Applied on next Start." + ) + rate_row.addWidget(self._rate_label) + self._rate_input = QSpinBox() + self._rate_input.setRange(*preferences.ALIGNMENT_RATE_HZ_RANGE) + self._rate_input.setValue(30) + self._rate_input.setStyleSheet(theme.input_style()) + self._rate_input.setToolTip(self._rate_label.toolTip()) + rate_row.addWidget(self._rate_input) + g.addLayout(rate_row) + + z_range_row = QHBoxLayout() + z_range_row.setSpacing(4) + self._z_range_label = QLabel("Z range") + self._z_range_label.setStyleSheet( + f"QLabel {{ color: {theme.TEXT_SECONDARY}; }} " f"QLabel:disabled {{ color: {theme.TEXT_MUTED}; }}" + ) + self._range_combo = QComboBox() + self._range_combo.addItems(["Auto", "Manual"]) + self._range_combo.setCurrentText("Auto") + self._range_combo.setStyleSheet(theme.input_style()) + self._range_combo.setToolTip( + "Auto: Z levels follow each frame's min/max.\n" + "Manual: drag the triangular handles on the colorbar to set min/max." + ) + self._z_range_label.setToolTip(self._range_combo.toolTip()) + self._range_combo.currentTextChanged.connect(self._on_range_mode_changed) + z_range_row.addWidget(self._z_range_label) + z_range_row.addStretch(1) + z_range_row.addWidget(self._range_combo) + _rate_w = self._rate_input.sizeHint().width() + self._range_combo.setCurrentText("Manual") + _manual_w = self._range_combo.sizeHint().width() + self._range_combo.setCurrentText("Auto") + _lo = max(_rate_w + 12, _manual_w + 6) + _hi = max(_lo + 4, _manual_w + 36) + self._range_combo.setFixedWidth((_lo + _hi) // 2) + g.addLayout(z_range_row) + + int_row = QHBoxLayout() + int_lbl = QLabel("Show integrated") + int_lbl.setStyleSheet( + f"QLabel {{ color: {theme.TEXT_SECONDARY}; }} " f"QLabel:disabled {{ color: {theme.TEXT_MUTED}; }}" + ) + int_tip = ( + "Off (default): display the latest flush only.\n" "On: cumulative sum since switched on. Reset on Start." + ) + int_lbl.setToolTip(int_tip) + int_row.addWidget(int_lbl) + int_row.addStretch() + self._integrated_chk = _AlignmentOnOffToggle(initial=False) + self._integrated_chk.set_option_tooltip(int_tip) + self._integrated_chk.toggled.connect(self._on_integrated_toggled) + int_row.addWidget(self._integrated_chk) + g.addLayout(int_row) + + bin_row = QHBoxLayout() + bin_lbl = QLabel("Binarize") + bin_lbl.setStyleSheet( + f"QLabel {{ color: {theme.TEXT_SECONDARY}; }} " f"QLabel:disabled {{ color: {theme.TEXT_MUTED}; }}" + ) + bin_tip = ( + "On: any pixel with count > 0 maps to brightest LUT color; zeros to darkest.\n" + "Makes single-hit visibility easy during alignment. Overrides Log and manual Z range." + ) + bin_lbl.setToolTip(bin_tip) + bin_row.addWidget(bin_lbl) + bin_row.addStretch() + self._binarize_chk = _AlignmentOnOffToggle(initial=True) + self._binarize_chk.set_option_tooltip(bin_tip) + self._binarize_chk.toggled.connect(self._on_binarize_toggled) + bin_row.addWidget(self._binarize_chk) + g.addLayout(bin_row) + + log_row = QHBoxLayout() + log_lbl = QLabel("Log") + log_lbl.setStyleSheet( + f"QLabel {{ color: {theme.TEXT_SECONDARY}; }} " f"QLabel:disabled {{ color: {theme.TEXT_MUTED}; }}" + ) + log_tip = ( + "On: apply log10(data + 1) before display so faint features pop.\n" + "Turning On resets Z range to Auto. No effect while Binarize is On." + ) + log_lbl.setToolTip(log_tip) + log_row.addWidget(log_lbl) + log_row.addStretch() + self._log_chk = _AlignmentOnOffToggle(initial=False) + self._log_chk.set_option_tooltip(log_tip) + self._log_chk.toggled.connect(self._on_log_toggled) + log_row.addWidget(self._log_chk) + g.addLayout(log_row) + + xh_row = QHBoxLayout() + xh_lbl = QLabel("Crosshair") + xh_lbl.setStyleSheet( + f"QLabel {{ color: {theme.TEXT_SECONDARY}; }} " f"QLabel:disabled {{ color: {theme.TEXT_MUTED}; }}" + ) + xh_tip = ( + "On: two red dotted squares — outer (50,50)–(206,206); inner centered with " + "side length 28/156 of the outer edge." + ) + xh_lbl.setToolTip(xh_tip) + xh_row.addWidget(xh_lbl) + xh_row.addStretch() + self._crosshair_chk = _AlignmentOnOffToggle(initial=True) + self._crosshair_chk.set_option_tooltip(xh_tip) + self._crosshair_chk.toggled.connect(self._on_crosshair_toggled) + xh_row.addWidget(self._crosshair_chk) + g.addLayout(xh_row) + + stats_group = QGroupBox("Statistics") + stats_group.setStyleSheet(theme.group_box_style()) + stats_group.setToolTip( + "Statistics of the most recent flush. Sum is total counts in the\n" + "displayed frame; Max is the brightest pixel. No signal means the\n" + "flush arrived but contained zero pixels." + ) + stats_layout = QVBoxLayout(stats_group) + stats_layout.setSpacing(4) + + self._stats_labels: dict[str, QLabel] = {} + _stat_val_style = f"font-family: monospace; color: {theme.TEXT_PRIMARY};" + for key, title in ( + ("frame", "Frame"), + ("sum", "Sum"), + ("max", "Max"), + ("status", "Status"), + ): + row = QHBoxLayout() + name_label = QLabel(title) + name_label.setStyleSheet(f"color: {theme.TEXT_MUTED};") + value_label = QLabel("—" if key != "status" else "idle") + value_label.setStyleSheet(_stat_val_style) + value_label.setAlignment(Qt.AlignmentFlag.AlignRight | Qt.AlignmentFlag.AlignVCenter) + if key == "status": + value_label.setWordWrap(True) + value_label.setAlignment(Qt.AlignmentFlag.AlignRight | Qt.AlignmentFlag.AlignTop) + row.addWidget(name_label) + row.addStretch() + row.addWidget(value_label) + stats_layout.addLayout(row) + self._stats_labels[key] = value_label + + left_layout.addWidget(group) + left_layout.addWidget(stats_group) + left_layout.addStretch() + return left_panel + + def _build_image_area(self) -> QWidget: + """Main XY plot (square) plus Z histogram column (colorbar / LUT only).""" + outer = QWidget() + outer_h = QHBoxLayout(outer) + outer_h.setContentsMargins(0, 0, 0, 0) + outer_h.setSpacing(8) + + self._gw = pg.GraphicsLayoutWidget() + self._gw.setBackground(theme.BG_DARK) + self._gw.setSizePolicy(QSizePolicy.Policy.Expanding, QSizePolicy.Policy.Expanding) + self._gw.setMinimumHeight(480) + + self._plot = self._gw.addPlot(row=0, col=0) + self._plot.showAxis("right") + self._plot.showAxis("top") + for ax_name in ("right", "top"): + ax = self._plot.getAxis(ax_name) + ax.setStyle(showValues=False) + self._plot.setLabel("left", "Y (pixel)") + self._plot.setLabel("bottom", "X (pixel)") + self._plot.hideButtons() + self._plot.setMenuEnabled(False) + + vb = self._plot.getViewBox() + vb.setMouseEnabled(x=False, y=False) + vb.setDefaultPadding(0) + vb.disableAutoRange() + vb.setRange(xRange=(0, _DETECTOR_X), yRange=(0, _DETECTOR_Y), padding=0) + vb.setBackgroundColor(theme.BG_DARK) + self._viewbox = vb + + self._image_item = pg.ImageItem( + np.zeros((_DETECTOR_Y, _DETECTOR_X), dtype=np.float32), + axisOrder="row-major", + ) + self._image_item.setLookupTable(_grayscale_lut()) + self._plot.addItem(self._image_item) + + self._target_overlay_items = _build_alignment_target_overlay(self._plot) + + self._waiting_text = pg.TextItem( + "Waiting for first flush…", + color=(255, 255, 255, 180), + anchor=(0.5, 0.5), + ) + self._waiting_text.setPos(_DETECTOR_X / 2, _DETECTOR_Y / 2) + self._plot.addItem(self._waiting_text, ignoreBounds=True) + + gl_layout = self._gw.ci.layout + gl_layout.setColumnStretchFactor(0, 1) + + right = QWidget() + right.setFixedWidth(100) + right_v = QVBoxLayout(right) + right_v.setContentsMargins(0, 0, 0, 0) + right_v.setSpacing(4) + + self._hist_gw = pg.GraphicsLayoutWidget() + self._hist_gw.setBackground(theme.BG_DARK) + self._hist_gw.setSizePolicy(QSizePolicy.Policy.Fixed, QSizePolicy.Policy.Expanding) + self._hist_gw.setMinimumWidth(88) + + self._hist = pg.HistogramLUTItem() + self._hist.gradient.setColorMap(pg.ColorMap([0.0, 1.0], [(0, 0, 0), (255, 255, 255)])) + self._hist.setImageItem(self._image_item) + self._image_item.setLookupTable(_grayscale_lut()) + self._hist.sigLevelsChanged.connect(self._on_hist_levels_changed) + self._hist_gw.addItem(self._hist, row=0, col=0) + right_v.addWidget(self._hist_gw, stretch=1) + + outer_h.addWidget(self._gw, stretch=1) + outer_h.addWidget(right) + + return outer + + # ------------------------------------------------------------------ + # Public slots / API used by MainWindow + # ------------------------------------------------------------------ + + @Slot(bool) + def set_acquiring(self, acquiring: bool) -> None: + """Toggle Start/Stop, rate controls, and integrated-buffer state on acquisition.""" + self._acquiring = acquiring + self._stop_btn.setEnabled(acquiring) + self._simulator_btn.setEnabled(not acquiring) + self._rate_label.setEnabled(not acquiring) + self._rate_input.setEnabled(not acquiring) + self._update_z_range_controls_enabled() + self._refresh_alignment_start_state() + if acquiring: + # Reset the integrated buffer on each Start so a fresh accumulation + # begins; if integrated was off, this is a no-op. + self._integrated_sum = None + self._clear_cps_history_plot() + # Re-arm the per-run diagnostics: log first few flushes again, and + # re-show the "Waiting for first flush…" overlay so the user gets + # immediate feedback on whether ZMQ is delivering. + self._flush_count = 0 + self._set_alignment_stats_row( + frame="—", + sum_s="—", + max_s="—", + status="waiting for first flush…", + ) + if self._waiting_text is None: + self._waiting_text = pg.TextItem( + "Waiting for first flush…", + color=(255, 255, 255, 180), + anchor=(0.5, 0.5), + ) + self._waiting_text.setPos(_DETECTOR_X / 2, _DETECTOR_Y / 2) + self._plot.addItem(self._waiting_text, ignoreBounds=True) + else: + self._clear_cps_history_plot() + self._set_alignment_stats_row( + frame="—", + sum_s="—", + max_s="—", + status="idle (stopped)", + ) + + def on_serval_process_running(self, running: bool) -> None: + """Forwarded from MainWindow when the Serval subprocess starts or stops.""" + self._serval_process_running = running + self._serval_hw_ready = False + self._refresh_alignment_start_state() + + def on_serval_chip_temps_line_seen(self) -> None: + """Forwarded from MainWindow when Serval logs chip temperatures (HW ready).""" + if not self._serval_process_running or self._serval_hw_ready: + return + self._serval_hw_ready = True + self._refresh_alignment_start_state() + + def _refresh_alignment_start_state(self) -> None: + """Start enabled when idle and (local simulator or Serval HW-ready).""" + if self._acquiring: + self._start_btn.setEnabled(False) + return + allow = self._simulator_btn.isChecked() or (self._serval_process_running and self._serval_hw_ready) + self._start_btn.setEnabled(allow) + + @Slot(bool) + def _on_simulator_toggled(self, _checked: bool) -> None: + self._refresh_alignment_start_state() + + @Slot(object) + def on_flush_received(self, flush_data: FlushData) -> None: + """Render a flush from the streaming server (alignment-mode shape (X, Y, 1)).""" + array = flush_data.array + metadata = flush_data.metadata + mode = metadata.get("mode") + + # Defensive: alignment flushes are always (X, Y, 1) but route by mode. + if mode != "alignment": + # If anything other than alignment mode arrives here, the routing + # in MainWindow._on_flush_received broke — log loudly so this + # surfaces in the engineering tab. + logger.warning( + "AlignmentTab.on_flush_received received non-alignment flush (mode=%r); ignoring", + mode, + ) + return + if array.ndim == 3 and array.shape[-1] == 1: + latest_2d = array[..., 0] + elif array.ndim == 2: + latest_2d = array + else: + logger.warning("Alignment tab got unexpected array shape %s; ignoring", array.shape) + return + + self._last_2d = latest_2d + + if self._integrated_chk.isChecked(): + if self._integrated_sum is None: + self._integrated_sum = latest_2d.astype(np.float64, copy=True) + else: + self._integrated_sum += latest_2d + + # Per-frame diagnostics. Computed once here, reused for the cps + # label, the sidebar Statistics group, and the system log. + frame_sum = float(latest_2d.sum()) + frame_max = float(latest_2d.max()) if latest_2d.size else 0.0 + flush_interval_s = float(metadata.get("flush_interval_s") or 0.0) + cps = frame_sum / flush_interval_s if flush_interval_s > 0 else 0.0 + self._cps_label.setText(_format_cps(cps)) + self._append_cps_sample(cps) + + # Sidebar stats — status answers "why do I see nothing?" at a glance. + if frame_sum == 0: + status = "No signal (flush arrived empty)" + else: + status = "OK" + self._set_alignment_stats_row( + frame=f"{latest_2d.shape[0]}×{latest_2d.shape[1]}", + sum_s=f"{frame_sum:.3e}", + max_s=f"{frame_max:.3e}", + status=status, + ) + + # Hide the "Waiting for first flush…" overlay on first flush. + if self._waiting_text is not None: + self._plot.removeItem(self._waiting_text) + self._waiting_text = None + + # Log the first 5 flushes always, then every 30th (≈ once a second + # at 30 Hz). Goes to the engineering tab's system log via the root + # logger; the per-flush ZMQ log message in MainWindow stays unchanged. + self._flush_count += 1 + if self._flush_count <= 5 or self._flush_count % 30 == 0: + logger.info( + "AlignmentTab flush #%d: shape=%s, sum=%g, max=%g, mode=%s, flush_interval=%s", + self._flush_count, + latest_2d.shape, + frame_sum, + frame_max, + mode, + flush_interval_s, + ) + + self._refresh_image() + + # ------------------------------------------------------------------ + # Internals + # ------------------------------------------------------------------ + + def _set_alignment_stats_row(self, *, frame: str, sum_s: str, max_s: str, status: str) -> None: + self._stats_labels["frame"].setText(frame) + self._stats_labels["sum"].setText(sum_s) + self._stats_labels["max"].setText(max_s) + self._stats_labels["status"].setText(status) + + def _clear_cps_history_plot(self) -> None: + """Drop all CPS timeline samples (e.g. on Stop / fresh Start).""" + self._cps_history.clear() + self._cps_history_curve.setData([], []) + self._cps_history_plot.setXRange(0, _CPS_HISTORY_WINDOW_S, padding=0) + self._cps_history_plot.setYRange(0, 1, padding=0) + + def _append_cps_sample(self, cps: float) -> None: + """Record one cps sample and refresh the rolling 60 s strip chart.""" + now = time.monotonic() + self._cps_history.append((now, float(cps))) + cutoff = now - _CPS_HISTORY_WINDOW_S + while self._cps_history and self._cps_history[0][0] < cutoff: + self._cps_history.popleft() + if not self._cps_history: + self._cps_history_curve.setData([], []) + self._cps_history_plot.setXRange(0, _CPS_HISTORY_WINDOW_S, padding=0) + self._cps_history_plot.setYRange(0, 1, padding=0) + return + t0 = now - _CPS_HISTORY_WINDOW_S + xs = np.fromiter((t - t0 for t, _ in self._cps_history), dtype=np.float64, count=len(self._cps_history)) + ys = np.fromiter((v for _, v in self._cps_history), dtype=np.float64, count=len(self._cps_history)) + self._cps_history_curve.setData(xs, ys) + self._cps_history_plot.setXRange(0, _CPS_HISTORY_WINDOW_S, padding=0) + vmin = float(np.nanmin(ys)) + vmax = float(np.nanmax(ys)) + if not (np.isfinite(vmin) and np.isfinite(vmax)): + return + if vmax <= vmin: + span = max(1.0, abs(vmin) * 0.05 + 1.0) + self._cps_history_plot.setYRange(vmin - 0.05 * span, vmax + span, padding=0) + else: + pad = (vmax - vmin) * 0.08 + lo = max(0.0, vmin - pad) + self._cps_history_plot.setYRange(lo, vmax + pad, padding=0) + + def _update_z_range_controls_enabled(self) -> None: + """Z range UI is inactive while Binarize overrides levels (pinned 0–1).""" + ok = not self._binarize_chk.isChecked() + self._z_range_label.setEnabled(ok) + self._range_combo.setEnabled(ok) + + @contextmanager + def _hist_levels_change_guard(self): + """Temporarily disconnect ``sigLevelsChanged`` so programmatic LUT updates do not flip Auto→Manual.""" + try: + self._hist.sigLevelsChanged.disconnect(self._on_hist_levels_changed) + except (TypeError, RuntimeError): + pass + try: + yield + finally: + self._hist.sigLevelsChanged.connect(self._on_hist_levels_changed) + + def _refresh_image(self) -> None: + """Push the current (latest or integrated, log or linear) frame to ImageItem. + + The entire method runs inside ``_hist_levels_change_guard`` so that the + ``setImage`` call (which triggers HistogramLUTItem.imageChanged → + region.setRegion → sigLevelsChanged) never reaches ``_on_hist_levels_changed`` + and cannot flip the Z range combo from Auto back to Manual. + """ + source = self._integrated_sum if self._integrated_chk.isChecked() else self._last_2d + if source is None: + return + + with self._hist_levels_change_guard(): + # Binarize takes precedence over Log/Manual: any pixel > 0 → 1.0, + # zeros stay 0.0, levels pinned to (0, 1) so the only colors shown + # are LUT-min (background) and LUT-max (any hit). + if self._binarize_chk.isChecked(): + disp = (source > 0).astype(np.float32) + self._image_item.setImage(disp.T, autoLevels=False, levels=(0.0, 1.0)) + self._hist.setLevels(0.0, 1.0) + self._hist.setHistogramRange(0.0, 1.0) + self._image_item.update() + return + + if self._log_chk.isChecked(): + disp = np.log10(source.astype(np.float32) + 1.0) + else: + disp = source.astype(np.float32, copy=False) + + # ImageItem with axisOrder='row-major' wants (rows=Y, cols=X). Our data + # is (X, Y), so transpose. We compute levels deterministically with + # numpy min/max (rather than passing autoLevels=True, which uses + # pyqtgraph's quickMinMax — subsampled and can miss sparse hot pixels + # on a 256x256 detector). + auto = self._range_combo.currentText() == "Auto" + if auto: + data_lo = float(disp.min()) + data_hi = float(disp.max()) + if data_hi <= data_lo: + # Fully-flat frame (e.g. all zeros). Pick a (0, 1) span so the + # LUT renders sensibly instead of mapping everything to a + # single end of the colormap. + data_lo = 0.0 + data_hi = 1.0 + self._image_item.setImage(disp.T, autoLevels=False, levels=(data_lo, data_hi)) + # Sync the LUT widget's histogram range to the new image so the + # handles sit at the data extents. + self._hist.setHistogramRange(data_lo, data_hi) + self._hist.setLevels(data_lo, data_hi) + else: + try: + lo, hi = self._hist.getLevels() + except Exception: + lo, hi = 0.0, 1.0 + lo, hi = float(lo), float(hi) + if hi <= lo: + hi = lo + 1.0 + self._image_item.setImage(disp.T, autoLevels=False, levels=(lo, hi)) + self._hist.setLevels(lo, hi) + + # Force an immediate repaint of the ImageItem. + self._image_item.update() + + @Slot(str) + def _on_range_mode_changed(self, text: str) -> None: + _ = text + self._refresh_image() + + @Slot() + def _on_hist_levels_changed(self) -> None: + """Dragging LUT handles switches to Manual and reapplies levels.""" + if not self._hist_lut_user_callbacks_enabled: + return + if self._binarize_chk.isChecked(): + return + if self._range_combo.currentText() != "Manual": + self._range_combo.setCurrentText("Manual") + self._refresh_image() + + @Slot(bool) + def _on_integrated_toggled(self, checked: bool) -> None: + # Reset the buffer whenever the toggle changes state, so cumulative + # always starts at zero from the moment the box is checked. + self._integrated_sum = None + self._refresh_image() + + @Slot(bool) + def _on_binarize_toggled(self, checked: bool) -> None: + """Disable Log + manual-range controls while binarize is active. + + They have no effect on the rendered image in binarize mode, so + greying them out makes the override semantics obvious. Their saved + states are preserved (we don't uncheck Log here), so the user gets + their previous settings back when they uncheck Binarize. + """ + # Log only matters in non-binarize mode; grey it out for clarity. + self._log_chk.setEnabled(not checked) + self._update_z_range_controls_enabled() + self._refresh_image() + + @Slot(bool) + def _on_log_toggled(self, checked: bool) -> None: + """Force range mode back to Auto when log is toggled. + + Linear-space manual levels (e.g. 0–100 counts) become nonsensical + after a log10 transform (data range 0–~5), and vice versa. Rather + than try to be clever about converting between the two, we snap + to Auto so the image is always visible after the toggle. The user + can then re-enter Manual once they see the actual data range. + """ + if self._range_combo.currentText() != "Auto": + self._range_combo.setCurrentText("Auto") + else: + self._refresh_image() + + @Slot(bool) + def _on_crosshair_toggled(self, checked: bool) -> None: + for it in getattr(self, "_target_overlay_items", ()): + it.setVisible(checked) + + @Slot() + def _on_start_clicked(self) -> None: + # Fresh accumulation buffer for "Show integrated" if it's already on. + self._integrated_sum = None + + if self._simulator_btn.isChecked(): + # Local simulator: synthetic flushes via QTimer; no start_requested. + logger.info("AlignmentTab: starting local simulator") + self.set_acquiring(True) + rate_hz = max(1.0, float(self._rate_input.value())) + self._fake_timer = QTimer(self) + self._fake_timer.setInterval(int(round(1000.0 / rate_hz))) + self._fake_timer.timeout.connect(self._emit_fake_flush) + self._fake_timer.start() + return + + params = { + "alignment_rate_hz": int(self._rate_input.value()), + # ``acq.py``'s own "longest possible" default (~220 days). The user + # stops via the Stop button; this just removes any acquisition-side + # auto-stop surprise. + "duration": 19_008_000, + } + self.start_requested.emit("alignment", params) + + @Slot() + def _on_stop_clicked(self) -> None: + if self._fake_timer is not None and self._fake_timer.isActive(): + logger.info("AlignmentTab: stopping local simulator") + self._fake_timer.stop() + self._fake_timer = None + self.set_acquiring(False) + return + self.stop_requested.emit() + + @Slot() + def _emit_fake_flush(self) -> None: + """Synthesize one alignment flush and feed it into ``on_flush_received``. + + Pattern: ~200–800 uniform-random hits per frame (1–3 counts each) plus + a Gaussian "beam spot" of ~2000 hits at a slowly-drifting center, so + the user sees both background noise and a clearly-recognizable feature. + """ + rng = self._fake_rng + arr = np.zeros((_DETECTOR_X, _DETECTOR_Y, 1), dtype=np.uint32) + + # Uniform background + n_bg = int(rng.integers(200, 800)) + bx = rng.integers(0, _DETECTOR_X, n_bg) + by = rng.integers(0, _DETECTOR_Y, n_bg) + np.add.at(arr, (bx, by, 0), 1) + + # Gaussian beam spot, slow drift via flush counter so it visibly moves. + n_spot = int(rng.integers(1500, 2500)) + drift_x = 128 + 30 * np.sin(self._flush_count / 60.0) + drift_y = 128 + 30 * np.cos(self._flush_count / 60.0) + sx = rng.normal(drift_x, 12.0, n_spot).clip(0, _DETECTOR_X - 1).astype(np.int32) + sy = rng.normal(drift_y, 12.0, n_spot).clip(0, _DETECTOR_Y - 1).astype(np.int32) + np.add.at(arr, (sx, sy, 0), 1) + + rate_hz = max(1.0, float(self._rate_input.value())) + flush_interval_s = 1.0 / rate_hz + meta = { + "mode": "alignment", + "flush_interval_s": flush_interval_s, + "flush_number": self._flush_count + 1, + "shape": (_DETECTOR_X, _DETECTOR_Y, 1), + "cycles_in_flush": 1, + } + self.on_flush_received(FlushData(array=arr, metadata=meta)) + + # ------------------------------------------------------------------ + # Preferences + # ------------------------------------------------------------------ + + def _build_preferences(self) -> dict: + """Snapshot of the alignment-tab widget state for the JSON file.""" + lo, hi = 0.0, 100.0 + try: + lo, hi = self._hist.getLevels() + except Exception: + pass + return { + "alignment_rate_hz": int(self._rate_input.value()), + "alignment_auto_range": self._range_combo.currentText() == "Auto", + "alignment_manual_min": float(lo), + "alignment_manual_max": float(hi), + "alignment_log": bool(self._log_chk.isChecked()), + "alignment_binarize": bool(self._binarize_chk.isChecked()), + "alignment_show_integrated": bool(self._integrated_chk.isChecked()), + "alignment_show_crosshair": bool(self._crosshair_chk.isChecked()), + "alignment_simulator": bool(self._simulator_btn.isChecked()), + } + + def load_alignment_preferences(self, path: Optional[Union[Path, str]] = None) -> None: + """Restore alignment widgets from the shared on-disk preferences.""" + prefs = preferences.load_operator_preferences(path) + self._rate_input.setValue(int(prefs["alignment_rate_hz"])) + self._range_combo.setCurrentText("Auto" if prefs["alignment_auto_range"] else "Manual") + self._log_chk.setChecked(bool(prefs["alignment_log"])) + self._binarize_chk.setChecked(bool(prefs["alignment_binarize"])) + self._integrated_chk.setChecked(bool(prefs["alignment_show_integrated"])) + self._crosshair_chk.setChecked(bool(prefs["alignment_show_crosshair"])) + self._simulator_btn.setChecked(bool(prefs["alignment_simulator"])) + + if not prefs["alignment_auto_range"]: + lo = float(prefs["alignment_manual_min"]) + hi = float(prefs["alignment_manual_max"]) + if hi <= lo: + hi = lo + 1.0 + with self._hist_levels_change_guard(): + self._hist.setLevels(lo, hi) + + self._on_crosshair_toggled(self._crosshair_chk.isChecked()) + self._on_binarize_toggled(self._binarize_chk.isChecked()) + self._update_z_range_controls_enabled() + self._refresh_image() + + def save_alignment_preferences(self, path: Optional[Union[Path, str]] = None) -> None: + """Persist alignment-tab widgets to the shared preferences file. + + The save merges with on-disk state (see ``preferences.save_operator_preferences``) + so this can run independently of ``OperatorTab.save_operator_preferences`` + without clobbering operator keys. + """ + preferences.save_operator_preferences(self._build_preferences(), path) diff --git a/src/splash_timepix/ui/assets/icon-2048.png b/src/splash_timepix/ui/assets/icon-2048.png new file mode 100644 index 0000000..746a3a2 Binary files /dev/null and b/src/splash_timepix/ui/assets/icon-2048.png differ diff --git a/src/splash_timepix/ui/assets/icon.png b/src/splash_timepix/ui/assets/icon.png new file mode 100644 index 0000000..56cc75e Binary files /dev/null and b/src/splash_timepix/ui/assets/icon.png differ diff --git a/src/splash_timepix/ui/centroider_tab.py b/src/splash_timepix/ui/centroider_tab.py new file mode 100644 index 0000000..7c00bfc --- /dev/null +++ b/src/splash_timepix/ui/centroider_tab.py @@ -0,0 +1,577 @@ +"""Centroider tab - sweep tpx3 clustering parameters and compare the results. + +The scientist picks a .tpx3 file and a list of eps-s (pixel) and eps-t (time) +values. Pressing "Centroid" runs the centroider sweep (tools/centroider) over +the full eps-s x eps-t grid, producing one clustered .h5 per combination plus a +PixelHits baseline. As each run finishes its x-histogram is loaded and overlaid +on the summary plot so the best eps-s / eps-t can be chosen by eye. + +All sweep orchestration / I/O lives in the centroider backend; this tab only +builds the UI and relays the CentroiderWorker's signals. +""" + +import logging +from pathlib import Path +from typing import Dict, List, Optional, Tuple + +import numpy as np +import pyqtgraph as pg +from PySide6.QtCore import Qt, Slot +from PySide6.QtWidgets import ( + QAbstractItemView, + QCheckBox, + QFileDialog, + QGroupBox, + QHBoxLayout, + QHeaderView, + QLabel, + QLineEdit, + QMessageBox, + QProgressBar, + QPushButton, + QSplitter, + QTableWidget, + QVBoxLayout, + QWidget, +) + +from . import theme +from .workers import CentroiderWorker, import_centroider_api + +logger = logging.getLogger(__name__) + +# Apply dark theme defaults to pyqtgraph (process-global; matches alignment tab). +pg.setConfigOptions(background=theme.BG_DARK, foreground=theme.TEXT_PRIMARY, antialias=False) + +# Curve colors cycled across combinations (baseline uses a separate grey). +_CURVE_COLORS = [ + theme.BLUE_LIGHT_1, + theme.TERTIARY_GREEN, + theme.TERTIARY_YELLOW, + theme.TERTIARY_ORANGE, + theme.TERTIARY_RED, + theme.TERTIARY_PURPLE, + theme.TERTIARY_TEAL, + theme.BLUE_LIGHT_2, + theme.TERTIARY_MUD, + theme.BLUE_LIGHT_3, +] + +# Number of steps the waterfall offset divides the visible dynamic range into. +_WATERFALL_DIVISIONS = 10 + + +def _format_eta(seconds: Optional[float]) -> str: + if seconds is None: + return "estimating..." + s = int(max(0.0, seconds)) + h, rem = divmod(s, 3600) + m, s = divmod(rem, 60) + if h: + return f"{h}h {m:02d}m {s:02d}s" + if m: + return f"{m}m {s:02d}s" + return f"{s}s" + + +class _ComboCell(QWidget): + """One cell of the combinations grid: a show/hide checkbox + status label.""" + + def __init__(self, eps_s: int, eps_t: str, on_toggled, parent=None): + super().__init__(parent) + self.eps_s = eps_s + self.eps_t = eps_t + + layout = QVBoxLayout(self) + layout.setContentsMargins(6, 4, 6, 4) + layout.setSpacing(2) + + self._checkbox = QCheckBox("show") + self._checkbox.setChecked(True) + self._checkbox.setEnabled(False) + self._checkbox.toggled.connect(lambda _checked: on_toggled()) + layout.addWidget(self._checkbox) + + self._status = QLabel("queued") + self._status.setStyleSheet(f"color: {theme.TEXT_MUTED}; font-size: 10px;") + layout.addWidget(self._status) + + def is_checked(self) -> bool: + return self._checkbox.isChecked() and self._checkbox.isEnabled() + + def set_running(self) -> None: + self._status.setText("running...") + self._status.setStyleSheet(f"color: {theme.STATUS_STREAMING}; font-size: 10px;") + + def set_status(self, status: str, wall_seconds: float = 0.0) -> None: + if status == "ok": + self._status.setText(f"ok ({wall_seconds:.1f}s)") + self._status.setStyleSheet(f"color: {theme.STATUS_OK}; font-size: 10px;") + self._checkbox.setEnabled(True) + elif status == "skipped": + self._status.setText("cached") + self._status.setStyleSheet(f"color: {theme.TEXT_SECONDARY}; font-size: 10px;") + self._checkbox.setEnabled(True) + else: + self._status.setText("failed") + self._status.setStyleSheet(f"color: {theme.STATUS_ERROR}; font-size: 10px;") + self._checkbox.setEnabled(False) + + +class CentroiderTab(QWidget): + """Tab for sweeping clustering parameters and comparing clustered outputs.""" + + def __init__(self, parent=None): + super().__init__(parent) + + self._worker: Optional[CentroiderWorker] = None + + # Plot state. + # baseline: dict with xs/counts/item, or None until the PixelHits run finishes. + self._baseline: Optional[dict] = None + # combo curves keyed by (eps_s, eps_t); _combo_order preserves insertion order. + self._combo_curves: Dict[Tuple[int, str], dict] = {} + self._combo_order: List[Tuple[int, str]] = [] + # grid cells keyed by (eps_s, eps_t). + self._cells: Dict[Tuple[int, str], _ComboCell] = {} + + self._setup_ui() + + # ------------------------------------------------------------------ + # UI construction + # ------------------------------------------------------------------ + + def _setup_ui(self) -> None: + root = QVBoxLayout(self) + root.setContentsMargins(10, 10, 10, 10) + root.setSpacing(10) + + root.addWidget(self._build_inputs_group()) + root.addWidget(self._build_progress_group()) + + splitter = QSplitter(Qt.Orientation.Horizontal) + splitter.addWidget(self._build_grid_group()) + splitter.addWidget(self._build_plot_group()) + splitter.setStretchFactor(0, 0) + splitter.setStretchFactor(1, 1) + splitter.setSizes([420, 900]) + root.addWidget(splitter, stretch=1) + + def _build_inputs_group(self) -> QGroupBox: + group = QGroupBox("Inputs") + group.setStyleSheet(theme.group_box_style()) + layout = QVBoxLayout(group) + layout.setContentsMargins(12, 16, 12, 12) + layout.setSpacing(8) + + # --- TPX3 file picker --- + file_row = QHBoxLayout() + file_label = QLabel("TPX3 File") + file_label.setMinimumWidth(110) + file_label.setStyleSheet(f"color: {theme.TEXT_PRIMARY};") + self._file_input = QLineEdit() + self._file_input.setPlaceholderText("/path/to/data.tpx3") + self._file_input.setStyleSheet(theme.input_style()) + browse_btn = QPushButton("Browse") + browse_btn.setStyleSheet(theme.secondary_button_style()) + browse_btn.clicked.connect(self._browse_file) + file_row.addWidget(file_label) + file_row.addWidget(self._file_input, stretch=1) + file_row.addWidget(browse_btn) + layout.addLayout(file_row) + + # --- eps-s --- + eps_s_row = QHBoxLayout() + eps_s_label = QLabel("eps-s (pixels)") + eps_s_label.setMinimumWidth(110) + eps_s_label.setStyleSheet(f"color: {theme.TEXT_PRIMARY};") + self._eps_s_input = QLineEdit("1,2,3") + self._eps_s_input.setPlaceholderText("comma-separated positive ints, e.g. 1,2,3") + self._eps_s_input.setStyleSheet(theme.input_style()) + eps_s_row.addWidget(eps_s_label) + eps_s_row.addWidget(self._eps_s_input, stretch=1) + layout.addLayout(eps_s_row) + + # --- eps-t --- + eps_t_row = QHBoxLayout() + eps_t_label = QLabel("eps-t (ns)") + eps_t_label.setMinimumWidth(110) + eps_t_label.setStyleSheet(f"color: {theme.TEXT_PRIMARY};") + self._eps_t_input = QLineEdit("5,10,20") + self._eps_t_input.setPlaceholderText("comma-separated integers in ns, e.g. 20,100,500") + self._eps_t_input.setStyleSheet(theme.input_style()) + eps_t_row.addWidget(eps_t_label) + eps_t_row.addWidget(self._eps_t_input, stretch=1) + layout.addLayout(eps_t_row) + + # --- Centroid / Stop buttons --- + _BTN_WIDTH = 110 + btn_row = QHBoxLayout() + btn_row.addStretch(1) + self._centroid_btn = QPushButton("Centroid") + self._centroid_btn.setFixedWidth(_BTN_WIDTH) + self._centroid_btn.setStyleSheet(theme.button_style(theme.BUTTON_START)) + self._centroid_btn.clicked.connect(self._on_centroid_clicked) + btn_row.addWidget(self._centroid_btn) + self._stop_btn = QPushButton("\u23f9 Stop") + self._stop_btn.setFixedWidth(_BTN_WIDTH) + self._stop_btn.setStyleSheet(theme.button_style(theme.BUTTON_STOP)) + self._stop_btn.setEnabled(False) + self._stop_btn.clicked.connect(self._on_stop_clicked) + btn_row.addWidget(self._stop_btn) + layout.addLayout(btn_row) + + return group + + def _build_progress_group(self) -> QGroupBox: + group = QGroupBox("Progress") + group.setStyleSheet(theme.group_box_style()) + layout = QVBoxLayout(group) + layout.setContentsMargins(12, 16, 12, 12) + layout.setSpacing(6) + + self._progress_bar = QProgressBar() + self._progress_bar.setRange(0, 100) + self._progress_bar.setValue(0) + self._progress_bar.setTextVisible(True) + self._progress_bar.setStyleSheet( + f""" + QProgressBar {{ + background-color: {theme.BG_DARK}; + color: {theme.TEXT_PRIMARY}; + border: 1px solid {theme.BORDER_SUBTLE}; + border-radius: 4px; + text-align: center; + height: 18px; + }} + QProgressBar::chunk {{ + background-color: {theme.BLUE_PRIMARY}; + border-radius: 3px; + }} + """ + ) + layout.addWidget(self._progress_bar) + + self._status_label = QLabel("Idle. Pick a .tpx3 file and press Centroid.") + self._status_label.setStyleSheet(f"color: {theme.TEXT_SECONDARY}; font-size: 11px;") + layout.addWidget(self._status_label) + + return group + + def _build_grid_group(self) -> QGroupBox: + group = QGroupBox("Combinations (check to show in plot)") + group.setStyleSheet(theme.group_box_style()) + layout = QVBoxLayout(group) + layout.setContentsMargins(12, 16, 12, 12) + + self._grid = QTableWidget(0, 0) + self._grid.setEditTriggers(QAbstractItemView.EditTrigger.NoEditTriggers) + self._grid.setSelectionMode(QAbstractItemView.SelectionMode.NoSelection) + self._grid.verticalHeader().setSectionResizeMode(QHeaderView.ResizeMode.ResizeToContents) + self._grid.horizontalHeader().setSectionResizeMode(QHeaderView.ResizeMode.Stretch) + self._grid.setStyleSheet( + f""" + QTableWidget {{ + background-color: {theme.BG_DARK}; + color: {theme.TEXT_PRIMARY}; + gridline-color: {theme.BORDER_SUBTLE}; + border: 1px solid {theme.BORDER_SUBTLE}; + }} + QHeaderView::section {{ + background-color: {theme.BG_PANEL}; + color: {theme.TEXT_PRIMARY}; + padding: 4px; + border: none; + font-weight: bold; + }} + """ + ) + layout.addWidget(self._grid) + + return group + + def _build_plot_group(self) -> QGroupBox: + group = QGroupBox("Summary plot") + group.setStyleSheet(theme.group_box_style()) + layout = QVBoxLayout(group) + layout.setContentsMargins(12, 16, 12, 12) + layout.setSpacing(6) + + toolbar = QHBoxLayout() + self._waterfall_btn = QPushButton("Waterfall") + self._waterfall_btn.setCheckable(True) + self._waterfall_btn.setStyleSheet(theme.checkable_button_style()) + self._waterfall_btn.setToolTip( + "Offset each visible curve vertically by a constant step\n" + f"(visible range / {_WATERFALL_DIVISIONS}) so overlapping histograms separate." + ) + self._waterfall_btn.toggled.connect(lambda _checked: self._refresh_plot()) + toolbar.addWidget(self._waterfall_btn) + toolbar.addStretch(1) + layout.addLayout(toolbar) + + self._plot = pg.PlotWidget() + self._plot.setBackground(theme.BG_DARK) + plot_item = self._plot.getPlotItem() + plot_item.showGrid(x=True, y=True, alpha=0.2) + plot_item.setLabel("bottom", "y (pixel / cluster centroid — dispersive axis)") + plot_item.setLabel("left", "counts") + self._legend = plot_item.addLegend(offset=(-10, 10)) + layout.addWidget(self._plot, stretch=1) + + return group + + # ------------------------------------------------------------------ + # Input handling + # ------------------------------------------------------------------ + + def _browse_file(self) -> None: + current = self._file_input.text() or str(Path.home()) + start_dir = current if Path(current).is_dir() else str(Path(current).parent) + file_path, _ = QFileDialog.getOpenFileName( + self, "Select TPX3 File", start_dir, "TPX3 Files (*.tpx3);;All Files (*)" + ) + if file_path: + self._file_input.setText(file_path) + + def _parse_eps_lists(self) -> Optional[Tuple[List[int], List[str]]]: + """Validate eps-s / eps-t via the centroider backend; show errors inline. + + The eps-t field accepts plain integers (e.g. ``20,100,500``); the ``ns`` + suffix is appended here before validation so the user never has to type it. + """ + try: + api = import_centroider_api() + except Exception as exc: # noqa: BLE001 + QMessageBox.critical(self, "Centroider unavailable", f"Could not load centroider backend:\n{exc}") + return None + try: + eps_s = api._normalize_eps_s(self._eps_s_input.text()) + # Append "ns" to each token so the user only enters bare integers. + raw_t = self._eps_t_input.text() + tokens_with_ns = ",".join( + t.strip() + "ns" if t.strip() and not t.strip().endswith("ns") else t.strip() + for t in raw_t.split(",") + if t.strip() + ) + eps_t = api._normalize_eps_t(tokens_with_ns) + except ValueError as exc: + QMessageBox.warning(self, "Invalid parameters", str(exc)) + return None + return eps_s, eps_t + + def _on_centroid_clicked(self) -> None: + if self._worker is not None and self._worker.isRunning(): + return + + file_path = self._file_input.text().strip() + if not file_path: + QMessageBox.warning(self, "Missing file", "Please select a .tpx3 file first.") + return + if not Path(file_path).exists(): + QMessageBox.warning(self, "File not found", f"Input file does not exist:\n{file_path}") + return + + parsed = self._parse_eps_lists() + if parsed is None: + return + eps_s_list, eps_t_list = parsed + + self._reset_outputs(eps_s_list, eps_t_list) + self._set_running(True) + + self._worker = CentroiderWorker( + input_file=file_path, + eps_s=",".join(str(s) for s in eps_s_list), + eps_t=",".join(eps_t_list), + parent=self, + ) + self._worker.progress.connect(self._on_progress) + self._worker.combo_finished.connect(self._on_combo_finished) + self._worker.sweep_finished.connect(self._on_sweep_finished) + self._worker.error_occurred.connect(self._on_error) + self._worker.start() + + def _on_stop_clicked(self) -> None: + if self._worker is None or not self._worker.isRunning(): + return + self._stop_btn.setEnabled(False) + self._status_label.setText("Stopping — waiting for tpx3dump to exit...") + self._worker.stop() + + # ------------------------------------------------------------------ + # Output / grid setup + # ------------------------------------------------------------------ + + def _reset_outputs(self, eps_s_list: List[int], eps_t_list: List[str]) -> None: + # Clear plot state. + self._plot.clear() + if self._legend is not None: + self._legend.clear() + self._baseline = None + self._combo_curves.clear() + self._combo_order.clear() + self._cells.clear() + + # Build the 2D grid: rows = eps-s, columns = eps-t. + self._grid.clear() + self._grid.setRowCount(len(eps_s_list)) + self._grid.setColumnCount(len(eps_t_list)) + self._grid.setHorizontalHeaderLabels([f"t={t}" for t in eps_t_list]) + self._grid.setVerticalHeaderLabels([f"s={s}" for s in eps_s_list]) + + for r, eps_s in enumerate(eps_s_list): + for c, eps_t in enumerate(eps_t_list): + key = (eps_s, eps_t) + cell = _ComboCell(eps_s, eps_t, on_toggled=self._refresh_plot) + self._cells[key] = cell + self._grid.setCellWidget(r, c, cell) + + self._progress_bar.setRange(0, max(1, len(eps_s_list) * len(eps_t_list) + 1)) + self._progress_bar.setValue(0) + + def _set_running(self, running: bool) -> None: + self._centroid_btn.setEnabled(not running) + self._stop_btn.setEnabled(running) + self._file_input.setEnabled(not running) + self._eps_s_input.setEnabled(not running) + self._eps_t_input.setEnabled(not running) + self._centroid_btn.setText("Centroiding..." if running else "Centroid") + + # ------------------------------------------------------------------ + # Worker signal handlers + # ------------------------------------------------------------------ + + @Slot(object) + def _on_progress(self, event) -> None: + """A run is starting (begin event).""" + self._progress_bar.setMaximum(max(1, event.total)) + self._status_label.setText( + f"[{event.index}/{event.total}] running {event.label} | ETA {_format_eta(event.eta_seconds)}" + ) + if event.eps_s is not None and event.eps_t is not None: + cell = self._cells.get((event.eps_s, event.eps_t)) + if cell is not None: + cell.set_running() + + @Slot(object) + def _on_combo_finished(self, event) -> None: + """A run finished (status set).""" + self._progress_bar.setMaximum(max(1, event.total)) + self._progress_bar.setValue(event.index) + self._status_label.setText( + f"[{event.index}/{event.total}] {event.label}: {event.status} | ETA {_format_eta(event.eta_seconds)}" + ) + + is_baseline = event.eps_t is None or event.eps_s is None + + if not is_baseline: + cell = self._cells.get((event.eps_s, event.eps_t)) + if cell is not None: + cell.set_status(event.status, event.wall_seconds) + + if event.status in ("ok", "skipped") and event.h5_path is not None: + self._load_curve(event, is_baseline) + + @Slot(object) + def _on_sweep_finished(self, result) -> None: + self._set_running(False) + n_ok = sum(1 for r in result.results if r.status == "ok") + n_failed = sum(1 for r in result.results if r.status == "failed") + n_skipped = sum(1 for r in result.results if r.status == "skipped") + n_cancelled = sum(1 for r in result.results if r.status == "cancelled") + was_cancelled = n_cancelled > 0 or (self._worker is not None and self._worker._cancel_event.is_set()) + if was_cancelled: + msg = f"Stopped. {n_ok} ok, {n_skipped} cached, {n_failed} failed, {n_cancelled} cancelled." + else: + self._progress_bar.setValue(self._progress_bar.maximum()) + msg = f"Done. {n_ok} ok, {n_skipped} cached, {n_failed} failed. Output: {result.run_dir}" + self._status_label.setText(msg) + self._refresh_plot() + + @Slot(str) + def _on_error(self, message: str) -> None: + self._set_running(False) + self._status_label.setText(f"Error: {message}") + QMessageBox.critical(self, "Centroider error", message) + + # ------------------------------------------------------------------ + # Plot management + # ------------------------------------------------------------------ + + def _load_curve(self, event, is_baseline: bool) -> None: + try: + api = import_centroider_api() + xs, counts, label = api.load_histogram(event.h5_path) + except Exception as exc: # noqa: BLE001 + logger.warning("Failed to load histogram for %s: %s", event.h5_path, exc) + return + + xs = np.asarray(xs, dtype=float) + counts = np.asarray(counts, dtype=float) + + if is_baseline: + item = self._plot.plot([], [], name="PixelHits (before)", pen=pg.mkPen(color=theme.TEXT_SECONDARY, width=2)) + self._baseline = {"xs": xs, "counts": counts, "item": item, "label": "PixelHits (before)"} + else: + key = (event.eps_s, event.eps_t) + color = _CURVE_COLORS[len(self._combo_order) % len(_CURVE_COLORS)] + item = self._plot.plot([], [], name=label, pen=pg.mkPen(color=color, width=1)) + self._combo_curves[key] = {"xs": xs, "counts": counts, "item": item, "label": label} + self._combo_order.append(key) + + self._refresh_plot() + + def _visible_curves(self) -> List[dict]: + """Curves currently selected for display, baseline first.""" + visible: List[dict] = [] + if self._baseline is not None: + visible.append(self._baseline) + for key in self._combo_order: + curve = self._combo_curves[key] + cell = self._cells.get(key) + if cell is not None and cell.is_checked(): + visible.append(curve) + return visible + + def _refresh_plot(self) -> None: + visible = self._visible_curves() + visible_items = {id(c["item"]) for c in visible} + + # Hide everything not currently selected. + for curve in self._all_curves(): + if id(curve["item"]) not in visible_items: + curve["item"].setVisible(False) + + # Compute the per-curve waterfall offset step from the visible range. + step = 0.0 + if self._waterfall_btn.isChecked() and visible: + gmax = max(float(np.max(c["counts"])) for c in visible if c["counts"].size) + gmin = min(float(np.min(c["counts"])) for c in visible if c["counts"].size) + step = (gmax - gmin) / _WATERFALL_DIVISIONS + if step <= 0: + step = 1.0 + + for i, curve in enumerate(visible): + offset = i * step + curve["item"].setData(curve["xs"], curve["counts"] + offset) + curve["item"].setVisible(True) + + def _all_curves(self) -> List[dict]: + curves: List[dict] = [] + if self._baseline is not None: + curves.append(self._baseline) + for key in self._combo_order: + curves.append(self._combo_curves[key]) + return curves + + # ------------------------------------------------------------------ + # Lifecycle + # ------------------------------------------------------------------ + + def shutdown(self) -> None: + """Stop the worker thread and any in-flight tpx3dump process (called on app close).""" + if self._worker is not None and self._worker.isRunning(): + self._worker.stop() + # Give tpx3dump up to 5 s to die (SIGTERM + SIGKILL already handled in runner). + self._worker.wait(5000) diff --git a/src/splash_timepix/ui/engineering_tab.py b/src/splash_timepix/ui/engineering_tab.py index 0d91508..2e70269 100644 --- a/src/splash_timepix/ui/engineering_tab.py +++ b/src/splash_timepix/ui/engineering_tab.py @@ -10,18 +10,27 @@ logger = logging.getLogger(__name__) -# Map QProcess names to terminal keys (simulator shares the source panel with live-cli). -_PROCESS_TO_TERMINAL = { +# Map QProcess / LogManager source keys → terminal widget keys. +# simulator shares the source panel with live-cli (mirrors the merged UI panel). +_SOURCE_TO_TERMINAL = { "simulator": "live-cli", + "zmq-backend": "zmq-backend", + "system": "system", + "serval": "serval", + "streaming": "streaming", + "live-cli": "live-cli", + "acquisition": "acquisition", } class EngineeringTab(QWidget): - """Engineering interface tab with 2x3 terminal grid. + """Logs tab — 2×3 grid of per-subsystem terminal panels. Layout: - [Serval] [Streaming] [live-cli] + [Serval] [Streaming] [live-cli / simulator] [Acquisition] [ZMQ Backend] [System Logs] + + The merged "All Logs (Integrated)" view lives in the separate Timeline tab. """ kill_all_requested = Signal() @@ -31,6 +40,7 @@ def __init__(self, parent=None): super().__init__(parent) self._terminals: dict[str, TerminalWidget] = {} self._setup_ui() + logger.debug("EngineeringTab initialized") def _setup_ui(self): layout = QVBoxLayout(self) @@ -41,7 +51,7 @@ def _setup_ui(self): kill_all_btn = QPushButton("Kill All Processes") kill_all_btn.setStyleSheet(theme.button_style(theme.BUTTON_STOP)) - kill_all_btn.clicked.connect(self.kill_all_requested.emit) + kill_all_btn.clicked.connect(self._on_kill_all_clicked) toolbar.addWidget(kill_all_btn) clear_btn = QPushButton("Clear Logs") @@ -88,29 +98,31 @@ def _setup_ui(self): layout.addLayout(grid) + def _on_kill_all_clicked(self) -> None: + logger.warning("Kill-all signal emitted from engineering tab") + self.kill_all_requested.emit() + def _clear_all_logs(self): + logger.info("Clear all logs requested from engineering tab") for terminal in self._terminals.values(): terminal.clear() self.clear_logs_requested.emit() - def _terminal_key(self, process_name: str) -> str: - return _PROCESS_TO_TERMINAL.get(process_name, process_name) + def _terminal_key(self, source: str) -> str: + return _SOURCE_TO_TERMINAL.get(source, source) @Slot(str, str) - def append_output(self, process_name: str, text: str): - key = self._terminal_key(process_name) - if key in self._terminals: - self._terminals[key].append_text(text) - else: - self._terminals["system"].append_text(f"[{process_name}] {text}") - - @Slot(str) - def append_system_log(self, text: str): - self._terminals["system"].append_text(text) - - @Slot(str) - def append_zmq_log(self, text: str): - self._terminals["zmq-backend"].append_text(text) + def on_log_line(self, source: str, formatted_line: str) -> None: + """Receive a pre-formatted line from LogManager and route it. + + ``formatted_line`` already contains the ISO timestamp and source tag — + it is written verbatim to the per-source terminal widget and to the + integrated "All Logs" view. No additional timestamp prefix is added + by TerminalWidget (see widgets.py). + """ + key = self._terminal_key(source) + target = self._terminals.get(key, self._terminals["system"]) + target.append_text(formatted_line) @Slot(str, bool) def set_process_status(self, process_name: str, running: bool): @@ -140,6 +152,6 @@ def _update_status_summary(self): self._status_label.setText("No processes running") self._status_label.setStyleSheet(f"color: {theme.TEXT_MUTED};") - def clear_terminal(self, process_name: str): - if process_name in self._terminals: - self._terminals[process_name].clear() + def clear_terminal(self, source: str): + if source in self._terminals: + self._terminals[source].clear() diff --git a/src/splash_timepix/ui/log_manager.py b/src/splash_timepix/ui/log_manager.py new file mode 100644 index 0000000..9396f41 --- /dev/null +++ b/src/splash_timepix/ui/log_manager.py @@ -0,0 +1,315 @@ +"""Unified log manager — captures all subsystem output to dated files on disk. + +All six subsystems (Serval, Streaming, live-cli/simulator, Acquisition, +ZMQ Backend, System) funnel through a single ``LogManager`` instance. + +On-disk layout:: + + /logs/ + └── YYYY-MM-DD/ + ├── all.log # merged, time-ordered, [source]-tagged + ├── serval.log + ├── streaming.log + ├── live-cli.log + ├── acquisition.log + ├── zmq-backend.log + └── system.log + +Every line (on disk and via the ``line_emitted`` signal) uses: + + 2026-05-15T10:07:42.318-0700 [streaming] Server started on tcp://*:5657 + +Midnight rollover is automatic — ``append()`` detects a date change and +opens the new day's folder without requiring a restart. +""" + +from __future__ import annotations + +import logging +import threading +from datetime import datetime +from pathlib import Path +from typing import Optional + +from PySide6.QtCore import QObject, Signal + +logger = logging.getLogger(__name__) + +# --------------------------------------------------------------------------- +# Repo root resolution — same anchor used by ProcessManager (workers.py:313) +# --------------------------------------------------------------------------- +_MODULE_DIR = Path(__file__).parent # …/ui/ +_REPO_ROOT = _MODULE_DIR.parent.parent.parent.parent # …/splash_timepix_dev/ +_LOG_ROOT_CANDIDATE = _REPO_ROOT / "logs" + +# Fallback when package is installed as a wheel (no writeable in-tree dir). +_LOG_ROOT_FALLBACK = Path.home() / ".local" / "state" / "splash_timepix" / "logs" + +# Timestamp format for each line. +_TS_FMT = "%Y-%m-%dT%H:%M:%S" + +# Known source → file-stem mapping (anything not listed maps to its own name). +_SOURCE_TO_FILE: dict[str, str] = { + "serval": "serval", + "streaming": "streaming", + "live-cli": "live-cli", + "simulator": "live-cli", # mirrors the merged UI panel + "acquisition": "acquisition", + "zmq-backend": "zmq-backend", + "system": "system", +} + + +def _log_root() -> Path: + """Return the root logs directory, falling back to XDG state home.""" + candidate = _LOG_ROOT_CANDIDATE + try: + candidate.mkdir(parents=True, exist_ok=True) + # Verify we can actually write there. + probe = candidate / ".write_probe" + probe.touch() + probe.unlink() + return candidate + except OSError: + return _LOG_ROOT_FALLBACK + + +def _format_line(source: str, line: str) -> str: + """Return ``line`` wrapped in an ISO-8601 timestamp + source tag.""" + now = datetime.now().astimezone() + ms = f"{now.microsecond // 1000:03d}" + tz = now.strftime("%z") + return f"{now.strftime(_TS_FMT)}.{ms}{tz} [{source}] {line}" + + +# --------------------------------------------------------------------------- +# LogManager +# --------------------------------------------------------------------------- + + +class LogManager(QObject): + """Central routing point for all subsystem log output. + + Signals + ------- + line_emitted(source, formatted_line) + Emitted for every non-empty line that passes through ``append()``. + ``source`` is the raw source key (e.g. ``"streaming"``). + ``formatted_line`` already includes the ISO timestamp + source tag + and is identical to what is written to disk. + """ + + line_emitted = Signal(str, str) # (source, formatted_line) + + def __init__(self, parent: Optional[QObject] = None) -> None: + super().__init__(parent) + + self._lock = threading.Lock() + self._log_root: Optional[Path] = None + self._current_date: str = "" # "YYYY-MM-DD" + self._handles: dict[str, object] = {} # stem → file handle + self._all_handle: Optional[object] = None + + self._using_fallback = False + # Paths opened during the last _ensure_today_open call, to be logged + # outside the lock to avoid a deadlock with the logging bridge. + self._pending_open_notifications: list[Path] = [] + self._install_python_logging_bridge() + + # ------------------------------------------------------------------ + # Public API + # ------------------------------------------------------------------ + + def append(self, source: str, text: str) -> None: + """Append ``text`` (possibly multi-line) under ``source``. + + Thread-safe — may be called from QProcess callbacks or background + threads. Each call holds the lock only for the duration of the + disk writes for the lines within ``text``. + """ + stripped = text.rstrip("\n") + if not stripped: + return + + lines = [ln for ln in stripped.split("\n") if ln.strip()] + if not lines: + return + + stem = _SOURCE_TO_FILE.get(source, source) + + with self._lock: + self._ensure_today_open() + for line in lines: + formatted = _format_line(source, line) + self._write_line(stem, formatted) + # Emit outside the lock would be safer for Qt but the signal + # emission is non-blocking here (direct connection from main + # thread triggers queued delivery on UI thread). Emitting + # inside the lock ensures ordering matches disk ordering. + self.line_emitted.emit(source, formatted) + + # Drain any newly-opened file notifications outside the lock to avoid + # a re-entrancy deadlock with the Python logging bridge. + newly_opened, self._pending_open_notifications = self._pending_open_notifications, [] + for path in newly_opened: + logger.info("Log file opened: %s", path) + + def session_marker(self, message: str = "--- session marker ---") -> None: + """Write a separator line to all open files (called on Clear Logs).""" + with self._lock: + self._ensure_today_open() + formatted = _format_line("system", message) + for stem in list(self._handles.keys()): + self._write_line(stem, formatted) + self.line_emitted.emit("system", formatted) + + newly_opened, self._pending_open_notifications = self._pending_open_notifications, [] + for path in newly_opened: + logger.info("Log file opened: %s", path) + + def close(self) -> None: + """Flush and close all open file handles (call from closeEvent).""" + with self._lock: + self._close_handles() + + # ------------------------------------------------------------------ + # Internal helpers + # ------------------------------------------------------------------ + + def _ensure_today_open(self) -> None: + """Open (or reopen after midnight) today's log folder/files.""" + today = datetime.now().strftime("%Y-%m-%d") + if today == self._current_date and self._handles: + return + + self._close_handles() + self._current_date = today + + # Resolve log root once; cache it. + if self._log_root is None: + self._log_root = _log_root() + using_fallback = self._log_root != _LOG_ROOT_CANDIDATE + if using_fallback and not self._using_fallback: + self._using_fallback = True + logger.warning( + "logs/ not writeable at %s; falling back to %s", + _LOG_ROOT_CANDIDATE, + self._log_root, + ) + + day_dir = self._log_root / today + try: + day_dir.mkdir(parents=True, exist_ok=True) + except OSError as exc: + logger.error("Cannot create log dir %s: %s", day_dir, exc) + return + + # Open per-source files (append mode so multiple sessions per day + # accumulate in the same file rather than overwriting). + stems = set(_SOURCE_TO_FILE.values()) + for stem in stems: + path = day_dir / f"{stem}.log" + try: + self._handles[stem] = open(path, "a", encoding="utf-8", buffering=1) # noqa: SIM115 + self._pending_open_notifications.append(path) + except OSError as exc: + logger.error("Cannot open log file %s: %s", path, exc) + + # The merged "all.log" is handled separately so we can keep it open + # alongside the per-source handles. + all_path = day_dir / "all.log" + try: + self._all_handle = open(all_path, "a", encoding="utf-8", buffering=1) # noqa: SIM115 + self._pending_open_notifications.append(all_path) + except OSError as exc: + logger.error("Cannot open all.log at %s: %s", all_path, exc) + self._all_handle = None + + def _write_line(self, stem: str, formatted: str) -> None: + """Write *formatted* to the per-source file and to all.log.""" + line_nl = formatted + "\n" + fh = self._handles.get(stem) + if fh is not None: + try: + fh.write(line_nl) + except OSError as exc: + logger.error("Log write failed (%s): %s", stem, exc) + + if self._all_handle is not None: + try: + self._all_handle.write(line_nl) + except OSError as exc: + logger.error("Log write failed (all.log): %s", exc) + + def _close_handles(self) -> None: + for fh in self._handles.values(): + try: + fh.flush() + fh.close() + except OSError: + pass + self._handles.clear() + + if self._all_handle is not None: + try: + self._all_handle.flush() + self._all_handle.close() + except OSError: + pass + self._all_handle = None + + # ------------------------------------------------------------------ + # Python logging bridge + # ------------------------------------------------------------------ + + def _install_python_logging_bridge(self) -> None: + """Attach a Handler to the root logger that feeds into this manager.""" + handler = _QtLoggingHandler(self) + handler.setLevel(logging.DEBUG) + # Avoid adding duplicates if LogManager is ever re-instantiated. + root = logging.getLogger() + for existing in list(root.handlers): + if isinstance(existing, _QtLoggingHandler): + root.removeHandler(existing) + root.addHandler(handler) + + +# --------------------------------------------------------------------------- +# Python logging → LogManager bridge +# --------------------------------------------------------------------------- + + +class _QtLoggingHandler(logging.Handler): + """Logging handler that routes Python log records into a ``LogManager``. + + Records from UI-internal loggers (``main``, ``workers``, ``operator_tab``, + etc.) are written to ``system.log`` so they appear in the System Logs + terminal and in ``all.log``. + """ + + # Avoid re-entrancy: if LogManager.append itself triggers a log record + # (e.g. an OSError caught with logger.error) we must not recurse. + _in_emit = threading.local() + + def __init__(self, manager: LogManager) -> None: + super().__init__() + self._manager = manager + fmt = logging.Formatter("%(name)s - %(levelname)s - %(message)s") + self.setFormatter(fmt) + + def emit(self, record: logging.LogRecord) -> None: + if getattr(self._in_emit, "active", False): + return + # INFO/DEBUG from this package are operational chatter (e.g. "Log file + # opened" after append). Routing them back into LogManager pollutes + # per-source logs and breaks line-count tests; WARNING+ still surface. + if record.name == __name__ and record.levelno < logging.WARNING: + return + self._in_emit.active = True + try: + msg = self.format(record) + self._manager.append("system", msg) + except Exception: # noqa: BLE001 + self.handleError(record) + finally: + self._in_emit.active = False diff --git a/src/splash_timepix/ui/main.py b/src/splash_timepix/ui/main.py index 6274d0c..7083c9a 100644 --- a/src/splash_timepix/ui/main.py +++ b/src/splash_timepix/ui/main.py @@ -4,19 +4,27 @@ """ import logging +import signal import sys from pathlib import Path from typing import Optional import numpy as np -from PySide6.QtCore import QTimer, Slot +import PySide6 +from PySide6.QtCore import QTimer, Slot, qVersion +from PySide6.QtGui import QIcon from PySide6.QtWidgets import QApplication, QMainWindow, QMessageBox, QStatusBar, QTabWidget, QVBoxLayout, QWidget from splash_timepix.serval_client import ServalClient -from . import theme +from . import single_instance, theme +from .alignment_tab import AlignmentTab +from .centroider_tab import CentroiderTab from .engineering_tab import EngineeringTab +from .log_manager import LogManager from .operator_tab import OperatorTab +from .preferences import load_operator_preferences, save_operator_preferences +from .timeline_tab import TimelineTab from .workers import ( FlushData, HeartbeatMonitorWorker, @@ -38,7 +46,7 @@ def __init__(self, autostart_serval: bool = False): super().__init__() self.setWindowTitle("TimePix3 Acquisition") - self.setMinimumSize(1200, 800) + self.setMinimumSize(1333, 1000) # State self._acquiring = False @@ -55,6 +63,8 @@ def __init__(self, autostart_serval: bool = False): self._waiting_for_streaming_ready = False self._serval_stdout_tail = "" + self._log_manager = LogManager(self) + self._setup_ui() self._setup_workers() if autostart_serval: @@ -68,19 +78,41 @@ def _setup_ui(self): layout = QVBoxLayout(central) layout.setContentsMargins(8, 8, 8, 8) - # Tab widget + # Tab widget. Order: Alignment (default landing tab — beam alignment is + # the first thing operators do each session) → Operator → Engineering. self._tabs = QTabWidget() + # Alignment tab (first / default). Auto-stopped when the user switches + # to the Operator tab — see _on_tab_changed below. + self._alignment_tab = AlignmentTab() + self._alignment_tab.start_requested.connect(self._on_start_requested) + self._alignment_tab.stop_requested.connect(self._on_stop_requested) + self._alignment_idx = self._tabs.addTab(self._alignment_tab, "Alignment") + # Operator tab self._operator_tab = OperatorTab() self._operator_tab.start_requested.connect(self._on_start_requested) self._operator_tab.stop_requested.connect(self._on_stop_requested) - self._tabs.addTab(self._operator_tab, "Operator") + self._operator_idx = self._tabs.addTab(self._operator_tab, "Operator") - # Engineering tab + # Logs tab (formerly Engineering) — viewing diagnostics during a live + # alignment run is explicitly allowed; this tab does NOT trigger auto-stop. self._engineering_tab = EngineeringTab() self._engineering_tab.kill_all_requested.connect(self._on_kill_all) - self._tabs.addTab(self._engineering_tab, "Engineering") + self._engineering_tab.clear_logs_requested.connect(self._log_manager.session_marker) + self._engineering_idx = self._tabs.addTab(self._engineering_tab, "Logs") + + # Timeline tab — integrated, time-ordered view of all subsystem logs. + self._timeline_tab = TimelineTab() + self._timeline_idx = self._tabs.addTab(self._timeline_tab, "Timeline") + + # Centroider tab — offline parameter sweep over clustering eps-s / eps-t. + # Self-contained (owns its own worker); does NOT trigger alignment auto-stop. + self._centroider_tab = CentroiderTab() + self._centroider_idx = self._tabs.addTab(self._centroider_tab, "Centroider") + + # Auto-stop alignment on Operator-tab entry. See _on_tab_changed. + self._tabs.currentChanged.connect(self._on_tab_changed) layout.addWidget(self._tabs) @@ -95,7 +127,13 @@ def _setup_workers(self): self._process_manager = ProcessManager(self) self._process_manager.process_started.connect(self._on_process_started) self._process_manager.process_stopped.connect(self._on_process_stopped) + # Persist to disk first, then handle Serval tail / UI via line_emitted. + self._process_manager.process_output.connect(self._log_manager.append) self._process_manager.process_output.connect(self._on_process_output) + self._log_manager.line_emitted.connect(self._engineering_tab.on_log_line) + self._log_manager.line_emitted.connect(self._timeline_tab.on_log_line) + # Clear the Timeline view together with the Logs tab terminals. + self._engineering_tab.clear_logs_requested.connect(self._timeline_tab.clear) # Serval poller (always runs) self._serval_worker = ServalPollerWorker(poll_interval=1.0) @@ -119,7 +157,7 @@ def _setup_workers(self): self._zmq_worker.error_occurred.connect(self._on_zmq_error) self._zmq_worker.start() self._engineering_tab.set_zmq_thread_status("running") - self._engineering_tab.append_zmq_log("ZMQ subscriber thread started.") + self._log_manager.append("zmq-backend", "ZMQ subscriber thread started.") def _start_serval(self): """Start Serval server on application startup.""" @@ -129,12 +167,12 @@ def _start_serval(self): return logger.info("Starting Serval server...") - self._engineering_tab.append_system_log("Starting Serval server...") + self._log_manager.append("system", "Starting Serval server...") if self._process_manager.start_serval(): self._status_bar.showMessage("Starting Serval server...") else: - self._engineering_tab.append_system_log("WARNING: Failed to start Serval - check if JAR exists") + self._log_manager.append("system", "WARNING: Failed to start Serval - check if JAR exists") QMessageBox.warning( self, "Serval Error", @@ -176,27 +214,47 @@ def _on_start_requested(self, mode: str, params: dict): except Exception: self._current_frame_number = None - tdc_freq = params["tdc_frequency"] - tdc_channel = params["tdc_channel"] - tdc_edge = params["tdc_edge"] - callback_batch_size = params.get("callback_batch_size", 10_000) - duration = params["duration"] + # Alignment uses a curated subset of params; fall back to operator-tab + # widget defaults for fields it doesn't carry (TDC freq/channel/edge, + # parse batch, n_bins). The streaming server ignores these in alignment + # mode but still wants well-formed CLI args. + if mode == "alignment": + op_params = self._operator_tab._get_params() + tdc_freq = op_params["tdc_frequency"] + tdc_channel = op_params["tdc_channel"] + tdc_edge = op_params["tdc_edge"] + callback_batch_size = op_params.get("callback_batch_size", 10_000) + n_bins = op_params.get("n_bins", 350) + duration = params["duration"] + alignment_rate_hz = float(params.get("alignment_rate_hz", 30.0)) + else: + tdc_freq = params["tdc_frequency"] + tdc_channel = params["tdc_channel"] + tdc_edge = params["tdc_edge"] + callback_batch_size = params.get("callback_batch_size", 10_000) + n_bins = params.get("n_bins", 350) + duration = params["duration"] + alignment_rate_hz = 30.0 logger.info( f"Starting {mode}: TDC={tdc_freq}Hz, ch={tdc_channel}, edge={tdc_edge}, " f"callback_batch_size={callback_batch_size}, duration={duration}s" ) - self._engineering_tab.append_system_log( - f"Starting {mode}: TDC={tdc_freq}Hz, parse_batch={callback_batch_size}, duration={duration}s" + self._log_manager.append( + "system", + f"Starting {mode}: TDC={tdc_freq}Hz, parse_batch={callback_batch_size}, duration={duration}s", ) - # Start streaming server (needed for all modes) + # Start streaming server (needed for all modes; alignment passes extra flags). if not self._process_manager.start_streaming_server( tdc_freq, tdc_channel, tdc_edge, callback_batch_size=callback_batch_size, + n_bins=n_bins, exit_on_disconnect=True, + alignment=(mode == "alignment"), + alignment_rate_hz=alignment_rate_hz, ): QMessageBox.warning(self, "Error", "Failed to start streaming server") return @@ -233,7 +291,7 @@ def _check_server_ready(self): elif self._ready_check_count > 60: # 30 second timeout self._waiting_for_streaming_ready = False self._stop_ready_check_timer() - self._engineering_tab.append_system_log("WARNING: Timeout waiting for server ready") + self._log_manager.append("system", "WARNING: Timeout waiting for server ready") QMessageBox.warning(self, "Timeout", "Streaming server did not become ready in time") self._process_manager.stop_process("streaming") @@ -241,11 +299,11 @@ def _continue_startup(self): """Continue startup sequence after server is ready.""" mode, params = self._start_params - self._engineering_tab.append_system_log("Streaming server ready") + self._log_manager.append("system", "Streaming server ready") if mode == "simulator": # Simulator mode: start simulator CLI (no Serval needed) - self._engineering_tab.append_system_log("Starting simulator...") + self._log_manager.append("system", "Starting simulator...") self._status_bar.showMessage("Starting simulator...") if not self._process_manager.start_simulator( @@ -259,11 +317,12 @@ def _continue_startup(self): self._acquiring = True self._operator_tab.set_acquiring(True) + self._alignment_tab.set_acquiring(True) elif mode == "replay": # Replay mode: start live-cli with source file (no Serval needed) replay_file = params.get("replay_file", "") - self._engineering_tab.append_system_log(f"Starting replay: {Path(replay_file).name}") + self._log_manager.append("system", f"Starting replay: {Path(replay_file).name}") self._status_bar.showMessage("Replaying file...") if not self._process_manager.start_live_cli(replay_file=replay_file): @@ -273,10 +332,25 @@ def _continue_startup(self): self._acquiring = True self._operator_tab.set_acquiring(True) + self._alignment_tab.set_acquiring(True) + + elif mode == "alignment": + # Alignment mode: identical pipeline to preview (live-cli + acq.py + # --preview), but the streaming server is already running with + # --alignment. acq.py kicks Serval into a continuous acquisition; + # without it the detector pushes nothing through live-cli. Output + # dir is unused in --preview mode but acq.py expects the flag, so + # pass an empty directory placeholder via the duration-only path. + self._log_manager.append("system", "Starting alignment live-cli...") + self._status_bar.showMessage("Starting live-cli (alignment)...") + QTimer.singleShot( + 1000, + lambda: self._start_live_cli_and_acq(params["duration"], "", True), + ) else: # Start/Preview mode: need live-cli and acquisition - self._engineering_tab.append_system_log("Starting live-cli...") + self._log_manager.append("system", "Starting live-cli...") self._status_bar.showMessage("Starting live-cli...") QTimer.singleShot( 1000, @@ -296,7 +370,7 @@ def _start_live_cli_and_acq(self, duration: int, output_dir: str, preview: bool) def _start_acquisition(self, duration: int, output_dir: str, preview: bool): """Start the acquisition script.""" - self._engineering_tab.append_system_log("Starting acquisition...") + self._log_manager.append("system", "Starting acquisition...") self._status_bar.showMessage("Acquiring..." if not preview else "Preview mode...") if not self._process_manager.start_acquisition(duration, output_dir, preview): @@ -306,6 +380,7 @@ def _start_acquisition(self, duration: int, output_dir: str, preview: bool): self._acquiring = True self._operator_tab.set_acquiring(True) + self._alignment_tab.set_acquiring(True) @Slot() def _on_stop_requested(self): @@ -314,7 +389,7 @@ def _on_stop_requested(self): return logger.info("Stop requested") - self._engineering_tab.append_system_log("Stop requested...") + self._log_manager.append("system", "Stop requested...") self._status_bar.showMessage("Stopping...") mode = getattr(self, "_current_mode", "start") @@ -324,78 +399,131 @@ def _on_stop_requested(self): self._save_on_stop(mode) if mode in ("simulator", "replay"): - # For simulator/replay, just kill the processes directly - self._engineering_tab.append_system_log(f"Stopping {mode} processes...") - self._process_manager.stop_process("simulator" if mode == "simulator" else "live-cli") - self._process_manager.stop_process("streaming") - self._acquiring = False - self._operator_tab.set_acquiring(False) - self._status_bar.showMessage(f"{mode.capitalize()} stopped") + # Kill only the data source (simulator or live-cli). The streaming + # server was launched with --exit-on-disconnect: once it detects the + # TCP disconnect it will do the final flush, publish the ZMQ stop + # message, and exit on its own. _on_acquisition_complete() fires + # when the "streaming" process exits (see _on_process_stopped). + proc_name = "simulator" if mode == "simulator" else "live-cli" + self._log_manager.append("system", f"Stopping {proc_name}...") + self._process_manager.stop_process(proc_name) + self._status_bar.showMessage("Waiting for streaming server to finish...") + # Safety net: force-kill streaming after 6 s if it has not exited. + QTimer.singleShot(6000, self._force_stop_streaming_if_still_running) else: # For real acquisition, call stop.py via Serval self._run_stop_script() def _save_on_stop(self, mode: str): """Save average data when stopping acquisition or replay.""" - # For replay, save next to the source file - # For acquisition, use the configured output dir + # For replay, save next to the source file using the replay file's stem. + # For acquisition, use the persistent scan counter as the prefix so every + # scan gets a unique, incrementing number independent of Serval's naming. + newest_tpx3: Optional[Path] = None + if mode == "replay" and self._current_replay_file: output_dir = str(Path(self._current_replay_file).parent) filename_base = Path(self._current_replay_file).stem + scan_counter = None # counter is not used / not incremented for replay else: output_dir = self._current_output_dir if not output_dir: output_dir = str(Path.home() / "Desktop") - self._engineering_tab.append_system_log(f"No output dir set, using {output_dir}") + self._log_manager.append("system", f"No output dir set, using {output_dir}") - # Find the newest/latest .tpx3 file in the output directory + # Find the newest/latest .tpx3 file in the output directory so we + # can rename it to match the derived output files. output_path = Path(output_dir) tpx3_files = list(output_path.glob("*.tpx3")) if tpx3_files: newest_tpx3 = max(tpx3_files, key=lambda f: f.stat().st_mtime) - filename_base = newest_tpx3.stem - self._engineering_tab.append_system_log(f"Found newest tpx3: {newest_tpx3}") + self._log_manager.append("system", f"Found newest tpx3: {newest_tpx3}") else: - self._engineering_tab.append_system_log( - "WARNING: No .tpx3 files found in output directory, skipping save" - ) + self._log_manager.append("system", "WARNING: No .tpx3 files found in output directory, skipping save") return - self._engineering_tab.append_system_log(f"Saving to {output_dir} as {filename_base}...") + # Use the persistent scan counter as filename_base so the prefix + # increments with every acquisition regardless of Serval's naming. + prefs = load_operator_preferences() + scan_counter = prefs.get("scan_counter", 1) + filename_base = f"{scan_counter:06d}" - png_path, csv_path, json_path = self._operator_tab.save_average_data(output_dir, filename_base) + self._log_manager.append("system", f"Saving to {output_dir} as {filename_base}...") + + png_path, csv_path, energy_path, time_path, json_path, slug = self._operator_tab.save_average_data( + output_dir, filename_base + ) if png_path: - self._engineering_tab.append_system_log(f"Saved: {png_path}") + self._log_manager.append("system", f"Saved: {png_path}") if csv_path: - self._engineering_tab.append_system_log(f"Saved: {csv_path}") + self._log_manager.append("system", f"Saved: {csv_path}") + if energy_path: + self._log_manager.append("system", f"Saved: {energy_path}") + if time_path: + self._log_manager.append("system", f"Saved: {time_path}") if json_path: - self._engineering_tab.append_system_log(f"Saved: {json_path}") - if not png_path and not csv_path and not json_path: - self._engineering_tab.append_system_log("No data to save") + self._log_manager.append("system", f"Saved: {json_path}") + if not any([png_path, csv_path, energy_path, time_path, json_path]): + self._log_manager.append("system", "No data to save") + + # Rename the .tpx3 source file to share the same slug as the derived files. + # Only applies to live acquisition (newest_tpx3 is None in replay mode). + if newest_tpx3 and slug and newest_tpx3.exists(): + try: + new_tpx3 = newest_tpx3.with_name(f"{filename_base}_{slug}.tpx3") + newest_tpx3.rename(new_tpx3) + self._log_manager.append("system", f"Renamed: {newest_tpx3.name} → {new_tpx3.name}") + logger.info("Renamed .tpx3: %s → %s", newest_tpx3.name, new_tpx3.name) + except OSError as e: + logger.warning("Could not rename .tpx3 file %s: %s", newest_tpx3, e) + self._log_manager.append("system", f"WARNING: Could not rename {newest_tpx3.name}: {e}") + + # Increment the scan counter only after a successful acquisition save + # (at least one output file was written). Replay saves do not consume + # a counter slot. + if scan_counter is not None and any([png_path, csv_path, energy_path, time_path, json_path]): + save_operator_preferences({"scan_counter": scan_counter + 1}) + logger.info("Scan counter advanced to %d", scan_counter + 1) + + def _force_stop_streaming_if_still_running(self): + """Safety net: force-kill streaming server if it has not exited on its own. + + Called ~6 s after the data source (simulator / live-cli) was stopped. + Under normal operation the streaming server exits well within that window + via --exit-on-disconnect; this only fires if something went wrong. + """ + if self._process_manager.is_running("streaming"): + logger.warning("Streaming server still running after timeout, force stopping") + self._log_manager.append( + "system", "WARNING: Streaming server did not exit after disconnect, force stopping..." + ) + self._process_manager.stop_process("streaming") + # _on_process_stopped("streaming") will fire and call _on_acquisition_complete. def _run_stop_script(self): """Stop acquisition via Serval.""" try: client = ServalClient() client.stop_acquisition() - self._engineering_tab.append_system_log("Stop command sent successfully") + self._log_manager.append("system", "Stop command sent successfully") except Exception as e: - self._engineering_tab.append_system_log(f"Stop failed: {e}, killing processes...") + self._log_manager.append("system", f"Stop failed: {e}, killing processes...") self._process_manager.stop_all() @Slot() def _on_kill_all(self): """Handle kill all request from engineering tab.""" logger.info("Killing all processes") - self._engineering_tab.append_system_log("Killing all processes...") + self._log_manager.append("system", "Killing all processes...") self._waiting_for_streaming_ready = False self._stop_ready_check_timer() self._process_manager.stop_all() self._acquiring = False self._operator_tab.set_acquiring(False) + self._alignment_tab.set_acquiring(False) self._status_bar.showMessage("All processes stopped") @Slot(str) @@ -403,19 +531,21 @@ def _on_process_started(self, name: str): """Handle process started signal.""" logger.info(f"Process started: {name}") self._engineering_tab.set_process_status(name, True) - self._engineering_tab.append_output(name, "--- Process started ---\n") + self._log_manager.append(name, "--- Process started ---") if name == "serval": self._serval_stdout_tail = "" self._operator_tab.on_serval_process_running(True) + self._alignment_tab.on_serval_process_running(True) @Slot(str, int) def _on_process_stopped(self, name: str, exit_code: int): """Handle process stopped signal.""" logger.info(f"Process stopped: {name} (exit code: {exit_code})") self._engineering_tab.set_process_status(name, False) - self._engineering_tab.append_output(name, f"\n--- Process exited (code: {exit_code}) ---\n") + self._log_manager.append(name, f"--- Process exited (code: {exit_code}) ---") if name == "serval": self._operator_tab.on_serval_process_running(False) + self._alignment_tab.on_serval_process_running(False) mode = getattr(self, "_current_mode", "start") @@ -423,72 +553,122 @@ def _on_process_stopped(self, name: str, exit_code: int): if self._acquiring: if name == "acquisition": self._on_acquisition_complete() - elif name == "simulator" and mode == "simulator": - self._on_acquisition_complete() - elif name == "live-cli" and mode == "replay": + elif name == "streaming" and mode in ("simulator", "replay"): + # Streaming server exited (either naturally via --exit-on-disconnect + # after the data source disconnected, or via the safety-net force-kill). + # This is the authoritative completion signal for these modes: by the + # time streaming exits, the final flush and ZMQ stop have been sent. self._on_acquisition_complete() @Slot(str, str) def _on_process_output(self, name: str, text: str): - """Handle process output signal.""" - self._engineering_tab.append_output(name, text) + """Maintain Serval stdout tail for chip-temp detection. + + Persistence and UI rendering are handled upstream by LogManager + (connected to the same process_output signal before this slot). + """ if name == "serval": self._serval_stdout_tail = (self._serval_stdout_tail + text)[-8192:] if "chip temps:" in self._serval_stdout_tail.lower(): self._operator_tab.on_serval_chip_temps_line_seen() + self._alignment_tab.on_serval_chip_temps_line_seen() def _on_acquisition_complete(self): """Handle acquisition completion - save average data.""" self._acquiring = False self._operator_tab.set_acquiring(False) + self._alignment_tab.set_acquiring(False) mode = getattr(self, "_current_mode", "start") - if mode in ("preview", "simulator", "replay"): + if mode in ("preview", "simulator", "replay", "alignment"): self._status_bar.showMessage(f"{mode.capitalize()} complete") - self._engineering_tab.append_system_log(f"{mode.capitalize()} complete") + self._log_manager.append("system", f"{mode.capitalize()} complete") return # Save average data for real acquisitions self._save_on_stop(mode) self._status_bar.showMessage("Acquisition complete") - self._engineering_tab.append_system_log("Acquisition complete") + self._log_manager.append("system", "Acquisition complete") @Slot(object) def _on_flush_received(self, flush_data: FlushData): - """Handle incoming flush from ZMQ worker.""" - self._operator_tab.on_flush_received(flush_data) + """Route incoming flushes by mode so each tab only sees its own shape. + + Alignment flushes are (X, Y, 1) uint32 arrays which would crash the + operator tab's heatmap math; timing flushes are (X, n_bins) or + (X, Y, n_bins) which the alignment tab cannot render. The metadata's + ``mode`` field (added in TimePixStart and the static metadata in + ``app.py``) is the authoritative router. Defaults to "timing" so any + old streaming server / pre-mode-field message routes to the operator + tab as before. + """ + meta = flush_data.metadata + mode = meta.get("mode", "timing") + if mode == "alignment": + self._alignment_tab.on_flush_received(flush_data) + else: + self._operator_tab.on_flush_received(flush_data) # Log to engineering tab - meta = flush_data.metadata flush_num = meta.get("flush_number", "?") cycles = meta.get("cycles_in_flush", "?") - self._engineering_tab.append_zmq_log( - f"Flush #{flush_num}: {cycles} cycles, {np.sum(flush_data.array):.2e} counts" + self._log_manager.append( + "zmq-backend", + f"[{mode}] Flush #{flush_num}: {cycles} cycles, {np.sum(flush_data.array):.2e} counts", ) + @Slot(int) + def _on_tab_changed(self, new_index: int) -> None: + """Auto-stop alignment when the user switches to the Operator tab. + + Engineering tab is intentionally exempt — viewing diagnostics during a + live alignment run is allowed (no auto-stop fires). The auto-stop is + silent (status-bar + engineering-log line) — no confirmation dialog. + """ + _tab_names = { + self._alignment_idx: "Alignment", + self._operator_idx: "Operator", + self._engineering_idx: "Logs", + self._timeline_idx: "Timeline", + } + tab_name = _tab_names.get(new_index, f"Tab {new_index}") + logger.debug("Tab changed to: %s", tab_name) + + if new_index != self._operator_idx: + return + if not self._acquiring: + return + if getattr(self, "_current_mode", None) != "alignment": + return + msg = "Alignment auto-stopped: switched to Operator" + logger.info(msg) + self._log_manager.append("system", msg) + self._status_bar.showMessage(msg) + self._on_stop_requested() + @Slot(bool) def _on_zmq_connection_changed(self, connected: bool): """Handle ZMQ connection state change for engineering tab.""" if connected: - self._engineering_tab.append_zmq_log("Receiving data from streaming server") + self._log_manager.append("zmq-backend", "Receiving data from streaming server") else: - self._engineering_tab.append_zmq_log("Not receiving data") + self._log_manager.append("zmq-backend", "Not receiving data") @Slot(bool) def _on_serval_connection_changed(self, connected: bool): """Handle Serval connection state change.""" if connected: - self._engineering_tab.append_system_log("Connected to Serval") + self._log_manager.append("system", "Connected to Serval") else: - self._engineering_tab.append_system_log("Disconnected from Serval") + self._log_manager.append("system", "Disconnected from Serval") @Slot(bool) def _on_heartbeat_connection_changed(self, connected: bool): """Handle heartbeat connection state change.""" if connected: - self._engineering_tab.append_system_log("Heartbeat connected") + self._log_manager.append("system", "Heartbeat connected") @Slot(object) def _on_heartbeat_status_for_state(self, status: HeartbeatStatus): @@ -498,7 +678,7 @@ def _on_heartbeat_status_for_state(self, status: HeartbeatStatus): @Slot(str) def _on_zmq_error(self, error: str): """Handle ZMQ error.""" - self._engineering_tab.append_zmq_log(f"Error: {error}") + self._log_manager.append("zmq-backend", f"Error: {error}") def closeEvent(self, event): """Handle window close - cleanup workers and processes.""" @@ -520,6 +700,20 @@ def closeEvent(self, event): event.ignore() return + # Save operator + alignment preferences before tearing down workers, so + # a save exception cannot leave workers/processes running. Both saves + # write into the same JSON file via a merge-on-write strategy in + # preferences.save_operator_preferences, so the order does not matter. + # Failures here must never block quit — log and continue. + try: + self._operator_tab.save_operator_preferences() + except Exception: + logger.exception("Failed to save operator preferences") + try: + self._alignment_tab.save_alignment_preferences() + except Exception: + logger.exception("Failed to save alignment preferences") + logger.info("Closing application...") self._waiting_for_streaming_ready = False @@ -539,19 +733,227 @@ def closeEvent(self, event): self._serval_worker.stop() self._serval_worker.wait(2000) + # Stop the centroider sweep worker if one is running. + try: + self._centroider_tab.shutdown() + except Exception: + logger.exception("Failed to shut down centroider tab") + # Stop all processes (including Serval) if self._process_manager: self._process_manager.stop_all() + self._log_manager.close() event.accept() +def _show_already_running_dialog(holder_pid: Optional[int]) -> str: + """Show the second-instance dialog; return the chosen action. + + Returns one of: + - ``"kill_and_reopen"`` — terminate the other instance, then start a + fresh UI here. + - ``"kill"`` — terminate the other instance and exit this process. + - ``"cancel"`` — do nothing; exit this process. + + The Kill / Kill & Reopen buttons are hidden when ``holder_pid`` + cannot be verified as our app via psutil, which protects against + PID reuse and cross-user signaling. In that case only Cancel is + offered. + """ + msg = QMessageBox() + msg.setIcon(QMessageBox.Icon.Warning) + msg.setWindowTitle("TimePix UI already running") + msg.setText("Another TimePix UI session is already running for this user.") + + can_verify = holder_pid is not None and single_instance.is_other_instance_alive(holder_pid) + + if can_verify: + msg.setInformativeText( + f"Close that window first, or click Kill to terminate it (pid {holder_pid}).\n\n" "Cancel to do nothing." + ) + else: + msg.setInformativeText( + "Could not verify the other process — Kill is disabled. " + "Close that window manually first.\n\n" + "Cancel to do nothing." + ) + + # ActionRole buttons are laid out in insertion order (RejectRole / + # AcceptRole would otherwise be re-arranged by QDialogButtonBox per + # platform style — e.g. AcceptRole leftmost on GNOME). We want Cancel + # always at the very left, then Kill, then Kill & Reopen on the right. + cancel_btn = msg.addButton("Cancel", QMessageBox.ButtonRole.ActionRole) + if can_verify: + kill_btn = msg.addButton("Kill", QMessageBox.ButtonRole.ActionRole) + kill_reopen_btn = msg.addButton("Kill && Reopen", QMessageBox.ButtonRole.ActionRole) + else: + kill_btn = None + kill_reopen_btn = None + msg.setDefaultButton(cancel_btn) + msg.setEscapeButton(cancel_btn) + + msg.exec() + clicked = msg.clickedButton() + if kill_reopen_btn is not None and clicked is kill_reopen_btn: + return "kill_and_reopen" + if kill_btn is not None and clicked is kill_btn: + return "kill" + return "cancel" + + +def _confirm_force_kill(pid: int) -> bool: + """Second-stage confirmation before SIGKILL escalation.""" + msg = QMessageBox() + msg.setIcon(QMessageBox.Icon.Critical) + msg.setWindowTitle("Force-kill TimePix UI?") + msg.setText(f"Process {pid} did not exit after SIGTERM.") + msg.setInformativeText( + "Force-killing skips graceful shutdown of Serval / streaming-server / " + "live-cli, which may leave them as orphaned processes. Only do this " + "if the other window is truly stuck." + ) + cancel_btn = msg.addButton("Cancel", QMessageBox.ButtonRole.RejectRole) + kill_btn = msg.addButton("Force kill", QMessageBox.ButtonRole.DestructiveRole) + msg.setDefaultButton(cancel_btn) + msg.exec() + return msg.clickedButton() is kill_btn + + +def _handle_singleton() -> bool: + """Acquire the singleton lock or run the recovery dialog flow. + + Returns ``True`` if the caller should continue starting the UI, or + ``False`` if the caller should ``sys.exit(0)`` (the user chose + Cancel, chose Kill without reopen, or a kill attempt failed). + + Must be called *before* :class:`MainWindow` is constructed (but + *after* :class:`QApplication` exists so dialog widgets can be + shown). Non-Linux platforms skip enforcement entirely. + """ + try: + single_instance.acquire_lock() + return True + except single_instance.AlreadyRunning as e: + action = _show_already_running_dialog(e.pid) + if action == "cancel" or e.pid is None: + return False + + # Both "kill" and "kill_and_reopen" terminate the other instance. + # The first instance's SIGTERM handler routes through closeEvent → + # ProcessManager.stop_all(), so its children are stopped before its + # FD closes and the kernel drops our lock. + if not single_instance.terminate_other_instance(e.pid): + # Either uid/cmdline check failed, or SIGTERM did not bring the + # process down within the timeout. Offer SIGKILL as a gated + # escalation; otherwise tell the user to retry. + if _confirm_force_kill(e.pid) and single_instance.force_kill_other_instance(e.pid): + pass # fall through + else: + _show_terminate_failed_dialog(e.pid) + return False + + if action == "kill": + # User asked us to terminate the other instance and exit; do + # not start a fresh UI here. They can relaunch when ready. + return False + + # "kill_and_reopen": retry the flock once and continue starting the UI. + try: + single_instance.acquire_lock() + return True + except single_instance.AlreadyRunning: + _show_retry_dialog() + return False + + +def _show_terminate_failed_dialog(pid: int) -> None: + msg = QMessageBox() + msg.setIcon(QMessageBox.Icon.Warning) + msg.setWindowTitle("Could not terminate other instance") + msg.setText(f"Failed to terminate the existing TimePix UI (pid {pid}).") + msg.setInformativeText("Close that window manually, then relaunch.") + msg.exec() + + +def _show_retry_dialog() -> None: + msg = QMessageBox() + msg.setIcon(QMessageBox.Icon.Information) + msg.setWindowTitle("Lock still held") + msg.setText("The other instance is exiting but still holds the lock.") + msg.setInformativeText("Wait a moment and relaunch.") + msg.exec() + + +def _load_app_icon() -> QIcon: + """Return the bundled window/taskbar icon, or an empty QIcon if missing. + + The asset lives next to this module so editable installs and built wheels + (via the ``[tool.setuptools.package-data]`` glob in ``pyproject.toml``) + resolve the same path. A missing file is a soft failure — log and fall back + to Qt's default rather than block UI startup. + """ + icon_path = Path(__file__).resolve().parent / "assets" / "icon.png" + if not icon_path.is_file(): + logger.warning("App icon not found at %s; falling back to Qt default", icon_path) + return QIcon() + return QIcon(str(icon_path)) + + def main(): """Application entry point.""" autostart_serval = "--autostart-serval" in sys.argv app = QApplication(sys.argv) app.setStyle("Fusion") + # Wayland compositors (GNOME, KDE Plasma) use the desktop-file name as the + # window's app_id for grouping in the taskbar / dock and for matching + # against an installed `splash_timepix.desktop`. Without it the window + # often shows up as a generic "Wayland Application". Harmless on X11. + app.setDesktopFileName("splash_timepix") + # Without this, X11 derives WM_CLASS from argv[0] — which becomes + # ".../main.py" under `python -m splash_timepix.ui.main`. Panels read + # WM_CLASS for taskbar grouping and the entry's "application name" + # tooltip, so the launcher would show "main.py" instead of our app. + # On Wayland this is also used as a fallback for app_id when + # setDesktopFileName is unset (it's not, but keeping both is cheap). + app.setApplicationName("splash_timepix") + # Human-readable label. Qt appends this to top-level window titles when + # the title does not already include it, which is what most Linux + # taskbars / Alt-Tab switchers display as the entry's tooltip. + app.setApplicationDisplayName("Splash TimePix") + # setWindowIcon on the QApplication propagates to every top-level widget + # constructed afterward, so MainWindow inherits it without an explicit + # call. Must run *before* MainWindow is built. + app.setWindowIcon(_load_app_icon()) + + # Route SIGTERM through Qt's event loop so MainWindow.closeEvent runs and + # ProcessManager.stop_all() takes the children with us. Without this the + # second-instance "Kill" path would orphan Serval/streaming-server/live-cli. + signal.signal(signal.SIGTERM, lambda *_: app.quit()) + + # Python signal handlers only run when the interpreter regains control. + # Qt's exec() blocks in C++, so without periodic Python invocations a + # SIGTERM may sit pending until the next user event. A no-op QTimer on a + # short interval forces Qt to call a Python slot, giving the interpreter + # a chance to dispatch pending signals. + _sig_pump = QTimer() + _sig_pump.start(200) + _sig_pump.timeout.connect(lambda: None) + + # Singleton enforcement must happen *before* constructing MainWindow so + # the second instance does not spin up workers it would immediately tear + # down. Non-Linux platforms skip enforcement (logs a warning). + if not _handle_singleton(): + sys.exit(0) + + logger.info( + "splash_timepix UI starting — Python %s, Qt %s, PySide6 %s, platform %s", + sys.version.split()[0], + qVersion(), + PySide6.__version__, + sys.platform, + ) # Apply base dark theme stylesheet app.setStyleSheet( diff --git a/src/splash_timepix/ui/operator_tab.py b/src/splash_timepix/ui/operator_tab.py index a577bd4..49a1421 100644 --- a/src/splash_timepix/ui/operator_tab.py +++ b/src/splash_timepix/ui/operator_tab.py @@ -2,8 +2,10 @@ import json import logging +import uuid +from datetime import datetime from pathlib import Path -from typing import Optional, Tuple +from typing import Optional, Tuple, Union import numpy as np from PySide6.QtCore import Qt, Signal, Slot @@ -12,6 +14,7 @@ QDoubleSpinBox, QFileDialog, QFrame, + QGridLayout, QGroupBox, QHBoxLayout, QLabel, @@ -23,8 +26,8 @@ QWidget, ) -from . import theme -from .widgets import HeatmapWidget, StatusIndicator, get_colormap +from . import preferences, theme +from .widgets import CURSOR_COLORS, HeatmapWidget, SpectrumPlotWidget, StatusIndicator, VerticalLabel, get_colormap from .workers import FlushData, HeartbeatStatus, ServalStatus logger = logging.getLogger(__name__) @@ -77,8 +80,15 @@ def __init__(self, parent=None): self._cumulative_sum: Optional[np.ndarray] = None self._total_cycles = 0 self._flush_count = 0 + self._last_avg_2d: Optional[np.ndarray] = None # most recent averaged heatmap data + self._n_energy: Optional[int] = None # number of energy pixels in current data self._setup_ui() + try: + self.load_operator_preferences() + except Exception: # pragma: no cover - defensive + logger.exception("Failed to load operator preferences; using widget defaults") + self._refresh_mode_buttons_state() def _setup_ui(self): layout = QVBoxLayout(self) @@ -152,7 +162,7 @@ def _setup_ui(self): view_layout.setContentsMargins(8, 6, 8, 6) view_layout.setSpacing(8) - cmap_label = QLabel("Colormap:") + cmap_label = QLabel("Colormap") cmap_label.setStyleSheet(f"color: {theme.TEXT_SECONDARY};") view_layout.addWidget(cmap_label) @@ -162,12 +172,63 @@ def _setup_ui(self): self._colormap_combo.currentTextChanged.connect(self._on_colormap_changed) view_layout.addWidget(self._colormap_combo) + cmap_sep = QFrame() + cmap_sep.setFrameShape(QFrame.Shape.VLine) + cmap_sep.setFixedWidth(1) + cmap_sep.setStyleSheet(f"background-color: {theme.BORDER_SUBTLE}; border: none;") + view_layout.addWidget(cmap_sep) + + self._vcursor_heatmap_btn = QPushButton("| Cursors") + self._vcursor_heatmap_btn.setCheckable(True) + self._vcursor_heatmap_btn.setChecked(True) + self._vcursor_heatmap_btn.setStyleSheet(theme.checkable_button_style()) + self._vcursor_heatmap_btn.setToolTip("Show/hide energy cursor lines in both heatmaps") + self._vcursor_heatmap_btn.toggled.connect(self._on_vcursor_heatmap_toggled) + view_layout.addWidget(self._vcursor_heatmap_btn) + view_layout.addSpacing(12) - self._reset_avg_btn = QPushButton("Reset Avg") - self._reset_avg_btn.setStyleSheet(theme.secondary_button_style()) - self._reset_avg_btn.clicked.connect(self._reset_average) - view_layout.addWidget(self._reset_avg_btn) + # Zoom button group + zoom_group = QFrame() + zoom_group.setStyleSheet( + f"QFrame {{ border: 1px solid {theme.BORDER_SUBTLE}; border-radius: 4px;" + f" background-color: {theme.BG_DARK}; }}" + ) + zoom_layout = QHBoxLayout(zoom_group) + zoom_layout.setContentsMargins(2, 2, 2, 2) + zoom_layout.setSpacing(2) + + self._zoom_rect_btn = QPushButton("⊞ Zoom") + self._zoom_rect_btn.setCheckable(True) + self._zoom_rect_btn.setChecked(True) + self._zoom_rect_btn.setStyleSheet(theme.checkable_button_style()) + self._zoom_rect_btn.clicked.connect(lambda: self._set_zoom_mode("rect")) + zoom_layout.addWidget(self._zoom_rect_btn) + + self._zoom_h_btn = QPushButton("↔ Hor") + self._zoom_h_btn.setCheckable(True) + self._zoom_h_btn.setStyleSheet(theme.checkable_button_style()) + self._zoom_h_btn.clicked.connect(lambda: self._set_zoom_mode("h")) + zoom_layout.addWidget(self._zoom_h_btn) + + self._zoom_v_btn = QPushButton("↕ Ver") + self._zoom_v_btn.setCheckable(True) + self._zoom_v_btn.setStyleSheet(theme.checkable_button_style()) + self._zoom_v_btn.clicked.connect(lambda: self._set_zoom_mode("v")) + zoom_layout.addWidget(self._zoom_v_btn) + + sep = QFrame() + sep.setFrameShape(QFrame.Shape.VLine) + sep.setFixedWidth(1) + sep.setStyleSheet(f"background-color: {theme.BORDER_SUBTLE}; border: none;") + zoom_layout.addWidget(sep) + + self._reset_view_btn = QPushButton("⟳ Reset") + self._reset_view_btn.setStyleSheet(theme.checkable_button_style()) + self._reset_view_btn.clicked.connect(self._reset_view) + zoom_layout.addWidget(self._reset_view_btn) + + view_layout.addWidget(zoom_group) top_layout.addWidget(view_group) top_layout.addStretch() @@ -214,7 +275,7 @@ def _setup_ui(self): ) # TDC Frequency tdc_row = QHBoxLayout() - tdc_label = QLabel("TDC Frequency (Hz):") + tdc_label = QLabel("TDC Frequency (Hz)") tdc_label.setStyleSheet(f"color: {theme.TEXT_SECONDARY};") tdc_label.setToolTip(_tt_tdc_freq) tdc_row.addWidget(tdc_label) @@ -233,7 +294,7 @@ def _setup_ui(self): ) # TDC Channel tdc_ch_row = QHBoxLayout() - tdc_ch_label = QLabel("TDC Channel:") + tdc_ch_label = QLabel("TDC Channel") tdc_ch_label.setStyleSheet(f"color: {theme.TEXT_SECONDARY};") tdc_ch_label.setToolTip(_tt_tdc_ch) tdc_ch_row.addWidget(tdc_ch_label) @@ -247,7 +308,7 @@ def _setup_ui(self): _tt_tdc_edge = "Trigger on the rising or falling edge of the selected TDC line." # TDC Edge tdc_edge_row = QHBoxLayout() - tdc_edge_label = QLabel("TDC Edge:") + tdc_edge_label = QLabel("TDC Edge") tdc_edge_label.setStyleSheet(f"color: {theme.TEXT_SECONDARY};") tdc_edge_label.setToolTip(_tt_tdc_edge) tdc_edge_row.addWidget(tdc_edge_label) @@ -260,7 +321,7 @@ def _setup_ui(self): # Parse batch size (CLI --callback-batch-size): packets per vectorized parse_batch call batch_row = QHBoxLayout() - batch_label = QLabel("Parse batch (pkts):") + batch_label = QLabel("Parse batch (pkts)") batch_label.setStyleSheet(f"color: {theme.TEXT_SECONDARY};") batch_label.setToolTip( "Target number of 12-byte packets per parse batch (vectorized parse_batch). " @@ -277,13 +338,31 @@ def _setup_ui(self): batch_row.addWidget(self._callback_batch_input) settings_layout.addLayout(batch_row) + # n_bins + n_bins_row = QHBoxLayout() + n_bins_label = QLabel("Time bins (n_bins)") + n_bins_label.setStyleSheet(f"color: {theme.TEXT_SECONDARY};") + n_bins_label.setToolTip( + "Number of time bins per TDC cycle. Used as the fallback when the streaming server " + "cannot derive bin count from t_delta_ns. Default: 10000." + ) + n_bins_row.addWidget(n_bins_label) + self._n_bins_input = QSpinBox() + self._n_bins_input.setRange(500, 50_000) + self._n_bins_input.setValue(10_000) + self._n_bins_input.setSingleStep(10) + self._n_bins_input.setStyleSheet(theme.input_style()) + self._n_bins_input.setToolTip(n_bins_label.toolTip()) + n_bins_row.addWidget(self._n_bins_input) + settings_layout.addLayout(n_bins_row) + _tt_duration = ( "Acquisition length in seconds for Serval-backed runs and the simulator. " "Preview and replay may ignore this depending on the workflow." ) # Duration dur_row = QHBoxLayout() - dur_label = QLabel("Duration (s):") + dur_label = QLabel("Duration (s)") dur_label.setStyleSheet(f"color: {theme.TEXT_SECONDARY};") dur_label.setToolTip(_tt_duration) dur_row.addWidget(dur_label) @@ -301,7 +380,7 @@ def _setup_ui(self): ) # Output directory out_row = QHBoxLayout() - out_label = QLabel("Output:") + out_label = QLabel("Output") out_label.setStyleSheet(f"color: {theme.TEXT_SECONDARY};") out_label.setToolTip(_tt_output_label) out_row.addWidget(out_label) @@ -343,7 +422,7 @@ def _setup_ui(self): pipeline_group.setToolTip( "Live depth vs capacity. Packet buffer fills when the parse thread cannot keep up with TCP—" "try lowering Parse batch (pkts) or increasing buffer-size on the server. " - "ZMQ PUB (SUB): 3D flush depth (publisher); value in parentheses is the start/stop control queue." + "ZMQ PUB (SUB) shows 3D flush depth (publisher); value in parentheses is the start/stop control queue." ) pipeline_layout = QVBoxLayout(pipeline_group) pipeline_layout.setSpacing(4) @@ -351,13 +430,13 @@ def _setup_ui(self): queue_rows = [ ( "packet_buffer", - "Packet buffer:", + "Packet buffer", "Queued raw TCP batches waiting for the parser thread (SocketDataServer message queue). " "Fills if parsing lags behind the network reader.", ), ( "zmq_pub", - "ZMQ PUB (SUB):", + "ZMQ PUB (SUB)", "3D flush queue (PUB path) and control queue depth in parentheses; " "denominator is flush-queue capacity.", ), @@ -385,12 +464,12 @@ def _setup_ui(self): self._stats_labels = {} stat_names = [ - ("pixel_rate", "Pixel Rate:"), - ("tdc1_rate", "TDC1 Rate:"), - ("tdc2_rate", "TDC2 Rate:"), - ("elapsed_remaining", "Elapsed / Remaining:"), - ("flushes_cycles", "Flushes (Cycles):"), - ("avg_counts", "Avg Counts/Cycle:"), + ("pixel_rate", "Pixel Rate"), + ("tdc1_rate", "TDC1 Rate"), + ("tdc2_rate", "TDC2 Rate"), + ("elapsed_remaining", "Elapsed / Remaining"), + ("flushes_cycles", "Flushes (Cycles)"), + ("avg_counts", "Avg Counts/Cycle"), ] for key, label in stat_names: @@ -410,17 +489,242 @@ def _setup_ui(self): content_layout.addWidget(left_panel) - # Heatmaps + # Heatmaps + spectrum panel right_panel = QWidget() right_layout = QHBoxLayout(right_panel) right_layout.setContentsMargins(0, 0, 0, 0) right_layout.setSpacing(10) + # Both heatmap columns use the same structure: heatmap (expanding) + bottom row (fixed). + # _SPECTRUM_ROW_H is shared so the two HeatmapWidgets always have equal height. + _SPECTRUM_ROW_H = 210 + + # Current Flush column: heatmap + cursor-readout / calibration panel + flush_col = QWidget() + flush_col_layout = QVBoxLayout(flush_col) + flush_col_layout.setContentsMargins(0, 0, 0, 0) + flush_col_layout.setSpacing(0) self._current_heatmap = HeatmapWidget("Current Flush") - self._average_heatmap = HeatmapWidget("Running Average") + flush_col_layout.addWidget(self._current_heatmap) + + # --- Cursor readout + calibration panel --- + readout_panel = QFrame() + readout_panel.setFixedHeight(_SPECTRUM_ROW_H) + readout_panel.setStyleSheet( + f"QFrame {{ background-color: {theme.BG_PANEL}; " + f"border: 1px solid {theme.BORDER_SUBTLE}; border-radius: 4px; }}" + ) + rp_outer = QHBoxLayout(readout_panel) + rp_outer.setContentsMargins(0, 0, 0, 0) + rp_outer.setSpacing(0) + + # Left 2/3: calibration controls + cursor readout + rp_left = QWidget() + rp_layout = QVBoxLayout(rp_left) + rp_layout.setContentsMargins(10, 8, 8, 8) + rp_layout.setSpacing(4) + + # Vertical divider between left and right sections + sep_v = QFrame() + sep_v.setFrameShape(QFrame.Shape.VLine) + sep_v.setFixedWidth(1) + sep_v.setStyleSheet(f"background-color: {theme.BORDER_SUBTLE}; border: none;") + + # Right 1/3: ROI pair legend + active toggle buttons + rp_right = QWidget() + rp_right_layout = QVBoxLayout(rp_right) + rp_right_layout.setContentsMargins(8, 6, 8, 6) + rp_right_layout.setSpacing(4) + + _ROI_LABELS = ("ROI 1", "ROI 2", "ROI 3", "ROI 4", "ROI 5") + self._roi_toggle_btns: list[QPushButton] = [] + + _toggle_style = f""" + QPushButton {{ + background-color: {theme.GREY_DARK}; + color: {theme.TEXT_MUTED}; + border: 1px solid {theme.BORDER_SUBTLE}; + border-radius: 3px; + padding: 1px 4px; + font-size: 10px; + }} + QPushButton:checked {{ + background-color: {theme.BLUE_PRIMARY}; + color: {theme.TEXT_PRIMARY}; + border-color: {theme.BLUE_LIGHT_2}; + }} + """ + + for i in range(5): + roi_row = QHBoxLayout() + roi_row.setSpacing(6) + + swatch = QLabel() + swatch.setFixedSize(16, 3) + swatch.setStyleSheet(f"background-color: {CURSOR_COLORS[i]}; border: none;") + roi_row.addWidget(swatch, alignment=Qt.AlignmentFlag.AlignVCenter) + + lbl = QLabel(_ROI_LABELS[i]) + lbl.setStyleSheet(f"color: {theme.TEXT_PRIMARY};") + roi_row.addWidget(lbl) + roi_row.addStretch() + + active_default = i < 2 + btn = QPushButton("On" if active_default else "Off") + btn.setCheckable(True) + btn.setChecked(active_default) + btn.setFixedWidth(36) + btn.setStyleSheet(_toggle_style) + roi_row.addWidget(btn) + rp_right_layout.addLayout(roi_row) + self._roi_toggle_btns.append(btn) + + rp_right_layout.addStretch() + + rp_outer.addWidget(rp_left, stretch=2) + rp_outer.addWidget(sep_v) + rp_outer.addWidget(rp_right, stretch=1) + + # Calibration controls: label above each spinbox, the two controls side by side + cal_row = QHBoxLayout() + cal_row.setSpacing(8) + + pxev_col = QVBoxLayout() + pxev_col.setSpacing(2) + ev_px_lbl = QLabel("pixel/eV") + ev_px_lbl.setStyleSheet(f"color: {theme.TEXT_MUTED}; font-size: 10px;") + pxev_col.addWidget(ev_px_lbl) + self._pixel_per_ev = QDoubleSpinBox() + self._pixel_per_ev.setRange(-10000.0, 10000.0) + self._pixel_per_ev.setDecimals(4) + self._pixel_per_ev.setValue(12.796) + self._pixel_per_ev.setSingleStep(0.001) + self._pixel_per_ev.setStyleSheet(theme.input_style()) + self._pixel_per_ev.setFixedWidth(90) + pxev_col.addWidget(self._pixel_per_ev) + cal_row.addLayout(pxev_col) + + evmid_col = QVBoxLayout() + evmid_col.setSpacing(2) + ev_mid_lbl = QLabel("eV @ midpoint") + ev_mid_lbl.setStyleSheet(f"color: {theme.TEXT_MUTED}; font-size: 10px;") + evmid_col.addWidget(ev_mid_lbl) + self._ev_at_mid = QDoubleSpinBox() + self._ev_at_mid.setRange(-1_000_000.0, 1_000_000.0) + self._ev_at_mid.setDecimals(2) + self._ev_at_mid.setValue(0.0) + self._ev_at_mid.setSingleStep(0.1) + self._ev_at_mid.setStyleSheet(theme.input_style()) + self._ev_at_mid.setFixedWidth(90) + evmid_col.addWidget(self._ev_at_mid) + cal_row.addLayout(evmid_col) + cal_row.addStretch() + rp_layout.addLayout(cal_row) + + # Separator + sep = QFrame() + sep.setFrameShape(QFrame.Shape.HLine) + sep.setStyleSheet(f"color: {theme.BORDER_SUBTLE};") + rp_layout.addWidget(sep) + + # Readout grid: columns = name | pixel | eV + grid = QGridLayout() + grid.setSpacing(2) + grid.setColumnStretch(0, 2) + grid.setColumnStretch(1, 3) + grid.setColumnStretch(2, 3) + + for col, text in enumerate(["", "Pixel", "eV"]): + hdr = QLabel(text) + hdr.setStyleSheet(f"color: {theme.TEXT_MUTED}; font-size: 9px;") + hdr.setAlignment(Qt.AlignmentFlag.AlignRight | Qt.AlignmentFlag.AlignVCenter) + grid.addWidget(hdr, 0, col) + + self._cursor_px_labels: list[QLabel] = [] + self._cursor_ev_labels: list[QLabel] = [] + row_names = ["Cursor A", "Cursor B", "Δ"] + for row_idx, name in enumerate(row_names): + name_lbl = QLabel(name) + name_lbl.setStyleSheet(f"color: {theme.TEXT_MUTED}; font-size: 9px;") + grid.addWidget(name_lbl, row_idx + 1, 0) + + px_lbl = QLabel("--") + px_lbl.setStyleSheet(f"font-family: monospace; color: {theme.TEXT_PRIMARY};") + px_lbl.setAlignment(Qt.AlignmentFlag.AlignRight | Qt.AlignmentFlag.AlignVCenter) + grid.addWidget(px_lbl, row_idx + 1, 1) + self._cursor_px_labels.append(px_lbl) + + ev_lbl = QLabel("--") + ev_lbl.setStyleSheet(f"font-family: monospace; color: {theme.TEXT_PRIMARY};") + ev_lbl.setAlignment(Qt.AlignmentFlag.AlignRight | Qt.AlignmentFlag.AlignVCenter) + grid.addWidget(ev_lbl, row_idx + 1, 2) + self._cursor_ev_labels.append(ev_lbl) + + rp_layout.addLayout(grid) + rp_layout.addStretch() + + self._pixel_per_ev.valueChanged.connect(self._update_cursor_readout) + self._ev_at_mid.valueChanged.connect(self._update_cursor_readout) + self._pixel_per_ev.valueChanged.connect(self._update_ruler_energy_scale) + self._ev_at_mid.valueChanged.connect(self._update_ruler_energy_scale) + + flush_col_layout.addWidget(readout_panel) + right_layout.addWidget(flush_col) + + # Running Average column: heatmap + spectrum panel + avg_col = QWidget() + avg_col_layout = QVBoxLayout(avg_col) + avg_col_layout.setContentsMargins(0, 0, 0, 0) + avg_col_layout.setSpacing(0) - right_layout.addWidget(self._current_heatmap) - right_layout.addWidget(self._average_heatmap) + self._average_heatmap = HeatmapWidget("Running Average") + self._average_heatmap.enable_cursors(True) + self._average_heatmap.cursors_changed.connect(self._on_cursor_changed) + avg_col_layout.addWidget(self._average_heatmap) + + # Spectrum wrapper — left spacer matches HeatmapWidget's y-label (20 px + 4 px spacing) + # so the plot x-axis aligns with the heatmap image area. + spectrum_outer = QWidget() + spectrum_outer.setFixedHeight(_SPECTRUM_ROW_H) + so_layout = QHBoxLayout(spectrum_outer) + so_layout.setContentsMargins(4, 4, 4, 4) + so_layout.setSpacing(4) + y_counts_label = VerticalLabel("Counts") + y_counts_label.setFixedWidth(20) + so_layout.addWidget(y_counts_label) + self._spectrum_plot = SpectrumPlotWidget() + self._spectrum_plot.cursors_changed.connect(self._on_spectrum_cursors_changed) + so_layout.addWidget(self._spectrum_plot) + avg_col_layout.addWidget(spectrum_outer) + + self._current_heatmap.view_changed.connect(self._average_heatmap.set_view) + self._average_heatmap.view_changed.connect(self._current_heatmap.set_view) + # Zoom on either heatmap must refresh the spectrum/readout because + # cursor fractions are remapped through the current view. Note that + # set_view does not re-emit view_changed, so we connect both signals. + self._current_heatmap.view_changed.connect(self._on_heatmap_view_changed) + self._average_heatmap.view_changed.connect(self._on_heatmap_view_changed) + right_layout.addWidget(avg_col) + + # Wire ROI toggle buttons now that both heatmap and spectrum plot exist + for _i, _btn in enumerate(self._roi_toggle_btns): + + def _make_toggle(pair_idx, button): + def handler(checked): + button.setText("On" if checked else "Off") + self._average_heatmap.set_cursor_pair_active(pair_idx, checked) + self._spectrum_plot.set_pair_active(pair_idx, checked) + self._sync_cursor_overlay() + + return handler + + _btn.toggled.connect(_make_toggle(_i, _btn)) + + self._average_heatmap.cursors_changed.connect(lambda *_: self._sync_cursor_overlay()) + self._sync_cursor_overlay() + + self._spectrum_plot.cursors_changed.connect(self._sync_vcursor_overlay) + self._sync_vcursor_overlay() content_layout.addWidget(right_panel, stretch=1) layout.addLayout(content_layout, stretch=1) @@ -462,10 +766,56 @@ def _get_params(self) -> dict: "tdc_channel": self._get_tdc_channel(), "tdc_edge": self._get_tdc_edge(), "callback_batch_size": int(self._callback_batch_input.value()), + "n_bins": int(self._n_bins_input.value()), "duration": self._duration_input.value(), "output_dir": self._output_input.text(), } + def _build_preferences(self) -> dict: + """Snapshot the operator-sidebar widget state for persistence. + + Stores the **displayed** combo text (not the converted forms used + by ``_get_params``) so restoration via ``QComboBox.setCurrentText`` + round-trips exactly. See ``preferences.py`` for the full schema. + """ + return { + "tdc_frequency": float(self._tdc_freq_input.value()), + "tdc_channel_text": self._tdc_ch_combo.currentText(), + "tdc_edge_text": self._tdc_edge_combo.currentText(), + "callback_batch_size": int(self._callback_batch_input.value()), + "n_bins": int(self._n_bins_input.value()), + "duration": int(self._duration_input.value()), + "output_dir": self._output_input.text(), + } + + def load_operator_preferences(self, path: Optional[Union[Path, str]] = None) -> None: + """Restore acquisition-sidebar widgets from on-disk preferences. + + Always succeeds: a missing or malformed file falls back to the + widget defaults (already set by ``_setup_ui``). Out-of-range + values are clamped before being applied. The output path tooltip + is refreshed to reflect the loaded path. + """ + prefs = preferences.load_operator_preferences(path) + self._tdc_freq_input.setValue(float(prefs["tdc_frequency"])) + self._tdc_ch_combo.setCurrentText(prefs["tdc_channel_text"]) + self._tdc_edge_combo.setCurrentText(prefs["tdc_edge_text"]) + self._callback_batch_input.setValue(int(prefs["callback_batch_size"])) + self._n_bins_input.setValue(int(prefs["n_bins"])) + self._duration_input.setValue(int(prefs["duration"])) + self._output_input.setText(str(prefs["output_dir"])) + self._sync_output_path_tooltip() + + def save_operator_preferences(self, path: Optional[Union[Path, str]] = None) -> None: + """Persist current sidebar widget state to disk (atomic write). + + Idempotent and side-effect-free aside from the JSON write. The + caller is responsible for swallowing exceptions if the goal is + to never block quit; this method itself raises on I/O failure + so test code can assert behavior. + """ + preferences.save_operator_preferences(self._build_preferences(), path) + def _on_mode_clicked(self, mode: str): params = self._get_params() if mode == "start" and not params["output_dir"]: @@ -486,6 +836,133 @@ def _on_replay_clicked(self): def _on_stop_clicked(self): self.stop_requested.emit() + @Slot(int, float, float) + def _on_cursor_changed(self, pair_idx: int, frac_a: float, frac_b: float) -> None: + """Recompute spectra whenever a heatmap ROI cursor is moved.""" + self._update_spectra() + + @Slot(int, int, int, int) + def _on_heatmap_view_changed(self, x0: int, x1: int, y0: int, y1: int) -> None: + """Refresh spectrum + readout when either heatmap zoom/view changes. + + Cursor fractions are remapped through the current view, so the spectrum + data and the energy readout must be recomputed on every zoom. + """ + self._update_spectra() + self._update_cursor_readout() + + @Slot(float, float) + def _on_spectrum_cursors_changed(self, frac_a: float, frac_b: float) -> None: + """Update pixel/eV readout whenever a spectrum vertical cursor is moved.""" + self._update_cursor_readout() + + def _on_vcursor_heatmap_toggled(self, checked: bool) -> None: + self._current_heatmap.set_vcursors_on_heatmap(checked) + self._average_heatmap.set_vcursors_on_heatmap(checked) + + def _sync_vcursor_overlay(self, frac_a=None, frac_b=None) -> None: + if frac_a is None: + frac_a, frac_b = self._spectrum_plot.get_cursor_fracs() + self._current_heatmap.set_vcursor_overlay(frac_a, frac_b) + self._average_heatmap.set_vcursor_overlay(frac_a, frac_b) + + def _sync_cursor_overlay(self) -> None: + """Push the average heatmap cursor positions into the current-flush heatmap as a read-only overlay.""" + fracs = self._average_heatmap.get_cursor_fracs() + active = [btn.isChecked() for btn in self._roi_toggle_btns] + self._current_heatmap.update_cursor_overlay(fracs, active) + + def _update_cursor_readout(self) -> None: + """Refresh the pixel and eV position labels for the two vertical cursors. + + Spectrum cursor fractions span the heatmap's *visible* x-range, so we + must map them through the current view to get the true pixel index. + When unzoomed (x0=0, x1=n) this reduces to ``frac * (n - 1)``. + """ + n = self._n_energy + if n is None or n < 2: + for lbl in self._cursor_px_labels + self._cursor_ev_labels: + lbl.setText("--") + return + + x0, x1, _, _ = self._average_heatmap.get_view() + x0 = max(0, min(x0, n - 1)) + x1 = max(x0 + 1, min(x1, n)) + x_span_eff = max(0, x1 - x0 - 1) + + frac_a, frac_b = self._spectrum_plot.get_cursor_fracs() + x_a = x0 + frac_a * x_span_eff + x_b = x0 + frac_b * x_span_eff + + pixel_per_ev = self._pixel_per_ev.value() + ev_at_mid = self._ev_at_mid.value() + eV_a = ev_at_mid + (x_a - n / 2) / pixel_per_ev + eV_b = ev_at_mid + (x_b - n / 2) / pixel_per_ev + + dx = abs(x_b - x_a) + deV = abs(eV_b - eV_a) + + self._cursor_px_labels[0].setText(f"{x_a:.1f}") + self._cursor_px_labels[1].setText(f"{x_b:.1f}") + self._cursor_px_labels[2].setText(f"{dx:.1f}") + self._cursor_ev_labels[0].setText(f"{eV_a:.2f}") + self._cursor_ev_labels[1].setText(f"{eV_b:.2f}") + self._cursor_ev_labels[2].setText(f"{deV:.2f}") + + def _update_ruler_energy_scale(self) -> None: + n = self._n_energy + if n is None: + return + pixel_per_ev = self._pixel_per_ev.value() + if pixel_per_ev == 0: + return + ev_at_mid = self._ev_at_mid.value() + ev_per_pixel = 1.0 / pixel_per_ev + ev_at_zero = ev_at_mid - (n / 2) * ev_per_pixel + for hm in (self._current_heatmap, self._average_heatmap): + hm.set_x_scale(ev_per_pixel, ev_at_zero) + + def _update_spectra(self) -> None: + """Bin the average heatmap between each cursor pair and update the spectrum plot. + + Data shape is (n_energy, n_time). The display shows ``flipud(data.T)``, + so display-row indices map to time bins as ``time_bin = (n_t - 1) - row``. + Cursor fractions are widget-relative and span the *visible* portion of + the heatmap (the current view), not the full data — so we must apply the + view to map them correctly when zoomed. We also slice the spectrum to + the visible x (energy) range so the bottom plot's x-axis matches the + heatmap's x-axis after a zoom. + """ + data = self._last_avg_2d + if data is None or data.ndim != 2 or data.shape[0] == 0 or data.shape[1] == 0: + return + n_e, n_t = data.shape + + # Visible region in data-index coords; clamp defensively. + x0, x1, y0, y1 = self._average_heatmap.get_view() + x0 = max(0, min(x0, n_e - 1)) + x1 = max(x0 + 1, min(x1, n_e)) + y0 = max(0, min(y0, n_t - 1)) + y1 = max(y0 + 1, min(y1, n_t)) + + # Map widget fraction f in [0, 1] → continuous display row → time bin. + # When unzoomed (y0=0, y1=n_t) this reduces to (1 - f) * (n_t - 1), + # matching the previous formula. ``y_span_eff`` uses (y1 - y0 - 1) so + # f=0 lands on row y0 and f=1 lands on row (y1 - 1), the last visible row. + y_span_eff = max(0, y1 - y0 - 1) + + def frac_to_time_bin(f: float) -> int: + row = y0 + f * y_span_eff + return int(round((n_t - 1) - row)) + + for pair_idx, (frac_a, frac_b) in enumerate(self._average_heatmap.get_cursor_fracs()): + ta = max(0, min(n_t - 1, frac_to_time_bin(frac_a))) + tb = max(0, min(n_t - 1, frac_to_time_bin(frac_b))) + t_lo, t_hi = min(ta, tb), max(ta, tb) + n_bins = t_hi - t_lo + 1 + spectrum = data[x0:x1, t_lo : t_hi + 1].sum(axis=1) / n_bins + self._spectrum_plot.set_spectrum(pair_idx, spectrum) + def _on_colormap_changed(self, name: str): self._current_heatmap.set_colormap(name) self._average_heatmap.set_colormap(name) @@ -494,10 +971,30 @@ def _reset_average(self): self._cumulative_sum = None self._total_cycles = 0 self._flush_count = 0 + self._last_avg_2d = None + self._n_energy = None + self._current_heatmap.clear() self._average_heatmap.clear() + self._spectrum_plot.clear() + for lbl in self._cursor_px_labels + self._cursor_ev_labels: + lbl.setText("--") self._update_flush_stats() logger.info("Running average reset") + def _set_zoom_mode(self, mode: str) -> None: + for btn, m in [ + (self._zoom_rect_btn, "rect"), + (self._zoom_h_btn, "h"), + (self._zoom_v_btn, "v"), + ]: + btn.setChecked(m == mode) + self._current_heatmap.set_zoom_mode(mode) + self._average_heatmap.set_zoom_mode(mode) + + def _reset_view(self) -> None: + self._current_heatmap.reset_view() + self._average_heatmap.reset_view() + def _update_flush_stats(self): self._stats_labels["flushes_cycles"].setText(f"{self._flush_count} ({self._total_cycles:,})") if self._total_cycles > 0 and self._cumulative_sum is not None: @@ -506,18 +1003,32 @@ def _update_flush_stats(self): else: self._stats_labels["avg_counts"].setText("--") + def _refresh_mode_buttons_state(self) -> None: + """Enable Start/Preview only when Serval has finished HW init; simulator/replay never need Serval.""" + acquiring = self._acquiring + serval_ok = self._serval_process_running and self._serval_hw_ready + if acquiring: + self._start_btn.setEnabled(False) + self._preview_btn.setEnabled(False) + self._simulator_btn.setEnabled(False) + self._replay_btn.setEnabled(False) + self._stop_btn.setEnabled(True) + else: + self._stop_btn.setEnabled(False) + self._start_btn.setEnabled(serval_ok) + self._preview_btn.setEnabled(serval_ok) + self._simulator_btn.setEnabled(True) + self._replay_btn.setEnabled(True) + @Slot(bool) def set_acquiring(self, acquiring: bool): self._acquiring = acquiring - self._start_btn.setEnabled(not acquiring) - self._preview_btn.setEnabled(not acquiring) - self._simulator_btn.setEnabled(not acquiring) - self._replay_btn.setEnabled(not acquiring) - self._stop_btn.setEnabled(acquiring) + self._refresh_mode_buttons_state() self._tdc_freq_input.setEnabled(not acquiring) self._tdc_ch_combo.setEnabled(not acquiring) self._tdc_edge_combo.setEnabled(not acquiring) self._callback_batch_input.setEnabled(not acquiring) + self._n_bins_input.setEnabled(not acquiring) self._duration_input.setEnabled(not acquiring) self._output_input.setEnabled(not acquiring) self._browse_output_btn.setEnabled(not acquiring) @@ -555,8 +1066,24 @@ def on_flush_received(self, flush_data: FlushData): avg_2d = np.sum(average, axis=1) avg_stats = f"Over {self._total_cycles} cycles | Avg: {np.sum(avg_2d):.2e}" + self._last_avg_2d = avg_2d + self._n_energy = avg_2d.shape[0] self._average_heatmap.set_data(avg_2d, avg_stats) self._update_flush_stats() + self._update_spectra() + self._update_cursor_readout() + t_delta_ns = metadata.get("t_delta_ns") + if t_delta_ns is None: + tdc_hz = metadata.get("tdc_frequency_hz") + n_t = avg_2d.shape[1] + n_bins_meta = metadata.get("n_bins") or n_t + if tdc_hz and tdc_hz > 0: + t_delta_ns = 1e9 / (tdc_hz * n_bins_meta) + if t_delta_ns: + n_t = avg_2d.shape[1] + for hm in (self._current_heatmap, self._average_heatmap): + hm.set_axis_info(t_delta_ns, n_t) + self._update_ruler_energy_scale() def _update_serval_indicator(self) -> None: """Red when JVM down; blinking green until chip temps log; then steady green + Idle/status.""" @@ -582,6 +1109,7 @@ def on_serval_process_running(self, running: bool) -> None: self._serval_process_running = running self._serval_hw_ready = False self._update_serval_indicator() + self._refresh_mode_buttons_state() def on_serval_chip_temps_line_seen(self) -> None: """Called when Serval stdout contains the chip temperature line (startup complete).""" @@ -589,6 +1117,7 @@ def on_serval_chip_temps_line_seen(self) -> None: return self._serval_hw_ready = True self._update_serval_indicator() + self._refresh_mode_buttons_state() @Slot(object) def on_serval_status(self, status: ServalStatus): @@ -666,11 +1195,16 @@ def get_cumulative_data(self) -> tuple[Optional[np.ndarray], int]: def save_average_data( self, output_dir: str, filename_base: str - ) -> tuple[Optional[Path], Optional[Path], Optional[Path]]: - """Save the average heatmap as PNG, CSV, and metadata as JSON.""" + ) -> tuple[Optional[Path], Optional[Path], Optional[Path], Optional[Path], Optional[Path], str]: + """Save heatmap as PNG, CSV, energy-axis CSV, time-axis CSV, metadata JSON (incl. scan_name). + + Returns (png_path, csv_path, energy_path, time_path, json_path, slug) where slug is + ``{uuid_short}_{local_timestamp}`` and is shared by all six output files including the + renamed .tpx3 source file. + """ if self._cumulative_sum is None or self._total_cycles == 0: logger.warning("No data to save") - return None, None, None + return None, None, None, None, None, "" output_path = Path(output_dir) output_path.mkdir(parents=True, exist_ok=True) @@ -681,17 +1215,62 @@ def save_average_data( else: avg_2d = np.sum(average, axis=1) - # Save CSV - csv_path = output_path / f"{filename_base}_avg.csv" + # Use scan_name from ZMQ metadata so it matches what the streaming server + # broadcast; fall back to a fresh UUID4 when metadata isn't available. + meta_for_uuid = self._last_metadata or {} + scan_uuid = meta_for_uuid.get("scan_name") or str(uuid.uuid4()) + scan_uuid_short = scan_uuid.replace("-", "")[:8] + save_ts = datetime.now().strftime("%Y%m%dT%H%M%S") + slug = f"{scan_uuid_short}_{save_ts}" + + # Save average CSV + csv_path = output_path / f"{filename_base}_{slug}_avg.csv" try: np.savetxt(csv_path, avg_2d, delimiter=",", fmt="%.6e") - logger.info(f"Saved CSV: {csv_path}") + logger.info("Saved CSV: %s (scan: %s)", csv_path, scan_uuid) except Exception as e: logger.error(f"Failed to save CSV: {e}") csv_path = None + # Save energy-axis CSV (one eV value per x-pixel) + energy_path = output_path / f"{filename_base}_{slug}_energy.csv" + try: + n_x = avg_2d.shape[0] + pixel_per_ev = self._pixel_per_ev.value() + ev_at_mid = self._ev_at_mid.value() + energy_axis = ev_at_mid + (np.arange(n_x) - n_x / 2) / pixel_per_ev + np.savetxt(energy_path, energy_axis, delimiter=",", fmt="%.6f") + logger.info("Saved energy axis CSV: %s (scan: %s)", energy_path, scan_uuid) + except Exception as e: + logger.error(f"Failed to save energy axis CSV: {e}") + energy_path = None + + # Save time-axis CSV (one ns value per time bin) + # t_delta_ns comes from the ZMQ metadata published by app.py; it is + # 1 / (tdc_frequency_hz * n_bins) * 1e9 and is the width of one time bin. + time_path = output_path / f"{filename_base}_{slug}_time.csv" + try: + n_t = avg_2d.shape[1] + meta = self._last_metadata or {} + t_delta_ns = meta.get("t_delta_ns") + if t_delta_ns is None: + # Fallback: reconstruct from tdc_frequency and n_bins if present + tdc_hz = meta.get("tdc_frequency_hz") + n_bins = meta.get("n_bins") or n_t + if tdc_hz and tdc_hz > 0: + t_delta_ns = 1e9 / (tdc_hz * n_bins) + else: + t_delta_ns = 1.0 # unknown — bin index only + logger.warning("t_delta_ns not in metadata; time axis will be bin-index units (1 ns/bin assumed)") + time_axis = np.arange(n_t) * t_delta_ns + np.savetxt(time_path, time_axis, delimiter=",", fmt="%.6f") + logger.info("Saved time axis CSV: %s (scan: %s)", time_path, scan_uuid) + except Exception as e: + logger.error(f"Failed to save time axis CSV: {e}") + time_path = None + # Save PNG - png_path = output_path / f"{filename_base}_avg.png" + png_path = output_path / f"{filename_base}_{slug}_avg.png" try: display_data = np.flipud(avg_2d.T.astype(np.float32)) vmin, vmax = display_data.min(), display_data.max() @@ -707,15 +1286,16 @@ def save_average_data( h, w = rgb.shape[:2] qimg = QImage(rgb.data, w, h, 3 * w, QImage.Format.Format_RGB888) qimg.save(str(png_path)) - logger.info(f"Saved PNG: {png_path}") + logger.info("Saved PNG: %s (scan: %s)", png_path, scan_uuid) except Exception as e: logger.error(f"Failed to save PNG: {e}") png_path = None # Save JSON metadata - json_path = output_path / f"{filename_base}_meta.json" + json_path = output_path / f"{filename_base}_{slug}_meta.json" try: meta = { + "scan_name": scan_uuid, "total_flushes": self._flush_count, "total_cycles": self._total_cycles, "total_counts": float(np.sum(self._cumulative_sum)), @@ -728,9 +1308,9 @@ def save_average_data( with open(json_path, "w") as f: json.dump(meta, f, indent=2) - logger.info(f"Saved JSON: {json_path}") + logger.info("Saved JSON: %s (scan: %s)", json_path, scan_uuid) except Exception as e: logger.error(f"Failed to save JSON: {e}") json_path = None - return png_path, csv_path, json_path + return png_path, csv_path, energy_path, time_path, json_path, slug diff --git a/src/splash_timepix/ui/preferences.py b/src/splash_timepix/ui/preferences.py new file mode 100644 index 0000000..662b1ef --- /dev/null +++ b/src/splash_timepix/ui/preferences.py @@ -0,0 +1,358 @@ +"""Persistent operator-sidebar preferences (JSON, atomic write, clamp on load). + +The acquisition-settings sidebar in :class:`splash_timepix.ui.operator_tab.OperatorTab` +is restored from ``operator_prefs.json`` at startup and written back on a clean +quit. This module owns the on-disk format, the path resolution, and the +validate/clamp logic; the tab itself just calls :func:`load_operator_preferences` +and :func:`save_operator_preferences`. + +Saves are atomic (write to ``.tmp``, then ``os.replace``) so a crash during +write cannot corrupt the existing file. Loads tolerate a missing file, +malformed JSON, and out-of-range / unknown values: the sanitized result is +always a complete, in-range preferences dict, never raising into the UI layer. + +Crash-time saves are intentionally lost — JSON only updates on a clean exit. +""" + +from __future__ import annotations + +import json +import logging +import os +from pathlib import Path +from typing import Any, Dict, Optional, Union + +logger = logging.getLogger(__name__) + +# Bumped from 1 → 2 when alignment_* keys were added. Purely informational — +# validate_and_clamp tolerates missing/unknown keys, so v1 files load cleanly. +PREFERENCES_VERSION = 2 + +# Combo *display text* values. These must match the items added to the combo +# boxes in OperatorTab._setup_ui, because restoration uses +# QComboBox.setCurrentText(...) which only matches displayed item text exactly. +TDC_CHANNEL_VALUES = ("Both", "1", "2") +TDC_EDGE_VALUES = ("Rising", "Falling") + +# Numeric ranges, mirrored from the QSpinBox / QDoubleSpinBox setRange(...) +# calls in OperatorTab._setup_ui. Keep these in sync with the widget config. +TDC_FREQUENCY_RANGE = (0.1, 1e9) +CALLBACK_BATCH_SIZE_RANGE = (1, 10_000_000) +N_BINS_RANGE = (500, 50_000) +DURATION_RANGE = (1, 19_008_000) + +# Alignment-tab numeric ranges, mirrored from AlignmentTab._setup_ui. +ALIGNMENT_RATE_HZ_RANGE = (1, 30) + +# Scan counter: monotonically increasing integer written back after every +# successful acquisition save. Gives each output file group a human-readable +# sequential prefix independent of Serval's internal file naming. +SCAN_COUNTER_RANGE = (1, 999_999) +# Manual min/max levels span the practical range of uint32 alignment counts; +# these are sanity bounds, not perceptually-tuned defaults. +ALIGNMENT_LEVEL_RANGE = (0, 2**31 - 1) + +_PREFS_FILENAME = "operator_prefs.json" +_TMP_SUFFIX = ".tmp" + + +def default_preferences() -> Dict[str, Any]: + """Return a fresh preferences dict matching the widgets' built-in defaults.""" + return { + "preferences_version": PREFERENCES_VERSION, + "tdc_frequency": 1000.0, + "tdc_channel_text": "Both", + "tdc_edge_text": "Rising", + "callback_batch_size": 10_000, + "n_bins": 10_000, + "duration": 60, + "output_dir": str(Path.home() / "Desktop" / "data"), + # Alignment-tab defaults. UI starts in auto-range with crosshair on, + # 30 Hz update, latest-only display, linear (non-log) intensity. + "alignment_rate_hz": 30, + "alignment_auto_range": True, + # manual_min/max are floats so they can express log-space levels too + # (typical max ~5 in log10 space). They're only consulted in Manual + # range mode; toggling Log forces Auto so the image stays visible. + "alignment_manual_min": 0.0, + "alignment_manual_max": 100.0, + "alignment_log": False, + # Binarize defaults ON: collapses any pixel > 0 to the brightest LUT + # color so even single hits are unambiguously visible. Operators + # routinely have very low count rates during initial alignment, where + # the linear LUT renders nearly all pixels as near-black. + "alignment_binarize": True, + "alignment_show_integrated": False, + "alignment_show_crosshair": True, + # Alignment-only local simulator (synthetic flushes; no Serval/live-cli). + "alignment_simulator": False, + # Monotonically increasing scan counter, incremented after every + # successful acquisition save. Stored here so it persists across + # app restarts and accumulates across a full beamtime session. + "scan_counter": 1, + } + + +def config_dir() -> Path: + """Resolve the per-user config dir, honoring ``XDG_CONFIG_HOME``.""" + base = os.environ.get("XDG_CONFIG_HOME") or str(Path.home() / ".config") + return Path(base) / "splash_timepix" + + +def operator_prefs_path(base: Optional[Path] = None) -> Path: + """Path to ``operator_prefs.json`` under the config dir (or override).""" + return (base if base is not None else config_dir()) / _PREFS_FILENAME + + +def _clamp_float(value: Any, lo: float, hi: float, fallback: float, name: str) -> float: + try: + v = float(value) + except (TypeError, ValueError): + logger.warning("Pref %s: cannot coerce %r to float; using default %s", name, value, fallback) + return fallback + if v != v: # NaN + logger.warning("Pref %s: NaN; using default %s", name, fallback) + return fallback + if v < lo: + logger.warning("Pref %s: %s below min %s; clamping", name, v, lo) + return lo + if v > hi: + logger.warning("Pref %s: %s above max %s; clamping", name, v, hi) + return hi + return v + + +def _clamp_int(value: Any, lo: int, hi: int, fallback: int, name: str) -> int: + try: + # Reject bools (which are ints in Python) and float-like inputs that + # would silently truncate. + if isinstance(value, bool): + raise TypeError + v = int(value) + if isinstance(value, float) and not value.is_integer(): + raise TypeError + except (TypeError, ValueError): + logger.warning("Pref %s: cannot coerce %r to int; using default %s", name, value, fallback) + return fallback + if v < lo: + logger.warning("Pref %s: %s below min %s; clamping", name, v, lo) + return lo + if v > hi: + logger.warning("Pref %s: %s above max %s; clamping", name, v, hi) + return hi + return v + + +def _validate_choice(value: Any, choices: tuple, fallback: str, name: str) -> str: + if isinstance(value, str) and value in choices: + return value + logger.warning("Pref %s: %r not in %s; using default %r", name, value, choices, fallback) + return fallback + + +def _validate_output_dir(value: Any, fallback: str, name: str) -> str: + if isinstance(value, str) and value.strip(): + return value + logger.warning("Pref %s: %r not a non-empty string; using default %r", name, value, fallback) + return fallback + + +def _validate_bool(value: Any, fallback: bool, name: str) -> bool: + if isinstance(value, bool): + return value + logger.warning("Pref %s: %r not a bool; using default %r", name, value, fallback) + return fallback + + +def validate_and_clamp(raw: Any) -> Dict[str, Any]: + """Return a complete, in-range preferences dict from arbitrary input. + + Unknown keys are dropped silently; missing keys fall back to defaults; + out-of-range values are clamped with a warning. Non-dict inputs (e.g. a + JSON file containing a list) yield the full default set. + """ + defaults = default_preferences() + if not isinstance(raw, dict): + if raw is not None: + logger.warning("Preferences: top-level not a dict (%r); using defaults", type(raw).__name__) + return defaults + + out: Dict[str, Any] = {"preferences_version": PREFERENCES_VERSION} + + out["tdc_frequency"] = _clamp_float( + raw.get("tdc_frequency", defaults["tdc_frequency"]), + *TDC_FREQUENCY_RANGE, + fallback=defaults["tdc_frequency"], + name="tdc_frequency", + ) + out["tdc_channel_text"] = _validate_choice( + raw.get("tdc_channel_text", defaults["tdc_channel_text"]), + TDC_CHANNEL_VALUES, + fallback=defaults["tdc_channel_text"], + name="tdc_channel_text", + ) + out["tdc_edge_text"] = _validate_choice( + raw.get("tdc_edge_text", defaults["tdc_edge_text"]), + TDC_EDGE_VALUES, + fallback=defaults["tdc_edge_text"], + name="tdc_edge_text", + ) + out["callback_batch_size"] = _clamp_int( + raw.get("callback_batch_size", defaults["callback_batch_size"]), + *CALLBACK_BATCH_SIZE_RANGE, + fallback=defaults["callback_batch_size"], + name="callback_batch_size", + ) + out["n_bins"] = _clamp_int( + raw.get("n_bins", defaults["n_bins"]), + *N_BINS_RANGE, + fallback=defaults["n_bins"], + name="n_bins", + ) + out["duration"] = _clamp_int( + raw.get("duration", defaults["duration"]), + *DURATION_RANGE, + fallback=defaults["duration"], + name="duration", + ) + out["output_dir"] = _validate_output_dir( + raw.get("output_dir", defaults["output_dir"]), + fallback=defaults["output_dir"], + name="output_dir", + ) + + # Alignment-tab keys. Same flat namespace as operator keys; adding them here + # rather than in a nested dict keeps the JSON shape and existing read/write + # codepaths simple. + out["alignment_rate_hz"] = _clamp_int( + raw.get("alignment_rate_hz", defaults["alignment_rate_hz"]), + *ALIGNMENT_RATE_HZ_RANGE, + fallback=defaults["alignment_rate_hz"], + name="alignment_rate_hz", + ) + out["alignment_auto_range"] = _validate_bool( + raw.get("alignment_auto_range", defaults["alignment_auto_range"]), + fallback=defaults["alignment_auto_range"], + name="alignment_auto_range", + ) + out["alignment_manual_min"] = _clamp_float( + raw.get("alignment_manual_min", defaults["alignment_manual_min"]), + float(ALIGNMENT_LEVEL_RANGE[0]), + float(ALIGNMENT_LEVEL_RANGE[1]), + fallback=float(defaults["alignment_manual_min"]), + name="alignment_manual_min", + ) + out["alignment_manual_max"] = _clamp_float( + raw.get("alignment_manual_max", defaults["alignment_manual_max"]), + float(ALIGNMENT_LEVEL_RANGE[0]), + float(ALIGNMENT_LEVEL_RANGE[1]), + fallback=float(defaults["alignment_manual_max"]), + name="alignment_manual_max", + ) + out["alignment_log"] = _validate_bool( + raw.get("alignment_log", defaults["alignment_log"]), + fallback=defaults["alignment_log"], + name="alignment_log", + ) + out["alignment_binarize"] = _validate_bool( + raw.get("alignment_binarize", defaults["alignment_binarize"]), + fallback=defaults["alignment_binarize"], + name="alignment_binarize", + ) + out["alignment_show_integrated"] = _validate_bool( + raw.get("alignment_show_integrated", defaults["alignment_show_integrated"]), + fallback=defaults["alignment_show_integrated"], + name="alignment_show_integrated", + ) + out["alignment_show_crosshair"] = _validate_bool( + raw.get("alignment_show_crosshair", defaults["alignment_show_crosshair"]), + fallback=defaults["alignment_show_crosshair"], + name="alignment_show_crosshair", + ) + out["alignment_simulator"] = _validate_bool( + raw.get("alignment_simulator", defaults["alignment_simulator"]), + fallback=defaults["alignment_simulator"], + name="alignment_simulator", + ) + out["scan_counter"] = _clamp_int( + raw.get("scan_counter", defaults["scan_counter"]), + *SCAN_COUNTER_RANGE, + fallback=defaults["scan_counter"], + name="scan_counter", + ) + return out + + +def load_operator_preferences(path: Optional[Union[Path, str]] = None) -> Dict[str, Any]: + """Load and sanitize preferences from ``path`` (defaults to the user config). + + Never raises: a missing file, unreadable file, malformed JSON, or + out-of-range values all collapse to the default-and-clamp path with a + warning. The returned dict is always complete and in-range. + """ + p = Path(path) if path is not None else operator_prefs_path() + try: + with open(p, "r", encoding="utf-8") as fh: + raw = json.load(fh) + except FileNotFoundError: + logger.info("Preferences: %s does not exist; using defaults", p) + return default_preferences() + except (OSError, json.JSONDecodeError) as e: + logger.warning("Preferences: failed to read %s (%s); using defaults", p, e) + return default_preferences() + result = validate_and_clamp(raw) + logger.info("Preferences loaded from %s (%d top-level keys)", p, len(result)) + return result + + +def save_operator_preferences(prefs: Dict[str, Any], path: Optional[Union[Path, str]] = None) -> None: + """Atomically persist ``prefs`` to ``path`` (defaults to user config). + + The supplied ``prefs`` dict is *merged* with the existing on-disk state + (current file wins for keys not in ``prefs``, ``prefs`` wins where + overlapping). This lets callers persist only their subset (e.g. just the + operator-tab keys, or just the alignment-tab keys) without zeroing out + keys owned by another part of the UI. + + Validates/clamps on the way out so a misuse in the UI layer cannot write + a malformed file. Creates the parent directory if missing. Writes to + ``.tmp`` then ``os.replace`` into the final name so a crash mid- + write cannot corrupt the existing file. + """ + p = Path(path) if path is not None else operator_prefs_path() + + # Read current on-disk state (best-effort) so partial saves preserve other + # tabs' keys. Any read failure collapses to an empty merge — equivalent to + # the pre-merge behavior. + existing: Dict[str, Any] = {} + try: + with open(p, "r", encoding="utf-8") as fh: + loaded = json.load(fh) + if isinstance(loaded, dict): + existing = loaded + except (FileNotFoundError, OSError, json.JSONDecodeError): + existing = {} + + merged: Dict[str, Any] = {**existing, **prefs} + sanitized = validate_and_clamp(merged) + + parent = p.parent + parent.mkdir(parents=True, exist_ok=True) + + tmp_path = p.with_name(p.name + _TMP_SUFFIX) + payload = json.dumps(sanitized, indent=2, sort_keys=True) + "\n" + + # Write + fsync the tmp file before replacing so the rename target's + # contents are guaranteed durable (modulo FS quirks). os.replace is + # atomic on POSIX within a single filesystem. + with open(tmp_path, "w", encoding="utf-8") as fh: + fh.write(payload) + fh.flush() + try: + os.fsync(fh.fileno()) + except OSError: + # Some filesystems (tmpfs in containers) don't support fsync; + # the replace below is still atomic on POSIX. + pass + os.replace(tmp_path, p) + logger.info("Preferences saved to %s", p) diff --git a/src/splash_timepix/ui/single_instance.py b/src/splash_timepix/ui/single_instance.py new file mode 100644 index 0000000..516ff47 --- /dev/null +++ b/src/splash_timepix/ui/single_instance.py @@ -0,0 +1,299 @@ +"""Linux-only per-user single-instance enforcement for the TimePix UI. + +Acquires a non-blocking exclusive ``fcntl.flock`` on +``/ui.lock`` opened with ``O_CLOEXEC`` so the lock FD is not +inherited by child processes spawned by :class:`ProcessManager` (Serval, +streaming-server, live-cli, simulator). The kernel drops the flock on +process exit — clean, segfault, or SIGKILL — so a crashed previous run +never bricks the next launch. + +The PID of the current holder is written into the lock file *only after* +acquiring the lock. The PID is consumed by the second-instance "Kill" UX +in ``main.py``; the authoritative "someone is running" signal is the +flock failure itself, not the file contents. + +Non-Linux platforms (e.g. macOS dev boxes) are intentionally not +enforced: :func:`acquire_lock` returns ``None`` and :func:`other_instance_pid` +returns ``None``. The plan calls out Linux-only as a hard scope decision. +""" + +from __future__ import annotations + +import logging +import os +import signal +import sys +import time +from pathlib import Path +from typing import Iterable, Optional, Tuple, Union + +import psutil + +from .preferences import config_dir as _default_config_dir + +logger = logging.getLogger(__name__) + +LOCK_FILENAME = "ui.lock" + +# Markers used to confirm the lock-file PID actually points at a TimePix UI +# process before signaling. Either the entry-point script name or the +# module path is sufficient — both are stable across the supported launch +# methods (``tpx-ui`` console-script and ``python -m splash_timepix.ui.main``). +_DEFAULT_APP_MARKERS: Tuple[str, ...] = ("tpx-ui", "splash_timepix.ui.main") + +# Module-level reference keeps the held FD alive for the entire process +# lifetime. Closing the FD (via __del__, atexit, or accidental gc of a +# wrapper) would drop the lock prematurely. The kernel will drop it for us +# at exit; that is the desired behavior. +_held_fd: Optional[int] = None + + +class AlreadyRunning(RuntimeError): + """Another process holds the singleton lock. + + ``pid`` is the PID written by the holder, or ``None`` if the file is + empty / unparseable (which can happen during the holder's startup + window between flock and write). Callers should treat ``None`` as + "lock is held but PID unknown" and skip the Kill path. + """ + + def __init__(self, pid: Optional[int]): + super().__init__(f"another instance is running (pid={pid})") + self.pid = pid + + +def lock_path(base: Optional[Union[Path, str]] = None) -> Path: + """Path to ``ui.lock`` under the config dir (or override).""" + base_path = Path(base) if base is not None else _default_config_dir() + return base_path / LOCK_FILENAME + + +def _is_linux() -> bool: + return sys.platform == "linux" + + +def acquire_lock(path: Optional[Union[Path, str]] = None) -> Optional[int]: + """Acquire the singleton lock; return the held FD or ``None`` (non-Linux). + + On Linux: opens ``path`` with ``O_RDWR | O_CREAT | O_CLOEXEC``, + attempts ``fcntl.flock(LOCK_EX | LOCK_NB)``, writes the current PID + on success, and stashes the FD in a module-global so it stays alive + for the process lifetime. Subsequent calls in the same process are + idempotent (returns the already-held FD). + + Raises :class:`AlreadyRunning` if another process holds the lock. + + On non-Linux platforms: logs a warning and returns ``None`` without + enforcing anything (matches the plan's explicit Linux-only scope). + """ + global _held_fd + if not _is_linux(): + logger.warning("Single-instance lock not enforced on platform %r", sys.platform) + return None + + if _held_fd is not None: + return _held_fd + + import fcntl + + p = Path(path) if path is not None else lock_path() + p.parent.mkdir(parents=True, exist_ok=True) + + fd = os.open(p, os.O_RDWR | os.O_CREAT | os.O_CLOEXEC, 0o600) + try: + fcntl.flock(fd, fcntl.LOCK_EX | fcntl.LOCK_NB) + except BlockingIOError: + os.close(fd) + # Read the holder's PID from a fresh FD (no flock) so we never + # block. The holder may be mid-startup with an empty file; we + # tolerate that by returning pid=None. + holder_pid = _read_pid(p) + raise AlreadyRunning(holder_pid) + except OSError: + os.close(fd) + raise + + try: + os.ftruncate(fd, 0) + os.write(fd, str(os.getpid()).encode("ascii")) + try: + os.fsync(fd) + except OSError: + pass + except OSError: + # Releasing the lock on the way out is fine here: we never returned + # success, so no caller is depending on it being held. + try: + fcntl.flock(fd, fcntl.LOCK_UN) + finally: + os.close(fd) + raise + + _held_fd = fd + logger.info("Acquired single-instance lock at %s (pid=%d)", p, os.getpid()) + return fd + + +def release_lock() -> None: + """Release the held lock if any (test-only; production relies on kernel). + + Production code should *not* call this — the kernel drops the lock at + process exit, and an early release window would let a second instance + start before child processes (Serval, live-cli) finish shutting down. + Tests use this to simulate process death. + """ + global _held_fd + if _held_fd is None: + return + try: + import fcntl + + fcntl.flock(_held_fd, fcntl.LOCK_UN) + except OSError: + pass + try: + os.close(_held_fd) + except OSError: + pass + _held_fd = None + + +def _read_pid(path: Path) -> Optional[int]: + try: + with open(path, "r", encoding="ascii") as fh: + text = fh.read().strip() + except (FileNotFoundError, OSError): + return None + if not text: + return None + try: + return int(text) + except ValueError: + return None + + +def other_instance_pid(path: Optional[Union[Path, str]] = None) -> Optional[int]: + """Return the PID written in the lock file, or ``None`` if unavailable.""" + if not _is_linux(): + return None + return _read_pid(Path(path) if path is not None else lock_path()) + + +def _proc_is_live(proc: psutil.Process) -> bool: + """Return True iff ``proc`` is running and not a zombie. + + ``psutil.Process.is_running()`` returns True for zombie processes + (terminated but not yet reaped by the parent), which in our context + means the process cannot accept further signals and is effectively + gone. Treat that the same as "not running". + """ + try: + if not proc.is_running(): + return False + return proc.status() != psutil.STATUS_ZOMBIE + except (psutil.NoSuchProcess, psutil.ZombieProcess): + return False + + +def is_other_instance_alive( + pid: int, + *, + app_markers: Iterable[str] = _DEFAULT_APP_MARKERS, +) -> bool: + """Return True iff ``pid`` is a live TimePix UI owned by the current user. + + Verifies (in order): + 1. ``psutil.Process(pid)`` resolves, is running, and is not a zombie. + 2. Real uid matches ``os.getuid()`` (no cross-user signaling). + 3. ``cmdline()`` contains one of ``app_markers`` (mitigates PID + reuse: a recycled PID belonging to an unrelated process is + vanishingly unlikely to match any of our entry-point names). + + Any psutil exception (NoSuchProcess, AccessDenied, ZombieProcess) is + treated as "not our process" and returns False. + """ + try: + proc = psutil.Process(pid) + if not _proc_is_live(proc): + return False + if proc.uids().real != os.getuid(): + return False + cmdline = " ".join(proc.cmdline()) + except (psutil.NoSuchProcess, psutil.AccessDenied, psutil.ZombieProcess): + return False + return any(marker in cmdline for marker in app_markers) + + +def terminate_other_instance( + pid: int, + *, + timeout_s: float = 5.0, + poll_interval_s: float = 0.1, + app_markers: Iterable[str] = _DEFAULT_APP_MARKERS, +) -> bool: + """Send SIGTERM to ``pid`` if it looks like our app; wait up to ``timeout_s``. + + Returns ``True`` iff the target process is no longer running by the + time we return. Refuses to signal anything that fails the + :func:`is_other_instance_alive` checks — the caller can present that + refusal to the user as "could not verify the other instance". + + SIGKILL escalation is **not** done here; that decision belongs to the + UI layer (gated behind a confirmation prompt because the first + instance may legitimately be inside Serval/live-cli teardown). + """ + if not is_other_instance_alive(pid, app_markers=app_markers): + logger.warning("Refusing to signal pid=%d: not a live TimePix UI process", pid) + return False + + logger.warning("Existing instance detected at pid=%d; sending SIGTERM", pid) + try: + os.kill(pid, signal.SIGTERM) + except ProcessLookupError: + return True + except PermissionError: + logger.warning("Permission denied sending SIGTERM to pid=%d", pid) + return False + + deadline = time.monotonic() + timeout_s + while time.monotonic() < deadline: + try: + if not _proc_is_live(psutil.Process(pid)): + return True + except psutil.NoSuchProcess: + return True + time.sleep(poll_interval_s) + + try: + return not _proc_is_live(psutil.Process(pid)) + except psutil.NoSuchProcess: + return True + + +def force_kill_other_instance(pid: int) -> bool: + """Send SIGKILL to ``pid`` (use only after :func:`terminate_other_instance`). + + Same uid + cmdline guard as the SIGTERM path. Returns True if the + process is gone after the signal. + """ + if not is_other_instance_alive(pid): + return True # already gone + + try: + os.kill(pid, signal.SIGKILL) + except ProcessLookupError: + return True + except PermissionError: + logger.warning("Permission denied sending SIGKILL to pid=%d", pid) + return False + + # SIGKILL is uncatchable; the process is gone by the time the syscall + # returns. A short poll handles zombie reaping race-windows. + for _ in range(10): + try: + if not _proc_is_live(psutil.Process(pid)): + return True + except psutil.NoSuchProcess: + return True + time.sleep(0.05) + return False diff --git a/src/splash_timepix/ui/theme.py b/src/splash_timepix/ui/theme.py index 858e2fa..26b547c 100644 --- a/src/splash_timepix/ui/theme.py +++ b/src/splash_timepix/ui/theme.py @@ -158,9 +158,35 @@ def input_style() -> str: QLineEdit:focus, QSpinBox:focus, QDoubleSpinBox:focus, QComboBox:focus {{ border-color: {BLUE_PRIMARY}; }} + QLineEdit:disabled, QSpinBox:disabled, QDoubleSpinBox:disabled, QComboBox:disabled {{ + background-color: {BG_BUTTON_GROUP}; + color: {TEXT_MUTED}; + border-color: {BORDER_SUBTLE}; + }} """ def heatmap_background_style() -> str: """Style for heatmap display area.""" return f"background-color: {BG_DARK}; border: 1px solid {BORDER_SUBTLE};" + + +def checkable_button_style() -> str: + """Style for mutually-exclusive checkable tool buttons (e.g. zoom modes).""" + return f""" + QPushButton {{ + background-color: {BG_WIDGET}; + color: {TEXT_PRIMARY}; + padding: 6px 12px; + border: 1px solid {BORDER_DEFAULT}; + border-radius: 4px; + }} + QPushButton:checked {{ + background-color: {BLUE_LIGHT_2}; + color: white; + border: 1px solid {BLUE_LIGHT_2}; + }} + QPushButton:hover:!checked {{ + background-color: {BG_BUTTON_GROUP}; + }} + """ diff --git a/src/splash_timepix/ui/timeline_tab.py b/src/splash_timepix/ui/timeline_tab.py new file mode 100644 index 0000000..1ae3d67 --- /dev/null +++ b/src/splash_timepix/ui/timeline_tab.py @@ -0,0 +1,63 @@ +"""Timeline tab — integrated, time-ordered view of all subsystem logs. + +Every line that passes through ``LogManager`` (all six sources) appears here +in arrival order, already carrying the ISO-8601 timestamp + ``[source]`` tag. +This gives a single scrollable transcript that can be used to reconstruct the +exact sequence of events across subsystems without opening individual files. +""" + +import logging + +from PySide6.QtCore import Slot +from PySide6.QtWidgets import QHBoxLayout, QLabel, QPushButton, QVBoxLayout, QWidget + +from . import theme +from .widgets import TerminalWidget + +logger = logging.getLogger(__name__) + + +class TimelineTab(QWidget): + """Full-width integrated log view (All Logs, time-ordered).""" + + def __init__(self, parent=None): + super().__init__(parent) + self._setup_ui() + + def _setup_ui(self): + layout = QVBoxLayout(self) + layout.setSpacing(6) + layout.setContentsMargins(8, 8, 8, 8) + + self._terminal = TerminalWidget("All Logs (Integrated)") + self._terminal.set_status("live", True) + + # Toolbar + toolbar = QHBoxLayout() + + info = QLabel("All subsystem logs — time-ordered, ISO-8601 timestamps") + info.setStyleSheet(f"color: {theme.TEXT_MUTED}; font-size: 11px;") + toolbar.addWidget(info) + + toolbar.addStretch() + + clear_btn = QPushButton("Clear View") + clear_btn.setStyleSheet(theme.secondary_button_style()) + clear_btn.clicked.connect(self._on_clear_clicked) + toolbar.addWidget(clear_btn) + + layout.addLayout(toolbar) + layout.addWidget(self._terminal) + + def _on_clear_clicked(self) -> None: + logger.info("Timeline tab: clear view requested") + self._terminal.clear() + + @Slot(str, str) + def on_log_line(self, source: str, formatted_line: str) -> None: + """Append a pre-formatted line from LogManager to the integrated view.""" + self._terminal.append_text(formatted_line) + + def clear(self): + """Clear the terminal widget (called when the Logs tab clears all).""" + self._terminal.clear() diff --git a/src/splash_timepix/ui/widgets.py b/src/splash_timepix/ui/widgets.py index 696bf62..042957f 100644 --- a/src/splash_timepix/ui/widgets.py +++ b/src/splash_timepix/ui/widgets.py @@ -1,13 +1,12 @@ """Custom widgets for the TimePix3 UI.""" import logging -from datetime import datetime from functools import lru_cache from typing import Optional import numpy as np -from PySide6.QtCore import Qt, QTimer -from PySide6.QtGui import QColor, QImage, QPainter, QPixmap +from PySide6.QtCore import QPoint, QPointF, QRect, Qt, QTimer, Signal +from PySide6.QtGui import QColor, QImage, QPainter, QPen, QPixmap, QPolygonF from PySide6.QtWidgets import QFrame, QHBoxLayout, QLabel, QPlainTextEdit, QSizePolicy, QVBoxLayout, QWidget from . import theme @@ -146,13 +145,513 @@ def paintEvent(self, event): painter.drawText(-text_width // 2, text_height // 4, self._text) +# ============================================================================= +# Ruler Widget +# ============================================================================= + + +class _Ruler(QWidget): + """Ruler strip. Horizontal: tick marks + normalized labels (mantissa only). Vertical: tick marks only.""" + + _THICKNESS_H = 26 # height of horizontal ruler (room for ticks + labels) + _THICKNESS_V = 14 # width of vertical ruler (ticks only) + _TICK_MAJOR = 8 + _TICK_MINOR = 4 + + def __init__(self, orientation: Qt.Orientation, parent=None): + super().__init__(parent) + self._orientation = orientation + self._view_start = 0 + self._view_stop = 100 + self._scale = 1.0 + self._offset = 0.0 + if orientation == Qt.Orientation.Horizontal: + self.setFixedHeight(self._THICKNESS_H) + self.setSizePolicy(QSizePolicy.Policy.Expanding, QSizePolicy.Policy.Fixed) + else: + self.setFixedWidth(self._THICKNESS_V) + self.setSizePolicy(QSizePolicy.Policy.Fixed, QSizePolicy.Policy.Expanding) + + def set_range(self, start: int, stop: int) -> None: + self._view_start = start + self._view_stop = stop + self.update() + + def set_scale(self, scale: float, offset: float = 0.0) -> None: + self._scale = scale + self._offset = offset + self.update() + + @staticmethod + def _nice_interval(span: int) -> int: + for scale in (1, 2, 5, 10, 20, 50, 100, 200, 500, 1000, 2000, 5000, 10000, 20000, 50000, 100000): + if span / scale <= 10: + return scale + return max(1, span // 8) + + def paintEvent(self, event): + painter = QPainter(self) + painter.fillRect(self.rect(), QColor(theme.BG_WIDGET)) + + is_h = self._orientation == Qt.Orientation.Horizontal + + if is_h: + font = painter.font() + font.setPixelSize(8) + painter.setFont(font) + metrics = painter.fontMetrics() + label_h = metrics.height() + + painter.setPen(QColor(theme.TEXT_MUTED)) + + span = max(1, self._view_stop - self._view_start) + length = self.width() if is_h else self.height() + thickness = self.height() if is_h else self.width() + + interval = self._nice_interval(span) + minor_interval = max(1, interval // 5) + + # Border line + if is_h: + painter.drawLine(0, 0, length - 1, 0) + else: + painter.drawLine(thickness - 1, 0, thickness - 1, length - 1) + + last_label_end = -999 + d = (self._view_start // minor_interval) * minor_interval + while d <= self._view_stop: + if d >= self._view_start: + frac = (d - self._view_start) / span + pos = int(frac * length) + is_major = d % interval == 0 + tick_len = self._TICK_MAJOR if is_major else self._TICK_MINOR + if is_h: + painter.drawLine(pos, 0, pos, tick_len - 1) + if is_major: + label = f"{self._offset + d * self._scale:.2f}" + lw = metrics.horizontalAdvance(label) + lx = max(0, min(pos - lw // 2, length - lw)) + if lx >= last_label_end + 2: + painter.drawText(lx, tick_len + 1, lw, label_h, Qt.AlignmentFlag.AlignLeft, label) + last_label_end = lx + lw + else: + painter.drawLine(thickness - tick_len, pos, thickness - 1, pos) + d += minor_interval + + +# ============================================================================= +# Heatmap Canvas (image + draggable cursor pairs) +# ============================================================================= + +CURSOR_COLORS = ("#00FFFF", "#FF8C00", "#39FF14", "#FF00FF", "#FFE600") # cyan, orange, lime, magenta, yellow + + +class _HeatmapCanvas(QWidget): + """Heatmap canvas with viewport-based downsampling, LabVIEW-style zoom, and right-click pan. + + Stores full-resolution float32 data for saving. Renders a downsampled UI copy + clipped to the current view [x0, x1, y0, y1] in data-index coordinates. + """ + + cursors_changed = Signal(int, float, float) # pair_idx, frac_a, frac_b + view_changed = Signal(int, int, int, int) # x0, x1, y0, y1 (data indices) + + _HIT_PX = 8 + _ZOOM_MODES = ("rect", "h", "v") + + def __init__(self, parent=None): + super().__init__(parent) + + # Full-resolution data (kept for saving; never downsampled) + self._data_full: Optional[np.ndarray] = None # (n_rows, n_cols) float32 + self._vmin = 0.0 + self._vmax = 1.0 + self._colormap: np.ndarray = get_colormap("viridis") + + # View: [x0, x1, y0, y1] in data-index coords (x=cols, y=rows) + self._view: list[int] = [0, 1, 0, 1] + + # Downsampled UI pixmap + self._pixmap_ui: Optional[QPixmap] = None + self._rgb_buf: Optional[np.ndarray] = None # keeps buffer alive for QImage + + # Cursor state + self._cursors_visible = False + self._cursors: list[list[float]] = [[0.05, 0.15], [0.25, 0.35], [0.45, 0.55], [0.65, 0.75], [0.85, 0.95]] + self._cursors_active: list[bool] = [True, True, False, False, False] + self._drag: Optional[tuple[int, int]] = None + # Read-only cursor overlay mirrored from the other heatmap + self._overlay_cursors: list[tuple[float, float]] = [] + self._overlay_active: list[bool] = [] + # Vertical cursor overlay from spectrum plot (x-fractions) + self._vcursor_fracs: Optional[tuple[float, float]] = None + self._vcursors_on_heatmap: bool = True + self._cursor_time_scale: float = 0.0 # 0 = no time labels + self._cursor_time_offset: float = 0.0 + + # Zoom state + self._zoom_mode: str = "rect" + self._zoom_start: Optional[QPoint] = None + self._zoom_end: Optional[QPoint] = None + self._zooming = False + + # Pan state (right-click drag) + self._panning = False + self._pan_start: Optional[QPoint] = None + self._pan_view_start: Optional[list[int]] = None + + self.setMinimumSize(200, 150) + sp = QSizePolicy(QSizePolicy.Policy.Expanding, QSizePolicy.Policy.Preferred) + sp.setHeightForWidth(True) + self.setSizePolicy(sp) + self.setMouseTracking(True) + self.setContextMenuPolicy(Qt.ContextMenuPolicy.PreventContextMenu) + + def hasHeightForWidth(self) -> bool: + return True + + def heightForWidth(self, width: int) -> int: + return width * 4 // 3 # 3:4 portrait + + # ------------------------------------------------------------------ + # Public API + # ------------------------------------------------------------------ + + def set_display_data(self, data: np.ndarray, vmin: float, vmax: float, colormap: np.ndarray) -> None: + """Accept full-res display data. Resets view only when shape changes.""" + shape_changed = self._data_full is None or data.shape != self._data_full.shape + self._data_full = data + self._vmin = vmin + self._vmax = vmax + self._colormap = colormap + if shape_changed: + n_rows, n_cols = data.shape + self._view = [0, n_cols, 0, n_rows] + self._render() + + def clear_data(self) -> None: + self._data_full = None + self._pixmap_ui = None + self._view = [0, 1, 0, 1] + self.update() + + def set_zoom_mode(self, mode: str) -> None: + self._zoom_mode = mode + self._zooming = False + self._zoom_start = self._zoom_end = None + self.setCursor(Qt.CursorShape.CrossCursor) + + def set_view(self, x0: int, x1: int, y0: int, y1: int) -> None: + """Set view externally (linked zoom). Does NOT re-emit view_changed.""" + self._view = [x0, x1, y0, y1] + self._render() + + def reset_view(self) -> None: + if self._data_full is None: + return + n_rows, n_cols = self._data_full.shape + self._view = [0, n_cols, 0, n_rows] + self._render() + self.view_changed.emit(*self._view) + + def get_view(self) -> tuple[int, int, int, int]: + """Current data-index view as (x0, x1, y0, y1).""" + x0, x1, y0, y1 = self._view + return (int(x0), int(x1), int(y0), int(y1)) + + def set_cursor_time_scale(self, scale: float, offset: float = 0.0) -> None: + self._cursor_time_scale = scale + self._cursor_time_offset = offset + self.update() + + def set_cursors_visible(self, visible: bool) -> None: + self._cursors_visible = visible + self.update() + + def set_cursor_pair_active(self, pair_idx: int, active: bool) -> None: + if 0 <= pair_idx < len(self._cursors_active): + self._cursors_active[pair_idx] = active + self.update() + + def set_overlay_cursors(self, cursors: list[tuple[float, float]], active: list[bool]) -> None: + self._overlay_cursors = cursors + self._overlay_active = active + self.update() + + def set_vcursor_overlay(self, frac_a: float, frac_b: float) -> None: + self._vcursor_fracs = (frac_a, frac_b) + self.update() + + def set_vcursors_on_heatmap(self, visible: bool) -> None: + self._vcursors_on_heatmap = visible + self.update() + + def get_cursor_fracs(self) -> list[tuple[float, float]]: + return [(c[0], c[1]) for c in self._cursors] + + # ------------------------------------------------------------------ + # Rendering + # ------------------------------------------------------------------ + + def _render(self) -> None: + if self._data_full is None or self.width() < 1 or self.height() < 1: + return + + n_rows, n_cols = self._data_full.shape + x0, x1, y0, y1 = self._view + + # Clamp view to data bounds + x0 = max(0, min(x0, n_cols - 1)) + x1 = max(x0 + 1, min(x1, n_cols)) + y0 = max(0, min(y0, n_rows - 1)) + y1 = max(y0 + 1, min(y1, n_rows)) + self._view = [x0, x1, y0, y1] + + dw, dh = self.width(), self.height() + + col_idx = np.linspace(x0, x1 - 1, dw).astype(np.int32) + row_idx = np.linspace(y0, y1 - 1, dh).astype(np.int32) + + view_data = self._data_full[np.ix_(row_idx, col_idx)] + self._rgb_buf = np.ascontiguousarray(apply_colormap(view_data, self._colormap, self._vmin, self._vmax)) + + qimg = QImage(self._rgb_buf.data, dw, dh, 3 * dw, QImage.Format.Format_RGB888) + self._pixmap_ui = QPixmap.fromImage(qimg) + self.update() + + def resizeEvent(self, event): + super().resizeEvent(event) + self._render() + + # ------------------------------------------------------------------ + # Coordinate helpers + # ------------------------------------------------------------------ + + def _widget_to_data(self, pos: QPoint) -> tuple[int, int]: + x0, x1, y0, y1 = self._view + x_d = int(x0 + pos.x() / max(1, self.width()) * (x1 - x0)) + y_d = int(y0 + pos.y() / max(1, self.height()) * (y1 - y0)) + return x_d, y_d + + def _frac_to_y(self, frac: float, rect: QRect) -> int: + return rect.top() + int(frac * rect.height()) + + def _y_to_frac(self, y: int, rect: QRect) -> float: + if rect.height() == 0: + return 0.0 + return max(0.0, min(1.0, (y - rect.top()) / rect.height())) + + def _find_nearest_cursor(self, y: int, rect: QRect) -> Optional[tuple[int, int]]: + best_dist = self._HIT_PX + 1 + best = None + for pair_idx, fracs in enumerate(self._cursors): + if not self._cursors_active[pair_idx]: + continue + for ci, frac in enumerate(fracs): + dist = abs(self._frac_to_y(frac, rect) - y) + if dist <= self._HIT_PX and dist < best_dist: + best_dist = dist + best = (pair_idx, ci) + return best + + # ------------------------------------------------------------------ + # Painting + # ------------------------------------------------------------------ + + def paintEvent(self, event): + painter = QPainter(self) + painter.setRenderHint(QPainter.RenderHint.Antialiasing) + painter.fillRect(self.rect(), QColor(theme.BG_DARK)) + + if self._pixmap_ui and not self._pixmap_ui.isNull(): + painter.drawPixmap(self.rect(), self._pixmap_ui) + + # Cursor overlays + if self._cursors_visible and self._pixmap_ui: + rect = self.rect() + lbl_font = painter.font() + lbl_font.setPixelSize(9) + has_time = self._cursor_time_scale != 0.0 + x0, x1, y0, y1 = self._view + y_span = max(1, y1 - y0) + for pair_idx, (frac_a, frac_b) in enumerate(self._cursors): + if not self._cursors_active[pair_idx]: + continue + color = QColor(CURSOR_COLORS[pair_idx]) + pen = QPen(color, 1.5, Qt.PenStyle.DotLine) + painter.setPen(pen) + for frac in (frac_a, frac_b): + y = self._frac_to_y(frac, rect) + painter.drawLine(rect.left(), y, rect.right(), y) + if has_time: + display_row = y0 + frac * y_span + t_val = self._cursor_time_offset + display_row * self._cursor_time_scale + label = f"{t_val:.3E}".replace("E", " E") + painter.setFont(lbl_font) + fm = painter.fontMetrics() + lw = fm.horizontalAdvance(label) + lh = fm.height() + lx = rect.left() + 4 + ly = max(1, y - lh - 1) + painter.fillRect(lx - 1, ly, lw + 4, lh, QColor(0, 0, 0, 130)) + painter.setPen(color) + painter.drawText(lx + 1, ly, lw, lh, Qt.AlignmentFlag.AlignLeft, label) + painter.setPen(pen) + + # Read-only cursor overlay (mirrored from the other heatmap) + if self._overlay_cursors and self._pixmap_ui: + rect = self.rect() + for pair_idx, (frac_a, frac_b) in enumerate(self._overlay_cursors): + if pair_idx >= len(self._overlay_active) or not self._overlay_active[pair_idx]: + continue + color = QColor(CURSOR_COLORS[pair_idx]) + pen = QPen(color, 1.5, Qt.PenStyle.DotLine) + painter.setPen(pen) + for frac in (frac_a, frac_b): + y = self._frac_to_y(frac, rect) + painter.drawLine(rect.left(), y, rect.right(), y) + + # Vertical cursor overlay from spectrum plot (read-only, not draggable). + # The spectrum widget x-axis spans the heatmap's *visible* x-range, so a + # spectrum cursor at widget fraction f maps to the same widget fraction of + # the heatmap canvas — no zoom math needed here. + if self._vcursor_fracs and self._vcursors_on_heatmap and self._data_full is not None and self._pixmap_ui: + rect = self.rect() + vc_pen = QPen(QColor("#D0D0D0"), 1.0, Qt.PenStyle.SolidLine) + painter.setPen(vc_pen) + for frac in self._vcursor_fracs: + screen_x = int(frac * rect.width()) + painter.drawLine(screen_x, rect.top(), screen_x, rect.bottom()) + + # Zoom rubber-band overlay + if self._zooming and self._zoom_start and self._zoom_end: + sx, sy = self._zoom_start.x(), self._zoom_start.y() + ex, ey = self._zoom_end.x(), self._zoom_end.y() + if self._zoom_mode == "h": + sy, ey = 0, self.height() + elif self._zoom_mode == "v": + sx, ex = 0, self.width() + rx, ry = min(sx, ex), min(sy, ey) + rw, rh = abs(ex - sx), abs(ey - sy) + painter.fillRect(rx, ry, rw, rh, QColor(255, 255, 255, 40)) + painter.setPen(QPen(QColor(255, 255, 255, 200), 1, Qt.PenStyle.DashLine)) + painter.drawRect(rx, ry, rw, rh) + + # ------------------------------------------------------------------ + # Mouse interaction + # ------------------------------------------------------------------ + + def mousePressEvent(self, event): + if event.button() == Qt.MouseButton.RightButton: + self._panning = True + self._pan_start = event.pos() + self._pan_view_start = list(self._view) + self.setCursor(Qt.CursorShape.ClosedHandCursor) + return + + if event.button() == Qt.MouseButton.LeftButton: + # Cursor drag takes priority when cursors are visible + if self._cursors_visible: + hit = self._find_nearest_cursor(event.pos().y(), self.rect()) + if hit: + self._drag = hit + return + self._zooming = True + self._zoom_start = event.pos() + self._zoom_end = event.pos() + + def mouseMoveEvent(self, event): + # Pan + if self._panning and self._pan_start and self._pan_view_start and self._data_full is not None: + dx = event.pos().x() - self._pan_start.x() + dy = event.pos().y() - self._pan_start.y() + x0, x1, y0, y1 = self._pan_view_start + x_span, y_span = x1 - x0, y1 - y0 + n_rows, n_cols = self._data_full.shape + dx_d = int(-dx / max(1, self.width()) * x_span) + dy_d = int(-dy / max(1, self.height()) * y_span) + new_x0 = max(0, min(x0 + dx_d, n_cols - x_span)) + new_y0 = max(0, min(y0 + dy_d, n_rows - y_span)) + self._view = [new_x0, new_x0 + x_span, new_y0, new_y0 + y_span] + self._render() + self.view_changed.emit(*self._view) + return + + # Cursor drag + if self._drag is not None: + frac = self._y_to_frac(event.pos().y(), self.rect()) + pair_idx, ci = self._drag + self._cursors[pair_idx][ci] = frac + self.update() + self.cursors_changed.emit(pair_idx, self._cursors[pair_idx][0], self._cursors[pair_idx][1]) + return + + # Zoom rubber-band + if self._zooming: + self._zoom_end = event.pos() + self.update() + return + + # Cursor hover + if self._cursors_visible: + hit = self._find_nearest_cursor(event.pos().y(), self.rect()) + self.setCursor(Qt.CursorShape.SizeVerCursor if hit else Qt.CursorShape.CrossCursor) + else: + self.setCursor(Qt.CursorShape.CrossCursor) + + def mouseReleaseEvent(self, event): + if event.button() == Qt.MouseButton.RightButton and self._panning: + self._panning = False + self._pan_start = self._pan_view_start = None + self.setCursor(Qt.CursorShape.CrossCursor) + return + + if event.button() == Qt.MouseButton.LeftButton: + if self._drag is not None: + self._drag = None + return + if self._zooming and self._zoom_start and self._zoom_end and self._data_full is not None: + self._apply_zoom() + self._zooming = False + self._zoom_start = self._zoom_end = None + + def _apply_zoom(self) -> None: + sd = self._widget_to_data(self._zoom_start) + ed = self._widget_to_data(self._zoom_end) + x0, x1, y0, y1 = self._view + n_rows, n_cols = self._data_full.shape + + new_x0 = max(0, min(sd[0], ed[0])) + new_x1 = min(n_cols, max(sd[0], ed[0])) + new_y0 = max(0, min(sd[1], ed[1])) + new_y1 = min(n_rows, max(sd[1], ed[1])) + + if self._zoom_mode == "h": + new_y0, new_y1 = y0, y1 + elif self._zoom_mode == "v": + new_x0, new_x1 = x0, x1 + + # Reject degenerate selections (< 2 data points) + if new_x1 - new_x0 < 2: + new_x0, new_x1 = x0, x1 + if new_y1 - new_y0 < 2: + new_y0, new_y1 = y0, y1 + + self._view = [new_x0, new_x1, new_y0, new_y1] + self._render() + self.view_changed.emit(*self._view) + + # ============================================================================= # Heatmap Widget # ============================================================================= class HeatmapWidget(QWidget): - """Widget that displays a 2D heatmap with colormap and axis labels.""" + """Heatmap with tick-mark rulers, linked zoom/pan, and cursor ROIs.""" + + cursors_changed = Signal(int, float, float) # pair_idx, frac_a, frac_b + view_changed = Signal(int, int, int, int) # x0, x1, y0, y1 — forwarded from canvas def __init__(self, title: str = "Heatmap", parent=None): super().__init__(parent) @@ -163,11 +662,11 @@ def __init__(self, title: str = "Heatmap", parent=None): self._auto_scale = True self._vmin = 0.0 self._vmax = 1.0 - - # Axis configuration self._time_bin_ns: Optional[float] = None self._n_bins: Optional[int] = None - + self._ev_scale: float = 1.0 # eV per pixel (raw, before normalization) + self._ev_offset: float = 0.0 # eV at pixel 0 + self._current_x_view: tuple[int, int] = (0, 1) self._setup_ui() def _setup_ui(self): @@ -179,31 +678,31 @@ def _setup_ui(self): self._title_label.setStyleSheet(f"font-weight: bold; color: {theme.TEXT_PRIMARY};") layout.addWidget(self._title_label) - # Heatmap with axis labels + # Heatmap area: [corner + Y-ruler] left | [X-ruler / canvas / x-label] right heatmap_container = QWidget() heatmap_layout = QHBoxLayout(heatmap_container) heatmap_layout.setContentsMargins(0, 0, 0, 0) - heatmap_layout.setSpacing(4) + heatmap_layout.setSpacing(2) - # Y-axis label (rotated -90 degrees) - self._y_label = VerticalLabel("Time") - self._y_label.setFixedWidth(20) + # Y axis title + self._y_label = VerticalLabel("Time (ns)") heatmap_layout.addWidget(self._y_label) - # Image and X-axis + # Center column: canvas + X ruler + x-axis label center_widget = QWidget() center_layout = QVBoxLayout(center_widget) center_layout.setContentsMargins(0, 0, 0, 0) center_layout.setSpacing(2) - self._image_label = QLabel() - self._image_label.setAlignment(Qt.AlignmentFlag.AlignCenter) - self._image_label.setMinimumSize(200, 150) - self._image_label.setSizePolicy(QSizePolicy.Policy.Expanding, QSizePolicy.Policy.Expanding) - self._image_label.setStyleSheet(theme.heatmap_background_style()) - center_layout.addWidget(self._image_label) + self._canvas = _HeatmapCanvas() + self._canvas.cursors_changed.connect(self.cursors_changed) + self._canvas.view_changed.connect(self._on_view_changed) + center_layout.addWidget(self._canvas) - self._x_label = QLabel("Energy (pixels)") + self._x_ruler = _Ruler(Qt.Orientation.Horizontal) + center_layout.addWidget(self._x_ruler) + + self._x_label = QLabel("Energy (eV)") self._x_label.setAlignment(Qt.AlignmentFlag.AlignCenter) self._x_label.setStyleSheet(f"color: {theme.TEXT_SECONDARY}; font-size: 10px;") center_layout.addWidget(self._x_label) @@ -216,70 +715,281 @@ def _setup_ui(self): self._stats_label.setStyleSheet(f"color: {theme.TEXT_MUTED}; font-size: 11px;") layout.addWidget(self._stats_label) + # ------------------------------------------------------------------ + # Internal + # ------------------------------------------------------------------ + + def _on_view_changed(self, x0: int, x1: int, y0: int, y1: int) -> None: + self._current_x_view = (x0, x1) + self._x_ruler.set_range(x0, x1) + self._update_x_axis_label(x0, x1) + self.view_changed.emit(x0, x1, y0, y1) + + def _update_x_axis_label(self, x0: int, x1: int) -> None: + v0 = self._ev_offset + x0 * self._ev_scale + v1 = self._ev_offset + x1 * self._ev_scale + vmax_abs = max(abs(v0), abs(v1)) + if vmax_abs < 1e-10: + self._x_ruler.set_scale(self._ev_scale, self._ev_offset) + self._x_label.setText("Energy (eV)") + return + exp = int(np.floor(np.log10(vmax_abs))) + factor = 10.0**exp + self._x_ruler.set_scale(self._ev_scale / factor, self._ev_offset / factor) + sign = "+" if exp >= 0 else "-" + self._x_label.setText(f"Energy (E{sign}{abs(exp):02d} eV)") + + def _update_display(self): + if self._data is None: + return + display_data = np.flipud(self._data.T.astype(np.float32)) + if self._auto_scale: + vmin, vmax = display_data.min(), display_data.max() + else: + vmin, vmax = self._vmin, self._vmax + self._canvas.set_display_data(display_data, vmin, vmax, self._colormap) + + # ------------------------------------------------------------------ + # Public API + # ------------------------------------------------------------------ + def set_colormap(self, name: str): self._colormap_name = name self._colormap = get_colormap(name) self._update_display() def set_axis_info(self, time_bin_ns: float, n_bins: int): - """Set axis scaling information for labels.""" self._time_bin_ns = time_bin_ns self._n_bins = n_bins - self._update_axis_labels() + if time_bin_ns and n_bins: + # Flipped display: row 0 = last bin → scale negative, offset = (n_bins-1)*dt + self._canvas.set_cursor_time_scale(-time_bin_ns, (n_bins - 1) * time_bin_ns) - def _update_axis_labels(self): - """Update axis labels with current scaling info.""" - if self._time_bin_ns and self._n_bins: - total_time_ns = self._time_bin_ns * self._n_bins - if total_time_ns >= 1e6: - self._y_label.set_text(f"Time ({self._time_bin_ns/1e3:.1f} µs/bin)") - else: - self._y_label.set_text(f"Time ({self._time_bin_ns:.0f} ns/bin)") + def set_x_scale(self, scale: float, offset: float = 0.0) -> None: + self._ev_scale = scale + self._ev_offset = offset + self._update_x_axis_label(*self._current_x_view) def set_data(self, data: np.ndarray, stats_text: Optional[str] = None): self._data = data - if stats_text: self._stats_label.setText(stats_text) else: total = np.sum(data) max_val = np.max(data) self._stats_label.setText(f"Total: {total:.2e} | Max: {max_val:.2e}") - self._update_display() + def enable_cursors(self, visible: bool) -> None: + self._canvas.set_cursors_visible(visible) + + def set_cursor_pair_active(self, pair_idx: int, active: bool) -> None: + self._canvas.set_cursor_pair_active(pair_idx, active) + + def update_cursor_overlay(self, cursors: list[tuple[float, float]], active: list[bool]) -> None: + self._canvas.set_overlay_cursors(cursors, active) + + def set_vcursor_overlay(self, frac_a: float, frac_b: float) -> None: + self._canvas.set_vcursor_overlay(frac_a, frac_b) + + def set_vcursors_on_heatmap(self, visible: bool) -> None: + self._canvas.set_vcursors_on_heatmap(visible) + + def get_cursor_fracs(self) -> list[tuple[float, float]]: + return self._canvas.get_cursor_fracs() + def clear(self): self._data = None - self._image_label.clear() - self._image_label.setStyleSheet(theme.heatmap_background_style()) + self._canvas.clear_data() self._stats_label.setText("No data") + self._current_x_view = (0, 1) + self._x_ruler.set_range(0, 1) + self._x_label.setText("Energy (eV)") - def _update_display(self): - if self._data is None: - return + def set_view(self, x0: int, x1: int, y0: int, y1: int) -> None: + """Receive linked view update. Updates rulers without re-emitting view_changed.""" + self._current_x_view = (x0, x1) + self._canvas.set_view(x0, x1, y0, y1) + self._x_ruler.set_range(x0, x1) + self._update_x_axis_label(x0, x1) - # Transpose and flip: time bin 0 at bottom - display_data = np.flipud(self._data.T.astype(np.float32)) + def reset_view(self) -> None: + self._canvas.reset_view() - if self._auto_scale: - vmin, vmax = display_data.min(), display_data.max() - else: - vmin, vmax = self._vmin, self._vmax + def get_view(self) -> tuple[int, int, int, int]: + """Current data-index view as (x0, x1, y0, y1).""" + return self._canvas.get_view() - rgb = apply_colormap(display_data, self._colormap, vmin, vmax) + def set_zoom_mode(self, mode: str) -> None: + self._canvas.set_zoom_mode(mode) - h, w = rgb.shape[:2] - bytes_per_line = 3 * w - qimg = QImage(rgb.data, w, h, bytes_per_line, QImage.Format.Format_RGB888) - pixmap = QPixmap.fromImage(qimg) - scaled = pixmap.scaled( - self._image_label.size(), - Qt.AspectRatioMode.KeepAspectRatio, - Qt.TransformationMode.FastTransformation, - ) +# ============================================================================= +# Spectrum Plot Widget +# ============================================================================= + + +class SpectrumPlotWidget(QWidget): + """Paints two 1D spectra (one per ROI cursor pair) with two draggable vertical cursors. + + Each spectrum is the counts-per-time-bin projection onto the energy (x) axis. + The vertical cursors emit ``cursors_changed(frac_a, frac_b)`` when moved so + callers can display pixel / eV positions. + """ + + cursors_changed = Signal(float, float) # frac_a, frac_b (x fractions in [0, 1]) - self._image_label.setPixmap(scaled) + _COLORS = CURSOR_COLORS # keep in sync with heatmap cursor colors + _LABELS = ("ROI 1", "ROI 2", "ROI 3", "ROI 4", "ROI 5") + _VCURSOR_COLOR = "#D0D0D0" # light gray — neutral against cyan/orange spectra + _HIT_PX = 8 + + # Vertical margins (px) inside the widget + _MT = 6 + _MB = 6 + + def __init__(self, parent=None): + super().__init__(parent) + self._spectra: list[Optional[np.ndarray]] = [None] * 5 + self._pair_active: list[bool] = [True, True, False, False, False] + # Vertical cursor x-fractions: 0.0 = left edge, 1.0 = right edge + self._vcursors: list[float] = [0.25, 0.75] + self._vdrag: Optional[int] = None # index of cursor being dragged + self.setSizePolicy(QSizePolicy.Policy.Expanding, QSizePolicy.Policy.Expanding) + self.setMouseTracking(True) + + # ------------------------------------------------------------------ + # Public API + # ------------------------------------------------------------------ + + def set_spectrum(self, pair_idx: int, spectrum: Optional[np.ndarray]) -> None: + self._spectra[pair_idx] = spectrum + self.update() + + def clear(self) -> None: + self._spectra = [None] * 5 + self.update() + + def set_pair_active(self, pair_idx: int, active: bool) -> None: + if 0 <= pair_idx < len(self._pair_active): + self._pair_active[pair_idx] = active + self.update() + + def get_cursor_fracs(self) -> tuple[float, float]: + """Return current vertical cursor positions as (frac_a, frac_b) in [0, 1].""" + return (self._vcursors[0], self._vcursors[1]) + + # ------------------------------------------------------------------ + # Geometry helpers + # ------------------------------------------------------------------ + + def _frac_to_cx(self, frac: float) -> int: + return int(frac * self.width()) + + def _cx_to_frac(self, x: int) -> float: + w = self.width() + if w == 0: + return 0.0 + return max(0.0, min(1.0, x / w)) + + def _find_vcursor(self, x: int) -> Optional[int]: + best_dist = self._HIT_PX + 1 + best = None + for i, frac in enumerate(self._vcursors): + dist = abs(self._frac_to_cx(frac) - x) + if dist <= self._HIT_PX and dist < best_dist: + best_dist = dist + best = i + return best + + # ------------------------------------------------------------------ + # Painting + # ------------------------------------------------------------------ + + def paintEvent(self, event): + painter = QPainter(self) + painter.setRenderHint(QPainter.RenderHint.Antialiasing) + painter.fillRect(self.rect(), QColor(theme.BG_DARK)) + + pl = 0 + pr = self.width() + pt = self._MT + pb = self.height() - self._MB + pw = pr - pl + ph = pb - pt + if pw < 4 or ph < 4: + return + + # Axes border + painter.setPen(QPen(QColor(theme.BORDER_SUBTLE), 1)) + painter.drawRect(pl, pt, pw - 1, ph - 1) + + # Valid spectra + valid = [(i, s) for i, s in enumerate(self._spectra) if s is not None and len(s) > 1 and self._pair_active[i]] + + if not valid: + f = painter.font() + f.setPixelSize(11) + painter.setFont(f) + painter.setPen(QColor(theme.TEXT_MUTED)) + painter.drawText(pl, pt, pw, ph, Qt.AlignmentFlag.AlignCenter, "Drag cursors on average heatmap") + # Still draw vertical cursors so user can position them before data arrives + else: + # Y scale across all active spectra — min to max + ymin = min(float(s.min()) for _, s in valid) + ymax = max(float(s.max()) for _, s in valid) + y_range = ymax - ymin + if y_range <= 0: + y_range = 1.0 + + # Horizontal grid line at 50 % + grid_y = pb - int(0.5 * ph) + painter.setPen(QPen(QColor(theme.BORDER_SUBTLE), 1, Qt.PenStyle.DotLine)) + painter.drawLine(pl + 1, grid_y, pr - 1, grid_y) + + # Draw each spectrum as a polyline + for pair_idx, spectrum in valid: + color = QColor(self._COLORS[pair_idx]) + painter.setPen(QPen(color, 1.5)) + n = len(spectrum) + poly = QPolygonF() + for i in range(n): + x = pl + i / (n - 1) * pw + y = pb - (float(spectrum[i]) - ymin) / y_range * ph + poly.append(QPointF(x, y)) + painter.drawPolyline(poly) + + # Vertical cursors — drawn last so they sit on top of everything + vc_color = QColor(self._VCURSOR_COLOR) + vc_pen = QPen(vc_color, 1.5, Qt.PenStyle.DashLine) + painter.setPen(vc_pen) + for frac in self._vcursors: + cx = self._frac_to_cx(frac) + painter.drawLine(cx, pt, cx, pb) + + # ------------------------------------------------------------------ + # Mouse interaction + # ------------------------------------------------------------------ + + def mousePressEvent(self, event): + if event.button() != Qt.MouseButton.LeftButton: + return + hit = self._find_vcursor(event.pos().x()) + if hit is not None: + self._vdrag = hit + + def mouseMoveEvent(self, event): + if self._vdrag is not None: + self._vcursors[self._vdrag] = self._cx_to_frac(event.pos().x()) + self.update() + self.cursors_changed.emit(self._vcursors[0], self._vcursors[1]) + else: + hit = self._find_vcursor(event.pos().x()) + self.setCursor(Qt.CursorShape.SizeHorCursor if hit is not None else Qt.CursorShape.ArrowCursor) + + def mouseReleaseEvent(self, event): + if event.button() == Qt.MouseButton.LeftButton: + self._vdrag = None # ============================================================================= @@ -448,12 +1158,18 @@ def _setup_ui(self): layout.addWidget(self._output) def append_text(self, text: str): + """Append ``text`` to the terminal. + + Lines arriving here are expected to be pre-formatted by ``LogManager`` + (ISO-8601 timestamp + source tag already present). The method accepts + multi-line text and skips blank lines, but adds no extra timestamp. + """ stripped = text.rstrip("\n") if not stripped: return for line in stripped.split("\n"): - ts = datetime.now().strftime("%H:%M:%S") - self._output.appendPlainText(f"[{ts}] {line}") + if line: + self._output.appendPlainText(line) scrollbar = self._output.verticalScrollBar() scrollbar.setValue(scrollbar.maximum()) diff --git a/src/splash_timepix/ui/workers.py b/src/splash_timepix/ui/workers.py index b9f5e7e..fa92f6a 100644 --- a/src/splash_timepix/ui/workers.py +++ b/src/splash_timepix/ui/workers.py @@ -3,10 +3,14 @@ Workers communicate with the UI via Qt signals to ensure thread safety. """ +import importlib.util import logging +import sys +import threading import time from dataclasses import dataclass from pathlib import Path +from types import ModuleType from typing import Any, Dict, Optional, Tuple import msgpack @@ -113,7 +117,11 @@ def run(self): # Control messages (start/stop) are single-part; data (event) messages are multi-part is_data_message = msg_type != "start" and msg_type != "stop" if not is_data_message: - # Start or stop: no second part, skip to next message + logger.info( + "ZMQ %s received for scan: %s", + msg_type, + metadata.get("scan_name", "?"), + ) continue # Data message: receive second part (array bytes) @@ -332,8 +340,11 @@ def start_streaming_server( tdc_channel: int = 1, tdc_edge: str = "rising", callback_batch_size: int = 10_000, + n_bins: int = 350, collapse_y: bool = True, exit_on_disconnect: bool = True, + alignment: bool = False, + alignment_rate_hz: float = 30.0, ) -> bool: args = [ "-m", @@ -346,15 +357,22 @@ def start_streaming_server( tdc_edge, "--callback-batch-size", str(int(callback_batch_size)), + "--n-bins", + str(n_bins), ] - if collapse_y: + if alignment: + # Alignment mode forces n_bins=1 and ignores collapse_y server-side; + # we still pass --n-bins above (harmless override below) so the CLI + # signature stays uniform across modes. + args += ["--alignment", "--alignment-rate-hz", str(float(alignment_rate_hz))] + elif collapse_y: args.append("--collapse-y") if exit_on_disconnect: args.append("--exit-on-disconnect") return self._start_process( name="streaming", - program="python", + program=sys.executable, args=args, working_dir=self._project_root, ) @@ -376,7 +394,7 @@ def start_simulator(self, tdc_frequency: float = 1.0, cps: float = 1000.0, durat return self._start_process( name="simulator", - program="python", + program=sys.executable, args=args, working_dir=self._project_root, ) @@ -389,10 +407,15 @@ def start_live_cli(self, replay_file: Optional[str] = None) -> bool: logger.error(f"live-cli not found: {live_cli}") return False - # parameters default - args = [] - # parameters recommended by Henrique 2025-12-15 - # args = ["--bin-width-exp", 0, "--max-delay-bins", 12] + # parameters recommended by Henrique 2025-12-15. Default would be [] + # (i.e. live-cli's own defaults, --bin-width-exp 2 --max-delay-bins 3). + # NOTE 2026-05-01: enabling these to test whether they affect the + # ~45 s upstream-bolus pattern we see in /tmp/flush_pacing_*.json. + # Math suggests they shouldn't (default and these both give ~5 ms + # of total sort-buffer latency — same depth, just different bin + # granularity), but enabling them removes one variable. Revert by + # setting `args = []` below. + args = ["--bin-width-exp", "0", "--max-delay-bins", "12"] if replay_file: args = ["--source-files", replay_file] @@ -419,7 +442,7 @@ def start_acquisition(self, duration: int, output_dir: str, preview: bool = Fals return self._start_process( name="acquisition", - program="python", + program=sys.executable, args=args, working_dir=self._project_root, ) @@ -481,3 +504,120 @@ def _on_output(self, name: str, proc: QProcess) -> None: data = proc.readAllStandardOutput().data().decode("utf-8", errors="replace") if data: self.process_output.emit(name, data) + + +# ============================================================================= +# Centroider sweep (tools/centroider) integration +# ============================================================================= + +# Project root: .../splash_timepix_dev (workers.py is at src/splash_timepix/ui/). +_PROJECT_ROOT = Path(__file__).resolve().parent.parent.parent.parent +_CENTROIDER_DIR = _PROJECT_ROOT / "tools" / "centroider" + +_centroider_api: Optional[ModuleType] = None + + +def import_centroider_api() -> ModuleType: + """Import the centroider backend (tools/centroider/api.py) in-process. + + The centroider package lives outside the installed ``splash_timepix`` + package, so its directory is added to ``sys.path`` (api.py also relies on + that for its sibling imports) and the module is loaded from its file path. + The module is cached after the first successful import. + """ + global _centroider_api + if _centroider_api is not None: + return _centroider_api + + api_path = _CENTROIDER_DIR / "api.py" + if not api_path.exists(): + raise FileNotFoundError(f"centroider backend not found at: {api_path}") + + if str(_CENTROIDER_DIR) not in sys.path: + sys.path.insert(0, str(_CENTROIDER_DIR)) + + spec = importlib.util.spec_from_file_location("centroider_api", api_path) + if spec is None or spec.loader is None: + raise ImportError(f"Could not load centroider backend from {api_path}") + module = importlib.util.module_from_spec(spec) + # Register before exec so dataclass annotation resolution (which looks the + # module up in sys.modules via cls.__module__) succeeds. + sys.modules[spec.name] = module + spec.loader.exec_module(module) + _centroider_api = module + return module + + +class CentroiderWorker(QThread): + """Runs a centroider sweep in a background thread, calling the centroider + backend (``tools/centroider/api.run_sweep``) and relaying its progress + callbacks as Qt signals. + + All sweep orchestration lives in the centroider backend; this worker is + only the threading + signal glue. + """ + + # ProgressEvent emitted when a run begins (status is None). + progress = Signal(object) + # ProgressEvent emitted when a run finishes (status is "ok"/"failed"/"skipped"). + combo_finished = Signal(object) + # SweepResult emitted once the whole sweep completes. + sweep_finished = Signal(object) + error_occurred = Signal(str) + + def __init__( + self, + input_file: str, + eps_s: str, + eps_t: str, + tpx3dump: Optional[str] = None, + output_parent: Optional[str] = None, + parent=None, + ): + super().__init__(parent) + self._input_file = input_file + self._eps_s = eps_s + self._eps_t = eps_t + self._tpx3dump = tpx3dump + self._output_parent = output_parent + self._cancel_event = threading.Event() + + def stop(self) -> None: + """Signal the sweep and any in-flight tpx3dump process to stop.""" + self._cancel_event.set() + + def run(self) -> None: + self._cancel_event.clear() + try: + api = import_centroider_api() + except Exception as exc: # noqa: BLE001 + self.error_occurred.emit(f"Failed to load centroider backend: {exc}") + return + + # Default the output parent to the input file's directory. + output_parent = self._output_parent or str(Path(self._input_file).parent) + + def _callback(event) -> None: + if event.phase == "done": + # Final sweep-complete marker; sweep_finished covers this. + return + if event.status is None: + self.progress.emit(event) + else: + self.combo_finished.emit(event) + + try: + result = api.run_sweep( + input_file=self._input_file, + output_parent=output_parent, + eps_t_list=self._eps_t, + eps_s_list=self._eps_s, + tpx3dump=self._tpx3dump, + progress_callback=_callback, + cancel_event=self._cancel_event, + ) + except Exception as exc: # noqa: BLE001 + self.error_occurred.emit(str(exc)) + return + + self.sweep_finished.emit(result) diff --git a/src/splash_timepix/workers.py b/src/splash_timepix/workers.py index 3559940..d887461 100644 --- a/src/splash_timepix/workers.py +++ b/src/splash_timepix/workers.py @@ -359,7 +359,8 @@ def zmq_worker( # Note: Start messages sent before this sleep might be missed by late-connecting subscribers time.sleep(1.0) # Increased to 1 second to help with slow joiner problem - flush_count = 0 + # flush_number comes from app.py flush_metadata — that is the single source of truth. + last_flush_num: int = 0 while not stop_event.is_set(): # First, check for control messages (start/stop) - higher priority @@ -413,18 +414,20 @@ def zmq_worker( socket.send(metadata_bytes, zmq.SNDMORE | zmq.DONTWAIT) socket.send(array_bytes, zmq.DONTWAIT) - flush_num = flush_metadata.get("flush_number", "?") + last_flush_num = flush_metadata.get("flush_number", last_flush_num) logger.info( - f"Published flush #{flush_num}: shape={array_data.shape}, " + f"Published flush #{last_flush_num}: shape={array_data.shape}, " f"total_counts={np.sum(array_data)}, " f"cycles={flush_metadata.get('cycles_in_flush', '?')}, " f"size={len(array_bytes)/1024/1024:.2f} MB" ) except zmq.Again: - logger.warning("ZMQ send would block (no subscribers or slow subscribers), dropping flush") + logger.warning( + "ZMQ send would block, dropping flush (scan=%s)", + flush_metadata.get("scan_name", "?"), + ) - flush_count += 1 xyt_queue.task_done() except queue.Empty: @@ -453,7 +456,7 @@ def zmq_worker( except queue.Empty: break - logger.info(f"ZMQ worker published {flush_count} flushes total") + logger.info(f"ZMQ worker finished; last flush_number from app.py = {last_flush_num}") except Exception as e: logger.error(f"Fatal error in ZMQ worker: {e}", exc_info=True) diff --git a/tests/conftest.py b/tests/conftest.py index 7ca5cf1..a7a5ee6 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -1,13 +1,60 @@ """Shared pytest fixtures for the splash_timepix test suite.""" +from __future__ import annotations + +import subprocess +import sys import time +from dataclasses import dataclass, field +from pathlib import Path +from typing import List, Optional +import msgpack import pytest +import zmq +from splash_timepix.heartbeat import wait_for_ready from splash_timepix.simulator import PacketSimulator, SimulatorConfig +from splash_timepix.simulator_cli import SimulatorSource from splash_timepix.socket_server import SocketDataServer from tests.port_utils import get_free_port +REPO_ROOT = Path(__file__).resolve().parent.parent + + +def collect_messages_until_stop(sock: zmq.Socket, timeout_s: float = 12.0) -> List[dict]: + """Drain ``sock`` until a ``stop`` message arrives or ``timeout_s`` elapses. + + Returns messages in chronological order. For two-part ``event`` messages + the second (array-bytes) frame is consumed but *not* parsed; the + metadata dict is extended with ``_array_bytes`` (length in bytes) so + tests can still assert sanity without deserializing ndarrays. + """ + messages: List[dict] = [] + sock.setsockopt(zmq.RCVTIMEO, 800) + deadline = time.time() + timeout_s + while time.time() < deadline: + try: + meta_bytes = sock.recv() + except zmq.Again: + if any(m.get("msg_type") == "stop" for m in messages): + break + continue + meta = msgpack.unpackb(meta_bytes) + msg_type = meta.get("msg_type") + if msg_type in ("start", "stop"): + messages.append(meta) + if msg_type == "stop": + break + else: + try: + array_bytes = sock.recv() + meta["_array_bytes"] = len(array_bytes) + except zmq.Again: + pass + messages.append(meta) + return messages + @pytest.fixture def test_port() -> int: @@ -41,3 +88,248 @@ def simulator(): include_control_packets=False, ) return PacketSimulator(config) + + +# --------------------------------------------------------------------------- +# Integration rig: streaming server subprocess + ZMQ sub + simulator factory +# --------------------------------------------------------------------------- + + +@dataclass +class StreamingRig: + """All the moving parts needed by a subprocess-backed integration test. + + The rig owns: + + - a ``splash_timepix.app`` subprocess on dynamic ports, + - a ZMQ SUB socket already connected and subscribed to the data PUB, + - a ZMQ SUB socket already connected and subscribed to the heartbeat PUB, + - the config values (tdc_frequency / flush_interval / cps) so simulator + factories here are guaranteed to match what the server was started with. + + Use :meth:`make_simulator` to build an in-process :class:`SimulatorSource` + preconfigured for this rig, or :meth:`spawn_simulator_cli` to launch + ``splash_timepix.simulator_cli`` as a subprocess (for CLI-path coverage). + """ + + server_proc: subprocess.Popen + tcp_port: int + zmq_port: int + hb_port: int + tdc_frequency: float + flush_interval: float + cps: float + collapse_y: bool + sub_sock: zmq.Socket + hb_sock: zmq.Socket + _ctx: zmq.Context + _simulators: List[SimulatorSource] = field(default_factory=list) + _sim_procs: List[subprocess.Popen] = field(default_factory=list) + + def make_simulator( + self, + *, + cps: Optional[float] = None, + tdc_frequency: Optional[float] = None, + counting: bool = False, + ) -> SimulatorSource: + """Return a fresh :class:`SimulatorSource` matched to this rig.""" + src = SimulatorSource(host="localhost", port=self.tcp_port) + src.set_counts_per_second(cps if cps is not None else self.cps) + src.set_tdc_frequency(tdc_frequency if tdc_frequency is not None else self.tdc_frequency) + src.set_counting(counting) + self._simulators.append(src) + return src + + def spawn_simulator_cli( + self, + *, + duration: float, + cps: Optional[float] = None, + tdc_frequency: Optional[float] = None, + counting: bool = False, + tcp_batch_interval_s: float = 0.0, + ) -> subprocess.Popen: + """Spawn ``splash_timepix.simulator_cli --auto-start`` against this rig. + + ``tcp_batch_interval_s`` controls wire-level bolus batching in the + simulator: 0 (default) sends each packet immediately (the real- + hardware profile the existing suite was designed around), >0 + buffers bytes for that many seconds then emits as one sendall + (reproduces the Serval+luna-iterator regime for the bursty-wire + regression test). + """ + cmd = [ + sys.executable, + "-m", + "splash_timepix.simulator_cli", + "--auto-start", + "--port", + str(self.tcp_port), + "--tdc-frequency", + str(tdc_frequency if tdc_frequency is not None else self.tdc_frequency), + "--cps", + str(cps if cps is not None else self.cps), + # simulator_cli's --duration is typed as int; round up so callers + # can still pass floats for ergonomics. + "--duration", + str(max(1, int(round(duration)))), + ] + if not counting: + cmd.append("--no-count") + if tcp_batch_interval_s > 0: + cmd += ["--tcp-batch-interval", str(tcp_batch_interval_s)] + proc = subprocess.Popen( + cmd, + stdout=subprocess.DEVNULL, + stderr=subprocess.PIPE, + cwd=REPO_ROOT, + ) + self._sim_procs.append(proc) + return proc + + +@pytest.fixture +def streaming_rig(): + """Factory fixture for a streaming-server integration rig. + + Usage:: + + def test_something(streaming_rig): + rig = streaming_rig(tdc_frequency=10, cps=1000, flush_interval=1.0) + src = rig.make_simulator() + src.run_blocking(2.0) + msgs = _collect_zmq_until_stop(rig.sub_sock) + ... + + Single source of truth for test config: the params passed here are + propagated to both the server subprocess CLI flags and any simulator + created via :meth:`StreamingRig.make_simulator` / + :meth:`StreamingRig.spawn_simulator_cli`, so the TDC frequency the + server expects can never drift from the one the simulator emits. + """ + rigs: List[StreamingRig] = [] + + def _factory( + *, + tdc_frequency: float = 10.0, + cps: float = 1000.0, + flush_interval: float = 1.0, + exit_on_disconnect: bool = False, + collapse_y: bool = False, + tdc_channel: int = 0, + tdc_edge: str = "rising", + ready_timeout: float = 10.0, + ) -> StreamingRig: + tcp_port = get_free_port() + zmq_port = get_free_port() + hb_port = get_free_port() + + cmd = [ + sys.executable, + "-m", + "splash_timepix.app", + "--host", + "localhost", + "--port", + str(tcp_port), + "--zmq-port", + str(zmq_port), + "--heartbeat-port", + str(hb_port), + "--tdc-frequency", + str(tdc_frequency), + "--flush-interval", + str(flush_interval), + "--tdc-ch", + str(tdc_channel), + "--tdc-edge", + tdc_edge, + ] + if collapse_y: + cmd.append("--collapse-y") + if exit_on_disconnect: + cmd.append("--exit-on-disconnect") + + # Server stdout/stderr → DEVNULL. PIPE would deadlock the server + # once its OS buffer fills (nothing in the test process drains it) + # and our high-rate tests run long enough to make that a real risk. + # To debug failures, swap these to an open log file handle (and set + # stderr=subprocess.STDOUT) — the server logs heartbeat state + # transitions, client connect/disconnect, and each flush. + server_proc = subprocess.Popen( + cmd, + stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL, + cwd=REPO_ROOT, + ) + + # Wait for the server's own READY heartbeat before proceeding. The + # server publishes READY *before* it enters its 2 s "wait for subs to + # connect" sleep, so seeing READY is only half the story: the main + # loop isn't iterating yet when this returns. + if not wait_for_ready(port=hb_port, timeout=ready_timeout): + server_proc.terminate() + pytest.fail(f"streaming server did not reach ready within {ready_timeout}s") + + ctx = zmq.Context() + sub_sock = ctx.socket(zmq.SUB) + sub_sock.connect(f"tcp://127.0.0.1:{zmq_port}") + sub_sock.setsockopt(zmq.SUBSCRIBE, b"") + + hb_sock = ctx.socket(zmq.SUB) + hb_sock.connect(f"tcp://127.0.0.1:{hb_port}") + hb_sock.setsockopt(zmq.SUBSCRIBE, b"") + + # Cover the server's internal `wait_after_start = 2 s` window (the + # deliberate slow-joiner grace period it sleeps between READY and the + # main loop's first iteration). Matches the old hardcoded + # time.sleep(2.5) in pre-rig tests but is now anchored to the READY + # signal above rather than to absolute wall-clock time since spawn. + time.sleep(2.2) + + rig = StreamingRig( + server_proc=server_proc, + tcp_port=tcp_port, + zmq_port=zmq_port, + hb_port=hb_port, + tdc_frequency=tdc_frequency, + flush_interval=flush_interval, + cps=cps, + collapse_y=collapse_y, + sub_sock=sub_sock, + hb_sock=hb_sock, + _ctx=ctx, + ) + rigs.append(rig) + return rig + + yield _factory + + for rig in rigs: + for sim_proc in rig._sim_procs: + if sim_proc.poll() is None: + sim_proc.terminate() + try: + sim_proc.wait(timeout=5) + except subprocess.TimeoutExpired: + sim_proc.kill() + for sim in rig._simulators: + try: + sim.stop_auto_sending() + except Exception: + pass + + if rig.server_proc.poll() is None: + rig.server_proc.terminate() + try: + rig.server_proc.wait(timeout=8) + except subprocess.TimeoutExpired: + rig.server_proc.kill() + + try: + rig.sub_sock.close() + rig.hb_sock.close() + rig._ctx.term() + except Exception: + pass diff --git a/tests/test_flush_burstiness.py b/tests/test_flush_burstiness.py new file mode 100644 index 0000000..b25a32f --- /dev/null +++ b/tests/test_flush_burstiness.py @@ -0,0 +1,505 @@ +"""Bursty-wire regression test for flush pacing (server commit 754c857). + +Asserts the post-fix invariants under a 2 s-bolus wire pattern that +reproduces the production Serval + luna-iterator regime: + +- CV < 0.5 — flush pacing must be near-uniform +- fraction_sub_50ms < 0.1 — no microsecond-clustered emits +- flush_numbers contiguous 1..N, total_cycles monotonic +- sum(event.cycles_in_flush) == stop.total_cycles (conservation) + +For exploratory grid diagnostics (5×5 cps/tdc sweep) see: + tools/diagnostics/flush_burstiness.py + +Running +------- + +:: + + pytest -v -s tests/test_flush_burstiness.py -m slow +""" + +from __future__ import annotations + +import json +import os +import statistics +import subprocess +import time +from dataclasses import dataclass, field +from typing import List, Tuple + +import msgpack +import pytest +import zmq + +# Wallclock budget for the simulator's wire stream to drain on the receiver. +# Must exceed the nominal DAQ window by enough headroom to cover TCP back- +# pressure at the highest-load combo (cps=100k, tdc=10k Hz, tcp_batch=2 s): +# the server's parse + per-flush ZMQ publish saturates the recv loop, which +# stalls the simulator's sendall and stretches a 15 s synthetic stream to +# 75 s+ of wallclock on slower hosts (observed locally; ~40 s on the GitHub +# Actions ubuntu-latest runner). This budget is only a *ceiling* — the +# collect loop breaks the instant the stop message arrives — so generous +# headroom is essentially free on passing runs and only bounds a genuine +# hang. 150 s ≈ 10x DAQ keeps comfortable margin across host speeds. +SIM_WAIT_BUDGET_S: float = 150.0 + +# After the sim closes its TCP socket the server still needs to flush its +# ingest queue, take the final cycle-count snapshot, and publish the stop +# message on the ZMQ PUB socket. Added on top of SIM_WAIT_BUDGET_S as the +# post-drain grace window for that final stop to land. +STOP_WAIT_BUDGET_S: float = 20.0 + + +# ============================================================================= +# Per-combo result container +# ============================================================================= + + +@dataclass +class ComboResult: + cps: float + tdc: float + flush_interval_s: float + expected_flushes: float + + n_events: int = 0 + n_starts: int = 0 + n_stops: int = 0 + event_arrival_monotonic: List[float] = field(default_factory=list) + cycles_in_flush_values: List[int] = field(default_factory=list) + total_cycles_last: int = 0 + # Populated from the stop control message (TimePixStop.total_cycles). + # Used by the bursty-wire regression test to verify + # sum(event.cycles_in_flush) == stop.total_cycles. + stop_total_cycles: int = 0 + flush_numbers: List[int] = field(default_factory=list) + total_cycles_history: List[int] = field(default_factory=list) + + notes: str = "" + + @property + def deltas_s(self) -> List[float]: + if len(self.event_arrival_monotonic) < 2: + return [] + ts = self.event_arrival_monotonic + return [ts[i + 1] - ts[i] for i in range(len(ts) - 1)] + + @property + def stream_duration_s(self) -> float: + if len(self.event_arrival_monotonic) < 2: + return 0.0 + return self.event_arrival_monotonic[-1] - self.event_arrival_monotonic[0] + + def summary_row(self) -> str: + d = self.deltas_s + if d: + mean = statistics.fmean(d) + median = statistics.median(d) + std = statistics.pstdev(d) if len(d) > 1 else 0.0 + dmin = min(d) + dmax = max(d) + cv = std / mean if mean > 0 else 0.0 + frac_fast = sum(1 for x in d if x < 0.050) / len(d) + else: + mean = median = std = dmin = dmax = cv = frac_fast = 0.0 + + return ( + f"cps={self.cps:>8g} tdc={self.tdc:>7g} Hz | " + f"flushes={self.n_events:>4d} (exp~{self.expected_flushes:>4.1f}) " + f"stream={self.stream_duration_s:>5.2f}s | " + f"Δt mean={mean*1000:>7.1f}ms med={median*1000:>7.1f}ms " + f"std={std*1000:>7.1f}ms min={dmin*1000:>7.1f}ms max={dmax*1000:>7.1f}ms | " + f"CV={cv:>5.2f} <50ms={frac_fast*100:>5.1f}% " + f"{self.notes}" + ) + + +# ============================================================================= +# Collector: record monotonic arrival time for every SUB frame +# ============================================================================= + + +def _teardown_rig_eagerly(rig) -> None: + """Wait for a rig's subprocesses to exit, close its ZMQ sockets. + + The ``streaming_rig`` fixture only tears down after the test function + returns; since we reuse the fixture many times in a single test we have + to unwind each combo ourselves, otherwise we keep N server subprocesses + alive simultaneously. + + We run with ``exit_on_disconnect=True`` so the server is expected to exit + by itself when the simulator closes its TCP socket at the end of its + duration. This module is also executed under a sandbox that denies + ``kill()`` on subprocesses, so we deliberately do NOT send SIGTERM/SIGKILL + — we just wait. If a process refuses to exit we leak it and move on (the + fixture's final teardown will try to terminate it, which will also fail + under the sandbox, but it is best-effort on its side too). + """ + for sim_proc in rig._sim_procs: + try: + sim_proc.wait(timeout=5) + except subprocess.TimeoutExpired: + pass # accept the leak + try: + rig.server_proc.wait(timeout=8) + except subprocess.TimeoutExpired: + pass # accept the leak + + # Close sockets with LINGER=0 so closure is immediate even if buffered + # frames are pending (the server is done at this point, so any in-flight + # frames are irrelevant to the next combo). + for s in (rig.sub_sock, rig.hb_sock): + try: + s.setsockopt(zmq.LINGER, 0) + s.close() + except Exception: + pass + # Do NOT term() the context — the fixture's final teardown does that + # once, and multiple term() calls would deadlock. + + +def _collect_with_timestamps( + sock: zmq.Socket, + *, + stop_deadline_monotonic: float, +) -> ComboResult: + """Drain ``sock`` until a stop arrives or the deadline elapses. + + Records ``time.monotonic()`` at every ``recv()`` of an event-message + metadata frame. Two-part event messages have their array-bytes frame + consumed but not parsed. + """ + result = ComboResult( + cps=0.0, # filled in by caller + tdc=0.0, + flush_interval_s=0.0, + expected_flushes=0.0, + ) + + sock.setsockopt(zmq.RCVTIMEO, 500) + while time.monotonic() < stop_deadline_monotonic: + try: + meta_bytes = sock.recv() + arrived = time.monotonic() + except zmq.Again: + continue + + try: + meta = msgpack.unpackb(meta_bytes) + except Exception: + continue + + msg_type = meta.get("msg_type") + if msg_type == "start": + result.n_starts += 1 + elif msg_type == "stop": + result.n_stops += 1 + stc = meta.get("total_cycles") + if isinstance(stc, int): + result.stop_total_cycles = stc + break + else: + # Event: drain the array-bytes second frame (we don't need it). + try: + sock.recv() + except zmq.Again: + pass + result.n_events += 1 + result.event_arrival_monotonic.append(arrived) + cif = meta.get("cycles_in_flush") + if isinstance(cif, int): + result.cycles_in_flush_values.append(cif) + tc = meta.get("total_cycles") + if isinstance(tc, int): + result.total_cycles_last = tc + result.total_cycles_history.append(tc) + fn = meta.get("flush_number") + if isinstance(fn, int): + result.flush_numbers.append(fn) + + return result + + +# ============================================================================= +# Bursty-wire regression test (post-fix: asserts the green-path baseline) +# ============================================================================= +# +# Reproduces the production Serval+luna-iterator wire pattern in CI by +# driving the simulator with ``tcp_batch_interval_s > 0``: bytes are +# accumulated in-memory for that many seconds, then emitted in one +# ``sendall`` per bolus. The server then chews through the whole bolus +# back-to-back on its TCP receive thread. +# +# Under the *original* cycle-count flush gate this produced multiple +# flushes within a few milliseconds of each other (one per cycle-count +# crossing inside a single ``data_callback`` invocation), then multi- +# second silence until the next bolus — i.e. bursty ZMQ output instead of +# the one-per-flush-interval cadence the UI needs. +# +# Server commit ``754c857`` switched the flush gate to wall-clock +# (``emit_flush_if_due`` checks ``time.monotonic() - last_flush_time >= +# flush_interval``), which decouples flush cadence from TDC arrival +# cadence. This test now asserts the post-fix invariants — CV < 0.5, +# no microsecond-clustered emits, monotonic 1..N flush numbers, and +# conservation of cycles — and so a failure here means a real +# regression in the gate semantics. +# ============================================================================= + +# Parameters chosen to reproduce the production "bolus" regime on the +# wire. The simulator emits 2 s-wide boluses; with flush_interval=1.0, +# each bolus carries ~2 flush-intervals worth of data. Under the +# wall-clock gate we expect the server to space those ~2 flushes evenly +# at ~1 s intervals regardless of when the bolus arrives. Intentionally +# deviates from the smooth-wire grid above (tcp_batch_interval_s=0) to +# isolate any wire-level burst leaking through to the ZMQ side. +_BATCHED_TCP_INTERVAL_S: float = float(os.environ.get("BATCHED_TCP_INTERVAL_S", 2.0)) +_BATCHED_FLUSH_INTERVAL_S: float = float(os.environ.get("BATCHED_FLUSH", 1.0)) +_BATCHED_DAQ_SECONDS: int = int(os.environ.get("BATCHED_DAQ", 15)) + +# Three combos covering low/mid/high rate. At the high-rate combo +# (cps=100k, tdc=10k Hz) TCP backpressure stretches the wire stream to +# several times the nominal DAQ window — the test's count tolerance +# scales off observed stream_duration so this is expected and benign. +_BATCHED_COMBOS: Tuple[Tuple[int, int], ...] = ( + # (cps, tdc_frequency_hz) + (1_000, 100), + (10_000, 1_000), + (100_000, 10_000), +) + + +def _compute_metrics(combo: ComboResult) -> dict: + """Derive burstiness / conservation metrics from a ComboResult. + + Keeps the artifact schema in one place so before/after JSON diffs + against /tmp/burstiness_*.json are trivial to compute. + """ + deltas = combo.deltas_s + if deltas: + cv = (statistics.pstdev(deltas) / statistics.fmean(deltas)) if len(deltas) > 1 else 0.0 + frac_sub_50ms = sum(1 for d in deltas if d < 0.050) / len(deltas) + max_gap = max(deltas) + else: + cv = 0.0 + frac_sub_50ms = 0.0 + max_gap = 0.0 + + return { + "cps": combo.cps, + "tdc": combo.tdc, + "flush_interval_s": combo.flush_interval_s, + "tcp_batch_interval_s": _BATCHED_TCP_INTERVAL_S, + "n_events": combo.n_events, + "n_starts": combo.n_starts, + "n_stops": combo.n_stops, + "expected_flushes": combo.expected_flushes, + "cv": cv, + "fraction_sub_50ms": frac_sub_50ms, + "max_gap_s": max_gap, + "stream_duration_s": combo.stream_duration_s, + "total_cycles_reported": combo.total_cycles_last, + "stop_total_cycles": combo.stop_total_cycles, + "sum_cycles_in_flush": sum(combo.cycles_in_flush_values), + "flush_numbers": list(combo.flush_numbers), + "total_cycles_history": list(combo.total_cycles_history), + "notes": combo.notes, + } + + +def _write_artifact(metrics: List[dict], path: str) -> None: + """Dump per-combo metrics to a JSON file for before/after diffing.""" + with open(path, "w") as f: + json.dump( + { + "tcp_batch_interval_s": _BATCHED_TCP_INTERVAL_S, + "flush_interval_s": _BATCHED_FLUSH_INTERVAL_S, + "daq_seconds": _BATCHED_DAQ_SECONDS, + "combos": metrics, + }, + f, + indent=2, + sort_keys=True, + ) + + +@pytest.mark.slow +@pytest.mark.integration +def test_batched_wire_regression(streaming_rig): + """Pass/fail regression test that exercises the bursty-wire regime. + + Asserts the post-fix (server commit 754c857, wall-clock flush gate) + invariants under a 2 s-bolus wire pattern: + + - CV < 0.5 — flush pacing must be near-uniform; a regression to + the cycle-count gate would push CV to 0.84-0.96 against this + fixture. + - fraction_sub_50ms < 0.1 — no microsecond-clustered emits; + the cycle-count gate produced 7-43% sub-50ms gaps here. + - n_flushes within 0.7-1.3 of (stream_duration / flush_interval) + — expected scales with observed wire-drain duration so heavy- + load combos under TCP backpressure are not flagged for the + backpressure itself, only for genuine pacing bugs. + - flush_numbers contiguous 1..N, total_cycles monotonic, and + sum(event.cycles_in_flush) == stop.total_cycles — conservation + invariants the UI's running averages depend on. + + The test writes a JSON artifact to ``/tmp/burstiness_latest.json`` + on every run. Historical workflow: rename to + ``/tmp/burstiness_before.json`` / ``burstiness_after.json`` to diff + pacing metrics across server changes that touch the flush path. + """ + results: List[ComboResult] = [] + + print("\n\n") + print("=" * 100) + print( + f"BURSTY-WIRE REGRESSION — {len(_BATCHED_COMBOS)} combos, " + f"{_BATCHED_DAQ_SECONDS}s each, " + f"flush_interval={_BATCHED_FLUSH_INTERVAL_S}s, " + f"tcp_batch_interval_s={_BATCHED_TCP_INTERVAL_S}s" + ) + print("=" * 100) + + for combo_idx, (cps, tdc) in enumerate(_BATCHED_COMBOS, start=1): + print( + f"\n[{combo_idx}/{len(_BATCHED_COMBOS)}] " + f"cps={cps} tdc={tdc} Hz batch={_BATCHED_TCP_INTERVAL_S}s ...", + flush=True, + ) + + rig = streaming_rig( + tdc_frequency=float(tdc), + cps=float(cps), + flush_interval=_BATCHED_FLUSH_INTERVAL_S, + exit_on_disconnect=True, + collapse_y=True, + ) + + sim_proc: subprocess.Popen = rig.spawn_simulator_cli( + duration=_BATCHED_DAQ_SECONDS, + cps=cps, + tdc_frequency=tdc, + counting=False, + tcp_batch_interval_s=_BATCHED_TCP_INTERVAL_S, + ) + + deadline = time.monotonic() + SIM_WAIT_BUDGET_S + STOP_WAIT_BUDGET_S + combo = _collect_with_timestamps( + rig.sub_sock, + stop_deadline_monotonic=deadline, + ) + combo.cps = cps + combo.tdc = tdc + combo.flush_interval_s = _BATCHED_FLUSH_INTERVAL_S + + # Wall-clock flush gate (server commit 754c857): the streaming server + # publishes one flush per flush_interval of wallclock during the + # stream, so expected count tracks the *observed* stream duration — + # not the nominal DAQ window. For low-cps combos stream ≈ DAQ; for + # the high-cps combo, TCP backpressure stretches stream to several + # times the DAQ duration, and expected scales accordingly. + combo.expected_flushes = max(0.0, combo.stream_duration_s / _BATCHED_FLUSH_INTERVAL_S) + + try: + sim_proc.wait(timeout=3) + except subprocess.TimeoutExpired: + pass + + if combo.n_stops == 0: + combo.notes = "(NO stop received)" + elif combo.n_events == 0: + combo.notes = "(NO events)" + + results.append(combo) + print(" " + combo.summary_row()) + _teardown_rig_eagerly(rig) + + # ------------------------------------------------------------------------- + # Write artifact before asserting so the JSON is available even when + # the test fails (which is the Phase-1 expectation). + # ------------------------------------------------------------------------- + all_metrics = [_compute_metrics(c) for c in results] + artifact_path = os.environ.get("BURSTINESS_ARTIFACT", "/tmp/burstiness_latest.json") + _write_artifact(all_metrics, artifact_path) + print(f"\nArtifact written to {artifact_path}") + + # ------------------------------------------------------------------------- + # Assertions. Collect all failures first so the output shows every + # combo's verdict in one shot rather than aborting at the first + # broken combo — this gives the operator a complete picture of the + # current state. + # ------------------------------------------------------------------------- + failures: List[str] = [] + + for combo, m in zip(results, all_metrics): + label = f"cps={combo.cps:g}/tdc={combo.tdc:g}" + + # --- Stop sanity --------------------------------------------------- + if combo.n_stops != 1: + failures.append(f"{label}: expected exactly 1 stop, got {combo.n_stops}") + continue # the rest depend on a well-formed acquisition + + if combo.n_events == 0: + failures.append(f"{label}: no event messages produced") + continue + + # --- Count tolerance ---------------------------------------------- + low = 0.7 * combo.expected_flushes + high = 1.3 * combo.expected_flushes + if not (low <= combo.n_events <= high): + failures.append( + f"{label}: n_flushes={combo.n_events} outside " + f"[{low:.1f}, {high:.1f}] (expected~{combo.expected_flushes:.1f})" + ) + + # --- CV (burstiness) ---------------------------------------------- + if m["cv"] >= 0.5: + failures.append(f"{label}: CV={m['cv']:.3f} exceeds 0.5 threshold (bursty pacing)") + + # --- Sub-50ms fraction -------------------------------------------- + if m["fraction_sub_50ms"] >= 0.1: + failures.append( + f"{label}: fraction_sub_50ms={m['fraction_sub_50ms']:.3f} >= 0.1 (microsecond-clustered emits)" + ) + + # --- Max gap ------------------------------------------------------ + if m["max_gap_s"] >= 2.0 * _BATCHED_FLUSH_INTERVAL_S: + failures.append( + f"{label}: max_gap={m['max_gap_s']:.3f}s >= 2 * flush_interval ({2*_BATCHED_FLUSH_INTERVAL_S:.3f}s)" + ) + + # --- Flush number sequence ---------------------------------------- + expected_seq = list(range(1, len(combo.flush_numbers) + 1)) + if combo.flush_numbers != expected_seq: + failures.append( + f"{label}: flush_numbers not 1..N: {combo.flush_numbers[:8]}... (expected {expected_seq[:8]}...)" + ) + + # --- Monotonic total_cycles --------------------------------------- + tch = combo.total_cycles_history + non_monotonic = [(i, tch[i], tch[i + 1]) for i in range(len(tch) - 1) if tch[i + 1] < tch[i]] + if non_monotonic: + failures.append(f"{label}: total_cycles not monotonic at indices {non_monotonic[:3]}") + + # --- Conservation of cycles (the killer invariant) ---------------- + # Catches: off-by-one in cycles_in_flush, double-emit from TOCTOU, + # lost-emit from any source. Running-average correctness in the + # UI depends on this exactly. + sum_cif = sum(combo.cycles_in_flush_values) + if sum_cif != combo.stop_total_cycles: + failures.append( + f"{label}: sum(cycles_in_flush)={sum_cif} != " + f"stop.total_cycles={combo.stop_total_cycles} " + f"(conservation violated)" + ) + + if failures: + pytest.fail( + "\n".join( + [f"Bursty-wire regression: {len(failures)} assertion(s) failed:"] + + [f" - {f}" for f in failures] + + [f"Artifact: {artifact_path}"] + ) + ) diff --git a/tests/test_log_manager.py b/tests/test_log_manager.py new file mode 100644 index 0000000..f8da5c4 --- /dev/null +++ b/tests/test_log_manager.py @@ -0,0 +1,345 @@ +"""Tests for LogManager — disk persistence, rollover, dispatch, and formatting. + +These tests are pure Python; they do not instantiate Qt widgets or start a +QApplication, keeping the test suite fast and headless-friendly. + +``LogManager`` is a QObject, so a minimal QCoreApplication is created once for +the module. All signal emissions in the tests use direct (synchronous) +connections so we can inspect results without an event-loop spin. +""" + +from __future__ import annotations + +import logging +import re +import sys +import threading +from datetime import datetime +from unittest.mock import patch + +# --------------------------------------------------------------------------- +# Minimal QCoreApplication — must be created before any QObject +# --------------------------------------------------------------------------- +import pytest + +pytest.importorskip( + "PySide6", + reason="PySide6 (ui extra) not installed; LogManager tests need QtCore", +) + +from PySide6.QtCore import QCoreApplication # noqa: E402 + +_qapp = QCoreApplication.instance() or QCoreApplication(sys.argv[:1]) + +from splash_timepix.ui.log_manager import LogManager, _format_line, _QtLoggingHandler # noqa: E402 + +# --------------------------------------------------------------------------- +# Helpers +# --------------------------------------------------------------------------- + +_ISO_RE = re.compile(r"^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}[+-]\d{4} \[.+?\] .+$") + + +def _collect_emissions(manager: LogManager) -> list[tuple[str, str]]: + """Return a list of (source, formatted_line) received via line_emitted.""" + received: list[tuple[str, str]] = [] + manager.line_emitted.connect(lambda src, line: received.append((src, line))) + return received + + +# --------------------------------------------------------------------------- +# Formatting +# --------------------------------------------------------------------------- + + +class TestFormatLine: + def test_iso_format_matches_regex(self): + line = _format_line("streaming", "Server started") + assert _ISO_RE.match(line), f"Format mismatch: {line!r}" + + def test_source_tag_present(self): + line = _format_line("serval", "chip temps: 28.1") + assert "[serval]" in line + + def test_message_present(self): + msg = "some unique message 99" + line = _format_line("system", msg) + assert msg in line + + def test_millisecond_precision(self): + line = _format_line("acquisition", "done") + # After the time part there should be a dot followed by exactly 3 digits + assert re.search(r"T\d{2}:\d{2}:\d{2}\.\d{3}", line) + + def test_timezone_offset_present(self): + line = _format_line("zmq-backend", "flush #1") + # ends with something like +0000 or -0700 before the space + assert re.search(r"[+-]\d{4} ", line) + + +# --------------------------------------------------------------------------- +# append() — per-source file dispatch +# --------------------------------------------------------------------------- + + +class TestAppendDispatch: + def test_streaming_writes_streaming_log(self, tmp_path): + with patch("splash_timepix.ui.log_manager._LOG_ROOT_CANDIDATE", tmp_path): + mgr = LogManager() + mgr.append("streaming", "hello streaming") + mgr.close() + + today = datetime.now().strftime("%Y-%m-%d") + log_file = tmp_path / today / "streaming.log" + assert log_file.exists(), "streaming.log not created" + content = log_file.read_text() + assert "hello streaming" in content + + def test_streaming_does_not_write_serval_log(self, tmp_path): + with patch("splash_timepix.ui.log_manager._LOG_ROOT_CANDIDATE", tmp_path): + mgr = LogManager() + mgr.append("streaming", "exclusive line") + mgr.close() + + today = datetime.now().strftime("%Y-%m-%d") + serval_file = tmp_path / today / "serval.log" + if serval_file.exists(): + assert "exclusive line" not in serval_file.read_text() + + def test_all_log_receives_every_source(self, tmp_path): + with patch("splash_timepix.ui.log_manager._LOG_ROOT_CANDIDATE", tmp_path): + mgr = LogManager() + mgr.append("serval", "serval line") + mgr.append("streaming", "streaming line") + mgr.append("zmq-backend", "zmq line") + mgr.close() + + today = datetime.now().strftime("%Y-%m-%d") + all_log = tmp_path / today / "all.log" + assert all_log.exists() + content = all_log.read_text() + assert "serval line" in content + assert "streaming line" in content + assert "zmq line" in content + + def test_simulator_maps_to_live_cli_log(self, tmp_path): + with patch("splash_timepix.ui.log_manager._LOG_ROOT_CANDIDATE", tmp_path): + mgr = LogManager() + mgr.append("simulator", "sim output") + mgr.close() + + today = datetime.now().strftime("%Y-%m-%d") + live_file = tmp_path / today / "live-cli.log" + assert live_file.exists() + assert "sim output" in live_file.read_text() + + def test_multiline_text_each_line_formatted(self, tmp_path): + with patch("splash_timepix.ui.log_manager._LOG_ROOT_CANDIDATE", tmp_path): + mgr = LogManager() + mgr.append("system", "line one\nline two\nline three") + mgr.close() + + today = datetime.now().strftime("%Y-%m-%d") + sys_file = tmp_path / today / "system.log" + lines = [ln for ln in sys_file.read_text().splitlines() if ln.strip()] + assert len(lines) == 3, f"Expected 3 lines, got {lines}" + for line in lines: + assert _ISO_RE.match(line), f"Bad format: {line!r}" + + def test_blank_text_produces_no_output(self, tmp_path): + with patch("splash_timepix.ui.log_manager._LOG_ROOT_CANDIDATE", tmp_path): + mgr = LogManager() + mgr.append("system", " \n\n") + mgr.close() + + today = datetime.now().strftime("%Y-%m-%d") + sys_file = tmp_path / today / "system.log" + if sys_file.exists(): + assert sys_file.read_text().strip() == "" + + +# --------------------------------------------------------------------------- +# line_emitted signal +# --------------------------------------------------------------------------- + + +class TestLineEmitted: + def test_signal_emitted_for_each_line(self, tmp_path): + with patch("splash_timepix.ui.log_manager._LOG_ROOT_CANDIDATE", tmp_path): + mgr = LogManager() + received = _collect_emissions(mgr) + mgr.append("serval", "alpha\nbeta") + mgr.close() + + # Filter out any system records emitted by the Python logging bridge + serval_lines = [line for src, line in received if src == "serval"] + assert len(serval_lines) == 2 + + def test_signal_source_matches_input(self, tmp_path): + with patch("splash_timepix.ui.log_manager._LOG_ROOT_CANDIDATE", tmp_path): + mgr = LogManager() + received = _collect_emissions(mgr) + mgr.append("acquisition", "done") + mgr.close() + + acq_lines = [line for src, line in received if src == "acquisition"] + assert acq_lines, "No acquisition lines received" + + def test_signal_formatted_line_is_iso(self, tmp_path): + with patch("splash_timepix.ui.log_manager._LOG_ROOT_CANDIDATE", tmp_path): + mgr = LogManager() + received = _collect_emissions(mgr) + mgr.append("streaming", "check format") + mgr.close() + + for src, line in received: + if src == "streaming": + assert _ISO_RE.match(line), f"Bad format in signal: {line!r}" + + +# --------------------------------------------------------------------------- +# Midnight rollover +# --------------------------------------------------------------------------- + + +class TestRollover: + def test_rollover_creates_new_day_folder(self, tmp_path): + day1 = "2026-01-01" + day2 = "2026-01-02" + + with patch("splash_timepix.ui.log_manager._LOG_ROOT_CANDIDATE", tmp_path): + mgr = LogManager() + + fake_day1 = datetime(2026, 1, 1, 12, 0, 0).astimezone() + fake_day2 = datetime(2026, 1, 2, 0, 0, 1).astimezone() + + with patch("splash_timepix.ui.log_manager.datetime") as mock_dt: + mock_dt.now.return_value = fake_day1 + mgr.append("system", "day 1 message") + + mock_dt.now.return_value = fake_day2 + mgr.append("system", "day 2 message") + + mgr.close() + + assert (tmp_path / day1).is_dir(), f"{day1}/ not created" + assert (tmp_path / day2).is_dir(), f"{day2}/ not created" + + def test_rollover_content_is_correct_day(self, tmp_path): + day1 = "2026-02-28" + day2 = "2026-03-01" + + with patch("splash_timepix.ui.log_manager._LOG_ROOT_CANDIDATE", tmp_path): + mgr = LogManager() + + fake_day1 = datetime(2026, 2, 28, 23, 59, 59).astimezone() + fake_day2 = datetime(2026, 3, 1, 0, 0, 0).astimezone() + + with patch("splash_timepix.ui.log_manager.datetime") as mock_dt: + mock_dt.now.return_value = fake_day1 + mgr.append("streaming", "before midnight") + + mock_dt.now.return_value = fake_day2 + mgr.append("streaming", "after midnight") + + mgr.close() + + content_day1 = (tmp_path / day1 / "streaming.log").read_text() + content_day2 = (tmp_path / day2 / "streaming.log").read_text() + assert "before midnight" in content_day1 + assert "after midnight" not in content_day1 + assert "after midnight" in content_day2 + assert "before midnight" not in content_day2 + + +# --------------------------------------------------------------------------- +# Thread safety — no interleaved bytes within a single line +# --------------------------------------------------------------------------- + + +class TestConcurrency: + def test_concurrent_appends_produce_complete_lines(self, tmp_path): + """Two threads writing simultaneously must not produce split lines.""" + n_lines = 50 + + with patch("splash_timepix.ui.log_manager._LOG_ROOT_CANDIDATE", tmp_path): + mgr = LogManager() + + def write_source(source: str): + for i in range(n_lines): + mgr.append(source, f"{source}-msg-{i}") + + t1 = threading.Thread(target=write_source, args=("serval",)) + t2 = threading.Thread(target=write_source, args=("streaming",)) + t1.start() + t2.start() + t1.join() + t2.join() + mgr.close() + + today = datetime.now().strftime("%Y-%m-%d") + all_log = tmp_path / today / "all.log" + for raw_line in all_log.read_text().splitlines(): + line = raw_line.strip() + if not line: + continue + assert _ISO_RE.match(line), f"Partial/corrupt line: {line!r}" + + +# --------------------------------------------------------------------------- +# session_marker +# --------------------------------------------------------------------------- + + +class TestSessionMarker: + def test_marker_written_to_all_open_files(self, tmp_path): + with patch("splash_timepix.ui.log_manager._LOG_ROOT_CANDIDATE", tmp_path): + mgr = LogManager() + mgr.append("serval", "before clear") + mgr.session_marker("--- cleared by user ---") + mgr.close() + + today = datetime.now().strftime("%Y-%m-%d") + all_content = (tmp_path / today / "all.log").read_text() + assert "cleared by user" in all_content + + def test_marker_emitted_as_system_signal(self, tmp_path): + with patch("splash_timepix.ui.log_manager._LOG_ROOT_CANDIDATE", tmp_path): + mgr = LogManager() + received = _collect_emissions(mgr) + mgr.session_marker("test marker") + mgr.close() + + system_lines = [line for src, line in received if src == "system"] + assert any("test marker" in ln for ln in system_lines) + + +# --------------------------------------------------------------------------- +# Python logging bridge +# --------------------------------------------------------------------------- + + +class TestLoggingBridge: + def test_python_logger_reaches_system_log(self, tmp_path): + with patch("splash_timepix.ui.log_manager._LOG_ROOT_CANDIDATE", tmp_path): + mgr = LogManager() + test_logger = logging.getLogger("test.bridge") + test_logger.warning("bridge-test-warning-message") + mgr.close() + + today = datetime.now().strftime("%Y-%m-%d") + sys_file = tmp_path / today / "system.log" + assert sys_file.exists() + assert "bridge-test-warning-message" in sys_file.read_text() + + def test_no_duplicate_handlers_on_reinstantiation(self, tmp_path): + """Creating two LogManagers must leave exactly one _QtLoggingHandler.""" + with patch("splash_timepix.ui.log_manager._LOG_ROOT_CANDIDATE", tmp_path): + mgr1 = LogManager() + mgr2 = LogManager() + mgr1.close() + mgr2.close() + + root = logging.getLogger() + count = sum(1 for h in root.handlers if isinstance(h, _QtLoggingHandler)) + assert count == 1, f"Expected exactly one _QtLoggingHandler on the root logger, found {count}" diff --git a/tests/test_main_workflow.py b/tests/test_main_workflow.py index 535f4e8..5dd5a32 100644 --- a/tests/test_main_workflow.py +++ b/tests/test_main_workflow.py @@ -5,19 +5,17 @@ - **Streaming server** (`python -m splash_timepix.app`): TCP packet ingest, time-resolved binning, ZMQ PUB for ``start`` / ``event`` / ``stop``, and heartbeat PUB. - **Ready gating**: ``MainWindow._check_server_ready`` treats heartbeat ``state`` of - ``ready`` or ``streaming`` as “server up”. + ``ready`` or ``streaming`` as "server up". - **ZMQ subscriber** (``ZmqSubscriberWorker``): ignores single-part control messages, reassembles multi-part ``event`` payloads for the Operator tab. Tests here exercise that end-to-end contract (not every socket-server knob). """ -import socket +from __future__ import annotations + import subprocess -import sys -import threading import time -from pathlib import Path import msgpack import pytest @@ -25,181 +23,105 @@ from splash_timepix.heartbeat import HeartbeatPublisher, ServerState from splash_timepix.schemas import TimePixStart, TimePixStop -from splash_timepix.simulator import PacketSimulator, SimulatorConfig +from tests.conftest import collect_messages_until_stop from tests.port_utils import get_free_port -def _recv_zmq_messages(socket: zmq.Socket, until_stop: bool, deadline: float) -> dict: - """Drain the SUB socket; return counts and last stop/start metadata.""" - out = { - "start": [], - "stop": [], - "events": 0, - "scan_names": set(), - } - socket.setsockopt(zmq.RCVTIMEO, 500) - while time.time() < deadline: - if until_stop and out["stop"]: - break - try: - meta_bytes = socket.recv() - except zmq.Again: - continue - meta = msgpack.unpackb(meta_bytes) - msg_type = meta.get("msg_type") - is_data = msg_type not in ("start", "stop") - if msg_type == "start": - out["start"].append(meta) - if meta.get("scan_name"): - out["scan_names"].add(meta["scan_name"]) - elif msg_type == "stop": - out["stop"].append(meta) - elif is_data: - try: - socket.recv() - except zmq.Again: - pass - else: - out["events"] += 1 - return out - - @pytest.mark.integration @pytest.mark.slow -def test_streaming_app_full_zmq_cycle_tcp_client_matches_ui_data_path(): +def test_streaming_app_full_zmq_cycle_tcp_client_matches_ui_data_path(streaming_rig): """Same ZMQ contract the UI expects: start → event(s) → stop after TCP disconnect. - Uses a local TCP sender (no external simulator process): fastest check that - ``app.main`` + ``zmq_worker`` + binning still match ``ZmqSubscriberWorker``. + Uses the ``streaming_rig`` factory so TDC / flush-interval / collapse_y + values live in one place (here) and are propagated to both the server + subprocess and the in-process simulator — they can never drift. """ - repo_root = Path(__file__).resolve().parent.parent - tcp_port = get_free_port() - zmq_port = get_free_port() - hb_port = get_free_port() - - cmd = [ - sys.executable, - "-m", - "splash_timepix.app", - "--host", - "localhost", - "--port", - str(tcp_port), - "--zmq-port", - str(zmq_port), - "--heartbeat-port", - str(hb_port), - "--tdc-frequency", - "100", - "--flush-interval", - "0.08", - "--tdc-ch", - "1", - "--tdc-edge", - "rising", - "--collapse-y", - "--exit-on-disconnect", - ] - proc = subprocess.Popen( - cmd, - cwd=repo_root, - stdout=subprocess.DEVNULL, - stderr=subprocess.PIPE, + # High TDC + CPS avoid a known pre-existing server race at low rates + # ( < ~50 Hz TDC ) where the main loop can enter its connect block after + # data has already flowed and wipe the accumulators — unrelated to the + # factory work here, parked for a follow-up. + rig = streaming_rig( + tdc_frequency=10000.0, + flush_interval=0.5, + cps=100000.0, + tdc_channel=1, + tdc_edge="rising", + collapse_y=True, + exit_on_disconnect=True, ) - ctx = zmq.Context() - data_sub = ctx.socket(zmq.SUB) - hb_sub = ctx.socket(zmq.SUB) - try: - data_sub.connect(f"tcp://127.0.0.1:{zmq_port}") - data_sub.setsockopt(zmq.SUBSCRIBE, b"") - hb_sub.connect(f"tcp://127.0.0.1:{hb_port}") - hb_sub.setsockopt(zmq.SUBSCRIBE, b"") - - # app.py sleeps ~2s after READY for ZMQ slow joiner - time.sleep(2.6) - if proc.poll() is not None: - err = proc.stderr.read().decode(errors="replace") if proc.stderr else "" - pytest.fail(f"streaming server exited early (code={proc.returncode}): {err}") - - hb_states: list[str] = [] - hb_deadline = time.time() + 5.0 - hb_sub.setsockopt(zmq.RCVTIMEO, 500) - while time.time() < hb_deadline and "ready" not in hb_states: - try: - raw = hb_sub.recv() - hb_states.append(msgpack.unpackb(raw).get("state", "")) - except zmq.Again: - continue - assert "ready" in hb_states, f"expected heartbeat ready, saw {hb_states!r}" - - def _send_stream() -> None: - cfg = SimulatorConfig( - pixel_count_rate=4000, - tdc_frequency=100.0, - include_control_packets=False, - ) - sim = PacketSimulator(cfg) - client = socket.socket(socket.AF_INET, socket.SOCK_STREAM) - try: - client.connect(("127.0.0.1", tcp_port)) - # app.py updates heartbeat from the main loop with ~1s sleeps; keep the - # socket open long enough that STREAMING is observed before disconnect. - time.sleep(1.2) - for chunk in sim.generate_stream(0.6): - client.sendall(chunk) - time.sleep(1.2) - finally: - client.close() - - threading.Thread(target=_send_stream, daemon=True).start() - - saw_streaming = False - hb_deadline = time.time() + 15.0 - while time.time() < hb_deadline: - try: - raw = hb_sub.recv() - st = msgpack.unpackb(raw).get("state", "") - if st == "streaming": - saw_streaming = True - break - except zmq.Again: - continue - assert saw_streaming, "heartbeat never reached streaming after TCP connect" - - deadline = time.time() + 40.0 - collected = _recv_zmq_messages(data_sub, until_stop=True, deadline=deadline) + # Open the TCP connection but *don't* stream yet: heartbeat publisher + # ticks every 1 s, so we need a > 1 s window to observe STREAMING. + src = rig.make_simulator() + assert src.connect(), "failed to connect simulator to rig TCP port" + time.sleep(1.2) + saw_streaming = False + rig.hb_sock.setsockopt(zmq.RCVTIMEO, 500) + hb_deadline = time.time() + 5.0 + while time.time() < hb_deadline: try: - proc.wait(timeout=25) - except subprocess.TimeoutExpired: - proc.kill() - raise AssertionError("streaming server did not exit after client disconnect") + raw = rig.hb_sock.recv() + except zmq.Again: + continue + if msgpack.unpackb(raw).get("state") == "streaming": + saw_streaming = True + break + assert saw_streaming, "heartbeat never reached streaming after TCP connect" - err = proc.stderr.read().decode(errors="replace") if proc.stderr else "" - assert proc.returncode == 0, err + # Stream enough data for the server to accumulate a handful of full + # flush cycles (flush_interval = 0.5 s → ~3 full flushes + a final + # partial flush on disconnect). _auto_send_worker auto-disconnects at + # the end which is what triggers the server-side stop publish. + src.start_auto_sending(1.6) + if src.send_thread: + src.send_thread.join(timeout=5.0) + src.stop_auto_sending() - assert len(collected["start"]) >= 1 - start = TimePixStart(**collected["start"][0]) - assert start.tdc_frequency_hz == 100.0 - assert start.collapse_y is True + messages = collect_messages_until_stop(rig.sub_sock, timeout_s=40.0) - assert collected["events"] >= 1, "no ZMQ event messages — UI would see no heatmap updates" + try: + rig.server_proc.wait(timeout=25) + except subprocess.TimeoutExpired: + rig.server_proc.kill() + pytest.fail("streaming server did not exit after client disconnect") + assert rig.server_proc.returncode == 0, f"server exited with code {rig.server_proc.returncode}" + + starts = [m for m in messages if m.get("msg_type") == "start"] + stops = [m for m in messages if m.get("msg_type") == "stop"] + events = [m for m in messages if m.get("msg_type") not in ("start", "stop")] + + # Gap 2a (stop side): exactly one stop per acquisition — no race here. + # Gap 2a (start side): >= 1 for now; a known race between data_callback's + # fallback and the main-loop connect block can emit a duplicate start with + # the same scan_name (parked for a follow-up fix). + assert starts, "no start message received" + assert len(stops) == 1, f"gap 2a: expected exactly one stop, got {len(stops)}" + assert events, "no ZMQ event messages — UI would see no heatmap updates" + + start = TimePixStart(**starts[0]) + stop = TimePixStop(**stops[0]) + + # Config round-trip through the streaming server (rig → CLI → start msg). + assert start.tdc_frequency_hz == rig.tdc_frequency + assert start.collapse_y is rig.collapse_y + assert stop.total_cycles >= 1 + + # ── Gap 2b: scan_name must match across every message in the acquisition ── + assert stop.scan_name == start.scan_name, "stop scan_name differs from start scan_name" + event_scan_names = {e.get("scan_name") for e in events} + assert event_scan_names == {start.scan_name}, ( + f"gap 2b: event scan_names {event_scan_names!r} " f"diverge from start scan_name {start.scan_name!r}" + ) - assert len(collected["stop"]) >= 1 - stop = TimePixStop(**collected["stop"][-1]) - assert stop.scan_name == start.scan_name - assert stop.total_cycles >= 1 - finally: - if proc.poll() is None: - proc.terminate() - try: - proc.wait(timeout=5) - except subprocess.TimeoutExpired: - proc.kill() - data_sub.close() - hb_sub.close() - ctx.term() + # ── Gap 2d: flush_number must be 1..N with no gaps and no duplicates ── + flush_numbers = [e["flush_number"] for e in events] + assert flush_numbers == list( + range(1, len(flush_numbers) + 1) + ), f"gap 2d: flush_number sequence is not a monotonic 1..N: {flush_numbers!r}" + assert stop.total_flushes == len( + flush_numbers + ), f"stop.total_flushes={stop.total_flushes} != events received={len(flush_numbers)}" @pytest.mark.unit diff --git a/tests/test_parser.py b/tests/test_parser.py index 4c23d14..f516c66 100644 --- a/tests/test_parser.py +++ b/tests/test_parser.py @@ -39,7 +39,8 @@ def test_parse_basic_pixel_packet(self, parser): reserved = 0 # Build packet data (36 bits) - pixel_data = reserved | (y << 6) | (x << 16) | (tot << 26) + # WHY DID YOU HARDCODE THIS, USE THE SIMULATOR DUHHH! + pixel_data = reserved | (x << 6) | (y << 16) | (tot << 26) # Build full 96-bit value full_value = pixel_data | (timestamp << 36) | (packet_type << 92) @@ -115,7 +116,7 @@ def _build_pixel_packet(timestamp, tot, x, y, reserved): y = y & 0x3FF # 10 bits reserved = reserved & 0x3F # 6 bits - pixel_data = reserved | (y << 6) | (x << 16) | (tot << 26) + pixel_data = reserved | (x << 6) | (y << 16) | (tot << 26) full_value = pixel_data | (timestamp << 36) | (packet_type << 92) return full_value.to_bytes(12, byteorder="big") diff --git a/tests/test_preferences.py b/tests/test_preferences.py new file mode 100644 index 0000000..accbee5 --- /dev/null +++ b/tests/test_preferences.py @@ -0,0 +1,236 @@ +"""Tests for the operator-sidebar preferences module. + +Covers the round-trip / malformed-JSON / clamp / atomic-write behavior +described in the plan. These tests exercise ``preferences.py`` directly +and do **not** instantiate Qt widgets — Qt is only loaded when a real +``OperatorTab`` is constructed in ``main.py``, and pulling it in here +would force PySide6 onto the test runner. +""" + +from __future__ import annotations + +import json +from pathlib import Path + +import pytest + +from splash_timepix.ui import preferences + + +def _full_prefs_payload() -> dict: + """A complete, in-range preferences dict that round-trips identically. + + Must include every key that ``validate_and_clamp`` emits (operator subset + *and* alignment subset); otherwise round-trip equality fails because the + sanitized on-disk file fills missing keys with their defaults. + """ + return { + "preferences_version": preferences.PREFERENCES_VERSION, + "tdc_frequency": 2500.5, + "tdc_channel_text": "1", + "tdc_edge_text": "Falling", + "callback_batch_size": 5_000, + "n_bins": 12_000, + "duration": 120, + "output_dir": "/tmp/splash_timepix_test_output", + "alignment_rate_hz": 15, + "alignment_auto_range": False, + "alignment_manual_min": 5.0, + "alignment_manual_max": 250.0, + "alignment_log": True, + "alignment_binarize": False, + "alignment_show_integrated": True, + "alignment_show_crosshair": False, + "alignment_simulator": False, + "scan_counter": 1, + } + + +def test_default_preferences_self_consistent(): + """Defaults are themselves in-range; saving them yields no warnings.""" + defaults = preferences.default_preferences() + sanitized = preferences.validate_and_clamp(defaults) + assert sanitized == defaults + + +def test_round_trip_full_payload(tmp_path: Path): + """A fully-populated payload survives save → load unchanged.""" + path = tmp_path / "operator_prefs.json" + payload = _full_prefs_payload() + preferences.save_operator_preferences(payload, path) + loaded = preferences.load_operator_preferences(path) + assert loaded == payload + + +def test_combo_text_persisted_verbatim(tmp_path: Path): + """The combo text fields must be the exact display strings (case-sensitive). + + The OperatorTab restores combos via ``QComboBox.setCurrentText`` which + matches displayed item text *exactly*; if we ever silently lowercase + or normalize them on save, the restoration would silently no-op. + """ + path = tmp_path / "operator_prefs.json" + payload = _full_prefs_payload() + payload["tdc_channel_text"] = "Both" + payload["tdc_edge_text"] = "Rising" + preferences.save_operator_preferences(payload, path) + loaded = preferences.load_operator_preferences(path) + assert loaded["tdc_channel_text"] == "Both" + assert loaded["tdc_edge_text"] == "Rising" + + +def test_load_missing_file_returns_defaults(tmp_path: Path): + """A missing file is not an error; defaults are returned.""" + path = tmp_path / "does_not_exist.json" + loaded = preferences.load_operator_preferences(path) + assert loaded == preferences.default_preferences() + + +def test_load_malformed_json_returns_defaults(tmp_path: Path): + """A corrupt file does not raise into the UI; defaults are returned.""" + path = tmp_path / "operator_prefs.json" + path.write_text("{this is not valid json") + loaded = preferences.load_operator_preferences(path) + assert loaded == preferences.default_preferences() + + +def test_load_non_dict_top_level_returns_defaults(tmp_path: Path): + """A JSON top-level that isn't a dict (e.g. a list) collapses to defaults.""" + path = tmp_path / "operator_prefs.json" + path.write_text(json.dumps([1, 2, 3])) + loaded = preferences.load_operator_preferences(path) + assert loaded == preferences.default_preferences() + + +def test_missing_keys_filled_from_defaults(tmp_path: Path): + """Partial payloads keep specified values and fill the rest from defaults.""" + path = tmp_path / "operator_prefs.json" + path.write_text(json.dumps({"tdc_frequency": 250.0, "n_bins": 750})) + loaded = preferences.load_operator_preferences(path) + assert loaded["tdc_frequency"] == 250.0 + assert loaded["n_bins"] == 750 + defaults = preferences.default_preferences() + assert loaded["duration"] == defaults["duration"] + assert loaded["tdc_channel_text"] == defaults["tdc_channel_text"] + + +@pytest.mark.parametrize( + "key,bad_value,expected", + [ + ("tdc_frequency", 0.0, preferences.TDC_FREQUENCY_RANGE[0]), + ("tdc_frequency", 1e12, preferences.TDC_FREQUENCY_RANGE[1]), + ("callback_batch_size", 0, preferences.CALLBACK_BATCH_SIZE_RANGE[0]), + ("callback_batch_size", 99_999_999, preferences.CALLBACK_BATCH_SIZE_RANGE[1]), + ("n_bins", 10, preferences.N_BINS_RANGE[0]), + ("n_bins", 100_000, preferences.N_BINS_RANGE[1]), + ("duration", -1, preferences.DURATION_RANGE[0]), + ("duration", 100_000_000, preferences.DURATION_RANGE[1]), + ], +) +def test_out_of_range_values_clamped(key: str, bad_value, expected, tmp_path: Path): + """Out-of-range numeric values are clamped to the nearest in-range bound.""" + payload = _full_prefs_payload() + payload[key] = bad_value + path = tmp_path / "operator_prefs.json" + path.write_text(json.dumps(payload)) + loaded = preferences.load_operator_preferences(path) + assert loaded[key] == expected + + +@pytest.mark.parametrize( + "key,bad_value", + [ + ("tdc_channel_text", "Three"), + ("tdc_channel_text", "BOTH"), + ("tdc_channel_text", 1), + ("tdc_edge_text", "rising"), + ("tdc_edge_text", None), + ], +) +def test_unknown_combo_value_falls_back_to_default(key: str, bad_value, tmp_path: Path): + """Unknown / wrong-type combo values revert to defaults (case-sensitive).""" + payload = _full_prefs_payload() + payload[key] = bad_value + path = tmp_path / "operator_prefs.json" + path.write_text(json.dumps(payload)) + loaded = preferences.load_operator_preferences(path) + assert loaded[key] == preferences.default_preferences()[key] + + +def test_non_numeric_value_falls_back_to_default(tmp_path: Path): + """Strings where numbers are expected revert to default with a warning.""" + payload = _full_prefs_payload() + payload["tdc_frequency"] = "fast" + payload["n_bins"] = "many" + path = tmp_path / "operator_prefs.json" + path.write_text(json.dumps(payload)) + loaded = preferences.load_operator_preferences(path) + defaults = preferences.default_preferences() + assert loaded["tdc_frequency"] == defaults["tdc_frequency"] + assert loaded["n_bins"] == defaults["n_bins"] + + +def test_unknown_keys_dropped(tmp_path: Path): + """Extra keys in the JSON file are ignored (forward-compat with older readers).""" + payload = _full_prefs_payload() + payload["future_field"] = {"some": "blob"} + payload["another_future_field"] = 42 + path = tmp_path / "operator_prefs.json" + path.write_text(json.dumps(payload)) + loaded = preferences.load_operator_preferences(path) + assert "future_field" not in loaded + assert "another_future_field" not in loaded + + +def test_atomic_write_no_tmp_file_left(tmp_path: Path): + """A successful save leaves no ``.tmp`` artefact behind.""" + path = tmp_path / "operator_prefs.json" + preferences.save_operator_preferences(_full_prefs_payload(), path) + leftover = path.with_name(path.name + ".tmp") + assert path.exists() + assert not leftover.exists() + + +def test_atomic_write_creates_parent_dir(tmp_path: Path): + """Save creates the config directory if it does not yet exist.""" + path = tmp_path / "nested" / "deeper" / "operator_prefs.json" + assert not path.parent.exists() + preferences.save_operator_preferences(_full_prefs_payload(), path) + assert path.exists() + + +def test_save_overwrites_existing_file(tmp_path: Path): + """A second save replaces the file contents (replace semantics).""" + path = tmp_path / "operator_prefs.json" + a = _full_prefs_payload() + b = _full_prefs_payload() + b["duration"] = 999 + preferences.save_operator_preferences(a, path) + preferences.save_operator_preferences(b, path) + loaded = preferences.load_operator_preferences(path) + assert loaded["duration"] == 999 + + +def test_save_sanitizes_input_before_writing(tmp_path: Path): + """save() writes through validate_and_clamp; bad inputs are not persisted.""" + path = tmp_path / "operator_prefs.json" + payload = _full_prefs_payload() + payload["duration"] = -1 + payload["tdc_channel_text"] = "Three" + preferences.save_operator_preferences(payload, path) + on_disk = json.loads(path.read_text()) + assert on_disk["duration"] == preferences.DURATION_RANGE[0] + assert on_disk["tdc_channel_text"] == preferences.default_preferences()["tdc_channel_text"] + + +def test_config_dir_honors_xdg_config_home(monkeypatch, tmp_path: Path): + """``config_dir()`` uses ``XDG_CONFIG_HOME`` when set.""" + monkeypatch.setenv("XDG_CONFIG_HOME", str(tmp_path)) + assert preferences.config_dir() == tmp_path / "splash_timepix" + + +def test_config_dir_falls_back_to_home_dot_config(monkeypatch, tmp_path: Path): + """Without XDG_CONFIG_HOME, falls back to ``~/.config/splash_timepix``.""" + monkeypatch.delenv("XDG_CONFIG_HOME", raising=False) + monkeypatch.setenv("HOME", str(tmp_path)) + assert preferences.config_dir() == tmp_path / ".config" / "splash_timepix" diff --git a/tests/test_simulator_batching.py b/tests/test_simulator_batching.py new file mode 100644 index 0000000..d48b228 --- /dev/null +++ b/tests/test_simulator_batching.py @@ -0,0 +1,155 @@ +"""Unit test for the simulator's ``tcp_batch_interval_s`` knob. + +False-green guard for the bursty-wire regression test +------------------------------------------------------ +The server-level regression test in ``test_flush_burstiness.py`` relies on +the simulator actually emitting *multi-packet boluses* on the wire to +reproduce the Serval+luna-iterator symptom. If the batching code is +subtly broken and every ``sendall`` still carries only one packet, the +server-level test could silently go green for the wrong reason (or red +for a different reason than we expect). This unit test closes that loop +by asserting the batching knob really does batch, before any server +subprocess ever runs. + +Scope +----- +- Runs fully in-process (no sockets, no subprocesses). +- Replaces ``self.socket`` with a fake whose ``sendall`` records + ``(monotonic_ts, len(data))`` per call. +- Asserts sendall invocation rate, payload size distribution, and + multi-packet payloads. +""" + +from __future__ import annotations + +import statistics +import time +from typing import List, Tuple + +import pytest + +from splash_timepix.simulator_cli import SimulatorSource + + +class _FakeSocket: + """Capture sendall calls with their arrival time + size. + + Not a real socket, just a duck-typed stand-in. ``SimulatorSource`` + calls ``.sendall(bytes)`` in the worker and ``.close()`` on + disconnect — we only implement those two surfaces. + """ + + def __init__(self) -> None: + self.sendalls: List[Tuple[float, int]] = [] + self.closed = False + + def sendall(self, data: bytes) -> None: + self.sendalls.append((time.monotonic(), len(data))) + + def close(self) -> None: + self.closed = True + + +def _run_worker_with_fake_socket( + *, + cps: float, + tdc_frequency: float, + tcp_batch_interval_s: float, + duration: float, +) -> _FakeSocket: + """Drive ``_auto_send_worker`` directly with a fake socket. + + Avoids ``start_auto_sending`` so no real TCP is touched. Sets + ``self.running = True`` first because the worker's loop exits as + soon as it sees ``running == False``. + """ + src = SimulatorSource(host="localhost", port=9) # port unused with fake sock + src.pixel_count_rate = cps + src.tdc_frequency = tdc_frequency + src.counting = False + src.tcp_batch_interval_s = tcp_batch_interval_s + fake = _FakeSocket() + src.socket = fake # type: ignore[assignment] + src.running = True + # _auto_send_worker flips running=False on exit and then calls + # self.disconnect() which invokes self.socket.close(). + src._auto_send_worker(duration) + return fake + + +@pytest.mark.unit +def test_tcp_batch_interval_zero_is_per_packet(): + """Default mode: each generated packet is its own sendall. + + Anchors the default behaviour so enabling batching is demonstrably + the thing that changes the wire profile. + """ + fake = _run_worker_with_fake_socket( + cps=2000.0, + tdc_frequency=100.0, + tcp_batch_interval_s=0.0, + duration=1.0, + ) + assert fake.sendalls, "worker produced no packets at all" + # Every packet is 12 bytes; in per-packet mode every sendall carries + # exactly one packet. + sizes = [n for _, n in fake.sendalls] + assert all( + s == 12 for s in sizes + ), f"tcp_batch_interval_s=0 should send 12 bytes per sendall, got sizes {sorted(set(sizes))}" + assert fake.closed, "socket should be closed at end of worker" + + +@pytest.mark.unit +def test_tcp_batch_interval_produces_multi_packet_boluses(): + """Batched mode: sendall fires ~every interval, each carrying many packets.""" + cps = 10_000.0 + tdc_freq = 1_000.0 + interval = 0.5 + duration = 3.0 + + fake = _run_worker_with_fake_socket( + cps=cps, + tdc_frequency=tdc_freq, + tcp_batch_interval_s=interval, + duration=duration, + ) + + # --- Invocation count ------------------------------------------------- + # duration/interval = 6 boluses; allow 4..10 to cover startup grace + # and the final tail flush at worker exit. + n_calls = len(fake.sendalls) + assert 4 <= n_calls <= 10, f"expected 4..10 sendall calls over {duration}s at interval={interval}s, got {n_calls}" + + # --- Payload alignment ----------------------------------------------- + # Every packet is 12 bytes; every bolus is a whole number of packets. + sizes = [n for _, n in fake.sendalls] + bad_align = [s for s in sizes if s % 12 != 0] + assert not bad_align, f"payloads not a multiple of 12 bytes: {bad_align}" + + # --- Boluses must actually be multi-packet --------------------------- + # Expected packets per bolus = cps*interval + 2*tdc_freq*interval + # = 5000 (pixel) + 1000 (tdc rise+fall) = ~6000 packets/bolus, i.e. + # ~72000 bytes. Be generous (>= 100 packets) to survive CI jitter. + mean_packets_per_bolus = statistics.fmean(sizes) / 12 + assert mean_packets_per_bolus >= 100, ( + f"mean bolus size is only {mean_packets_per_bolus:.1f} packets — " + f"batching is not actually accumulating (batcher is broken)" + ) + + # --- Inter-send pacing -------------------------------------------- + # Inter-sendall median should be close to the configured interval + # (exclude the startup sendall which can arrive early, and the tail + # flush which can arrive anywhere relative to the last boundary). + ts = [t for t, _ in fake.sendalls] + deltas = [ts[i + 1] - ts[i] for i in range(len(ts) - 1)] + if len(deltas) >= 3: + # drop min and max to suppress start/tail artefacts + trimmed = sorted(deltas)[1:-1] + median_dt = statistics.median(trimmed) + assert 0.3 <= median_dt <= 0.8, ( + f"median inter-sendall interval {median_dt:.3f}s is far from " + f"configured {interval}s (expected within [0.3, 0.8])" + ) + + assert fake.closed, "socket should be closed at end of worker" diff --git a/tests/test_single_instance.py b/tests/test_single_instance.py new file mode 100644 index 0000000..1fc9f0e --- /dev/null +++ b/tests/test_single_instance.py @@ -0,0 +1,260 @@ +"""Tests for the Linux-only per-user single-instance lock. + +These tests cover: + +- Successful acquire / release round-trip. +- ``O_CLOEXEC`` is set on the held FD (so spawned children — Serval, + streaming-server, live-cli, simulator — don't inherit the flock). +- The PID written into ``ui.lock`` is the holder's PID. +- ``flock`` contention raises :class:`AlreadyRunning` with the holder's PID. +- ``psutil`` cmdline / uid checks reject unrelated processes. + +Pure-Python tests where possible — flock contention is simulated with a +second FD opened from the same process (per ``man 2 flock``: "An attempt +to lock the file using one of these file descriptors may be denied by a +lock that the calling process has already placed via another file +descriptor."). +""" + +from __future__ import annotations + +import os +import signal +import subprocess +import sys +import time +from pathlib import Path + +import pytest + +# Linux-only by design — flock semantics on macOS/Windows differ enough +# that the production code does not enforce there. Skipping early avoids +# accidental import-time failures of fcntl on hostile platforms. +pytestmark = pytest.mark.skipif(sys.platform != "linux", reason="single-instance is Linux-only") + +import fcntl # noqa: E402 + +from splash_timepix.ui import single_instance # noqa: E402 + + +def _can_signal_subprocess() -> bool: + """Return True if this process can SIGTERM a subprocess it spawned. + + Some sandboxed environments restrict ptrace/signal delivery even to + child processes owned by the same UID. The four tests that spawn a + ``sleep`` helper and then SIGTERM it are meaningless in those + environments and should be skipped rather than fail noisily. + """ + try: + proc = subprocess.Popen(["sleep", "5"]) + time.sleep(0.05) + os.kill(proc.pid, signal.SIGTERM) + proc.wait(timeout=2) + return True + except PermissionError: + return False + except Exception: + return False + + +_HAS_SIGNAL_PERMISSION = _can_signal_subprocess() +_skip_no_signal = pytest.mark.skipif( + not _HAS_SIGNAL_PERMISSION, + reason="cannot SIGTERM own subprocess in this environment (sandbox restriction)", +) + + +@pytest.fixture(autouse=True) +def _release_held_lock_after_test(): + """Release any module-held FD so tests don't leak state across cases.""" + yield + single_instance.release_lock() + + +@pytest.fixture +def lock_path(tmp_path: Path) -> Path: + return tmp_path / "ui.lock" + + +def test_acquire_returns_fd(lock_path: Path): + fd = single_instance.acquire_lock(lock_path) + assert isinstance(fd, int) + assert fd >= 0 + + +def test_acquire_idempotent_within_process(lock_path: Path): + """A second call from the same process returns the already-held FD.""" + fd1 = single_instance.acquire_lock(lock_path) + fd2 = single_instance.acquire_lock(lock_path) + assert fd1 == fd2 + + +def test_acquire_release_acquire_round_trip(lock_path: Path): + """After release, the next acquire succeeds (simulates clean restart).""" + single_instance.acquire_lock(lock_path) + single_instance.release_lock() + fd = single_instance.acquire_lock(lock_path) + assert isinstance(fd, int) + + +def test_lock_fd_has_o_cloexec(lock_path: Path): + """The held FD must not be inherited by spawned children. + + Without ``O_CLOEXEC`` (or ``set_inheritable(False)``), children like + Serval / streaming-server would keep the lock alive past the UI's + death — a fresh launch would then see the lock as busy until those + children also exit. + """ + fd = single_instance.acquire_lock(lock_path) + flags = fcntl.fcntl(fd, fcntl.F_GETFD) + assert flags & fcntl.FD_CLOEXEC, "lock FD must have FD_CLOEXEC set" + + +def test_pid_written_to_file_is_holders_pid(lock_path: Path): + """The lock file contains our PID after acquisition (read via separate FD).""" + single_instance.acquire_lock(lock_path) + pid_from_file = single_instance.other_instance_pid(lock_path) + assert pid_from_file == os.getpid() + + +def test_contention_raises_already_running_with_pid(lock_path: Path): + """A second acquire while another FD holds the flock raises with the PID. + + Simulated within one process: open + flock a sentinel FD that is *not* + the one ``acquire_lock`` will use, then write a fake PID into the file + (mirroring what a real first instance would have done). The next + ``acquire_lock`` call must see the contention and surface that PID. + """ + sentinel_pid = 424242 + + holder_fd = os.open(lock_path, os.O_RDWR | os.O_CREAT, 0o600) + try: + fcntl.flock(holder_fd, fcntl.LOCK_EX | fcntl.LOCK_NB) + os.write(holder_fd, str(sentinel_pid).encode("ascii")) + + with pytest.raises(single_instance.AlreadyRunning) as exc: + single_instance.acquire_lock(lock_path) + assert exc.value.pid == sentinel_pid + finally: + try: + fcntl.flock(holder_fd, fcntl.LOCK_UN) + finally: + os.close(holder_fd) + + +def test_contention_with_empty_lock_file_yields_pid_none(lock_path: Path): + """Lock held mid-startup (file empty) → AlreadyRunning(pid=None). + + The plan calls out this race window explicitly: the holder grabs flock + *before* writing its PID, so a second instance arriving in that gap + sees the file empty. The Kill UX must be disabled in that case + (``main._show_already_running_dialog`` does the right thing). + """ + holder_fd = os.open(lock_path, os.O_RDWR | os.O_CREAT, 0o600) + try: + fcntl.flock(holder_fd, fcntl.LOCK_EX | fcntl.LOCK_NB) + # Deliberately do not write a PID. + + with pytest.raises(single_instance.AlreadyRunning) as exc: + single_instance.acquire_lock(lock_path) + assert exc.value.pid is None + finally: + try: + fcntl.flock(holder_fd, fcntl.LOCK_UN) + finally: + os.close(holder_fd) + + +def test_other_instance_pid_returns_none_for_missing_file(tmp_path: Path): + assert single_instance.other_instance_pid(tmp_path / "absent.lock") is None + + +def test_other_instance_pid_returns_none_for_unparseable(lock_path: Path): + lock_path.write_text("not a pid\n") + assert single_instance.other_instance_pid(lock_path) is None + + +def test_is_other_instance_alive_rejects_dead_pid(): + """A PID that has never existed (very large) is not a live instance.""" + # PID_MAX on Linux defaults to 2^22; pick something well above any real PID. + bogus_pid = 4_194_303 + assert single_instance.is_other_instance_alive(bogus_pid) is False + + +@_skip_no_signal +def test_is_other_instance_alive_rejects_unrelated_cmdline(): + """A live process whose cmdline lacks our markers is rejected. + + This is the PID-reuse mitigation: even if the lock file's PID happens + to match an unrelated process, the cmdline check ensures we never + SIGTERM something that isn't ours. + """ + proc = subprocess.Popen(["sleep", "10"]) + try: + # Give the kernel a moment to set up cmdline. + time.sleep(0.05) + assert single_instance.is_other_instance_alive(proc.pid) is False + finally: + proc.terminate() + try: + proc.wait(timeout=2) + except subprocess.TimeoutExpired: + proc.kill() + proc.wait() + + +@_skip_no_signal +def test_is_other_instance_alive_accepts_matching_marker(): + """When markers match the cmdline, a live same-uid process is accepted.""" + proc = subprocess.Popen(["sleep", "10"]) + try: + time.sleep(0.05) + assert single_instance.is_other_instance_alive(proc.pid, app_markers=("sleep",)) is True + finally: + proc.terminate() + try: + proc.wait(timeout=2) + except subprocess.TimeoutExpired: + proc.kill() + proc.wait() + + +@_skip_no_signal +def test_terminate_other_instance_refuses_unrelated_process(): + """Refuse SIGTERM if cmdline doesn't match our markers (no signal sent).""" + proc = subprocess.Popen(["sleep", "10"]) + try: + time.sleep(0.05) + terminated = single_instance.terminate_other_instance(proc.pid, timeout_s=0.2) + assert terminated is False + # Process must still be alive — we refused to signal it. + assert proc.poll() is None + finally: + proc.terminate() + try: + proc.wait(timeout=2) + except subprocess.TimeoutExpired: + proc.kill() + proc.wait() + + +@_skip_no_signal +def test_terminate_other_instance_signals_matching_process(): + """When markers match, SIGTERM is delivered and the process exits.""" + proc = subprocess.Popen(["sleep", "10"]) + try: + time.sleep(0.05) + terminated = single_instance.terminate_other_instance(proc.pid, timeout_s=3.0, app_markers=("sleep",)) + assert terminated is True + # Reap the now-dead process so it doesn't leak as a zombie. + proc.wait(timeout=1) + finally: + if proc.poll() is None: + proc.kill() + proc.wait() + + +def test_lock_path_under_xdg_config_home(monkeypatch, tmp_path: Path): + """``lock_path()`` honors XDG_CONFIG_HOME (shared with preferences).""" + monkeypatch.setenv("XDG_CONFIG_HOME", str(tmp_path)) + assert single_instance.lock_path() == tmp_path / "splash_timepix" / "ui.lock" diff --git a/tests/test_start_stop_messages.py b/tests/test_start_stop_messages.py index 8bb4300..8409206 100644 --- a/tests/test_start_stop_messages.py +++ b/tests/test_start_stop_messages.py @@ -6,161 +6,155 @@ - ``start`` / ``event`` / ``stop`` messages on the data PUB socket (port 5657 by default). - The subscriber connecting before the data source starts (ZMQ slow joiner). -A single end-to-end test exercises that path with **dynamic ports** so tests can run -in parallel and avoid collisions with a developer's local UI session. +The integration tests share a ``streaming_rig`` factory fixture (see +``tests/conftest.py``) so TDC / flush-interval / port parameters live in one +place and can never drift between the streaming server and the simulator that +feeds it. """ +from __future__ import annotations + import subprocess -import sys import time -from pathlib import Path +from typing import List -import msgpack import numpy as np import pytest -import zmq from splash_timepix.schemas import TimePixEvent, TimePixStart, TimePixStop -from tests.port_utils import get_free_port +from tests.conftest import collect_messages_until_stop + + +def _split(messages: List[dict]): + """Partition a message stream into (starts, stops, events).""" + starts = [m for m in messages if m.get("msg_type") == "start"] + stops = [m for m in messages if m.get("msg_type") == "stop"] + events = [m for m in messages if m.get("msg_type") not in ("start", "stop")] + return starts, stops, events + + +def _assert_baseline_run_invariants( + starts: List[dict], + stops: List[dict], + events: List[dict], + run_label: str, +) -> str: + """Parity invariants: start+stop+event presence, start.scan == stop.scan. + + These match what the pre-rig tests asserted. We intentionally do *not* + check event-level scan_name cohesion nor strict monotonic flush_number: + a pre-existing race between ``data_callback``'s flush path and the + main-loop client-connect block (which polls once per second) can publish + flushes with ``scan_name=None`` and/or a pre-reset ``flush_number`` + before the reset lands. That bug is parked for a follow-up; see the + ``test_final_flush_sent_before_stop`` and ``test_streaming_app_full_…`` + tests for stricter assertions under race-free configurations. + """ + assert starts, f"{run_label}: no start message received" + assert len(stops) == 1, f"{run_label}: expected exactly one stop, got {len(stops)}" + assert events, f"{run_label}: no event messages received" + + scan = starts[0]["scan_name"] + assert ( + stops[0]["scan_name"] == scan + ), f"{run_label}: stop.scan_name {stops[0]['scan_name']!r} != start.scan_name {scan!r}" + return scan + + +def _assert_strict_run_invariants( + starts: List[dict], + stops: List[dict], + events: List[dict], + run_label: str, +) -> str: + """Stricter invariants for tests whose timing avoids the race: + + - ≥ 1 start (duplicate starts with same scan_name allowed), exactly 1 stop. + - scan_name cohesion across start(s), every event, and stop. + - flush_number is a contiguous 1..N (no gaps, no duplicates). + - stop.total_flushes == len(events). + + Requires the test to wait long enough after connect for the server's + 1 s main-loop poll to land the reset *before* any data flows — or to + use a ``flush_interval`` large enough that no full flush can fire in + the 1 s race window. + """ + assert starts, f"{run_label}: no start message received" + assert len(stops) == 1, f"{run_label}: expected exactly one stop, got {len(stops)}" + assert events, f"{run_label}: no event messages received" + + scan = starts[0]["scan_name"] + start_scans = {s["scan_name"] for s in starts} + assert start_scans == {scan}, f"{run_label}: start messages disagree on scan_name: {start_scans!r}" + assert ( + stops[0]["scan_name"] == scan + ), f"{run_label}: stop.scan_name {stops[0]['scan_name']!r} != start.scan_name {scan!r}" + + event_scans = {e.get("scan_name") for e in events} + assert event_scans == {scan}, f"{run_label}: event scan_names {event_scans!r} diverge from start {scan!r}" + + flush_numbers = [e["flush_number"] for e in events] + assert flush_numbers == list( + range(1, len(flush_numbers) + 1) + ), f"{run_label}: flush_number sequence is not monotonic 1..N: {flush_numbers!r}" + assert stops[0]["total_flushes"] == len( + flush_numbers + ), f"{run_label}: stop.total_flushes={stops[0]['total_flushes']} != events={len(flush_numbers)}" + return scan @pytest.mark.integration @pytest.mark.slow -def test_simulator_pipeline_delivers_start_and_flush_events(): - """Server + simulator subprocesses: start message, event payloads, schema fields.""" - repo_root = Path(__file__).resolve().parent.parent - tcp_port = get_free_port() - zmq_port = get_free_port() - hb_port = get_free_port() - - server_cmd = [ - sys.executable, - "-m", - "splash_timepix.app", - "--host", - "localhost", - "--port", - str(tcp_port), - "--zmq-port", - str(zmq_port), - "--heartbeat-port", - str(hb_port), - "--tdc-frequency", - "10", - "--flush-interval", - "1.0", - "--exit-on-disconnect", - ] - server_proc = subprocess.Popen( - server_cmd, - stdout=subprocess.DEVNULL, - stderr=subprocess.PIPE, - cwd=repo_root, +def test_simulator_pipeline_delivers_start_and_flush_events(streaming_rig): + """Server + simulator subprocesses: start message, event payloads, schema fields. + + Covers the ``simulator_cli`` CLI parsing path (hence the subprocess + launch) plus schema validity on every received ZMQ message. We use + :func:`_assert_baseline_run_invariants` rather than the strict variant + because this test sends data immediately on connect, so the 1 s + main-loop poll race can fire a flush before ``scan_name`` is assigned + (parked for follow-up). + """ + # High rates keep the absolute throughput up; the flush_interval of 10 s + # is what actually dodges the race — it's larger than the 1 s main-loop + # poll window, so no full flush can fire pre-reset. + rig = streaming_rig( + tdc_frequency=10000.0, + flush_interval=10.0, + cps=100000.0, + collapse_y=True, + exit_on_disconnect=True, ) - ctx = zmq.Context() - socket_sub = ctx.socket(zmq.SUB) + sim_proc = rig.spawn_simulator_cli(duration=5.0) + + # The simulator will auto-disconnect when its 5 s duration expires, which + # triggers the server-side stop publish. + messages = collect_messages_until_stop(rig.sub_sock, timeout_s=25.0) + try: - time.sleep(2.2) - if server_proc.poll() is not None: - err = server_proc.stderr.read().decode(errors="replace") if server_proc.stderr else "" - pytest.fail(f"server exited early: {err}") - - socket_sub.connect(f"tcp://127.0.0.1:{zmq_port}") - socket_sub.setsockopt(zmq.SUBSCRIBE, b"") - time.sleep(0.35) - - sim_cmd = [ - sys.executable, - "-m", - "splash_timepix.simulator_cli", - "--auto-start", - "--port", - str(tcp_port), - "--tdc-frequency", - "10", - "--cps", - "1000", - "--duration", - "5", - "--no-count", - ] - sim_proc = subprocess.Popen( - sim_cmd, - stdout=subprocess.DEVNULL, - stderr=subprocess.PIPE, - cwd=repo_root, - ) - - start_received = False - events_received = [] - deadline = time.time() + 25.0 - - while time.time() < deadline and len(events_received) < 2: - socket_sub.setsockopt(zmq.RCVTIMEO, 4000) - try: - metadata_bytes = socket_sub.recv() - except zmq.Again: - continue - metadata = msgpack.unpackb(metadata_bytes) - msg_type = metadata.get("msg_type") - is_data_message = msg_type not in ("start", "stop") - - if msg_type == "start": - start_msg = TimePixStart(**metadata) - assert start_msg.scan_name - assert start_msg.tdc_frequency_hz == 10.0 - assert start_msg.detector_size_x > 0 - assert start_msg.detector_size_y > 0 - for field in ( - "scan_name", - "tdc_frequency_hz", - "detector_size_x", - "detector_size_y", - ): - assert field in metadata - start_received = True - continue - - if msg_type == "stop": - continue - - if is_data_message: - socket_sub.setsockopt(zmq.RCVTIMEO, 2000) - try: - array_bytes = socket_sub.recv() - except zmq.Again: - continue - shape = tuple(metadata["shape"]) - dtype = metadata["dtype"] - array = np.frombuffer(array_bytes, dtype=dtype).reshape(shape) - assert array.shape == shape - assert array.dtype == np.dtype(dtype) - events_received.append( - { - "flush_number": metadata.get("flush_number"), - "shape": shape, - "total_counts": float(np.sum(array)), - } - ) - - sim_proc.terminate() - try: - sim_proc.wait(timeout=8) - except subprocess.TimeoutExpired: - sim_proc.kill() - - assert start_received, "Start message not received within timeout" - assert len(events_received) >= 1, f"Expected at least one flush event, got {events_received!r}" - assert all(e["flush_number"] is not None for e in events_received) - finally: - server_proc.terminate() - try: - server_proc.wait(timeout=8) - except subprocess.TimeoutExpired: - server_proc.kill() - socket_sub.close() - ctx.term() + sim_proc.wait(timeout=8) + except subprocess.TimeoutExpired: + sim_proc.kill() + + starts, stops, events = _split(messages) + + # Parity with pre-rig assertions: start/event/stop present, schemas valid, + # stop.scan_name matches start.scan_name. + _assert_baseline_run_invariants(starts, stops, events, run_label="sim-subprocess") + start_msg = TimePixStart(**starts[0]) + assert start_msg.tdc_frequency_hz == rig.tdc_frequency + assert start_msg.detector_size_x > 0 + assert start_msg.detector_size_y > 0 + + # Event-frame sanity: the (detector_size_x, detector_size_y) fields live + # on the start message only; the event frames carry the run-time stats. + # Assert the fields the UI actually reads per-event. + for ev_meta in events: + assert ev_meta["shape"] + assert ev_meta["dtype"] == "uint32" + assert ev_meta["flush_number"] >= 1 + assert ev_meta["cycles_in_flush"] > 0 @pytest.mark.unit @@ -241,5 +235,115 @@ def test_timepix_event_schema(self): assert event.array.shape == (256, 256, 100) +@pytest.mark.integration +@pytest.mark.slow +def test_simulator_stop_sends_zmq_stop_and_second_run_resets_counters(streaming_rig): + """Bug #11 / issues 1 & 2: stop_auto_sending() publishes a ZMQ stop; second run + reconnects with a fresh scan_name and flush_number starting at 1. + + Asserts strict invariants (single stop, scan_name cohesion across every + message, flush_number == 1..N) plus the cross-run invariants that + scan_name must change and the new run's flush_number restarts at 1. + + To keep the invariants strict we *explicitly* ``connect()`` and then + sleep 1.2 s before each run's ``start_auto_sending`` call, so the + server's 1 s-period main-loop poll has time to observe the new client, + assign the scan_name, and reset counters *before* any data flows. + Without that window a pre-existing race can leak a flush with + ``scan_name=None`` / pre-reset ``flush_number`` into the stream (parked + for a follow-up; see ``_assert_baseline_run_invariants``). + """ + rig = streaming_rig( + tdc_frequency=10000.0, + flush_interval=0.5, + cps=100000.0, + collapse_y=True, + ) + + source = rig.make_simulator() + + # ── Run 1 ────────────────────────────────────────────────────────── + assert source.connect(), "failed initial connect" + time.sleep(1.2) # let the server observe connect and reset before data flows + source.start_auto_sending(2.0) + if source.send_thread: + source.send_thread.join(timeout=6.0) + source.stop_auto_sending() # disconnects (triggers stop publish) + + run1 = collect_messages_until_stop(rig.sub_sock, timeout_s=10.0) + starts1, stops1, events1 = _split(run1) + scan1 = _assert_strict_run_invariants(starts1, stops1, events1, run_label="run1") + + # ── Run 2: explicit reconnect so we can sleep before sending ────── + assert source.connect(), "failed reconnect for run 2" + time.sleep(1.2) + source.start_auto_sending(2.0) + if source.send_thread: + source.send_thread.join(timeout=6.0) + source.stop_auto_sending() + + run2 = collect_messages_until_stop(rig.sub_sock, timeout_s=10.0) + starts2, stops2, events2 = _split(run2) + scan2 = _assert_strict_run_invariants(starts2, stops2, events2, run_label="run2") + + # ── Bug #11/2: cross-run invariants ────────────────────────────── + assert scan2 != scan1, "Bug #11/2: scan_name must change across connect/disconnect cycles" + assert ( + events2[0]["flush_number"] == 1 + ), f"Bug #11/2: flush_number must restart at 1 on reconnect, got {events2[0]['flush_number']}" + + +@pytest.mark.integration +@pytest.mark.slow +def test_final_flush_sent_before_stop(streaming_rig): + """Bug #11 / issue 3: accumulated data from the last partial TDC cycles is + published as one final event message before the stop. + + Setup: tdc=10 kHz, flush_interval=10 s. + Stream: 5 s of wallclock. + Under the wall-clock flush gate (server commit 754c857): the regular + gate fires when ``time.monotonic() - last_flush_time >= flush_interval`` + so a 5 s stream with a 10 s gate emits 0 regular flushes. The final + partial-cycle flush at stop is therefore the *only* event message — + if it is missing we get 0 events, which is the bug. + + Without the fix : 0 event messages received. + With the fix : at least 1 event message (the final flush) received. + + High TDC/CPS avoids the known low-rate race (parked for follow-up); the + large flush_interval is what keeps the "no regular flush" property. + """ + rig = streaming_rig( + tdc_frequency=10000.0, + flush_interval=10.0, + cps=100000.0, + collapse_y=True, + exit_on_disconnect=True, + ) + + source = rig.make_simulator() + assert source.connect(), "failed to connect" + + # Stream for less than one flush cycle (5 s < 10 s) so only the final + # partial-cycle flush can fire. + source.start_auto_sending(5.0) + if source.send_thread: + source.send_thread.join(timeout=10.0) + source.stop_auto_sending() + + messages = collect_messages_until_stop(rig.sub_sock, timeout_s=15.0) + starts, stops, events = _split(messages) + + assert events, ( + "Bug #11/3: no event messages received — final flush was not sent. " + f"All message types: {[m.get('msg_type') for m in messages]}" + ) + + # Strict invariants are safe here: flush_interval=10 s >> the 1 s race + # window in the server's main-loop poll, so no full flush can fire + # before the client-connect block resets state. + _assert_strict_run_invariants(starts, stops, events, run_label="final-flush") + + if __name__ == "__main__": pytest.main([__file__, "-v", "-s"]) diff --git a/tools/centroider/README.md b/tools/centroider/README.md new file mode 100644 index 0000000..67fe8be --- /dev/null +++ b/tools/centroider/README.md @@ -0,0 +1,329 @@ +# tpx3 Sweep Optimizer + +A standalone command-line tool that wraps `tpx3dump` and runs it over every +`(eps-t, eps-s)` combination, producing one HDF5 file per combo with live +progress and an adaptive ETA. + +Optionally generates a CSV histogram per HDF5 file and a PixelHits baseline +for before/after comparison — all driven by a single progress bar. + +No extra Python packages required for the sweep itself. +`histogramify.py` additionally needs `h5py` and `numpy` (install via `.venv`). + +--- + +## Quick start + +The fastest way to try it is the convenience script, which uses the 21 MB +sample file and 9 combinations by default: + +```bash +# Preview — prints planned commands without running anything +bash tools/centroider/run_example.sh --dry-run + +# Full sweep (9 combinations + 1 baseline, ~few minutes) +bash tools/centroider/run_example.sh + +# Sweep + generate histogram CSVs in one pass +bash tools/centroider/run_example.sh --histogram +``` + +Or use the built-in `--example` preset directly: + +```bash +python tools/centroider/sweep.py --example --dry-run +python tools/centroider/sweep.py --example +python tools/centroider/sweep.py --example --histogram +``` + +--- + +## Full usage + +``` +python tools/centroider/sweep.py [OPTIONS] + +Input / output: + -i, --input PATH Path to the .tpx3 input file + -o, --output-dir PATH Directory for HDF5 outputs and summary files + +Sweep parameters: + -t, --eps-t LIST Comma-separated time gaps (e.g. 20ns,100ns,500ns) + -s, --eps-s LIST Comma-separated pixel distances (e.g. 1,2,3) + +Converter: + --tpx3dump PATH Path to tpx3dump binary (default: $TPX3DUMP env or built-in path) + --extra-args "ARGS" Extra flags forwarded to every tpx3dump call + --log-level LEVEL tpx3dump log verbosity: off/trace/debug/info/warn/error (default: warn) + +Behaviour: + --skip-existing Skip combos whose .h5 output already exists (resume) + --keep-going Continue sweep if a run fails (default: abort on first failure) + --dry-run Print planned commands and exit without running + --example Fill in SampleData defaults for any missing -i/-o/-t/-s + +Post-processing: + --histogram Generate CSV histograms (histogramify) for every produced .h5 + --no-baseline Skip the PixelHits baseline run (--disable-clustering) + --keep-pixel-data Do NOT pass --discard-pixel-data to clustered runs +``` + +--- + +## Example: the 9-file sweep + baseline + histograms + +```bash +python tools/centroider/sweep.py \ + -i /home/tpx/Desktop/tpx3LOCAL/SampleData/tpx3/sample21MB.tpx3 \ + -o /home/tpx/Desktop/tpx3LOCAL/SampleData/h5s \ + -t 20ns,100ns,500ns \ + -s 1,2,3 \ + --histogram +``` + +> **Important:** Keep `--eps-t` in the **ns to low-µs range** for Tpx3 data. +> The DFSCluster algorithm becomes exponentially slower as eps-t grows, because +> it tries to merge more hits into bigger and bigger clusters. +> Values above ~100µs can make a single run take minutes or hang entirely. +> +> Measured scaling on `sample21MB.tpx3`: +> - 20ns → ~7s · 100ns → ~9s · 500ns → ~8s · 1µs → ~9s · 10µs → ~12s · 100µs → ~28s · 1ms+ → hangs + +--- + +## Run directory & reproducibility + +To avoid mixing outputs into the folder you point `-o` at, the tool treats +`-o/--output-dir` as a **parent** and writes everything into a deterministic +per-input subfolder: + +``` +/_centroided/ +``` + +The name has **no timestamp**, so re-running the same input writes to the same +folder (idempotent) and the parent directory stays clean. + +Each run directory contains a `luna.meta` JSON file capturing everything needed +to unambiguously recreate the run: + +```json +{ + "luna_version": "Tpx3Dump 0.3.2", + "tpx3dump_path": "/home/tpx/.../luna/0.4.3/bin/tpx3dump", + "generated_at": "2026-05-29T14:20:00-07:00", + "command_line": "tools/centroider/sweep.py --example --histogram", + "input_file": "/home/tpx/.../SampleData/tpx3/sample21MB.tpx3", + "output_parent": "/home/tpx/.../SampleData/h5s", + "run_dir": "/home/tpx/.../SampleData/h5s/sample21MB_centroided", + "parameters": { + "eps_t": ["20ns", "100ns", "500ns"], + "eps_s": [1, 2, 3], + "log_level": "warn", + "discard_pixel_data": true, + "baseline_pixelhits": true, + "histogram": true, + "skip_existing": false, + "keep_going": false, + "extra_args": [] + }, + "tpx3dump_flags": { + "clustered": ["--discard-pixel-data"], + "baseline": ["--disable-clustering"] + } +} +``` + +--- + +## Pipeline phases + +`sweep.py` drives all three phases under a **single progress bar** (`[k/N]`): + +| Phase | Task | Flag | +|---|---|---| +| 1 — clustered conversions | `tpx3dump ... --eps-t T --eps-s S --discard-pixel-data` | always | +| 2 — PixelHits baseline | `tpx3dump ... --disable-clustering` → `_PixelHits.h5` | default on; `--no-baseline` to skip | +| 3 — histograms | `histogramify` in-process → one `.csv` per `.h5` | opt-in via `--histogram` | + +--- + +## Outputs + +### Per-combination HDF5 files (clustered, Phase 1) + +Named `_t_s.h5`, written into `--output-dir`. + +By default these contain **only the `Clusters` dataset** (`--discard-pixel-data`), +which saves significant disk space. Pass `--keep-pixel-data` to retain `PixelHits` +in each clustered file. + +### PixelHits baseline HDF5 (Phase 2) + +Named `_PixelHits.h5`. Contains the `PixelHits` dataset (unsorted, +unclastered) and is used as the "before clustering" reference for histogram comparisons. +Skip with `--no-baseline`. + +### Histogram CSVs (Phase 3, requires `--histogram`) + +One CSV per HDF5, same stem, written into `--output-dir`: + +**Clustered files** → `.csv`: + +``` +x,s=1&t=20ns +0.5,3 +1.0,12 +1.5,7 +... +``` + +- `x`: unique cluster-x centroid (float), sorted ascending. +- count column: number of clusters at that centroid (y and t collapsed). +- Column header: `s=&t=` (from `summary.json` or filename). + +**PixelHits baseline** → `_PixelHits.csv`: + +``` +x,PixelHits +0,1024 +1,2310 +... +255,640 +``` + +- `x`: fixed integer pixel column, every value `0..255` (256 rows always). +- count column: raw pixel-hit count at that x (y and t collapsed). + +### summary.json / summary.csv + +One row per conversion with: + +| Field | Description | +|---|---| +| `eps_t` | Time gap used (null for baseline) | +| `eps_s` | Pixel distance used (null for baseline) | +| `output_file` | Full path to the HDF5 | +| `status` | `ok`, `failed`, or `skipped` | +| `wall_seconds` | Measured elapsed time (Python) | +| `reported_seconds` | Time reported by tpx3dump itself | +| `output_bytes` | HDF5 file size | +| `exit_code` | tpx3dump exit code | +| `command` | Full command that was run | +| `error_message` | Last lines of stderr on failure | + +### Completion signal + +`sweep.py` exits with code `0` when all phases complete successfully, +non-zero if any conversion or histogram step failed. The final summary +table and `summary.json`/`summary.csv` are the machine-readable "done" signal. + +--- + +## Output directory layout (example) + +``` +SampleData/h5s/ # parent (-o), kept clean + sample21MB_centroided/ # deterministic run dir (_centroided) + luna.meta # reproducibility metadata (luna version + params) + sample21MB_t20ns_s1.h5 # Clusters only (--discard-pixel-data) + sample21MB_t20ns_s2.h5 + sample21MB_t20ns_s3.h5 + sample21MB_t100ns_s1.h5 + sample21MB_t100ns_s2.h5 + sample21MB_t100ns_s3.h5 + sample21MB_t500ns_s1.h5 + sample21MB_t500ns_s2.h5 + sample21MB_t500ns_s3.h5 + sample21MB_PixelHits.h5 # PixelHits baseline (--disable-clustering) + sample21MB_t20ns_s1.csv # Histogram CSVs (--histogram) + sample21MB_t20ns_s2.csv + ... + sample21MB_PixelHits.csv + summary.json + summary.csv +``` + +--- + +## Post-processing standalone: histogramify.py + +`histogramify.py` can also be run on its own against any existing `.h5` directory: + +```bash +# Point -i at a sweep run dir (where the .h5 files live) +.venv/bin/python3.12 tools/centroider/histogramify.py \ + -i /home/tpx/Desktop/tpx3LOCAL/SampleData/h5s/sample21MB_centroided +``` + +``` +histogramify.py [OPTIONS] + + -i, --input-dir PATH Directory containing .h5 files (default: SampleData/h5s) + -o, --output-dir PATH Directory for CSV outputs (default: same as --input-dir) + --glob PATTERN Glob pattern for input files (default: *.h5) +``` + +Mode is auto-detected: +- `*_PixelHits.h5` → `PixelHits['x']` mode, fixed bins `0..255` +- all others → `Clusters['cx']` mode, unique float centroids + +Requires: `h5py`, `numpy` (install into `.venv` with `pip install h5py`). + +--- + +## Progress bar example + +``` +Starting sweep of 20 combination(s)... + [OK] eps_t=20ns eps_s=1 wall=7s ETA remaining: 2m 9s + [OK] eps_t=20ns eps_s=2 wall=7s ETA remaining: 2m 2s + [2/20] [███░░░░░░░░░░░░░░░░░░░░░░░░░░░] 10% elapsed: 14s ETA: 2m 2s | next: eps_t=20ns eps_s=3 + ... + [OK] PixelHits baseline wall=5s ETA remaining: 3s + [OK] histogram: sample21MB_t20ns_s1.h5 wall=0s ETA remaining: done + ... +Sweep complete — 20 ok, 0 skipped, 0 failed | total elapsed: 2m 45s +``` + +--- + +## Resume a partial sweep + +If the sweep was interrupted, use `--skip-existing` to skip already-converted +files and only run the missing combinations: + +```bash +python tools/centroider/sweep.py \ + -i SampleData/tpx3/sample21MB.tpx3 \ + -o SampleData/h5s \ + -t 20ns,100ns,500ns -s 1,2,3 \ + --skip-existing +``` + +--- + +## Override paths in run_example.sh + +The convenience script respects these env vars: + +```bash +TPX3_INPUT=/path/to/other.tpx3 \ +H5_OUTPUT_DIR=/path/to/output \ +TPX3DUMP=/path/to/tpx3dump \ + bash tools/centroider/run_example.sh +``` + +--- + +## File layout + +``` +tools/centroider/ + sweep.py — CLI entry point and orchestration loop (all 3 phases) + runner.py — run_one(): subprocess wrapper, timing, RunResult dataclass + progress.py — ProgressReporter: adaptive ETA, TTY bar, non-TTY fallback + histogramify.py — histogram CSVs from .h5 files (importable + standalone CLI) + run_example.sh — zero-arg convenience script + README.md — this file +``` diff --git a/tools/centroider/api.py b/tools/centroider/api.py new file mode 100644 index 0000000..7aba700 --- /dev/null +++ b/tools/centroider/api.py @@ -0,0 +1,413 @@ +""" +Programmatic API for the tpx3 sweep optimizer. +============================================= + +An argv-free, print-free entry point that mirrors ``sweep.main`` but is meant +to be driven in-process (e.g. from a GUI worker thread) rather than the CLI. + +It reuses the orchestration helpers from :mod:`sweep`, the single-run wrapper +from :mod:`runner`, and the histogram readers from :mod:`histogramify`, and +reports progress through an optional ``progress_callback`` instead of the TTY +:class:`progress.ProgressReporter` (which stays as-is for CLI use). + +Typical usage:: + + from api import run_sweep, ProgressEvent + + def on_progress(event: ProgressEvent) -> None: + print(f"[{event.index}/{event.total}] {event.label} {event.status or ''}") + + result = run_sweep( + input_file="sample.tpx3", + output_parent="h5s", + eps_t_list="20ns,100ns,500ns", + eps_s_list="1,2,3", + progress_callback=on_progress, + ) + +Nothing here writes to stdout/stderr, raises ``SystemExit``, or parses argv, so +it is safe to call from a long-lived process. +""" + +from __future__ import annotations + +import re +import sys +import threading +from dataclasses import dataclass, field +from pathlib import Path +from types import SimpleNamespace +from typing import Callable, List, Optional, Sequence, Tuple, Union + +import numpy as np + +# --------------------------------------------------------------------------- +# Make the sibling modules importable when this file is loaded directly +# (without installing anything), mirroring sweep.py's own path shim. +# --------------------------------------------------------------------------- +sys.path.insert(0, str(Path(__file__).parent)) + +import sweep as _sweep # noqa: E402 +from histogramify import histogramify_clusters, histogramify_pixelhits, label_for, load_summary # noqa: E402 +from runner import Combo, RunResult, run_one # noqa: E402 + +# --------------------------------------------------------------------------- +# Validation (raises ValueError instead of calling sys.exit, so callers in a +# long-lived process can handle bad input gracefully). +# --------------------------------------------------------------------------- + +_VALID_UNIT_RE = re.compile(r"^[0-9]+(\.[0-9]+)?ns$") + +EpsT = Union[str, Sequence[str]] +EpsS = Union[str, Sequence[Union[str, int]]] + + +def _normalize_eps_t(value: EpsT) -> List[str]: + """Accept a comma-separated string or a sequence; return validated tokens. + + Only nanosecond values are accepted (e.g. ``20ns``, ``100ns``). Larger + units (ms, s, …) would make each tpx3dump run impractically slow, so they + are rejected explicitly. + """ + if isinstance(value, str): + tokens = [t.strip() for t in value.split(",") if t.strip()] + else: + tokens = [str(t).strip() for t in value if str(t).strip()] + if not tokens: + raise ValueError("eps-t must not be empty") + bad = [t for t in tokens if not _VALID_UNIT_RE.match(t)] + if bad: + raise ValueError( + f"Invalid eps-t value(s): {bad}. " + "Only nanosecond values are accepted, e.g. 20ns, 100ns, 500ns" + ) + return tokens + + +def _normalize_eps_s(value: EpsS) -> List[int]: + """Accept a comma-separated string or a sequence; return validated positive ints.""" + if isinstance(value, str): + tokens = [t.strip() for t in value.split(",") if t.strip()] + else: + tokens = [str(t).strip() for t in value if str(t).strip()] + if not tokens: + raise ValueError("eps-s must not be empty") + result: List[int] = [] + bad: List[str] = [] + for t in tokens: + try: + v = int(t) + if v < 1: + raise ValueError + result.append(v) + except ValueError: + bad.append(t) + if bad: + raise ValueError(f"Invalid eps-s value(s): {bad} (must be positive integers, e.g. 1,2,3)") + return result + + +# --------------------------------------------------------------------------- +# Result / progress payloads +# --------------------------------------------------------------------------- + + +@dataclass +class ProgressEvent: + """One progress update emitted by :func:`run_sweep`. + + A ``begin`` event has ``status is None``; a ``finish`` event carries the + run's ``status`` ("ok" | "failed" | "skipped"), ``wall_seconds`` and the + produced ``h5_path`` (when available). A final event with ``phase == "done"`` + is emitted once the whole sweep finishes. + """ + + index: int # 1-based task index across the whole sweep + total: int + label: str + phase: str # "clustered" | "baseline" | "done" + eps_t: Optional[str] = None + eps_s: Optional[int] = None + status: Optional[str] = None # None on begin; "ok"/"failed"/"skipped" on finish + wall_seconds: float = 0.0 + eta_seconds: Optional[float] = None + h5_path: Optional[Path] = None + + +@dataclass +class SweepResult: + """Final outcome of a sweep.""" + + run_dir: Path + results: List[RunResult] = field(default_factory=list) + baseline_h5: Optional[Path] = None + + +ProgressCallback = Callable[[ProgressEvent], None] + + +# --------------------------------------------------------------------------- +# Internal ETA tracker (in-process analogue of progress.ProgressReporter) +# --------------------------------------------------------------------------- + + +class _EtaTracker: + def __init__(self, total: int) -> None: + self.total = total + self._completed = 0 + self._skipped = 0 + self._failed = 0 + self._cumulative_wall = 0.0 + + def finish(self, wall_seconds: float, status: str) -> None: + if status == "ok": + self._cumulative_wall += wall_seconds + self._completed += 1 + elif status == "skipped": + self._skipped += 1 + elif status == "cancelled": + pass # Don't count cancelled runs against ETA + else: + self._failed += 1 + + def eta_seconds(self) -> Optional[float]: + if self._completed == 0: + return None + avg = self._cumulative_wall / self._completed + remaining = self.total - (self._completed + self._skipped + self._failed) + if remaining <= 0: + return 0.0 + return avg * remaining + + +# --------------------------------------------------------------------------- +# Main programmatic entry point +# --------------------------------------------------------------------------- + + +def run_sweep( + input_file: Union[str, Path], + output_parent: Union[str, Path], + eps_t_list: EpsT, + eps_s_list: EpsS, + tpx3dump: Optional[Union[str, Path]] = None, + progress_callback: Optional[ProgressCallback] = None, + run_baseline: bool = True, + skip_existing: bool = True, + keep_going: bool = True, + keep_pixel_data: bool = False, + log_level: str = "warn", + cancel_event: Optional[threading.Event] = None, +) -> SweepResult: + """Run tpx3dump over every (eps-t, eps-s) combo, reporting progress in-process. + + Parameters mirror the CLI in :mod:`sweep`. ``eps_t_list`` / ``eps_s_list`` + accept either a comma-separated string (``"20ns,100ns"``) or a sequence. + ``progress_callback`` is invoked on every begin/finish and once on + completion. Returns a :class:`SweepResult`. + + Raises ``ValueError`` / ``FileNotFoundError`` on bad inputs (never calls + ``sys.exit``). + """ + input_file = Path(input_file) + output_parent = Path(output_parent) + tpx3dump = Path(tpx3dump) if tpx3dump else Path(_sweep._DEFAULT_TPX3DUMP) + + if not input_file.exists(): + raise FileNotFoundError(f"Input file not found: {input_file}") + if not tpx3dump.exists(): + raise FileNotFoundError( + f"tpx3dump not found at: {tpx3dump}. Set the tpx3dump argument or the TPX3DUMP env var." + ) + + eps_t_values = _normalize_eps_t(eps_t_list) + eps_s_values = _normalize_eps_s(eps_s_list) + + run_dir = _sweep._run_dir_for(output_parent, input_file) + combos = _sweep._build_combos(input_file, run_dir, eps_t_values, eps_s_values) + baseline_h5 = run_dir / f"{input_file.stem}_PixelHits.h5" + + clustered_extra = [] if keep_pixel_data else ["--discard-pixel-data"] + baseline_extra = ["--disable-clustering"] + + n_baseline = 1 if run_baseline else 0 + total = len(combos) + n_baseline + + run_dir.mkdir(parents=True, exist_ok=True) + + # Reproducibility metadata (best-effort; never fatal for the GUI use case). + try: + luna_version = _sweep._get_luna_version(tpx3dump) + meta_args = SimpleNamespace( + log_level=log_level, + keep_pixel_data=keep_pixel_data, + no_baseline=not run_baseline, + histogram=False, + skip_existing=skip_existing, + keep_going=keep_going, + ) + _sweep._write_meta( + run_dir=run_dir, + input_file=input_file, + output_parent=output_parent, + tpx3dump=tpx3dump, + luna_version=luna_version, + eps_t_list=eps_t_values, + eps_s_list=eps_s_values, + extra_args=[], + clustered_extra=clustered_extra, + baseline_extra=baseline_extra, + args=meta_args, + ) + except Exception: # noqa: BLE001 - metadata is non-essential + pass + + tracker = _EtaTracker(total) + results: List[RunResult] = [] + + def _emit(event: ProgressEvent) -> None: + if progress_callback is not None: + progress_callback(event) + + def _begin(index: int, label: str, phase: str, eps_t, eps_s) -> None: + _emit( + ProgressEvent( + index=index, + total=total, + label=label, + phase=phase, + eps_t=eps_t, + eps_s=eps_s, + status=None, + eta_seconds=tracker.eta_seconds(), + ) + ) + + def _finish(index, label, phase, eps_t, eps_s, status, wall, h5_path) -> None: + tracker.finish(wall, status) + _emit( + ProgressEvent( + index=index, + total=total, + label=label, + phase=phase, + eps_t=eps_t, + eps_s=eps_s, + status=status, + wall_seconds=wall, + eta_seconds=tracker.eta_seconds(), + h5_path=h5_path, + ) + ) + + index = 0 + + def _is_cancelled() -> bool: + return cancel_event is not None and cancel_event.is_set() + + # ---- Phase 1: clustered conversions ---- + for combo in combos: + if _is_cancelled(): + _sweep._write_summary(results, run_dir) + _emit(ProgressEvent(index=index, total=total, label="cancelled", phase="done", status="cancelled")) + return SweepResult(run_dir=run_dir, results=results, baseline_h5=None) + + index += 1 + label = f"s={combo.eps_s}&t={combo.eps_t}" + _begin(index, label, "clustered", combo.eps_t, combo.eps_s) + + if skip_existing and combo.output_file.exists(): + results.append(RunResult(combo=combo, status="skipped")) + _finish(index, label, "clustered", combo.eps_t, combo.eps_s, "skipped", 0.0, combo.output_file) + continue + + result = run_one( + tpx3dump=tpx3dump, + combo=combo, + input_file=input_file, + log_level=log_level, + extra_args=clustered_extra, + cancel_event=cancel_event, + ) + results.append(result) + produced = combo.output_file if result.status in ("ok", "skipped") else None + _finish(index, label, "clustered", combo.eps_t, combo.eps_s, result.status, result.wall_seconds, produced) + + if result.status == "cancelled": + _sweep._write_summary(results, run_dir) + _emit(ProgressEvent(index=index, total=total, label="cancelled", phase="done", status="cancelled")) + return SweepResult(run_dir=run_dir, results=results, baseline_h5=None) + + if result.status == "failed" and not keep_going: + _sweep._write_summary(results, run_dir) + _emit(ProgressEvent(index=index, total=total, label="aborted", phase="done", status="failed")) + return SweepResult(run_dir=run_dir, results=results, baseline_h5=None) + + # ---- Phase 2: PixelHits baseline ---- + if run_baseline: + if _is_cancelled(): + _sweep._write_summary(results, run_dir) + _emit(ProgressEvent(index=index, total=total, label="cancelled", phase="done", status="cancelled")) + return SweepResult(run_dir=run_dir, results=results, baseline_h5=None) + + index += 1 + baseline_combo = Combo(eps_t=None, eps_s=None, output_file=baseline_h5) + label = "PixelHits baseline" + _begin(index, label, "baseline", None, None) + + if skip_existing and baseline_h5.exists(): + results.append(RunResult(combo=baseline_combo, status="skipped")) + _finish(index, label, "baseline", None, None, "skipped", 0.0, baseline_h5) + else: + result = run_one( + tpx3dump=tpx3dump, + combo=baseline_combo, + input_file=input_file, + log_level=log_level, + extra_args=baseline_extra, + cancel_event=cancel_event, + ) + results.append(result) + produced = baseline_h5 if result.status in ("ok", "skipped") else None + _finish(index, label, "baseline", None, None, result.status, result.wall_seconds, produced) + + if result.status == "cancelled": + _sweep._write_summary(results, run_dir) + _emit(ProgressEvent(index=index, total=total, label="cancelled", phase="done", status="cancelled")) + return SweepResult(run_dir=run_dir, results=results, baseline_h5=None) + + _sweep._write_summary(results, run_dir) + + produced_baseline = baseline_h5 if (run_baseline and baseline_h5.exists()) else None + _emit(ProgressEvent(index=total, total=total, label="done", phase="done", status="ok")) + + return SweepResult(run_dir=run_dir, results=results, baseline_h5=produced_baseline) + + +# --------------------------------------------------------------------------- +# Plot data loading +# --------------------------------------------------------------------------- + + +def load_histogram( + h5_path: Union[str, Path], + summary: Optional[List[dict]] = None, +) -> Tuple[np.ndarray, np.ndarray, str]: + """Read an x-histogram straight from a sweep ``.h5`` file. + + Auto-detects baseline vs clustered files by the ``_PixelHits`` suffix and + returns ``(xs, counts, label)`` where *label* is the legend/column string + (``"PixelHits"`` or ``"s=&t="``). + """ + h5_path = Path(h5_path) + if h5_path.stem.endswith("_PixelHits"): + xs, counts = histogramify_pixelhits(h5_path) + else: + xs, counts = histogramify_clusters(h5_path) + label = label_for(h5_path, summary) + return xs, counts, label + + +def load_run_summary(run_dir: Union[str, Path]) -> Optional[List[dict]]: + """Load the ``summary.json`` written into *run_dir*, if present.""" + return load_summary(Path(run_dir)) diff --git a/tools/centroider/centroider.md b/tools/centroider/centroider.md new file mode 100644 index 0000000..354db45 --- /dev/null +++ b/tools/centroider/centroider.md @@ -0,0 +1,411 @@ +# Tpx3 Sweep Optimizer — Architecture & Developer Guide + +This document explains what the optimizer does, why it exists, how the code is +structured, and how to run it. It is intended for anyone joining this project +who needs to understand the full pipeline from raw `.tpx3` data to x-histogram +CSVs. + +--- + +## Background & Objective + +A **Timepix3 (Tpx3)** detector records individual pixel hits: for each photon +or particle that arrives, the detector registers the pixel coordinates `(x, y)`, +a timestamp (time-of-arrival, ToA), and a time-over-threshold (ToT) proxy for +energy. The raw data format is `.tpx3` — a binary stream of these hit records. + +The raw data is not immediately useful for most analyses. The key processing +step is **clustering**: nearby hits in both space and time are grouped into +**clusters**, where each cluster represents a single physical event (one +photon, one particle track). The cluster is summarised by its centroid +position `(cx, cy)`, giving sub-pixel position resolution. + +The clustering result depends on two algorithm parameters: + +| Parameter | Meaning | CLI flag | +|---|---|---| +| `eps-t` | Maximum time gap between hits in the same cluster | `--eps-t` (e.g. `100ns`) | +| `eps-s` | Maximum pixel distance between hits in the same cluster | `--eps-s` (e.g. `2`) | + +Different values of these parameters produce different cluster assignments. +**The Sweep Optimizer exists to run the clustering over a grid of +`(eps-t, eps-s)` combinations automatically**, so we can later compare results +and choose the best parameters for a given experiment. + +--- + +## The External Tool: `tpx3dump` (Luna) + +All actual `.tpx3 → .h5` conversion and clustering is done by a compiled Rust +binary called `tpx3dump`, part of the **Luna** software package (version +`Tpx3Dump 0.3.2` at the time of writing): + +``` +/home/tpx/Desktop/tpx3LOCAL/software/luna/0.4.3/bin/tpx3dump +``` + +The Python code in `tools/centroider/` does **not** reimplement clustering. +It is a **wrapper and orchestrator** that calls `tpx3dump` repeatedly with +different parameters and handles progress reporting, file organisation, and +post-processing. + +### What `tpx3dump` produces + +`tpx3dump` writes an HDF5 (`.h5`) file containing two compound datasets: + +- **`PixelHits`** — every raw hit as recorded. Columns include `x`, `y`, `tot`, + `toa`. Present when clustering is disabled (`--disable-clustering`). +- **`Clusters`** — one row per cluster. Columns include the centroid `cx`, `cy`, + cluster size, total ToT, and cluster time. Present when clustering runs + (the default). + +The detector is **256 × 256 pixels**, so `x` and `y` are in `[0, 255]`. +Cluster centroids `cx`, `cy` are floats (sub-pixel precision). + +### Key `tpx3dump` flags used by the optimizer + +| Flag | Effect | +|---|---| +| `--eps-t VALUE` | Time gap threshold for clustering (e.g. `100ns`) | +| `--eps-s VALUE` | Pixel distance threshold for clustering (integer) | +| `--discard-pixel-data` | Write only the `Clusters` dataset; omit `PixelHits` (saves disk space) | +| `--disable-clustering` | Skip clustering; write only `PixelHits` (the "raw" baseline) | +| `-g LEVEL` | Log verbosity: `off`, `warn`, `info`, etc. | + +--- + +## Pipeline Overview + +The full pipeline has three sequential phases, all orchestrated by `sweep.py` +under a single progress bar: + +``` + .tpx3 file + │ + ├─── Phase 1: Clustered conversions ──────────────────────────────────────────┐ + │ For every (eps-t, eps-s) combination: │ + │ tpx3dump process --eps-t T --eps-s S --discard-pixel-data │ + │ → _tT_sS.h5 (contains only Clusters dataset) │ + │ │ + ├─── Phase 2: PixelHits baseline ────────────────────────────────────────────┤ + │ tpx3dump process --disable-clustering │ + │ → _PixelHits.h5 (contains only PixelHits dataset) │ + │ │ + └─── Phase 3: Histograms (optional, --histogram) ────────────────────────────┘ + For each .h5 produced above: + histogramify.py reads Clusters['cx'] or PixelHits['x'] + → .csv (x-axis histogram, one column per file) +``` + +All output files land in a **deterministic run directory**: + +``` +/_centroided/ +``` + +--- + +## Phase 1 — Clustered Conversions + +### What happens + +`sweep.py` builds the Cartesian product of all `eps-t` × `eps-s` values +supplied by the user and runs `tpx3dump` once per combination, sequentially. + +For `eps-t = [20ns, 100ns, 500ns]` and `eps-s = [1, 2, 3]` this produces +**9 runs → 9 HDF5 files**: + +``` +sample21MB_t20ns_s1.h5 +sample21MB_t20ns_s2.h5 +sample21MB_t20ns_s3.h5 +sample21MB_t100ns_s1.h5 +... +sample21MB_t500ns_s3.h5 +``` + +### Filename convention + +``` +_t_s.h5 +``` + +The double underscore (`__`) separates the input stem from the parameter +suffix; `t` and `s` prefix each value. This is deterministic and +filesystem-safe for all valid `eps-t` unit strings. + +### `--discard-pixel-data` is on by default + +Because the downstream analysis only needs the `Clusters` dataset (to build +x-histograms), the clustered `.h5` files are generated with +`--discard-pixel-data`. This drops the `PixelHits` table from the clustered +files and can reduce their size substantially (raw pixel hits are the bulk of +the data). Pass `--keep-pixel-data` to disable this default. + +### Performance note + +The Luna `DFSCluster` algorithm's runtime grows dramatically with `eps-t`. +Observed scaling on a 21 MB sample file: + +| eps-t | ~runtime | +|---|---| +| 20 ns | 5–7 s | +| 100 ns | 7–10 s | +| 500 ns | 7–10 s | +| 1 µs | ~10 s | +| 10 µs | ~15 s | +| 100 µs | ~30 s | +| 1 ms | hangs | + +Keep `eps-t` in the **ns to low-µs range** for practical sweeps. + +--- + +## Phase 2 — PixelHits Baseline + +### Why we create this file + +The clustered `.h5` files do **not** contain `PixelHits` (because of +`--discard-pixel-data`). But we need a reference representing the data +**before any clustering** — i.e. the raw detector response — so we can compare: + +> How does the x-distribution of raw pixel hits compare to the x-distribution +> of cluster centroids under different (eps-t, eps-s) settings? + +The baseline run answers this. It uses `--disable-clustering`, which makes +`tpx3dump` skip the clustering step entirely and write only `PixelHits`. + +### Output file + +``` +_PixelHits.h5 +``` + +This file is detected by `histogramify.py` via its `_PixelHits` suffix and +processed differently from the clustered files (see Phase 3). + +Skip the baseline with `--no-baseline` if you only care about clustered +outputs. + +--- + +## Phase 3 — Histogram CSVs + +### Why y-histograms (labelled "x" in output) + +We collapse the 2D cluster/hit data onto the **y-axis only** (tpx3dump column +names ``cy`` / ``y``), producing a 1D histogram suitable for comparing +clustering parameter choices without handling full 2D arrays. + +**Important — axis naming convention:** tpx3dump's ``y``/``cy`` column +corresponds to the *dispersive* (physically horizontal) axis of the detector +as mounted. This is the axis that shows the expected two-peak structure from +the beam. Despite reading ``cy``/``y`` from the HDF5 file, the output CSV +column is still named ``x`` and the plot axis is labelled accordingly, so that +"x" consistently means "the spatial axis we care about" throughout the +pipeline. tpx3dump's ``x``/``cx`` column (physical column direction) is +collapsed and discarded. + +### Two histogram modes (auto-detected by filename) + +**Clustered files** (`*_t*_s*.h5`): + +- Read `Clusters['cx']` — the float x-centroid of each cluster. +- Group by unique `cx` value and count occurrences + (`numpy.unique(cx, return_counts=True)`). +- The x-axis differs per file (cluster positions depend on the parameters). +- Output CSV: one row per unique centroid value. + +``` +x,s=1&t=100ns +28.0,4 +31.0,1 +32.0,3 +... +``` + +**PixelHits baseline** (`*_PixelHits.h5`): + +- Read `PixelHits['x']` — integer pixel x-coordinate of every raw hit. +- Bin into **fixed** bins `0..255` (256 rows always present, even if count is 0) + using `numpy.bincount`. +- The x-axis is always `0, 1, 2, …, 255` — no variation between files. + +``` +x,PixelHits +0,0 +1,0 +2,1024 +... +255,640 +``` + +### Column header format + +The count column header encodes the parameters, so a single CSV is +self-describing: + +- Clustered: `s=&t=` (e.g. `s=2&t=500ns`) +- Baseline: `PixelHits` + +The label is sourced from `summary.json` if available (canonical), falling back +to parsing the filename. + +### Phase 3 is opt-in + +Add `--histogram` to `sweep.py` to enable it. It can also be run standalone +at any time: + +```bash +.venv/bin/python3.12 tools/centroider/histogramify.py \ + -i SampleData/h5s/sample21MB_centroided +``` + +--- + +## Run Directory & Reproducibility + +### Directory layout + +`-o/--output-dir` is treated as a **parent**. All outputs for one input file +land in a deterministic subdirectory: + +``` +/ + _centroided/ ← created automatically, never clutters parent + luna.meta ← reproducibility metadata (JSON) + sample21MB_t20ns_s1.h5 + sample21MB_t20ns_s2.h5 + ... + sample21MB_PixelHits.h5 + sample21MB_t20ns_s1.csv ← (if --histogram) + ... + sample21MB_PixelHits.csv + summary.json ← one row per tpx3dump run (timing, status, command) + summary.csv +``` + +The name has **no timestamp**, so re-running the same input writes to the same +folder (idempotent) and the parent directory stays clean across many +experiments. + +### `luna.meta` — the reproducibility file + +Written at the start of every real run, this JSON captures: + +- `luna_version` — `tpx3dump --version` output (e.g. `"Tpx3Dump 0.3.2"`) +- `tpx3dump_path` — absolute path to the binary used +- `generated_at` — ISO-8601 timestamp with timezone +- `command_line` — the exact Python invocation that produced this run +- `input_file`, `output_parent`, `run_dir` — resolved absolute paths +- `parameters` — **all effective** values (not just what was passed on the + command line): `eps_t`, `eps_s`, `log_level`, `discard_pixel_data`, + `baseline_pixelhits`, `histogram`, `skip_existing`, `keep_going`, `extra_args` +- `tpx3dump_flags` — per-phase flags actually forwarded to the binary + +This means you can reproduce any run exactly, even months later, by reading +`luna.meta` and re-invoking `sweep.py` with those parameters. + +--- + +## Code Architecture + +``` +tools/centroider/ + sweep.py orchestrator — CLI, all three phases, single progress bar + runner.py one tpx3dump invocation — builds command, runs subprocess, captures timing + progress.py ProgressReporter — adaptive ETA, TTY bar, non-TTY fallback + histogramify.py CSV histogram generation — importable by sweep.py, also standalone CLI + run_example.sh zero-argument convenience script using SampleData defaults + README.md reference: usage, flags, output formats, examples +``` + +### Module responsibilities + +**`sweep.py`** is the single entry point. It: +1. Parses and validates all CLI arguments. +2. Computes the total task count across all three phases up front, so + `ProgressReporter` can display an accurate bar from the start. +3. Derives the run dir and writes `luna.meta` before any conversions start. +4. Calls `run_one()` from `runner.py` for each conversion (Phases 1 & 2). +5. Calls `histogramify.process_file()` in-process for each histogram (Phase 3). +6. Writes `summary.json` and `summary.csv` after every phase (so partial + results survive an abort). + +**`runner.py`** is a pure subprocess wrapper. It knows nothing about sweeps +or progress. It builds the `tpx3dump` command, runs it, captures stdout/stderr, +measures wall time, and parses the `"Full tpx3dump run took Xs"` log line from +the binary's own timing report. Returns a `RunResult` dataclass. + +**`progress.py`** owns all terminal output during the sweep. It renders a +`[k/N] [████░░░░] 55% elapsed: 58s ETA: 3s` bar on TTY stderr (in-place, +using `\r`) and falls back to line-by-line output when stderr is not a +terminal (e.g. when piped to a log file). ETA is an adaptive running +average of completed task times. + +**`histogramify.py`** is designed to be both importable (called in-process by +`sweep.py` in Phase 3, keeping the progress bar in control of the whole +pipeline) and runnable standalone (for post-hoc processing of existing `.h5` +files). Auto-detects histogram mode from the filename suffix. + +--- + +## Inputs + +| Argument | Flag | Example | Description | +|---|---|---|---| +| Input file | `-i` | `SampleData/tpx3/sample21MB.tpx3` | The raw Tpx3 binary file | +| Output parent | `-o` | `SampleData/h5s` | Parent dir; run subdir created inside it | +| Time gaps | `-t` | `20ns,100ns,500ns` | Comma-separated, with unit (ns/µs/ms/s) | +| Space gaps | `-s` | `1,2,3` | Comma-separated positive integers (pixels) | +| tpx3dump path | `--tpx3dump` | (built-in default) | Override if luna is in a different location | + +All other flags are optional. Run `python tools/centroider/sweep.py --help` for +the full list. + +--- + +## Outputs + +| File | When created | Contents | +|---|---|---| +| `luna.meta` | Start of every run | JSON reproducibility record | +| `_t_s.h5` | Phase 1 | HDF5 with `Clusters` dataset only | +| `_PixelHits.h5` | Phase 2 | HDF5 with `PixelHits` dataset only | +| `_t_s.csv` | Phase 3 (`--histogram`) | x-histogram for clustered file | +| `_PixelHits.csv` | Phase 3 (`--histogram`) | x-histogram for baseline file | +| `summary.json` | After each phase | Per-run timing, status, command, file size | +| `summary.csv` | After each phase | Same as above in CSV format | + +--- + +## Quick Start + +```bash +# Preview planned commands (no files created) +python tools/centroider/sweep.py --example --dry-run --histogram + +# Full run: 9 combos + baseline + histograms (~2 min on sample21MB.tpx3) +python tools/centroider/sweep.py --example --histogram + +# Resume after an interruption (skip already-converted files) +python tools/centroider/sweep.py --example --histogram --skip-existing + +# Convenience script (same as --example) +bash tools/centroider/run_example.sh --histogram +``` + +--- + +## Dependencies + +| Package | Used by | Notes | +|---|---|---| +| `tpx3dump` (Luna) | `runner.py` | External binary, not a Python package | +| `numpy` | `histogramify.py` | Already in `.venv` | +| `h5py` | `histogramify.py` | Install: `.venv/bin/python3.12 -m pip install h5py` | +| Python stdlib | everything else | `argparse`, `csv`, `json`, `subprocess`, `pathlib` | + +The sweep itself (`sweep.py`, `runner.py`, `progress.py`) uses **only the +Python standard library** — no extra packages needed to run conversions. +`h5py` + `numpy` are required only for the histogram phase (`--histogram`). diff --git a/tools/centroider/histogramify.py b/tools/centroider/histogramify.py new file mode 100644 index 0000000..ee34b24 --- /dev/null +++ b/tools/centroider/histogramify.py @@ -0,0 +1,285 @@ +""" +histogramify.py +=============== + +Turns sweep-generated .h5 files into per-file CSV histograms over the +cluster y-centroid (or raw pixel-y for the PixelHits baseline). + +Two modes, auto-detected by filename suffix: + + *_PixelHits.h5 -> reads PixelHits['y'], fixed bins 0..255 (256 rows always) + all others -> reads Clusters['cy'], np.unique(return_counts=True) + +Note: tpx3dump labels the dispersive (physically "horizontal") detector axis +as ``y``/``cy``. We read that column and call it "x" in the CSV output so +that downstream consumers see a consistently named spatial axis. + +Can be imported and called in-process by sweep.py, or used standalone:: + + .venv/bin/python3.12 tools/centroider/histogramify.py + .venv/bin/python3.12 tools/centroider/histogramify.py -i SampleData/h5s + .venv/bin/python3.12 tools/centroider/histogramify.py -i SampleData/h5s -o SampleData/h5s +""" + +from __future__ import annotations + +import argparse +import csv +import json +import re +import sys +import textwrap +import time +from pathlib import Path +from typing import Dict, List, Optional, Tuple + +import numpy as np + +try: + import h5py +except ImportError: + sys.exit( + "ERROR: h5py is not installed.\n" + " Run: .venv/bin/python3.12 -m pip install h5py" + ) + +# --------------------------------------------------------------------------- +# Constants +# --------------------------------------------------------------------------- + +_SAMPLEDATA_DEFAULT = Path("/home/tpx/Desktop/tpx3LOCAL/SampleData/h5s") + +# Matches sweep output names: _t_s +_SWEEP_NAME_RE = re.compile(r"_t(?P[^_]+)_s(?P\d+)$") + +# --------------------------------------------------------------------------- +# Core processing functions (importable) +# --------------------------------------------------------------------------- + + +def histogramify_clusters(h5_path: Path) -> Tuple[np.ndarray, np.ndarray]: + """ + Read ``Clusters['cy']`` from an HDF5 file. + + Returns ``(xs, counts)`` where *xs* are unique cluster-y centroids sorted + ascending and *counts* are the number of clusters at each position. + Each cluster contributes 1; x and t dimensions are collapsed. + + Note: tpx3dump's ``cy`` (physical row) corresponds to the dispersive axis + of the detector as mounted — the axis that shows the two-peak structure + expected from the beam. Despite the column name, we label the output axis + "x" throughout the pipeline to mean "the spatial axis we care about". + """ + with h5py.File(h5_path, "r") as f: + cy = f["Clusters"]["cy"][:] + cy = np.asarray(cy, dtype=float) + xs, counts = np.unique(cy, return_counts=True) + return xs, counts.astype(np.int64) + + +def histogramify_pixelhits(h5_path: Path) -> Tuple[np.ndarray, np.ndarray]: + """ + Read ``PixelHits['y']`` from an HDF5 file. + + Returns ``(xs, counts)`` where *xs* is ``np.arange(256)`` (always 256 + rows) and *counts* is the pixel-hit count at each position. Bins with no + hits are 0. x and time dimensions are collapsed. + + Note: tpx3dump's ``y`` (physical row) is the dispersive axis — see + ``histogramify_clusters`` for the rationale. + """ + with h5py.File(h5_path, "r") as f: + y = f["PixelHits"]["y"][:] + y = np.asarray(y, dtype=int) + counts = np.bincount(y, minlength=256)[:256] + xs = np.arange(256, dtype=int) + return xs, counts.astype(np.int64) + + +def label_for(h5_path: Path, summary: Optional[List[Dict]] = None) -> str: + """ + Build the count-column header string for a given ``.h5`` file. + + - Files whose stem ends with ``_PixelHits`` return ``"PixelHits"``. + - Clustered files return ``"s=&t="`` — sourced from + *summary* (a parsed ``summary.json`` list) when available, otherwise + parsed from the filename convention ``_t_s.h5``. + """ + stem = h5_path.stem + if stem.endswith("_PixelHits"): + return "PixelHits" + + # Try summary.json lookup first (canonical source of truth) + if summary: + for row in summary: + row_file = Path(row.get("output_file", "")) + if row_file.name == h5_path.name: + eps_t = row.get("eps_t") + eps_s = row.get("eps_s") + if eps_t and eps_s is not None: + return f"s={eps_s}&t={eps_t}" + + # Fallback: parse from filename + m = _SWEEP_NAME_RE.search(stem) + if m: + return f"s={m.group('eps_s')}&t={m.group('eps_t')}" + + return stem + + +def write_csv(out_path: Path, xs: np.ndarray, counts: np.ndarray, label: str) -> None: + """Write a two-column CSV with header ``x,