diff --git a/AGENTS.md b/AGENTS.md index 34a5b77..345d907 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -68,6 +68,7 @@ Use this workflow when the user has an existing data model in their own system a - **api.py**: HTTP client layer (`Client`, `Response` dataclass). Handles authentication, timeouts, error mapping. - **endpoint_*.py**: Feature modules (About, Service, ModelChain, TerminateAsync). Each exports one main function. - **weather.py**: Meteorological file format specification and conversion utilities. Exports `TSV_COLUMNS` (format spec), `from_dataframe()`, `from_pvlib()`, `from_solcast()` (converters from common data sources), and `check_sequential_year_timestamps()` (TMY validator). Use when users need to convert weather data from pandas, pvlib, or Solcast into SolarFarmer's TSV format. +- **custom_rotations.py**: CSV import and protobuf serialization for custom tracker rotation schedules. Exports `from_csv()`, `csv_to_protobuf()`, and `validate_tracker_rotation_ids()`. Datasets that exceed `max_timesteps_per_file` (default 40,000) are automatically split into numbered `*001of002.gz` files. Use when users need to supply a custom rotation schedule to a 3D calculation. - **models/**: Two distinct kinds of model: - **Pydantic models** (`SolarFarmerBaseModel` subclasses, `frozen=True`): `EnergyCalculationInputs`, `PVPlant`, `Location`, `Inverter`, `Layout`, `Transformer`, etc. These are **immutable** — mutations raise `ValidationError`. Serialize with `model_dump(by_alias=True, exclude_none=True)`. - **Results models**: `CalculationResults` (in `energy_calculation_results.py`) wraps API outputs and provides convenience properties and accessors such as `performance_ratio_bifacial`, `get_performance()`, `print_annual_results()`, `loss_tree_timeseries()`, and `pvsyst_timeseries()`. @@ -190,6 +191,7 @@ The following are **named workflows** for structuring developer work. They are N | Add polling timeout logic | EndpointDev workflow | Requires config constants, async pattern | | Help SDK user run a calculation | Default referencing Quickstart | Refer to copilot-instructions.md quickstart | | Convert weather data to SF format | Default | Use `sf.from_dataframe()`, `sf.from_pvlib()`, or `sf.from_solcast()` | +| Convert custom tracker rotation CSV to protobuf | Default | Use `sf.custom_rotations.from_csv()` + `to_protobuf_file()`, or one-step `sf.custom_rotations.csv_to_protobuf()` | ## Tool Restrictions diff --git a/docs/api.md b/docs/api.md index d2513d4..4c47e08 100644 --- a/docs/api.md +++ b/docs/api.md @@ -18,6 +18,7 @@ The SolarFarmer SDK is organized into the following main categories: - [**Endpoint Functions**](#endpoint-functions): Core functions for making API calls - [**Main Classes**](#main-classes): Key data models for calculations and plant design - [**Weather Utilities**](#weather-utilities): Convert DataFrames to SolarFarmer weather files (requires `pandas`) +- [**Custom Tracker Rotations**](#custom-tracker-rotations): Convert CSV rotation schedules to protobuf format for 3D calculations ### Configuration & Design @@ -105,6 +106,45 @@ Data dictionary describing the SolarFarmer TSV weather file format: required and --- +## Custom Tracker Rotations + +Convert a CSV schedule of tracker rotation angles to the gzip-compressed +protobuf file required by SolarFarmer 3D calculations. + +### `custom_rotations.from_csv()` + +::: solarfarmer.custom_rotations.from_csv + options: + extra: + show_root_toc_entry: false + show_root_members: true + +### `custom_rotations.from_csv_folder()` + +::: solarfarmer.custom_rotations.from_csv_folder + options: + extra: + show_root_toc_entry: false + show_root_members: true + +### `custom_rotations.csv_to_protobuf()` + +::: solarfarmer.custom_rotations.csv_to_protobuf + options: + extra: + show_root_toc_entry: false + show_root_members: true + +### `custom_rotations.validate_tracker_rotation_ids()` + +::: solarfarmer.custom_rotations.validate_tracker_rotation_ids + options: + extra: + show_root_toc_entry: false + show_root_members: true + +--- + ## Main Classes The core classes handle the complete workflow from plant design to results analysis: diff --git a/docs/getting-started/custom-tracker-rotations.md b/docs/getting-started/custom-tracker-rotations.md new file mode 100644 index 0000000..52afeb1 --- /dev/null +++ b/docs/getting-started/custom-tracker-rotations.md @@ -0,0 +1,172 @@ +--- +title: Custom Tracker Rotations +description: Convert and validate custom tracker rotation CSV schedules +--- + +# Custom Tracker Rotations + +Use a CSV schedule to create the gzip-compressed protobuf file required for +custom tracker rotations in a SolarFarmer 3D calculation. + +## Convert a CSV Schedule + +```python +import solarfarmer as sf + +dataset = sf.custom_rotations.from_csv( + "tracker_rotations.csv", + offset_from_utc=1.0, + rotations_are_at_middle_of_period=False, +) +written = dataset.to_protobuf_file("TrackersConditionsDatasetDto_Protobuf.gz") +``` + +`to_protobuf_file` returns a list of the paths it wrote. Datasets with up to +40,000 timesteps produce a single file (`TrackersConditionsDatasetDto_Protobuf.gz`). +Larger datasets are split automatically into numbered parts: + +``` +TrackersConditionsDatasetDto_Protobuf001of002.gz +TrackersConditionsDatasetDto_Protobuf002of002.gz +``` + +You can override the split threshold: + +```python +dataset.to_protobuf_file("TrackersConditionsDatasetDto_Protobuf.gz", max_timesteps_per_file=20_000) +``` + +For a one-step conversion, use: + +```python +import solarfarmer as sf + +sf.custom_rotations.csv_to_protobuf( + "tracker_rotations.csv", + "TrackersConditionsDatasetDto_Protobuf.gz", + offset_from_utc=1.0, +) +``` + +Place the resulting file(s) beside the calculation JSON and other input files. +`run_energy_calculation(inputs_folder_path=...)` discovers both the single-file +and the multi-part naming patterns automatically. + +## Converting CSV Files in a Folder + +When the rotation schedule is split across multiple CSV files (for example one +file per month or per week), use `from_csv_folder` instead of `from_csv`. It +reads every `*.csv` file in the directory, validates each one independently, +sorts them by their first timestamp regardless of filename order, and merges +them into a single dataset: + +```python +import solarfarmer as sf + +dataset = sf.custom_rotations.from_csv_folder( + "rotation_csvs/", + offset_from_utc=1.0, + rotations_are_at_middle_of_period=False, +) +written = dataset.to_protobuf_file("TrackersConditionsDatasetDto_Protobuf.gz") +``` + +All files must contain **exactly the same tracker columns in the same order**. +The importer raises `ValueError` if any file has different or reordered IDs, +overlapping timestamps, or an inconsistent time period compared with the other +files. All keyword arguments accepted by `from_csv` (`offset_from_utc`, +`flip_sign`, `energy_calculation_inputs`, etc.) are forwarded to every file. + +## CSV Format + +The CSV has separate local timestamp fields followed by one column for each +tracker rotation ID: + +```text +Year,Month,Day,Hour,Minute,Second,Tracker0,Tracker1 +2025,1,1,8,0,0,-1.5,-1.5 +2025,1,1,8,5,0,-3.0,-3.0 +``` + +`Year`, `Month`, `Day`, `Hour`, and `Minute` are required. `Second` is +optional. `Azimuth` and `Zenith` are accepted only for compatibility with old +exports and are ignored. Tracker rotation IDs are matched case-insensitively, +so the CSV cannot contain both `Group1` and `group1`. + +The importer validates calendar timestamps, numeric finite angles, the valid +range from -89.90 to 89.90 degrees, duplicate headers, and strictly increasing +timestamps. It infers the base period from the timestamps; a larger gap is +allowed when a CSV contains daytime rotations only, and every output record +uses the inferred base period. + +Timestamps use the fixed `offset_from_utc` supplied to the importer. No +daylight-saving-time adjustment is applied. + +## Rotation Direction + +SolarFarmer expects negative angles in the morning and positive angles in the +afternoon. If a schedule clearly has the opposite convention, the importer +emits a warning. Reverse it explicitly when needed: + +```python +dataset = sf.custom_rotations.from_csv( + "tracker_rotations.csv", + offset_from_utc=1.0, + flip_sign=True, +) +``` + +## Validate Against Calculation Inputs + +When the calculation payload is available, check that the CSV tracker IDs +match the `trackerRotationID` values in its `pvPlant.trackers` collection. +Pass the JSON file path directly — the SDK loads and parses it automatically: + +```python +from pathlib import Path +import solarfarmer as sf + +dataset = sf.custom_rotations.from_csv( + "tracker_rotations.csv", + offset_from_utc=1.0, + energy_calculation_inputs=Path("EnergyCalculationInputs.json"), +) +``` + +You can also pass a pre-loaded dict or `EnergyCalculationInputs` object instead +of a path. The check is optional because a CSV can be prepared before the full +payload is available. IDs are compared case-insensitively but the original CSV +spelling is preserved in the generated protobuf file. + +## Things to Verify Before Running + +The following checks are not enforced by the SDK but are worth confirming +before submitting a calculation. + +**Rotation timestamps use the same fixed UTC offset as the weather file.** +The importer applies a fixed `offset_from_utc` to all timestamps; DST +transitions are not applied. Use the same offset you supplied to the weather +file converter. + +**The weather data covers the full simulation period.** +The tracker rotation timestamps define the simulation period. If a full-year +weather file is provided but the rotation data covers only three months, +SolarFarmer will simulate those three months only. Verify that weather data +is available for every timestamp in the rotation schedule. + +```python +# weather_timestamps: list of datetime objects from your weather source +# (e.g. df.index.to_pydatetime() for a pandas DataFrame) +sf.custom_rotations.check_weather_covers_rotation_period(weather_timestamps, dataset) +``` + +**The weather and rotation time resolutions are compatible.** +Both files may share the same resolution (e.g. 5-minute weather and +5-minute rotations) or use proportional resolutions (e.g. hourly weather +and 5-minute rotations). In the latter case SolarFarmer interpolates the +weather data to match the rotation timestamps. Be aware that interpolated +irradiance values may differ slightly from the original data. + +```python +sf.custom_rotations.check_compatible_time_resolutions(weather_timestamps, dataset) +``` \ No newline at end of file diff --git a/mkdocs.yml b/mkdocs.yml index 0309551..4164464 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -89,6 +89,7 @@ nav: - Workflow 1 - Load Existing API Files: getting-started/workflow-1-existing-api-files.md - Workflow 2 - Design Plants with PVSystem: getting-started/workflow-2-pvplant-builder.md - Workflow 3 - Advanced Integration: getting-started/workflow-3-plantbuilder-advanced.md + - Custom Tracker Rotations: getting-started/custom-tracker-rotations.md - Quick Start Examples: getting-started/quick-start-examples.md - End-to-End Examples: getting-started/end-to-end-examples.md - API Reference: api.md diff --git a/solarfarmer/__init__.py b/solarfarmer/__init__.py index b0b6b0d..ae92649 100644 --- a/solarfarmer/__init__.py +++ b/solarfarmer/__init__.py @@ -1,3 +1,4 @@ +from . import custom_rotations from .__version__ import __version__ from .api import SolarFarmerAPIError from .config import ( @@ -21,6 +22,16 @@ SF_PORTAL_URL, TERMINATE_ASYNC_ENDPOINT_URL, ) +from .custom_rotations import ( + RotationSignConventionWarning, + validate_tracker_rotation_ids, +) +from .custom_rotations import ( + csv_to_protobuf as custom_rotations_csv_to_protobuf, +) +from .custom_rotations import ( + from_csv as from_custom_rotations_csv, +) from .endpoint_about import about from .endpoint_modelchains import run_energy_calculation from .endpoint_service import service @@ -87,8 +98,11 @@ __all__ = [ "__version__", + "custom_rotations", + "custom_rotations_csv_to_protobuf", "about", "configure_logging", + "from_custom_rotations_csv", # Config constants "ABOUT_ENDPOINT_URL", "ANNUAL_MONTHLY_RESULTS_FILENAME", @@ -168,4 +182,6 @@ "from_pvlib", "from_solcast", "check_sequential_year_timestamps", + "RotationSignConventionWarning", + "validate_tracker_rotation_ids", ] diff --git a/solarfarmer/custom_rotations.py b/solarfarmer/custom_rotations.py new file mode 100644 index 0000000..fc07bd3 --- /dev/null +++ b/solarfarmer/custom_rotations.py @@ -0,0 +1,652 @@ +"""Import custom tracker rotation schedules from SolarFarmer CSV files. + +CSV files use separate timestamp columns followed by one column per tracker +rotation ID:: + + Year,Month,Day,Hour,Minute,Second,Tracker0,Tracker1 + 2025,1,1,8,0,0,-1.5,-1.5 + +The ``Second`` column is optional. ``Azimuth`` and ``Zenith`` columns from +older SolarFarmer exports are ignored. Rotation values are decimal degrees and +are converted to the integer centidegrees required by the protobuf format. +""" + +from __future__ import annotations + +import csv +import json +import math +import warnings +from collections.abc import Mapping, Sequence +from datetime import datetime, timedelta, timezone +from decimal import ROUND_HALF_UP, Decimal, InvalidOperation +from pathlib import Path +from typing import Any + +from .models.energy_calculation_inputs import EnergyCalculationInputs +from .models.trackers_conditions_dataset import TrackerCondition, TrackersConditionsDataset + +__all__ = [ + "RotationSignConventionWarning", + "check_compatible_time_resolutions", + "check_weather_covers_rotation_period", + "csv_to_protobuf", + "from_csv", + "from_csv_folder", + "validate_tracker_rotation_ids", +] + + +_REQUIRED_TIMESTAMP_COLUMNS = ("year", "month", "day", "hour", "minute") +_OPTIONAL_TIMESTAMP_COLUMNS = {"second"} +_IGNORED_COLUMNS = {"azimuth", "zenith"} +_MAX_ROTATION_CENTIDEGREES = 8990 + + +class RotationSignConventionWarning(UserWarning): + """Warn when a CSV appears to use the opposite tracker angle convention.""" + + +def from_csv( + path: str | Path, + *, + offset_from_utc: float = 0.0, + rotations_are_at_middle_of_period: bool = False, + flip_sign: bool = False, + energy_calculation_inputs: EnergyCalculationInputs + | Mapping[str, Any] + | Path + | str + | None = None, +) -> TrackersConditionsDataset: + """Read a custom tracker rotation CSV file into a validated dataset. + + Parameters + ---------- + path : str or Path + CSV file with ``Year, Month, Day, Hour, Minute[, Second]`` columns + followed by one or more tracker rotation columns. Header matching is + case-insensitive; legacy ``Azimuth`` and ``Zenith`` columns are ignored. + offset_from_utc : float, default 0.0 + Fixed UTC offset in hours used for all CSV timestamps. Fixed offsets do + not apply daylight-saving time. + rotations_are_at_middle_of_period : bool, default False + Whether each rotation value represents the middle rather than the start + of its period. + flip_sign : bool, default False + Negate all input angles before conversion. Use when a schedule uses + positive morning and negative afternoon rotations. + energy_calculation_inputs : EnergyCalculationInputs, mapping, or path, optional + Parsed calculation inputs, a mapping using either API camel-case or + Python snake-case keys, or a path to a JSON file containing the + calculation inputs. Used to verify that CSV tracker rotation IDs + match the plant's tracker IDs. + + Returns + ------- + TrackersConditionsDataset + Dataset ready for :meth:`TrackersConditionsDataset.to_protobuf_file`. + + Raises + ------ + ValueError + If headers, timestamps, rotation values, or the inferred time cadence + are invalid. + """ + tz = _timezone_from_offset(offset_from_utc) + csv_path = Path(path) + + with csv_path.open("r", encoding="utf-8-sig", newline="") as file_handle: + reader = csv.DictReader(file_handle) + tracker_ids = _parse_headers(reader.fieldnames) + rows = list(reader) + + if len(rows) < 2: + raise ValueError("CSV must contain at least two timestamped data rows to infer a period") + + conditions: list[TrackerCondition] = [] + timestamps: list[datetime] = [] + morning_values: list[float] = [] + afternoon_values: list[float] = [] + + for line_number, row in enumerate(rows, start=2): + if None in row: + raise ValueError(f"row {line_number} contains more values than its header") + if all(value is None or not value.strip() for value in row.values()): + continue + + timestamp = _parse_timestamp(row, line_number, tz) + rotations = [ + _parse_rotation(row[tracker_id], line_number, tracker_id, flip_sign) + for tracker_id in tracker_ids + ] + timestamps.append(timestamp) + if timestamp.hour < 12: + morning_values.extend(rotations) + else: + afternoon_values.extend(rotations) + conditions.append( + TrackerCondition( + period_in_minutes=1.0, + start_of_period=timestamp, + tracker_rotations_array_values=[_to_centidegrees(value) for value in rotations], + ) + ) + + if len(conditions) < 2: + raise ValueError("CSV must contain at least two non-empty data rows to infer a period") + + period_in_minutes = _infer_period_in_minutes(timestamps) + conditions = [ + condition.model_copy(update={"period_in_minutes": period_in_minutes}) + for condition in conditions + ] + + if not flip_sign and _uses_opposite_sign_convention(morning_values, afternoon_values): + warnings.warn( + "Rotation angles appear positive in the morning and negative in the afternoon. " + "SolarFarmer expects negative morning and positive afternoon rotations; " + "pass flip_sign=True to reverse the schedule.", + RotationSignConventionWarning, + stacklevel=2, + ) + + dataset = TrackersConditionsDataset( + data=conditions, + offset_from_utc=offset_from_utc, + rotations_are_at_middle_of_period=rotations_are_at_middle_of_period, + tracker_rotation_ids=tracker_ids, + ) + if energy_calculation_inputs is not None: + validate_tracker_rotation_ids(dataset, energy_calculation_inputs) + return dataset + + +def from_csv_folder( + folder: str | Path, + *, + offset_from_utc: float = 0.0, + rotations_are_at_middle_of_period: bool = False, + flip_sign: bool = False, + energy_calculation_inputs: EnergyCalculationInputs + | Mapping[str, Any] + | Path + | str + | None = None, +) -> TrackersConditionsDataset: + """Read all CSV files in a folder and merge them into a single dataset. + + Each CSV file is parsed with :func:`from_csv` using the same keyword + arguments. Files may cover different date ranges (e.g. one file per month + or per day) and are sorted chronologically by their first timestamp before + being merged. Filesystem or alphabetical order is therefore irrelevant. + + All files must contain exactly the same set of tracker-ID columns **in the + same order**. The merged dataset uses the tracker-ID list from the first + file (after chronological sorting). + + Parameters + ---------- + folder : str or Path + Directory that contains the ``*.csv`` rotation files. + offset_from_utc : float, default 0.0 + Fixed UTC offset in hours applied to all files. + rotations_are_at_middle_of_period : bool, default False + Whether each rotation value represents the middle rather than the start + of its period. + flip_sign : bool, default False + Negate all input angles before conversion. + energy_calculation_inputs : EnergyCalculationInputs, mapping, or path, optional + When provided, validates that the merged tracker-rotation IDs match the + tracker IDs in the calculation inputs. + + Returns + ------- + TrackersConditionsDataset + Merged dataset covering the combined date range of all files. + + Raises + ------ + ValueError + If no CSV files are found, tracker-ID columns differ between files, + timestamps overlap across file boundaries, or time periods are + inconsistent across files. + """ + folder_path = Path(folder) + csv_files = sorted(folder_path.glob("*.csv")) + if not csv_files: + raise ValueError(f"No CSV files found in folder '{folder_path}'") + + # Parse each file independently — from_csv validates headers, timestamps, + # and rotation values per file. Keep (path, dataset) together so that file + # names remain correct after chronological sorting. + parsed: list[tuple[Path, TrackersConditionsDataset]] = [ + ( + csv_file, + from_csv( + csv_file, + offset_from_utc=offset_from_utc, + rotations_are_at_middle_of_period=rotations_are_at_middle_of_period, + flip_sign=flip_sign, + ), + ) + for csv_file in csv_files + ] + + # Sort by first timestamp so file naming does not dictate order. + parsed.sort(key=lambda item: item[1].data[0].start_of_period) + + # Validate tracker-ID columns are identical (same IDs, same order) across all files. + reference_path, reference_dataset = parsed[0] + reference_ids = reference_dataset.tracker_rotation_ids + for file_path, dataset in parsed[1:]: + if dataset.tracker_rotation_ids != reference_ids: + raise ValueError( + f"'{file_path.name}' has different tracker-ID columns than '{reference_path.name}'. " + "All CSV files in the folder must have the same tracker columns in the same order." + ) + + # Validate all files inferred the same period. + periods = {dataset.data[0].period_in_minutes for _, dataset in parsed} + if len(periods) > 1: + raise ValueError( + f"CSV files have inconsistent time periods (minutes): {sorted(periods)}. " + "All files in the folder must use the same time resolution." + ) + + # Merge and verify no overlapping timestamps at file boundaries. + merged_data: list[TrackerCondition] = [] + for _, dataset in parsed: + if merged_data: + last_ts = merged_data[-1].start_of_period + first_ts = dataset.data[0].start_of_period + if first_ts <= last_ts: + raise ValueError( + f"Timestamps overlap at file boundary: " + f"{last_ts} (end of previous file) >= {first_ts} (start of next file)" + ) + merged_data.extend(dataset.data) + + merged = TrackersConditionsDataset( + data=merged_data, + offset_from_utc=offset_from_utc, + rotations_are_at_middle_of_period=rotations_are_at_middle_of_period, + tracker_rotation_ids=reference_ids, + ) + if energy_calculation_inputs is not None: + validate_tracker_rotation_ids(merged, energy_calculation_inputs) + return merged + + +def csv_to_protobuf( + csv_path: str | Path, + output_path: str | Path, + max_timesteps_per_file: int = 40_000, + **kwargs: Any, +) -> TrackersConditionsDataset: + """Convert a custom rotations CSV file directly to a protobuf gzip file. + + Parameters + ---------- + csv_path : str or Path + Source CSV path accepted by :func:`from_csv`. + output_path : str or Path + Destination path for the gzip-compressed protobuf output. When the + dataset contains more than *max_timesteps_per_file* timesteps this + path is used as the base name and multiple files are written with the + ``{part:03d}of{total:03d}`` suffix (e.g. + ``TrackersConditionsDatasetDto_Protobuf001of002.gz``). + max_timesteps_per_file : int, default 40_000 + Maximum timesteps per protobuf file. Passed to + :meth:`TrackersConditionsDataset.to_protobuf_file`. + **kwargs + Keyword arguments accepted by :func:`from_csv`. + + Returns + ------- + TrackersConditionsDataset + The validated dataset that was serialized. + """ + dataset = from_csv(csv_path, **kwargs) + dataset.to_protobuf_file(output_path, max_timesteps_per_file=max_timesteps_per_file) + return dataset + + +def validate_tracker_rotation_ids( + dataset: TrackersConditionsDataset, + energy_calculation_inputs: EnergyCalculationInputs | Mapping[str, Any] | Path | str, +) -> None: + """Validate that dataset rotation IDs match tracker IDs in calculation inputs. + + Comparisons are case-insensitive, but IDs remain unchanged in the dataset. + + Parameters + ---------- + dataset : TrackersConditionsDataset + Dataset whose ``tracker_rotation_ids`` are validated. + energy_calculation_inputs : EnergyCalculationInputs, mapping, or path + Parsed calculation inputs, a mapping using either API camel-case or + Python snake-case keys, or a path to a JSON file containing the + calculation inputs. + + Raises + ------ + ValueError + If the payload has no trackers or the two ID sets differ. + """ + inputs = _coerce_energy_calculation_inputs(energy_calculation_inputs) + payload_ids = _tracker_rotation_ids_from_inputs(inputs) + dataset_ids = { + rotation_id.casefold(): rotation_id for rotation_id in dataset.tracker_rotation_ids + } + + missing_from_payload = sorted( + rotation_id for key, rotation_id in dataset_ids.items() if key not in payload_ids + ) + missing_from_csv = sorted( + rotation_id for key, rotation_id in payload_ids.items() if key not in dataset_ids + ) + if missing_from_payload or missing_from_csv: + details: list[str] = [] + if missing_from_payload: + details.append(f"CSV IDs not found in calculation inputs: {missing_from_payload}") + if missing_from_csv: + details.append(f"calculation-input IDs missing from CSV: {missing_from_csv}") + raise ValueError("Tracker rotation IDs do not match: " + "; ".join(details)) + + +def check_weather_covers_rotation_period( + weather_timestamps: Sequence[datetime], + dataset: TrackersConditionsDataset, +) -> None: + """Verify that weather timestamps span the full rotation dataset period. + + The tracker rotation timestamps define the simulation period. If weather + data does not cover the full rotation period, the simulation will be + truncated to the overlapping window. + + Weather and rotation timestamps must use the same reference time (both + timezone-aware with the same UTC offset, or both timezone-naive). + + Parameters + ---------- + weather_timestamps : Sequence[datetime] + Timestamps from the weather file, in any order. + dataset : TrackersConditionsDataset + Rotation dataset whose first and last timestamps define the simulation + window. + + Raises + ------ + ValueError + If *weather_timestamps* is empty or does not fully cover the rotation + period. + """ + if not dataset.data: + return + + rotation_start = dataset.data[0].start_of_period + rotation_end = dataset.data[-1].start_of_period + + if not weather_timestamps: + raise ValueError( + f"weather_timestamps is empty; cannot verify coverage of rotation period " + f"{rotation_start} \u2013 {rotation_end}" + ) + + weather_start = min(weather_timestamps) + weather_end = max(weather_timestamps) + + errors: list[str] = [] + if weather_start > rotation_start: + errors.append( + f"weather starts at {weather_start} which is after " + f"the first rotation timestamp {rotation_start}" + ) + if weather_end < rotation_end: + errors.append( + f"weather ends at {weather_end} which is before " + f"the last rotation timestamp {rotation_end}" + ) + if errors: + raise ValueError( + "Weather data does not fully cover the rotation period: " + "; ".join(errors) + ) + + +def check_compatible_time_resolutions( + weather_timestamps: Sequence[datetime], + dataset: TrackersConditionsDataset, +) -> None: + """Verify that the weather and rotation time resolutions are compatible. + + Compatible means one period is a whole multiple of the other (e.g. hourly + weather with 5-minute rotations, or both at the same resolution). When the + resolutions differ, SolarFarmer interpolates the coarser dataset to match + the finer one. + + Parameters + ---------- + weather_timestamps : Sequence[datetime] + Timestamps from the weather file, in ascending order. At least two + timestamps are required to infer the period. When using a pandas + DataFrame, pass ``df.index.to_pydatetime()``. + dataset : TrackersConditionsDataset + Rotation dataset whose ``period_in_minutes`` is checked against the + inferred weather period. + + Raises + ------ + ValueError + If fewer than two weather timestamps are supplied, the weather + timestamps are not strictly increasing, or the resolutions are not + compatible. + """ + if not dataset.data: + return + + if len(weather_timestamps) < 2: + raise ValueError("at least two weather timestamps are required to infer the resolution") + + sorted_ts = sorted(weather_timestamps) + weather_deltas_s = [ + int((later - earlier).total_seconds()) + for earlier, later in zip(sorted_ts, sorted_ts[1:], strict=False) + ] + weather_period_s = min(weather_deltas_s) + if weather_period_s <= 0: + raise ValueError("weather timestamps must be strictly increasing") + + rotation_period_s = round(dataset.data[0].period_in_minutes * 60) + + longer = max(weather_period_s, rotation_period_s) + shorter = min(weather_period_s, rotation_period_s) + + if longer % shorter != 0: + weather_min = weather_period_s / 60 + rotation_min = dataset.data[0].period_in_minutes + raise ValueError( + f"Incompatible time resolutions: weather period is {weather_min:g} min, " + f"rotation period is {rotation_min:g} min. " + "One must be a whole multiple of the other." + ) + + +def _parse_headers(fieldnames: list[str] | None) -> list[str]: + if not fieldnames: + raise ValueError("CSV must contain a header row") + + normalized_headers: dict[str, str] = {} + for header in fieldnames: + if header is None or not header.strip(): + raise ValueError("CSV contains an empty column header") + cleaned = header.strip() + normalized = cleaned.casefold() + if normalized in normalized_headers: + raise ValueError( + f"CSV contains duplicate case-insensitive headers: " + f"'{normalized_headers[normalized]}' and '{cleaned}'" + ) + normalized_headers[normalized] = cleaned + + missing = [ + name.title() for name in _REQUIRED_TIMESTAMP_COLUMNS if name not in normalized_headers + ] + if missing: + raise ValueError(f"CSV is missing required timestamp columns: {missing}") + + excluded = { + *_REQUIRED_TIMESTAMP_COLUMNS, + *_OPTIONAL_TIMESTAMP_COLUMNS, + *_IGNORED_COLUMNS, + } + tracker_ids = [ + header.strip() for header in fieldnames if header.strip().casefold() not in excluded + ] + if not tracker_ids: + raise ValueError("CSV must contain at least one tracker rotation column") + return tracker_ids + + +def _parse_timestamp(row: Mapping[str, str | None], line_number: int, tz: timezone) -> datetime: + normalized = {key.strip().casefold(): value for key, value in row.items() if key is not None} + parts: dict[str, int] = {} + for name in _REQUIRED_TIMESTAMP_COLUMNS: + parts[name] = _parse_integer(normalized.get(name), line_number, name.title()) + parts["second"] = _parse_integer(normalized.get("second", "0"), line_number, "Second") + try: + return datetime(tzinfo=tz, **parts) + except ValueError as exc: + raise ValueError(f"row {line_number} has an invalid timestamp: {exc}") from None + + +def _parse_integer(value: str | None, line_number: int, column: str) -> int: + if value is None or not value.strip(): + raise ValueError(f"row {line_number}, column '{column}' must be an integer") + try: + return int(value) + except ValueError: + raise ValueError(f"row {line_number}, column '{column}' must be an integer") from None + + +def _parse_rotation(value: str | None, line_number: int, tracker_id: str, flip_sign: bool) -> float: + if value is None or not value.strip(): + raise ValueError(f"row {line_number}, tracker '{tracker_id}' must have a rotation value") + try: + rotation = float(value) + except ValueError: + raise ValueError( + f"row {line_number}, tracker '{tracker_id}' rotation must be numeric" + ) from None + if not math.isfinite(rotation): + raise ValueError(f"row {line_number}, tracker '{tracker_id}' rotation must be finite") + if flip_sign: + rotation = -rotation + if abs(rotation) > _MAX_ROTATION_CENTIDEGREES / 100: + raise ValueError( + f"row {line_number}, tracker '{tracker_id}' rotation must be in [-89.90, 89.90] degrees" + ) + return rotation + + +def _to_centidegrees(rotation: float) -> int: + try: + return int((Decimal(str(rotation)) * 100).quantize(Decimal("1"), rounding=ROUND_HALF_UP)) + except InvalidOperation: + raise ValueError("rotation cannot be represented as centidegrees") from None + + +def _infer_period_in_minutes(timestamps: list[datetime]) -> float: + deltas = [ + int((later - earlier).total_seconds()) + for earlier, later in zip(timestamps, timestamps[1:], strict=False) + ] + if any(delta <= 0 for delta in deltas): + raise ValueError("CSV timestamps must be strictly increasing without duplicates") + + cadence_seconds = min(deltas) + if any(delta % cadence_seconds for delta in deltas): + raise ValueError( + "CSV timestamps must use a consistent cadence; larger daytime-only gaps must be " + "whole multiples of the inferred period" + ) + return cadence_seconds / 60 + + +def _timezone_from_offset(offset_from_utc: float) -> timezone: + if not math.isfinite(offset_from_utc) or not -12 <= offset_from_utc <= 14: + raise ValueError("offset_from_utc must be a finite value in the range [-12, 14]") + return timezone(timedelta(hours=offset_from_utc)) + + +def _uses_opposite_sign_convention( + morning_values: list[float], afternoon_values: list[float] +) -> bool: + morning_nonzero = [value for value in morning_values if value != 0] + afternoon_nonzero = [value for value in afternoon_values if value != 0] + if not morning_nonzero or not afternoon_nonzero: + return False + + morning_positive = sum(value > 0 for value in morning_nonzero) / len(morning_nonzero) + afternoon_negative = sum(value < 0 for value in afternoon_nonzero) / len(afternoon_nonzero) + return morning_positive >= 0.75 and afternoon_negative >= 0.75 + + +def _coerce_energy_calculation_inputs( + energy_calculation_inputs: EnergyCalculationInputs | Mapping[str, Any] | Path | str, +) -> EnergyCalculationInputs | Mapping[str, Any]: + """Load calculation inputs from a JSON file path if needed. + + If *energy_calculation_inputs* is a :class:`~pathlib.Path` or :class:`str`, + it is treated as a path to a JSON file and parsed into a mapping. + Otherwise the value is returned unchanged. + """ + if isinstance(energy_calculation_inputs, (str, Path)): + path = Path(energy_calculation_inputs) + try: + data: Mapping[str, Any] = json.loads(path.read_text(encoding="utf-8")) + except OSError as exc: + raise ValueError( + f"Could not read energy_calculation_inputs from '{path}': {exc}" + ) from exc + except json.JSONDecodeError as exc: + raise ValueError(f"Could not parse '{path}' as JSON: {exc}") from exc + if not isinstance(data, Mapping): + raise ValueError(f"Expected a JSON object in '{path}', got {type(data).__name__}") + return data + return energy_calculation_inputs + + +def _tracker_rotation_ids_from_inputs( + energy_calculation_inputs: EnergyCalculationInputs | Mapping[str, Any], +) -> dict[str, str]: + if isinstance(energy_calculation_inputs, EnergyCalculationInputs): + trackers = energy_calculation_inputs.pv_plant.trackers or [] + ids = [tracker.tracker_rotation_id for tracker in trackers] + else: + plant = energy_calculation_inputs.get("pvPlant") or energy_calculation_inputs.get( + "pv_plant" + ) + if not isinstance(plant, Mapping): + raise ValueError("calculation inputs must contain a pvPlant/pv_plant mapping") + trackers = plant.get("trackers") or [] + if not isinstance(trackers, list): + raise ValueError("calculation inputs pvPlant.trackers must be a list") + ids = [] + for index, tracker in enumerate(trackers): + if not isinstance(tracker, Mapping): + raise ValueError(f"calculation inputs tracker {index} must be a mapping") + rotation_id = tracker.get("trackerRotationID") or tracker.get("tracker_rotation_id") + if not isinstance(rotation_id, str) or not rotation_id.strip(): + raise ValueError( + f"calculation inputs tracker {index} must define trackerRotationID/tracker_rotation_id" + ) + ids.append(rotation_id) + + if not ids: + raise ValueError("calculation inputs do not define any tracker rotation IDs") + normalized_ids: dict[str, str] = {} + for rotation_id in ids: + normalized = rotation_id.casefold() + normalized_ids.setdefault(normalized, rotation_id) + return normalized_ids diff --git a/solarfarmer/models/trackers_conditions_dataset.py b/solarfarmer/models/trackers_conditions_dataset.py index a4be80b..6d49cf9 100644 --- a/solarfarmer/models/trackers_conditions_dataset.py +++ b/solarfarmer/models/trackers_conditions_dataset.py @@ -1,6 +1,7 @@ from __future__ import annotations import gzip +import math import re from collections.abc import Iterable from datetime import datetime, timedelta, timezone @@ -266,23 +267,70 @@ def from_protobuf_dir(cls, directory: Path | str) -> TrackersConditionsDataset: f"No TrackersConditionsDatasetDto_Protobuf*.gz files found in {base}" ) - def to_protobuf_file(self, path: Path | str) -> None: - """Serialize to a gzip-compressed protobuf file. + def to_protobuf_file( + self, + path: Path | str, + max_timesteps_per_file: int = 40_000, + ) -> list[Path]: + """Serialize to one or more gzip-compressed protobuf files. + + When the dataset contains more than *max_timesteps_per_file* timesteps + the data is split across multiple files using the SolarFarmer multi-part + naming convention: + + * **Single file** (data fits in one part): the file is written to + *path* unchanged, e.g. ``TrackersConditionsDatasetDto_Protobuf.gz``. + * **Multiple files**: the stem of *path* is used as the base name and + a ``{part:03d}of{total:03d}`` suffix is inserted before ``.gz``, e.g. + ``TrackersConditionsDatasetDto_Protobuf001of002.gz``. Parameters ---------- path : Path or str - Destination path. The file is written (or overwritten) atomically - using gzip compression. + Destination path (or base path for multi-part output). + max_timesteps_per_file : int, default 40_000 + Maximum number of timesteps written to a single file before + splitting into an additional file. + + Returns + ------- + list[Path] + Paths of every file that was written, in order. + + Raises + ------ + ValueError + If *max_timesteps_per_file* is not a positive integer. """ + if max_timesteps_per_file <= 0: + raise ValueError("max_timesteps_per_file must be greater than 0") + from solarfarmer.models._proto.trackers_conditions_dataset_pb2 import ( # noqa: PLC0415 TrackersConditionsDatasetDto, ) - dto = _dataset_to_dto(self, TrackersConditionsDatasetDto) - raw = dto.SerializeToString() - with gzip.open(Path(path), "wb") as fh: - fh.write(raw) + n = len(self.data) + base = Path(path) + total_files = max(1, math.ceil(n / max_timesteps_per_file)) + + if total_files == 1: + dto = _dataset_to_dto(self, TrackersConditionsDatasetDto) + _write_dto_to_gz(dto, base) + return [base] + + # Split into multiple files + stem = base.stem # strips the final .gz suffix + parent = base.parent + written: list[Path] = [] + for i in range(total_files): + chunk_path = parent / f"{stem}{i + 1:03d}of{total_files:03d}.gz" + start = i * max_timesteps_per_file + end = min(start + max_timesteps_per_file, n) + chunk = self.model_copy(update={"data": self.data[start:end]}) + dto = _dataset_to_dto(chunk, TrackersConditionsDatasetDto) + _write_dto_to_gz(dto, chunk_path) + written.append(chunk_path) + return written # --------------------------------------------------------------------------- @@ -291,6 +339,13 @@ def to_protobuf_file(self, path: Path | str) -> None: # --------------------------------------------------------------------------- +def _write_dto_to_gz(dto: object, path: Path) -> None: + """Serialize a protobuf DTO to a gzip-compressed file.""" + raw = dto.SerializeToString() # type: ignore[union-attr] + with gzip.open(path, "wb") as fh: + fh.write(raw) + + def _dto_to_dataset(dto: object) -> TrackersConditionsDataset: """Convert a ``TrackersConditionsDatasetDto`` protobuf message to the domain model.""" n = len(dto.start_of_period) # type: ignore[union-attr] diff --git a/tests/test_custom_rotations.py b/tests/test_custom_rotations.py new file mode 100644 index 0000000..057c32b --- /dev/null +++ b/tests/test_custom_rotations.py @@ -0,0 +1,636 @@ +"""Tests for custom tracker rotations CSV parsing and protobuf export.""" + +import json +from datetime import datetime, timedelta, timezone + +import pytest + +from solarfarmer.custom_rotations import ( + RotationSignConventionWarning, + check_compatible_time_resolutions, + check_weather_covers_rotation_period, + csv_to_protobuf, + from_csv, + from_csv_folder, + validate_tracker_rotation_ids, +) +from solarfarmer.models import ( + DiffuseModel, + EnergyCalculationInputs, + EnergyCalculationOptions, + Location, + MonthlyAlbedo, + PVPlant, + Tracker, + TrackerCondition, + TrackersConditionsDataset, + Vector3Double, +) + + +def _write_csv(tmp_path, content: str): + path = tmp_path / "rotations.csv" + path.write_text(content, encoding="utf-8") + return path + + +def _make_dataset_with_timestamps( + timestamps: list[datetime], period_minutes: float = 5.0 +) -> TrackersConditionsDataset: + """Create a minimal TrackersConditionsDataset with given timestamps.""" + conditions = [ + TrackerCondition( + period_in_minutes=period_minutes, + start_of_period=ts, + tracker_rotations_array_values=[-100], + ) + for ts in timestamps + ] + return TrackersConditionsDataset(data=conditions, tracker_rotation_ids=["T0"]) + + +class TestFromCsv: + def test_parses_rotations_and_ignores_legacy_columns(self, tmp_path) -> None: + path = _write_csv( + tmp_path, + "\n".join( + [ + "Year,Month,Day,Hour,Minute,Second,Azimuth,Zenith,North,South", + "2025,4,1,8,0,0,120,45,-1.25,-2.50", + "2025,4,1,8,5,0,121,46,-3.50,-4.75", + "2025,4,1,8,10,0,122,47,-5.00,-6.25", + ] + ), + ) + + dataset = from_csv( + path, + offset_from_utc=5.5, + rotations_are_at_middle_of_period=True, + ) + + assert dataset.tracker_rotation_ids == ["North", "South"] + assert dataset.offset_from_utc == 5.5 + assert dataset.rotations_are_at_middle_of_period is True + assert dataset.data[0].start_of_period.tzinfo == timezone(timedelta(hours=5.5)) + assert [condition.period_in_minutes for condition in dataset.data] == [5.0, 5.0, 5.0] + assert dataset.data[0].tracker_rotations_array_values == [-125, -250] + assert dataset.data[1].tracker_rotations_array_values == [-350, -475] + + def test_accepts_csv_without_second_column(self, tmp_path) -> None: + path = _write_csv( + tmp_path, + "\n".join( + [ + "Year,Month,Day,Hour,Minute,Row_A", + "2025,4,1,8,0,-1", + "2025,4,1,8,10,-2", + ] + ), + ) + + dataset = from_csv(path) + + assert dataset.data[0].start_of_period.second == 0 + assert dataset.data[0].period_in_minutes == 10.0 + + def test_daytime_only_gap_uses_base_period(self, tmp_path) -> None: + path = _write_csv( + tmp_path, + "\n".join( + [ + "Year,Month,Day,Hour,Minute,Tracker_A", + "2025,4,1,8,0,-1", + "2025,4,1,8,5,-2", + "2025,4,1,17,0,3", + ] + ), + ) + + dataset = from_csv(path) + + assert [condition.period_in_minutes for condition in dataset.data] == [5.0, 5.0, 5.0] + + @pytest.mark.parametrize("offset_from_utc", [-12, 14]) + def test_accepts_civil_timezone_boundaries(self, tmp_path, offset_from_utc) -> None: + path = _write_csv( + tmp_path, + "\n".join( + [ + "Year,Month,Day,Hour,Minute,Tracker_A", + "2025,4,1,8,0,-1", + "2025,4,1,8,5,-2", + ] + ), + ) + + dataset = from_csv(path, offset_from_utc=offset_from_utc) + + assert dataset.offset_from_utc == offset_from_utc + + def test_rejects_offset_before_civil_timezone_range(self, tmp_path) -> None: + path = _write_csv( + tmp_path, + "\n".join( + [ + "Year,Month,Day,Hour,Minute,Tracker_A", + "2025,4,1,8,0,-1", + "2025,4,1,8,5,-2", + ] + ), + ) + + with pytest.raises(ValueError, match=r"\[-12, 14\]"): + from_csv(path, offset_from_utc=-12.5) + + def test_reversed_sign_warns_and_flip_sign_corrects(self, tmp_path) -> None: + path = _write_csv( + tmp_path, + "\n".join( + [ + "Year,Month,Day,Hour,Minute,Tracker_A", + "2025,4,1,9,0,15", + "2025,4,1,9,5,20", + "2025,4,1,13,0,-15", + "2025,4,1,13,5,-20", + ] + ), + ) + + with pytest.warns(RotationSignConventionWarning, match="flip_sign=True"): + from_csv(path) + + dataset = from_csv(path, flip_sign=True) + assert dataset.data[0].tracker_rotations_array_values == [-1500] + assert dataset.data[-1].tracker_rotations_array_values == [2000] + + @pytest.mark.parametrize( + ("header", "rows", "match"), + [ + ( + "Year,Month,Day,Hour,Second,Tracker_A", + ["2025,4,1,8,0,-1", "2025,4,1,8,5,-2"], + "required timestamp", + ), + ( + "Year,Month,Day,Hour,Minute,Tracker_A,tracker_a", + ["2025,4,1,8,0,-1,-2", "2025,4,1,8,5,-2,-3"], + "duplicate case-insensitive", + ), + ( + "Year,Month,Day,Hour,Minute,Tracker_A", + ["2025,4,1,8,0,not-a-number", "2025,4,1,8,5,-2"], + "must be numeric", + ), + ], + ) + def test_rejects_invalid_csv(self, tmp_path, header, rows, match) -> None: + path = _write_csv(tmp_path, "\n".join([header, *rows])) + + with pytest.raises(ValueError, match=match): + from_csv(path) + + def test_rejects_duplicate_timestamps(self, tmp_path) -> None: + path = _write_csv( + tmp_path, + "\n".join( + [ + "Year,Month,Day,Hour,Minute,Tracker_A", + "2025,4,1,8,0,-1", + "2025,4,1,8,0,-2", + ] + ), + ) + + with pytest.raises(ValueError, match="strictly increasing"): + from_csv(path) + + def test_rejects_decreasing_timestamps(self, tmp_path) -> None: + path = _write_csv( + tmp_path, + "\n".join( + [ + "Year,Month,Day,Hour,Minute,Tracker_A", + "2025,4,1,8,5,-1", + "2025,4,1,8,0,-2", + ] + ), + ) + + with pytest.raises(ValueError, match="strictly increasing"): + from_csv(path) + + @pytest.mark.parametrize("rotation", [89.90, -89.90]) + def test_allows_rotation_at_boundary(self, tmp_path, rotation) -> None: + path = _write_csv( + tmp_path, + "\n".join( + [ + "Year,Month,Day,Hour,Minute,Tracker_A", + f"2025,4,1,8,0,{rotation}", + "2025,4,1,8,5,0", + ] + ), + ) + + dataset = from_csv(path) + + assert abs(dataset.data[0].tracker_rotations_array_values[0]) == 8990 + + @pytest.mark.parametrize("rotation", [89.91, -89.91]) + def test_rejects_rotation_above_boundary(self, tmp_path, rotation) -> None: + path = _write_csv( + tmp_path, + "\n".join( + [ + "Year,Month,Day,Hour,Minute,Tracker_A", + f"2025,4,1,8,0,{rotation}", + "2025,4,1,8,5,0", + ] + ), + ) + + with pytest.raises(ValueError, match=r"\[-89.90, 89.90\]"): + from_csv(path) + + +class TestPayloadValidation: + def test_matches_payload_ids_case_insensitively(self, tmp_path) -> None: + path = _write_csv( + tmp_path, + "\n".join( + [ + "Year,Month,Day,Hour,Minute,North,South", + "2025,4,1,8,0,-1,-2", + "2025,4,1,8,5,-2,-3", + ] + ), + ) + payload = { + "pvPlant": { + "trackers": [ + {"trackerRotationID": "NORTH"}, + {"trackerRotationID": "south"}, + ] + } + } + + dataset = from_csv(path, energy_calculation_inputs=payload) + validate_tracker_rotation_ids(dataset, payload) + + def test_rejects_payload_id_mismatch(self, tmp_path) -> None: + path = _write_csv( + tmp_path, + "\n".join( + [ + "Year,Month,Day,Hour,Minute,North", + "2025,4,1,8,0,-1", + "2025,4,1,8,5,-2", + ] + ), + ) + payload = {"pvPlant": {"trackers": [{"trackerRotationID": "South"}]}} + + with pytest.raises(ValueError, match="do not match"): + from_csv(path, energy_calculation_inputs=payload) + + def test_accepts_energy_calculation_inputs_model(self, tmp_path) -> None: + path = _write_csv( + tmp_path, + "\n".join( + [ + "Year,Month,Day,Hour,Minute,Group_A", + "2025,4,1,8,0,-1", + "2025,4,1,8,5,-2", + ] + ), + ) + point = Vector3Double(x=0, y=0, z=0) + tracker = Tracker( + id=1, + mounting_type_id="mt", + north_point=point, + south_point=point, + tracker_rotation_id="Group_A", + tracker_system_id="ts", + ) + inputs = EnergyCalculationInputs( + location=Location(latitude=0, longitude=0), + pv_plant=PVPlant( + transformers=[], + mounting_type_specifications={}, + trackers=[tracker], + ), + monthly_albedo=MonthlyAlbedo(values=[0.2] * 12), + energy_calculation_options=EnergyCalculationOptions( + diffuse_model=DiffuseModel.PEREZ, + include_horizon=False, + ), + ) + + dataset = from_csv(path, energy_calculation_inputs=inputs) + + assert dataset.tracker_rotation_ids == ["Group_A"] + + def test_accepts_json_file_path(self, tmp_path) -> None: + csv_path = _write_csv( + tmp_path, + "\n".join( + [ + "Year,Month,Day,Hour,Minute,North,South", + "2025,4,1,8,0,-1,-2", + "2025,4,1,8,5,-2,-3", + ] + ), + ) + payload = { + "pvPlant": { + "trackers": [ + {"trackerRotationID": "North"}, + {"trackerRotationID": "South"}, + ] + } + } + json_path = tmp_path / "inputs.json" + json_path.write_text(json.dumps(payload), encoding="utf-8") + + dataset = from_csv(csv_path, energy_calculation_inputs=json_path) + + assert dataset.tracker_rotation_ids == ["North", "South"] + + +class TestProtobufExport: + def test_writes_readable_protobuf(self, tmp_path) -> None: + csv_path = _write_csv( + tmp_path, + "\n".join( + [ + "Year,Month,Day,Hour,Minute,Tracker_A", + "2025,4,1,8,0,-1.25", + "2025,4,1,8,5,2.50", + ] + ), + ) + output_path = tmp_path / "TrackersConditionsDatasetDto_Protobuf.gz" + + dataset = csv_to_protobuf(csv_path, output_path) + loaded = TrackersConditionsDataset.from_protobuf_file(output_path) + + assert output_path.exists() + assert loaded == dataset + + def test_single_file_returns_one_path(self, tmp_path) -> None: + csv_path = _write_csv( + tmp_path, + "\n".join( + [ + "Year,Month,Day,Hour,Minute,Tracker_A", + "2025,4,1,8,0,-1", + "2025,4,1,8,5,2", + ] + ), + ) + dataset = from_csv(csv_path) + output_path = tmp_path / "TrackersConditionsDatasetDto_Protobuf.gz" + + written = dataset.to_protobuf_file(output_path) + + assert written == [output_path] + assert output_path.exists() + + def test_splits_into_multiple_files_when_limit_exceeded(self, tmp_path) -> None: + # 5 rows split into files of 2 → 3 files + rows = [f"2025,4,1,{8 + i // 12},{(i * 5) % 60},{-i}" for i in range(5)] + csv_path = _write_csv( + tmp_path, + "\n".join(["Year,Month,Day,Hour,Minute,Tracker_A", *rows]), + ) + dataset = from_csv(csv_path) + output_path = tmp_path / "TrackersConditionsDatasetDto_Protobuf.gz" + + written = dataset.to_protobuf_file(output_path, max_timesteps_per_file=2) + + assert len(written) == 3 + assert written[0].name == "TrackersConditionsDatasetDto_Protobuf001of003.gz" + assert written[1].name == "TrackersConditionsDatasetDto_Protobuf002of003.gz" + assert written[2].name == "TrackersConditionsDatasetDto_Protobuf003of003.gz" + assert all(p.exists() for p in written) + + # round-trip: merge all files and compare to the original dataset + merged = TrackersConditionsDataset.from_protobuf_files(written) + assert merged.data == dataset.data + assert merged.tracker_rotation_ids == dataset.tracker_rotation_ids + + def test_splits_evenly_when_divisible(self, tmp_path) -> None: + rows = [f"2025,4,1,{8 + i // 12},{(i * 5) % 60},{-i}" for i in range(4)] + csv_path = _write_csv( + tmp_path, + "\n".join(["Year,Month,Day,Hour,Minute,Tracker_A", *rows]), + ) + dataset = from_csv(csv_path) + output_path = tmp_path / "TrackersConditionsDatasetDto_Protobuf.gz" + + written = dataset.to_protobuf_file(output_path, max_timesteps_per_file=2) + + assert len(written) == 2 + assert written[0].name == "TrackersConditionsDatasetDto_Protobuf001of002.gz" + assert written[1].name == "TrackersConditionsDatasetDto_Protobuf002of002.gz" + + def test_rejects_nonpositive_max_timesteps(self, tmp_path) -> None: + csv_path = _write_csv( + tmp_path, + "\n".join( + ["Year,Month,Day,Hour,Minute,Tracker_A", "2025,4,1,8,0,-1", "2025,4,1,8,5,2"] + ), + ) + dataset = from_csv(csv_path) + + with pytest.raises(ValueError, match="greater than 0"): + dataset.to_protobuf_file(tmp_path / "out.gz", max_timesteps_per_file=0) + + +class TestWeatherChecks: + def test_coverage_passes_when_weather_spans_rotation(self) -> None: + ts_rotation = [datetime(2025, 1, 1, 8, 0), datetime(2025, 1, 1, 8, 5)] + ts_weather = [datetime(2025, 1, 1, 0, 0), datetime(2025, 1, 1, 23, 55)] + dataset = _make_dataset_with_timestamps(ts_rotation) + + check_weather_covers_rotation_period(ts_weather, dataset) + + def test_coverage_fails_when_weather_starts_late(self) -> None: + ts_rotation = [datetime(2025, 1, 1, 8, 0), datetime(2025, 1, 1, 8, 5)] + ts_weather = [datetime(2025, 1, 1, 8, 5), datetime(2025, 1, 1, 23, 55)] + dataset = _make_dataset_with_timestamps(ts_rotation) + + with pytest.raises(ValueError, match="starts at"): + check_weather_covers_rotation_period(ts_weather, dataset) + + def test_coverage_fails_when_weather_ends_early(self) -> None: + ts_rotation = [datetime(2025, 1, 1, 8, 0), datetime(2025, 1, 1, 8, 5)] + ts_weather = [datetime(2025, 1, 1, 0, 0), datetime(2025, 1, 1, 8, 0)] + dataset = _make_dataset_with_timestamps(ts_rotation) + + with pytest.raises(ValueError, match="ends at"): + check_weather_covers_rotation_period(ts_weather, dataset) + + def test_coverage_passes_for_empty_dataset(self) -> None: + check_weather_covers_rotation_period([datetime(2025, 1, 1)], TrackersConditionsDataset()) + + def test_coverage_raises_for_empty_weather_timestamps(self) -> None: + dataset = _make_dataset_with_timestamps( + [datetime(2025, 1, 1, 8, 0), datetime(2025, 1, 1, 8, 5)] + ) + + with pytest.raises(ValueError, match="empty"): + check_weather_covers_rotation_period([], dataset) + + def test_resolution_passes_for_same_period(self) -> None: + ts = [datetime(2025, 1, 1, 8, 0), datetime(2025, 1, 1, 8, 5)] + dataset = _make_dataset_with_timestamps(ts, period_minutes=5.0) + + check_compatible_time_resolutions(ts, dataset) + + def test_resolution_passes_for_coarser_weather(self) -> None: + # 60-minute weather, 5-minute rotation + ts_rotation = [datetime(2025, 1, 1, 8, 0), datetime(2025, 1, 1, 8, 5)] + ts_weather = [datetime(2025, 1, 1, 8, 0), datetime(2025, 1, 1, 9, 0)] + dataset = _make_dataset_with_timestamps(ts_rotation, period_minutes=5.0) + + check_compatible_time_resolutions(ts_weather, dataset) + + def test_resolution_passes_for_finer_weather(self) -> None: + # 5-minute weather, 60-minute rotation + ts_rotation = [datetime(2025, 1, 1, 8, 0), datetime(2025, 1, 1, 9, 0)] + ts_weather = [datetime(2025, 1, 1, 8, 0), datetime(2025, 1, 1, 8, 5)] + dataset = _make_dataset_with_timestamps(ts_rotation, period_minutes=60.0) + + check_compatible_time_resolutions(ts_weather, dataset) + + def test_resolution_fails_for_incompatible_periods(self) -> None: + # 7-minute weather, 5-minute rotation → not a whole multiple + ts_rotation = [datetime(2025, 1, 1, 8, 0), datetime(2025, 1, 1, 8, 5)] + ts_weather = [datetime(2025, 1, 1, 8, 0), datetime(2025, 1, 1, 8, 7)] + dataset = _make_dataset_with_timestamps(ts_rotation, period_minutes=5.0) + + with pytest.raises(ValueError, match="Incompatible"): + check_compatible_time_resolutions(ts_weather, dataset) + + def test_resolution_passes_for_empty_dataset(self) -> None: + ts_weather = [datetime(2025, 1, 1), datetime(2025, 1, 1, 0, 5)] + + check_compatible_time_resolutions(ts_weather, TrackersConditionsDataset()) + + def test_resolution_raises_for_insufficient_weather_timestamps(self) -> None: + dataset = _make_dataset_with_timestamps( + [datetime(2025, 1, 1, 8, 0), datetime(2025, 1, 1, 8, 5)] + ) + + with pytest.raises(ValueError, match="at least two"): + check_compatible_time_resolutions([datetime(2025, 1, 1)], dataset) + + +class TestFromCsvFolder: + """from_csv_folder merges multiple CSV files from a directory.""" + + _JAN = "\n".join( + [ + "Year,Month,Day,Hour,Minute,T0,T1", + "2025,1,1,8,0,-1.0,-2.0", + "2025,1,1,8,5,-1.5,-2.5", + "2025,1,1,8,10,-2.0,-3.0", + ] + ) + _FEB = "\n".join( + [ + "Year,Month,Day,Hour,Minute,T0,T1", + "2025,2,1,8,0,-3.0,-4.0", + "2025,2,1,8,5,-3.5,-4.5", + "2025,2,1,8,10,-4.0,-5.0", + ] + ) + + def test_merges_two_files(self, tmp_path) -> None: + (tmp_path / "jan.csv").write_text(self._JAN, encoding="utf-8") + (tmp_path / "feb.csv").write_text(self._FEB, encoding="utf-8") + + dataset = from_csv_folder(tmp_path) + + assert len(dataset.data) == 6 + assert dataset.tracker_rotation_ids == ["T0", "T1"] + + def test_files_sorted_chronologically_not_by_name(self, tmp_path) -> None: + # Name the February file so it sorts alphabetically before January. + (tmp_path / "aaa_feb.csv").write_text(self._FEB, encoding="utf-8") + (tmp_path / "zzz_jan.csv").write_text(self._JAN, encoding="utf-8") + + dataset = from_csv_folder(tmp_path) + + # January data must come first regardless of filename order. + first_ts = dataset.data[0].start_of_period + assert first_ts == datetime(2025, 1, 1, 8, 0, tzinfo=timezone.utc) + assert len(dataset.data) == 6 + + def test_single_file_folder(self, tmp_path) -> None: + (tmp_path / "only.csv").write_text(self._JAN, encoding="utf-8") + + dataset = from_csv_folder(tmp_path) + + assert len(dataset.data) == 3 + + def test_empty_folder_raises(self, tmp_path) -> None: + with pytest.raises(ValueError, match="No CSV files found"): + from_csv_folder(tmp_path) + + def test_mismatched_tracker_ids_raises(self, tmp_path) -> None: + different_ids = "\n".join( + [ + "Year,Month,Day,Hour,Minute,T0,T9", + "2025,2,1,8,0,-3.0,-4.0", + "2025,2,1,8,5,-3.5,-4.5", + ] + ) + (tmp_path / "jan.csv").write_text(self._JAN, encoding="utf-8") + (tmp_path / "feb.csv").write_text(different_ids, encoding="utf-8") + + with pytest.raises(ValueError, match="different tracker-ID columns"): + from_csv_folder(tmp_path) + + def test_mismatched_column_order_raises(self, tmp_path) -> None: + reversed_order = "\n".join( + [ + "Year,Month,Day,Hour,Minute,T1,T0", + "2025,2,1,8,0,-3.0,-4.0", + "2025,2,1,8,5,-3.5,-4.5", + ] + ) + (tmp_path / "jan.csv").write_text(self._JAN, encoding="utf-8") + (tmp_path / "feb.csv").write_text(reversed_order, encoding="utf-8") + + with pytest.raises(ValueError, match="different tracker-ID columns"): + from_csv_folder(tmp_path) + + def test_overlapping_timestamps_raises(self, tmp_path) -> None: + overlap = "\n".join( + [ + "Year,Month,Day,Hour,Minute,T0,T1", + # Starts at the same timestamp as the last row of _JAN. + "2025,1,1,8,10,-3.0,-4.0", + "2025,1,1,8,15,-3.5,-4.5", + ] + ) + (tmp_path / "jan.csv").write_text(self._JAN, encoding="utf-8") + (tmp_path / "overlap.csv").write_text(overlap, encoding="utf-8") + + with pytest.raises(ValueError, match="overlap"): + from_csv_folder(tmp_path) + + def test_kwargs_forwarded(self, tmp_path) -> None: + (tmp_path / "jan.csv").write_text(self._JAN, encoding="utf-8") + (tmp_path / "feb.csv").write_text(self._FEB, encoding="utf-8") + + dataset = from_csv_folder(tmp_path, flip_sign=True) + + # All centidegree values should be positive (signs flipped). + for condition in dataset.data: + for value in condition.tracker_rotations_array_values: + assert value > 0