diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..2ee352b --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,61 @@ +# AGENTS.md + +## Repository Overview + +`regi-headless` now uses a **Python-first execution model**. + +- `regi-headless/src/main/python/regi_python` is the runtime bridge. It starts the JVM with **JPype** and exposes `regi_session()` and `run_headless(calculation_callback)`. +- `district-scripts/` and the example scripts under `regi-headless/src/test/resources/...` are now Python entrypoints that call into Java through the bridge. + +Treat this repo as a Python orchestration layer over REGI Java libraries. + +## Current Architecture + +- `regi-headless/src/main/python/regi_python/regi_python.py` is the current Python entrypoint. + - `regi_session()` owns JVM startup/shutdown. + - `run_headless()` creates the REGI domain and invokes a Python callback with a Java-backed registry. +- `regi-headless/src/main/java/...` contains the Java support layer that the Python bridge calls through JPype. +- `district-scripts/` contains active district scripts written calling Java-backed calculations inside the callback. +- `regi-headless/src/test/resources/usace/rowcps/headless/examples` contains the same pattern for example scripts used by tests. + +## File Groups To Treat Differently + +- `regi-headless/src/main/python/` + - python to java bridge code + - prefer modern Python style and small, testable helpers +- `district-scripts/` + - active district-owned operational scripts + - preserve behavior unless the task explicitly changes calculation results or API usage +- `regi-headless/src/test/resources/usace/rowcps/headless/examples/` + - example scripts used by tests + - keep them aligned with the district-script pattern +- `docs/` + - useful for historical context and documentation of features moving forward + - `docs/agent-guides/` is the shared source of truth for agent-facing workflow guidance + +## Working Rules + +- Keep edits scoped to the requested migration target. +- When a script needs a new Java method, update the Java public API and the test harness together; the script test suite validates against the Java source-defined scriptable API. +- Avoid broad refactors outside the requested script family. + +## Verification + +Common checks in this repo: + +- `./gradlew buildPythonWheel` +- `./gradlew testPythonWheel` + +The Python package is built from `regi-headless/src/main/python`, and the wheel test verifies the package metadata, importability, public API exposure, and bundled Java jars. + +## Environment Notes + +The Python bridge expects the Java environment to be available, and the repo documentation currently references these variables: + +- `JAVA_HOME` +- `CDA_URL` +- `CDA_API_KEY` +- `OFFICE_ID` +- `REGI_LOG_LEVEL` + +Use the repo's current bridge code and tests as the source of truth for behavior; use the docs for orientation, not as a strict contract. `regi-headless/src/test/python/test_district_scripts.py` is the main contract for migrated script shape and allowed API calls. diff --git a/README.md b/README.md index cd6752e..f35a3ca 100644 --- a/README.md +++ b/README.md @@ -1,75 +1,105 @@ # REGI-Headless -> [!IMPORTANT] -> **Notice: Project Refactor in Progress** -> This project is undergoing a large refactor for [CWMS Data API](https://github.com/USACE/cwms-data-api) support. -> It will transition from a Java project consuming Jython scripts to a **Python project** that -> utilizes **JPype** to call underlying REGI Java libraries. +`regi-python` is the Python bridge over the REGI Java libraries. -`REGI-Headless` is a Java-based command-line tool and library designed to run -**REGI** calculations in a headless environment. -It allows users to execute complex hydrological calculations and manage gate settings via Jython -scripts without the need for a graphical interface. +The client-facing Python API is documented in [docs/PYTHON_API.md](docs/PYTHON_API.md). +For script migration guidance, see [docs/JYTHON_TO_JPYPE_MIGRATION.md](docs/JYTHON_TO_JPYPE_MIGRATION.md). -## Features +## What Lives Where -- **Headless Execution**: Run REGI calculations as part of automated workflows or on servers. -- **Database Integration**: Connects to CWMS data retrieval and storage. -- **Modular Calculations**: Includes support for: - - Inflow calculations (Clone, Compute, Auto-Adjust, Balance All, etc.) - - Flow Group and gate settings calculations. +- `regi-headless/src/main/python/regi_python/` + - Python bridge code + - Public entry points: `regi_session()` and `run_headless(calculation_callback)` +- `regi-headless/src/main/java/` + - Java support layer used by the bridge calling into REGI calculation and data access libraries +- `district-scripts/` + - Copy of district-owned Python scripts used as examples for smoke tests -## Project Structure +## Requirements -- `regi-headless/`: Core Java implementation, including `RegiCLI`. -- `district-scripts/`: Example scripts and district-specific configurations. -- `docs/`: Additional documentation. +- Java JDK 21 or higher +- Python 3.11 or higher -## Getting Started +### Environment variables +- `JAVA_HOME` set for JPype startup +- `CDA_URL` url for CDA instance to connect to +- `CDA_API_KEY` required for accessing and storing data in CDA +- `OFFICE_ID` session scoped office for data access +- `REGI_LOG_LEVEL` for logging verbosity +- `REGI_LOG_FORMAT` overrides the default Python log format -### Prerequisites +## Building The Wheel -- Java JDK 21 or higher. +Build the Python wheel with Gradle: -### CWMS Data API Configuration +```powershell +./gradlew buildPythonWheel +``` + +The wheel is written to `regi-headless/build/install/regi_python/dist/`. -The library reads the following environment variables when establishing the CWMS Data API data source: +Release tags become wheel versions and must be PEP 440 compatible. For example, use `0.0.2a0`, `0.0.2b0`, or `0.0.2rc0` instead of `0.0.2-alpha`, `0.0.2-beta`, or `0.0.2-rc`. -- `CDA_URL`: Base URL for the CWMS Data API endpoint. -- `API_KEY`: API key used for CWMS Data API authentication and authorization. -- `OFFICE_ID`: CWMS office identifier used to scope the session. +Install the built wheel into a Python environment: + +```powershell +pip install regi_python-*.whl +``` -The factory uses these values to authenticate, resolve the current user, and persist the connected office and time zone into the REGI project. +### Notes -### Building +- The Python package name is `regi_python`. +- The wheel metadata name is `regi-python`. +- The bundled Java jars are packaged inside `regi_python/lib/`. +## Releases + +Releases are published from the GitHub repository at [USACE-WaterManagement/regi-python](https://github.com/USACE-WaterManagement/regi-python). A release build attaches the Python wheel and checksum file to the GitHub Release for the matching tag. + +To consume a published wheel, download the wheel asset from the release and install it with your package manager. Use the exact wheel filename from the release asset URL. ```powershell -./gradlew build +pip install https://github.com/USACE-WaterManagement/regi-python/releases/download//regi_python--py3-none-any.whl ``` -Details TBD. +```powershell +uv pip install https://github.com/USACE-WaterManagement/regi-python/releases/download//regi_python--py3-none-any.whl +``` -## Usage +```powershell +poetry add https://github.com/USACE-WaterManagement/regi-python/releases/download//regi_python--py3-none-any.whl +``` + +```powershell +pdm add https://github.com/USACE-WaterManagement/regi-python/releases/download//regi_python--py3-none-any.whl +``` + +If your tool does not support direct wheel URLs, download the asset from the release page and install it from the local `.whl` file instead. -TBD +## Using The Bridge -### Command Line Options +See [docs/PYTHON_API.md](docs/PYTHON_API.md) for the full Python-facing contract. -TBD +## Verification -### Example +Run the wheel smoke tests: -TBD +```powershell +./gradlew testPythonWheel +``` + +Run the script/API compatibility smoke test: -## Testing +```powershell +./gradlew smokeTestDistrictScripts +``` -TBD +`./gradlew check` runs both along with Java unit tests. ## Maintainers -See [MAINTAINERS.md](MAINTAINERS.md) for a list of project maintainers. +See [MAINTAINERS.md](MAINTAINERS.md). ## License -See [LICENSE](LICENSE) for licensing information. +See [LICENSE](LICENSE). diff --git a/docs/JYTHON_TO_JPYPE_MIGRATION.md b/docs/JYTHON_TO_JPYPE_MIGRATION.md new file mode 100644 index 0000000..a1d10ce --- /dev/null +++ b/docs/JYTHON_TO_JPYPE_MIGRATION.md @@ -0,0 +1,122 @@ +# Migrating REGI Headless District Jython Scripts to `regi_python` + +This guide captures the script migration pattern used in the district-scripts history, +moving from old Python 2 Jython-style top-level scripts to the current JPype-based `regi_python` bridge. + +## Core Migration Pattern + +Old scripts typically: + +- imported Java classes at module import time +- accessed global `registry` and calculation objects at top level +- executed the calculation immediately when the file was loaded +- relied on shell wrappers such as `RunHeadlessJython.bat` or `.sh` + +New scripts should: + +- import `regi_session` and `run_headless` from `regi_python` +- move Java imports inside a callback that runs after the JVM starts +- wrap the calculation logic in a function such as `run_calculations(registry)` +- keep `registry.getCalculation(...)` and Java method calls unchanged +- end with `if __name__ == "__main__": with regi_session(): run_headless(run_calculations)` + +## Before And After + +### Before + +```python +from java.util import Calendar +from java.util import TimeZone + +gateCalc = registry.getCalculation(1.0, "Gate Flow") +timeZone = TimeZone.getTimeZone("US/Central") +startCal = Calendar.getInstance(timeZone) +endCal = Calendar.getInstance(timeZone) +gateCalc.computeFlowGroup("SWT", "FOSS", startCal.getTime(), endCal.getTime(), "Flow.FOSS.Project_Total") +``` + +### After + +```python +from regi_python import regi_session, run_headless + + +def run_calculations(registry): + from java.util import Calendar + from java.util import TimeZone + + gateCalc = registry.getCalculation(1.0, "Gate Flow") + timeZone = TimeZone.getTimeZone("US/Central") + startCal = Calendar.getInstance(timeZone) + endCal = Calendar.getInstance(timeZone) + gateCalc.computeFlowGroup("SWT", "FOSS", startCal.getTime(), endCal.getTime(), "Flow.FOSS.Project_Total") + + +if __name__ == "__main__": + with regi_session(): + run_headless(run_calculations) +``` + +## Translation Rules + +### 1. Move Java imports inside the callback + +JPype-backed imports should happen after the JVM starts. +Put them inside `run_calculations(registry)` or a nested helper only called from that callback. + +This is the biggest behavioral difference from Jython scripts. + +### 2. Keep the registry lookup pattern + +The migration does not change the scriptable REGI API. Calls like these stay the same: + +- `registry.getNames(1.0)` +- `registry.getCalculation(1.0, "Inflow")` +- `registry.getCalculation(1.0, "Gate Flow")` + +The change is only the Python wrapper around those calls. + +### 3. Convert top-level execution into a callback + +Scripts must be converted from immediate execution to a `run_calculations()` function. +That keeps import side effects out of the module. + +### 4. Replace Python 2 `print` statements + +Old scripts often contain lines like: + +```python +print "Error Computing Flow Group" +``` + +Update these to Python 3 syntax: + +```python +print("Error Computing Flow Group") +``` + +### 5. Keep optional Java-side logging calls + +Calls such as `LoggingOptions.setDbMessageLevel(2)` and `LoggingOptions.setMetricsEnabled(True)` still belong in the script if the district workflow depends on them. They do not move to `regi_python`; they simply live inside the callback now. + +### 6. Remove the shell wrapper + +The old `RunHeadlessJython` launchers are no longer needed. The Python file itself becomes the entry point: + +```python +if __name__ == "__main__": + with regi_session(): + run_headless(run_calculations) +``` + +## Project-Specific Examples + +The current district scripts show the same migration shape across several script families: + +- `SWF/InflowCalcComputedInflow.py` +- `SWF/InflowCalcComputeEvapAsFlow.py` +- `SWF/GateSettings.py` +- `SWT/GateFlowGroup1.py` +- `SWL/Big3-GateFlow.py` + +The differences between them are the calculation names, location lists, and optional logging or flow-group loops. The migration pattern itself is the same. diff --git a/docs/PYTHON_API.md b/docs/PYTHON_API.md new file mode 100644 index 0000000..70d27c0 --- /dev/null +++ b/docs/PYTHON_API.md @@ -0,0 +1,116 @@ +# REGI Python API + +`regi_python` is the client-facing Python package for the REGI headless bridge. It is intentionally small: import the package, open a JVM session, and run your callback against a REGI registry. + +## Package Surface + +```python +from regi_python import regi_session, run_headless, __version__ +``` + +- `regi_session()`: context manager that starts and stops the JVM for the bridge +- `run_headless(calculation_callback)`: executes a callback against a Java-backed registry +- `__version__`: installed package version, or `"unknown"` when the package is not installed from metadata + +The package name on import is `regi_python`. The wheel metadata name is `regi-python`. + +## Runtime Requirements + +- Python 3.11 or newer +- Java JDK 21 or newer +- `JAVA_HOME` set before the bridge starts + +The wheel bundles the REGI jars inside `regi_python/lib/`, and the bridge loads those jars at JVM startup. + +## `regi_session()` + +Use `regi_session()` as the outer lifecycle boundary for any bridge work. + +Behavior: + +- starts the JVM lazily if it is not already running +- configures Java logging to flow into Python logging +- shuts the JVM down when the context exits + +Example: + +```python +from regi_python import regi_session, run_headless + +with regi_session(): + run_headless(my_callback) +``` + +Treat this context manager as the owner of the JVM lifecycle for the process. + +## `run_headless(calculation_callback)` + +`run_headless()` creates a headless REGI domain and calls your callback with a `RegiCalcRegistry` instance. + +Callback shape: + +```python +def my_callback(registry): + ... +``` + +Behavior: + +- creates the REGI domain through `HeadlessRegiDomainFactory` +- builds a `RegiCalcRegistry` with the current manager id +- calls `calculation_callback(registry)` +- commits the domain only after the callback succeeds +- always shuts down the executor and closes the domain in `finally` +- logs the failure and re-raises any exception from the callback + +The bridge is orchestration code. The callback is where client logic should live. + +## Example + +```python +from regi_python import regi_session, run_headless +from java.util import Calendar, TimeZone + + +def calculate_gate_flow(registry): + gate_calc = registry.getCalculation(1.0, "Gate Flow") + tz = TimeZone.getTimeZone("US/Central") + start = Calendar.getInstance(tz) + end = Calendar.getInstance(tz) + start.set(2025, 0, 1) + end.set(2025, 0, 2) + gate_calc.computeAll("OFFICE", "PROJECT", start.getTimeInMillis(), end.getTimeInMillis()) + + +with regi_session(): + run_headless(calculate_gate_flow) +``` + +## Logging + +The bridge uses Python logging for client-visible output. + +Environment variables: + +- `REGI_LOG_LEVEL`: accepted Python log level name such as `DEBUG`, `INFO`, `WARNING`, or `ERROR` +- `REGI_LOG_FORMAT`: overrides the default Python log format +- `AWS_BATCH_JOB_ID` and `AWS_BATCH_JOB_ATTEMPT`: injected into the default log format when present + +Java JUL records are forwarded into the same Python logger once the JVM starts. + +## Compatibility + +| Component | Supported | +| --- | --- | +| Python | 3.11+ | +| Java | JDK 21+ | +| Packaging | `regi-python` wheel with bundled jars | +| Import name | `regi_python` | + +## Troubleshooting + +- If JVM startup fails immediately, verify `JAVA_HOME` points at a JDK installation and not just a JRE. +- If imports fail after installation, confirm the wheel includes `regi_python/lib/*.jar` and that the package was installed from the built wheel. +- If logs do not appear, lower `REGI_LOG_LEVEL` or override `REGI_LOG_FORMAT`. +- If you are calling the package from a larger application, make sure some other code is not starting and stopping the JVM out from under `regi_session()`. + diff --git a/docs/agent-guides/README.md b/docs/agent-guides/README.md new file mode 100644 index 0000000..a1f6dfd --- /dev/null +++ b/docs/agent-guides/README.md @@ -0,0 +1,5 @@ +# Agent Guides + +These files are the shared, agent-agnostic source of truth for repetitive repo workflows. + +- [regi-python-bridge-maintenance.md](regi-python-bridge-maintenance.md) diff --git a/docs/agent-guides/regi-python-bridge-maintenance.md b/docs/agent-guides/regi-python-bridge-maintenance.md new file mode 100644 index 0000000..c62a410 --- /dev/null +++ b/docs/agent-guides/regi-python-bridge-maintenance.md @@ -0,0 +1,21 @@ +# REGI Python Bridge Maintenance + +Use this guide when editing the Python bridge or troubleshooting runtime behavior. + +## Focus files + +- `regi-headless/src/main/python/regi_python/regi_python.py` +- `regi-headless/src/main/python/regi_python/regi_python_logging.py` + +## Rules + +- Start the JVM only inside `regi_session()`. +- Keep runtime-only JPype imports inside functions that run after JVM startup. +- Keep `run_headless(calculation_callback)` responsible for creating the domain, running the callback, committing, and shutting down cleanly. +- Keep Python logging and Java logging forwarding aligned. +- Keep wheel packaging and bundled jar checks working with the existing tests. + +## Common checks + +- `./gradlew buildPythonWheel` +- `./gradlew testPythonWheel`