Skip to content

fix: load canonical V1 COPY fixtures in integration tests #92

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
kcrmin merged 4 commits into from
Aug 14, 2026
Merged

fix: load canonical V1 COPY fixtures in integration tests #92

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
650d4c7
Reset migration contributions for the launch baseline
kcrmin Aug 13, 2026
a103411
Support long market data paths on Windows
kcrmin Aug 13, 2026
fbbf048
fix: load V1 COPY fixtures through psycopg
kcrmin Aug 14, 2026
6a8cb41
style: normalize market data imports
kcrmin Aug 14, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
102 changes: 4 additions & 98 deletions db/migration-contributions/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,103 +1,9 @@
# Backtest migration contributions

This directory is the machine-readable handoff boundary from `backtest-engine` to the
central Flyway bundle assembled by `backend/db-migration` (owner A). It is the
COM07 mechanism only — **this repository never executes Flyway and never applies DDL
at runtime**.
This directory is the machine-readable handoff boundary from `backtest-engine` to the central Flyway bundle in `backend/db-migration`.

## contribution.properties
The launch schema was rebased on 2026-08-13. All schema and seed changes through that point are contained in the immutable central `V1__initial_schema.sql`, so `migrations/` intentionally has no timestamped migration yet. The vendored central fixture contains that same V1 for standalone tests.

| key | value | meaning |
|---|---|---|
| `contract.version` | `1` | contribution format version |
| `owner` | `backtest` | must equal the owner token in every contributed filename; one of the tokens in the central `MigrationOwner` enum |
| `schemas` | `backtest` | schemas this repository may mutate; central `DatabaseAccessPolicy` remains authoritative at table level |
| `migrations.directory` | `migrations` | the only directory that may contribute production Flyway SQL |
| `fixtures.directory` | `fixtures` | test-only material the central bundle must never scan |
| `filename.regex` | `^V[0-9]{14}__backtest_[a-z0-9]+(?:_[a-z0-9]+)*[.]sql$` | complete filename contract |
| `runtime.flyway.enabled` | `false` | migration execution belongs to the central one-shot deployment step |
Flyway remains the forward migration mechanism. After development starts, add reviewed changes as `V<UTC timestamp>__backtest_<slug>.sql`. This repository contributes SQL but never runs Flyway at application startup; `runtime.flyway.enabled` must remain `false`.

`V001`-style legacy numbering is rejected centrally and by
`tests/persistence/test_migration_contribution.py`. New files use
`V<YYYYMMDDHHMMSS>__backtest_<slug>.sql` with a UTC, globally unique timestamp.

## Current contents

All nine original `backtest.*` tables already exist in the applied central baseline
`V1__initial_schema.sql` (`CREATE TABLE "backtest"."runs"` and following). Applied
migrations are immutable, so this repository never restates them; it contributes only
later changes, and both files below leave those nine tables exactly as the baseline
defines them.

`migrations/` contains exactly one bundle-eligible file:

| file | adds | why |
|---|---|---|
| `V20260805170000__backtest_run_outcome_detail.sql` | three nullable columns on `backtest.runs` — `result_manifest_id`, `retryable`, `missing_requirements` | Forward, globally ordered adoption of the approved outcome projection. It deliberately contains none of the retired legacy input-pin fields. |

The active contribution alters a baseline table by adding columns only. The normalized
`run_input_pins` table comes from the root provider-owned `V20260805130000` migration,
mirrored here only as `fixtures/pending-root/*.sql.fixture` for integration tests.

The historical consumer-owned `V20260802094500__backtest_run_input_pins.sql` is
preserved byte-for-byte as
`fixtures/superseded-proposals/V20260802094500__backtest_run_input_pins.sql.fixture`.
Keeping it outside `migrations/` prevents a fresh central bundle from creating the
provider-owned table twice.

The original `V20260802143000__backtest_run_outcome_detail.sql` proposal is preserved
byte-for-byte as
`fixtures/superseded-proposals/V20260802143000__backtest_run_outcome_detail.sql.fixture`.
Its fixture suffix keeps it out of fresh Flyway bundle assembly after newer central
versions made the original timestamp unsafe.

**Canonical model.** The root superproject's `db/schema.dbml` is authoritative and is
not part of this repository, so it cannot be edited from here. Each migration
therefore *proposes* its canonical-model change rather than making it, and both
proposals are **proposed, not approved**:

- `run_input_pins` →
[`change-requests/2026-08-02-backtest-run-input-pins.md`](change-requests/2026-08-02-backtest-run-input-pins.md)
- the three `runs` columns → `docs/dbml-change-request-run-outcome-detail.md`, stated
verbatim as DBML

Each must land in the same reviewed change as its migration.

`change-requests/` holds those proposals. It is documentation: the central assembler
never scans it, and nothing there is SQL.

`fixtures/` holds the vendored copy of the central migration bundle that the
Testcontainers integration tests apply, plus its recorded SHA-256 digests. It contains
no bundle-eligible filenames. `tests/conftest.py` applies the vendored bundle **and
then** `migrations/`, in the order the central assembler would, so the integration
suite runs against the schema the deployment will have.

## Open item: `storage` schema ownership is contradictory

`docs/backend-implementation-master-checklist.md` treats `storage` as D-owned, but
`backend/db-migration/.../DatabaseAccessPolicy.java:36` registers it as
`MigrationOwner.SHARED`. The two disagree and the disagreement is **not** resolved
here.

Consequences, deliberately chosen:

- `schemas=backtest` only. This contribution does **not** claim `storage`.
- No `storage` DDL is authored in this repository, now or as part of this rebuild.
- The backtest persistence layer treats `storage.objects` as **read-only**
(`StorageObjectReader`), even though `DatabaseAccessPolicy.allowsBacktest` would
permit `INSERT`. Writing rows for backtest result objects is deferred to BT3, which
must first get the ownership question answered by owner A.

This needs a decision from the central migration owner before any repository writes
`storage`. Until then, treat any `storage` write from this repository as a contract
violation.

## Related caveat: the access policy is not enforced at runtime

`DatabaseAccessPolicy` is a static helper used by central unit tests. The applied
baseline contains no `GRANT`/`REVOKE` statements and no per-service roles, so nothing
in the live database prevents a service from writing a schema it does not own — the
backend already inserts into `backtest.runs` directly. This repository compensates
only for itself: `backtest_engine.persistence` restricts its own statements to the
declared schemas and refuses DDL, and `tests/persistence/test_runtime_no_ddl.py`
proves it.
The root `db/schema.dbml` remains the canonical model. A new migration and its DBML change must be reviewed together.
43 changes: 1 addition & 42 deletions db/migration-contributions/fixtures/central-migration.sha256
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,42 +1 @@
79c9bb70d6ba517fad9f85dd5ad129a5d99da2253c4ecd42248bb1b03df6f80e V1__initial_schema.sql.fixture
4ea6b22533487e4c0a216c01abfb15e3912a86105b8159edf4a6c8ba718f8ae6 V20260801112341__backend_identity_email_auth.sql.fixture
30c67dd8ab775da32a89d1c7b06f0a1eeb0b7a35f96c8230ea1979becded26cf V20260801153000__backend_bot_continuation_deadlines.sql.fixture
8ac86fd04cc71390a74ef846187d1086485ea133dc590dfa995ec569eadb3822 V20260802050054__backend_account_preferences_theme.sql.fixture
a53734c7d01ed1d531b7d6790ad7204ce206e53fbc37b0e6cc5285ff22951a46 V20260802060000__backend_account_lifecycle_dormant_status.sql.fixture
6fb2bf77c9bfce6c3accc4bbf188e2d6d9acd30f48cf8c648539b5e558665a75 V20260802060100__backend_account_lifecycle_contract.sql.fixture
ae205fed746cbddbd2dc02f3674eb5d4db5affeda66addfebbb73576dedbd376 V20260802060200__backend_account_lifecycle_command_receipts.sql.fixture
7c96bddc516580aa1ca18c1d0903660de9c4ebb6559585cae87c6fcb3295b411 V20260802060300__backend_oidc_step_up_nonces.sql.fixture
dc75455794d1c810c171e3c49af68ccc617868c181f869bbf8a2705be23e8372 V20260802060400__backend_account_closure_coordination.sql.fixture
25658142315af9efc102d2bb22f6dd971f5928ff00b8f598848bb8b86925c43f V20260802194500__backend_final_leaderboard_unranked_entries.sql.fixture
cbc828c7d9244a31cd189a80d109f9448f17fa45b2d1cd665b16e440b6c15c20 V20260802213500__backend_room_final_access_grants.sql.fixture
cb482f2f87332e99b73fff4db610edd3577d8261b8260b30d8cd2d29b2610be2 V20260802220000__backend_retention_category_split.sql.fixture
c28a0d2f88e9379d985b223285e5b21f95e03d60f4eb5f371d64173ebcc1af98 V20260802220100__trading_private_bot_runtime_cleanup.sql.fixture
e2fefdb87d57c15ae760dfe9f86c8ebd63d7fa5d6f2579adf38afbf317e61fe6 V20260802220200__backend_retention_execution.sql.fixture
87ba58e2965f7fbae85e0df368cb13b57f3f63d41fd53ca2f7e86f65c944f9be V20260802220300__backtest_competition_owner_anonymization.sql.fixture
cd7f87a24464b11911db855abe005960117616afe7039afd9a450c1ad7c587f4 V20260802230000__backend_operator_room_permissions.sql.fixture
e0e604a224b609aa9a3e16f799c3d757465af80831dc947870789183dbaee4e8 V20260802231000__backend_leaderboard_result_source_guard.sql.fixture
bdd36d675fe2c0b85e89ffc6cdf74c89ae0290dbd45d6b802225cb2c969713f8 V20260802231100__backend_transactional_outbox.sql.fixture
bdbff070ad9825f8c074a5345a31b97aa1010c91cfd179fd77f6d343af95e36b V20260802231200__backend_delegated_strategy_scope.sql.fixture
0dc6b20c9455cea23709cc6dc9c2cb9f8cb7868c8ad7c15380a327db2f1ee42e V20260802231300__backend_user_case_contract.sql.fixture
bda8bc0c99097c3ce82b64aff3e7d1bd70cea6c8d615706b5b94b44a14ed3688 V20260802231400__backend_operator_rbac.sql.fixture
3994c8c0fe5337f561503df9e65f2c5c5253e14bc504ff8a2f809f73b2476ce4 V20260802231500__backend_notification_delivery.sql.fixture
8b7b1958ee67619e901f9c54a5894ced0944debbc1f1e8cb9ba1e2a68c516bd5 V20260802231600__backend_account_sanction_commands.sql.fixture
ac93d09cded9b1be72c040a1f69381435e1eb68d29bc0d147cb69307f67b6997 V20260802231700__backend_operator_case_workflow.sql.fixture
aa81c7bd0c1f1c9d41c5065c2c1106cfd5fc094d94492434aece9cc3afacbd76 V20260802231800__backend_case_response_deadline.sql.fixture
2ca75d80b0948adcf35b53ca65197503ed9b9c9cd8d61613ae575cc818b84fee V20260802231900__backend_durable_batch_execution.sql.fixture
41bf4b61bddbe33fc968220fc2c7a1b93a569dc92cf629e9ddbc31e66d8acbe8 V20260802232000__backend_operator_trust.sql.fixture
357c7ceff14ef6da5090fb728d10b88cbbebf57f3b0c450c9de49d61e598508d V20260802232100__backend_operator_bootstrap_evidence.sql.fixture
d6ac0c8d6c93053d7f7734c3a846aea7727615c6a483c6917ad06b3baa621313 V20260804090000__backend_seed_basic_element_catalog.sql.fixture
a6e3ab7e66573ae57261d615efa8a05608770104eb0d18b771e223575cb0e389 V20260804090100__pipeline_register_official_rsi_14.sql.fixture
8e818d077476f6ff83d118adbe0adf2065d1ab7ee1ec7d0e494969234c0f7ee5 V20260804100000__backend_bot_launch_contract_plans.sql.fixture
5e32112d03c36b0c6c46cd952c2c5e17e296ecb8a10013add9dbd1ebfcf4c376 V20260804145900__backend_room_ledger_handoff.sql.fixture
f79840ae0642c8319bf994a9d27254994f15485beb18836aec8b86d4cdb55ddc V20260804160000__backtest_runtime_ownership_expand.sql.fixture
2ed973f73dbd9daf9a152b7d7b1847307cf0ac397f4a40157d59577b2bd51acc V20260804160010__backend_backtest_competition_link.sql.fixture
bea14651c4c8ae595b383c91565f15be71ccca20e66d29d78f797b434290134c V20260804160020__pipeline_dataset_manifest_empty_hash.sql.fixture
ec1ce83eda58025656617bbd26e497a15f307465ffb149bdeeef3322f3b3cd47 V20260804160100__backtest_runtime_ownership_constrain.sql.fixture
0311a2767e99ead8910afa5f5b5f578e6ca1e0fd7a41f1c321888a61eac0a022 V20260805130000__backtest_run_input_pins.sql.fixture
3e63766bdaa38995bbefa6b541a73eab8037dd93ef54f3e8e481d2a8c8ed0b71 V20260806100000__backend_google_oidc_provider.sql.fixture
ac96729db9ef421940a2b7c73ac691a0a2d7f9b5a1eebba47802515914e831ec V20260807120000__backend_customer_refresh_token_families.sql.fixture
b4f894e257a809a4e7b6d8c635ec8a66751f0d6b70e2c93a6072fb1f0a22f205 V20260807130000__backend_publish_full_basic_element_catalog.sql.fixture
87392d260f13a8e2ad90e025bb3afd6fd95dc7abdc2c223afbfbdfaa046cb6c7 V20260808000000__backend_publish_live_strategy_timeframes.sql.fixture
94ae572554a7cef8b0b9f91a24a6fd14b7c2ad93feccd283da7047bcb3239f30 V20260808120000__backend_publish_production_backtest_resolutions.sql.fixture
e3bec37557b570d9a33c10a07e3e5c706ab56105dec409ad5ddb800720517282 V1__initial_schema.sql.fixture
Loading
Loading