Wind energy workshop - #13
Open
sergioferragut wants to merge 2 commits into
Open
Conversation
Teaches PostGIS and TimescaleDB together on fixed-location assets: a portfolio
of wind plants across six real regions, laid out on a geodesic grid with
ST_Project and modelling how the turbines steal each other's wind.
Contents:
* 14 numbered SQL steps, run top to bottom in psql
* reset_demo.sh, orchestration only — it runs the SQL and prints verification
* four provisioned Grafana dashboards forming a global > region > plant >
turbine drill-through with cascading filters
Notable design decisions, each measured rather than assumed:
* Turbine layout is a STAGGERED checkerboard, not a plain grid. Aligned rows
put the worst wake exactly on the prevailing wind: 25.8% mean power loss at
the design bearing, against 0.0% once staggered, with the loss reappearing
as narrow 29% spikes at the +/-17 degree diagonal alignment.
* The turbine view selects its own source — raw hypertable, hourly aggregate
or daily aggregate — from the current zoom level, with the untaken UNION ALL
branches constant-folded out of the plan.
* The backfill pre-creates chunks before fanning out. Chunk creation takes
ShareUpdateExclusiveLock on the parent and holds it to commit, so workers
that create their own chunks serialise: 1.09x on four workers against 3.67x
with the chunks already in place.
* The aggregate fill parallelises by WINDOW, aligned to each aggregate's
materialization chunk interval, with the aggregates themselves kept in
dependency order — refreshing a child before its parent leaves the child
silently empty.
* All data is generated by SQL functions an attendee can read and re-run.
Volume is parameterised (--plants, --turbines-per-plant, --days), so the
same workshop runs at 5,000 rows or 13 million.
Also adds a repo-root .env.example: one set of connection settings shared by
every workshop, which each reset_demo.sh reads as ../.env.
Three repo-root working notes come with it. These are for us, not for workshop
attendees, and they are deliberately WORKSHOP-AGNOSTIC: they describe how the
platforms behave, in neutral vocabulary, so they stay useful whatever the next
workshop is about. They exist because this material is easy to get wrong in ways
that produce no error — a stale doc claim or a mis-set panel field just renders
the wrong answer.
* CLAUDE.md — how to work in this repo. Conventions (one workshop per
top-level directory, SQL house style, the README skeleton), the standing
rule to verify platform behaviour against the docs and source rather than
from recall, and the non-negotiables: everything must work self-service on a
fresh Tiger Cloud service, iterate at the smallest data volume that
reproduces the behaviour, and data generation lives in SQL.
It also carries the rules for maintaining the two platform files, since they
are the part of this repo that accumulates: every workshop is an occasion to
extend them, an entry must generalise away from the workshop that prompted
it, confidence labels have to be earned, and a disproven entry gets corrected
in place rather than left to be trusted.
* TIGER_PLATFORM.md — durable facts about TimescaleDB, Tiger Cloud and PostGIS
as they actually behave, each entry labelled VERIFIED, DOCUMENTED or
ASSUMED, with the measurement or source that earned the label. Covers
continuous aggregates, Hypercore and the columnstore, retention and tiering,
the real extension inventory and which features are gated, idempotent
generation, and PostGIS pitfalls.
* GRAFANA_PLATFORM.md — the same, for Grafana: panel JSON and the grid,
provisioning and datasource wiring, geomap view and layer behaviour, and the
SQL that has to live inside a panel. Separate from the database notes
because a wrong field name in panel JSON raises nothing at all — the panel
simply renders wrong, so these findings need somewhere to live.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Step 08 filled its eight aggregates through a generated \gexec block guarded by a config flag, with a companion cagg_fill_plan() window planner that only reset_demo.sh ever called. That obscured the lesson it was supposed to teach, so the fill is now eight plain CALL refresh_continuous_aggregate() statements, parents before children. Removing the parallel path was a deliberate trade, not an oversight. Since TimescaleDB 2.28 the call batches incrementally on its own, which retires the MEMORY argument for hand-rolled windowing and leaves only speed — and speed has to be judged against the whole build. The measured 2.83x applied to ~16s of a ~100s build, so it saved ~10s in exchange for a window planner, a config key, a conditional in the SQL and 70 lines of shell that the default path never ran. The raw-data backfill keeps its --jobs parallelism, where the same arithmetic comes out the other way (3.67x on the dominant phase). The prose now states what the defaults actually are, because both are easy to misread: buckets_per_batch is 10 BUCKETS (not chunks, not rows), and the batches are SEQUENTIAL, each in its own transaction — that is what bounds memory, and it is not concurrency. Includes an xact_commit recipe to observe it: 1202 commits by default against 6 with buckets_per_batch: 0. Fix a real bug found while verifying the above. A refresh range's upper bound snaps DOWN to a bucket boundary, but a refresh ending inside an OPEN bucket materialises that whole bucket and pushes the watermark to its end — leaving the watermark ahead of the wall clock. Real-time aggregation only unions rows above the watermark, so rows written into that bucket afterwards are invisible, and force => true cannot reach them because its upper bound snaps down identically. Step 10's NULL,NULL refresh set the condition; step 12 then wrote at now() and refreshed to now(), losing exactly one row per turbine it had just added (721 raw against 720 aggregated). Step 12 now bounds at the end of the current bucket, and reset_demo.sh asserts SUM(readings) = COUNT(*), which nothing else here caught. Verified sensitive: injecting one row into a materialised bucket reports -1. Explain the backfill worker count instead of just reporting "serial", which reads as a missing feature. It falls to 1 on small services on purpose: the load is CPU-bound in one backend, so extra connections need spare cores, and below 2 CPU they cost time. Measured at 563,588 rows/hypertable — 0.5 CPU: 50s serial vs 68s at 2 jobs; 1 CPU: 21s vs 24s vs 30s at 4; 4 CPU: 20s vs 15s vs 13s. Also: answer the enable_merge_on_cagg_refresh question in the file, since anyone reading the docs will reach for it. It is silently ignored on any aggregate with columnstore enabled (the gate is compression_enabled on the AGGREGATE, not whether the chunk is compressed yet), and does nothing for a cold fill either. Correct two stale comments — the backfill batches on chunk boundaries, not one-month windows — and give the migration's aggregates IF NOT EXISTS, which continuous aggregates do honour, so it is re-runnable. TIGER_PLATFORM.md gains the generalised versions: the open-bucket hole and how to assert against it, the batching defaults, the merge GUC's two conditions, and the rule that connection-level parallelism for a generated load scales with cores and is negative below two. Verified at 6.73M rows per hypertable (12 plants x 8 turbines x 730 days) in 126s with no OOM, all eight aggregates non-empty, step 08 and the migration both re-runnable under ON_ERROR_STOP=1. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See commits for detailed descriptions.
This PR adds a static location PostGIS + timescaleDB example to the workshop library.
The use case is Wind Energy Production monitoring across multiple plants across the world.