Skip to content

feat: ingest OSM ways as places alongside nodes - #119

Merged
koinsaari merged 1 commit into
mainfrom
feat/osm-way-ingestion
Aug 1, 2026
Merged

feat: ingest OSM ways as places alongside nodes#119
koinsaari merged 1 commit into
mainfrom
feat/osm-way-ingestion

Conversation

@koinsaari

Copy link
Copy Markdown
Contributor

Summary

  • OSM ways (buildings/areas) that qualify under the existing POI allowlist are now ingested as Place rows, same as nodes — fixes venues like shopping malls being silently dropped (internal/sources/osm/pbf.go only handled Node elements before).
  • StreamNodes replaced with StreamElements (node + way scanning; relations still skipped). A way's Lat/Lng is the centroid of its resolved orb.LineString — no polygon is persisted.
  • TransformNode split into TransformOSMNode/TransformOSMWay, both delegating to an unexported transform() — removes a stringly-typed/positional osmType argument from the public API.
  • Requires the input .osm.pbf to be preprocessed with osmium add-locations-to-ways so way node coordinates resolve; unannotated ways are skipped with a slog.Warn, run doesn't fail.
  • No parent/child linking (e.g. mall → shops inside it) — every way-derived place is flat and independent, same as nodes.
  • Test fixture testdata/andorra-sample.osm.pbf re-generated with osmium add-locations-to-ways so way tests have real annotated geometry; node data unaffected.

Test plan

  • go test ./... — all unit tests pass
  • go test -tags integration ./... — all integration tests pass (real Postgres/PostGIS via testcontainers), including a new assertion that a way-derived place (Olympia Centre, shop=mall, way 54858980) upserts with osm_type='way' and parent_id IS NULL
  • go vet ./... and golangci-lint run ./... clean
  • Manually verified way categories aren't shop-specific — 394 way-derived places across 14 categories in the Andorra fixture (restaurants, shops, worship, education, government, transport, healthcare, etc.)

🤖 Generated with Claude Code

Buildings/venues mapped as OSM ways (e.g. shopping malls) were silently
dropped since StreamNodes only handled node elements. Adds Way/WaySink,
StreamElements (node+way scanning), and centroid-based Lat/Lng for ways,
reusing the existing POI allowlist and transform pipeline. Ways require
osmium add-locations-to-ways preprocessing to resolve node coordinates;
unannotated ways are skipped with a warning.

TransformNode is split into TransformOSMNode/TransformOSMWay wrappers
over an unexported transform() to remove the stringly-typed/positional
osmType argument from the public API.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@claude

claude Bot commented Aug 1, 2026

Copy link
Copy Markdown

Claude finished @koinsaari's task in 2m 5s —— View job


Review complete

  • Read REVIEW.md and CLAUDE.md for review instructions/context
  • Review diff against origin/main
  • Check core invariants (pure data layer, deterministic AuditFlags, validation ordering, no HTTP framework)
  • Check SQL/GORM parameterization, JSONB Valuer/Scanner symmetry, test coverage for new code
  • Post final review and set commit status

No blocking issues. This is a clean, well-scoped extension of the OSM canonical pipeline.

What I checked:

  • StreamElements (internal/sources/osm/pbf.go) correctly replaces StreamNodes, dispatching nodes/ways via separate sinks and skipping relations. The empty-LineString guard (len(ls) == 0) runs before centroid() is called, so there's no divide-by-zero when a way's nodes aren't annotated — it warns and skips instead, matching the documented osmium add-locations-to-ways precondition.
  • TransformOSMNode/TransformOSMWaytransform() refactor is a clean de-duplication; the natural upsert key (osm_id, osm_type) (internal/place/repository.go) already discriminates by OSMType, so way-derived places (osm_type='way') can't collide with node-derived places at the same coordinates — confirmed by the new integration test asserting parent_id IS NULL on the way-derived Olympia Centre row.
  • No SQL string concatenation introduced; the new integration assertion uses GORM's parameterized Where("osm_id = ? AND osm_type = ?", ...).
  • No path in this diff touches internal/a11y, validation ordering, or HTTP handlers — the invariants in REVIEW.md around AuditFlags, validation-before-a11y, and the "pure data layer" boundary aren't implicated by this change.
  • New exported surface (StreamElements, Way, WaySink, TransformOSMWay, centroid) is covered by new unit tests (pbf_test.go, transformer_test.go) and a new integration test asserting the full ingest → upsert path for a way.
  • go.mod/go.sum changes are just promoting paulmach/orb from indirect to direct (now imported directly in pbf.go) plus routine transitive bumps — no unexpected additions.

🟡 Nit (non-blocking): In internal/sources/osm/source.go, the processed%10000==0 progress log only fires in the node sink, not the way sink (source.go:36-85). Way-heavy extracts will under-report progress during long runs. Not worth blocking on.

Fix this →

Notes on tooling limitations:

  • I wasn't able to run go build/go vet/go test in this sandbox (Bash commands required approval that wasn't available in this run) — the PR description states these were run locally and pass; I verified correctness by reading the diff directly instead.
  • I was unable to set the claude-code-review commit status on 0779a9c — the API call returned 403 Resource not accessible by integration. This may need a workflow permissions update (statuses: write) for the GitHub App token.

@koinsaari
koinsaari merged commit 99e2a03 into main Aug 1, 2026
9 checks passed
@koinsaari
koinsaari deleted the feat/osm-way-ingestion branch August 1, 2026 14:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant