Skip to content

feat: fort map-data consumer — gyms, stations & pokestops - with DNF filtering#1228

Open
jfberry wants to merge 50 commits into
WatWowMap:mainfrom
jfberry:feat/fort-consumer
Open

feat: fort map-data consumer — gyms, stations & pokestops - with DNF filtering#1228
jfberry wants to merge 50 commits into
WatWowMap:mainfrom
jfberry:feat/fort-consumer

Conversation

@jfberry

@jfberry jfberry commented Jul 16, 2026

Copy link
Copy Markdown

Deploying this (end users)

Two small config changes — one on Golbat, one on ReactMap. Once both are set, fort map-data (gyms, pokestops, stations) is served from Golbat's in-memory fort API instead of SQL, with automatic SQL fallback if Golbat is ever unreachable — so it's safe to enable on a dual (endpoint + DB) source.

1 · Golbat — run Golbat with UnownHash/Golbat#385 (it already includes the reward type-20 quest decode from #382) and turn on the in-memory fort index. fort_in_memory is a top-level option and defaults to false:

# golbat config.toml  (top level, above the [sections])
fort_in_memory = true

Make sure your Golbat API secret is set. Sanity-check the endpoint is live:

curl -H "X-Golbat-Secret: <secret>" http://<golbat-host>:<port>/api/fort/available
# 200 + { "pokestops": {...}, "gyms": {...}, "stations": {...} }

2 · ReactMap — add endpoint + secret to the scanner DB source whose useFor includes your fort types (gym/pokestop/station). This is the same Golbat endpoint most instances already point their pokémon source at; the DB source simply becomes dual (endpoint + DB):

// config/local.json → database.schemas[]   (your existing scanner DB source)
{
  "host": "127.0.0.1", "port": 3306,
  "username": "...", "password": "...", "database": "golbat_db",
  "endpoint": "http://<golbat-host>:<port>",   // ← add
  "secret": "<golbat api secret>",             // ← add (must match Golbat's)
  // "httpAuth": { "username": "...", "password": "..." },  // ← optional, if Golbat is behind Basic auth
  "useFor": ["gym", "pokestop", "station", "spawnpoint", "weather", "..."]
}

Restart ReactMap — this is a server-only change (no client rebuild). Gyms/pokestops/stations now fetch from Golbat; markers, popups, filters and deep-links keep working; and if Golbat is down or fort_in_memory is off, each fort type transparently falls back to SQL on the same DB.

Prefer to scope it tighter? Split a useFor: ["gym","pokestop","station"]-only copy of the DB source with the endpoint added and drop those types from the original — but it isn't necessary.

Confirm it's live (logs): each ~15-min availability refresh logs a line like [GYM] loaded available from http://<golbat>:<port>/api/fort/available — …, and map pans stop issuing fort SQL on endpoint-backed sources. A Golbat outage instead logs a fallback warn and reverts to SQL.


What this does

Routes fort map data — gyms, pokestops, and stations (getAll, getOne, getAvailable) — through Golbat's in-memory fort API (UnownHash/Golbat#385) when a scanner source has an endpoint, with transparent SQL fallback. Map pans stop issuing fort SQL entirely on endpoint-backed sources, and Golbat's DNF filtering returns only the forts that will render.

Architecture

  • Endpoint branch per model (mem set on the source): POST /api/<type>/scan → per-record pure mapper → the existing, unchanged secondaryFilter. On any failure (503 / network / bad shape) the model logs and falls through to SQL — a dual source degrades gracefully.
  • DNF is a superset narrow; secondaryFilter guarantees exactness. Pure backends (server/src/filters/fort/*) translate the map filters into Golbat DNF clauses (OR across clauses, AND within). The one hard invariant is never under-return; anything inexpressible (quest title/target conditions, gender, gym badges, time-window cutoffs) poisons to match-all or stays a client-side residual.
  • Exact key semantics in the translation (form-exact pokemon pairs, per-reward-type clauses, amount-exact mega/stardust/xp, reward type-20 treated as mega, grunt-class exclusion subtraction, station liveness) so the DNF result matches what renders.
  • Access-control parity: the endpoint paths reproduce the SQL path's gates — strict area-restriction denial, freshness, secondaryFilter's per-perm field gating — and getOne returns only {lat, lon} like SQL.
  • Observability: each fetch logs the narrowing (DNF(n): X matched -> Y after secondaryFilter (-residual) | <clause shapes>), so any translation gap is visible in production logs.
  • Deep-link parity: an off-viewport onlyManualId fort is fetched via the by-id endpoint and joins the candidate set, mirroring SQL's (bbox) OR id = ?.

Source shapes

A scanner source can be pure-DB (unchanged), dual (endpoint + DB — fort reads use Golbat, everything else and the fallback use the bound DB), or pure-endpoint (endpoint only, no DB — degrades gracefully where a query would need a DB). Enabling the endpoint on an existing DB source (the common case above) makes it dual.

Verified live (dual source, endpoint vs SQL)

Path Result
Pokestop quests 2373 forts scanned → 8 returned → 8 rendered (−0 residual)
Pokestop invasions (+ exclude grunts) 2806 scanned → 107 → 107 (−0)
Pokestop lures / showcases validated
Stations (incl. station_active, gmax) validated; inactive-viewing unaffected (match-all preserved)
Gyms (raids/teams/badges) validated, residual 0

Requirements

  • Golbat with UnownHash/Golbat#385 (fort_in_memory = true), deployed before this — Golbat's scan decoder ignores unknown request fields, so an out-of-date Golbat degrades to match-all/SQL rather than erroring.
  • Per scanner source: endpoint + secret (and optional httpAuth) as for pokémon; DB config unchanged (used for fallback + un-migrated queries).

No test framework exists in this repo; verification was throwaway node goldens per pure module (mappers, DNF backends), eslint/prettier, a full correctness/security/concurrency review of the branch, and the live parity runs above.

🤖 Generated with Claude Code

turtlesocks-bot and others added 15 commits June 5, 2026 13:43
…wMap#1225)

* fix(scanArea): prevent crash when area feature has no name/key

Guard the scan area search filter against features missing a properties.key (which happens when a scan area polygon has no name set), instead of throwing TypeError: Cannot read properties of undefined (reading 'toLowerCase').

Also fixes a longstanding typo (geoJsonFilName / geoJsonFilname -> geoJsonFileName) in the multi-domain example config and docs.

* fix: copilot comments

---------

Co-authored-by: Mygod <contact-git@mygod.be>
@jfberry

jfberry commented Jul 16, 2026

Copy link
Copy Markdown
Author

Validating in a test environment

Scope: this branch routes only gyms through Golbat (getAll/getOne/getAvailable). Pokestop markers/popups and stations still use SQL. Live validation needs Golbat #385 deployed with fort_in_memory = true.

1. Golbat prereqs — deploy #385, fort_in_memory = true; sanity-check (S=secret, G=golbat url):

curl -H "X-Golbat-Secret: $S" "$G/api/gym/available"
curl -H "X-Golbat-Secret: $S" -H 'Content-Type: application/json' -XPOST "$G/api/gym/scan" \
  -d '{"min":{"latitude":LAT1,"longitude":LON1},"max":{"latitude":LAT2,"longitude":LON2},"limit":1000,"filters":[]}'
curl -H "X-Golbat-Secret: $S" "$G/api/gym/id/<gym-id>"

2. ReactMapgit checkout feat/fort-consumer && yarn install, restart (server-only changes — no client rebuild).

3. Config — add endpoint + secret to the scanner DB source whose useFor includes gym (makes it a dual source). If you already did this for the Phase-1 pokestop test, gyms are already routed.

4. Confirm the endpoint path (logs):

  • ReactMap, each ~15-min refresh: [GYMS] [GYM] loaded available from Golbat endpoint …/api/gym/available — …
  • getAll/getOne are silent on success; a [GYM] /api/gym/scan … falling back to SQL warn = the endpoint failed → SQL.
  • Golbat: available-gyms built in … each time getAvailable is hit.

5. Functional — gym + raid markers render; a popup shows team/slots/defenders and raid boss + moves + CP; the filter drawer populates teams, tiers (r), eggs (e), raid bosses.

6. Golden parity (endpoint vs SQL): curl "$R/api/v1/available/gyms?current=1" with fort_in_memory on (endpoint) vs off (SQL fallback); diff the two → expect match (bar time-sensitive active-raid keys).

7. Fallback — stop Golbat or set fort_in_memory=false → gyms should still render via SQL + the fallback warn appears (no hard endpoint dependency; a timeout falls back too).

8. Area restrictions — as an area-restricted user, confirm gyms appear only within permitted areas (exercises the new client-side filterRTree) — the one behavior that isn't a straight SQL mirror, so the most important to verify.

jfberry and others added 7 commits July 17, 2026 15:37
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…st-layer availability fix

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@jfberry
jfberry force-pushed the feat/fort-consumer branch from dadba73 to af62c30 Compare July 17, 2026 14:38
@jfberry jfberry changed the title feat: fort map-data consumer — gyms (stations/pokestops/DNF to follow) feat: fort map-data consumer — gyms, stations & pokestops - with DNF filtering Jul 17, 2026
jfberry and others added 2 commits July 17, 2026 16:06
… badge poison

Review findings:
- tier-override mode matches raid_level alone (curated boss/egg keys
  under-returned tier raids on endpoint sources)
- b<display_type> keys move to the onlyEventStops gate (secondaryFilter's
  events branch consumes them, not the invasions branch)
- endpoint rows resolve the quest layer as dual-capable (pure-endpoint ctx
  flags made effectiveQuestLayer resolve to 'both')
- poison on onlyGymBadges only: onlyBadge defaults to 'all' for every user
  with the gymBadges perm and was silently disabling gym DNF entirely

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replaces the all-gyms/ex/in-battle poison with real narrowing: every gym-layer
display requires the team/slot match (hasGym = enabler && (team || slot)), so
team_id/available_slots clauses mirroring finalTeams/finalSlots are a tight
superset for all four enablers — ex/ar/in-battle narrowing stays residual.
The standalone is_ar_scan_eligible clause is subsumed (ar-shown gyms also
need the team match). Badge viewing still poisons. Power-up narrowing is gone
for good — power-ups are no longer in the game.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@jfberry
jfberry marked this pull request as ready for review July 17, 2026 15:42
@jfberry

jfberry commented Jul 17, 2026

Copy link
Copy Markdown
Author

This is now in my production and I have verified the DNFs through manual testing

jfberry and others added 3 commits July 20, 2026 12:05
The endpoint (in-memory) getAll paths filter rows with filterRTree, which
returns true for empty area inputs — so unlike the SQL path's getAreaSql it did
NOT deny a user who has no assigned areas while strictAreaRestrictions is on and
restrictions are configured. Such a user received every fort in the viewport
(access-control bypass). Add areaRestrictionsDenyAll (mirroring getAreaSql's
strict-deny and empty-consolidated deny) and short-circuit each fort model's
endpoint branch to [] before accepting rows.

Addresses Mygod review comment WatWowMap#1 (P1).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…resilience

Three endpoint-source correctness fixes from the Mygod review:

- WatWowMap#4 (P1): a pure-endpoint pokestop source now reports hasConfirmed:true. The
  Golbat scan always returns confirmed incident data, but no DB schema check
  runs for a knex-less source, so onlyConfirmed and confirmed rocket-reward (a)
  filters were silently degrading to the grunt possible-encounter pool.
- WatWowMap#6 (P2): historicalRarity now skips only pure-endpoint sources (no bound
  knex), not dual sources. Testing source.mem alone dropped a dual source DB
  and cleared the historical rarity map on every refresh.
- WatWowMap#8 (P2): DbManager.getOne uses Promise.allSettled so a pure-endpoint source
  whose by-id fetch misses (falling through to an unbound this.query() that
  throws) no longer fails the whole single-fort lookup.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Golbat now exposes confirmed invasion slots 2/3 in /api/fort/available. Read
them in the availability mapper, adding an a<id>-<form> key per slot the event
config marks as a reward (second/thirdReward) — matching the SQL path, which
already advertises all three confirmed slots. Leaders/Giovanni (41-44) stay
excluded.

Addresses Mygod review comment WatWowMap#9 (P2). Requires Golbat with the slots-2/3
availability change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@jfberry

jfberry commented Jul 20, 2026

Copy link
Copy Markdown
Author

Thanks for the thorough review, @Mygod. Walking through all nine — seven are addressed (across this PR and the Golbat companion #385), and two look like they were reviewed against a Golbat without #385's changes. Details below.

Fixed

  • Add GraphQL Support #1 — strict-area denial before endpoint scans (P1). Confirmed bug: the endpoint paths filter with filterRTree, which returns true for empty area inputs, so a no-area user under strictAreaRestrictions was not denied the way the SQL getAreaSql denies. Added areaRestrictionsDenyAll (mirrors getAreaSql's strict-deny and empty-consolidated deny) and short-circuit each fort model's endpoint branch to []. — 15d82178
  • Add Devices #3 — type-20 mega retrievable (P1). Root cause was upstream: Golbat's vendored proto didn't decode reward type 20 at all (empty info), which is what the m150-150 empty-info stopgap worked around. With Golbat Enable new Pokemon by default #382 merged (decodes TEMP_EVO_BRANCH_RESOURCEpokemon_id/amount), type 20 is now treated as mega everywhere — SQL match/advertise/search, parseRdmRewards, and the endpoint DNF (quest_reward_type: [12, 20]) — and the GoFest stopgap is retired. — ReactMap cade9f68, Golbat c3acb26
  • Refactoring #4 — endpoint incidents confirmed-capable (P1). A pure-endpoint pokestop source now reports hasConfirmed: true (the Golbat scan always returns confirmed lineup data; no DB schema check runs for a knex-less source). — f51f662e
  • Add Ingress Portals #6 — historical rarity on dual sources (P2). historicalRarity() now skips only pure-endpoint sources (no bound knex) rather than any source with mem, so a dual source keeps contributing. — f51f662e
  • Add weather #7 — fort scan result limit (P2). On the Golbat side, the fort scan traversal cap read Tuning.MaxPokemonResults; added a dedicated max_fort_results (default 9000) so the fort API can be tuned independently of pokemon. The per-request Limit still min()s it. — Golbat a3c82e5
  • Add S2 Scan Cells #8getOne on endpoint-only misses (P2). DbManager.getOne now uses Promise.allSettled, so a pure-endpoint source whose by-id fetch misses (falling through to an unbound this.query()) no longer fails the whole single-fort lookup. — f51f662e
  • Add Submission Cells #9 — confirmed rewards from every rocket slot (P2). Golbat's invasion availability exposed only slot 1; it now exposes slots 2/3 (FortLookupIncident/invasionKey/ApiPokestopInvasionAvailable), and the mapper advertises an a key per slot the event marks (second/thirdReward). Index cardinality is unchanged — a confirmed grunt's lineup is deterministic per character, so (character, slot1) already determines slots 2/3. — Golbat e8c4505, ReactMap fe24b481

Not reproduced (appear to predate Golbat #385)

  • Add Initial Filtering Menus #2station_active unsupported filter. station_active is part of the Golbat ApiFortDnfFilter contract in the companion Correct Scrolling Issue #385: it's a declared field with an evaluator (isFortDnfMatch: *StationActive != (StationEndTimestamp > now)) and a dedicated test. Huma rejects only unknown properties, and this one is declared — so endpoint station sources are not rejected. Correct Scrolling Issue #385 adds both stationed_gmax and station_active.
  • Add Spawnpoints #5 — multi-incident DNF drops stops. FortLookup.Incidents is a slice ([]FortLookupIncident), and isFortDnfMatch iterates all non-expired incidents with match-any semantics, not a single flat incident. ReactMap also emits incident_character and incident_display_type as separate clauses (never AND-ed within one), so a stop with the selected invasion in any active incident still matches. If you saw a drop against Correct Scrolling Issue #385's matcher I'm happy to dig into a specific repro.

All changes build/lint clean; Golbat decoder tests pass (incl. new slots-2/3 coverage). #3, #9, and the type-20 path assume a deployed Golbat ≥ #385 + #382.

@jfberry

jfberry commented Jul 20, 2026

Copy link
Copy Markdown
Author

I've taken the change to support slot2/slot3 availability and querying (which I previously had not added because these fields are always empty). Just like AR searching, these fields are legacy fields at the moment - but slot2/3 data coming back is something we would hope for though unlikely, whereas AR is something which is likely gone forever. A focused change later to drop AR data capture & reporting will be a wider impact in golbat

@Mygod

Mygod commented Jul 21, 2026

Copy link
Copy Markdown
Collaborator

Type-20 workaround has been removed in 9acf867. Please merge main and resolve conflicts. Hopefully it will make things easier.

turtlesocks-bot and others added 7 commits July 21, 2026 17:48
develop's canonical type-20 handling (9acf867 + reward-definition refactor)
superseded my SQL-side type-20 work, which was reverted before the merge. The
endpoint DNF builder is ReactMap-fort-consumer-only (not in develop), so
re-apply the mega clause quest_reward_type [12, 20] so Golbat returns type-20
(temp-evo branch) mega stops for the in-memory scan path.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@jfberry

jfberry commented Jul 22, 2026

Copy link
Copy Markdown
Author

Merged develop (through 9acf867) into the branch — thanks, that did make it simpler.

My type-20 handling has converged on develop's: I reverted my SQL-side type-20 generalization (it was redundant with 9acf867 + the reward-definition refactor) and took develop's TEMP_EVOLUTION_RESOURCE_REWARD_TYPES / QUEST_REWARD_FILTER_DEFINITIONS approach wholesale. The only fort-consumer-specific pieces that remain are:

  • Endpoint DNF (filters/fort/pokestop.js): mega clauses emit quest_reward_type: [12, 20] so Golbat returns type-20 (temp-evo branch) mega stops for the in-memory scan — develop doesn't have this file.
  • parseRdmRewards guard: the merge resolution keeps typeof quest.quest_rewards === 'string' ? JSON.parse(...) : quest.quest_rewards (+ an Array.isArray guard). The endpoint path hands parseRdmRewards an already-parsed rewards array rather than a JSON string, so develop's JSON.parse(quest.quest_rewards)[0] alone would throw on endpoint sources. Flagging it as the integration seam if parseRdmRewards gets touched again.

No conflict markers, lint clean; the DNF, type-20 key, and rocket slots-2/3 mapper pass their unit checks. This assumes deployed Golbat ≥ #385 + #382 for the type-20 / slots-2/3 paths.

@Mygod

Mygod commented Jul 22, 2026

Copy link
Copy Markdown
Collaborator

The endpoint Pokestop path currently returns no markers because it omits a required filter limit, and one supported pure-endpoint configuration can fail during startup. Additional endpoint capability, availability, manual-ID, performance, and source-maintainability regressions also remain.

Full review comments:

  • [P1] Pass the result limit to endpoint filtering — server/src/models/Pokestop.js:939-943
    For every successful endpoint-backed Pokestop scan, this call omits secondaryFilter's final resultLimit argument. Its loop condition is filteredResults.length < resultLimit; with undefined this is false immediately, so even non-empty endpoint responses produce no markers. Pass queryLimits.pokestops as the final argument, as the SQL call below does.

  • [P1] Keep pure endpoint filter context off the SQL path — server/src/services/DbManager.js:285-285
    When fallbackRocketPokemonFiltering is disabled and Pokestop uses a pure endpoint source, setting hasConfirmed here makes Pokestop.getFilterContext() bypass its !hasConfirmed return and execute this.query() on an unbound model. state.loadLocalContexts() awaits that during startup, so initialization rejects; make the filter-context path recognize mem and return endpoint capability without SQL.

  • [P2] Preserve the appended manual Pokestop at the limit — server/src/models/Pokestop.js:929-931
    When the scan already returns queryLimits.pokestops candidates and onlyManualId points outside the bounding box, the by-ID record is appended to the end and then this truncation immediately removes it. Deep links therefore fail in full viewports; retain the manual row when capping or apply the final limit after secondary filtering.

  • [P2] Drop incomplete type-20 rewards from availability — server/src/models/pokestopAvailableMapper.js:108-108
    When Golbat reports a type-20 tuple without both a positive pokemon_id and amount, this emits u20 or m{id}-0, but secondaryFilter treats type 20 as a dedicated mega reward and drops it when its computed mega key is empty. The drawer consequently advertises a filter that can never return a marker; match the SQL path by adding type 20 only when both fields are present.

  • [P2] Mark dual endpoint Pokestops as confirmed-capable — server/src/services/DbManager.js:292-295
    When a dual source's database lacks the confirmed column, schemaCheck leaves hasConfirmed false and this overlay copies only endpoint credentials, even though subsequent getAll calls use Golbat rows carrying confirmation and lineup slots. Consequently onlyConfirmed is ignored and reward filtering falls back to possible grunt encounters instead of the actual lineup; override this capability whenever the endpoint is active.

  • [P2] Do not send unsupported station_active clauses — server/src/filters/fort/station.js:37-37
    For endpoint-backed station scans, Golbat's ApiFortDnfFilter has no station_active field and its JSON decoder silently ignores unknown members. In onlyAllStations mode this therefore becomes an empty clause matching every expired station, so the intended cache and payload reduction does not occur; add the corresponding Golbat predicate or leave liveness as an explicit residual match-all.

  • [P2] Replace literal NULs in the cache key — server/src/utils/fortAvailable.js:31-31
    The separators in this template literal are literal NUL bytes, causing Git to classify the entire JavaScript file as binary so normal source diffs and some text tooling cannot inspect it. Use escaped separators such as \0 in the source to preserve the runtime key while keeping the file reviewable.

Also please change the merge target to main branch, also there has been some additional changes introduced in the main branch meanwhile.

jfberry and others added 2 commits July 23, 2026 07:57
- [P1] Endpoint Pokestop scan passed no resultLimit to secondaryFilter, whose
  loop runs while filteredResults.length < resultLimit — so undefined returned
  zero markers. Pass queryLimits.pokestops (mirroring the SQL call) and drop the
  pre-truncation, which also fixes [P2] dropping the appended off-viewport
  manual-id row before filtering.
- [P1] getFilterContext ran this.query() on an unbound (pure-endpoint) model
  when fallbackRocketPokemonFiltering is off and hasConfirmed is set, rejecting
  at startup. Recognize mem and return endpoint capability without SQL.
- [P2] Dual endpoint sources now marked hasConfirmed:true even when the bound DB
  lacks the confirmed column (getAll uses Golbat rows with confirmation).
- [P2] Availability mapper advertises type-20 mega only when both pokemon_id and
  amount are present (never u20 or m<id>-0), matching what secondaryFilter keys.
- [P2] Cache-key separators are the \0 escape instead of literal NUL bytes, so
  git no longer classifies fortAvailable.js as binary.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
# Conflicts:
#	server/src/services/EventManager.js
@jfberry
jfberry changed the base branch from develop to main July 23, 2026 07:03
@jfberry

jfberry commented Jul 23, 2026

Copy link
Copy Markdown
Author

Thanks — all seven addressed, plus retargeted to main and merged the latest main.

P1

  • Result limit not passed to endpoint filtering — the endpoint secondaryFilter call now passes queryLimits.pokestops (its loop is filteredResults.length < resultLimit, so undefined returned zero markers). Removed the pre-truncation entirely, mirroring the SQL call.
  • Pure-endpoint filter context off the SQL pathgetFilterContext now recognizes mem and returns { hasConfirmedInvasions: true } without touching this.query(), so startup no longer rejects on an unbound model when fallbackRocketPokemonFiltering is off.

P2

  • Manual Pokestop at the limit — fixed by the same change: no pre-truncation, so the appended off-viewport by-id row survives into secondaryFilter (parity with SQL's OR id = manualId + LIMIT).
  • Incomplete type-20 in availabilityquestRewardKey/the mapper now emit m<id>-<amt> only when both pokemon_id and amount are present, and never u20 (type 20 has a dedicated filter). Matches what secondaryFilter keys, so no dead drawer entries.
  • Dual endpoint confirmed-capability — the dual overlay now also sets hasConfirmed: true, so onlyConfirmed and confirmed rocket-reward filters use the Golbat lineup even when the bound DB lacks the confirmed column.
  • Literal NULs in the cache key — switched to the \0 escape in source; git no longer classifies fortAvailable.js as binary.

station_active — I'll push back gently here: it is in the Golbat companion (#385) — a declared ApiFortDnfFilter field with an evaluator (*StationActive != (StationEndTimestamp > now)) and a test, not an unknown member. You're right that Huma silently ignores genuinely-unknown fields (no additionalProperties: false), but that's the general DNF-superset contract — any field a given Golbat doesn't recognize is ignored and secondaryFilter/the station time-gate still finalizes correctness (just without the payload reduction). Against a Golbat that predates #385's station changes it degrades to residual match-all rather than breaking. The consumer already requires #385, so it resolves optimally there. Happy to add a hard residual fallback if you'd rather not depend on the field.

Base is now main; note the diff currently carries develop's commits that aren't yet in main — those drop out once develop lands on main.

P1:
- getOne (gym/pokestop) endpoint now projects to {lat, lon} like the SQL path,
  instead of returning the raw Golbat record. The client controls the GraphQL
  selection, so returning the full record leaked raid/team/lure/detail past the
  sub-perm split and area restrictions for any fort id.
- Pokestop endpoint onlyLevels power-up gate now guarded on !onlyAllPokestops
  (like the gym sibling). SQL only applies it under onlyAllPokestops, so the
  endpoint was under-returning the entire quest/invasion/lure layer for a user
  with a non-all levels filter.

P2:
- DbManager.search / submissionCells use Promise.allSettled: the fort
  search/getSubmissions methods have no endpoint branch, so a pure-endpoint
  source rejected the whole batch (crashing search/submissions) instead of
  degrading. Mirrors the getOne fix.
- fetchJson redacts Authorization / X-Golbat-Secret before the debug log and
  the failed-request payload dump, so credentials are no longer written to disk.
- EventManager only arms the availability TTL on a non-empty refresh, so a
  failed (empty) endpoint refresh no longer suppresses recovery for the window.

P3:
- encodeURIComponent on all fort by-id fetch URLs (getOne, manual-id,
  getDynamaxMons) — no path traversal via id/onlyManualId.
- historicalRarity uses Promise.allSettled so a dual source whose DB lacks
  pokemon_stats no longer fails the whole batch and blanks every rarity map.

(Skipped review P3: force does not bypass the 30s combined-availability cache —
that cache deliberately caches failures to avoid hammering; the TTL fix already
cuts the drawer-empty window to <=30s, and force-bypass needs invasive threading
for a minor gain.)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@jfberry

jfberry commented Jul 23, 2026

Copy link
Copy Markdown
Author

Whole-PR review + fixes

Given how much moved recently (the develop/main merges, type-20 convergence, two review rounds), I ran a full review across three dimensions — correctness/SQL-parity, security/access-control, concurrency/integration — over the ~2,050-line feature diff, verifying each finding against the code. 8 issues surfaced (2 P1, 3 P2, 3 P3); 7 are fixed in 09ac0769, 1 is a documented accept.

P1 (fixed)

  • getOne returned the full Golbat record — SQL getOne projects to {lat, lon}; the endpoint returned the raw record, and since the client controls the GraphQL selection, a gyms-but-not-raids user could read raid/team/detail for any fort id, outside their areas. Now projected to {lat, lon} to match SQL.
  • Pokestop onlyLevels under-return — the endpoint applied the power-up gate to every row, but SQL applies it only under onlyAllPokestops (the gym sibling has the !onlyAllGyms || guard). A user with a non-all levels filter lost the whole quest/invasion/lure layer. Added the !onlyAllPokestops || guard.

P2 (fixed)

  • search / submissionCells crashed for pure-endpoint sources — those fort methods have no endpoint branch, and DbManager batched them with Promise.all, so a pure-endpoint source rejected the whole batch. Switched to Promise.allSettled (mirrors the getOne fix), degrading that source to no results.
  • Credentials written to diskfetchJson dumped request options (incl. X-Golbat-Secret / Basic auth) to logs/ on any failed request. Now redacts those headers before the debug log and the payload dump.
  • Availability TTL armed on empty refresh — a failed (empty) endpoint refresh armed the 60s TTL and suppressed recovery. TTL is now only armed on a non-empty result.

P3

  • Fixed: encodeURIComponent on all fort by-id fetch URLs (no path traversal via id/onlyManualId); historicalRarityallSettled so a dual DB lacking pokemon_stats doesn't blank every source's rarity map.
  • Accepted (not changed): force doesn't bypass the 30s combined-availability cache — that cache deliberately caches failures to avoid hammering an old/down Golbat, and the TTL fix already cuts the drawer-empty window to ≤30s; a force-bypass needs invasive threading for a minor, drawer-only gain.

Verified solid (no action)

  • All three DNF builders correctly mirror SQL as superset-narrows — layer-gating, exact quest keys (form:0/bare, amount-exactness, type-20 [12,20]), rocket→incident_character expansion, onlyGymBadges[], station station_active. No under-returns.
  • All availability mappers reproduce the SQL getAvailable keys (incl. type-20 complete-only, rocket slots 1/2/3).
  • Strict-area guard is complete on all three getAll endpoints and faithfully mirrors getAreaSql; manual-id preserves area filtering; cache key is credential-isolated.
  • Golbat's DNF matches both AR and non-AR quest layers (arMatch || noArMatch) — no systematic layer under-return.

node --check + ESLint clean across all changed files. As before, worth a functional smoke test on a pokestop source (quests + confirmed invasions) once pulled, given there's no server test suite. Requires Golbat ≥ #385 + #382.

@jfberry

jfberry commented Jul 23, 2026

Copy link
Copy Markdown
Author

(running in my production, will also ask my other testers of this PR to update after a short period of no further review comments)

@Mygod

Mygod commented Jul 24, 2026

Copy link
Copy Markdown
Collaborator

The endpoint paths can omit valid forts, break advertised dual-source fallback, and replace cached availability or rarity with failure-derived empty data. Lint and production build pass, but they do not cover these runtime contract and failure-mode regressions.

Full review comments:

  • [P1] Apply the fort result cap after residual filters — server/src/models/Pokestop.js:886-886
    When an endpoint-backed query uses residual-only conditions such as area restrictions, freshness, advanced quest conditions, gym badges, or station activity, Golbat stops after the broader DNF reaches its result cap. Rows later rejected by filterRTree or secondaryFilter consume that cap, so valid forts appearing later in the scan are omitted; use pagination/over-fetching or exact server-side predicates before enforcing the ReactMap limit. The same pre-filter cap exists in Gym.js and Station.js.

  • [P1] Keep multi-incident stops out of flat DNF filters — server/src/filters/fort/pokestop.js:281-282
    When a stop has multiple simultaneous incidents and the selected character or display type belongs to a non-indexed incident, these clauses compare against Golbat's single flat FortLookup incident tuple. The scan drops the stop before with_incidents can return the full invasions[] that secondaryFilter would match; leave these predicates residual or make Golbat test every active incident.

  • [P1] Preserve cached availability when refresh fails — server/src/services/EventManager.js:172-177
    When a pure-endpoint /api/fort/available request fails, Db.getAvailable returns [], but this branch runs only after the cached options have already been replaced and Pokestop quest conditions reset. The drawer is therefore emptied instead of retaining the last good result; conversely, a genuinely successful empty result is never timestamped and is re-scanned on every session. Propagate refresh success separately and only commit or arm the TTL for a successful refresh.

  • [P1] Preserve DB capability flags for endpoint fallback — server/src/services/DbManager.js:296-300
    For a dual endpoint-plus-DB source whose incident table lacks confirmed or lineup columns, this also changes the context used by the SQL fallback. If /api/pokestop/scan or /api/fort/available is unavailable, fallback queries reference confirmed and slot_1/2/3_*, reject, and are discarded by runScannerSources, leaving no pokestops or availability. Keep endpoint capabilities separate from schema-derived fallback flags.

  • [P2] Remove the unsupported station_active clause — server/src/filters/fort/station.js:37-37
    For endpoint-backed station queries outside inactive mode, this emits station_active, but Golbat's ApiFortDnfFilter has no such field; the planned producer extension only adds stationed_gmax. The current decoder ignores the unknown key, so cases such as onlyAllStations become match-all and expired or stale stations can exhaust the endpoint's hard result cap before the local time gate. Keep liveness residual or add it to the producer contract first.

  • [P2] Preserve conditions for generic quest rewards — server/src/models/pokestopAvailableMapper.js:172-175
    When Golbat reports a reward type mapping to u<type>, this branch omits its title and target from conditions. The SQL path calls process for every generic quest row, so endpoint-only deployments lose the advanced quest-condition selector for these rewards and existing adv selections can be cleared by the client. Run generic keys through the same condition collector.

  • [P2] Retain historical rarity after total query failure — server/src/services/DbManager.js:375-377
    If every eligible pokemon_stats query rejects, such as during a temporary outage of the sole dual-source DB, filtering rejected promises leaves results empty and setRarity clears the cached historical map. Previously Promise.all entered the catch path and retained the last good data. Track whether any eligible query succeeded and avoid replacing rarity state on total failure.

Four ReactMap-side regressions from Mygod review; the three that would modify
Golbat or broaden the Golbat queries are held (see PR discussion).

- Availability mapper runs generic u<type> quest keys through process() so they
  carry their title/target conditions, matching develop SQL genericQuests
  (endpoint deployments were losing the advanced quest-condition selector).
- EventManager #refreshAvailable returns early on an empty result: a failed
  (empty) refresh no longer replaces the last-good drawer + conditions or arms
  the TTL — it retains the cache and retries next call.
- DbManager no longer forces hasConfirmed:true on a dual source. That flag gates
  the SQL fallback confirmed-column query, so a dual DB lacking confirmed would
  reject on fallback. The endpoint scan branch marks itself confirmed-capable
  locally instead, keeping endpoint capability separate from the fallback flag.
- historicalRarity retains the last-good rarity map when every eligible
  pokemon_stats query fails (total DB outage) instead of clearing it — the
  allSettled change had lost Promise.all catch-path retention.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@jfberry

jfberry commented Jul 24, 2026

Copy link
Copy Markdown
Author

Thanks — four of these were real ReactMap-side regressions and are fixed in 4b278a3; the other three reference the Golbat companion contract and are addressed there rather than by broadening the queries.

Fixed (4b278a3)

  • Generic quest conditions — the availability mapper now runs every key, including generic u<type> fallbacks, through process(), so they carry their title/target conditions to match develop's SQL genericQuests.forEach(process). Endpoint deployments no longer lose the advanced quest-condition selector.
  • Preserve cached availability on failed refresh#refreshAvailable now returns early on an empty result, so a failed (empty) /api/fort/available no longer replaces the last-good drawer + conditions or arms the TTL; it keeps the cache and retries on the next call.
  • Endpoint capability vs. SQL-fallback flags — reverted the dual-source hasConfirmed = true overlay (it poisoned the SQL fallback into querying a confirmed column the DB may lack). The endpoint scan branch now marks itself confirmed-capable locally, leaving the schema-derived flag for the fallback.
  • Retain historical rarity on total failure — if every eligible pokemon_stats query rejects, the last-good rarity map is retained instead of cleared (restoring the old Promise.all catch-path behaviour that the allSettled change had dropped).

Addressed on the Golbat side / not broadened

These three are contract points against Golbat #385 (the companion), and #385 already covers them — so the fix is to rely on the contract, not to widen what Golbat returns:

  • Result cap before residual filters — handled by a dedicated, separately-tunable Golbat cap (max_fort_results, default 9000, independent of max_pokemon_results) rather than over-fetching/pagination, which would broaden every scan. The residual gates (area polygons, freshness, adv conditions, badges) can't be pushed server-side, so the deliberate choice is a high, operator-tunable ceiling.
  • Multi-incident stops#385's FortLookup.Incidents is a slice ([]FortLookupIncident), not a single flat tuple, and isFortDnfMatch iterates every non-expired incident (for _, inc := range fortLookup.Incidents → match-any). A stop matches if any active incident satisfies the clause, and ReactMap emits incident_character / incident_display_type as separate clauses (never AND-ed), so no valid stop is dropped.
  • station_active — it is a declared field in #385's ApiFortDnfFilter (decoder/api_fort.go) with an evaluator (*StationActive != (StationEndTimestamp > now)) and a test, so it's part of the producer contract — not an unknown key. Leaving liveness residual would instead ship every expired station and defeat the payload reduction. (#385 adds both stationed_gmax and station_active.)

All four fixes are lint/node --check clean and the mapper's generic-condition behaviour is unit-verified. As always this assumes Golbat ≥ #385 + #382.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants