Qualitative failure taxonomy: bucket each model's FP/FN from cached detections

[jonfroehlich]

Motivation

The P/R/F1 table tells us that the challengers fail; it doesn't tell us how. The chat VLMs are "FP-heavy" (119–293 FP vs RampNet's 9) — but we don't know how much of that flood is genuine hallucination vs measurement artifact. Several confounds could be inflating FP (and FN) without the models being as wrong as the numbers imply:

• Box→center + tight radius double-penalizes loose localization: a correct-but-loose box whose center lands >0.022 from the GT point is scored as an FP and the ramp as an FN — one miss, counted twice.
• Hood/nadir (pitch_deg=-30 → ~⅓ of every view is vehicle hood + black cap; GDINO's top box is the hood — see companion nadir-mask issue).
• Cross-view dedup double-counting at tile seams (360 pano seam: curb ramps split across the left/right edge get double-counted in GT scoring #43).

What to build

A script over .model_cache/ that, for each model + city, classifies every FP and FN against the derived GT (rampnet/detection_eval.py):

• FP buckets: on-hood/nadir · near-GT-but-outside-match-radius (localization) · cross-view dedup duplicate · on driveway / crosswalk paint / stairs · genuine hallucination (far from anything).
• FN buckets: small/distant · occluded · seam-truncated (360 pano seam: curb ramps split across the left/right edge get double-counted in GT scoring #43) · inside the hood region.
• Output: per-model bucket counts + a worst-cases gallery, reusing the dump_detections.py overlay machinery.

Why it's cheap and high-value

• No model re-runs — cached detection points + derived GT already contain everything the geometric buckets need (the gallery also needs the local pano images).
• Turns "119 FPs" into "40 localization / 25 hood / 30 driveway / 24 real," which is what tells us whether the RampNet lead is as clean as it looks — and lets us qualify the paper's numbers precisely.
• De-risks the cascade (Experiment: hybrid RampNet + open VLM (Qwen3-VL) — ensemble / cascade / distillation #35): if OWLv2's ~8,800 FPs are mostly obvious junk (hood/driveway), an arbiter kills them trivially; if they're ambiguous concrete, the arbiter struggles too. The taxonomy sets the cascade's ceiling before we build it.

Refs #39, #43, #35.

[jonfroehlich]

Measurable half done in #94 (branch analysis/miss-taxonomy-46). Summary of what the buckets say,
and what is left.

FN side. Every miss bucketed from the committed low-floor caches (0.05 floor, well under the
0.30 operating point). Pooled, seven US splits, 2,060 GT / 427 misses:

bucket	misses	share	recall pts
merged	124	29.0%	0.060
sub_threshold	166	38.9%	0.081
localization	9	2.1%	0.004
silent	128	30.0%	0.062

The consequence is for #59, which sized the near-field population at 0.087 recall points and flagged
the "appearance/vocabulary" label as an inference. Bucketed, only 0.023 of that is
sourcing-addressable — the rest is confidence (0.041) and adjacent-pair peak merging (0.023).
#59''s bound over-states the reachable population by 3.8×.

Both confounds this issue named came back negative:

• "one miss, counted twice" / the matcher. Max-cardinality bipartite matching vs the deployed
  greedy matcher is a wash — 10 ramps hit only under optimal, 10 only under greedy, net zero.
• "a peak was there" could be density. RampNet emits 4.2 floor-level peaks/pano; azimuth-randomized
  null puts near-field sub_threshold at 46.7% real vs 4.7% chance.

merged is a target problem, not an extractor one — 63% of merged pairs sit above
min_distance=10 in Chebyshev, so the extractor was free to emit two peaks and the heatmap simply
had one mode. That retires the min_distance=3 idea for most of the bucket, on top of #62. And it
explains paterson: 80 of its 111 misses (72%) are merged, vs 3–24% elsewhere. The paired tactile
surfaces are a σ problem, not a vocabulary one.

FP side, all 8 models × 7 splits, read from .model_cache with no GPU and no model load:

model	FP	duplicate	near_gt	hood	isolated	excess over chance
rampnet @0.30	181	2.8%	19.3%	3.3%	74.6%	+5.3%
gemini-3.6-flash	798	1.5%	24.2%	1.6%	72.7%	+11.6%
gemini-3.1-pro-preview	615	2.6%	30.9%	0.7%	65.9%	+15.1%
Qwen3-VL-8B	1,550	0.6%	16.8%	3.7%	79.0%	+5.4%
Qwen3-VL-32B	248	2.8%	35.9%	1.6%	59.7%	+19.1%
molmo2-8B	1,228	0.6%	16.8%	2.0%	80.7%	+4.1%
owlv2-large	55,538	1.4%	8.2%	15.4%	75.0%	−0.2%
grounding-dino-base	61,586	0.8%	5.6%	26.1%	67.5%	−0.7%

Density had to be corrected here too, exactly as it did for recall. OWLv2 and Grounding DINO are
the only two models whose near-ramp FPs are entirely chance — every sparse model shows a real
excess. Their boxes are not "nearly right", they are everywhere. Different method from
null_recall.py (exact arc geometry, not cross-pano shuffling), same conclusion.

The GDINO hood prediction in this issue was right: 26.1% of its 61,586 FPs are on the vehicle,
free to remove with a nadir mask.

For the #35 cascade — the question this issue posed — the answer is partly the easy case, but
67–75% of every model''s FPs land in isolated, and that residue sets the cascade''s ceiling.

Leaving this issue open for the gallery half. silent and isolated are both upper bounds that
geometry cannot split: silent still holds occlusion / deep shadow / debris / GT disagreement, and
isolated still holds driveways, crosswalk paint and stairs. Both need the panorama imagery — the
dump_detections.py overlay work this issue originally scoped. The 128 silent misses and a sample
of the isolated FPs are the obvious first galleries.

🤖 Generated with Claude Code (claude-opus-5[1m])

[jonfroehlich]

Gallery half now built (fd6ea38, 15f791b on #94). Both galleries render; nobody has tagged
them, so this stays open. Two things worth flagging beyond the crops themselves.

1. Much of silent no longer needs eyes

silent_witness.py asks, for each of the 128 silent misses, whether any other model detected a
ramp there. If one did, the imagery demonstrably contains a recognizable ramp — so RampNet''s
failure is RampNet-specific, which is the strongest evidence for a real vocabulary failure
obtainable without a reviewer, and it needs no gallery at all.

Density correction was mandatory for the third time in this issue: OWLv2 witnesses 121 of 128,
and chance accounts for 76.9 of them.

witness	raw	by chance	excess
union, 5 sparse models	69	30.0	+39.0
union, 2 dense detectors	127	102.2	+24.8

Near-field: 32/45 raw (71%), chance 13.0, ~19 corrected. That brackets the sourcing-addressable
population at 0.009–0.022 recall points against #59''s original 0.087 — worth 1–2 recall points,
not 9. The test is one-directional and the docs say so: unwitnessed ≠ confirmed invisible, since
every challenger is weaker than RampNet here.

The gallery''s work queue is therefore the 59 unwitnessed misses, not all 128 —
analysis_out/silent_witness.json carries the list.

2. The instrument check changed the gallery''s design, and would have changed its conclusions

geom() sizes ramps at the model''s 4096-px input. The stored panoramas do not agree — with
that, or with each other:

split	stored width	vs model
bend / paterson / gainesville	16384	4.0×
richmond	4096–12288	mixed within the split
annapolis	8000	2.0×
clovis	5760	1.4×
morgantown	4096	1.0×

So "I can see the ramp" does not imply "the model should have" — on a 4× split those pixels
never reached it. Morgantown is the inverse and quietly the most useful split in the benchmark for
this question: no resolution headroom, so a ramp visible there is an appearance failure.

Left uncorrected this would have licensed appearance verdicts on 14 of 45 near-field silent
misses while appearing to license all 45. The fix is a third panel per crop — context / detail /
as the model saw it, the last rendered from the pano downsampled to 4096 and blown back up with
NEAREST so it looks exactly as coarse as it really was. The reviewer compares budgets instead of
inferring one.

Other instrument decisions: gnomonic views centred on each point''s own lon/lat (removes the
equirectangular stretch that is worst exactly where ramps sit, and centres every subject so framing
can''t bias one judgment against another); panel widths derived from fov × the pano''s own width,
so a crop never invents or discards detail; items under 30 source px rendered but excluded from
any rate rather than labelled (15 of 128, all far-field).

3. FP gallery

fp_gallery.py renders worst-N isolated false positives per model through the same instrument
and manifest, ranked by the model''s own confidence where it has one. Sampling is always reported —
the isolated population is ~41,000 boxes for OWLv2 alone, so any gallery is a sample and says which
and how many it left out.

The instrument caveat cuts the other way here: on a 4× split, calling a box "obviously not a ramp"
may be using detail the model never had.

One spot-check, offered as an illustration and explicitly not a finding (n=1): Gemini fired on a
circular manhole cover with radial grating beside a red-painted curb — a truncated-dome tactile-pad
lookalike, still legible at the model''s 182-px budget, so not a resolution artifact. If that
pattern generalizes it is a nameable failure mode and it bears directly on #35''s arbiter ceiling,
but that is exactly what the tagging pass is for.

Manifests are keyed like benchmark/<city>/incremental_fp_tags.json, so verdicts commit and join
the same way — including the same fragility, that the key encodes coordinates and a re-extraction
orphans a tag.

I have deliberately produced no bulk labels of my own: it is a judgment task, and it is the one
I have previously gotten backwards on this benchmark by judging before checking the instrument.

🤖 Generated with Claude Code (claude-opus-5[1m])