Skip to content

fix(enrich): normalize the click tolerance ring per axis - #55

Open
Sarcastic-Soul wants to merge 1 commit into
nossa-y:mainfrom
Sarcastic-Soul:fix/click-tolerance-vertical-axis
Open

fix(enrich): normalize the click tolerance ring per axis#55
Sarcastic-Soul wants to merge 1 commit into
nossa-y:mainfrom
Sarcastic-Soul:fix/click-tolerance-vertical-axis

Conversation

@Sarcastic-Soul

Copy link
Copy Markdown
Contributor

Fixes #50

The bug

_resolve_click normalizes a click's coordinates per axisxn by screen width, yn by screen height — but derived its tolerance ring from screen_w only, then applied that one value to both:

xn, yn = x / screen_w, y / screen_h
...
tol_n = _TOL_PX / screen_w
if l - tol_n <= xn <= l + w + tol_n and t - tol_n <= yn <= t + h + tol_n:

tol_n is in normalized-x units. Using it against yn scales it by the display aspect ratio, so on the default 1728x1117 the documented 40px ring is really 25.9px vertically:

40 / 1728 = 0.023148   ->   0.023148 * 1117 = 25.9 px

What that costs

A click 30px past an element's edge — comfortably inside the 40px ring:

direction before after
30px right of the right edge ('Btn', 'tolerance') ('Btn', 'tolerance')
30px below the bottom edge ('main content', 'zone') ('Btn', 'tolerance')

The click was never lost — the resolution chain always ends somewhere. But it fell through to the coarse screen-zone branch, so the label degraded from the element's real name to "main content" and _confidence dropped from medium to low. Clicks near the bottom edge of small controls were the most affected, since those are exactly the ones that land in the ring rather than the box.

The change

One tolerance per axis:

tol_x = _TOL_PX / screen_w
tol_y = _TOL_PX / screen_h

Plus a short comment recording why they can't be shared, so the next reader doesn't re-collapse them.

Tests

Two new tests in tests/test_enrich.py, both driven by a tiny helper that builds a one-frame/one-element database so the geometry is explicit rather than inferred from the shared fixture:

  • test_tolerance_ring_is_symmetric_in_pixels — equal pixel offsets past the right and bottom edges must both resolve as tolerance. This is the regression guard: it fails on main (the vertical case returns zone) and passes here.
  • test_tolerance_ring_still_has_an_edge — well outside the ring on either axis still falls through to zone, so the fix widens the ring correctly rather than removing its boundary.

Full suite: 109 passed (107 before, +2).

Scope

enrich.py only, inside the tolerance branch. Exact containment, the zone fallback, the rescue-window logic, and _confidence are all untouched — a click that resolved exactly before still resolves exactly, with the same label.

_resolve_click compares against per-axis normalized coordinates (xn by
width, yn by height) but derived a single tolerance from screen_w, so the
vertical ring was shrunk by the display aspect ratio: 40px became 26px on
1728x1117. Vertical near-misses fell through to the coarse zone branch,
losing the element's real label and dropping confidence to low.

Derive tol_x and tol_y separately. Adds a symmetry test (equal pixel
offsets past the right and bottom edges must both resolve as "tolerance")
and a companion test that the ring still has an outer edge.

Fixes nossa-y#50

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Click-resolution tolerance ring is ~35% too small vertically

1 participant