Skip to content

fix(skill-creator): handle eval_id=None crash in eval-viewer sort#1316

Open
Alex-ai-future wants to merge 1 commit into
anthropics:mainfrom
Alex-ai-future:fix/skill-creator
Open

fix(skill-creator): handle eval_id=None crash in eval-viewer sort#1316
Alex-ai-future wants to merge 1 commit into
anthropics:mainfrom
Alex-ai-future:fix/skill-creator

Conversation

@Alex-ai-future

@Alex-ai-future Alex-ai-future commented Jun 15, 2026

Copy link
Copy Markdown

Both Python and JS sort logic in the skill-creator eval-viewer crash when eval_id is None/null. This fix makes the sort handle that case gracefully by placing those runs last.

Bug description

When the skill-creator eval-viewer scans a workspace, build_run() returns eval_id=None for any run directory that does not have an eval_metadata.json file (or the file exists but lacks an eval_id field). The sort code then crashes:

Python (skills/skill-creator/eval-viewer/generate_review.py):

runs.sort(key=lambda r: (r.get("eval_id", float("inf")), r["id"]))

r.get("eval_id", float("inf")) returns None — not inf — because .get() only falls back to the default when the key is absent, not when it exists with a None value. Comparing (1, "...") < (None, "...") raises TypeError: '<' not supported between instances of 'int' and 'NoneType'.

JS (skills/skill-creator/eval-viewer/viewer.html):

runs.map(r => r.eval_id).sort((a, b) => a - b)

null - int produces NaN, making the sort order unpredictable.

Why this is not a dirty-data problem

Scenario eval_metadata.json exists? eval_id Is this dirty data?
Normal run with metadata Yes 1 No
Manual test directory No None No
Run shared by another team No None No
Early version of skill-creator output No None No

The producer (build_run()) and consumer (find_runs()) are in the same file (generate_review.py). build_run() was explicitly designed to return eval_id=None as a valid output rather than raising or skipping — the author chose to treat runs without metadata as legitimate entries. This is an internal contract: one module produces None, another module does not handle it.

The fix is to make the sort defensive: handle None explicitly instead of assuming all eval_id values are integers.

Fix

  • skills/skill-creator/eval-viewer/generate_review.py: Changed to r["eval_id"] if r["eval_id"] is not None else float("inf")
  • skills/skill-creator/eval-viewer/viewer.html: Changed to explicit null-aware comparator that places null values last

Both Python and JS sort logic assumed eval_id is always an int. When
eval_metadata.json is missing, build_run() returns eval_id=None which
is a valid output. The sort code now explicitly handles None by placing
those runs last.

- generate_review.py: r.get() doesn't work when key exists with None value
- viewer.html: a - b produces NaN when either operand is null

Signed-off-by: Alex <alex.tech.lab@outlook.com>
@Alex-ai-future Alex-ai-future changed the title Fix eval_id=None crash in eval-viewer sort fix(skill-creator): handle eval_id=None crash in eval-viewer sort Jun 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant