Skip to content

fix(spur-cli): resolve first allocated node from compressed hostlist - #589

Draft
maybeharshit wants to merge 1 commit into
ROCm:mainfrom
maybeharshit:fix/Issue575
Draft

fix(spur-cli): resolve first allocated node from compressed hostlist#589
maybeharshit wants to merge 1 commit into
ROCm:mainfrom
maybeharshit:fix/Issue575

Conversation

@maybeharshit

Copy link
Copy Markdown
Contributor

What

srun and sattach connect directly to a job's first allocated node to stream
live output. They extracted that node from JobInfo.nodelist with
nodelist.split(',').next() — but the controller reports nodelist as a
compressed Slurm-style hostlist (built by spur_core::hostlist::compress),
not a raw comma list. For any job on 2+ nodes this yields a non-hostname:

  • node[001-002] has no top-level comma, so the split returns the whole literal
    node[001-002].
  • node[001,003] has a comma only inside the brackets, so the split returns the
    broken fragment node[001.
  • multi-prefix (gpu[001-004],cpu[001-002]) splits into the fragment gpu[001-004.

The result was used directly as a host: sattach dialed
http://node[001-002]:6818 and failed to connect, and srun's get_node()
probe failed so live output was silently skipped. Fixes #575.

Approach

  • Add spur_core::hostlist::expand_first, which expands a pattern only far
    enough to yield its first hostname — the inverse of the controller's
    compress, and cheaper than expand(..).next() for large allocations.
  • Add a shared spur_cli::nodelist::first_allocated_node helper that both call
    sites now use, replacing the duplicated split(',') logic. It falls back to a
    comma-split for a malformed pattern so a bad input still yields something
    connectable rather than nothing (mirroring expand_hostlist_or_split).
  • Wire it into sattach (clear error if unresolvable) and srun
    try_stream_output (keeps the existing skip-on-failure behavior).

The unrelated srun::first_node helper (for user-typed -w, resolved
server-side) is intentionally left unchanged.

Design choices

  • Uses the exact inverse of what produced the string; [1] PR fix(spur-cli): compress sinfo NODELIST as a Slurm-style hostlist #574
    hardened compress() and verified expand(compress(x)) round-trips, so
    relying on expand/expand_first is guaranteed correct even for mixed
    zero-padding (node[9,010-011]) and multi-prefix lists.
  • Client-side only: no proto, persisted Raft/WAL, or config change, so there is
    no upgrade/compat hazard.

Testing

  • Unit: new expand_first tests (range, gap, multi-prefix, mixed padding, plain
    list, single, empty, suffix bracket, unmatched-bracket error, and parity with
    expand(..).next()), plus first_allocated_node tests. cargo clippy
    (spur-core, spur-cli) clean; cargo test for spur-core and spur-cli pass.
  • E2E on a 2-node LXD cluster with hostnames compressing to spurnode[1-2]:
    before the fix, sattach failed with a DNS error on spurnode[1-2]; after,
    sattach connects to spurnode1 and streams live output, and srun -N2
    streams from both nodes with no controller-side connect errors.

Made with Cursor

srun and sattach extracted the job's first node with `nodelist.split(',')`,
but the controller reports `JobInfo.nodelist` as a compressed Slurm-style
hostlist (e.g. `node[001-002]`). Splitting that on commas yields a bracket
fragment, so both tools targeted a non-existent host: sattach failed to
connect and srun silently skipped live output streaming.

Add `spur_core::hostlist::expand_first`, which expands a pattern only far
enough to yield its first hostname (the inverse of the controller's
`compress`), and a shared `nodelist::first_allocated_node` helper that both
call sites now use. Falls back to a comma-split for malformed patterns.

Co-authored-by: Cursor <cursoragent@cursor.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

srun/sattach pick the wrong first node from a compressed nodelist

1 participant