Lifecycle tags that aren't facts: `spawn:version` is the literal "0.1.0", and `spawn:completion-file` is written from a flag default with no `spawn:on-complete` — both mislead the #502 triage

[scttfrdmn]

Summary

Two independent one-line defects in pkg/aws/tags.go, filed together because I hit both while diagnosing the same event and both fail the same way: a spawn: tag asserts something about the instance that is not true, and the operator's only view into a non-reaping fleet is those tags.

This is the follow-up #502 offered ("spawn:version reads 0.1.0 while the binary is spored version 0.100.0; happy to file separately"), plus a second one found next to it. #502's root cause is fixed; these are what remain, and they are what made that diagnosis slower than it needed to be.

Context: a c8g.8xlarge build node launched with --ttl 4h ran 16.8 h past its own spawn:ttl-deadline and was stopped by a human, ~$21 at the tagged spawn:price-per-hour=1.276160. The cause was #502 (the --iam-policy-file role had no spored baseline, so DescribeTags 403'd and TTL=0s). While working out why nothing reaped it, the tag set was the primary evidence — and two of the tags in it were misinformation.

Bug 1 — spawn:version is a hard-coded literal, in the "always present" identity block

pkg/aws/tags.go:50, in the section commented "base identity (always present)":

tags := []types.Tag{
    {Key: aws.String("spawn:managed"),        Value: aws.String("true")},
    {Key: aws.String("spawn:root"),           Value: aws.String("true")},
    {Key: aws.String("spawn:created-by"),     Value: aws.String("spawn")},
    {Key: aws.String("spawn:version"),        Value: aws.String("0.1.0")},   // <-- literal
    ...
}

cmd/root.go:25 has var Version = "", populated by ldflags — the real value, printed correctly by spawn version (0.100.2). buildTags doesn't reference it. So every instance spawn has ever launched carries spawn:version=0.1.0, regardless of the binary.

Measured: the affected node's tag read spawn:version=0.1.0; the daemon's own log line read spored v0.100.0 starting....

Why it isn't cosmetic: "was this instance launched by a spawn that predates the fix?" is the first question when a fleet doesn't reap itself, and it is exactly the question this tag looks like it answers. Answering it required SSHing in to read /var/log/spored.log, on a node whose access path was itself under suspicion. A stale-by-construction version tag is worse than an absent one, because absence sends you to the log immediately.

The value is also unreachable by any reader. The only other spawn:version reference is pkg/aws/ami_mgmt.go:170, which reads it off AMIs (surfaced as spawn ami list --version, documented in docs-gen/ami.md). Nothing consumes the instance tag — so today it is pure misinformation with a documented-looking name, and the moment something does consume it, it will consume 0.1.0.

Fix: write cmd.Version (or whatever buildTags can reach without an import cycle — passing it in alongside accountID/userARN matches the existing signature). It has a producer; it should be derived from it, not restated. A test asserting tag("spawn:version") == <the injected version> is the thing that keeps it from re-rotting.

Bug 2 — spawn:completion-file is always written; spawn:on-complete only when asked

--completion-file has a non-empty default (cmd/launch_flags.go:236):

launchCmd.Flags().StringVar(&completionFile, "completion-file", "/tmp/SPAWN_COMPLETE", "File to watch for completion signal")

buildTags writes each lifecycle tag if its config field is non-empty (pkg/aws/tags.go:179-190):

if config.OnComplete != "" {      // only when --on-complete is passed
    tags = append(tags, ...{Key: "spawn:on-complete",     Value: config.OnComplete})
}
if config.CompletionFile != "" {  // ALWAYS -- the flag default is non-empty
    tags = append(tags, ...{Key: "spawn:completion-file", Value: config.CompletionFile})
}

So a launch that passes --ttl and nothing else gets:

spawn:completion-file = /tmp/SPAWN_COMPLETE
spawn:on-complete     = (absent)

That is a watch with no action attached. It reads — to an operator, and to anything that summarises tags — as "this instance has a completion path", when touching /tmp/SPAWN_COMPLETE does nothing. On the node above it cost a real detour: spawn:completion-file present sent me looking for why the completion path hadn't fired, before noticing there was no spawn:on-complete for it to fire.

The asymmetry is arbitrary rather than intended — the two tags are consecutive if blocks written identically, and only one of the two flags happens to have a default. Note that spored's own configureInstance treats on-complete="" as "empty to disable" (cmd/spored/main.go:730), so the daemon already has a name for this state; the tag set doesn't express it.

Fix, in preference order:

1. Only write spawn:completion-file when config.OnComplete != "" — the file is meaningless without an action, and the pair should be atomic.
2. Or drop the flag default to "" and have spored fall back to /tmp/SPAWN_COMPLETE when on-complete is set but no file is named. Keeps the ergonomics, moves the default to where it is actionable.
3. Or, cheapest and weakest, warn at launch when --completion-file (explicit or defaulted) is set with no --on-complete.

(1) also fixes the reverse direction for free: --completion-file explicitly passed with no --on-complete is currently silent, and is a plausible user error rather than a defaulting artefact.

Note, not a bug report: a pre-#502 role stays broken until the next launch

CreateOrGetInstanceProfile now calls ensureSporedBaselinePolicy unconditionally (pkg/aws/iam.go:427), including for pre-existing roles — the heal is right, and its comment says so. But it only runs inside a launch. Measured today on 0.100.2, hours after the fix shipped:

$ aws iam list-role-policies --role-name spawn-instance-d1029e0a
{"PolicyNames": ["spawn-custom-policy"]}
$ aws iam list-attached-role-policies --role-name spawn-instance-d1029e0a
{"AttachedPolicies": [{"PolicyName": "AmazonSSMManagedInstanceCore", ...}]}

Still no ec2:* of any kind — the same role, the same account, the same shape as when it produced TTL=0s. That is expected given where the heal lives, and the next --iam-role spawn-instance-d1029e0a launch will fix it. But there is no way to check or repair a cached role short of committing to an instance, which means the only way to find out whether your fleet can reap itself is to launch it. If a spawn doctor --iam-role NAME or an --iam-preflight ever lands, this is the case for it. Low priority; recorded because the role is only latent, not fixed.

Environment

• spawn 0.100.2, commit 8086471e597f6957388c1cd3cd54bccd516dec05, build 2026-08-18T08:05:35Z
• affected node: c8g.8xlarge, us-east-1, Amazon Linux 2023 aarch64, launched via spawn launch --ttl 4h --iam-role spawn-instance-d1029e0a
• spored v0.100.0 on the instance; spawn:version=0.1.0 in its tags
• both defects are in pkg/aws/tags.go and are independently fixable