Skip to content

Move history persistence's fsync off the collector's tick goroutine - #119

Merged
LarsLaskowski merged 1 commit into
mainfrom
claude/issue-110-6h3cdk
Aug 21, 2026
Merged

Move history persistence's fsync off the collector's tick goroutine#119
LarsLaskowski merged 1 commit into
mainfrom
claude/issue-110-6h3cdk

Conversation

@LarsLaskowski

@LarsLaskowski LarsLaskowski commented Aug 21, 2026

Copy link
Copy Markdown
Owner

📖 Description

persistHistory() ran synchronously inside Run's select loop: encoding
and atomically writing (with fsync) the history file blocked fastTick
from running for the whole duration of every write — once per slowTick
and once more on shutdown. time.Ticker drops rather than queues missed
ticks, so a slow write (e.g. an SD-card fsync stall under
updates_check_minutes-driven writes, or while the root-privileged apt
timer is doing I/O) silently dropped fast-tick samples instead of merely
delaying them.

The snapshot (c.History()) is still taken on the caller's goroutine so it
stays a consistent point-in-time view, but the encode-and-write now runs in
a background goroutine tracked by a new Collector.persistWG
(sync.WaitGroup). A buffered try-lock (Collector.persisting, capacity 1)
skips — rather than queues — an overlapping flush while a previous write is
still in flight, since the next flush writes newer data anyway. Run's
ctx.Done() branch now calls persistWG.Wait() after the final
persistHistory(), so the last flush is still guaranteed to land before
Run returns; this remains bounded by cmd/pimonitor/main.go's existing
10s shutdown context (collDone vs. shutdownCtx.Done()), so a stuck
fsync still cannot hang the process.

Collector.writeFile is now a field (defaulting to writeFileAtomic) so
tests can substitute a controllable write function without touching real
disk I/O.

Not folded in (per the issue): fsyncing the parent directory after
os.Rename for full crash durability — a separate, minor concern.

🎫 Issues

Closes #110

👩‍💻 Reviewer Notes

Focus areas: the try-lock in persistHistory (internal/collector/persist.go)
and the shutdown join in Run (internal/collector/collector.go). Verified
only against fixtures/synthetic slow writes — no real Pi/SD-card hardware
was used to reproduce the fsync stall itself, per the issue's own caveat
that the stall durations are general platform knowledge rather than
measurements from this project.

📑 Test Plan

Added to internal/collector/persist_test.go, all driven by channels (no
time.Sleep-based synchronization) and run under -race:

  • TestCollector_PersistHistory_DoesNotBlockOnSlowWritepersistHistory()
    returns promptly while the write is still blocked; the data still lands
    once the write completes and is decoded back.
  • TestCollector_PersistHistory_SkipsOverlappingWrites — two flushes
    triggered while a write is in flight are skipped, not queued (write
    function invoked exactly once).
  • TestCollector_Run_ShutdownWaitsForFinalFlushRun blocks until the
    in-flight final flush completes before returning, and the flushed data is
    present afterwards.
  • Updated TestCollector_PersistAndLoadHistory and
    TestCollector_PersistHistory_OverwritesExistingFile to persistWG.Wait()
    after each persistHistory() call, since writes are no longer synchronous.

go build ./..., go vet ./..., go test ./... -race -cover, and
golangci-lint run all pass locally.

✅ Checklist

General

  • I have added/updated tests for my changes (go test ./... -race -cover passes locally).
  • go vet ./... and golangci-lint run are clean.
  • I have tested my changes.
  • I have read the CONTRIBUTING documentation and followed the project's code style guidelines.
  • I have updated ARCHITECTURE.md if this changes a documented design decision.

REST API / configuration / packaging

Not applicable — no API, config, or packaging change.

⏭ Next Steps

The issue separately notes writeFileAtomic doesn't fsync the parent
directory after os.Rename, a minor durability gap on power loss
immediately after rename. Left out of this PR per the issue's own guidance,
as a follow-up.

persistHistory() ran synchronously inside Run's select loop: encoding and
atomically writing (with fsync) the history file blocked fastTick from
running on every slowTick and on shutdown. time.Ticker drops rather than
queues missed ticks, so a slow write (e.g. an SD-card fsync stall) silently
dropped fast-tick samples.

The snapshot is still taken on the caller's goroutine for a consistent
point-in-time view, but the encode-and-write now runs in a background
goroutine tracked by a new persistWG. A buffered try-lock (persisting) skips
an overlapping flush rather than queuing it, since the next flush writes
newer data anyway. Run's ctx.Done() branch now waits on persistWG before
returning, so the final flush is still guaranteed to land before shutdown
completes (bounded by main.go's existing 10s shutdown context).

Closes #110
@sonarqubecloud

Copy link
Copy Markdown

@LarsLaskowski
LarsLaskowski merged commit 51dbc85 into main Aug 21, 2026
5 checks passed
@LarsLaskowski
LarsLaskowski deleted the claude/issue-110-6h3cdk branch August 21, 2026 20:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

History persistence blocks the collector tick: encode + fsync run inline in Run()

2 participants