Move history persistence's fsync off the collector's tick goroutine - #119
Merged
Conversation
persistHistory() ran synchronously inside Run's select loop: encoding and atomically writing (with fsync) the history file blocked fastTick from running on every slowTick and on shutdown. time.Ticker drops rather than queues missed ticks, so a slow write (e.g. an SD-card fsync stall) silently dropped fast-tick samples. The snapshot is still taken on the caller's goroutine for a consistent point-in-time view, but the encode-and-write now runs in a background goroutine tracked by a new persistWG. A buffered try-lock (persisting) skips an overlapping flush rather than queuing it, since the next flush writes newer data anyway. Run's ctx.Done() branch now waits on persistWG before returning, so the final flush is still guaranteed to land before shutdown completes (bounded by main.go's existing 10s shutdown context). Closes #110
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



📖 Description
persistHistory()ran synchronously insideRun'sselectloop: encodingand atomically writing (with
fsync) the history file blockedfastTickfrom running for the whole duration of every write — once per
slowTickand once more on shutdown.
time.Tickerdrops rather than queues missedticks, so a slow write (e.g. an SD-card
fsyncstall underupdates_check_minutes-driven writes, or while the root-privileged apttimer is doing I/O) silently dropped fast-tick samples instead of merely
delaying them.
The snapshot (
c.History()) is still taken on the caller's goroutine so itstays a consistent point-in-time view, but the encode-and-write now runs in
a background goroutine tracked by a new
Collector.persistWG(
sync.WaitGroup). A buffered try-lock (Collector.persisting, capacity 1)skips — rather than queues — an overlapping flush while a previous write is
still in flight, since the next flush writes newer data anyway.
Run'sctx.Done()branch now callspersistWG.Wait()after the finalpersistHistory(), so the last flush is still guaranteed to land beforeRunreturns; this remains bounded bycmd/pimonitor/main.go's existing10s shutdown context (
collDonevs.shutdownCtx.Done()), so a stuckfsyncstill cannot hang the process.Collector.writeFileis now a field (defaulting towriteFileAtomic) sotests can substitute a controllable write function without touching real
disk I/O.
Not folded in (per the issue): fsyncing the parent directory after
os.Renamefor full crash durability — a separate, minor concern.🎫 Issues
Closes #110
👩💻 Reviewer Notes
Focus areas: the try-lock in
persistHistory(internal/collector/persist.go)and the shutdown join in
Run(internal/collector/collector.go). Verifiedonly against fixtures/synthetic slow writes — no real Pi/SD-card hardware
was used to reproduce the fsync stall itself, per the issue's own caveat
that the stall durations are general platform knowledge rather than
measurements from this project.
📑 Test Plan
Added to
internal/collector/persist_test.go, all driven by channels (notime.Sleep-based synchronization) and run under-race:TestCollector_PersistHistory_DoesNotBlockOnSlowWrite—persistHistory()returns promptly while the write is still blocked; the data still lands
once the write completes and is decoded back.
TestCollector_PersistHistory_SkipsOverlappingWrites— two flushestriggered while a write is in flight are skipped, not queued (write
function invoked exactly once).
TestCollector_Run_ShutdownWaitsForFinalFlush—Runblocks until thein-flight final flush completes before returning, and the flushed data is
present afterwards.
TestCollector_PersistAndLoadHistoryandTestCollector_PersistHistory_OverwritesExistingFiletopersistWG.Wait()after each
persistHistory()call, since writes are no longer synchronous.go build ./...,go vet ./...,go test ./... -race -cover, andgolangci-lint runall pass locally.✅ Checklist
General
go test ./... -race -coverpasses locally).go vet ./...andgolangci-lint runare clean.ARCHITECTURE.mdif this changes a documented design decision.REST API / configuration / packaging
Not applicable — no API, config, or packaging change.
⏭ Next Steps
The issue separately notes
writeFileAtomicdoesn'tfsyncthe parentdirectory after
os.Rename, a minor durability gap on power lossimmediately after rename. Left out of this PR per the issue's own guidance,
as a follow-up.