perf(lava): WAL journaling for the LAVA database, gated on local storage#359
perf(lava): WAL journaling for the LAVA database, gated on local storage#359abrignoni wants to merge 1 commit into
Conversation
Speeds up LAVA database writes dramatically (per-row commits in the media helpers go from ~2k to ~68k commits/s in local benchmarks) by enabling WAL + synchronous=NORMAL, but only when the output path is affirmatively local storage: - scripts/storage_safety.py: network-aware journal mode selection. WAL is unsafe over network filesystems (per SQLite docs), and examiners commonly write LEAPP output to NAS/mapped drives/UNC paths. Detection covers Windows (GetDriveTypeW + UNC/extended-length \\?\ path handling), Linux (/proc/mounts fstype) and macOS (`mount` output, MNT_LOCAL flag). Undetermined paths stay on the safe default rollback journal. - lava_finalize_output() now checkpoints the WAL and switches the database back to journal_mode=DELETE before close. This guarantees the delivered _lava_artifacts.db is complete and standalone (no -wal/-shm sidecars) and remains openable from read-only/write-blocked media, which a WAL-header database is not (the LAVA viewer opens it OPEN_READONLY). - WAL take-effect verification before relaxing synchronous mode, per @JamesHabben's suggestion. Supersedes #285. Original work by @OneSixForensics; this squash adds the finalize-time checkpoint/journal reset, macOS local-disk detection (WAL previously never engaged on macOS), and narrowed exception handling to keep pylint clean. Co-Authored-By: OneSixForensics <kyle@onesixforensics.com>
|
@JamesHabben @Johann-PLW — requesting your review before this merges, since it touches the LAVA framework ( Two areas worth the most scrutiny:
No rush — holding the merge until one of you has looked. Supersedes #285 (original work by @OneSixForensics). |
Summary
Enables SQLite WAL journaling (+
synchronous=NORMAL) for_lava_artifacts.dbwhen — and only when — the output path is confirmed local storage. The media insert helpers commit per row, so media-heavy runs pay thousands of fsync'd rollback-journal commits; in a local benchmark this change moves per-row commit throughput from ~2,100/s to ~68,500/s.Supersedes #285 — original work by @OneSixForensics. This PR keeps his design (safe-by-default network detection, WAL take-effect verification per @JamesHabben) and adds three fixes found during review:
Fixes on top of #285
1. Finalize-time checkpoint + journal reset (data-completeness fix).
journal_mode=WALis persisted in the database header. Functional testing showed the finalized db from #285, copied to read-only/write-blocked media, fails to open withattempt to write a readonly database— SQLite must create a-shmfile to read a WAL database, and the LAVA viewer opens the db withOPEN_READONLY.lava_finalize_output()now runsPRAGMA wal_checkpoint(TRUNCATE)+PRAGMA journal_mode=DELETEbefore close, so the delivered.dbis complete, standalone (no-wal/-shmsidecars), back on the universal delete journal, and openable from archived or write-blocked report media.2. macOS local-disk detection. #285 detected local storage on Windows (GetDriveTypeW) and Linux (
/proc/mounts) but returned "undetermined" on macOS, so WAL never engaged there. Added detection viamountoutput using thelocaloption (the MNT_LOCAL flag); SMB/NFS/AFP volumes and unrecognized mounts stay on the safe rollback journal.3. Narrowed exception handling so the PR introduces no new pylint warnings (
--disable=C,Rclean except pre-existing lavafuncs debt).Forensic safety notes
storage_safety.pyonly ever touches the LAVA report database connection (report output, never evidence); it opens no files in write mode itself.\\?\C:\...is local,\\?\UNC\...is network (RLEAPP prefixes\\?\to all output paths on Windows).Validation
initialize_lava→ per-row commits →lava_finalize_output: WAL active during the run, sidecars removed at finalize, header back to delete mode, a.db-only copy contains all rows,PRAGMA integrity_checkok, and read-only open from a read-only directory succeeds.\\?\extended paths).pylint --disable=C,R: no new warnings vs main.Supersedes #285 — original work by @OneSixForensics.
🤖 Generated with Claude Code