Skip to content

perf(lava): WAL journaling for the LAVA database, gated on local storage#359

Open
abrignoni wants to merge 1 commit into
mainfrom
lava-wal-perf-update
Open

perf(lava): WAL journaling for the LAVA database, gated on local storage#359
abrignoni wants to merge 1 commit into
mainfrom
lava-wal-perf-update

Conversation

@abrignoni

Copy link
Copy Markdown
Owner

Summary

Enables SQLite WAL journaling (+ synchronous=NORMAL) for _lava_artifacts.db when — and only when — the output path is confirmed local storage. The media insert helpers commit per row, so media-heavy runs pay thousands of fsync'd rollback-journal commits; in a local benchmark this change moves per-row commit throughput from ~2,100/s to ~68,500/s.

Supersedes #285 — original work by @OneSixForensics. This PR keeps his design (safe-by-default network detection, WAL take-effect verification per @JamesHabben) and adds three fixes found during review:

Fixes on top of #285

1. Finalize-time checkpoint + journal reset (data-completeness fix). journal_mode=WAL is persisted in the database header. Functional testing showed the finalized db from #285, copied to read-only/write-blocked media, fails to open with attempt to write a readonly database — SQLite must create a -shm file to read a WAL database, and the LAVA viewer opens the db with OPEN_READONLY. lava_finalize_output() now runs PRAGMA wal_checkpoint(TRUNCATE) + PRAGMA journal_mode=DELETE before close, so the delivered .db is complete, standalone (no -wal/-shm sidecars), back on the universal delete journal, and openable from archived or write-blocked report media.

2. macOS local-disk detection. #285 detected local storage on Windows (GetDriveTypeW) and Linux (/proc/mounts) but returned "undetermined" on macOS, so WAL never engaged there. Added detection via mount output using the local option (the MNT_LOCAL flag); SMB/NFS/AFP volumes and unrecognized mounts stay on the safe rollback journal.

3. Narrowed exception handling so the PR introduces no new pylint warnings (--disable=C,R clean except pre-existing lavafuncs debt).

Forensic safety notes

  • storage_safety.py only ever touches the LAVA report database connection (report output, never evidence); it opens no files in write mode itself.
  • Network/undetermined paths keep SQLite's default rollback journal — WAL is never enabled on NAS/UNC/mapped-drive output, per SQLite's documented WAL limitation.
  • Windows extended-length paths are handled: \\?\C:\... is local, \\?\UNC\... is network (RLEAPP prefixes \\?\ to all output paths on Windows).

Validation

  • End-to-end test driving initialize_lava → per-row commits → lava_finalize_output: WAL active during the run, sidecars removed at finalize, header back to delete mode, a .db-only copy contains all rows, PRAGMA integrity_check ok, and read-only open from a read-only directory succeeds.
  • Detection unit-tested with injected mount tables (macOS smb/apfs/autofs, Linux nfs4/ext4, UNC and \\?\ extended paths).
  • pylint --disable=C,R: no new warnings vs main.
  • PluginLoader smoke test: 316 plugins on this branch, identical to main.

Supersedes #285 — original work by @OneSixForensics.

🤖 Generated with Claude Code

Speeds up LAVA database writes dramatically (per-row commits in the media
helpers go from ~2k to ~68k commits/s in local benchmarks) by enabling
WAL + synchronous=NORMAL, but only when the output path is affirmatively
local storage:

- scripts/storage_safety.py: network-aware journal mode selection. WAL is
  unsafe over network filesystems (per SQLite docs), and examiners commonly
  write LEAPP output to NAS/mapped drives/UNC paths. Detection covers
  Windows (GetDriveTypeW + UNC/extended-length \\?\ path handling), Linux
  (/proc/mounts fstype) and macOS (`mount` output, MNT_LOCAL flag).
  Undetermined paths stay on the safe default rollback journal.

- lava_finalize_output() now checkpoints the WAL and switches the database
  back to journal_mode=DELETE before close. This guarantees the delivered
  _lava_artifacts.db is complete and standalone (no -wal/-shm sidecars)
  and remains openable from read-only/write-blocked media, which a
  WAL-header database is not (the LAVA viewer opens it OPEN_READONLY).

- WAL take-effect verification before relaxing synchronous mode, per
  @JamesHabben's suggestion.

Supersedes #285. Original work by @OneSixForensics; this squash adds the
finalize-time checkpoint/journal reset, macOS local-disk detection (WAL
previously never engaged on macOS), and narrowed exception handling to
keep pylint clean.

Co-Authored-By: OneSixForensics <kyle@onesixforensics.com>
@abrignoni

Copy link
Copy Markdown
Owner Author

@JamesHabben @Johann-PLW — requesting your review before this merges, since it touches the LAVA framework (lavafuncs.py) and the delivered report DB.

Two areas worth the most scrutiny:

  1. lava_finalize_output()finalize_lava_journal_mode (the forensic-completeness part). During a run the DB is in WAL, so data lives in -wal/-shm sidecars; at finalize we PRAGMA wal_checkpoint(TRUNCATE) + PRAGMA journal_mode=DELETE so the delivered .db is a single standalone file with no sidecars, and opens read-only from write-blocked/archived media. Without this, copying the .db alone could silently drop uncheckpointed rows, or fail to open read-only (attempt to write a readonly database). I verified end-to-end: 500 per-row commits under WAL, finalize back to delete mode, sidecars gone, .db-only copy in a read-only dir opens with all rows and integrity_check=ok.

  2. Local-vs-network detection in storage_safety.py. WAL only engages on confirmed local storage; SMB/NFS/AFP/UNC/mapped drives and any undetermined path stay on the default rollback journal (SQLite's documented WAL-over-network limitation). Detection covers Windows (GetDriveTypeW, incl. \\?\ extended paths), Linux (/proc/mounts), and macOS (mount local / MNT_LOCAL). Please sanity-check the platform edge cases against setups you run.

No rush — holding the merge until one of you has looked. Supersedes #285 (original work by @OneSixForensics).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant