Expand SQLite3 data validation

[PimSanders]

This PR close #16 by expanding the data validation capabilities in SQLite3.

The SQLite3 WAL file can store multiple versions of the same frame, when reading only valid frames should be returned. The docs define a valid frame as follows:

A frame is considered valid if and only if the following conditions are true:

1. The salt-1 and salt-2 values in the frame-header match salt values in the wal-header
2. The checksum values in the final 8 bytes of the frame-header exactly match the checksum computed consecutively on the first 24 bytes of the WAL header and the first 8 bytes and the content of all frames up to and including the current frame.

The first check was already implemented, I have interpreted the second check as:

The checksum values in the final 8 bytes of the frame-header (checksum-1 and checksum-2) exactly match the computed checksum over:

1. the first 24 bytes of the WAL header
2. the first 8 bytes of each frame header (up to and including this frame)
3. the page data of each frame (up to and including this frame)

When initializing a database the option validate_checksum can be passed to use the new validation. I have chosen to only calculate the salts by default (just like before) as this will probably be good enough, and a lot faster. See the example below for the time impact:

In [1]: %timeit -n10 list(list(sqlite3.SQLite3(Path("./big.sqlite"), Path("./big.sqlite-wal"), validate_checksum=False).tables())[0].rows())
33 ms ± 1.06 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

In [2]: %timeit -n10 list(list(sqlite3.SQLite3(Path("./big.sqlite"), Path("./big.sqlite-wal"), validate_checksum=True).tables())[0].rows())
1.05 s ± 4.88 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

[codecov]

Codecov Report

❌ Patch coverage is 0% with 48 lines in your changes missing coverage. Please review.
✅ Project coverage is 0.00%. Comparing base (62f6301) to head (ec7b77e).

Files with missing lines	Patch %	Lines
dissect/database/sqlite3/wal.py	0.00%	46 Missing ⚠️
dissect/database/sqlite3/sqlite3.py	0.00%	2 Missing ⚠️

Additional details and impacted files

@@          Coverage Diff          @@
##            main     #23   +/-   ##
=====================================
  Coverage   0.00%   0.00%           
=====================================
  Files        152     152           
  Lines       4716    4756   +40     
=====================================
- Misses      4716    4756   +40     

Flag	Coverage Δ
unittests	0.00% <0.00%> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[codspeed-hq]

Merging this PR will not alter performance

🎉 Hooray! pytest-codspeed just leveled up to 5.0.3!

A heads-up, this is a breaking change and it might affect your current performance baseline a bit. But here's the exciting part - it's packed with new, cool features and promises improved result stability 🥳!
Curious about what's new? Visit our releases page to delve into all the awesome details about this new version.

✅ 6 untouched benchmarks
🆕 2 new benchmarks

Performance Changes

	Benchmark	BASE	HEAD	Efficiency
🆕	test_benchmark_wal_checksum_validation[False]	N/A	153.7 ms	N/A
🆕	test_benchmark_wal_checksum_validation[True]	N/A	136.1 ms	N/A

---

_{Comparing PimSanders:improvement/expand-wal-validation (ec7b77e) with main (62f6301)}

[Schamper]

Maybe add a benchmark test too? I'll look at the actual checksum checking part later when I have a bit more time.

[PimSanders]

Take your time, I don't think I will be doing a whole lot of Dissect dev to coming weeks ...

[Schamper]

Can you add a benchmark test too, so that we can track future changes to this algorithm?

[Schamper]

LGTM! The codspeed comment suggests that with validation is actually faster than without. Any idea why? Perhaps we should enable validation by default? 😂

[PimSanders]

Yeah uuuh not sure why that is the case now. Enabled it by default, it did need to be disabled for some tests.

[Schamper]

Claude found this:

diff --git i/dissect/database/sqlite3/wal.py w/dissect/database/sqlite3/wal.py
index 04a538a..0a79899 100644
--- i/dissect/database/sqlite3/wal.py
+++ w/dissect/database/sqlite3/wal.py
@@ -39,7 +39,8 @@ class WAL:
             raise InvalidDatabase("Invalid WAL header magic")
 
         self.checksum_endian = "<" if self.header.magic == WAL_HEADER_MAGIC_LE else ">"
-        self._checksum_struct = struct.Struct(f"{self.checksum_endian}2I")
+        # Checksum values are always stored in big-endian format
+        self._checksum_struct = struct.Struct(">2I")
 
         self.frame = lru_cache(1024)(self.frame)
         self.frame_size = len(c_sqlite3.wal_frame) + self.header.page_size
@@ -107,8 +108,8 @@ class WAL:
             if len(frame_hdr_bytes) < len(c_sqlite3.wal_frame):
                 raise EOFError("Incomplete frame header while calculating checksum")
 
-            # Checksum first 16 bytes of frame header
-            seed = calculate_checksum(frame_hdr_bytes[:16], seed=seed, endian=self.checksum_endian)
+            # Checksum first 8 bytes of frame header (page number and page count)
+            seed = calculate_checksum(frame_hdr_bytes[:8], seed=seed, endian=self.checksum_endian)
 
             # Read and checksum page data
             page_data = self.fh.read(self.header.page_size)
diff --git i/tests/sqlite3/test_wal.py w/tests/sqlite3/test_wal.py
index 8c71f71..67a7559 100644
--- i/tests/sqlite3/test_wal.py
+++ w/tests/sqlite3/test_wal.py
@@ -1,5 +1,6 @@
 from __future__ import annotations
 
+import io
 from typing import TYPE_CHECKING
 
 import pytest
@@ -59,26 +60,48 @@ def test_sqlite_wal_checkpoint(sqlite_db: Path, sqlite_wal: Path, db_as_path: bo
     [pytest.param(True, id="wal_as_path"), pytest.param(False, id="wal_as_fh")],
 )
 def test_sqlite_wal_checksum_validation(sqlite_db: Path, sqlite_wal: Path, db_as_path: bool, wal_as_path: bool) -> None:
-    # Test that the WAL checksum validation works as expected
-    # When validate_checksums=True, only entries before the last checkpoint are visible
+    # With an intact WAL, checksum validation should not change the result:
+    # both modes show the live database state, matching real SQLite behaviour.
     db = sqlite3.SQLite3(
         sqlite_db if db_as_path else sqlite_db.open("rb"),
         sqlite_wal if wal_as_path else sqlite_wal.open("rb"),
         validate_checksums=True,
     )
 
-    _assert_valid_checksum(db)
+    _assert_live_state(db)
 
     db.close()
 
-    # When validate_checksums=False, entries after the last checkpoint are also visible
     db = sqlite3.SQLite3(
         sqlite_db if db_as_path else sqlite_db.open("rb"),
         sqlite_wal if wal_as_path else sqlite_wal.open("rb"),
         validate_checksums=False,
     )
 
-    _assert_invalid_checksum(db)
+    _assert_live_state(db)
+
+    db.close()
+
+
+def test_sqlite_wal_checksum_validation_corrupt(sqlite_db: Path, sqlite_wal: Path) -> None:
+    # Corrupt the stored checksum of the first frame of the current WAL generation.
+    # The WAL header is 32 bytes and a frame header is 24 bytes, with checksum1 at frame offset 16.
+    wal_data = bytearray(sqlite_wal.read_bytes())
+    wal_data[32 + 16] ^= 0xFF
+
+    # With validation, the corrupted frame and all frames after it are rejected,
+    # so the database reflects the state at the last checkpoint.
+    db = sqlite3.SQLite3(sqlite_db, io.BytesIO(bytes(wal_data)), validate_checksums=True)
+
+    _assert_checkpoint_state(db)
+
+    db.close()
+
+    # Without validation, the corrupted frames are still applied (salts match),
+    # so the post-checkpoint delete and update are visible.
+    db = sqlite3.SQLite3(sqlite_db, io.BytesIO(bytes(wal_data)), validate_checksums=False)
+
+    _assert_live_state(db)
 
     db.close()
 
@@ -202,8 +225,8 @@ def _assert_checkpoint_3(s: sqlite3.SQLite3) -> None:
 
 
 # Assertion functions for test_sqlite_wal_checksum_validation()
-def _assert_valid_checksum(s: sqlite3.SQLite3) -> None:
-    # If the checksum validation is correct, all entries BEFORE the last checkpoint should be present
+def _assert_checkpoint_state(s: sqlite3.SQLite3) -> None:
+    # State as of the last checkpoint: the post-checkpoint delete and update are not applied
     table = next(iter(s.tables()))
     rows = list(table.rows())
 
@@ -244,8 +267,8 @@ def _assert_valid_checksum(s: sqlite3.SQLite3) -> None:
     assert rows[10].value == 101
 
 
-def _assert_invalid_checksum(s: sqlite3.SQLite3) -> None:
-    # If the checksum validation is incorrect, all entries AFTER the last checkpoint should be present
+def _assert_live_state(s: sqlite3.SQLite3) -> None:
+    # Live database state: the post-checkpoint delete and update are applied
     table = next(iter(s.tables()))
     rows = list(table.rows())

[Schamper]

I was in a bit of a rush but basically it concluded that the current code considered every frame invalid, skipping all of them. Which is of course quite fast.