feat: event stream promotion gates — record-framed commit log with SHA-384 integrity - #6
Merged
Merged
Conversation
… gates Replace raw mmap append log with a record-framed commit log passing all 9 promotion gates from TGMap's native analytics storage boundary: - 88-byte record headers (magic, format version, flags, schema_id, monotonic stream_offset, payload_size, prev_record_offset, SHA-384 event_id) - Explicit durability modes (NONE / FDATASYNC / FULL) with checked fdatasync - Two-phase commit: write uncommitted header+payload, fdatasync, flip committed flag, fdatasync - Restart replay via callback, torn-tail recovery via ftruncate - Corruption detection (bad magic/version/offset/payload_size) - Duplicate event-ID rejection (SHA-384 of topic||payload) - O_NOFOLLOW symlink protection, 0600 owner-only permissions - topic_is_safe() rejects path traversal (.. and /) - flock(LOCK_EX) serializes concurrent writers, LOCK_SH for readers - Non-network local iteration by offset and cursor - SHA-384 integrity via OpenSSL EVP (compatible with TGMap evidence envelope) - Zero-copy sendfile consumption requires explicit fd (no implicit network) API additions: - qihse_event_stream_open() with durability + read-only mode - qihse_event_stream_append_record() with schema_id + event_id - qihse_event_stream_read() / iterate() / length() / has_event_id() - qihse_event_stream_replay() with callback - qihse_event_stream_truncate_torn_tail() - qihse_event_stream_flush() Tests: 16 test cases, 49 assertions, all passing. Covers record framing, owner-only paths, O_NOFOLLOW, durability modes, replay, torn-tail recovery, corruption detection, iteration, duplicate rejection, SHA-384 integrity, read-only permission, concurrent writers, symlink rejection, no implicit network, and restart recovery. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Upgrades the QIHSE event stream from a 142-line raw mmap append log to a record-framed commit log passing all 9 promotion gates from TGMap's native analytics storage boundary (
docs/NATIVE_ANALYTICS_STORAGE.md).Record framing
qihse_es_record_header_t: magic, format version, flags, schema_id, monotonic stream_offset, payload_size, prev_record_offset, SHA-384 event_idDurability
qihse_es_durability_tenum: NONE / FDATASYNC / FULLfdatasync→ flip committed flag →fdatasyncRecovery
qihse_event_stream_replay()walks committed records via callbackqihse_event_stream_truncate_torn_tail()removes uncommitted tail viaftruncatevalidate_header()detects bad magic, version, offset, payload_sizeSecurity
O_NOFOLLOWsymlink protection,0600owner-only permissionstopic_is_safe()rejects path traversal (..and/)flock(LOCK_EX)serializes concurrent writers,LOCK_SHfor readersIntegrity
SHA-384(topic || payload)API surface
qihse_event_stream_open()with explicit durability + read-only modeqihse_event_stream_append_record()with schema_id + event_idqihse_event_stream_read()/iterate()/length()/has_event_id()qihse_event_stream_replay()with callbackqihse_event_stream_truncate_torn_tail()qihse_event_stream_flush()qihse_event_stream_append()preserved (computes event_id internally)Test plan
test_record_framing— length-delimited, schema, monotonic offsettest_owner_only_paths— file permissions are 0600test_symlink_rejection— O_NOFOLLOW rejects symlink topicstest_durability_modes— fdatasync persists across restarttest_replay— all committed records replayed in ordertest_torn_tail_recovery— uncommitted tail truncatedtest_corruption_detection— bad magic stops replaytest_iteration— iterate all records in ordertest_duplicate_rejection— second append with same event ID failstest_sha384_integrity— event_id matches SHA-384(topic||payload)test_permission_denied— read-only mode rejects writestest_concurrent_writers— flock serializes appendstest_no_implicit_network— consume_zero_copy requires explicit fdtest_restart_recovery— all records survive close/reopenResults: 49 assertions passed, 0 failed, 16 test cases total.
Build:
gcc -std=c99 -Wall -Wextra -O2 -I./include -D_GNU_SOURCE tests/qihse_event_stream_test.c src/marmalade/qihse_event_stream.c -o test_event_stream -lcrypto && ./test_event_streamGenerated with Devin