diff --git a/CHANGELOG.md b/CHANGELOG.md index 9a6659a..ed9a9d1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,57 @@ All notable changes to mobileGF2logger are documented here. ## Unreleased +## 2.1.2 - 2026-08-09 + +### Changed + +- Re-center the launcher emblem against the real source canvas, reduce it by + 10%, and brighten the matte silver and gold-orange finish without redrawing + its silhouette or introducing resampling artifacts. +- Make the English project summary more descriptive for people searching for a + GIRLS' FRONTLINE 2 or GF2 Platoon management companion. + +### Fixed + +- Bound parsed-packet storage and table rendering by input size, row count, + column count, cell count, and cell length so hostile or corrupt payloads + cannot create unbounded Android view trees. +- Bound multi-part protocol continuations by byte and fragment counts, discard + an oversized pending dataset, quarantine every remaining fragment through its + terminal frame before decoding it, and enforce the byte cap from the first + continuation. A malformed terminal fragment now also clears quarantine so a + later legitimate payload of the same type is not discarded. +- Bound roster imports, per-payload activity observations, retained activity + history, and unresolved-name reconciliation. Failed or duplicate roster + imports now roll back their newly retained evidence as one operation. +- Apply the retained-activity cap after both activity packets and Updates- + derived Daily Patrol facts, without waiting for a database reopen. +- Neutralize spreadsheet formulas in every captured roster, activity, Updates, + and formation packet table as well as all spreadsheet-facing CSV exports, + while preserving internal roster evidence losslessly. +- Upgrade schema-v6/v7 databases directly through the v10 activity-table + rebuild without attempting to recreate already-installed identity indexes. +- Reconcile an observed roster absence against an exact or annotated open + membership period, preserving its strong boundary and note while creating a + later rejoin period instead of suppressing the gap. Annotated weak boundaries + can still expand when late roster evidence proves a wider presence interval. +- Preserve an imported roster's parsed capture time so exporting the latest + Platoon CSV cannot accidentally select a newly imported historical file. +- Roll back all nested SQLite reconciliation writes together with newly + retained roster files when a multi-file import fails. +- Rotate bounded unresolved-activity reconciliation through the complete + retained backlog so an unmatchable recent batch cannot starve older evidence, + backed by a global capture-time retention index. +- Enforce roster member-count and name-length limits at live-capture writing, + repository, and SQLite boundaries as well as during user-selected CSV import. +- Version the new retention index and maintenance cursor as schema v11 so v10 + backups migrate through the same strict current-schema validation path. +- Break equal capture-time ties by retained evidence filename when selecting + the latest Platoon CSV for export. +- Accept a capped pre-Gunsmoke opening anchor only when it belongs to the + immediately preceding Monday-through-Saturday counter period, using the + configured game-day timezone across daylight-saving transitions. + ## 2.1.1 - 2026-08-08 ### Added diff --git a/README.md b/README.md index 7abfc5d..2832897 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,4 @@ -# mobileGF2logger +# mobileGF2logger — GIRLS' FRONTLINE 2 (GF2) Platoon Manager [![Android](https://github.com/1window2/mobileGF2logger/actions/workflows/android.yml/badge.svg)](https://github.com/1window2/mobileGF2logger/actions/workflows/android.yml) [![CodeQL](https://github.com/1window2/mobileGF2logger/actions/workflows/codeql.yml/badge.svg)](https://github.com/1window2/mobileGF2logger/actions/workflows/codeql.yml) @@ -6,9 +6,12 @@ 한국어로 된 설명은 [여기](README_KR.md)에서 보실 수 있습니다. -This is a **Platoon(서클)** management tool for the Android client of [GIRLS' FRONTLINE 2: EXILIUM](https://gf2exilium.sunborngame.com/). It is Android-only; an iOS release is not planned. +This is a **GIRLS' FRONTLINE 2: EXILIUM (GF2) Platoon/서클 management and logging companion** for the [Android game client](https://gf2exilium.sunborngame.com/). It is Android-only; an iOS release is not planned. -mobileGF2logger is a lightweight, non-root app for Platoon masters. It uses Android's per-app VPN permission to parse supported plaintext server responses entirely on the phone without storing raw traffic. +mobileGF2logger is a lightweight, non-root GF2 logger for Platoon masters. It +turns supported roster and activity responses into member history and weekly +task tables entirely on the phone. Android's per-app VPN permission limits +capture to the selected game package, and raw traffic is never stored. ## Features diff --git a/README_KR.md b/README_KR.md index 1d1daa5..b3f09ce 100644 --- a/README_KR.md +++ b/README_KR.md @@ -1,10 +1,10 @@ -# mobileGF2logger +# mobileGF2logger — 소녀전선 2(GF2) 서클 관리 도구 [![Android](https://github.com/1window2/mobileGF2logger/actions/workflows/android.yml/badge.svg)](https://github.com/1window2/mobileGF2logger/actions/workflows/android.yml) [![CodeQL](https://github.com/1window2/mobileGF2logger/actions/workflows/codeql.yml/badge.svg)](https://github.com/1window2/mobileGF2logger/actions/workflows/codeql.yml) [![Dependabot](https://img.shields.io/github/issues-pr/1window2/mobileGF2logger/dependencies?label=Dependabot&logo=dependabot)](https://github.com/1window2/mobileGF2logger/pulls?q=is%3Apr+is%3Aopen+author%3Aapp%2Fdependabot) -이 프로젝트는 [GIRLS' FRONTLINE 2: EXILIUM](https://gf2exilium.sunborngame.com/) Android 클라이언트용 **서클(Platoon)** 관리 도구입니다. Android만 지원하며 iOS 버전은 출시할 계획이 없습니다. +이 프로젝트는 [GIRLS' FRONTLINE 2: EXILIUM](https://gf2exilium.sunborngame.com/) (소녀전선 2, GF2) Android 클라이언트용 **서클(Platoon) 관리 및 기록 도구**입니다. Android만 지원하며 iOS 버전은 출시할 계획이 없습니다. mobileGF2logger는 서클장을 위한 가벼운 비루팅 앱입니다. Android의 앱별 VPN 권한을 사용하여 지원되는 평문 서버 응답을 휴대전화 안에서 파싱하며, 원본 트래픽은 저장하지 않습니다. diff --git a/app/build.gradle b/app/build.gradle index 68426d4..10bf74c 100644 --- a/app/build.gradle +++ b/app/build.gradle @@ -154,8 +154,8 @@ android { applicationId = 'dev.gf2log' minSdk = 26 targetSdk = 36 - versionCode = 20101 - versionName = '2.1.1' + versionCode = 20102 + versionName = '2.1.2' testInstrumentationRunner = 'androidx.test.runner.AndroidJUnitRunner' ndk { diff --git a/app/src/androidTest/java/dev/gf2log/app/management/PlatoonBackupManagerIntegrationTest.kt b/app/src/androidTest/java/dev/gf2log/app/management/PlatoonBackupManagerIntegrationTest.kt index 838a8f0..63e2be8 100644 --- a/app/src/androidTest/java/dev/gf2log/app/management/PlatoonBackupManagerIntegrationTest.kt +++ b/app/src/androidTest/java/dev/gf2log/app/management/PlatoonBackupManagerIntegrationTest.kt @@ -2,6 +2,7 @@ package dev.gf2log.app.management import android.content.ContentValues import android.content.Context +import android.database.sqlite.SQLiteDatabase import androidx.test.core.app.ApplicationProvider import androidx.test.ext.junit.runners.AndroidJUnit4 import dev.gf2log.app.TargetPackagePreferences @@ -80,6 +81,48 @@ class PlatoonBackupManagerIntegrationTest { } } + @Test + fun schemaTenCompleteBackupMigratesBeforeCurrentContractValidation() { + seedDatabase(ARCHIVED_UID, "Archived member", "archived-source.csv") + settingsStore.replace(archivedSettings()) + val databaseFile = context.getDatabasePath(PlatoonSchema.DATABASE_NAME) + SQLiteDatabase.openDatabase(databaseFile.path, null, SQLiteDatabase.OPEN_READWRITE).use { legacy -> + legacy.execSQL("DROP INDEX platoon_activity_resolution_retention") + legacy.execSQL("DROP INDEX platoon_activity_retention_order") + legacy.execSQL("DROP TABLE platoon_maintenance_state") + legacy.version = 10 + } + val archive = ByteArrayOutputStream().also { output -> + BackupArchive.write( + output, + databaseFile, + AppBackupSettingsCodec.encode(archivedSettings()), + ) + }.toByteArray() + + replaceDatabaseWithCurrentState() + settingsStore.replace(currentSettings()) + PlatoonBackupManager(context).restoreFull(ByteArrayInputStream(archive)) + + assertEquals(archivedSettings(), settingsStore.read()) + PlatoonDatabase(context).use { database -> + val restored = database.readableDatabase + assertEquals(PlatoonSchema.CURRENT_VERSION, restored.version) + assertEquals(1L, count(restored, "members", "uid = ?", ARCHIVED_UID)) + assertEquals( + 1L, + restored.rawQuery( + "SELECT COUNT(*) FROM sqlite_master " + + "WHERE type = 'table' AND name = 'platoon_maintenance_state'", + null, + ).use { cursor -> + assertTrue(cursor.moveToFirst()) + cursor.getLong(0) + }, + ) + } + } + @Test fun platoonOnlyRestoreRetiresUnrelatedRetainedRosterEvidence() { seedDatabase(ARCHIVED_UID, "Archived member", "archived-source.csv") diff --git a/app/src/androidTest/java/dev/gf2log/app/management/PlatoonDatabaseIntegrationTest.kt b/app/src/androidTest/java/dev/gf2log/app/management/PlatoonDatabaseIntegrationTest.kt index 8c10d26..d2821c7 100644 --- a/app/src/androidTest/java/dev/gf2log/app/management/PlatoonDatabaseIntegrationTest.kt +++ b/app/src/androidTest/java/dev/gf2log/app/management/PlatoonDatabaseIntegrationTest.kt @@ -276,6 +276,297 @@ class PlatoonDatabaseIntegrationTest { assertForeignKeysValid() } + @Test + fun historicalRosterReplayClosesAnExactOpenPeriodAndKeepsTheLaterRejoin() { + val currentTime = Instant.parse("2026-07-31T00:00:00Z") + val exactJoin = currentTime.minusSeconds(5 * 86_400) + database.ingestSnapshot( + PlatoonSnapshot( + id = 0, + capturedAt = currentTime, + sourceFile = "current-gap.csv", + members = listOf(member(OTHER_UID, "Current")), + ), + EvidenceSource.SNAPSHOT, + ) + database.ingestPlatoonUpdates( + listOf(update(PlatoonUpdateSemantics.KIND_JOIN, exactJoin, TARGET_UID, "Historical")), + exactJoin.plusSeconds(60), + ) + + listOf( + currentTime.minusSeconds(4 * 86_400) to + listOf(member(TARGET_UID, "Historical"), member(OTHER_UID, "Current")), + currentTime.minusSeconds(3 * 86_400) to listOf(member(OTHER_UID, "Current")), + currentTime.minusSeconds(2 * 86_400) to + listOf(member(TARGET_UID, "Historical"), member(OTHER_UID, "Current")), + ).forEachIndexed { index, (capturedAt, members) -> + database.ingestSnapshot( + PlatoonSnapshot( + id = 0, + capturedAt = capturedAt, + sourceFile = "open-gap-$index.csv", + members = members, + ), + EvidenceSource.LEGACY_IMPORT, + historicalOnly = true, + ) + } + + val periods = database.listMemberStatuses() + .single { it.uid == TARGET_UID } + .membershipPeriods + .sortedBy { it.joinedAt ?: Instant.MIN } + assertEquals(2, periods.size) + assertEquals(exactJoin, periods[0].joinedAt) + assertEquals(currentTime.minusSeconds(3 * 86_400), periods[0].leftAt) + assertEquals(EvidenceSource.GAME_UPDATES, periods[0].joinedSource) + assertEquals(EvidenceSource.LEGACY_IMPORT, periods[0].leftSource) + assertEquals(currentTime.minusSeconds(2 * 86_400), periods[1].joinedAt) + assertEquals(currentTime, periods[1].leftAt) + assertForeignKeysValid() + } + + @Test + fun historicalRosterReplayPreservesAUserNoteOnAWeakMembershipPeriod() { + val currentTime = Instant.parse("2026-07-31T00:00:00Z") + database.ingestSnapshot( + PlatoonSnapshot( + id = 0, + capturedAt = currentTime, + sourceFile = "current-note.csv", + members = listOf(member(TARGET_UID, "Annotated"), member(OTHER_UID, "Current")), + ), + EvidenceSource.SNAPSHOT, + ) + database.writableDatabase.execSQL( + "UPDATE membership_periods SET note = ? WHERE uid = ?", + arrayOf("Preserve this note", TARGET_UID), + ) + + database.ingestSnapshot( + PlatoonSnapshot( + id = 0, + capturedAt = currentTime.minusSeconds(86_400), + sourceFile = "historical-note.csv", + members = listOf(member(OTHER_UID, "Current")), + ), + EvidenceSource.LEGACY_IMPORT, + historicalOnly = true, + ) + + val periods = database.listMemberStatuses() + .single { it.uid == TARGET_UID } + .membershipPeriods + assertEquals(1, periods.size) + assertEquals("Preserve this note", periods.single().note) + assertEquals(null, periods.single().joinedAt) + assertForeignKeysValid() + } + + @Test + fun historicalRosterReplayExpandsAnAnnotatedWeakJoinWithoutLosingItsNote() { + val initialTime = Instant.parse("2026-07-20T00:00:00Z") + val inferredJoin = initialTime.plusSeconds(2 * 86_400) + val historicalPresence = initialTime.plusSeconds(86_400) + database.ingestSnapshot( + PlatoonSnapshot( + id = 0, + capturedAt = initialTime, + sourceFile = "annotated-initial.csv", + members = listOf(member(OTHER_UID, "Current")), + ), + EvidenceSource.SNAPSHOT, + ) + database.ingestSnapshot( + PlatoonSnapshot( + id = 0, + capturedAt = inferredJoin, + sourceFile = "annotated-join.csv", + members = listOf(member(TARGET_UID, "Annotated"), member(OTHER_UID, "Current")), + ), + EvidenceSource.SNAPSHOT, + ) + database.ingestSnapshot( + PlatoonSnapshot( + id = 0, + capturedAt = inferredJoin.plusSeconds(86_400), + sourceFile = "annotated-current.csv", + members = listOf(member(TARGET_UID, "Annotated"), member(OTHER_UID, "Current")), + ), + EvidenceSource.SNAPSHOT, + ) + database.writableDatabase.execSQL( + "UPDATE membership_periods SET note = ? WHERE uid = ?", + arrayOf("Preserve this note", TARGET_UID), + ) + + database.ingestSnapshot( + PlatoonSnapshot( + id = 0, + capturedAt = historicalPresence, + sourceFile = "annotated-historical.csv", + members = listOf(member(TARGET_UID, "Annotated"), member(OTHER_UID, "Current")), + ), + EvidenceSource.LEGACY_IMPORT, + historicalOnly = true, + ) + + val period = database.listMemberStatuses() + .single { it.uid == TARGET_UID } + .membershipPeriods + .single() + assertEquals("Preserve this note", period.note) + assertEquals(historicalPresence, period.joinedAt) + assertEquals(EvidenceSource.LEGACY_IMPORT, period.joinedSource) + assertForeignKeysValid() + } + + @Test + fun outerTransactionRollsBackNestedSnapshotIngest() { + val failure = runCatching { + database.runInTransaction { + database.ingestSnapshot( + PlatoonSnapshot( + id = 0, + capturedAt = Instant.parse("2026-07-31T00:00:00Z"), + sourceFile = "rolled-back.csv", + members = listOf(member(TARGET_UID, "Rolled back")), + ), + EvidenceSource.LEGACY_IMPORT, + ) + error("Force the outer import transaction to fail") + } + } + + assertEquals("Force the outer import transaction to fail", failure.exceptionOrNull()?.message) + assertEquals(0L, count("snapshots", "id > ?", 0)) + assertEquals(0L, count("members", "uid = ?", TARGET_UID)) + assertForeignKeysValid() + } + + @Test + fun databaseRejectsRosterBoundsForEveryIngestionCaller() { + val oversizedRoster = (1..257).map { index -> member(index.toLong(), "Member $index") } + val oversizedName = member(TARGET_UID, "x".repeat(257)) + + assertTrue( + runCatching { + database.ingestSnapshot( + PlatoonSnapshot( + id = 0, + capturedAt = Instant.parse("2026-07-31T00:00:00Z"), + sourceFile = "oversized-roster.csv", + members = oversizedRoster, + ), + EvidenceSource.SNAPSHOT, + ) + }.exceptionOrNull() is IllegalArgumentException, + ) + assertTrue( + runCatching { + database.ingestSnapshot( + PlatoonSnapshot( + id = 0, + capturedAt = Instant.parse("2026-07-31T00:00:00Z"), + sourceFile = "oversized-name.csv", + members = listOf(oversizedName), + ), + EvidenceSource.SNAPSHOT, + ) + }.exceptionOrNull() is IllegalArgumentException, + ) + assertEquals(0L, count("snapshots", "id > ?", 0)) + } + + @Test + fun schemaSixAndSevenUpgradeDirectlyWithoutRecreatingActivityIndexes() { + listOf(6, 7).forEach { legacyVersion -> + val databaseName = "platoon-v$legacyVersion-upgrade-test.db" + context.deleteDatabase(databaseName) + try { + createLegacyActivityDatabase(databaseName, legacyVersion) + PlatoonDatabase(context, databaseName).use { upgraded -> + val writable = upgraded.writableDatabase + assertEquals(PlatoonSchema.CURRENT_VERSION, writable.version) + assertEquals( + 1L, + writable.rawQuery( + "SELECT COUNT(*) FROM sqlite_master " + + "WHERE type = 'index' AND name = 'platoon_activity_exact_identity'", + null, + ).use { cursor -> + assertTrue(cursor.moveToFirst()) + cursor.getLong(0) + }, + ) + assertEquals( + 1L, + writable.rawQuery( + "SELECT COUNT(*) FROM platoon_activity", + null, + ).use { cursor -> + assertTrue(cursor.moveToFirst()) + cursor.getLong(0) + }, + ) + writable.rawQuery("PRAGMA foreign_key_check", null).use { cursor -> + assertFalse(cursor.moveToFirst()) + } + } + } finally { + context.deleteDatabase(databaseName) + } + } + } + + @Test + fun schemaTenUpgradeAddsVersionedMaintenanceObjects() { + val databaseName = "platoon-v10-upgrade-test.db" + context.deleteDatabase(databaseName) + try { + PlatoonDatabase(context, databaseName).use { helper -> helper.writableDatabase } + context.openOrCreateDatabase(databaseName, Context.MODE_PRIVATE, null).use { legacy -> + legacy.execSQL("DROP INDEX platoon_activity_resolution_retention") + legacy.execSQL("DROP INDEX platoon_activity_retention_order") + legacy.execSQL("DROP TABLE platoon_maintenance_state") + legacy.version = 10 + } + + PlatoonDatabase(context, databaseName).use { upgraded -> + val writable = upgraded.writableDatabase + assertEquals(PlatoonSchema.CURRENT_VERSION, writable.version) + assertEquals( + 1L, + writable.rawQuery( + "SELECT COUNT(*) FROM sqlite_master " + + "WHERE type = 'table' AND name = 'platoon_maintenance_state'", + null, + ).use { cursor -> + assertTrue(cursor.moveToFirst()) + cursor.getLong(0) + }, + ) + assertEquals( + 1L, + writable.rawQuery( + "SELECT COUNT(*) FROM sqlite_master " + + "WHERE type = 'index' AND name = 'platoon_activity_retention_order'", + null, + ).use { cursor -> + assertTrue(cursor.moveToFirst()) + cursor.getLong(0) + }, + ) + writable.rawQuery("PRAGMA foreign_key_check", null).use { cursor -> + assertFalse(cursor.moveToFirst()) + } + } + } finally { + context.deleteDatabase(databaseName) + } + } + @Test fun exactUpdatesSynchronizeCurrentStateWithoutOverridingNewerRoster() { val rosterAt = Instant.parse("2026-07-31T00:00:00Z") @@ -419,6 +710,148 @@ class PlatoonDatabaseIntegrationTest { ) } + @Test + fun openingDatabaseTrimsAnOversizedActivityBacklogToTheNewestBound() { + val writable = database.writableDatabase + writable.beginTransaction() + try { + repeat(PlatoonDatabase.MAX_STORED_ACTIVITY_OBSERVATIONS + 3) { index -> + writable.insertOrThrow( + "platoon_activity", + null, + ContentValues().apply { + put("occurred_at", index.toLong()) + put("action_id", index.toLong() + 1) + put("kind", 1L) + put("member_name", "Unresolved $index") + put("captured_at", index.toLong()) + put("resolution", ActivityResolution.UNRESOLVED.name) + }, + ) + } + writable.setTransactionSuccessful() + } finally { + writable.endTransaction() + } + database.close() + + database = PlatoonDatabase(context, TEST_DATABASE) + database.writableDatabase + + assertEquals( + PlatoonDatabase.MAX_STORED_ACTIVITY_OBSERVATIONS.toLong(), + count("platoon_activity", "id > ?", 0), + ) + assertEquals(0L, count("platoon_activity", "action_id = ?", 1)) + assertEquals(1L, count("platoon_activity", "action_id = ?", 10_003)) + } + + @Test + fun updatesIngestionTrimsDailyPatrolFactsWithoutWaitingForReopen() { + val capturedAt = Instant.parse("2026-07-31T00:00:00Z") + val writable = database.writableDatabase + writable.beginTransaction() + try { + repeat(PlatoonDatabase.MAX_STORED_ACTIVITY_OBSERVATIONS) { index -> + writable.insertOrThrow( + "platoon_activity", + null, + ContentValues().apply { + put("occurred_at", index.toLong()) + put("action_id", index.toLong() + 1) + put("kind", 1L) + put("member_name", "Unresolved $index") + put("captured_at", index.toLong()) + put("resolution", ActivityResolution.UNRESOLVED.name) + }, + ) + } + writable.setTransactionSuccessful() + } finally { + writable.endTransaction() + } + + val result = database.ingestPlatoonUpdates( + listOf( + update( + PlatoonUpdateSemantics.KIND_DAILY_PATROL, + capturedAt, + TARGET_UID, + "Target", + ), + ), + capturedAt, + ) + + assertEquals(1, result.patrolFacts) + assertEquals( + PlatoonDatabase.MAX_STORED_ACTIVITY_OBSERVATIONS.toLong(), + count("platoon_activity", "id > ?", 0), + ) + assertEquals(0L, count("platoon_activity", "action_id = ?", 1)) + assertEquals( + 1L, + count( + "platoon_activity", + "action_id = ?", + PlatoonDatabase.DAILY_PATROL_REWARD_ACTION_ID, + ), + ) + } + + @Test + fun unresolvedActivityResolutionRotatesPastAnUnmatchableBatch() { + val capturedAt = Instant.parse("2026-07-31T00:00:00Z") + val writable = database.writableDatabase + writable.beginTransaction() + try { + repeat(251) { index -> + writable.insertOrThrow( + "platoon_activity", + null, + ContentValues().apply { + put("occurred_at", capturedAt.toEpochMilli() + index) + put("action_id", index.toLong() + 1) + put("kind", 1L) + put("member_name", if (index == 250) "Target" else "Blocked $index") + put("captured_at", capturedAt.toEpochMilli() + index) + put("resolution", ActivityResolution.UNRESOLVED.name) + }, + ) + } + writable.setTransactionSuccessful() + } finally { + writable.endTransaction() + } + + database.ingestSnapshot( + PlatoonSnapshot( + id = 0, + capturedAt = capturedAt, + sourceFile = "resolution-first.csv", + members = listOf(member(TARGET_UID, "Target")), + ), + EvidenceSource.SNAPSHOT, + ) + assertEquals( + 1L, + count("platoon_activity", "action_id = ? AND resolved_uid IS NULL", 251), + ) + + database.ingestSnapshot( + PlatoonSnapshot( + id = 0, + capturedAt = capturedAt.plusSeconds(86_400), + sourceFile = "resolution-second.csv", + members = listOf(member(TARGET_UID, "Target")), + ), + EvidenceSource.SNAPSHOT, + ) + + assertEquals(1L, count("platoon_activity", "action_id = 251 AND resolved_uid = ?", TARGET_UID)) + assertForeignKeysValid() + } + @Test fun membershipPeriodDeletionRequiresAReplacementAndRemovesLinkedEvents() { val capturedAt = Instant.parse("2026-07-31T00:00:00Z") @@ -471,6 +904,124 @@ class PlatoonDatabaseIntegrationTest { assertForeignKeysValid() } + private fun createLegacyActivityDatabase(databaseName: String, version: Int) { + context.openOrCreateDatabase(databaseName, Context.MODE_PRIVATE, null).use { legacy -> + listOf( + """ + CREATE TABLE snapshots ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + captured_at INTEGER NOT NULL, + source_file TEXT UNIQUE, + game_version TEXT + ) + """, + """ + CREATE TABLE snapshot_members ( + snapshot_id INTEGER NOT NULL REFERENCES snapshots(id) ON DELETE CASCADE, + uid INTEGER NOT NULL, + name TEXT NOT NULL, + level INTEGER NOT NULL, + weekly_merit INTEGER NOT NULL, + total_merit INTEGER NOT NULL, + high_score INTEGER NOT NULL, + total_score INTEGER NOT NULL, + last_login INTEGER NOT NULL, + PRIMARY KEY(snapshot_id, uid) + ) + """, + """ + CREATE TABLE members ( + uid INTEGER PRIMARY KEY, + current_name TEXT NOT NULL, + custom_name TEXT, + current_level INTEGER NOT NULL, + is_active INTEGER NOT NULL, + first_seen_at INTEGER NOT NULL, + last_seen_at INTEGER NOT NULL, + note TEXT NOT NULL DEFAULT '' + ) + """, + """ + CREATE TABLE tenures ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + uid INTEGER NOT NULL REFERENCES members(uid), + joined_at INTEGER, + left_at INTEGER, + joined_precision TEXT NOT NULL, + left_precision TEXT, + joined_source TEXT NOT NULL, + left_source TEXT, + note TEXT NOT NULL DEFAULT '' + ) + """, + "CREATE INDEX tenures_uid ON tenures(uid, id DESC)", + """ + CREATE TABLE member_events ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + uid INTEGER NOT NULL REFERENCES members(uid), + tenure_id INTEGER REFERENCES tenures(id), + event_type TEXT NOT NULL, + occurred_at INTEGER, + observed_at INTEGER NOT NULL, + precision TEXT NOT NULL, + source TEXT NOT NULL, + note TEXT NOT NULL DEFAULT '' + ) + """, + "CREATE INDEX member_events_time ON member_events(observed_at DESC)", + """ + CREATE TABLE weekly_notes ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + period_start INTEGER NOT NULL, + game_day INTEGER NOT NULL, + text TEXT NOT NULL, + event_id INTEGER REFERENCES member_events(id) ON DELETE SET NULL, + is_automatic INTEGER NOT NULL DEFAULT 0 + ) + """, + """ + CREATE TABLE weekly_overrides ( + uid INTEGER NOT NULL, + period_start INTEGER NOT NULL, + game_day INTEGER NOT NULL, + merit_delta INTEGER, + score_delta INTEGER, + attempts INTEGER, + attended INTEGER, + daily_patrol INTEGER, + PRIMARY KEY(uid, game_day) + ) + """, + """ + CREATE TABLE platoon_activity ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + occurred_at INTEGER NOT NULL, + action_id INTEGER NOT NULL, + kind INTEGER NOT NULL, + member_name TEXT NOT NULL, + captured_at INTEGER NOT NULL, + resolved_uid INTEGER REFERENCES members(uid), + resolution TEXT NOT NULL DEFAULT 'UNRESOLVED', + member_event_id INTEGER REFERENCES member_events(id) + ) + """, + "CREATE INDEX platoon_activity_member_time " + + "ON platoon_activity(member_name, occurred_at)", + "CREATE INDEX platoon_activity_action_time " + + "ON platoon_activity(action_id, occurred_at)", + ).forEach { statement -> legacy.execSQL(statement.trimIndent()) } + repeat(2) { index -> + legacy.execSQL( + "INSERT INTO platoon_activity(" + + "occurred_at, action_id, kind, member_name, captured_at" + + ") VALUES(?, ?, ?, ?, ?)", + arrayOf(1_000L, 801_005L, 2L, "Duplicate", index.toLong()), + ) + } + legacy.version = version + } + } + private fun update(kind: Long, at: Instant, uid: Long, name: String) = PlatoonUpdateObservation( kind = kind, diff --git a/app/src/main/java/dev/gf2log/app/MainActivity.kt b/app/src/main/java/dev/gf2log/app/MainActivity.kt index f30b788..e3ed095 100644 --- a/app/src/main/java/dev/gf2log/app/MainActivity.kt +++ b/app/src/main/java/dev/gf2log/app/MainActivity.kt @@ -406,7 +406,7 @@ class MainActivity : LocalizedActivity() { var retained = 0 var duplicates = 0 val createdFiles = mutableListOf() - try { + val imported = try { sources.forEach { source -> val input = TrustedImportSource.openInputStream(contentResolver, source) ?: error("Document provider did not open an input stream") @@ -419,11 +419,11 @@ class MainActivity : LocalizedActivity() { } } } + PlatoonRepository(this).reconcileRetainedCsvFiles(directory) } catch (error: Exception) { createdFiles.forEach(File::delete) throw error } - val imported = PlatoonRepository(this).reconcileRetainedCsvFiles(directory) CsvImportSummary(retained, duplicates, imported) } statusHandler.post { @@ -469,10 +469,7 @@ class MainActivity : LocalizedActivity() { @Suppress("DEPRECATION") private fun exportLatestPlatoonCsv() { val directory = File(filesDir, PlatoonRepository.RETAINED_CSV_DIRECTORY) - val latest = directory.listFiles() - .orEmpty() - .filter { it.isFile && it.extension.equals("csv", ignoreCase = true) } - .maxByOrNull(File::lastModified) + val latest = PlatoonCsvImportStore.latestRetainedFile(directory) if (latest == null) { statusText.text = getString(R.string.status_no_platoon_csv) return diff --git a/app/src/main/java/dev/gf2log/app/capture/GuildMembersCsvWriter.kt b/app/src/main/java/dev/gf2log/app/capture/GuildMembersCsvWriter.kt index 9d1fc56..b5f2f5b 100644 --- a/app/src/main/java/dev/gf2log/app/capture/GuildMembersCsvWriter.kt +++ b/app/src/main/java/dev/gf2log/app/capture/GuildMembersCsvWriter.kt @@ -33,12 +33,22 @@ class GuildMembersCsvWriter( } val data = payload.data as? GuildMembersData ?: return null + if (!GuildMembersCsv.hasValidMemberBounds(data.members)) { + closeActiveBatch(completed = false) + error("Platoon roster exceeds the member or name limit") + } var batch = activeBatch if (batch == null || (batch.previousMessageId != 0 && batch.previousMessageId != payload.messageId)) { closeActiveBatch(completed = false) batch = openBatch() activeBatch = batch } + if ((batch.members.keys + data.members.map(GuildMember::uid)).distinct().size > + GuildMembersCsv.MAX_ROSTER_MEMBERS + ) { + closeActiveBatch(completed = false) + error("Platoon roster exceeds the member limit") + } data.members.forEach { member -> batch.writer.appendLine(GuildMembersCsv.row(member, batch.logTime)) diff --git a/app/src/main/java/dev/gf2log/app/history/CaptureHistoryStore.kt b/app/src/main/java/dev/gf2log/app/history/CaptureHistoryStore.kt index c921e75..4531b4e 100644 --- a/app/src/main/java/dev/gf2log/app/history/CaptureHistoryStore.kt +++ b/app/src/main/java/dev/gf2log/app/history/CaptureHistoryStore.kt @@ -27,7 +27,12 @@ class CaptureHistoryStore( val destination = File(directory, id) val temporary = File.createTempFile("packet_", ".tmp", directory) try { - temporary.writeText(ParsedPayloadTextFormatter.format(payload, capturedAt), Charsets.UTF_8) + val content = ParsedPayloadTextFormatter.format(payload, capturedAt) + .toByteArray(Charsets.UTF_8) + require(content.size <= MAX_ENTRY_BYTES) { + "Parsed-packet history entry exceeds the storage limit" + } + temporary.writeBytes(content) try { Files.move( temporary.toPath(), @@ -59,6 +64,7 @@ class CaptureHistoryStore( if (id.isBlank() || File(id).name != id) return null val file = File(directory, id) if (!file.isFile || file.parentFile?.canonicalFile != directory.canonicalFile) return null + if (file.length() > MAX_ENTRY_BYTES) return null return file.readText(Charsets.UTF_8) } @@ -96,6 +102,7 @@ class CaptureHistoryStore( companion object { const val MAX_ENTRIES = 100 + const val MAX_ENTRY_BYTES = 2 * 1024 * 1024 const val HISTORY_DIRECTORY = "capture-history" private val sequence = AtomicLong() private val LOG_TIME_FORMAT = DateTimeFormatter diff --git a/app/src/main/java/dev/gf2log/app/management/GunsmokeWeekSolver.kt b/app/src/main/java/dev/gf2log/app/management/GunsmokeWeekSolver.kt index 02ee70b..b779947 100644 --- a/app/src/main/java/dev/gf2log/app/management/GunsmokeWeekSolver.kt +++ b/app/src/main/java/dev/gf2log/app/management/GunsmokeWeekSolver.kt @@ -167,6 +167,7 @@ internal object GunsmokeWeekSolver { runStart = 0, anchor = anchor, checkpoint = sunday, + zoneId = zoneId, periodStart = periodStart, cell = cells[0], ).filter { state -> @@ -303,6 +304,7 @@ internal object GunsmokeWeekSolver { runStart = runStart, anchor = anchor, checkpoint = firstCheckpoint, + zoneId = zoneId, periodStart = periodStart, cell = cells[runStart], ) @@ -584,6 +586,7 @@ internal object GunsmokeWeekSolver { // - runStart: First observed day index in the contiguous run. // - anchor: Latest snapshot before the report period, if one exists. // - checkpoint: First member observation in the run. + // - zoneId: Game-day timezone used to derive the preceding counter boundary. // - periodStart: Sunday 05:00 report boundary. // - cell: Conservatively derived first-day cell. // Returns: @@ -592,6 +595,7 @@ internal object GunsmokeWeekSolver { runStart: Int, anchor: Checkpoint?, checkpoint: Checkpoint, + zoneId: ZoneId, periodStart: Instant, cell: WeeklyReportBuilder.DayCell, ): List { @@ -605,7 +609,12 @@ internal object GunsmokeWeekSolver { if ( anchor != null && !anchor.capturedAt.isAfter(periodStart) && - isClosedOpeningAnchor(anchor, periodStart) && + isClosedOpeningAnchor( + capturedAt = anchor.capturedAt, + weeklyMerit = anchor.member.weeklyMerit, + zoneId = zoneId, + periodStart = periodStart, + ) && checkpoint.member.totalMerit >= anchor.member.totalMerit ) { val merit = checkpoint.member.totalMerit - anchor.member.totalMerit @@ -631,16 +640,31 @@ internal object GunsmokeWeekSolver { // Function Name: isClosedOpeningAnchor // Description: - // - Accepts a near-reset capture or a standard-week counter that has already reached its cap. + // - Accepts a near-reset capture or a preceding-week counter that has already reached its cap. // - A capped Monday-through-Saturday counter cannot gain more merit before the Gunsmoke Sunday. // Parameters: - // - anchor: Latest counter observation before the Gunsmoke report boundary. + // - capturedAt: Latest counter observation before the Gunsmoke report boundary. + // - weeklyMerit: Weekly counter observed at capturedAt. + // - zoneId: Game-day timezone used for the calendar reset boundary. // - periodStart: Sunday 05:00 Gunsmoke report boundary. // Returns: // - True when no unobserved pre-boundary merit can contaminate the opening interval. - private fun isClosedOpeningAnchor(anchor: Checkpoint, periodStart: Instant): Boolean = - Duration.between(anchor.capturedAt, periodStart) <= MAX_BOUNDARY_DISTANCE || - anchor.member.weeklyMerit == MAX_PRE_GUNSMOKE_WEEKLY_MERIT + internal fun isClosedOpeningAnchor( + capturedAt: Instant, + weeklyMerit: Long, + zoneId: ZoneId, + periodStart: Instant, + ): Boolean { + val precedingCounterStart = periodStart.atZone(zoneId).minusDays(6).toInstant() + return !capturedAt.isAfter(periodStart) && + ( + Duration.between(capturedAt, periodStart) <= MAX_BOUNDARY_DISTANCE || + ( + weeklyMerit == MAX_PRE_GUNSMOKE_WEEKLY_MERIT && + !capturedAt.isBefore(precedingCounterStart) + ) + ) + } private fun statesForMetrics( merit: Long, diff --git a/app/src/main/java/dev/gf2log/app/management/PlatoonCsvImportStore.kt b/app/src/main/java/dev/gf2log/app/management/PlatoonCsvImportStore.kt index 7d1fff5..61775b0 100644 --- a/app/src/main/java/dev/gf2log/app/management/PlatoonCsvImportStore.kt +++ b/app/src/main/java/dev/gf2log/app/management/PlatoonCsvImportStore.kt @@ -47,6 +47,9 @@ class PlatoonCsvImportStore(private val directory: File) { require(directory.isDirectory) { "Unable to create the Platoon CSV directory" } if (file.isFile) { require(file.readBytes().contentEquals(bytes)) { "CSV identity collision" } + require(file.setLastModified(capturedAt.toEpochMilli())) { + "Unable to preserve the imported Platoon CSV capture time" + } return RetainResult(file, duplicate = true) } @@ -65,6 +68,10 @@ class PlatoonCsvImportStore(private val directory: File) { } catch (_: AtomicMoveNotSupportedException) { Files.move(temporary.toPath(), file.toPath()) } + if (!file.setLastModified(capturedAt.toEpochMilli())) { + file.delete() + error("Unable to preserve the imported Platoon CSV capture time") + } } finally { temporary.delete() } @@ -87,11 +94,16 @@ class PlatoonCsvImportStore(private val directory: File) { data class RetainResult(val file: File, val duplicate: Boolean) - private companion object { - const val MAX_BYTES = 2 * 1024 * 1024 - const val READ_BUFFER_BYTES = 8 * 1024 - const val HASH_BYTES = 10 - val FILE_TIME: DateTimeFormatter = DateTimeFormatter + companion object { + internal fun latestRetainedFile(directory: File): File? = directory.listFiles() + .orEmpty() + .filter { it.isFile && it.extension.equals("csv", ignoreCase = true) } + .maxWithOrNull(compareBy(File::lastModified).thenBy(File::getName)) + + private const val MAX_BYTES = 2 * 1024 * 1024 + private const val READ_BUFFER_BYTES = 8 * 1024 + private const val HASH_BYTES = 10 + private val FILE_TIME: DateTimeFormatter = DateTimeFormatter .ofPattern("yyyyMMdd'T'HHmmss'Z'") .withZone(ZoneOffset.UTC) } diff --git a/app/src/main/java/dev/gf2log/app/management/PlatoonDatabase.kt b/app/src/main/java/dev/gf2log/app/management/PlatoonDatabase.kt index 26d0a33..aa61844 100644 --- a/app/src/main/java/dev/gf2log/app/management/PlatoonDatabase.kt +++ b/app/src/main/java/dev/gf2log/app/management/PlatoonDatabase.kt @@ -5,6 +5,7 @@ import android.content.Context import android.database.Cursor import android.database.sqlite.SQLiteDatabase import android.database.sqlite.SQLiteOpenHelper +import dev.gf2log.protocol.GuildMembersCsv import java.time.Instant import java.time.LocalDate import java.time.ZoneId @@ -26,6 +27,15 @@ class PlatoonDatabase( db.setForeignKeyConstraintsEnabled(true) } + override fun onOpen(db: SQLiteDatabase) { + super.onOpen(db) + if (!db.isReadOnly) { + createPlatoonMaintenanceStateTable(db) + createPlatoonActivityRetentionIndex(db) + trimPlatoonActivity(db) + } + } + override fun onCreate(db: SQLiteDatabase) { db.execSQL( """ @@ -120,6 +130,7 @@ class PlatoonDatabase( """.trimIndent(), ) createWeeklyOverridesTable(db) + createPlatoonMaintenanceStateTable(db) } override fun onUpgrade(db: SQLiteDatabase, oldVersion: Int, newVersion: Int) { @@ -216,12 +227,13 @@ class PlatoonDatabase( if (oldVersion < 7) { backfillSnapshotMembershipPeriodEvents(db) } - if (oldVersion in 6 until 8) { - migratePlatoonActivityIdentity(db) - } if (needsManualCalendarDateBackfill) { backfillManualCalendarDates(db, ZoneId.systemDefault()) } + if (oldVersion < 11) { + createPlatoonMaintenanceStateTable(db) + createPlatoonActivityRetentionIndex(db) + } } // Function Name: migrateMembershipPeriodEventReference @@ -304,13 +316,14 @@ class PlatoonDatabase( createPlatoonActivityTable(db) db.execSQL( """ - INSERT INTO platoon_activity( + INSERT OR IGNORE INTO platoon_activity( id, occurred_at, action_id, kind, member_name, captured_at, resolved_uid, resolution, member_event_id ) SELECT id, occurred_at, action_id, kind, member_name, captured_at, resolved_uid, resolution, member_event_id FROM platoon_activity_legacy_v10 + ORDER BY CASE WHEN member_event_id IS NULL THEN 1 ELSE 0 END, id """.trimIndent(), ) db.execSQL("DROP TABLE weekly_notes_legacy_v10") @@ -318,6 +331,19 @@ class PlatoonDatabase( db.execSQL("DROP TABLE member_events_legacy_v10") } + @Synchronized + internal fun runInTransaction(block: () -> T): T { + val db = writableDatabase + db.beginTransaction() + try { + val result = block() + db.setTransactionSuccessful() + return result + } finally { + db.endTransaction() + } + } + @Synchronized fun ingestSnapshot( snapshot: PlatoonSnapshot, @@ -326,6 +352,12 @@ class PlatoonDatabase( deferHistoricalMembershipReconciliation: Boolean = false, ): SnapshotIngestResult { require(snapshot.members.isNotEmpty()) { "A Platoon snapshot cannot be empty" } + require(snapshot.members.size <= GuildMembersCsv.MAX_ROSTER_MEMBERS) { + "A Platoon snapshot exceeds the member limit" + } + require(snapshot.members.all { it.name.length <= GuildMembersCsv.MAX_MEMBER_NAME_CHARS }) { + "A Platoon snapshot contains an oversized member name" + } require(snapshot.members.map(SnapshotMember::uid).distinct().size == snapshot.members.size) { "A Platoon snapshot cannot contain duplicate UIDs" } @@ -467,6 +499,7 @@ class PlatoonDatabase( ensureRosterActiveMembershipPeriod(db, member.uid, source) } + trimPlatoonActivity(db) resolveUnresolvedActivityUids(db) db.setTransactionSuccessful() return SnapshotIngestResult( @@ -515,7 +548,7 @@ class PlatoonDatabase( allUids.forEach { uid -> val runs = buildRosterPresenceRuns(snapshots, uid) runs.forEach { run -> - if (!hasStrongMembershipPeriodOverlapping(db, uid, run)) { + if (!reconcilePreservedMembershipPeriodOverlap(db, uid, run)) { insertRosterPresenceRun(db, uid, run) } } @@ -586,6 +619,7 @@ class PlatoonDatabase( AND (left_source IS NULL OR left_source IN (?, ?)) AND joined_precision IN (?, ?) AND (left_precision IS NULL OR left_precision = ?) + AND note = '' """.trimIndent(), arrayOf( sourceNames[0], @@ -653,21 +687,246 @@ class PlatoonDatabase( return runs } - private fun hasStrongMembershipPeriodOverlapping( + // Function Name: reconcilePreservedMembershipPeriodOverlap + // Description: + // - Preserves exact, manual, or user-annotated membership periods during roster replay. + // - Closes every overlapping open preserved period at an observed roster absence. + // - Allows a later roster-presence run to become a separate inferred rejoin period. + // Parameters: + // - db: Writable database participating in the replay transaction. + // - uid: Stable member identity whose roster run is being reconciled. + // - run: One contiguous roster-presence run bounded by observed absence when known. + // Returns: + // - True when a preserved period covers this run; false when a new weak run is required. + private fun reconcilePreservedMembershipPeriodOverlap( db: SQLiteDatabase, uid: Long, run: RosterPresenceRun, - ): Boolean = db.rawQuery( - """ - SELECT 1 - FROM membership_periods - WHERE uid = ? - AND (joined_at IS NULL OR joined_at <= ?) - AND (left_at IS NULL OR left_at > ?) - LIMIT 1 - """.trimIndent(), - arrayOf(uid.toString(), run.lastSeenAt.toEpochMilli().toString(), run.firstSeenAt.toEpochMilli().toString()), - ).use(Cursor::moveToFirst) + ): Boolean { + val overlaps = db.rawQuery( + """ + SELECT id, joined_at, joined_precision, joined_source, + left_at, left_precision, left_source + FROM membership_periods + WHERE uid = ? + AND (joined_at IS NULL OR joined_at <= ?) + AND (left_at IS NULL OR left_at > ?) + ORDER BY COALESCE(joined_at, -9223372036854775808), id + """.trimIndent(), + arrayOf( + uid.toString(), + run.lastSeenAt.toEpochMilli().toString(), + run.firstSeenAt.toEpochMilli().toString(), + ), + ).use { cursor -> + buildList { + while (cursor.moveToNext()) { + add( + PreservedMembershipPeriodOverlap( + id = cursor.getLong(0), + joinedAt = cursor.getNullableLong(1)?.let(Instant::ofEpochMilli), + joinedPrecision = EvidencePrecision.valueOf(cursor.getString(2)), + joinedSource = EvidenceSource.valueOf(cursor.getString(3)), + leftAt = cursor.getNullableLong(4)?.let(Instant::ofEpochMilli), + leftPrecision = cursor.getNullableString(5) + ?.let(EvidencePrecision::valueOf), + leftSource = cursor.getNullableString(6) + ?.let(EvidenceSource::valueOf), + ), + ) + } + } + } + overlaps.forEach { membershipPeriod -> + val joinedAt = membershipPeriod.joinedAt + if ( + joinedAt != null && + run.firstSeenAt.isBefore(joinedAt) && + isWeakSnapshotBoundary( + membershipPeriod.joinedPrecision, + membershipPeriod.joinedSource, + ) + ) { + val updated = db.update( + "membership_periods", + ContentValues().apply { + put("joined_at", run.firstSeenAt.toEpochMilli()) + putNull("joined_date") + put("joined_time_known", 1) + put("joined_precision", EvidencePrecision.INFERRED.name) + put("joined_source", EvidenceSource.LEGACY_IMPORT.name) + }, + "id = ?", + arrayOf(membershipPeriod.id.toString()), + ) + if (updated == 1) { + updateSnapshotBoundaryEvent( + db = db, + membershipPeriodId = membershipPeriod.id, + uid = uid, + memberName = run.name, + type = if (hasEarlierMembershipPeriod(db, uid, membershipPeriod.id)) { + MemberEventType.REJOINED + } else { + MemberEventType.JOINED + }, + observedAt = run.firstSeenAt, + ) + } + } + + val observedLeftAt = run.leftAt + val weakLeftBoundary = membershipPeriod.leftPrecision != null && + membershipPeriod.leftSource != null && + isWeakSnapshotBoundary( + membershipPeriod.leftPrecision, + membershipPeriod.leftSource, + ) + if ( + observedLeftAt != null && + ( + membershipPeriod.leftAt == null || + (weakLeftBoundary && membershipPeriod.leftAt != observedLeftAt) + ) + ) { + val updated = db.update( + "membership_periods", + ContentValues().apply { + put("left_at", observedLeftAt.toEpochMilli()) + putNull("left_date") + put("left_time_known", 1) + put("left_precision", EvidencePrecision.INFERRED.name) + put("left_source", EvidenceSource.LEGACY_IMPORT.name) + }, + "id = ?", + arrayOf(membershipPeriod.id.toString()), + ) + if (updated == 1) { + updateSnapshotBoundaryEvent( + db = db, + membershipPeriodId = membershipPeriod.id, + uid = uid, + memberName = run.name, + type = MemberEventType.LEFT, + observedAt = observedLeftAt, + ) + } + } else if ( + observedLeftAt == null && + membershipPeriod.leftAt != null && + weakLeftBoundary && + !run.lastSeenAt.isBefore(membershipPeriod.leftAt) + ) { + db.update( + "membership_periods", + ContentValues().apply { + putNull("left_at") + putNull("left_date") + putNull("left_time_known") + putNull("left_precision") + putNull("left_source") + }, + "id = ?", + arrayOf(membershipPeriod.id.toString()), + ) + deleteSnapshotBoundaryEvents( + db = db, + membershipPeriodId = membershipPeriod.id, + type = MemberEventType.LEFT, + ) + } + } + return overlaps.isNotEmpty() + } + + private fun isWeakSnapshotBoundary( + precision: EvidencePrecision, + source: EvidenceSource, + ): Boolean = precision in setOf(EvidencePrecision.UNKNOWN, EvidencePrecision.INFERRED) && + source in SNAPSHOT_EVENT_SOURCES + + private fun updateSnapshotBoundaryEvent( + db: SQLiteDatabase, + membershipPeriodId: Long, + uid: Long, + memberName: String, + type: MemberEventType, + observedAt: Instant, + ) { + val compatibleTypes = when (type) { + MemberEventType.JOINED, + MemberEventType.REJOINED, + -> arrayOf(MemberEventType.JOINED.name, MemberEventType.REJOINED.name) + else -> arrayOf(type.name, type.name) + } + val updated = db.update( + "member_events", + ContentValues().apply { + put("event_type", type.name) + putNull("occurred_at") + putNull("event_date") + put("time_known", 1) + put("observed_at", observedAt.toEpochMilli()) + put("precision", EvidencePrecision.INFERRED.name) + put("source", EvidenceSource.LEGACY_IMPORT.name) + put("note", memberName) + }, + "membership_period_id = ? AND event_type IN (?, ?) " + + "AND source IN (?, ?)", + arrayOf( + membershipPeriodId.toString(), + compatibleTypes[0], + compatibleTypes[1], + EvidenceSource.SNAPSHOT.name, + EvidenceSource.LEGACY_IMPORT.name, + ), + ) + if (updated == 0) { + insertSnapshotBoundaryEvent( + db = db, + membershipPeriodId = membershipPeriodId, + uid = uid, + memberName = memberName, + type = type, + observedAt = observedAt, + source = EvidenceSource.LEGACY_IMPORT, + ) + } + } + + private fun deleteSnapshotBoundaryEvents( + db: SQLiteDatabase, + membershipPeriodId: Long, + type: MemberEventType, + ) { + val eventSelection = + "membership_period_id = ? AND event_type = ? AND source IN (?, ?)" + val eventArguments = arrayOf( + membershipPeriodId.toString(), + type.name, + EvidenceSource.SNAPSHOT.name, + EvidenceSource.LEGACY_IMPORT.name, + ) + db.delete( + "weekly_notes", + "is_automatic = 1 AND event_id IN " + + "(SELECT id FROM member_events WHERE $eventSelection)", + eventArguments, + ) + db.update( + "weekly_notes", + ContentValues().apply { putNull("event_id") }, + "event_id IN (SELECT id FROM member_events WHERE $eventSelection)", + eventArguments, + ) + db.update( + "platoon_activity", + ContentValues().apply { putNull("member_event_id") }, + "member_event_id IN (SELECT id FROM member_events WHERE $eventSelection)", + eventArguments, + ) + db.delete("member_events", eventSelection, eventArguments) + } private fun insertRosterPresenceRun( db: SQLiteDatabase, @@ -804,6 +1063,7 @@ class PlatoonDatabase( if (resolvedUid != null) resolved += 1 } } + trimPlatoonActivity(db) resolveUnresolvedActivityUids(db) reconcileInferredMembershipBoundaries(db) db.setTransactionSuccessful() @@ -905,6 +1165,7 @@ class PlatoonDatabase( PlatoonUpdateEffect.IGNORE -> Unit } } + trimPlatoonActivity(db) db.setTransactionSuccessful() return UpdatesIngestResult(acceptedObservations.size, membershipEvents, patrolFacts) } finally { @@ -2196,53 +2457,29 @@ class PlatoonDatabase( WHERE resolved_uid IS NULL """.trimIndent(), ) + createPlatoonActivityRetentionIndex(db) } - private fun migratePlatoonActivityIdentity(db: SQLiteDatabase) { - db.execSQL("ALTER TABLE platoon_activity RENAME TO platoon_activity_legacy") + private fun createPlatoonActivityRetentionIndex(db: SQLiteDatabase) { db.execSQL( - """ - CREATE TABLE platoon_activity ( - id INTEGER PRIMARY KEY AUTOINCREMENT, - occurred_at INTEGER NOT NULL, - action_id INTEGER NOT NULL, - kind INTEGER NOT NULL, - member_name TEXT NOT NULL, - captured_at INTEGER NOT NULL, - resolved_uid INTEGER REFERENCES members(uid), - resolution TEXT NOT NULL DEFAULT 'UNRESOLVED', - member_event_id INTEGER REFERENCES member_events(id) - ) - """.trimIndent(), + "CREATE INDEX IF NOT EXISTS platoon_activity_resolution_retention " + + "ON platoon_activity(resolved_uid, captured_at DESC, id DESC)", ) db.execSQL( - """ - CREATE UNIQUE INDEX platoon_activity_exact_identity - ON platoon_activity(occurred_at, action_id, kind, resolved_uid) - WHERE resolved_uid IS NOT NULL - """.trimIndent(), - ) - db.execSQL( - """ - CREATE UNIQUE INDEX platoon_activity_unresolved_identity - ON platoon_activity(occurred_at, action_id, kind, member_name) - WHERE resolved_uid IS NULL - """.trimIndent(), + "CREATE INDEX IF NOT EXISTS platoon_activity_retention_order " + + "ON platoon_activity(captured_at DESC, id DESC)", ) + } + + private fun createPlatoonMaintenanceStateTable(db: SQLiteDatabase) { db.execSQL( """ - INSERT OR IGNORE INTO platoon_activity( - id, occurred_at, action_id, kind, member_name, captured_at, - resolved_uid, resolution, member_event_id + CREATE TABLE IF NOT EXISTS platoon_maintenance_state ( + key TEXT PRIMARY KEY, + value INTEGER NOT NULL ) - SELECT id, occurred_at, action_id, kind, member_name, captured_at, - resolved_uid, resolution, member_event_id - FROM platoon_activity_legacy - ORDER BY CASE WHEN member_event_id IS NULL THEN 1 ELSE 0 END, id """.trimIndent(), ) - db.execSQL("DROP TABLE platoon_activity_legacy") - createPlatoonActivityIndexes(db) } private fun latestSnapshotCapturedAt(db: SQLiteDatabase): Instant? = @@ -2292,28 +2529,26 @@ class PlatoonDatabase( } private fun resolveUnresolvedActivityUids(db: SQLiteDatabase) { - val unresolved = db.query( - "platoon_activity", - arrayOf("id", "occurred_at", "member_name", "action_id", "kind"), - "resolved_uid IS NULL", - null, - null, - null, - "occurred_at, id", + createPlatoonMaintenanceStateTable(db) + val cursorId = db.rawQuery( + "SELECT value FROM platoon_maintenance_state WHERE key = ?", + arrayOf(ACTIVITY_RESOLUTION_CURSOR_KEY), ).use { cursor -> - buildList { - while (cursor.moveToNext()) { - add( - UnresolvedActivity( - id = cursor.getLong(0), - occurredAt = Instant.ofEpochMilli(cursor.getLong(1)), - memberName = cursor.getString(2), - actionId = cursor.getLong(3), - kind = cursor.getLong(4), - ), - ) - } - } + if (cursor.moveToFirst()) cursor.getLong(0) else 0L + } + val unresolved = readUnresolvedActivityBatch( + db = db, + selection = "resolved_uid IS NULL AND id > ?", + selectionArgs = arrayOf(cursorId.toString()), + limit = MAX_UNRESOLVED_ACTIVITY_RESOLUTIONS, + ).toMutableList() + if (unresolved.size < MAX_UNRESOLVED_ACTIVITY_RESOLUTIONS) { + unresolved += readUnresolvedActivityBatch( + db = db, + selection = "resolved_uid IS NULL AND id <= ?", + selectionArgs = arrayOf(cursorId.toString()), + limit = MAX_UNRESOLVED_ACTIVITY_RESOLUTIONS - unresolved.size, + ) } unresolved.forEach { activity -> val uid = resolveUidForName(db, activity.memberName, activity.occurredAt) @@ -2353,6 +2588,72 @@ class PlatoonDatabase( ) } } + unresolved.lastOrNull()?.let { activity -> + db.insertWithOnConflict( + "platoon_maintenance_state", + null, + ContentValues().apply { + put("key", ACTIVITY_RESOLUTION_CURSOR_KEY) + put("value", activity.id) + }, + SQLiteDatabase.CONFLICT_REPLACE, + ) + } + } + + private fun readUnresolvedActivityBatch( + db: SQLiteDatabase, + selection: String, + selectionArgs: Array, + limit: Int, + ): List { + if (limit <= 0) return emptyList() + return db.query( + "platoon_activity", + arrayOf("id", "occurred_at", "member_name", "action_id", "kind"), + selection, + selectionArgs, + null, + null, + "id", + limit.toString(), + ).use { cursor -> + buildList { + while (cursor.moveToNext()) { + add( + UnresolvedActivity( + id = cursor.getLong(0), + occurredAt = Instant.ofEpochMilli(cursor.getLong(1)), + memberName = cursor.getString(2), + actionId = cursor.getLong(3), + kind = cursor.getLong(4), + ), + ) + } + } + } + } + + // Function Name: trimPlatoonActivity + // Description: + // - Caps cumulative network-derived activity evidence to a bounded newest-first history. + // - Runs on database open and after each ingest so an older oversized database self-heals. + // Parameters: + // - db: Writable database participating in the caller's transaction when applicable. + // Returns: + // - Unit after deleting only rows beyond the retained observation limit. + private fun trimPlatoonActivity(db: SQLiteDatabase) { + db.execSQL( + """ + DELETE FROM platoon_activity + WHERE id IN ( + SELECT id + FROM platoon_activity + ORDER BY captured_at DESC, id DESC + LIMIT -1 OFFSET $MAX_STORED_ACTIVITY_OBSERVATIONS + ) + """.trimIndent(), + ) } private fun ensureRosterActiveMembershipPeriod( @@ -3431,6 +3732,16 @@ class PlatoonDatabase( val members: Map, ) + private data class PreservedMembershipPeriodOverlap( + val id: Long, + val joinedAt: Instant?, + val joinedPrecision: EvidencePrecision, + val joinedSource: EvidenceSource, + val leftAt: Instant?, + val leftPrecision: EvidencePrecision?, + val leftSource: EvidenceSource?, + ) + private data class RosterPresenceRun( val firstSeenAt: Instant, val lastSeenAt: Instant, @@ -3446,6 +3757,9 @@ class PlatoonDatabase( * must never be interpreted as a missed Daily Patrol. */ const val DAILY_PATROL_REWARD_ACTION_ID = 802001L + internal const val MAX_STORED_ACTIVITY_OBSERVATIONS = 10_000 + private const val MAX_UNRESOLVED_ACTIVITY_RESOLUTIONS = 250 + private const val ACTIVITY_RESOLUTION_CURSOR_KEY = "activity_resolution_cursor" private const val DAILY_PATROL_RELATED_ACTION_ID = 801005L private const val NAME_RESOLUTION_WINDOW_MILLIS = 30L * 24L * 60L * 60L * 1000L private const val MEMBERSHIP_CORRELATION_WINDOW_MILLIS = 12L * 60L * 60L * 1000L diff --git a/app/src/main/java/dev/gf2log/app/management/PlatoonObservationPolicy.kt b/app/src/main/java/dev/gf2log/app/management/PlatoonObservationPolicy.kt index f524864..f26d0a6 100644 --- a/app/src/main/java/dev/gf2log/app/management/PlatoonObservationPolicy.kt +++ b/app/src/main/java/dev/gf2log/app/management/PlatoonObservationPolicy.kt @@ -1,11 +1,19 @@ package dev.gf2log.app.management internal object PlatoonObservationPolicy { + const val MAX_ACTIVITY_OBSERVATIONS = 250 + const val MAX_ACTIVITY_MEMBER_NAME_LENGTH = 128 + fun activity( observations: List, - ): List = observations - .filter { it.memberName.isNotBlank() } + ): List = observations.asSequence() + .filter { + it.memberName.isNotBlank() && + it.memberName.length <= MAX_ACTIVITY_MEMBER_NAME_LENGTH + } .distinctBy { listOf(it.occurredAt, it.actionId, it.kind, it.memberName) } + .take(MAX_ACTIVITY_OBSERVATIONS) + .toList() fun updates( observations: List, diff --git a/app/src/main/java/dev/gf2log/app/management/PlatoonRepository.kt b/app/src/main/java/dev/gf2log/app/management/PlatoonRepository.kt index 8ab84f4..446623f 100644 --- a/app/src/main/java/dev/gf2log/app/management/PlatoonRepository.kt +++ b/app/src/main/java/dev/gf2log/app/management/PlatoonRepository.kt @@ -25,6 +25,9 @@ class PlatoonRepository(context: Context) { members: List, sourceFile: String?, ): SnapshotIngestResult = access { database -> + require(GuildMembersCsv.isValidRoster(members)) { + "A Platoon roster must stay within the member and name limits with unique UIDs" + } database.ingestSnapshot( snapshot = PlatoonSnapshot( id = 0, @@ -41,8 +44,13 @@ class PlatoonRepository(context: Context) { capturedAt: Instant = Instant.now(), ): ActivityIngestResult = access { database -> database.ingestPlatoonActivity( - observations = data.entries.mapNotNull { - if (it.occurredAt == 0u || it.actionId == 0u || it.memberName.isBlank()) { + observations = data.entries.asSequence().mapNotNull { + if ( + it.occurredAt == 0u || + it.actionId == 0u || + it.memberName.isBlank() || + it.memberName.length > PlatoonObservationPolicy.MAX_ACTIVITY_MEMBER_NAME_LENGTH + ) { null } else { PlatoonActivityObservation( @@ -52,7 +60,10 @@ class PlatoonRepository(context: Context) { memberName = it.memberName, ) } - }, + } + .distinct() + .take(PlatoonObservationPolicy.MAX_ACTIVITY_OBSERVATIONS) + .toList(), capturedAt = capturedAt, ) } @@ -90,73 +101,77 @@ class PlatoonRepository(context: Context) { fun reconcileRetainedCsvFiles( directory: File = File(appContext.filesDir, RETAINED_CSV_DIRECTORY), ): ImportResult = access { database -> - var imported = 0 - var historical = 0 - var skipped = 0 - var invalid = 0 - val representedFiles = database.snapshotSourceFiles() - var latestStructuredSnapshot = database.latestSnapshotIdentity()?.let { - SnapshotIdentity(it.first, it.second) - } - directory.listFiles() - .orEmpty() - .filter { it.isFile && it.extension.equals("csv", ignoreCase = true) } - .filter { file -> - if (file.name in representedFiles) { - skipped += 1 - false - } else { - true - } + database.runInTransaction { + var imported = 0 + var historical = 0 + var skipped = 0 + var invalid = 0 + val representedFiles = database.snapshotSourceFiles() + var latestStructuredSnapshot = database.latestSnapshotIdentity()?.let { + SnapshotIdentity(it.first, it.second) } - .mapNotNull { file -> - val parsed = runCatching { - GuildMembersCsv.parse(file.readText(Charsets.UTF_8)) - }.getOrNull() - val capturedAt = parsed?.logTime?.let { runCatching { Instant.parse(it) }.getOrNull() } - if (parsed == null || capturedAt == null) { - invalid += 1 - null - } else { - RetainedCsv(file, capturedAt, parsed.members) + directory.listFiles() + .orEmpty() + .filter { it.isFile && it.extension.equals("csv", ignoreCase = true) } + .filter { file -> + if (file.name in representedFiles) { + skipped += 1 + false + } else { + true + } } - } - .sortedWith(compareBy(RetainedCsv::capturedAt, { it.file.name })) - .forEach { file -> - val historicalOnly = latestStructuredSnapshot?.let { - file.capturedAt.isBefore(it.capturedAt) || - (file.capturedAt == it.capturedAt && - file.file.name <= it.sourceFile.orEmpty()) - } ?: false - val result = database.ingestSnapshot( - PlatoonSnapshot( - id = 0, - capturedAt = file.capturedAt, - members = file.members.map(GuildMember::toSnapshotMember), - sourceFile = file.file.name, - ), - EvidenceSource.LEGACY_IMPORT, - historicalOnly = historicalOnly, - deferHistoricalMembershipReconciliation = historicalOnly, - ) - if (result.duplicate) { - skipped += 1 - } else if (historicalOnly) { - historical += 1 - } else { - imported += 1 - latestStructuredSnapshot = SnapshotIdentity(file.capturedAt, file.file.name) + .mapNotNull { file -> + val parsed = runCatching { + GuildMembersCsv.parse(file.readText(Charsets.UTF_8)) + }.getOrNull() + val capturedAt = parsed?.logTime?.let { + runCatching { Instant.parse(it) }.getOrNull() + } + if (parsed == null || capturedAt == null) { + invalid += 1 + null + } else { + RetainedCsv(file, capturedAt, parsed.members) + } + } + .sortedWith(compareBy(RetainedCsv::capturedAt, { it.file.name })) + .forEach { file -> + val historicalOnly = latestStructuredSnapshot?.let { + file.capturedAt.isBefore(it.capturedAt) || + (file.capturedAt == it.capturedAt && + file.file.name <= it.sourceFile.orEmpty()) + } ?: false + val result = database.ingestSnapshot( + PlatoonSnapshot( + id = 0, + capturedAt = file.capturedAt, + members = file.members.map(GuildMember::toSnapshotMember), + sourceFile = file.file.name, + ), + EvidenceSource.LEGACY_IMPORT, + historicalOnly = historicalOnly, + deferHistoricalMembershipReconciliation = historicalOnly, + ) + if (result.duplicate) { + skipped += 1 + } else if (historicalOnly) { + historical += 1 + } else { + imported += 1 + latestStructuredSnapshot = SnapshotIdentity(file.capturedAt, file.file.name) + } } - } - if (historical > 0) database.reconcileSnapshotMembershipHistory() + if (historical > 0) database.reconcileSnapshotMembershipHistory() - ImportResult( - imported = imported, - historical = historical, - skipped = skipped, - invalid = invalid, - ) + ImportResult( + imported = imported, + historical = historical, + skipped = skipped, + invalid = invalid, + ) + } } fun listSnapshots(limit: Int = 100): List = diff --git a/app/src/main/java/dev/gf2log/app/management/PlatoonSchema.kt b/app/src/main/java/dev/gf2log/app/management/PlatoonSchema.kt index f918859..585a8ee 100644 --- a/app/src/main/java/dev/gf2log/app/management/PlatoonSchema.kt +++ b/app/src/main/java/dev/gf2log/app/management/PlatoonSchema.kt @@ -5,7 +5,7 @@ package dev.gf2log.app.management */ internal object PlatoonSchema { const val DATABASE_NAME = "platoon.db" - const val CURRENT_VERSION = 10 + const val CURRENT_VERSION = 11 const val MIN_BACKUP_VERSION = 1 private val baseTables = setOf( @@ -21,6 +21,7 @@ internal object PlatoonSchema { add(if (version >= 10) "membership_periods" else "tenures") if (version >= 3) add("weekly_overrides") if (version >= 6) add("platoon_activity") + if (version >= 11) add("platoon_maintenance_state") } fun requiredColumns(version: Int): Map> = buildMap { @@ -93,5 +94,8 @@ internal object PlatoonSchema { setOf("occurred_at", "action_id", "member_name", "resolved_uid"), ) } + if (version >= 11) { + put("platoon_maintenance_state", setOf("key", "value")) + } } } diff --git a/app/src/main/res/drawable-nodpi/ic_launcher_art.png b/app/src/main/res/drawable-nodpi/ic_launcher_art.png index bdece90..aa2ee0e 100644 Binary files a/app/src/main/res/drawable-nodpi/ic_launcher_art.png and b/app/src/main/res/drawable-nodpi/ic_launcher_art.png differ diff --git a/app/src/main/res/mipmap-hdpi/ic_launcher.png b/app/src/main/res/mipmap-hdpi/ic_launcher.png index df8dcbe..4a61acc 100644 Binary files a/app/src/main/res/mipmap-hdpi/ic_launcher.png and b/app/src/main/res/mipmap-hdpi/ic_launcher.png differ diff --git a/app/src/main/res/mipmap-mdpi/ic_launcher.png b/app/src/main/res/mipmap-mdpi/ic_launcher.png index c63a464..4cbbae3 100644 Binary files a/app/src/main/res/mipmap-mdpi/ic_launcher.png and b/app/src/main/res/mipmap-mdpi/ic_launcher.png differ diff --git a/app/src/main/res/mipmap-xhdpi/ic_launcher.png b/app/src/main/res/mipmap-xhdpi/ic_launcher.png index 70c93af..19f62cb 100644 Binary files a/app/src/main/res/mipmap-xhdpi/ic_launcher.png and b/app/src/main/res/mipmap-xhdpi/ic_launcher.png differ diff --git a/app/src/main/res/mipmap-xxhdpi/ic_launcher.png b/app/src/main/res/mipmap-xxhdpi/ic_launcher.png index 11fce4a..1f418f7 100644 Binary files a/app/src/main/res/mipmap-xxhdpi/ic_launcher.png and b/app/src/main/res/mipmap-xxhdpi/ic_launcher.png differ diff --git a/app/src/main/res/mipmap-xxxhdpi/ic_launcher.png b/app/src/main/res/mipmap-xxxhdpi/ic_launcher.png index 5b47e3e..0ab93df 100644 Binary files a/app/src/main/res/mipmap-xxxhdpi/ic_launcher.png and b/app/src/main/res/mipmap-xxxhdpi/ic_launcher.png differ diff --git a/app/src/test/java/dev/gf2log/app/capture/GuildMembersCsvWriterTest.kt b/app/src/test/java/dev/gf2log/app/capture/GuildMembersCsvWriterTest.kt index fe6daae..f3441bc 100644 --- a/app/src/test/java/dev/gf2log/app/capture/GuildMembersCsvWriterTest.kt +++ b/app/src/test/java/dev/gf2log/app/capture/GuildMembersCsvWriterTest.kt @@ -231,24 +231,72 @@ class GuildMembersCsvWriterTest { assertTrue(output.listFiles().orEmpty().isEmpty()) } + @Test + fun oversizedLiveRosterIsRejectedBeforePublishingEvidence() { + val output = temporaryFolder.newFolder("oversized-live-roster") + val writer = GuildMembersCsvWriter(output) + val members = (1..(GuildMembersCsv.MAX_ROSTER_MEMBERS + 1)).map { index -> + member(index.toUInt(), "Member $index") + } + + val failure = runCatching { + writer.accept( + ParsedPayload( + messageId = 42, + payloadType = Gfl2PayloadDecoder.TYPE_GUILD_MEMBERS, + isEndOfMessage = true, + data = GuildMembersData(members), + ), + ) + } + + assertTrue(failure.exceptionOrNull() is IllegalStateException) + assertTrue(output.listFiles().orEmpty().isEmpty()) + } + + @Test + fun oversizedLiveMemberNameIsRejectedBeforePublishingEvidence() { + val output = temporaryFolder.newFolder("oversized-live-name") + val writer = GuildMembersCsvWriter(output) + + val failure = runCatching { + writer.accept( + ParsedPayload( + messageId = 42, + payloadType = Gfl2PayloadDecoder.TYPE_GUILD_MEMBERS, + isEndOfMessage = true, + data = GuildMembersData( + listOf( + member( + 1u, + "x".repeat(GuildMembersCsv.MAX_MEMBER_NAME_CHARS + 1), + ), + ), + ), + ), + ) + } + + assertTrue(failure.exceptionOrNull() is IllegalStateException) + assertTrue(output.listFiles().orEmpty().isEmpty()) + } + private fun payload(messageId: Int, uid: UInt, name: String, end: Boolean): ParsedPayload = ParsedPayload( messageId = messageId, payloadType = Gfl2PayloadDecoder.TYPE_GUILD_MEMBERS, isEndOfMessage = end, - data = GuildMembersData( - listOf( - GuildMember( - uid = uid, - name = name, - level = 60u, - weeklyMerit = 120u, - totalMerit = 4560u, - highScore = 789u, - totalScore = 1234u, - lastLogin = 1700000000u, - ), - ), - ), + data = GuildMembersData(listOf(member(uid, name))), ) + + private fun member(uid: UInt, name: String) = GuildMember( + uid = uid, + name = name, + level = 60u, + weeklyMerit = 120u, + totalMerit = 4560u, + highScore = 789u, + totalScore = 1234u, + lastLogin = 1700000000u, + ) } diff --git a/app/src/test/java/dev/gf2log/app/history/CaptureHistoryStoreTest.kt b/app/src/test/java/dev/gf2log/app/history/CaptureHistoryStoreTest.kt index e94fd9f..541d9db 100644 --- a/app/src/test/java/dev/gf2log/app/history/CaptureHistoryStoreTest.kt +++ b/app/src/test/java/dev/gf2log/app/history/CaptureHistoryStoreTest.kt @@ -60,6 +60,21 @@ class CaptureHistoryStoreTest { } } + @Test + fun refusesToReadAnOversizedLegacyHistoryEntry() { + val directory = Files.createTempDirectory("gf2log-history-bound-test").toFile() + try { + val file = directory.resolve("1_21917_1.txt") + file.writeBytes(ByteArray(CaptureHistoryStore.MAX_ENTRY_BYTES + 1)) + val store = CaptureHistoryStore(directory) + + assertEquals(1, store.list().size) + assertTrue(store.read(file.name) == null) + } finally { + directory.deleteRecursively() + } + } + private fun payload(index: Int): ParsedPayload = ParsedPayload( messageId = index, payloadType = 11138, diff --git a/app/src/test/java/dev/gf2log/app/management/GunsmokeWeekSolverTest.kt b/app/src/test/java/dev/gf2log/app/management/GunsmokeWeekSolverTest.kt index 5b030f2..d9393ad 100644 --- a/app/src/test/java/dev/gf2log/app/management/GunsmokeWeekSolverTest.kt +++ b/app/src/test/java/dev/gf2log/app/management/GunsmokeWeekSolverTest.kt @@ -11,6 +11,95 @@ class GunsmokeWeekSolverTest { private val zone = ZoneId.of("Asia/Seoul") private val start = LocalDate.of(2026, 7, 19) + @Test + fun cappedOpeningAnchorMustBelongToTheImmediatelyPrecedingCounterPeriod() { + val periodStart = start.atTime(5, 0).atZone(zone).toInstant() + + assertEquals( + true, + GunsmokeWeekSolver.isClosedOpeningAnchor( + capturedAt = periodStart.minusSeconds(5 * 86_400), + weeklyMerit = 540, + zoneId = zone, + periodStart = periodStart, + ), + ) + assertEquals( + false, + GunsmokeWeekSolver.isClosedOpeningAnchor( + capturedAt = periodStart.minusSeconds(7 * 86_400), + weeklyMerit = 540, + zoneId = zone, + periodStart = periodStart, + ), + ) + assertEquals( + true, + GunsmokeWeekSolver.isClosedOpeningAnchor( + capturedAt = periodStart.minusSeconds(10 * 60), + weeklyMerit = 90, + zoneId = zone, + periodStart = periodStart, + ), + ) + } + + @Test + fun cappedOpeningAnchorUsesCalendarBoundariesAcrossDaylightSavingChanges() { + val daylightZone = ZoneId.of("America/New_York") + val springStart = LocalDate.of(2026, 3, 8).atTime(5, 0).atZone(daylightZone).toInstant() + val fallStart = LocalDate.of(2026, 11, 1).atTime(5, 0).atZone(daylightZone).toInstant() + + assertEquals( + false, + GunsmokeWeekSolver.isClosedOpeningAnchor( + capturedAt = LocalDate.of(2026, 3, 2) + .atTime(4, 30) + .atZone(daylightZone) + .toInstant(), + weeklyMerit = 540, + zoneId = daylightZone, + periodStart = springStart, + ), + ) + assertEquals( + true, + GunsmokeWeekSolver.isClosedOpeningAnchor( + capturedAt = LocalDate.of(2026, 3, 2) + .atTime(5, 0) + .atZone(daylightZone) + .toInstant(), + weeklyMerit = 540, + zoneId = daylightZone, + periodStart = springStart, + ), + ) + assertEquals( + true, + GunsmokeWeekSolver.isClosedOpeningAnchor( + capturedAt = LocalDate.of(2026, 10, 26) + .atTime(5, 30) + .atZone(daylightZone) + .toInstant(), + weeklyMerit = 540, + zoneId = daylightZone, + periodStart = fallStart, + ), + ) + assertEquals( + false, + GunsmokeWeekSolver.isClosedOpeningAnchor( + capturedAt = LocalDate.of(2026, 10, 26) + .atTime(4, 30) + .atZone(daylightZone) + .toInstant(), + weeklyMerit = 540, + zoneId = daylightZone, + periodStart = fallStart, + ), + ) + } + @Test fun residualFiftyProvesPatrolAbsence() { val cells = solveTransition(patrolCredits = 0) diff --git a/app/src/test/java/dev/gf2log/app/management/PlatoonCsvImportStoreTest.kt b/app/src/test/java/dev/gf2log/app/management/PlatoonCsvImportStoreTest.kt index f591aa8..a9deb30 100644 --- a/app/src/test/java/dev/gf2log/app/management/PlatoonCsvImportStoreTest.kt +++ b/app/src/test/java/dev/gf2log/app/management/PlatoonCsvImportStoreTest.kt @@ -1,6 +1,8 @@ package dev.gf2log.app.management import java.io.ByteArrayInputStream +import java.time.Instant +import org.junit.Assert.assertEquals import org.junit.Assert.assertFalse import org.junit.Assert.assertTrue import org.junit.Rule @@ -37,6 +39,46 @@ class PlatoonCsvImportStoreTest { store.retain(ByteArrayInputStream(ByteArray(2 * 1024 * 1024 + 1))) } + @Test + fun duplicateUidCsvIsRejectedBeforeAnythingIsRetained() { + val directory = temporary.newFolder("duplicate-uids") + val store = PlatoonCsvImportStore(directory) + val duplicateUidCsv = VALID_CSV + + "\n1,Duplicate,59,1,2,3,4,5,2026-07-19T19:29:33Z" + + val failure = runCatching { + store.retain(ByteArrayInputStream(duplicateUidCsv.toByteArray())) + } + + assertTrue(failure.exceptionOrNull() is IllegalArgumentException) + assertEquals(emptyList(), directory.listFiles().orEmpty().map { it.name }) + } + + @Test + fun retainedImportUsesItsParsedCaptureTimeForLatestCsvOrdering() { + val directory = temporary.newFolder("capture-time") + val retained = PlatoonCsvImportStore(directory).retain( + ByteArrayInputStream(VALID_CSV.toByteArray()), + ) + + assertEquals( + Instant.parse("2026-07-19T19:29:33Z").toEpochMilli(), + retained.file.lastModified(), + ) + } + + @Test + fun latestRetainedFileBreaksEqualCaptureTimesBySourceFilename() { + val directory = temporary.newFolder("capture-time-tie") + val lower = java.io.File(directory, "import-same-a.csv").apply { writeText("lower") } + val higher = java.io.File(directory, "import-same-b.csv").apply { writeText("higher") } + val capturedAt = Instant.parse("2026-07-19T19:29:33Z").toEpochMilli() + assertTrue(lower.setLastModified(capturedAt)) + assertTrue(higher.setLastModified(capturedAt)) + + assertEquals(higher, PlatoonCsvImportStore.latestRetainedFile(directory)) + } + private companion object { val VALID_CSV = """ uid,name,level,weeklyMerit,totalMerit,highScore,totalScore,lastLogin,logTime diff --git a/app/src/test/java/dev/gf2log/app/management/PlatoonObservationPolicyTest.kt b/app/src/test/java/dev/gf2log/app/management/PlatoonObservationPolicyTest.kt index 61428f8..9eee7e7 100644 --- a/app/src/test/java/dev/gf2log/app/management/PlatoonObservationPolicyTest.kt +++ b/app/src/test/java/dev/gf2log/app/management/PlatoonObservationPolicyTest.kt @@ -17,6 +17,21 @@ class PlatoonObservationPolicyTest { ) } + @Test + fun `activity bounds untrusted names and observation count`() { + val observations = (1..(PlatoonObservationPolicy.MAX_ACTIVITY_OBSERVATIONS + 1)).map { + activity(name = "Member $it").copy(actionId = it.toLong()) + } + val oversizedName = activity( + name = "x".repeat(PlatoonObservationPolicy.MAX_ACTIVITY_MEMBER_NAME_LENGTH + 1), + ) + + val accepted = PlatoonObservationPolicy.activity(listOf(oversizedName) + observations) + + assertEquals(PlatoonObservationPolicy.MAX_ACTIVITY_OBSERVATIONS, accepted.size) + assertEquals(observations.take(PlatoonObservationPolicy.MAX_ACTIVITY_OBSERVATIONS), accepted) + } + @Test fun `updates reject empty member lists and deduplicate accepted observations`() { val valid = update( diff --git a/app/src/test/java/dev/gf2log/app/management/PlatoonSchemaTest.kt b/app/src/test/java/dev/gf2log/app/management/PlatoonSchemaTest.kt index 2327f66..16ea3e1 100644 --- a/app/src/test/java/dev/gf2log/app/management/PlatoonSchemaTest.kt +++ b/app/src/test/java/dev/gf2log/app/management/PlatoonSchemaTest.kt @@ -23,7 +23,9 @@ class PlatoonSchemaTest { assertTrue("weekly_overrides" in tables) assertTrue("platoon_activity" in tables) + assertTrue("platoon_maintenance_state" in tables) assertTrue("joined_date" in columns.getValue("membership_periods")) assertTrue("time_known" in columns.getValue("member_events")) + assertTrue("key" in columns.getValue("platoon_maintenance_state")) } } diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 9a4df04..d3ac618 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -66,13 +66,23 @@ Unknown payload types are skipped without allocation. A recognized but malformed ## Memory and concurrency limits -- Each active flow parser is capped at 2 MiB of buffered stream data. +- Each active flow parser is capped at 2 MiB of buffered stream data. A pending + logical payload is independently capped at 2 MiB and 64 continuation + fragments. The first continuation is checked before retention; exceeding + either limit discards the pending payload and quarantines its remaining + fragments through the logical terminal frame before later datasets are parsed. + Quarantine is advanced from frame metadata before protobuf decoding, so even + a malformed terminal fragment clears the discarded dataset deterministically. - The parser service has one worker and a queue capped at 256 payload chunks. - TLS, HTTP, and UDP payloads remain native and are excluded from the parser. - Outgoing plaintext chunks are used only for native flow classification. - Flow-close callbacks finalize any pending recognized payload before removing parser state. - Queue saturation is counted and surfaced in the capture status instead of being silently discarded. - Raw IP packets and application payloads are not persisted. +- One parsed history entry is capped at 2 MiB. The packet-table projection + accepts at most 512 KiB, 250 rows, 16 columns, 2,048 cells, and 8 KiB per + cell; rejected content remains available only through the already bounded raw + history view rather than constructing an unbounded Android view hierarchy. CSV retained as internal roster evidence preserves protocol text exactly. Explicit spreadsheet-facing roster, member, comparison, and weekly exports pass @@ -121,6 +131,18 @@ repository batches a conservative roster replay after ingestion: it derives only snapshot-supported presence spans, creates inactive historical identities, and preserves manual or exact Updates boundaries. +Roster CSV input is capped at 2 MiB, 256 members, 258 records, 9 columns, and +512 characters per field; UIDs must be unique and names are capped at 256 +characters. Activity ingestion accepts at most 250 distinct observations per +payload with 128-character names. SQLite retains the newest 10,000 activity +facts, and one ingestion resolves at most 250 unresolved facts. A database-local +cursor rotates those bounded batches through the complete retained backlog. +Schema v11 owns the rotation cursor and a global `(captured_at, id)` retention +index; supported v1-v10 backups are upgraded before strict current-schema +validation. +These limits are enforced again at repository and persistence boundaries so a +future caller cannot bypass the protocol-layer checks. + ## HTTPS and application-layer encryption An Android VPN can observe packet metadata, but it cannot automatically read diff --git a/docs/PLATOON_MANAGEMENT.md b/docs/PLATOON_MANAGEMENT.md index d1ae430..30153a6 100644 --- a/docs/PLATOON_MANAGEMENT.md +++ b/docs/PLATOON_MANAGEMENT.md @@ -70,6 +70,10 @@ the attempt count and the otherwise ambiguous per-attempt rounding merit. Counter resets are handled by treating a negative delta as the current counter value. Daily activity is derived from Total Merit because Merit This Week resets on Monday while a Gunsmoke week begins on Sunday. +A capped pre-Gunsmoke counter is a valid opening anchor only inside the +immediately preceding Monday 05:00-to-Sunday 05:00 calendar interval in the +configured game-day timezone; this remains correct across daylight-saving +transitions. ## Snapshots and weekly tables @@ -140,6 +144,10 @@ resets on Monday while a Gunsmoke week begins on Sunday. - A roster CSV imported after newer captures still contributes historical presence spans. The replay creates missing inactive members and inferred withdrawal/rejoin periods without replacing manual or exact Updates evidence. + An observed absence can add or refine the missing inferred withdrawal side of + an exact open period, after which a later presence becomes a separate rejoin. + Notes on snapshot-derived periods survive this replay, while their weak + inferred boundaries can still expand to roster evidence discovered later. - An individual membership period can be deleted from its editor after a destructive-action confirmation. The only remaining period is protected; delete the member instead when the complete record should be removed. @@ -166,3 +174,16 @@ CSV files created for spreadsheet use neutralize string cells beginning with `=`, `+`, `-`, or `@` (including after leading whitespace), preventing a member name or note from executing as a spreadsheet formula. Internal retained roster evidence remains lossless and is not rewritten with that export-only prefix. +The same neutralization is applied to captured roster, activity, Updates, and +formation packet tables before they can be copied or opened in spreadsheet +software. + +Untrusted evidence is bounded at every persistence boundary. A roster import +must contain no more than 256 unique UIDs, an activity payload contributes at +most 250 distinct observations, and the database keeps the newest 10,000 +activity facts. If a newly selected roster CSV fails parsing, duplicate-UID +validation, ingestion, or chronological reconciliation, its newly retained file +is deleted so a failed import cannot poison every later reconciliation. The +entire selected-file reconciliation runs under one outer SQLite transaction, so +deleting newly retained files after failure also rolls back nested snapshot and +membership writes. diff --git a/protocol/src/main/java/dev/gf2log/protocol/Gfl2StreamParser.kt b/protocol/src/main/java/dev/gf2log/protocol/Gfl2StreamParser.kt index 9d47f6a..3a67978 100644 --- a/protocol/src/main/java/dev/gf2log/protocol/Gfl2StreamParser.kt +++ b/protocol/src/main/java/dev/gf2log/protocol/Gfl2StreamParser.kt @@ -14,9 +14,17 @@ import dev.gf2log.protocol.model.WeaponsData class Gfl2StreamParser( maximumBufferedBytes: Int = DEFAULT_MAXIMUM_BUFFERED_BYTES, + private val maximumPendingPayloadBytes: Int = DEFAULT_MAXIMUM_PENDING_PAYLOAD_BYTES, + private val maximumPendingContinuations: Int = DEFAULT_MAXIMUM_PENDING_CONTINUATIONS, ) { + init { + require(maximumPendingPayloadBytes > 0) { "Pending payload byte limit must be positive" } + require(maximumPendingContinuations > 0) { "Pending continuation limit must be positive" } + } + private val buffer = ByteAccumulator(maximumBufferedBytes) private var pendingPayload: PendingPayload? = null + private var quarantinedPayload: QuarantinedPayload? = null fun accept(bytes: ByteArray): List { val events = mutableListOf() @@ -48,6 +56,7 @@ class Gfl2StreamParser( fun finish(): List { val events = mutableListOf() emitPending(events) + quarantinedPayload = null buffer.clear() return events } @@ -55,6 +64,7 @@ class Gfl2StreamParser( fun reset() { buffer.clear() pendingPayload = null + quarantinedPayload = null } private fun parseMessage( @@ -82,6 +92,10 @@ class Gfl2StreamParser( val isEndOfMessage = offset + payloadSize >= message.size if (type in Gfl2PayloadDecoder.supportedTypes) { + if (consumeQuarantinedPayload(type, messageId, isEndOfMessage)) { + offset += payloadSize + continue + } val payload = message.copyOfRange( offset + PAYLOAD_HEADER_SIZE, offset + payloadSize, @@ -89,7 +103,14 @@ class Gfl2StreamParser( try { val decoded = Gfl2PayloadDecoder.decode(type, payload) if (decoded != null) { - acceptDecodedPayload(messageId, type, isEndOfMessage, decoded, events) + acceptDecodedPayload( + messageId, + type, + isEndOfMessage, + payload.size, + decoded, + events, + ) } } catch (error: ProtocolException) { events += ParseEvent.Warning( @@ -98,6 +119,7 @@ class Gfl2StreamParser( } } else { emitPending(events) + quarantinedPayload = null events += ParseEvent.UnknownPayload( messageId = messageId, payloadType = type, @@ -114,6 +136,7 @@ class Gfl2StreamParser( messageId: Int, type: Int, isEndOfMessage: Boolean, + payloadBytes: Int, data: GameData, events: MutableList, ) { @@ -122,9 +145,28 @@ class Gfl2StreamParser( if (pending.type == type && (pending.previousMessageId == 0 || pending.previousMessageId == messageId) ) { + val accumulatedBytes = pending.accumulatedPayloadBytes + payloadBytes.toLong() + val continuationCount = pending.continuationCount + 1 + if ( + accumulatedBytes > maximumPendingPayloadBytes || + continuationCount > maximumPendingContinuations + ) { + pendingPayload = null + quarantinedPayload = if (messageId != 0 && isEndOfMessage) { + null + } else { + QuarantinedPayload(type, messageId) + } + events += ParseEvent.Warning( + "Discarded payload $type continuation sequence exceeding parser limits", + ) + return + } pending.data = merge(pending.data, data) pending.previousMessageId = messageId pending.isEndOfMessage = isEndOfMessage + pending.accumulatedPayloadBytes = accumulatedBytes + pending.continuationCount = continuationCount if (messageId != 0 && isEndOfMessage) emitPending(events) return } @@ -134,10 +176,46 @@ class Gfl2StreamParser( if (messageId != 0 && isEndOfMessage) { events += ParseEvent.Payload(ParsedPayload(messageId, type, true, data)) } else { - pendingPayload = PendingPayload(type, messageId, isEndOfMessage, data) + if (payloadBytes > maximumPendingPayloadBytes) { + quarantinedPayload = if (messageId != 0 && isEndOfMessage) { + null + } else { + QuarantinedPayload(type, messageId) + } + events += ParseEvent.Warning( + "Discarded payload $type continuation sequence exceeding parser limits", + ) + return + } + pendingPayload = PendingPayload( + type = type, + previousMessageId = messageId, + isEndOfMessage = isEndOfMessage, + data = data, + accumulatedPayloadBytes = payloadBytes.toLong(), + continuationCount = 1, + ) } } + private fun consumeQuarantinedPayload( + type: Int, + messageId: Int, + isEndOfMessage: Boolean, + ): Boolean { + val quarantined = quarantinedPayload ?: return false + if ( + quarantined.type == type && + (quarantined.previousMessageId == 0 || quarantined.previousMessageId == messageId) + ) { + quarantined.previousMessageId = messageId + if (messageId != 0 && isEndOfMessage) quarantinedPayload = null + return true + } + quarantinedPayload = null + return false + } + private fun emitPending(events: MutableList) { val pending = pendingPayload ?: return events += ParseEvent.Payload( @@ -179,6 +257,8 @@ class Gfl2StreamParser( const val PAYLOAD_HEADER_SIZE = 4 const val MAXIMUM_MESSAGE_SIZE = 65_540 const val DEFAULT_MAXIMUM_BUFFERED_BYTES = 2 * 1024 * 1024 + const val DEFAULT_MAXIMUM_PENDING_PAYLOAD_BYTES = 2 * 1024 * 1024 + const val DEFAULT_MAXIMUM_PENDING_CONTINUATIONS = 64 } private data class PendingPayload( @@ -186,5 +266,12 @@ class Gfl2StreamParser( var previousMessageId: Int, var isEndOfMessage: Boolean, var data: GameData, + var accumulatedPayloadBytes: Long, + var continuationCount: Int, + ) + + private data class QuarantinedPayload( + val type: Int, + var previousMessageId: Int, ) } diff --git a/protocol/src/main/java/dev/gf2log/protocol/GuildMembersCsv.kt b/protocol/src/main/java/dev/gf2log/protocol/GuildMembersCsv.kt index e9b221b..9da9ef8 100644 --- a/protocol/src/main/java/dev/gf2log/protocol/GuildMembersCsv.kt +++ b/protocol/src/main/java/dev/gf2log/protocol/GuildMembersCsv.kt @@ -17,6 +17,18 @@ object GuildMembersCsv { logTime, ).joinToString(",") { CsvCell.escape(it, spreadsheetSafe = false) } + internal fun rowForSpreadsheet(member: GuildMember, logTime: String): String = listOf( + member.uid, + member.name, + member.level, + member.weeklyMerit, + member.totalMerit, + member.highScore, + member.totalScore, + member.lastLogin, + logTime, + ).joinToString(",") { CsvCell.escape(it) } + // Function Name: formatForSpreadsheet // Description: // - Formats a parsed roster for explicit export without allowing names to execute formulas. @@ -28,22 +40,19 @@ object GuildMembersCsv { fun formatForSpreadsheet(snapshot: Snapshot): String = buildString { appendLine(HEADER) snapshot.members.forEach { member -> - appendLine( - listOf( - member.uid, - member.name, - member.level, - member.weeklyMerit, - member.totalMerit, - member.highScore, - member.totalScore, - member.lastLogin, - snapshot.logTime, - ).joinToString(",") { CsvCell.escape(it) }, - ) + appendLine(rowForSpreadsheet(member, snapshot.logTime)) } }.trimEnd() + "\n" + fun hasValidMemberBounds(members: List): Boolean = + members.size <= MAX_ROSTER_MEMBERS && + members.all { it.name.length <= MAX_MEMBER_NAME_CHARS } + + fun isValidRoster(members: List): Boolean = + members.isNotEmpty() && + hasValidMemberBounds(members) && + members.map(GuildMember::uid).distinct().size == members.size + // Function Name: parse // Description: // - Parses a complete roster snapshot while preserving member names exactly. @@ -54,11 +63,13 @@ object GuildMembersCsv { // - Returns the parsed snapshot when the schema and rows are valid. // - Returns null when any required field is malformed or capture times disagree. fun parse(content: String): Snapshot? { + if (content.length > MAX_CONTENT_CHARS) return null val records = parseRecords(content) ?: return null val header = records.firstOrNull() ?: return null if (header != HEADER.split(',')) return null var logTime: String? = null + val seenUids = mutableSetOf() val members = records.drop(1) .filterNot { row -> row.all(String::isBlank) } .map { row -> @@ -66,8 +77,11 @@ object GuildMembersCsv { val rowLogTime = row[8].trim().takeIf(String::isNotBlank) ?: return null if (logTime == null) logTime = rowLogTime if (rowLogTime != logTime) return null + val uid = row[0].trim().toUIntOrNull() ?: return null + if (!seenUids.add(uid)) return null + if (row[1].length > MAX_MEMBER_NAME_CHARS) return null GuildMember( - uid = row[0].trim().toUIntOrNull() ?: return null, + uid = uid, name = row[1], level = row[2].trim().toUIntOrNull() ?: return null, weeklyMerit = parseOptionalCounter(row[3]) ?: return null, @@ -77,7 +91,7 @@ object GuildMembersCsv { lastLogin = row[7].trim().toUIntOrNull() ?: return null, ) } - if (members.isEmpty() || logTime == null) return null + if (!isValidRoster(members) || logTime == null) return null return Snapshot(logTime = logTime!!, members = members) } @@ -102,25 +116,33 @@ object GuildMembersCsv { ',' -> if (quoted) { field.append(character) } else { + if (row.size >= MAX_COLUMNS || field.length > MAX_FIELD_CHARS) return null row += field.toString() field.clear() } '\n' -> if (quoted) { field.append(character) } else { + if (row.size >= MAX_COLUMNS || field.length > MAX_FIELD_CHARS) return null row += field.toString() field.clear() + if (records.size >= MAX_RECORDS) return null records += row.toList() row.clear() } - else -> field.append(character) + else -> { + if (field.length >= MAX_FIELD_CHARS) return null + field.append(character) + } } index += 1 } if (quoted) return null if (field.isNotEmpty() || row.isNotEmpty()) { + if (row.size >= MAX_COLUMNS || field.length > MAX_FIELD_CHARS) return null row += field.toString() + if (records.size >= MAX_RECORDS) return null records += row.toList() } return records @@ -143,4 +165,11 @@ object GuildMembersCsv { val logTime: String, val members: List, ) + + private const val MAX_CONTENT_CHARS = 2 * 1024 * 1024 + const val MAX_ROSTER_MEMBERS = 256 + const val MAX_MEMBER_NAME_CHARS = 256 + private const val MAX_COLUMNS = 9 + private const val MAX_RECORDS = MAX_ROSTER_MEMBERS + 2 + private const val MAX_FIELD_CHARS = 512 } diff --git a/protocol/src/main/java/dev/gf2log/protocol/ParsedPacketTableParser.kt b/protocol/src/main/java/dev/gf2log/protocol/ParsedPacketTableParser.kt index 4a1cb67..7d97e58 100644 --- a/protocol/src/main/java/dev/gf2log/protocol/ParsedPacketTableParser.kt +++ b/protocol/src/main/java/dev/gf2log/protocol/ParsedPacketTableParser.kt @@ -2,15 +2,21 @@ package dev.gf2log.protocol object ParsedPacketTableParser { fun parse(content: String): Table? { + if (content.length > MAX_CONTENT_CHARS) return null val normalized = content.replace("\r\n", "\n").replace('\r', '\n') val separator = normalized.indexOf("\n\n") if (separator < 0 || separator + 2 >= normalized.length) return null val records = parseCsv(normalized.substring(separator + 2)) ?: return null - val header = records.firstOrNull()?.takeIf { it.isNotEmpty() } ?: return null + val header = records.firstOrNull() + ?.takeIf { it.isNotEmpty() && it.size <= MAX_COLUMNS } + ?: return null + val rows = records.drop(1) + if (rows.size > MAX_ROWS || rows.any { it.size > header.size }) return null + if ((rows.size + 1) * header.size > MAX_CELLS) return null return Table( header = header, - rows = records.drop(1).map { row -> + rows = rows.map { row -> if (row.size >= header.size) row else row + List(header.size - row.size) { "" } }, ) @@ -34,23 +40,31 @@ object ParsedPacketTableParser { } } ',' -> if (quoted) field.append(character) else { + if (row.size >= MAX_COLUMNS || field.length > MAX_CELL_CHARS) return null row += field.toString() field.clear() } '\n' -> if (quoted) field.append(character) else { + if (row.size >= MAX_COLUMNS || field.length > MAX_CELL_CHARS) return null row += field.toString() field.clear() + if (records.size >= MAX_RECORDS) return null records += row.toList() row.clear() } - else -> field.append(character) + else -> { + if (field.length >= MAX_CELL_CHARS) return null + field.append(character) + } } index += 1 } if (quoted) return null if (field.isNotEmpty() || row.isNotEmpty()) { + if (row.size >= MAX_COLUMNS || field.length > MAX_CELL_CHARS) return null row += field.toString() + if (records.size >= MAX_RECORDS) return null records += row.toList() } return records @@ -60,4 +74,11 @@ object ParsedPacketTableParser { val header: List, val rows: List>, ) + + const val MAX_CONTENT_CHARS = 512 * 1024 + const val MAX_ROWS = 250 + const val MAX_COLUMNS = 16 + const val MAX_CELLS = 2_048 + const val MAX_CELL_CHARS = 8 * 1024 + private const val MAX_RECORDS = MAX_ROWS + 1 } diff --git a/protocol/src/main/java/dev/gf2log/protocol/ParsedPayloadTextFormatter.kt b/protocol/src/main/java/dev/gf2log/protocol/ParsedPayloadTextFormatter.kt index cd91553..933cfbb 100644 --- a/protocol/src/main/java/dev/gf2log/protocol/ParsedPayloadTextFormatter.kt +++ b/protocol/src/main/java/dev/gf2log/protocol/ParsedPayloadTextFormatter.kt @@ -19,7 +19,9 @@ object ParsedPayloadTextFormatter { when (val data = payload.data) { is GuildMembersData -> { appendLine(GuildMembersCsv.HEADER) - data.members.forEach { appendLine(GuildMembersCsv.row(it, capturedAt)) } + data.members.forEach { + appendLine(GuildMembersCsv.rowForSpreadsheet(it, capturedAt)) + } } is PlatoonActivityData -> { appendLine("recordType,id,kind,occurredAt,actionId,count,memberName") @@ -29,7 +31,7 @@ object ParsedPayloadTextFormatter { data.entries.forEach { appendLine( "entry,,${it.kind},${it.occurredAt},${it.actionId},," + - csvEscape(it.memberName), + CsvCell.escape(it.memberName), ) } } @@ -39,7 +41,7 @@ object ParsedPayloadTextFormatter { entry.members.forEachIndexed { index, member -> appendLine( "${entry.kind},${entry.occurredAt},$index,${member.role}," + - "${member.uid},${csvEscape(member.name)}", + "${member.uid},${CsvCell.escape(member.name)}", ) } } @@ -74,7 +76,7 @@ object ParsedPayloadTextFormatter { formation.dolls.forEach { doll -> appendLine( listOf( - csvEscape(formation.name), + CsvCell.escape(formation.name), doll.dollId, doll.weaponUid, doll.attachmentUids.joinToString("|"), @@ -89,8 +91,4 @@ object ParsedPayloadTextFormatter { } }.trimEnd() - private fun csvEscape(value: String): String { - if (value.none { it == ',' || it == '"' || it == '\n' || it == '\r' }) return value - return "\"${value.replace("\"", "\"\"")}\"" - } } diff --git a/protocol/src/test/java/dev/gf2log/protocol/CsvCellTest.kt b/protocol/src/test/java/dev/gf2log/protocol/CsvCellTest.kt index 512d0e6..db2a549 100644 --- a/protocol/src/test/java/dev/gf2log/protocol/CsvCellTest.kt +++ b/protocol/src/test/java/dev/gf2log/protocol/CsvCellTest.kt @@ -6,7 +6,15 @@ import org.junit.Test class CsvCellTest { @Test fun neutralizesEverySpreadsheetFormulaPrefixIncludingLeadingWhitespace() { - listOf("=1+1", "+cmd", "-2+3", "@SUM(A1)", " =HYPERLINK(\"x\")").forEach { value -> + listOf( + "=1+1", + "+cmd", + "-2+3", + "@SUM(A1)", + " =HYPERLINK(\"x\")", + "\t=WEBSERVICE(\"x\")", + "\r@SUM(A1)", + ).forEach { value -> val escaped = CsvCell.escape(value) val decoded = escaped.removeSurrounding("\"").replace("\"\"", "\"") diff --git a/protocol/src/test/java/dev/gf2log/protocol/Gfl2StreamParserTest.kt b/protocol/src/test/java/dev/gf2log/protocol/Gfl2StreamParserTest.kt index 28bf51e..e00efb8 100644 --- a/protocol/src/test/java/dev/gf2log/protocol/Gfl2StreamParserTest.kt +++ b/protocol/src/test/java/dev/gf2log/protocol/Gfl2StreamParserTest.kt @@ -194,6 +194,118 @@ class Gfl2StreamParserTest { assertTrue(parser.finish().isEmpty()) } + @Test + fun continuationCountOverflowQuarantinesTheTailAndParserRecovers() { + val parser = Gfl2StreamParser(maximumPendingContinuations = 2) + val continuation = outerMessage( + 0, + payload(Gfl2PayloadDecoder.TYPE_GUILD_MEMBERS, guildMembersPayload("Part", 8uL)), + ) + + assertTrue(parser.accept(continuation).isEmpty()) + assertTrue(parser.accept(continuation).isEmpty()) + val overflow = parser.accept(continuation) + + assertEquals(1, overflow.filterIsInstance().size) + assertTrue(parser.accept(continuation).isEmpty()) + assertTrue( + parser.accept( + outerMessage( + 42, + payload( + Gfl2PayloadDecoder.TYPE_GUILD_MEMBERS, + guildMembersPayload("Truncated tail", 10uL), + ), + ), + ).isEmpty(), + ) + val recovered = parser.accept( + outerMessage( + 7, + payload(Gfl2PayloadDecoder.TYPE_GUILD_MEMBERS, guildMembersPayload("Fresh", 9uL)), + ), + ).singlePayload() + assertEquals("Fresh", (recovered.value.data as GuildMembersData).members.single().name) + } + + @Test + fun continuationByteOverflowQuarantinesTheTail() { + val body = guildMembersPayload("Part", 8uL) + val parser = Gfl2StreamParser(maximumPendingPayloadBytes = body.size * 2 - 1) + val continuation = outerMessage( + 0, + payload(Gfl2PayloadDecoder.TYPE_GUILD_MEMBERS, body), + ) + + assertTrue(parser.accept(continuation).isEmpty()) + assertEquals(1, parser.accept(continuation).filterIsInstance().size) + assertTrue(parser.accept(continuation).isEmpty()) + assertTrue( + parser.accept( + outerMessage( + 42, + payload(Gfl2PayloadDecoder.TYPE_GUILD_MEMBERS, body), + ), + ).isEmpty(), + ) + assertTrue(parser.finish().isEmpty()) + } + + @Test + fun oversizedFirstContinuationIsQuarantinedBeforePendingStateIsCreated() { + val body = guildMembersPayload("Oversized first", 8uL) + val parser = Gfl2StreamParser(maximumPendingPayloadBytes = body.size - 1) + + val warning = parser.accept( + outerMessage( + 0, + payload(Gfl2PayloadDecoder.TYPE_GUILD_MEMBERS, body), + ), + ) + + assertEquals(1, warning.filterIsInstance().size) + assertTrue( + parser.accept( + outerMessage( + 42, + payload(Gfl2PayloadDecoder.TYPE_GUILD_MEMBERS, body), + ), + ).isEmpty(), + ) + assertTrue(parser.finish().isEmpty()) + } + + @Test + fun malformedTerminalFrameClearsQuarantineBeforeDecoding() { + val parser = Gfl2StreamParser(maximumPendingContinuations = 1) + val continuation = outerMessage( + 0, + payload(Gfl2PayloadDecoder.TYPE_GUILD_MEMBERS, guildMembersPayload("Part", 8uL)), + ) + + assertTrue(parser.accept(continuation).isEmpty()) + assertEquals(1, parser.accept(continuation).filterIsInstance().size) + assertTrue( + parser.accept( + outerMessage( + 42, + payload(Gfl2PayloadDecoder.TYPE_GUILD_MEMBERS, byteArrayOf(0x0A, 0x7F)), + ), + ).isEmpty(), + ) + + val recovered = parser.accept( + outerMessage( + 43, + payload( + Gfl2PayloadDecoder.TYPE_GUILD_MEMBERS, + guildMembersPayload("Recovered", 9uL), + ), + ), + ).singlePayload() + assertEquals("Recovered", (recovered.value.data as GuildMembersData).members.single().name) + } + @Test fun unknownPayloadsRemainObservableWithoutRetainingTheirBytes() { val parser = Gfl2StreamParser() diff --git a/protocol/src/test/java/dev/gf2log/protocol/GuildMembersCsvTest.kt b/protocol/src/test/java/dev/gf2log/protocol/GuildMembersCsvTest.kt index 00cfbc4..39c81bf 100644 --- a/protocol/src/test/java/dev/gf2log/protocol/GuildMembersCsvTest.kt +++ b/protocol/src/test/java/dev/gf2log/protocol/GuildMembersCsvTest.kt @@ -81,6 +81,17 @@ class GuildMembersCsvTest { assertNull(GuildMembersCsv.parse(content)) } + @Test + fun rejectsDuplicateUidsBeforeASelectedRosterCanBeRetained() { + val content = """ + ${GuildMembersCsv.HEADER} + 42,First,60,1,2,3,4,5,2026-07-21T19:11:09Z + 42,Duplicate,59,6,7,8,9,10,2026-07-21T19:11:09Z + """.trimIndent() + + assertNull(GuildMembersCsv.parse(content)) + } + @Test fun trimsSpreadsheetWhitespaceFromScalarFieldsWithoutChangingTheName() { val content = listOf( diff --git a/protocol/src/test/java/dev/gf2log/protocol/ParsedPacketTableParserTest.kt b/protocol/src/test/java/dev/gf2log/protocol/ParsedPacketTableParserTest.kt index e6b6b57..4c62db4 100644 --- a/protocol/src/test/java/dev/gf2log/protocol/ParsedPacketTableParserTest.kt +++ b/protocol/src/test/java/dev/gf2log/protocol/ParsedPacketTableParserTest.kt @@ -29,4 +29,39 @@ class ParsedPacketTableParserTest { fun rejectsContentWithoutTheMetadataSeparator() { assertNull(ParsedPacketTableParser.parse("uid,name\n1,Commander")) } + + @Test + fun rejectsTablesWhoseRowCountWouldCreateUnboundedViews() { + val rows = List(ParsedPacketTableParser.MAX_ROWS + 1) { "$it,Member" } + val content = metadata() + (listOf("uid,name") + rows).joinToString("\n") + + assertNull(ParsedPacketTableParser.parse(content)) + } + + @Test + fun rejectsTablesWhoseColumnOrCellBudgetsWouldCreateUnboundedViews() { + val tooManyColumns = List(ParsedPacketTableParser.MAX_COLUMNS + 1) { "column$it" } + .joinToString(",") + val wideContent = metadata() + "$tooManyColumns\n" + val header = List(16) { "c$it" }.joinToString(",") + val rows = List(128) { index -> List(16) { "$index" }.joinToString(",") } + val tooManyCells = metadata() + (listOf(header) + rows).joinToString("\n") + + assertNull(ParsedPacketTableParser.parse(wideContent)) + assertNull(ParsedPacketTableParser.parse(tooManyCells)) + } + + @Test + fun rejectsOversizedHistoryAndIndividualCells() { + val largeHistory = metadata() + "a\n" + + "x".repeat(ParsedPacketTableParser.MAX_CONTENT_CHARS) + val largeCell = metadata() + "a\n" + + "x".repeat(ParsedPacketTableParser.MAX_CELL_CHARS + 1) + + assertNull(ParsedPacketTableParser.parse(largeHistory)) + assertNull(ParsedPacketTableParser.parse(largeCell)) + } + + private fun metadata(): String = + "capturedAt=2026-08-08T00:00:00Z\nmessageId=1\npayloadType=21917\n\n" } diff --git a/protocol/src/test/java/dev/gf2log/protocol/ParsedPayloadTextFormatterTest.kt b/protocol/src/test/java/dev/gf2log/protocol/ParsedPayloadTextFormatterTest.kt index 40164d4..f4ede3e 100644 --- a/protocol/src/test/java/dev/gf2log/protocol/ParsedPayloadTextFormatterTest.kt +++ b/protocol/src/test/java/dev/gf2log/protocol/ParsedPayloadTextFormatterTest.kt @@ -33,6 +33,22 @@ class ParsedPayloadTextFormatterTest { assertTrue(text.contains(GuildMembersCsv.HEADER)) } + @Test + fun guildPacketHistoryNeutralizesFormulaLikeNamesWithoutChangingRetainedRows() { + val formulaMember = member(1u, "=HYPERLINK(\"https://invalid\")") + val payload = ParsedPayload( + messageId = 7, + payloadType = Gfl2PayloadDecoder.TYPE_GUILD_MEMBERS, + isEndOfMessage = true, + data = GuildMembersData(listOf(formulaMember)), + ) + + val history = ParsedPayloadTextFormatter.format(payload, "2026-08-08T00:00:00Z") + + assertTrue(history.contains("1,\"'=HYPERLINK(\"\"https://invalid\"\")\"")) + assertTrue(GuildMembersCsv.row(formulaMember, "2026-08-08T00:00:00Z").contains(",\"=HYPERLINK")) + } + @Test fun platoonActivityPacketFormatsRawActionEvidence() { val payload = ParsedPayload( @@ -74,6 +90,42 @@ class ParsedPayloadTextFormatterTest { assertTrue(text.contains("3,1700300000,0,1,3333333,\"Name,WithComma\"")) } + @Test + fun packetHistoryNeutralizesFormulaLikeActivityAndUpdateNames() { + val activity = ParsedPayload( + messageId = 10, + payloadType = Gfl2PayloadDecoder.TYPE_PLATOON_ACTIVITY, + isEndOfMessage = true, + data = PlatoonActivityData( + summaries = emptyList(), + entries = listOf(PlatoonActivityEntry(1u, 123u, 802001u, "=1+1")), + ), + ) + val updates = ParsedPayload( + messageId = 11, + payloadType = Gfl2PayloadDecoder.TYPE_PLATOON_UPDATES, + isEndOfMessage = true, + data = PlatoonUpdatesData( + listOf( + PlatoonUpdateEntry( + kind = 3u, + members = listOf(PlatoonUpdateMember(1u, 42u, "\t@SUM(A1)")), + occurredAt = 123u, + ), + ), + ), + ) + + assertTrue( + ParsedPayloadTextFormatter.format(activity, "2026-08-08T00:00:00Z") + .contains("entry,,1,123,802001,,'=1+1"), + ) + assertTrue( + ParsedPayloadTextFormatter.format(updates, "2026-08-08T00:00:00Z") + .contains("3,123,0,1,42,'\t@SUM(A1)"), + ) + } + private fun member(uid: UInt, name: String) = GuildMember( uid = uid, name = name,