From abb4d1e8e8dc4667d4f47ee22ce4be68e7fc9150 Mon Sep 17 00:00:00 2001 From: BRUNER Patrick Date: Thu, 25 Jun 2026 14:40:41 +0200 Subject: [PATCH 1/3] Handle all line endings exactly in Direct stream reader PositionAwareStreamReaderDirect now detects the actual terminator per line instead of guessing one constant for the whole file. It scans with IndexOfAny(SearchValues.Create("\r\n")) and classifies each hit as \n, \r\n, or a bare \r (classic-Mac), including the \r-at-block-boundary straddle. Byte Position advances by the content bytes plus the real terminator's bytes, so it stays exact on files that interleave line endings. This folds the two capabilities that previously only System/Legacy had into the default reader: - bare \r no longer renders a classic-Mac file as one giant line - Position no longer drifts on mixed \n/\r\n files, keeping seeks into flushed buffers correct The guessed _newLineSequenceLength field and GuessNewLineSequenceLength (with its seek-reset-reread) are replaced by a lazy EnsureInitialized that fills the first block from the current stream position, plus a ResetReader override that resets scan state on a mid-stream seek (also fixing a latent stale-block scan after a Position change). Tests: 11 new TDD cases covering bare \r, mixed endings, repeated terminators, trailing \r and \r\n, \n\r, multibyte exact positions, and \r / \r\n landing exactly on the 32 KB block boundary. Full reader suite green (97 tests); Direct throughput benchmark shows no regression. Adds byte-exact manual/GUI fixtures (LF, CRLF, CR, Mixed) under TestData with a regenerator and README, pinned `binary` in .gitattributes so core.autocrlf cannot corrupt their terminators. --- .gitattributes | 9 + .github/CLAUDE.md | 61 ++++++- CONTEXT.md | 155 ++++++++++++++++++ .../PositionAwareStreamReaderDirect.cs | 146 +++++++++-------- src/LogExpert.Tests/LogExpert.Tests.csproj | 15 ++ .../PositionAwareStreamReaderDirectTests.cs | 153 ++++++++++++++++- .../TestData/LineEndings_CR.txt | 1 + .../TestData/LineEndings_CRLF.txt | 7 + .../TestData/LineEndings_LF.txt | 7 + .../TestData/LineEndings_Mixed.txt | 5 + .../TestData/LineEndings_README.md | 24 +++ 11 files changed, 506 insertions(+), 77 deletions(-) create mode 100644 CONTEXT.md create mode 100644 src/LogExpert.Tests/TestData/LineEndings_CR.txt create mode 100644 src/LogExpert.Tests/TestData/LineEndings_CRLF.txt create mode 100644 src/LogExpert.Tests/TestData/LineEndings_LF.txt create mode 100644 src/LogExpert.Tests/TestData/LineEndings_Mixed.txt create mode 100644 src/LogExpert.Tests/TestData/LineEndings_README.md diff --git a/.gitattributes b/.gitattributes index 56b578ee..8ba9a84c 100644 --- a/.gitattributes +++ b/.gitattributes @@ -5,3 +5,12 @@ # (Pattern has no slash so it matches the file regardless of path — its directory # "Solution Items" contains a space, which .gitattributes cannot express in a path.) usedComponents.json text eol=lf + +# Line-ending test fixtures (src/LogExpert.Tests/TestData). Their whole point is +# byte-exact terminators (bare \r, mixed \n/\r\n/\r), so git must NOT normalize +# EOLs on them — otherwise core.autocrlf would rewrite \r\n<->\n and corrupt the +# fixtures. Mark binary to store and check out the exact bytes. +LineEndings_LF.txt binary +LineEndings_CRLF.txt binary +LineEndings_CR.txt binary +LineEndings_Mixed.txt binary diff --git a/.github/CLAUDE.md b/.github/CLAUDE.md index da1bb1a1..131ea004 100644 --- a/.github/CLAUDE.md +++ b/.github/CLAUDE.md @@ -13,6 +13,26 @@ LogExpert is a Windows log file viewer and analyzer built with C# and Windows Fo - NUnit for testing - Plugin-based architecture +## Environment + +This is a Windows .NET/WinForms repo. Python is often unavailable — prefer PowerShell, `tshark`, and built-in tooling. Check whether `gh` CLI (and any other external tool) is actually installed before building a plan around it. The active GitHub account is `brunerpat`. + +### Line Endings + +The working tree is CRLF on Windows (only `usedComponents.json` is pinned to LF via [.gitattributes](.gitattributes); there is no global normalization). Never use `sed`-based or other text-tool renames that strip line endings — they produce a whole-file diff of pure line-ending churn. After any bulk rename, verify no references were left unrenamed (a missed reference will break the build). + +## Testing + +Follow TDD for all bug fixes: write a failing test that reproduces the issue first, then fix, then verify the **full suite** passes — not just the new test in isolation (timing/filesystem tests can pass alone but fail alongside the suite). + +## Documentation + +Docs must be code-grounded and reconciled against the actual source and live config, then reader-tested before delivery. Don't document from assumption. + +## Refactoring + +For "behavior-preserving" refactors, explicitly confirm equivalence and flag any unintended logic changes before applying. + ## Build Commands ### Using Nuke Build (Recommended) @@ -31,6 +51,15 @@ LogExpert is a Windows log file viewer and analyzer built with C# and Windows Fo ./build.ps1 --target Clean Pack CreateSetup --configuration Release ``` +#### How the Nuke build works + +- `build.ps1` bootstraps and runs the Nuke build defined in [build/Build.cs](build/Build.cs). Targets and their dependencies are declared there in C#. +- **Default target is `Test`** (which depends on `Compile`), so a bare `./build.ps1` restores, compiles, and runs the tests. +- Select targets with `--target ` (space-separated for multiple). Common targets: `Clean`, `Restore`, `Compile`, `Test`, `Pack`, `CreateSetup`, `GeneratePluginHashes`, `Publish`. +- List all available targets: `./build.ps1 --help` +- Preview the execution plan without running it: `./build.ps1 --plan` +- Pick build configuration with `--configuration Debug|Release` (defaults to `Debug` locally, `Release` on CI). + ### Using .NET CLI Directly ```bash @@ -144,7 +173,7 @@ LogExpert/ - Use localization resources from `LogExpert.Resources` project - Windows Forms designer files: `*.designer.cs` -### Testing +### Test Setup & Conventions - Unit tests use NUnit framework with Moq for mocking - Test projects follow naming pattern `*.Tests` @@ -156,7 +185,7 @@ LogExpert/ ### Code Style - Nullable reference types enabled (`enable`) -- Comprehensive `.editorconfig` with 4000+ rules +- `.editorconfig` is the source of truth for formatting and analyzer severities — match it rather than imposing your own style; don't fight or override its rules - ImplicitUsings enabled - Assembly signing enabled (Key.snk) @@ -174,9 +203,9 @@ LogExpert/ ### Git Workflow - Default branch: `Development` (use for PRs) -- Commit format: Include "Co-Authored-By: Claude Sonnet 4.5 " +- Commit format: **Do not** add a `Co-Authored-By` trailer (or any other agent/AI attribution) to commit messages - GitHub Actions run on push to Development branch -- AppVeyor for CI builds and artifact creation +- AppVeyor is the second CI for builds and artifact creation. It is currently disabled and needs to be re-added alongside GitHub Actions ## Plugin Security System @@ -198,6 +227,15 @@ LogExpert/ 6. **Encoding detection**: BOM-less files default to encoding from EncodingOptions 7. **Plugin hashes**: Only verified in Release builds; Debug builds skip verification +## Dont Do that + +Hard rules — never do these: + +- **Don't add a `Co-Authored-By` trailer** (or any other agent/AI attribution) to commit messages. +- **Don't attempt Linux/macOS builds.** This is Windows-only (Windows Desktop SDK + Windows Forms). +- **Don't use `AutoScaleMode` or `AutoScaleDimensions` on individual controls** — only on forms (High DPI). +- **Don't override or fight `.editorconfig` rules** — it is the source of truth for formatting and analyzer severities. + ## Key Dependencies - **NLog**: Logging framework @@ -212,10 +250,23 @@ LogExpert/ - Main README: [README.md](README.md) - Plugin Development: [PLUGIN_DEVELOPMENT_GUIDE.md](src/docs/PLUGIN_DEVELOPMENT_GUIDE.md) - Plugin Hash System: [PLUGIN_HASH_MANAGEMENT.md](src/docs/PLUGIN_HASH_MANAGEMENT.md) -- Performance Benchmarks: [BENCHMARK_SUMMARY.md](src/docs/performance/BENCHMARK_SUMMARY.md) - GitHub Wiki: https://github.com/LogExperts/LogExpert/wiki - Discord: https://discord.gg/SjxkuckRe9 +## Agent skills + +### Issue tracker + +Issues are tracked in `LogExperts/LogExpert` GitHub Issues, read-only for skills via the `gh` CLI — analysis only; only humans post. See `docs/agents/issue-tracker.md`. + +### Triage labels + +Canonical triage roles map to repo labels (`needs-info` → `question`; the rest use default names), used to read/filter issues — skills don't apply labels. See `docs/agents/triage-labels.md`. + +### Domain docs + +Single-context: one `CONTEXT.md` + `docs/adr/` at the repo root. See `docs/agents/domain.md`. + # Update Rules File To update this file, ensure that all sections are kept current with the latest architectural decisions, build processes, and development workflows. Follow these guidelines: - Review and update build commands if there are changes in the build system. diff --git a/CONTEXT.md b/CONTEXT.md new file mode 100644 index 00000000..0d045a82 --- /dev/null +++ b/CONTEXT.md @@ -0,0 +1,155 @@ +# LogExpert — Context & Ubiquitous Language + +Glossary of domain terms used in this codebase. Terms here have a single, agreed +meaning; do not redefine them locally. + +## Highlighting & triggers + +- **Highlight Entry** (`HighlightEntry`) — One search-text rule plus the visual + styling and trigger flags applied to lines that match it. Belongs to exactly + one Highlight Group. +- **Highlight Group** (`HighlightGroup`) — A named, ordered list of Highlight + Entries. The Log Window evaluates lines against the currently selected group. +- **Trigger** — A side-effecting action attached to a Highlight Entry that fires + when a line matches. Current triggers: Set Bookmark, Stop Tail, Don't Dirty + LED, Plugin (keyword action), and Audio Alert. +- **Action Entry** (`ActionEntry`) — Plugin name + parameters bound to the + Plugin trigger of a Highlight Entry. +- **Tail trigger path** — The single code path in `LogWindow.CheckFilterAndHighlight` + that evaluates highlight entries against *newly appended* lines. Triggers + that have user-perceivable side effects (currently: Audio Alert) fire **only** + on this path; bulk/scanner paths intentionally skip them. + +## Audio alerts + +- **Audio Alert** — A sound played when a tail-only highlight match occurs. + Toggled per Highlight Entry via `AlertOnHit`. +- **Sound File** (`SoundFilePath`) — Per-entry absolute path to an audio file + (WAV/MP3/WMA/AIFF, anything NAudio can decode). Empty path means "default + system beep". A missing or unreadable file falls back silently to the system + beep and is logged at warn level. +- **Cooldown** (`CooldownSeconds`) — Per-entry minimum seconds between two + audio alerts. `0` disables throttling. +- **Active Cooldown** — The cooldown value of the *most recently played* alert. + It gates every subsequent attempt across the whole application until it + expires, regardless of which entry's cooldown was passed in for the new + attempt. See ADR 0001. +- **Audio Player** (`LogExpert.Audio.AudioPlayer`) — Static, process-wide + fire-and-forget player. Owns the single global last-played timestamp and + active-cooldown state. + +## Sessions + +- **Session** (`.lxj`) — Persisted workspace: a list of log file paths plus a + tab/window layout XML blob. Loading a session reopens those files in their + saved arrangement. Per-file state (bookmarks, filters, columnizer, …) is + *not* in the session — it loads independently from each file's **Session + File** (`.lxp`). A session is created or restored only by explicit user + action ("Save Project" / "Load Project" menu items, to be renamed to + Session terminology). +- **Session File** (`.lxp`) — JSON file saved beside (or in a central + directory next to) a log file, restoring per-file state on reload: + columnizer name, bookmarks, filter pipes, highlight group, multi-file + state, follow-tail, encoding. Created automatically on save; one per log + file. Composed by a Session via its `FileNames` list. The columnizer + field of a Session File is one of several sources that can be selected + by **Columnizer Selection Priority**; the other fields always load when + the Session File exists. + +*Avoid*: "project" / "project file" / "workspace" (use **Session**), +"persistence file" / "per-file persistence" (use **Session File**), +bare "session file" when you mean the workspace (that's a **Session**). + +## Control character display + +- **Control Character Substitution** — Display-only replacement of selected + control characters with visible glyphs, performed inside `LogWindow.PaintCell`. + Never modifies raw column data, never affects search, filter, highlight + regex, or columnizer parsing. Off by default. See ADR 0003. +- **Substitution Style** — One of five rendering modes applied globally to + every enabled control character: Caret notation (`^G`), C escape (`\a` / + `\xNN`), Abbreviation (`BEL`), Unicode Control Pictures (`␇`), ISO 2047 + (`⍾`). Default: Control Pictures. +- **Enabled Code Points** (`ControlCharSettings.EnabledCodepoints`) — The + subset of C0 + DEL the user has opted in to substituting. First-time + default is the "non-whitespace preset": all 33 characters except `HT`, + `LF`, `CR`. +- **Substitution Style Fallback** — Per-style rule for characters the style + has no defined glyph for. C escape falls back to `\xNN`; ISO 2047 falls + back to the Control Pictures glyph; the other three styles cover all 33 + in-scope characters. +- **Copy Displayed Form** (`ControlCharSettings.CopyDisplayedForm`) — + Opt-in setting that makes clipboard copy (and other "export selection" + paths) use the substituted text instead of the raw bytes. Default: off, + i.e. clipboard always carries raw data. + +*Avoid*: "control char rendering" (use **Substitution**), "escape" alone +(ambiguous between the **C escape** style and the general concept). + +## Columnizer selection + +- **Columnizer** (`ILogLineMemoryColumnizer`) — A plugin that parses a log + line into columns. Each loaded log window has exactly one active + columnizer at a time. The set of available columnizers is owned by + `PluginRegistry`. +- **Columnizer Mask Entry** (`ColumnizerMaskEntry`) — One user-configured + row on the Settings → Columnizers tab. Pairs a **Mask**, a **Mask Type**, + and a **Columnizer Name**. Stored in `Preferences.ColumnizerMaskList`. +- **Mask Type** — `Glob` or `Regex`. Glob uses `*` and `?` wildcards and + matches against the short file name; Regex uses .NET regular-expression + syntax. New rows default to `Glob`; rows that existed in `settings.json` + before this field was introduced deserialize as `Regex` for backward + compatibility. See ADR 0004. +- **Stale Mask Entry** — A Columnizer Mask Entry whose `ColumnizerName` + does not resolve to any currently-registered columnizer. Stale entries + are skipped at match time (the loop continues to the next entry), kept + in the settings file (the plugin may return), and flagged in the + Settings dialog with a leading warning icon. +- **Columnizer History** — Auto-maintained list (`Settings.ColumnizerHistoryList`, + capped at 40 entries) recording the columnizer last used per absolute + file path. Stale entries (columnizer no longer registered) are removed + on lookup; this list, unlike the Mask list, is not user-curated. +- **Columnizer Selection Priority** (`ColumnizerSelectionPriority` enum) — + The user-configured rule that orders the four sources of a "which + columnizer should this file open with?" decision: Session File, + Columnizer History, Columnizer Mask Entry, and AutoPick. Three modes, + mutually exclusive: + - `HistoryThenMask` *(default; today's behaviour)* — Session File → + History → Mask → AutoPick → built-in default. + - `MaskThenHistory` — Session File → Mask → History → AutoPick → default. + - `MaskOverridesPersistence` — Mask → Session File → History → AutoPick + → default. The only mode in which a matching mask outranks an + existing Session File. Only the columnizer field of the Session File + is overridden; bookmarks, filters, etc. still restore. + + See ADR 0005. Replaces the deprecated `Preferences.MaskPrio` bool. + (Note: the enum member name `MaskOverridesPersistence` retains the old + "Persistence" wording for backward compatibility of serialized settings; + the user-facing concept is **Session File**.) +- **AutoPick** (`Preferences.AutoPick`) — When on, runs + `ColumnizerPicker.FindBetterMemoryColumnizer` against the loaded file + content to auto-detect a columnizer. Fires only when the Columnizer + Selection Priority chain above produced no result — never overrides + an explicit Mask, History, or Session File hit. + +*Avoid*: "file mask" alone (ambiguous between glob and regex — say +**Glob Mask** or **Regex Mask**), "mask priority" (use **Columnizer +Selection Priority**). + +## Flagged ambiguities + +- "**session file**" was used by old UI resource keys (e.g. + `LoadProject_UI_Message_Error_Title_FailedToUpdateSessionFile`) to mean + the workspace (`.lxj`). Resolved: from now on **Session File** = `.lxp` + (per-file state) and **Session** = `.lxj` (workspace). Existing resource + keys named `*SessionFile*` today refer to the workspace and need + renaming. +- "**project**" / "**project file**" historically referred to `.lxj`. + Resolved: use **Session**. The `ProjectFileHandler` / + `ProjectPersister` type names and `OnLoadProjectToolStripMenuItemClick` + handler are scheduled for renaming. +- "**Per-file Persistence**" was the prior term for `.lxp`. Resolved: + use **Session File**. The enum member + `ColumnizerSelectionPriority.MaskOverridesPersistence` keeps its old + name for serialization compatibility — internal name only, not a + domain term. diff --git a/src/LogExpert.Core/Classes/Log/Streamreaders/PositionAwareStreamReaderDirect.cs b/src/LogExpert.Core/Classes/Log/Streamreaders/PositionAwareStreamReaderDirect.cs index e3282c88..5ae5f779 100644 --- a/src/LogExpert.Core/Classes/Log/Streamreaders/PositionAwareStreamReaderDirect.cs +++ b/src/LogExpert.Core/Classes/Log/Streamreaders/PositionAwareStreamReaderDirect.cs @@ -20,6 +20,8 @@ public class PositionAwareStreamReaderDirect : PositionAwareStreamReaderBase, IL private const char CHAR_LF = '\n'; private const char CHAR_CR = '\r'; + private static readonly SearchValues _lineTerminators = SearchValues.Create("\r\n"); + #endregion #region Fields @@ -28,7 +30,8 @@ public class PositionAwareStreamReaderDirect : PositionAwareStreamReaderBase, IL private int _readBlockLength; // valid chars in _readBlock private int _scanOffset; // current scan position in _readBlock private bool _eof; - private int _newLineSequenceLength; + private bool _initialized; // first block filled from the current stream position + private int _terminatorCharByteSize; // bytes for a single '\r' or '\n' in the active encoding private readonly List _completedBlocks = []; public override bool IsDisposed { get; protected set; } @@ -56,55 +59,73 @@ public override string ReadLine () } /// - /// Reads the next line by scanning the current block for '\n'. If the block is exhausted, - /// tail-copies the partial line to a new block and refills. Returns a zero-copy - /// ReadOnlyMemory<char> slice into the pooled block. + /// Reads the next line by scanning the current block for the next line terminator + /// (\n, \r\n, or a bare \r). If the block is exhausted, tail-copies + /// the partial line to a new block and refills. Returns a zero-copy + /// ReadOnlyMemory<char> slice into the pooled block. The byte position advances by the + /// content bytes plus the bytes of the actual terminator, so it stays exact on + /// files with mixed line endings. /// public bool TryReadLine (out ReadOnlyMemory lineMemory) { var reader = GetStreamReader(); - if (_newLineSequenceLength == 0) - { - _newLineSequenceLength = GuessNewLineSequenceLength(reader); - } + EnsureInitialized(reader); while (true) { - // If we have data to scan, look for \n + // If we have data to scan, look for the next \r or \n if (_scanOffset < _readBlockLength) { var searchSpan = _readBlock.AsSpan(_scanOffset, _readBlockLength - _scanOffset); - var lfIndex = searchSpan.IndexOf(CHAR_LF); + var hitIndex = searchSpan.IndexOfAny(_lineTerminators); - if (lfIndex >= 0) + if (hitIndex >= 0) { - // Found a line boundary - var lineLength = lfIndex; + // Number of char cells the terminator occupies (1 for \n or bare \r, 2 for \r\n). + int terminatorChars; - // Strip \r if present before \n - if (lineLength > 0 && searchSpan[lineLength - 1] == CHAR_CR) + if (searchSpan[hitIndex] == CHAR_LF) + { + terminatorChars = 1; + } + else if (hitIndex + 1 < searchSpan.Length) { - lineLength--; + // \r with a known following char: \r\n if it's \n, otherwise a bare \r. + terminatorChars = searchSpan[hitIndex + 1] == CHAR_LF ? 2 : 1; + } + else if (_eof) + { + // \r is the very last char in the file: a bare \r. + terminatorChars = 1; + } + else + { + // \r is the last char of the block but more data follows. Refill so the + // next char becomes available (the tail-copy carries the \r forward), + // then re-scan to classify it as \r\n or a bare \r. + RefillBlock(reader); + continue; } - // Enforce MaximumLineLength - var cappedLength = Math.Min(lineLength, MaximumLineLength); + var lineLength = hitIndex; + // Enforce MaximumLineLength on the returned slice, but count the full + // content for the byte position. + var cappedLength = Math.Min(lineLength, MaximumLineLength); lineMemory = _readBlock.AsMemory(_scanOffset, cappedLength); - // Update byte position: line chars + lineLength var contentSpan = _readBlock.AsSpan(_scanOffset, lineLength); - MovePosition(Encoding.GetByteCount(contentSpan) + _newLineSequenceLength); + MovePosition(Encoding.GetByteCount(contentSpan) + (terminatorChars * _terminatorCharByteSize)); - // Advance scan past the \n - _scanOffset += lfIndex + 1; + // Advance scan past the content and its terminator. + _scanOffset += hitIndex + terminatorChars; return true; } } - // No \n found (or no data at all). Need to refill. + // No terminator found (or no data at all). Need to refill. if (_eof) { // Emit remaining content as final line (no trailing newline) @@ -227,61 +248,50 @@ private void RefillBlock (StreamReader reader) } } - private int GuessNewLineSequenceLength (StreamReader reader) + /// + /// Lazily fills the first block from the current stream position. Called on the first read + /// after construction and after any seek + /// (which resets _initialized via ). + /// + private void EnsureInitialized (StreamReader reader) { - var currentPos = Position; - - try + if (_initialized) { - // Fill initial block - var charsRead = reader.Read(_readBlock, 0, BLOCK_SIZE); - _readBlockLength = charsRead; - _scanOffset = 0; + return; + } - if (charsRead == 0) - { - _eof = true; - return 0; - } + _initialized = true; - // Find first \n to determine newline sequence - var span = _readBlock.AsSpan(0, _readBlockLength); - var lfIndex = span.IndexOf(CHAR_LF); + // A single '\r' and a single '\n' encode to the same number of bytes in every encoding + // LogExpert uses (ASCII control chars), so one value covers \n, \r and (×2) \r\n. + Span singleTerminator = [CHAR_LF]; + _terminatorCharByteSize = Encoding.GetByteCount(singleTerminator); - if (lfIndex < 0) - { - // No newline found in first block — assume single-byte - return 1; - } - - if (lfIndex > 0 && span[lfIndex - 1] == CHAR_CR) - { - // \r\n - Span newline = ['\r', '\n']; - return Encoding.GetByteCount(newline); - } + var charsRead = reader.Read(_readBlock, 0, BLOCK_SIZE); + _readBlockLength = charsRead; + _scanOffset = 0; - // \n only - Span singleLf = ['\n']; - return Encoding.GetByteCount(singleLf); - } - finally + if (charsRead == 0) { - // Reset position — the filled block data is kept, no re-read needed - // Position is reset to original so byte tracking starts clean - Position = currentPos; - - // Re-read the block since Position setter resets the stream - _scanOffset = 0; - var charsRead = reader.Read(_readBlock, 0, BLOCK_SIZE); - _readBlockLength = charsRead; - if (charsRead == 0) - { - _eof = true; - } + _eof = true; } } + /// + /// Resets scan state so the next read re-fills from the (just seeked) stream position. + /// Only touches value-type fields, which is safe under the base constructor's virtual call + /// to this method (the pooled _readBlock is left untouched). + /// + protected override void ResetReader () + { + _scanOffset = 0; + _readBlockLength = 0; + _eof = false; + _initialized = false; + + base.ResetReader(); + } + protected override void Dispose (bool disposing) { if (disposing) diff --git a/src/LogExpert.Tests/LogExpert.Tests.csproj b/src/LogExpert.Tests/LogExpert.Tests.csproj index 1ce9e53f..ec1f5398 100644 --- a/src/LogExpert.Tests/LogExpert.Tests.csproj +++ b/src/LogExpert.Tests/LogExpert.Tests.csproj @@ -42,6 +42,21 @@ PreserveNewest + + PreserveNewest + + + PreserveNewest + + + PreserveNewest + + + PreserveNewest + + + PreserveNewest + diff --git a/src/LogExpert.Tests/StreamReaderTests/PositionAwareStreamReaderDirectTests.cs b/src/LogExpert.Tests/StreamReaderTests/PositionAwareStreamReaderDirectTests.cs index a3390aa3..0d44d3be 100644 --- a/src/LogExpert.Tests/StreamReaderTests/PositionAwareStreamReaderDirectTests.cs +++ b/src/LogExpert.Tests/StreamReaderTests/PositionAwareStreamReaderDirectTests.cs @@ -461,10 +461,8 @@ public void TryReadLine_LineLongerThanBlockSize_PositionParityWithSystemReader ( break; } - Assert.That(directLine.Span.ToString(), Is.EqualTo(systemLine.Span.ToString()), - $"Line {lineNum}: content mismatch"); - Assert.That(directReader.Position, Is.EqualTo(systemReader.Position), - $"Line {lineNum}: position mismatch (System={systemReader.Position}, Direct={directReader.Position})"); + Assert.That(directLine.Span.ToString(), Is.EqualTo(systemLine.Span.ToString()), $"Line {lineNum}: content mismatch"); + Assert.That(directReader.Position, Is.EqualTo(systemReader.Position), $"Line {lineNum}: position mismatch (System={systemReader.Position}, Direct={directReader.Position})"); lineNum++; } @@ -601,4 +599,151 @@ public void TryReadLine_LineLongerThanBlockSize_DetachBlocksWithLargeTail () } #endregion + + #region Line ending handling (bare CR, mixed) + + private static List ReadAllLines (string text, Encoding? encoding = null, int maximumLineLength = 500) + { + var enc = encoding ?? Encoding.UTF8; + using var stream = CreateStream(text, enc); + using var reader = new PositionAwareStreamReaderDirect(stream, new EncodingOptions { Encoding = enc }, maximumLineLength); + var result = new List(); + while (reader.TryReadLine(out var line)) + { + result.Add(line.Span.ToString()); + } + + return result; + } + + private static List<(string Content, long Position)> ReadAllWithPositions (string text, Encoding? encoding = null) + { + var enc = encoding ?? Encoding.UTF8; + using var stream = CreateStream(text, enc); + using var reader = new PositionAwareStreamReaderDirect(stream, new EncodingOptions { Encoding = enc }, 500); + var result = new List<(string, long)>(); + while (reader.TryReadLine(out var line)) + { + result.Add((line.Span.ToString(), reader.Position)); + } + + return result; + } + + [Test] + [System.Diagnostics.CodeAnalysis.SuppressMessage("Globalization", "CA1303:Do not pass literals as localized parameters", Justification = "Unit Tests")] + public void TryReadLine_BareCarriageReturn_SplitsLines () + { + var lines = ReadAllLines("Line 1\rLine 2\rLine 3"); + Assert.That(lines, Is.EqualTo(["Line 1", "Line 2", "Line 3"])); + } + + [Test] + [System.Diagnostics.CodeAnalysis.SuppressMessage("Globalization", "CA1303:Do not pass literals as localized parameters", Justification = "Unit Tests")] + public void TryReadLine_MixedLineEndings_SplitsLines () + { + // \r\n, \n, bare \r all in one file + var lines = ReadAllLines("a\r\nb\nc\rd\n"); + Assert.That(lines, Is.EqualTo(["a", "b", "c", "d"])); + } + + [Test] + [TestCase("\r\r\r", 3)] + [TestCase("\r\n\r\n\r\n", 3)] + [TestCase("\n\n\n", 3)] + public void TryReadLine_RepeatedTerminators_YieldEmptyLines (string text, int expectedCount) + { + var lines = ReadAllLines(text); + Assert.That(lines.Count, Is.EqualTo(expectedCount)); + Assert.That(lines, Has.All.Empty); + } + + [Test] + [System.Diagnostics.CodeAnalysis.SuppressMessage("Globalization", "CA1303:Do not pass literals as localized parameters", Justification = "Unit Tests")] + public void TryReadLine_TrailingBareCR_NoSpuriousEmptyLine () + { + var lines = ReadAllLines("abc\r"); + Assert.That(lines, Is.EqualTo(["abc"])); + } + + [Test] + [System.Diagnostics.CodeAnalysis.SuppressMessage("Globalization", "CA1303:Do not pass literals as localized parameters", Justification = "Unit Tests")] + public void TryReadLine_TrailingCrLf_NoSpuriousEmptyLine () + { + var lines = ReadAllLines("abc\r\n"); + Assert.That(lines, Is.EqualTo(["abc"])); + } + + [Test] + [System.Diagnostics.CodeAnalysis.SuppressMessage("Globalization", "CA1303:Do not pass literals as localized parameters", Justification = "Unit Tests")] + public void TryReadLine_LfThenTrailingCr_YieldsEmptyLineBetween () + { + // "abc\n\r": \n terminates "abc", then a standalone \r yields an empty final line + var lines = ReadAllLines("abc\n\r"); + Assert.That(lines, Is.EqualTo(["abc", ""])); + } + + [Test] + [System.Diagnostics.CodeAnalysis.SuppressMessage("Globalization", "CA1303:Do not pass literals as localized parameters", Justification = "Unit Tests")] + public void Position_BareCarriageReturn_ExactBytes () + { + // "abc\rdef" UTF-8: "abc"(3) + \r(1) = 4; "def"(3) at EOF = 7 + var result = ReadAllWithPositions("abc\rdef"); + Assert.That(result.Count, Is.EqualTo(2)); + Assert.That(result[0], Is.EqualTo(("abc", 4L))); + Assert.That(result[1], Is.EqualTo(("def", 7L))); + } + + [Test] + [System.Diagnostics.CodeAnalysis.SuppressMessage("Globalization", "CA1303:Do not pass literals as localized parameters", Justification = "Unit Tests")] + public void Position_MixedLineEndings_ExactBytes () + { + // "a\r\nb\nc\rd\n": a+\r\n=3, b+\n=5, c+\r=7, d+\n=9 + var result = ReadAllWithPositions("a\r\nb\nc\rd\n"); + Assert.That(result.Select(r => r.Position).ToArray(), Is.EqualTo([3L, 5L, 7L, 9L])); + Assert.That(result.Select(r => r.Content).ToArray(), Is.EqualTo(["a", "b", "c", "d"])); + } + + [Test] + [System.Diagnostics.CodeAnalysis.SuppressMessage("Globalization", "CA1303:Do not pass literals as localized parameters", Justification = "Unit Tests")] + public void Position_MixedLineEndings_Multibyte_ExactBytes () + { + // "Héllo\rwörld\n" UTF-8: "Héllo"=6 bytes + \r(1) = 7; "wörld"=6 bytes + \n(1) = 14 + var result = ReadAllWithPositions("Héllo\rwörld\n"); + Assert.That(result.Count, Is.EqualTo(2)); + Assert.That(result[0], Is.EqualTo(("Héllo", 7L))); + Assert.That(result[1], Is.EqualTo(("wörld", 14L))); + } + + [Test] + public void TryReadLine_BareCrAtBlockBoundary_SplitsCorrectly () + { + // \r lands exactly at the last char of the first 32 KB block, followed by more data. + const int blockSize = 32_768; + var firstLine = new string('A', blockSize - 1); // \r will be char at index blockSize-1 (last in block) + var text = firstLine + "\r" + "rest\nlast\n"; + + var lines = ReadAllLines(text, maximumLineLength: blockSize + 100); + Assert.That(lines.Count, Is.EqualTo(3)); + Assert.That(lines[0], Is.EqualTo(firstLine)); + Assert.That(lines[1], Is.EqualTo("rest")); + Assert.That(lines[2], Is.EqualTo("last")); + } + + [Test] + public void TryReadLine_CrLfStraddlesBlockBoundary_SplitsCorrectly () + { + // \r is the last char of the first block, \n is the first char of the next block. + const int blockSize = 32_768; + var firstLine = new string('A', blockSize - 1); + var text = firstLine + "\r\n" + "rest\nlast\n"; + + var lines = ReadAllLines(text, maximumLineLength: blockSize + 100); + Assert.That(lines.Count, Is.EqualTo(3)); + Assert.That(lines[0], Is.EqualTo(firstLine)); + Assert.That(lines[1], Is.EqualTo("rest")); + Assert.That(lines[2], Is.EqualTo("last")); + } + + #endregion } \ No newline at end of file diff --git a/src/LogExpert.Tests/TestData/LineEndings_CR.txt b/src/LogExpert.Tests/TestData/LineEndings_CR.txt new file mode 100644 index 00000000..41256b08 --- /dev/null +++ b/src/LogExpert.Tests/TestData/LineEndings_CR.txt @@ -0,0 +1 @@ +2026-06-25 10:00:01.000 [INFO] Line 1 - start of file 2026-06-25 10:00:02.000 [INFO] Line 2 - second line 2026-06-25 10:00:03.000 [WARN] Line 3 - multibyte: ü 世界 2026-06-25 10:00:04.000 [INFO] Line 4 - line before the empty one 2026-06-25 10:00:06.000 [ERROR] Line 6 - line after the empty one 2026-06-25 10:00:07.000 [INFO] Line 7 - final line, no trailing newline \ No newline at end of file diff --git a/src/LogExpert.Tests/TestData/LineEndings_CRLF.txt b/src/LogExpert.Tests/TestData/LineEndings_CRLF.txt new file mode 100644 index 00000000..09aed648 --- /dev/null +++ b/src/LogExpert.Tests/TestData/LineEndings_CRLF.txt @@ -0,0 +1,7 @@ +2026-06-25 10:00:01.000 [INFO] Line 1 - start of file +2026-06-25 10:00:02.000 [INFO] Line 2 - second line +2026-06-25 10:00:03.000 [WARN] Line 3 - multibyte: ü 世界 +2026-06-25 10:00:04.000 [INFO] Line 4 - line before the empty one + +2026-06-25 10:00:06.000 [ERROR] Line 6 - line after the empty one +2026-06-25 10:00:07.000 [INFO] Line 7 - final line, no trailing newline \ No newline at end of file diff --git a/src/LogExpert.Tests/TestData/LineEndings_LF.txt b/src/LogExpert.Tests/TestData/LineEndings_LF.txt new file mode 100644 index 00000000..8c0453dc --- /dev/null +++ b/src/LogExpert.Tests/TestData/LineEndings_LF.txt @@ -0,0 +1,7 @@ +2026-06-25 10:00:01.000 [INFO] Line 1 - start of file +2026-06-25 10:00:02.000 [INFO] Line 2 - second line +2026-06-25 10:00:03.000 [WARN] Line 3 - multibyte: ü 世界 +2026-06-25 10:00:04.000 [INFO] Line 4 - line before the empty one + +2026-06-25 10:00:06.000 [ERROR] Line 6 - line after the empty one +2026-06-25 10:00:07.000 [INFO] Line 7 - final line, no trailing newline \ No newline at end of file diff --git a/src/LogExpert.Tests/TestData/LineEndings_Mixed.txt b/src/LogExpert.Tests/TestData/LineEndings_Mixed.txt new file mode 100644 index 00000000..93ba7d08 --- /dev/null +++ b/src/LogExpert.Tests/TestData/LineEndings_Mixed.txt @@ -0,0 +1,5 @@ +2026-06-25 10:00:01.000 [INFO] Line 1 - start of file +2026-06-25 10:00:02.000 [INFO] Line 2 - second line +2026-06-25 10:00:03.000 [WARN] Line 3 - multibyte: ü 世界 2026-06-25 10:00:04.000 [INFO] Line 4 - line before the empty one + +2026-06-25 10:00:06.000 [ERROR] Line 6 - line after the empty one 2026-06-25 10:00:07.000 [INFO] Line 7 - final line, no trailing newline \ No newline at end of file diff --git a/src/LogExpert.Tests/TestData/LineEndings_README.md b/src/LogExpert.Tests/TestData/LineEndings_README.md new file mode 100644 index 00000000..a0d2b723 --- /dev/null +++ b/src/LogExpert.Tests/TestData/LineEndings_README.md @@ -0,0 +1,24 @@ +# Line-ending test files + +Hand-crafted fixtures for verifying that the stream readers split lines correctly +regardless of line-ending style. Drag-and-drop or open each in the GUI and confirm +it shows **7 lines**, the 5th of which is **empty**, with the multibyte chars on +line 3 (`ü 世界`) intact. + +All files are UTF-8 **without BOM**. The last line has **no trailing terminator** +(exercises the end-of-file tail path). The expected logical content is identical +across every file — only the terminators differ. + +| File | Terminator(s) | Notes | +|------|---------------|-------| +| `LineEndings_LF.txt` | `\n` only | Unix | +| `LineEndings_CRLF.txt` | `\r\n` only | Windows | +| `LineEndings_CR.txt` | `\r` only | Classic Mac — a reader that scans for `\n` only shows the whole file as ONE line | +| `LineEndings_Mixed.txt` | `\n`, `\r\n`, `\r` interleaved | Different terminator after each line | + +The default `SystemDirect` reader handles all four. The `CR` and `Mixed` files are +the interesting ones: before `Direct` learned to detect the actual terminator per +line, `CR` rendered as a single giant line and `Mixed` drifted the byte position +(visible as wrong seeking when buffers are flushed and reloaded). + +Regenerate with `scratchpad/gen-lineendings.ps1` (writes byte-exact terminators). From bf4eb7ff4905769f46006b6e92924f66918feeaf Mon Sep 17 00:00:00 2001 From: BRUNER Patrick Date: Thu, 25 Jun 2026 14:41:16 +0200 Subject: [PATCH 2/3] missing files --- .../TestData/LineEndings_README.md | 2 +- .../TestData/gen-lineendings.ps1 | 40 +++++++++++++++++++ 2 files changed, 41 insertions(+), 1 deletion(-) create mode 100644 src/LogExpert.Tests/TestData/gen-lineendings.ps1 diff --git a/src/LogExpert.Tests/TestData/LineEndings_README.md b/src/LogExpert.Tests/TestData/LineEndings_README.md index a0d2b723..6b7d99db 100644 --- a/src/LogExpert.Tests/TestData/LineEndings_README.md +++ b/src/LogExpert.Tests/TestData/LineEndings_README.md @@ -21,4 +21,4 @@ the interesting ones: before `Direct` learned to detect the actual terminator pe line, `CR` rendered as a single giant line and `Mixed` drifted the byte position (visible as wrong seeking when buffers are flushed and reloaded). -Regenerate with `scratchpad/gen-lineendings.ps1` (writes byte-exact terminators). +Regenerate with `gen-lineendings.ps1` in this folder (writes byte-exact terminators). diff --git a/src/LogExpert.Tests/TestData/gen-lineendings.ps1 b/src/LogExpert.Tests/TestData/gen-lineendings.ps1 new file mode 100644 index 00000000..8cbfcbda --- /dev/null +++ b/src/LogExpert.Tests/TestData/gen-lineendings.ps1 @@ -0,0 +1,40 @@ +# Regenerates the LineEndings_*.txt fixtures with byte-exact terminators. +# Run from anywhere: paths resolve relative to this script's own location. +# The fixtures are pinned `binary` in .gitattributes so git never normalizes EOLs. +$dir = $PSScriptRoot +$enc = New-Object System.Text.UTF8Encoding($false) # UTF-8, no BOM + +# Build multibyte chars from code points so the script stays pure-ASCII on disk. +$umlaut = [char]0x00FC # u-umlaut +$cjk = [string][char]0x4E16 + [char]0x754C # CJK "world" + +$lines = @( + "2026-06-25 10:00:01.000 [INFO] Line 1 - start of file", + "2026-06-25 10:00:02.000 [INFO] Line 2 - second line", + "2026-06-25 10:00:03.000 [WARN] Line 3 - multibyte: $umlaut $cjk", + "2026-06-25 10:00:04.000 [INFO] Line 4 - line before the empty one", + "", + "2026-06-25 10:00:06.000 [ERROR] Line 6 - line after the empty one", + "2026-06-25 10:00:07.000 [INFO] Line 7 - final line, no trailing newline" +) + +function Write-File($name, $sep) { + # Last line gets no trailing terminator (exercises the EOF tail path). + $text = [string]::Join($sep, $lines) + [System.IO.File]::WriteAllBytes((Join-Path $dir $name), $enc.GetBytes($text)) +} + +Write-File "LineEndings_LF.txt" "`n" +Write-File "LineEndings_CRLF.txt" "`r`n" +Write-File "LineEndings_CR.txt" "`r" + +# Mixed: a different terminator after each line, last line unterminated. +$terms = @("`n", "`r`n", "`r", "`n", "`r`n", "`r") +$sb = New-Object System.Text.StringBuilder +for ($i = 0; $i -lt $lines.Count; $i++) { + [void]$sb.Append($lines[$i]) + if ($i -lt $terms.Count) { [void]$sb.Append($terms[$i]) } +} +[System.IO.File]::WriteAllBytes((Join-Path $dir "LineEndings_Mixed.txt"), $enc.GetBytes($sb.ToString())) + +Write-Output "Done." From 54fa19f2082e3694dc6397d4a7e08a0eeeffaaab Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" Date: Thu, 25 Jun 2026 12:45:04 +0000 Subject: [PATCH 3/3] chore: update plugin hashes [skip ci] --- .../PluginHashGenerator.Generated.cs | 32 +++++++++---------- 1 file changed, 16 insertions(+), 16 deletions(-) diff --git a/src/PluginRegistry/PluginHashGenerator.Generated.cs b/src/PluginRegistry/PluginHashGenerator.Generated.cs index 633650c5..d5472889 100644 --- a/src/PluginRegistry/PluginHashGenerator.Generated.cs +++ b/src/PluginRegistry/PluginHashGenerator.Generated.cs @@ -10,7 +10,7 @@ public static partial class PluginValidator { /// /// Gets pre-calculated SHA256 hashes for built-in plugins. - /// Generated: 2026-06-24 08:06:11 UTC + /// Generated: 2026-06-25 12:45:02 UTC /// Configuration: Release /// Plugin count: 21 /// @@ -18,27 +18,27 @@ public static Dictionary GetBuiltInPluginHashes() { return new Dictionary(StringComparer.OrdinalIgnoreCase) { - ["AutoColumnizer.dll"] = "36D3E82056BBE33D1F6A32ED28107475A6E466DA11366D92BF2FC75468659BAB", + ["AutoColumnizer.dll"] = "97CB14C8E6F7A714A74E92152CD947B2DF26742AC4C6F1550366B5C2586FB7F8", ["BouncyCastle.Cryptography.dll"] = "E5EEAF6D263C493619982FD3638E6135077311D08C961E1FE128F9107D29EBC6", ["BouncyCastle.Cryptography.dll (x86)"] = "E5EEAF6D263C493619982FD3638E6135077311D08C961E1FE128F9107D29EBC6", - ["CsvColumnizer.dll"] = "6AB202B1E664F587A0655C2426831FB3E5640AECC6205C4C6C147A8DC9DBEFC9", - ["CsvColumnizer.dll (x86)"] = "6AB202B1E664F587A0655C2426831FB3E5640AECC6205C4C6C147A8DC9DBEFC9", - ["DefaultPlugins.dll"] = "05783A26E389565F7D262C005F7D5CD7938738CD28A6BB5C7E7E275291616E73", - ["FlashIconHighlighter.dll"] = "C68A44AC9542B23F852B681909C1D6810246474081092432FBC507CF1E9D4038", - ["GlassfishColumnizer.dll"] = "A30C04E7C61309977731327B26AB88B01386EC96CF8D84266ACF8434E4257575", - ["JsonColumnizer.dll"] = "8600F78EF0930211D8986737D7AAC968F4EB7213CE691AFF0691A8682DF82D52", - ["JsonCompactColumnizer.dll"] = "0F77645DA79291D3CC20FD254798C0D6AB3E27DB9BA96BB04F73D566F49FDCDD", - ["Log4jXmlColumnizer.dll"] = "B502ED950944FC439634AC3F3FB45E7B2308E945CBFBEA5A3CAF1257ACF49457", - ["LogExpert.Resources.dll"] = "D44708FD8FC9F9485D8D36068A131903954E385786F0FAE78AC6EF9BDC0ED576", + ["CsvColumnizer.dll"] = "9C9F3914A4ECD1E9A45A0B87845E942162DB8203BCEF05FCA8E8E4A6C10D6AC2", + ["CsvColumnizer.dll (x86)"] = "9C9F3914A4ECD1E9A45A0B87845E942162DB8203BCEF05FCA8E8E4A6C10D6AC2", + ["DefaultPlugins.dll"] = "8EA6195A67840581DA4987B3B2893C68D8DC6DCA919B180896258386B9DB2713", + ["FlashIconHighlighter.dll"] = "5095E95DD6AEED1747BB6CF259D550B2B3D530B6F925C47A7458ECEDBF2EBC60", + ["GlassfishColumnizer.dll"] = "66730D4646D90568BFDCE500363CF2215B52F538B1E9172A78ECC6CAE0DD1C27", + ["JsonColumnizer.dll"] = "7C064A5DE3A970C79518280D43E8C041322B4FBC27D0D8A77BC387DFD738936F", + ["JsonCompactColumnizer.dll"] = "5EEAFD0882DD1DBE5F0FFF92A65580737B775E43CA7F3A0DC4EFDE6A28B0DE5E", + ["Log4jXmlColumnizer.dll"] = "C8458E855FE59F799E2394500BF4991A7E5E7F8AA6B030E2B7A733322E2AFAEF", + ["LogExpert.Resources.dll"] = "F765E294A0EEEC525FA8992D22B1A666880CE2D0F720EFAF154D01EBB7A8AEED", ["Microsoft.Extensions.DependencyInjection.Abstractions.dll"] = "67FA4325000DB017DC0C35829B416F024F042D24EFB868BCF17A895EE6500A93", ["Microsoft.Extensions.DependencyInjection.Abstractions.dll (x86)"] = "67FA4325000DB017DC0C35829B416F024F042D24EFB868BCF17A895EE6500A93", ["Microsoft.Extensions.Logging.Abstractions.dll"] = "BB853130F5AFAF335BE7858D661F8212EC653835100F5A4E3AA2C66A4D4F685D", ["Microsoft.Extensions.Logging.Abstractions.dll (x86)"] = "BB853130F5AFAF335BE7858D661F8212EC653835100F5A4E3AA2C66A4D4F685D", - ["RegexColumnizer.dll"] = "9F18D9269DF2B8C482F1C82A1FDA93D1D445F0C2CEEEFA72124A9B6C4B5C1E9C", - ["SftpFileSystem.dll"] = "301D74FC8683B8D8B71A286711152ACA5581CF78D32CF9DCF218802F81D7D02B", - ["SftpFileSystem.dll (x86)"] = "2FE8EE1C55CC85AA82F4CE3B19B5E2734BC00E928D65C94E8B88695D9434AEF9", - ["SftpFileSystem.Resources.dll"] = "C39F867FFA577867BE6A7101F66D0EBA3207E2060FDE03646AC1F8D8ECCC140E", - ["SftpFileSystem.Resources.dll (x86)"] = "C39F867FFA577867BE6A7101F66D0EBA3207E2060FDE03646AC1F8D8ECCC140E", + ["RegexColumnizer.dll"] = "009C8C7E56BB89C9448091A294894FE69C661DF35F9082CD2A092EEAF85E23A7", + ["SftpFileSystem.dll"] = "74624F9B6659F83B53B7FEC45267345B84BE3DB13C811E04926884E65C32D56E", + ["SftpFileSystem.dll (x86)"] = "03A1F8E1A3DF8915404A2653FE55B39B196EFD19C6DE42586D0B9822EFFBC617", + ["SftpFileSystem.Resources.dll"] = "09F468603D3EAF56B13058BCE104CF9EA3F7B757352E9199D54486C8734AC47B", + ["SftpFileSystem.Resources.dll (x86)"] = "09F468603D3EAF56B13058BCE104CF9EA3F7B757352E9199D54486C8734AC47B", }; }