Improve GNU compatibility through byte processing by dspinellis · Pull Request #487 · uutils/sed

dspinellis · 2026-07-07T10:55:25Z

The original Rust implementation assummed a UTF-8 environment, as is common in modern systems. This commit improves GNU sed compatibility by changing most processing to run on bytes rather than Rust's Unicode characters and strings. This change also improves the handling of binary files. The commit adds a simple POSIX-compatible locale configuration to support UTF-8 processing when specified. While at it, also fix script-specified file handling to take place through OsString.

The change fixes two failing GNU test suite cases.

github-actions · 2026-07-07T10:58:36Z

GNU sed testsuite comparison:

Test results comparison:
  Current:   TOTAL: 65 / PASSED: 12 / FAILED: 43 / SKIPPED: 10
  Reference: TOTAL: 65 / PASSED: 10 / FAILED: 45 / SKIPPED: 10

Changes from main branch:
  TOTAL: +0
  PASSED: +2
  FAILED: -2

Test improvements (2):
  + 8bit
  + mac-mf

github-actions · 2026-07-07T11:02:01Z

GNU sed testsuite comparison:

Test results comparison:
  Current:   TOTAL: 65 / PASSED: 12 / FAILED: 43 / SKIPPED: 10
  Reference: TOTAL: 65 / PASSED: 10 / FAILED: 45 / SKIPPED: 10

Changes from main branch:
  TOTAL: +0
  PASSED: +2
  FAILED: -2

Test improvements (2):
  + 8bit
  + mac-mf

codecov · 2026-07-07T11:03:23Z

Codecov Report

❌ Patch coverage is 84.12017% with 185 lines in your changes missing coverage. Please review.
✅ Project coverage is 84.98%. Comparing base (30607c4) to head (c58cecc).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
src/sed/processor.rs	44.73%	126 Missing ⚠️
src/sed/compiler.rs	92.47%	21 Missing ⚠️
src/sed/fast_regex.rs	91.71%	14 Missing ⚠️
src/sed/fast_io.rs	86.44%	8 Missing ⚠️
src/sed/named_writer.rs	61.90%	8 Missing ⚠️
src/sed/delimited_parser.rs	96.65%	7 Missing ⚠️
src/sed/mod.rs	98.21%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #487      +/-   ##
==========================================
+ Coverage   83.05%   84.98%   +1.92%     
==========================================
  Files          13       13              
  Lines        6120     6760     +640     
  Branches      358      378      +20     
==========================================
+ Hits         5083     5745     +662     
+ Misses       1034     1012      -22     
  Partials        3        3

Flag	Coverage Δ
macos_latest	`85.79% <84.92%> (+2.02%)`	⬆️
ubuntu_latest	`85.83% <84.92%> (+1.98%)`	⬆️
windows_latest	`0.00% <0.00%> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

codspeed-hq · 2026-07-07T11:07:22Z

Merging this PR will improve performance by 18.22%

⚡ 5 improved benchmarks
✅ 6 untouched benchmarks

Performance Changes

	Benchmark	`BASE`	`HEAD`	Efficiency
⚡	`access_log_translit`	785.6 ms	459.5 ms	+70.98%
⚡	`remove_cr`	261.7 ms	222.4 ms	+17.68%
⚡	`genome_subst`	210.6 ms	194 ms	+8.59%
⚡	`number_fix`	582.9 ms	565.1 ms	+3.16%
⚡	`access_log_subst`	1.1 s	1.1 s	+2.46%

Tip

Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.

_{Comparing dspinellis:switch-to-byte-handling (c58cecc) with main (15def35)}

github-actions · 2026-07-07T11:18:03Z

GNU sed testsuite comparison:

Test results comparison:
  Current:   TOTAL: 65 / PASSED: 12 / FAILED: 43 / SKIPPED: 10
  Reference: TOTAL: 65 / PASSED: 10 / FAILED: 45 / SKIPPED: 10

Changes from main branch:
  TOTAL: +0
  PASSED: +2
  FAILED: -2

Test improvements (2):
  + 8bit
  + mac-mf

github-actions · 2026-07-07T12:00:22Z

GNU sed testsuite comparison:

Test results comparison:
  Current:   TOTAL: 65 / PASSED: 12 / FAILED: 43 / SKIPPED: 10
  Reference: TOTAL: 65 / PASSED: 10 / FAILED: 45 / SKIPPED: 10

Changes from main branch:
  TOTAL: +0
  PASSED: +2
  FAILED: -2

Test improvements (2):
  + 8bit
  + mac-mf

Copilot

Pull request overview

This PR refactors the sed implementation to improve GNU sed compatibility and binary-file handling by switching most internal processing from Rust String/Unicode semantics to byte-oriented (Vec<u8>) semantics, while still enabling UTF-8-aware behavior when the environment locale indicates it.

Changes:

Introduces locale-driven CharacterMode (byte vs UTF-8) and updates processing paths accordingly (regex, transliteration, list output, and script parsing).
Migrates key data structures (pattern/hold space, script providers, writers, output buffering) to operate on bytes instead of UTF-8 strings.
Expands/adjusts integration and unit tests to explicitly cover both byte-mode (LC_ALL=C) and UTF-8-mode (LC_ALL=C.UTF-8) behavior, including invalid UTF-8 cases.

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
tests/tests.rs	Sets default test locale to UTF-8 to keep fixture-based tests stable while allowing explicit byte-mode overrides.
tests/by-util/test_sed.rs	Adds/updates locale-explicit tests and extends coverage for byte/UTF-8 mode differences (backrefs, escapes, listing, transliteration).
src/sed/script_line_provider.rs	Returns raw script lines as bytes and preserves invalid UTF-8 from file sources.
src/sed/script_char_provider.rs	Switches script scanning from chars to bytes; adds APIs for raw byte access.
src/sed/processor.rs	Updates core processing to use byte pattern spaces; adds byte/UTF-8 aware list and transliteration paths.
src/sed/named_writer.rs	Adds byte-writing support for `w`-style outputs to preserve arbitrary bytes.
src/sed/mod.rs	Adds locale parsing and initializes `ProcessingContext` with environment-derived `CharacterMode`.
src/sed/fast_regex.rs	Reworks regex abstraction to accept byte patterns, support byte-mode restrictions, and return matches as bytes.
src/sed/fast_io.rs	Converts IO chunk storage and output scheduling to bytes; updates readers/writers to avoid UTF-8 assumptions.
src/sed/delimited_parser.rs	Parses delimited constructs into bytes, adds byte→OsString conversion utility, and introduces mode-aware parsing helpers.
src/sed/compiler.rs	Compiles regex/transliteration/replacements in a character-mode-aware way and switches script file paths to byte-safe handling.
src/sed/command.rs	Introduces `CharacterMode`, migrates replacement/template handling to bytes, and extends transliteration for byte and UTF-8 modes.
README.md	Documents new locale-driven behavior, limitations vs GNU sed locales, and byte-mode backref restriction.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

github-actions · 2026-07-07T12:38:11Z

GNU sed testsuite comparison:

Test results comparison:
  Current:   TOTAL: 65 / PASSED: 12 / FAILED: 43 / SKIPPED: 10
  Reference: TOTAL: 65 / PASSED: 10 / FAILED: 45 / SKIPPED: 10

Changes from main branch:
  TOTAL: +0
  PASSED: +2
  FAILED: -2

Test improvements (2):
  + 8bit
  + mac-mf

github-actions · 2026-07-07T14:55:59Z

GNU sed testsuite comparison:

Test results comparison:
  Current:   TOTAL: 65 / PASSED: 12 / FAILED: 43 / SKIPPED: 10
  Reference: TOTAL: 65 / PASSED: 10 / FAILED: 45 / SKIPPED: 10

Changes from main branch:
  TOTAL: +0
  PASSED: +2
  FAILED: -2

Test improvements (2):
  + 8bit
  + mac-mf

The original Rust implementation assummed a UTF-8 environment, as is common in modern systems. This commit improves GNU sed compatibility by changing most processing to run on bytes rather than Rust's Unicode characters and strings. This change also improves the handling of binary files. The commit adds a simple POSIX-compatible locale configuration to support UTF-8 processing when specified. While at it, also fix script-specified file handling to take place through OsString and add a few missing unit tests. The change fixes two failing GNU test suite cases.

github-actions · 2026-07-08T07:17:29Z

GNU sed testsuite comparison:

Test results comparison:
  Current:   TOTAL: 65 / PASSED: 12 / FAILED: 43 / SKIPPED: 10
  Reference: TOTAL: 65 / PASSED: 10 / FAILED: 45 / SKIPPED: 10

Changes from main branch:
  TOTAL: +0
  PASSED: +2
  FAILED: -2

Test improvements (2):
  + 8bit
  + mac-mf

dspinellis · 2026-07-08T07:31:23Z

I force-pushed a few unit tests to also pass codecov/patch.

dspinellis force-pushed the switch-to-byte-handling branch from 9258d95 to 39f18c2 Compare July 7, 2026 11:00

dspinellis force-pushed the switch-to-byte-handling branch from 39f18c2 to 60aa15b Compare July 7, 2026 11:14

dspinellis force-pushed the switch-to-byte-handling branch from 60aa15b to 58f773c Compare July 7, 2026 11:58

dspinellis requested a review from Copilot July 7, 2026 12:13

Copilot started reviewing on behalf of dspinellis July 7, 2026 12:14 View session

Copilot AI reviewed Jul 7, 2026

View reviewed changes

Comment thread src/sed/fast_io.rs Outdated

Comment thread src/sed/processor.rs

Copilot AI mentioned this pull request Jul 7, 2026

Fix UB in IOChunk::as_str(): only cache utf8_verified on successful UTF-8 validation #488

Closed

dspinellis force-pushed the switch-to-byte-handling branch from ef35132 to 431c083 Compare July 7, 2026 14:54

dspinellis assigned sylvestre Jul 7, 2026

dspinellis force-pushed the switch-to-byte-handling branch from 431c083 to c58cecc Compare July 8, 2026 07:15

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Improve GNU compatibility through byte processing#487

Improve GNU compatibility through byte processing#487
dspinellis wants to merge 1 commit into
uutils:mainfrom
dspinellis:switch-to-byte-handling

dspinellis commented Jul 7, 2026

Uh oh!

github-actions Bot commented Jul 7, 2026

Uh oh!

github-actions Bot commented Jul 7, 2026

Uh oh!

codecov Bot commented Jul 7, 2026 •

edited

Loading

Uh oh!

codspeed-hq Bot commented Jul 7, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jul 7, 2026

Uh oh!

github-actions Bot commented Jul 7, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jul 7, 2026

Uh oh!

github-actions Bot commented Jul 7, 2026

Uh oh!

github-actions Bot commented Jul 8, 2026

Uh oh!

dspinellis commented Jul 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

dspinellis commented Jul 7, 2026

Uh oh!

github-actions Bot commented Jul 7, 2026

Uh oh!

github-actions Bot commented Jul 7, 2026

Uh oh!

codecov Bot commented Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

codspeed-hq Bot commented Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will improve performance by 18.22%

Performance Changes

Uh oh!

github-actions Bot commented Jul 7, 2026

Uh oh!

github-actions Bot commented Jul 7, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jul 7, 2026

Uh oh!

github-actions Bot commented Jul 7, 2026

Uh oh!

github-actions Bot commented Jul 8, 2026

Uh oh!

dspinellis commented Jul 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

codecov Bot commented Jul 7, 2026 •

edited

Loading

codspeed-hq Bot commented Jul 7, 2026 •

edited

Loading