Skip to content

Speed up object child lookup - #173

Merged
rgerhards merged 3 commits into
masterfrom
codex/libfastjson-performance-analysis
Jul 23, 2026
Merged

Speed up object child lookup#173
rgerhards merged 3 commits into
masterfrom
codex/libfastjson-performance-analysis

Conversation

@rgerhards

@rgerhards rgerhards commented Jul 22, 2026

Copy link
Copy Markdown
Member

What changed

  • replace iterator API calls in the private _fjson_find_child() hot path with direct traversal of the existing child pages
  • preserve case comparison, insertion order, deleted-slot handling, and key/value ownership
  • add multipage, deletion-hole, replacement, and comparison-mode regression coverage
  • add a reusable paired benchmark suite with normalized current-main and candidate evidence

Why

Rsyslog's regular $!varname path performs repeated libfastjson object lookups. Profiling attributed about 13-16% of the variable-heavy path to child lookup. The existing implementation paid iterator setup and function-call overhead for every comparison even though the private page representation is directly available here.

Performance

Two standalone paired sessions, each with one calibration and 11 measured pairs, found:

  • first-key lookup: about 3.12x faster
  • width-8 last-key lookup: 2.31-2.36x faster
  • width-16 last-key lookup: 1.90-1.94x faster
  • replacement writes: 1.17-1.88x faster
  • no core workload regression above 5%

Two full rsyslog sessions using 2,097,152 messages per trial found the following repeatable single-worker throughput changes:

  • flat $! reads: +9.68% / +13.53%
  • flat $! writes: +6.84% / +6.18%
  • three-level mixed access: +17.19% / +17.31%
  • 4-KiB object copies: +7.49% / +7.74%

Eight-worker results were neutral or noisy, and no conclusive rsyslog workload regressed above 5%. The change should therefore be presented as a useful lookup improvement, not a general double-digit rsyslog throughput claim.

Validation

  • libfastjson test suite: 17/17
  • ASAN + UBSAN test suite: 17/17
  • full Valgrind test suite
  • Clang static analysis
  • exported ABI symbol comparison
  • make distcheck
  • deterministic benchmark self-tests and style checks
  • two independent standalone paired benchmark sessions
  • two independent full-lifecycle rsyslog paired sessions
  • local Cubic review: no actionable findings

The existing tests/parse_flags.c -Wsizeof-array-div warning was demoted from error during full local validation; it is unrelated to this change.


Summary by cubic

Speeds up libfastjson object child lookup by walking child pages directly in _fjson_find_child(), removing iterator overhead. Adds a reproducible microbenchmark suite and documents the traversal semantics.

  • Refactors

    • Replace iterator API in _fjson_find_child() with direct child‑page traversal.
    • Preserve behavior: case sensitivity, insertion order, deleted‑slot handling, and key/value ownership.
    • Add inline docs in _fjson_find_child() explaining traversal rationale and case‑mode semantics.
  • New Features

    • Add a reproducible paired microbenchmark suite (benchmarks/benchmark.c, runner.py, run.sh, README.md) with self‑tests; records calibration, medians/MAD, outliers, raw JSON, and build/source fingerprints; requires clean Git checkouts; ship runner, results, and reports via Makefile.am.
    • Add regression tests for multipage objects, deletion holes, replacements, and both comparison modes, including a new lookup test in tests/test_charcase.c.

Written for commit 9babea4. Summary will update on new commits.

Review in cubic

@rgerhards

Copy link
Copy Markdown
Member Author

@cubic-dev-ai review

@cubic-dev-ai

cubic-dev-ai Bot commented Jul 22, 2026

Copy link
Copy Markdown

@cubic-dev-ai review

@rgerhards I have started the AI code review. It will take a few minutes to complete.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 23 files

Heads up: you’re close to your included review allowance. Set a flex budget so reviews don’t pause.

Tip: instead of fixing issues one by one fix them all with cubic

Re-trigger cubic

Comment thread benchmarks/benchmark.c
Comment thread benchmarks/results/direct-lookup-b-rerun.json
Comment thread benchmarks/results/direct-lookup-findings.json
Comment thread benchmarks/results/direct-lookup-b-rerun.json
Comment thread Makefile.am Outdated
@rgerhards rgerhards self-assigned this Jul 23, 2026
@rgerhards
rgerhards force-pushed the codex/libfastjson-performance-analysis branch from f131504 to d02e65a Compare July 23, 2026 08:13
@github-advanced-security

Copy link
Copy Markdown

You are seeing this message because GitHub Code Scanning has recently been set up for this repository, or this pull request contains the workflow file for the Code Scanning tool.

What Enabling Code Scanning Means:

  • The 'Security' tab will display more code scanning analysis results (e.g., for the default branch).
  • Depending on your configuration and choice of analysis tool, future pull requests will be annotated with code scanning analysis results.
  • You will be able to see the analysis results for the pull request's branch on this overview once the scans have completed and the checks have passed.

For more information about GitHub Code Scanning, check out the documentation.

@rgerhards rgerhards added this to the 1.2607.0 milestone Jul 23, 2026
@rgerhards
rgerhards force-pushed the codex/libfastjson-performance-analysis branch 3 times, most recently from d02e65a to 6b11f09 Compare July 23, 2026 09:18
@rgerhards
rgerhards marked this pull request as ready for review July 23, 2026 11:05
@rgerhards
rgerhards merged commit de077f5 into master Jul 23, 2026
21 checks passed
@rgerhards
rgerhards deleted the codex/libfastjson-performance-analysis branch July 23, 2026 11:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants