multiline: add built-in json parser for JSON multiline objects - #12264
multiline: add built-in json parser for JSON multiline objects#12264lecaros wants to merge 2 commits into
Conversation
Signed-off-by: lecaros <lecaros@chronosphere.io>
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
📝 WalkthroughWalkthroughThe change adds built-in JSON multiline parser registration, exposes its factory, lists ChangesJSON multiline parser support
Estimated code review effort: 2 (Simple) | ~10 minutes Sequence Diagram(s)sequenceDiagram
participant Config
participant BuiltinFactory
participant JSONParser
participant JSONTest
Config->>BuiltinFactory: selects json parser
BuiltinFactory->>JSONParser: creates parser
JSONTest->>JSONParser: appends fixtures and flushes pending data
JSONParser-->>JSONTest: emits expected records
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: cc6f17f9da
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tests/internal/multiline.c`:
- Around line 367-400: Add invalid-boundary fixtures to json_input and
json_output covering an opening brace followed by an unindented line and a
closing-brace line with trailing content. Assert that each malformed
continuation does not merge into the active JSON group, and include both the
resulting valid records and invalid payload outcomes, including the relevant
boundary-size cases.
- Around line 1229-1230: Restore the flb_time_get(&tm) call immediately before
flb_ml_append_text in the affected test flow, ensuring tm is initialized before
being passed to the append operation and removing the invalid `-.` artifact.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 5fe3bb36-a478-4ea5-9d3a-cc31c23d07ad
📒 Files selected for processing (6)
include/fluent-bit/multiline/flb_ml_parser.hplugins/filter_multiline/ml.cplugins/in_tail/tail.csrc/multiline/CMakeLists.txtsrc/multiline/flb_ml_parser.ctests/internal/multiline.c
| /* JSON (pretty-printed and single-line objects) */ | ||
| struct record_check json_input[] = { | ||
| {"{\"id\":101,\"level\":\"info\",\"msg\":\"single-line record A\"}"}, | ||
| {"{"}, | ||
| {" \"id\": 102,"}, | ||
| {" \"level\": \"warn\","}, | ||
| {" \"msg\": \"multiline record B\""}, | ||
| {"}"}, | ||
| {"{\"id\":103,\"level\":\"info\",\"msg\":\"single-line record C\"}"}, | ||
| {"{"}, | ||
| {" \"id\": 104,"}, | ||
| {" \"level\": \"error\","}, | ||
| {" \"msg\": \"multiline record D\""}, | ||
| {"}"}, | ||
| }; | ||
|
|
||
| struct record_check json_output[] = { | ||
| {"{\"id\":101,\"level\":\"info\",\"msg\":\"single-line record A\"}\n"}, | ||
| { | ||
| "{\n" | ||
| " \"id\": 102,\n" | ||
| " \"level\": \"warn\",\n" | ||
| " \"msg\": \"multiline record B\"\n" | ||
| "}\n" | ||
| }, | ||
| {"{\"id\":103,\"level\":\"info\",\"msg\":\"single-line record C\"}\n"}, | ||
| { | ||
| "{\n" | ||
| " \"id\": 104,\n" | ||
| " \"level\": \"error\",\n" | ||
| " \"msg\": \"multiline record D\"\n" | ||
| "}\n" | ||
| }, | ||
| }; |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Add invalid continuation boundary fixtures.
json_input only tests records that must group. Add cases where { is followed by an unindented line and where a closing-brace line has extra content. Assert that the parser does not merge those records with the active JSON group.
As per coding guidelines, validate both success and failure paths, including invalid payloads and boundary sizes.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tests/internal/multiline.c` around lines 367 - 400, Add invalid-boundary
fixtures to json_input and json_output covering an opening brace followed by an
unindented line and a closing-brace line with trailing content. Assert that each
malformed continuation does not merge into the active JSON group, and include
both the resulting valid records and invalid payload outcomes, including the
relevant boundary-size cases.
Source: Coding guidelines
cc6f17f to
61b3d44
Compare
Signed-off-by: lecaros <lecaros@chronosphere.io>
61b3d44 to
c95af44
Compare
1 participant
Add built-in multiline parser
jsonfor tail inputs that read JSON objects split across lines.The parser groups lines only (regex start
^\{.*, continuation onwhitespace-indented lines or a lone
}). It does not embed a JSON parser;per-line JSON parsing fails on partial lines such as
{or"id": 102,.Expected usage:
multiline.parser: jsonon tail, thenfilter_parserwithkey_name: logandparser: json.parser: jsonon tail; no multilineparser.
Addresses #2418, #8232
Enter
[N/A]in the box, if an item is not applicable to your change.Testing
Before we can approve your change; please submit the following in a comment:
config
oneline.json
{"id":1,"time":"2026-08-11T10:00:00.000","level":"info","msg":"oneline record one"} {"id":2,"time":"2026-08-11T10:00:01.000","level":"warn","msg":"oneline record two"} {"id":3,"time":"2026-08-11T10:00:02.000","level":"error","msg":"oneline record three"}mixed.json
{"id":101,"time":"2026-08-11T11:00:00.000","level":"info","msg":"single-line record A"} { "id": 102, "time": "2026-08-11T11:00:01.000", "level": "warn", "msg": "multiline record B", "details": { "user": "alice", "action": "login" } } {"id":103,"time":"2026-08-11T11:00:02.000","level":"info","msg":"single-line record C"} {"id": 104, "time": "2026-08-11T11:00:03.000", "level": "error", "msg": "multiline record D", "stack": [ "frame1", "frame2" ] } {"id":105,"time":"2026-08-11T11:00:04.000","level":"info","msg":"single-line record E"}debug output
Documentation
Backporting