input_chunk: storage limit eviction with more than 32 outputs discards records from other routes

[terrorobe]

Bug

With more than 32 outputs, filesystem storage-limit eviction stores over-limit output IDs in a 32-bit mask. On Linux/amd64, higher output IDs are effectively reduced modulo 32, so different outputs reuse the same bit. When one output reaches its limit, Fluent Bit can therefore remove a chunk's routes to other outputs that remain below their limits. If that leaves the chunk without any routes, Fluent Bit destroys it and discards the log record.

I reproduced this with the official linux/amd64 Fluent Bit 5.1.0 container image, which is the latest upstream release.

Test setup

Fluent Bit assigns output IDs in configuration order. The reproduction uses that deterministic ordering to create these instances:

• file.0 matches both test records and has a 1 MiB storage limit, leaving it well below capacity throughout the test.
• null.1 through null.31 match only unused filler.* tags. They do not receive either test record; they only place the final output at ID 32.
• http.32 matches both test records, points to an unreachable local port so it cannot drain its queue, and has an 8 KiB storage limit.

Two dummy inputs each emit one record under distinct many.* tags, creating separate filesystem-backed chunks. Both chunks are routed to file.0 and http.32. On the official amd64 image, the 8 KiB limit admits the first chunk but is crossed when Fluent Bit places the second chunk.

Correct limit enforcement should remove only http.32's route from the oldest chunk. The chunk must remain queued for file.0, which still has capacity, so the file output should receive both records. The bug also clears file.0's route, leaves that chunk with no destinations, and destroys it before the file output can flush it.

Version

$ docker run --rm --platform linux/amd64 fluent/fluent-bit:5.1.0 --version
Fluent Bit v5.1.0
Git commit: fe293d4270b27bc078e0bb82e7aa2afe08e723f8

Reproduction

set -euo pipefail

root=$(mktemp -d)
name="flb-storage-limit-repro-$$"
mkdir -p "$root/storage" "$root/out"
conf="$root/repro.conf"

cat >"$conf" <<'EOF'
[SERVICE]
    Flush          2
    Grace          1
    Log_Level      debug
    storage.path   /tmp/storage

[INPUT]
    Name           dummy
    Tag            many.one
    Samples        1
    Dummy          {"message":"one"}
    storage.type   filesystem

[INPUT]
    Name           dummy
    Tag            many.two
    Samples        1
    Dummy          {"message":"two"}
    storage.type   filesystem

# Output ID 0: both records should remain routed here.
[OUTPUT]
    Name                       file
    Match                      many.*
    Path                       /tmp/out
    File                       records.log
    Format                     plain
    storage.total_limit_size   1M
EOF

# Create unrelated output IDs 1 through 31.
for i in $(seq 1 31); do
    cat >>"$conf" <<EOF

[OUTPUT]
    Name           null
    Match          filler.$i
EOF
done

# Output ID 32: its limit should affect only this route.
cat >>"$conf" <<'EOF'

[OUTPUT]
    Name                       http
    Match                      many.*
    Host                       127.0.0.1
    Port                       9
    Retry_Limit                false
    storage.total_limit_size   8K
EOF

docker run --rm --platform linux/amd64 --name "$name" \
    -v "$conf:/fluent-bit/etc/repro.conf:ro" \
    -v "$root/storage:/tmp/storage" \
    -v "$root/out:/tmp/out" \
    fluent/fluent-bit:5.1.0 \
    -c /fluent-bit/etc/repro.conf \
    >"$root/fluent-bit.log" 2>&1 &
pid=$!

sleep 6
docker stop -t 1 "$name" >/dev/null 2>&1 || true
wait "$pid" || true

grep 'drop chunk.*with no output route' "$root/fluent-bit.log"
wc -l "$root/out/records.log"
cat "$root/out/records.log"

Actual result

Fluent Bit reports that it destroyed a chunk after removing all of its routes:

[debug] [input chunk] drop chunk ... with no output route from input plugin dummy.1

The unconstrained file output receives only one record:

1 /tmp/.../out/records.log
{"message":"two"}

The {"message":"one"} record is discarded.

Expected result

Output ID 32 may evict its own route from the oldest chunk to remain under its 8 KiB limit. It must not remove the route to output ID 0.

The file output should receive both records:

{"message":"one"}
{"message":"two"}

Cause

The storage-limit path represents over-limit outputs in an int bitmask:

overlimit |= (1 << o_ins->id);

Chunk routing masks support more than 32 outputs, but this fixed-width mask cannot represent them. Shifting a signed int into or beyond its sign bit is undefined in C. On Linux/amd64 in this reproduction, the shift count is effectively reused modulo 32: output ID 32 aliases ID 0, ID 33 aliases ID 1, and so on. Higher IDs can therefore collide with any lower ID, not only ID 0; the exact behavior is not portable because the shifts are undefined.

flb_input_chunk_find_space_new_data() consequently treats both outputs 0 and 32 as over limit in this reproduction. It clears both routes from the oldest chunk, and the chunk is destroyed once its route mask becomes empty.

This can discard log data from an unrelated output that has not reached its configured limit.

Proposed fix

Check each routed output directly against its current filesystem usage and storage.total_limit_size instead of representing over-limit outputs in a fixed-width integer.

Related implementation and regression coverage:

• input_chunk: fix storage limit eviction with 32 or more outputs #12235