enhancement(app): add support for always-on compressed storage of debug logs

[tobz]

Summary

This PR adds support for always-on compressed storage of debug logs to allow for "looking back" at previous debug logs without having to have explicitly set the application's log level to DEBUG or higher.

In many debugging scenarios, it can often be necessary to change configuration settings to collect more verbose output to assist in said debugging. However, this can often be suboptimal because it means the adverse conditions must be triggered again in order to be captured... when if only you had diagnostics/debug output from the initial adverse condition, you could avoid having to spend the time to recreate the issue.

This PR introduces a new logging mechanism that aims to keep as many of the most recent debug logs as possible in a fixed amount of space, in a rotating fashion, such that when we want to debug an adverse condition that recently occurred, we can skip first enabling debug logging and then having to recreate the condition... the logs are just already there, and automatically present in flare output. It employs a number of techniques that allow it to store a large number of debug logs, in a nearly lossless form, in a small amount of space.

As part of this PR, we've done the following:

• created a new CompressedRingBuffer layer implementation for tracing
• wired it up in saluki-app to collect debug logs by default (configurable), up to 1MiB (that is, it uses a maximum of 1MiB of actual heap to store the values in memory)
• wired up a diagnostic collector such that the log is present in flare output from ADP

Architecturally, this implementation works by:

• taking individual log events and breaking them into their constituent pieces -- level, target, message, structured fields -- and then storing them in the most appropriate way: delta/RLE encoding, string interner, log message template extraction, and so on
• "condensed" events are packed into "segments" with enough information to extract an equivalent represent of the original log events, and then the segments are compressed
• segments are kept around until we exceed our target maximum in-memory size, with the oldest segments being dropped
• all of this happens on a dedicated background thread, decoupled by a fixed-size channel (with pre-allocation/string reuse tricks), in order to minimize overhead/latency in the caller-facing side of the logging system, while also ensuring deterministic memory usage

Change Type

• Bug fix
• New feature
• Non-functional (chore, refactoring, docs)
• Performance

How did you test this PR?

New and existing tests.

References

DADP-2

[datadog-datadog-us1-prod]

✨ Fix all issues with BitsAI

⚠️ Warnings

🚦 3 Pipeline jobs failed

DataDog/saluki | check-docs     

DataDog/saluki | check-licenses     

Semantic PR Title Check | Check For Semantic PR Title     

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 54ecf80 | Docs | Datadog PR Page | Give us feedback!}