feat(prometheus): stream active connections, termination status and bandwidth - #13796
Open
AlinsRan wants to merge 3 commits into
Open
feat(prometheus): stream active connections, termination status and bandwidth#13796AlinsRan wants to merge 3 commits into
AlinsRan wants to merge 3 commits into
Conversation
…andwidth A TCP or UDP proxy could not be monitored from the stream metrics we had. The only one was a per route connection counter, nginx reports a stream $status of 200 for every failure that happens after the upstream connection is established, and no byte counter of a live session is reachable from Lua, so a long-lived connection produced nothing until it ended. Add three metrics, keyed by listen_addr because a session can end before any stream route is matched and because the byte counters come from nginx, which only knows the listening address: - apisix_stream_active_connections, live concurrency per listening address - apisix_stream_status, one count per finished session. $stream_session_reason narrows 200 down to genuine closes and folds timeouts and resets onto the codes nginx itself uses; plugin rejections are carried on ctx because a stream plugin rejects by closing the session, which never sets $status - apisix_stream_bandwidth, bytes per direction and side, read once a second from the nginx zone so it grows while a connection is still open apisix_stream_metrics_zone, $stream_session_reason and $stream_listen_addr come from apisix-nginx-module 1.19.9, so .requirements moves to the first runtime carrying it. Everything degrades on an older runtime: the collector logs once and stays off, and the status metric falls back to $status.
dosubot
Bot
added
size:XL
ci/linux-install-openresty.sh guards the debug deb with a hardcoded version and sha256 per architecture, and bails out when .requirements moves without them. Every job that installs the runtime failed on that guard. Digests taken from the published 1.3.14 release assets.
The live-gauge case scraped the metrics endpoint while its session was open and found no series, although the later case asserting 0 on the same series passed -- so the collector had published, just not before this block scraped. Whether it has ticked by then depends on when the stream plugins finish loading, which is not what this case is about. Read the zone directly, which is where the count lives while the session is open; the published gauge stays covered by the case after it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
A TCP or UDP proxy cannot be monitored from the Stream metrics APISIX has today.
apisix_stream_connection_totalis a per-Route connection counter, and three things are missing behind it:$statusof 200 for every failure that happens after the upstream connection is established — every post-connect path ends inngx_stream_proxy_finalize(s, NGX_STREAM_OK)— so an idle timeout or a reset is indistinguishable from a normal close.$bytes_sentand friends only exist at log time, and Lua has no handle on a live session, so a long-lived connection reports nothing until it ends.This adds three metrics, all keyed by
listen_addrrather than by Route: a session can end before any Stream Route is matched, and the byte counters come from NGINX, which only knows the listening address.apisix_stream_active_connectionslisten_addrapisix_stream_statuscode,listen_addr,nodeapisix_stream_bandwidthlisten_addr,type,sidecodeonly ever takes values NGINX itself uses for Stream sessions — no synthetic codes.$stream_session_reasonnarrows 200 down to genuine closes and folds timeouts and resets onto 400/502. A plugin rejection is carried onctxinstead, because a Stream plugin rejects by closing the session (plugin.lua'srun_plugincallsngx_exit(1)), so the code it returned never reaches$status; it is mapped through the same whitelist so an operator-suppliedrejected_codecannot put an arbitrary value on the metric.apisix_stream_bandwidthandapisix_stream_active_connectionsare read once a second from an NGINX shared memory zone, so they keep moving while a connection is open. Zone size isnginx_config.stream.metrics_zone_size, default1m.Runtime dependency
apisix_stream_metrics_zone,$stream_session_reasonand$stream_listen_addrcome from apisix-nginx-module 1.19.9, so.requirementsmoves to1.3.14, the first APISIX-Runtime carrying it.Everything degrades on an older runtime rather than breaking: the collector logs once and stays off, and the status metric falls back to
$status.Which issue(s) this PR fixes
N/A
Checklist