[logging] Add opt-in TX capture health summaries for TCI handoffs

[jensenpat]

Summary

Adds bounded TX microphone capture-health diagnostics for #4230 without changing audio behavior.

The diagnostics are off by default. TX capture-health summaries are written only when Help → Support's CAT/rigctld logging toggle—the existing category used by TCI debug logging—is enabled. The lightweight in-memory tracker and get audio fields remain available for automation and troubleshooting snapshots, but they do not produce support-log records unless that TCI debug category is on.

The #4230 field log confirmed the failure signature: while TCI audio keeps the 200 ms suppression window fresh, AudioEngine::onTxAudioReady() returns without consuming the local QAudioSource. The Linux/PipeWire capture ring reached 12000/12000 bytes while Qt still reported Active/NoError; the last successful microphone read was 14,173 ms old. This PR detects that full-while-Active condition directly, while retaining Active-to-Idle with unread bytes as a fallback when capacity is unavailable.

Merge order and follow-up fix

Merge this diagnostic PR before #4251. The Linux-only behavioral fix in #4251 is intentionally stacked after this work. Once this PR lands, #4251 reduces to its single behavior commit while these opt-in diagnostics remain available to validate the fix on affected PipeWire systems.

What is recorded

• Tracks suppressed microphone callbacks and peak unread bytes while TCI audio is fresh.
• Records saturation when buffered bytes reach the reported source capacity, even if Qt continues reporting the source as Active.
• Retains the Active-to-Idle-with-unread-bytes signature as a fallback when source capacity is unavailable.
• Counts later local, non-DAX TX starts only while the capture source is currently saturated.
• Clears current saturation after a successful microphone read, while retaining lifecycle history and bounded one-warning-per-class rate limiting.
• Excludes initial Idle state, TX owned by another client on the same radio, DAX TX, and TX while TCI audio is still fresh.
• Includes device, Qt state/error, buffer availability/capacity, source lifetime, last successful microphone-read age, and aggregate counters.

AetherSDR's own local TX remains tracked normally when the radio is operating in Multi-Flex mode. Only TX owned by another client is excluded, because that remote transmission does not consume this AetherSDR process's microphone capture stream and must not be counted as a local capture failure.

The existing get audio automation snapshot exposes the same TX endpoint evidence:

• buffer_bytes_available
• buffer_capacity_bytes
• source_was_active
• saturation_observed
• tci_suppressed_callbacks
• idle_during_tci_transitions
• full_buffer_during_tci_observations
• post_tci_local_tx_while_saturated
• last_mic_read_age_ms

Review findings addressed

• audioEndpointDiagnostics() now self-marshals with a blocking queued invocation when called off the AudioEngine thread. This removes the troubleshooting-dialog race on QAudioSource, QIODevice, QElapsedTimer, and the non-atomic tracker snapshot while preserving direct execution for callers already on the owner thread.
• Current saturation is now separate from lifecycle-level observation/rate-limiting. A successful microphone read clears the current condition, so later healthy local TX attempts do not inflate post_tci_local_tx_while_saturated; historical evidence remains available in the lifecycle summary.

Related PipeWire / Linux audio tracking

These references are the working set of PipeWire/Linux audio failure and lifecycle reports relevant to field correlation. They are not asserted to share one root cause; the diagnostics distinguish local QAudioSource capture saturation from DAX routing, stream ownership, clocking, and device-change failures.

Current reports:

• Refs Linux PipeWire: TCI TX audio leaves PC Audio SSB transmit path stuck after transmission #4230 — TCI TX leaves the subsequent PC Audio SSB path stuck. The new log confirms local capture saturation while the source remains Active.
• Refs DAX-TX forward power drops to ~0 W during digital TX; re-establishing the dax_tx stream restores it #4152 — Linux/PipeWire DAX TX falls to near-zero RF until the dax_tx stream is recreated.
• Refs DAX TX audio reaches AetherSDR but is not transmitted in SSB modes (works in DIGU) #3805 — PipeWire DAX TX reaches AetherSDR but produces no SSB RF while the PC Audio PipeWire input works.

Prior PipeWire/Linux audio lifecycle and quality reports:

• Refs PipeWire DAX TX produces smeared FT8 signal vs clean TCI; DAX TX also doesn't release MOX cleanly at end of transmission #2286 — PipeWire DAX TX smearing and incomplete end-of-transmission teardown.
• Refs Low-Latency DAX TX drops audio when PTT is triggered by external client #752 — PipeWire DAX TX audio dropped when an external client owns PTT.
• Refs WSJT-x DT drift on receives during/after transmitting cycles. #537 — WSJT-X receive timing drift after repeated TX cycles on Linux.
• Refs Current DAX Implementation higher latency than SSDR DAX Latency #1008 — DAX latency on Linux, leading to the native PipeWire stream work.
• Refs No audio on Ubuntu 24.04 LTS with gstreamer1.0-pulseaudio and gstreamer1.0-pipewire installed #75 — Ubuntu PipeWire/PulseAudio compatibility produced no enumerated PC audio devices.
• Refs Linux Pipewire causes pop-up for device changes #2864 — PipeWire device-change churn repeatedly reopened the audio-device warning.
• Refs DAX virtual audio: only slice 0 gets audio client on Linux PipeWire headless (RPi 5, Ubuntu 24.04, FLEX-8600) #2895 — headless Linux/PipeWire DAX channels remained Idle except for slice 0.
• Refs No tci audio again and no dax audio #3363 — TCI RX and DAX audio both stopped during operation.
• Refs TCI audio (DAX 1) does not start on connect; requires mode change + DAX toggle to recover #3669 — TCI/DAX audio required mode and DAX toggles to re-arm.
• Refs DAX + TCI re-assert storm: continuous "slice set N dax=<ch>" spam when WSJT-X connects via TCI with DAX active (Linux/macOS) #4009 — Linux/macOS DAX+TCI re-assert loop caused stream churn and audio jitter.

Issues that only concern packaging, PipeWire node naming, or channel-pair selection are intentionally not listed as capture-stall candidates.

Validation

• Configured and built the complete application with Ninja and 8 build jobs.
• Full CTest suite: 114/114 passed; the quarantined clean-room test was skipped by its existing policy.
• Hardware-free TX capture-health state-machine test: 22/22 assertions passed, including Active/full-buffer saturation, the Idle fallback, recovery after a successful read, rate limiting, DAX exclusion, and exclusion of TX owned by another client.
• tools/check_engine_boundary.py --strict: zero blocking findings.
• git diff --check: clean.
• Agent automation bridge: ping and get audio passed; the TCI/CAT logging category was toggled off and restored; Slice Troubleshooting opened and refreshed repeatedly through the formerly racy path with the app remaining responsive. No radio connection or TX was performed.

Field-validation result

The first affected PipeWire support log supplied the decisive evidence this PR was designed to collect: the ring was full while the Qt source remained Active. #4251 applies the narrow Linux-only drain/discard fix; affected users can enable the TCI debug logging category when validating that they no longer reach saturation.

Screenshots

Not applicable: this is opt-in support logging and diagnostics instrumentation with no visual UI change.

👨🏼‍💻 Generated with OpenAI Codex (GPT-5.6 Sol) and tested by @jensenpat

[aethersdr-agent]

Nicely scoped diagnostic PR. The TxCaptureHealthTracker is a clean hardware-free state machine, the conservative saturation signature (Active→Idle + suppressed callbacks + unread bytes + TCI fresh) is well-reasoned, the anomaly rate-limiting is correct, and the state-machine test pins each class (initial-Idle-is-healthy, saturation, DAX exclusion, foreign-owner exclusion, one-log-per-lifecycle). Threading care is evident — recordTxCaptureLocalTxAttempt is marshaled onto the owning thread and the stateChanged lambda guards against a stale source. All files are in scope, no AppSettings/QSettings misuse, no new deps, commit is signed. Default-on + bounded + no per-callback logging matches the stated field-evidence goal.

Non-blocking notes

• audioEndpointDiagnostics() reads the tracker (and m_micDevice->bytesAvailable()) from the caller's thread while the tracker mutates on the audio thread — inline comment. Consistent with the existing pattern in that function, so not a blocker.

Thanks for the thorough writeup and the field-validation boundary being called out honestly — that's exactly the right framing for a diagnostic-only change.

---

_{🤖 aethersdr-agent · cost: $4.9911 · model: claude-opus-4-8}

[aethersdr-agent]

audioEndpointDiagnostics() is const and invoked from AutomationServer / DeviceDiagnostics on the main thread, but m_txCaptureHealth and m_micDevice are owned and mutated on the audio thread (onTxAudioReady, the queued stateChanged lambda, recordMicRead). So snapshot() here — and txCaptureBufferedBytes()'s m_micDevice->bytesAvailable() call one line below — race the audio thread's writes on non-atomic members.

This matches the pre-existing pattern in this same function (it already reads m_audioSink->state(), m_micDevice->isOpen(), etc. cross-thread), so it's not a regression and the torn reads are diagnostic-only. Worth flagging only because you deliberately marshaled recordLocalTxAttempt onto the owning thread "so diagnostics cannot race readyRead/stateChanged" — the read side of the same data isn't marshaled. Fine to leave as-is for a support-log snapshot; just noting the asymmetry.

[NF0T]

Reviewed this alongside #4251 (which stacks on top of this one). Two findings that are specifically this PR's own — reproducible with no drain/Linux-specific code involved at all:

1. Cross-thread read of audioEndpointDiagnostics() is a real race, not just a "widened surface." The stacked PR's bot review flagged this as pre-existing and non-blocking, but I traced both call paths: AutomationServer's get audio correctly marshals via Qt::BlockingQueuedConnection onto the audio thread before calling audioEndpointDiagnostics() — but SliceTroubleshootingDialog::refreshSnapshot() → DeviceDiagnostics::buildAudioDevicesSnapshot() calls it directly from the GUI thread with no marshaling at all. AudioEngine lives on a dedicated audio thread (moveToThread in MainWindow.cpp), so this is genuine UAF risk on m_audioSource (nulled and deleted on the audio thread with no lock while the GUI thread can be mid-read), and genuine UB on the new TxCaptureHealthTracker::Snapshot this PR adds (9 plain non-atomic fields — torn/inconsistent reads are possible) and on the new m_txCaptureHealthClock (a QElapsedTimer raced between .elapsed() on one thread and .restart() on the other). The m_audioSource UAF pattern pre-dates this PR, but this PR adds three new non-atomic fields and a second racy timer into that exact unsafe path. Worth fixing by marshaling SliceTroubleshootingDialog's path the same way AutomationServer already does, rather than carrying it forward as non-blocking.

2. m_saturationReported latches for the entire source lifecycle. In TxCaptureHealthTracker::recordLocalTxAttempt(), saturated is m_saturationReported || (...), and nothing resets that flag except a full reset(). So once any saturation event fires once, every later local-TX attempt for the rest of the session gets counted as "stalled" regardless of whether the buffer has since recovered — inflating exactly the field evidence this instrumentation exists to collect cleanly. If "sticky forever" is intentional, worth a comment saying so; if not, it probably wants a way to clear once the buffer's healthy again.

Both are independent of #4251's Linux drain — they reproduce in this PR alone. I posted two different, drain-interaction-specific findings on #4251 itself, since those only manifest once the drain commit is layered on top of this one.

[jensenpat]

@NF0T Thanks for tracing these. I’m addressing both #4233 findings on the existing author branch now: self-marshaling audioEndpointDiagnostics() onto the AudioEngine thread, and separating current saturation state from lifecycle rate-limiting so a successful mic read clears the current stall. I’m also gating the TX capture-health summaries behind the existing TCI debug logging category per maintainer direction.

[ten9876]

Approving — clean, well-tested diagnostic

High-effort pass. TxCaptureHealthTracker is a clean hardware-free state machine, reset per source lifecycle, and this hardened version correctly uses the transient m_currentlySaturated (cleared on recordMicRead) rather than the lifecycle-level flag — so a recovered buffer isn't flagged as still-stalled on a later healthy TX. Instrumentation-only (no audio-path behavior change), one-log-per-lifecycle rate-limiting is correct, the state-machine test pins each class, and there's no AppSettings/QSettings misuse.

The one finding — audioEndpointDiagnostics() reads m_txCaptureHealth/m_micDevice->bytesAvailable() cross-thread — is genuinely non-blocking: diagnostic-only torn reads, consistent with the pre-existing state()/error()/isOpen() reads in the same method. The asymmetry with the marshaled recordLocalTxAttempt write is worth a future tightening (marshal the snapshot or expose atomics) but doesn't gate a support-log snapshot.

This is the correct base for #4251 — merging first, after which #4251 reduces to the Linux drain commit and inherits this hardened tracker. Thanks for the field-evidence writeup and the honest validation boundary, @jensenpat.

[ten9876]

Maintainer approval (Tier 1) — with two nit fixes as suggestions

High-effort pass. This is a clean, well-tested opt-in diagnostic: TxCaptureHealthTracker is a solid hardware-free state machine (saturation, DAX/foreign-owner/TCI-fresh exclusions, one-log-per-lifecycle, recovery, idle-fallback all covered in tx_capture_health_test), the mutation-side threading is careful, and the earlier read-race the bot flagged is already addressed by marshaling audioEndpointDiagnostics() onto the owning thread. No correctness bug — it's observability-only.

Two low-severity nits, posted inline as one-click suggestions (I built + ran tx_capture_health_test (22/22) and a full app build with both applied — clean):

1. Per-callback tracker work (incl. bytesAvailable()/bufferSize()) runs on the onTxAudioReady hot path even when the diagnostic is off — gate it on the same lcCat().isDebugEnabled() the logger already checks.
2. formatTxCaptureHealth()'s first line uses a multi-arg .arg() followed by a numeric .arg(), so a stray %N in a hardware device description could be re-substituted — collapse to a single-pass .arg().

Intentionally not changed: the audioEndpointDiagnostics() BlockingQueuedConnection — that's the correct, consistent Qt pattern for reading the tracker + sink/mic state together; a non-blocking read would reintroduce the torn reads this PR deliberately closed. Keep it as-is.

Approving now. Apply the two suggestions when you get a chance and I'll arm auto-merge once they're in (or say the word and I'll merge as-is — they're non-blocking). Thanks for the careful writeup and honest field-validation boundary, @jensenpat.

[ten9876]

Nit fix (per-callback work when the diagnostic is off): gate the hot-path tracker updates + their device queries on the same category logTxCaptureHealthSummary already checks, so a default-off support toggle does no per-callback work on the audio thread. The suppression return below stays unconditional. Built + tx_capture_health_test green.

Suggested change

	const TxCaptureHealthTracker::Event event = m_txCaptureHealth.recordSuppressedCallback(
	txCaptureBufferedBytes(), txCaptureBufferCapacityBytes());
	if (event != TxCaptureHealthTracker::Event::None) {
	logTxCaptureHealthEvent(event);
	}
	if (m_audioSource) {
	observeTxCaptureState(m_audioSource->state());
	}
	// Per-callback capture-health tracking is only ever surfaced when the
	// opt-in support toggle is on (logTxCaptureHealthSummary gates on the
	// same category), so skip the tracker updates and their device queries
	// on this hot path when the diagnostic is disabled. The suppression
	// early-return below is core behavior and must always run.
	if (lcCat().isDebugEnabled()) {
	const TxCaptureHealthTracker::Event event = m_txCaptureHealth.recordSuppressedCallback(
	txCaptureBufferedBytes(), txCaptureBufferCapacityBytes());
	if (event != TxCaptureHealthTracker::Event::None) {
	logTxCaptureHealthEvent(event);
	}
	if (m_audioSource) {
	observeTxCaptureState(m_audioSource->state());
	}
	}

[ten9876]

Same gate for the normal-path consume marker.

Suggested change

	m_txCaptureHealth.recordMicRead(txCaptureNowMs());
	if (lcCat().isDebugEnabled()) {
	m_txCaptureHealth.recordMicRead(txCaptureNowMs());
	}

[ten9876]

Nit fix (%N in a device name): substitute the first line in one multi-arg .arg() so a stray %N in a hardware device description can't be re-substituted by the trailing numeric .arg(lifecycleMs).

Suggested change

	<< QStringLiteral(" reason=\"%1\" %2 state=%3 error=%4 lifetime=%5ms")
	.arg(valueOrUnknown(summary.reason),
	field(QStringLiteral("device"), summary.deviceDescription),
	valueOrUnknown(summary.state),
	valueOrUnknown(summary.error))
	.arg(summary.lifecycleMs)
	// Single-pass multi-arg substitution: a stray "%N" inside a
	// hardware device description must not be re-substituted by a
	// trailing numeric .arg().
	<< QStringLiteral(" reason=\"%1\" %2 state=%3 error=%4 lifetime=%5ms")
	.arg(valueOrUnknown(summary.reason),
	field(QStringLiteral("device"), summary.deviceDescription),
	valueOrUnknown(summary.state),
	valueOrUnknown(summary.error),
	QString::number(summary.lifecycleMs))