avoid panics in TestPublisherStopShutdownInactive by using in memory logs

[aelnahas]

Motivation/summary

TestPublisherStopShutdownInactive is flaky due to a potential for a routine to write to the logs after the test function has returned. The shutdown sequence used in the test cleanup:
https://github.com/elastic/beats/blob/50ef57e75fbd5b2923a308da39a27d1ce24185c3/libbeat/publisher/pipeline/consumer.go#L237 might happen before the queueReader had a chance to log its stop message. Therefore, the cleanup might exit too early. This commit bypasses this problem by using logp.NewTestingLoggerWithObserver which still returns a logging output using *testing.T but an in memory slice of the logs to bypass the cleanup panics.

Reference for ogp.NewTestingLoggerWithObserver: https://github.com/elastic/elastic-agent-libs/blob/main/logp/logptest/logptest.go#L46

Checklist

• Update CHANGELOG.asciidoc
• Documentation has been updated

For functional changes, consider:

• Is it observable through the addition of either logging or metrics?
• Is its use being published in telemetry to enable product improvement?
• Have system tests been added to avoid regression?

How to test these changes

I used the following steps to reproduce the problem and also check if the fix works

1. Clone beats and add the replace directive + go mod tidy
2. Add the delay to queueReader after the "stop" log: https://github.com/elastic/beats/blob/50ef57e75fbd5b2923a308da39a27d1ce24185c3/libbeat/publisher/pipeline/queue_reader.go#L76
3. In newTestLogger, swap the body to return logptest.NewTestingLogger(t, selector)
4. Run go test -count=2 ./internal/publish/... and observe the panic
5. Revert step 3 and re-run to confirm it's fixed

Related issues

#21375

[github-actions]

🤖 GitHub comments

Just comment with:

• run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

[infra-vault-gh-plugin-prod]

💚 Build Succeeded

• Buildkite Build
• Commit: f62c8b1

[carsonip]

given that this is likely caused by a regression from beats, could you gauge how much more effort it is to address the root cause (or are we using the pipeline correctly in our usage), before we jump to a workaround?

[mergify]

This pull request does not have a backport label. Could you fix it @aelnahas? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-8.19 is the label to automatically backport to the 8.19 branch.
• backport-9./d is the label to automatically backport to the 9./d branch. /d is the digit.
• backport-active-all is the label that automatically backports to all active branches.
• backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.