chore(crashtracking): harden multi thread ptrace collection#2216
chore(crashtracking): harden multi thread ptrace collection#2216gyuheon0h wants to merge 5 commits into
Conversation
📚 Documentation Check Results📦
|
Clippy Allow Annotation ReportComparing clippy allow annotations between branches:
Summary by Rule
Annotation Counts by File
Annotation Stats by Crate
About This ReportThis report tracks Clippy allow annotations for specific rules, showing how they've changed in this PR. Decreasing the number of these annotations generally improves code quality. |
|
🔒 Cargo Deny Results📦
|
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: edd03dc8f1
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
gyuheon0h
changed the title
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: edd03dc8f1
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| match capture_thread_context(tid, resolve_frames, addr_space, stop_deadline) { | ||
| Ok(ctx) => return Some(ctx), | ||
| Err(_) if attempt < CAPTURE_RETRIES => { | ||
| std::thread::sleep(RETRY_DELAY); |
There was a problem hiding this comment.
When ptrace is permanently denied for the target process (for example PTRACE_SEIZE returns EPERM because PR_SET_PTRACER was not granted or Yama rejects the attach), capture_thread_context fails immediately for every TID, but this branch still sleeps 5 ms before retrying an operation that cannot succeed. Since stream_thread_contexts stops once the shared overall_deadline is reached, a many-thread process can burn a large part of the crash receiver budget on these retries and stop invoking callbacks for later threads; before this change those failures were reported quickly as incomplete thread entries. Consider retrying only transient errno values or skipping the delay when the failure is permanent.
Useful? React with 👍 / 👎.
Artifact Size Benchmark Reportaarch64-alpine-linux-musl
aarch64-unknown-linux-gnu
libdatadog-x64-windows
libdatadog-x86-windows
x86_64-alpine-linux-musl
x86_64-unknown-linux-gnu
|
BenchmarksComparisonBenchmark execution time: 2026-07-09 20:48:33 Comparing candidate commit 39e3d72 in PR branch Found 0 performance improvements and 0 performance regressions! Performance is the same for 1 metrics, 0 unstable metrics.
|
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz | 39e3d72 | 1783629609 | gyuheon0h/multi-thread-collection-harden |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| receiver_entry_point/report/2644 | execution_time | 3.360ms | 3.401ms ± 0.039ms | 3.387ms ± 0.014ms | 3.412ms | 3.484ms | 3.508ms | 3.661ms | 8.08% | 2.453 | 9.807 | 1.14% | 0.003ms | 1 | 200 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| receiver_entry_point/report/2644 | execution_time | [3.396ms; 3.406ms] or [-0.158%; +0.158%] | None | None | None |
Baseline
Baseline benchmark details
Group 1
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz | e026a3c | 1783610792 | main |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| receiver_entry_point/report/2644 | execution_time | 3.363ms | 3.400ms ± 0.030ms | 3.390ms ± 0.010ms | 3.405ms | 3.464ms | 3.502ms | 3.517ms | 3.74% | 1.744 | 2.730 | 0.87% | 0.002ms | 1 | 200 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| receiver_entry_point/report/2644 | execution_time | [3.396ms; 3.404ms] or [-0.121%; +0.121%] | None | None | None |
gyuheon0h
force-pushed
the
gyuheon0h/multi-thread-collection-harden
branch
from
d292c2f to
dc76d1f
Compare
1 participant
PROF-15363
What does this PR do?
More effort to harden the ptrace all thread collection crashtracking mechanism
attach_threadinstead of silently proceeding with unreadable registersPTRACE_PEEKUSERon ARM64; this shouldnt matter since the issue in CI we are seeing is on x86 CentOS but it is worth doing since there is only improvement here.capture_with_retryto retry transient ptrace attach failures (1 retry, 5ms delay)-The 5ms delay on failure is not that significant, and worth the cost of getting good stacks vs failing
Motivation
What inspired you to submit this pull request?
Additional Notes
Anything else we should know when reviewing?
How to test the change?
Describe here in detail how the change can be validated.