Skip to content

[FFI] Complete Arrow shutdown before free returns - #736

Open
nikolaobradovic-db wants to merge 2 commits into
mainfrom
nikola-obradovic_data/zerobus/ffi_use_after_free_fix
Open

[FFI] Complete Arrow shutdown before free returns#736
nikolaobradovic-db wants to merge 2 commits into
mainfrom
nikola-obradovic_data/zerobus/ffi_use_after_free_fix

Conversation

@nikolaobradovic-db

Copy link
Copy Markdown
Contributor

What changes are proposed in this pull request?

  • Update zerobus_arrow_stream_free to wait until Arrow background work stops, request bodies reach EOF or are dropped, and retained batches are released before returning.
  • Previously, tonic could retain an imported C Data batch after the stream handle was freed. Its release callback could then access producer state that had already been destroyed.
  • The change adds request-body lifecycle tracking, supervisor reaper waiting, runtime-aware blocking shutdown, recurring warnings for stalled shutdown, and fail-stop handling when shutdown cannot complete safely. Native Rust Drop remains nonblocking; only C FFI destruction waits for complete shutdown.
  • Go finalizers delegate blocking cleanup to a goroutine. Go and C++ documentation describes the updated destruction and concurrency behavior. No C ABI signatures change.

How is this tested?

  • Added deterministic C Data lifetime tests covering:
  • release before free returns;
  • transport-retained owners;
  • release outside SDK mutexes;
  • multi-thread Tokio runtime destruction.
  • Added request-body EOF/fusion and supervisor-reaper regression tests.

@nikolaobradovic-db
nikolaobradovic-db force-pushed the nikola-obradovic_data/zerobus/ffi_use_after_free_fix branch from 42b4240 to d5e950f Compare August 18, 2026 08:16
Comment thread rust/ffi/src/arrow.rs
{
tokio::task::block_in_place(|| abort_and_drop_arrow_stream(stream));
}
Ok(_) => abort_and_drop_arrow_stream_on_thread(stream),

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This blocking branch can deadlock for a direct FFI consumer calling free from a single-thread Tokio runtime when a C Data release callback waits for work on that same runtime. I reproduced this locally. Our wrappers call free, but they use IPC ingestion only, so they take the earlier nonblocking path.
Just flagging this, myb its okay to leave it as is

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants