Skip to content

gazette: surface fragment store diagnostic on FRAGMENT_STORE_UNHEALTHY#3159

Merged
jgraettinger merged 1 commit into
masterfrom
johnny/gazette-fragment-store-errors-057e0d
Jul 17, 2026
Merged

gazette: surface fragment store diagnostic on FRAGMENT_STORE_UNHEALTHY#3159
jgraettinger merged 1 commit into
masterfrom
johnny/gazette-fragment-store-errors-057e0d

Conversation

@jgraettinger

Copy link
Copy Markdown
Member

Problem

The broker's AppendResponse and FragmentStoreHealthResponse both carry a store_health_error string — the actual diagnostic from the failing fragment store — populated when the status is FRAGMENT_STORE_UNHEALTHY. But the gazette crate collapsed every non-OK status into Error::BrokerStatus(status), which holds only the enum value. The response (and its store_health_error) was discarded, so upstack callers saw only the generic "unexpected broker status: FragmentStoreUnhealthy".

Fix

  • Add a dedicated Error::FragmentStoreUnhealthy(String) variant (Display: "fragment store is unhealthy: {0}"). A new variant, rather than a field on BrokerStatus, avoids churning the ~15 call sites that pattern-match BrokerStatus(broker::Status::X) as a tuple.
  • Classify it as non-transient in is_transient(), matching prior behavior (the status means the store has failed its health check for an extended period).
  • The two response types carrying store_health_error (append and fragment_store_health) now special-case FRAGMENT_STORE_UNHEALTHY and move that string into the new error before the response is dropped.

Bonus

This also fixes the storage_mappings.rs GraphQL health-check consumer. Its existing Ok(Err(err)) => Some(err.to_string()) arm previously produced the useless generic status string; it now surfaces the store's real diagnostic.

Retry semantics are unchanged: the append Stream still retries on its own loop regardless of is_transient().

cargo check passes for gazette and dependent crates (publisher, shuffle, dekaf, control-plane-api, migrate); cargo fmt applied.

The broker's AppendResponse and FragmentStoreHealthResponse both carry a
`store_health_error` string with the failing fragment store's own
diagnostic, populated when status is FRAGMENT_STORE_UNHEALTHY. The gazette
crate collapsed every non-OK status into Error::BrokerStatus(status),
which holds only the enum value, so the diagnostic was dropped and callers
saw only the generic "unexpected broker status: FragmentStoreUnhealthy".

Add a dedicated Error::FragmentStoreUnhealthy(String) variant and populate
it from `store_health_error` in the append and fragment_store_health
paths. A new variant (rather than a field on BrokerStatus) avoids churning
the many call sites that pattern-match BrokerStatus as a tuple.

This also fixes the storage-mappings GraphQL health check, whose existing
error arm now surfaces the store's real diagnostic instead of the generic
status string.
@jgraettinger
jgraettinger merged commit e5f8029 into master Jul 17, 2026
15 of 17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants