Description: Each submit copy is pinned to the provider selected when the run starts: resume re-binds the same providerId, and there is no way to move a copy elsewhere. When a provider is unhealthy, that copy can never complete — observed live: a secondary's provider failed every AddPieces with its own node-RPC i/o timeout across many hours and retries, leaving its data set at 0 pieces while the primary copy finished completely. The only escape today is 'Discard previous submit', which forgets the whole run including the healthy copy's progress.
Impact: One sick provider permanently blocks a copy; redundancy (the point of multiple copies) is hostage to the initial selection.
Steps to Reproduce: Submit 2 copies; have the secondary's provider fail commits persistently; observe resume retries the same provider forever.
Expected Behavior: A per-copy 'switch provider' action on a failed context: pick (or let the SDK pick) a different provider — excluding ones already used by this run — and restart that copy as a fresh context with replanned chunks and a new create-and-add. Pieces already committed to the abandoned provider's data set stay on chain (note them; payment rails for a partial data set are the operator's to clean up). The healthy copies are untouched.
Additional Context: The chunked SavedSubmitContext (run-store v2) makes this tractable: a replacement is a new context entry with fresh chunks; nothing in the resume key changes. Check whether the SDK's provider selection (storage.createContexts) can exclude provider ids, or use an explicit provider picker (the SP registry list is an account-less read). Applies to the hosted console first; the local serve flow targets one data set explicitly and can already be pointed at a new one.
Description: Each submit copy is pinned to the provider selected when the run starts: resume re-binds the same
providerId, and there is no way to move a copy elsewhere. When a provider is unhealthy, that copy can never complete — observed live: a secondary's provider failed every AddPieces with its own node-RPC i/o timeout across many hours and retries, leaving its data set at 0 pieces while the primary copy finished completely. The only escape today is 'Discard previous submit', which forgets the whole run including the healthy copy's progress.Impact: One sick provider permanently blocks a copy; redundancy (the point of multiple copies) is hostage to the initial selection.
Steps to Reproduce: Submit 2 copies; have the secondary's provider fail commits persistently; observe resume retries the same provider forever.
Expected Behavior: A per-copy 'switch provider' action on a failed context: pick (or let the SDK pick) a different provider — excluding ones already used by this run — and restart that copy as a fresh context with replanned chunks and a new create-and-add. Pieces already committed to the abandoned provider's data set stay on chain (note them; payment rails for a partial data set are the operator's to clean up). The healthy copies are untouched.
Additional Context: The chunked
SavedSubmitContext(run-store v2) makes this tractable: a replacement is a new context entry with fresh chunks; nothing in the resume key changes. Check whether the SDK's provider selection (storage.createContexts) can exclude provider ids, or use an explicit provider picker (the SP registry list is an account-less read). Applies to the hosted console first; the local serve flow targets one data set explicitly and can already be pointed at a new one.