fix: surface transfer errors when caching blobs (COMP-2142)

[stefanoboriero]

A blob transfer job could report success while uploading a 0-byte object to the blob cache, producing a cache entry marked COMPLETED that holds no bytes. The failure only surfaced later, client-side, as a digest mismatch on pull - and because the bad entry is cached, it is sticky: every subsequent pull of that layer failed the same way.

The download is not performed by Wave but by a separate Kubernetes Job with its own pod, DNS config and node selector. The upstream registry is therefore contacted twice, by two processes in two network contexts: once by Wave's own probe, once by the transfer pod. Those two requests can always disagree - a DNS blip, a transient partition, an auth token expiring in between, a registry 5xx on the second request - and nothing on the return path checked that the bytes actually arrived.

Four layers each independently swallowed the failure; close all four:

1. ProxyClient.curl() emitted only '-s -X GET', so an HTTP error was written to stdout with a zero exit status. Add '-f' so HTTP errors become non-zero exits, plus '--retry' and '--connect-timeout' (sourced from the existing HttpClientConfig) so transient failures are retried. Note '--connect-timeout' bounds connection setup only: an upstream that connects and then stalls mid-body is still capped by the job timeout, not by this option. '--retry-all-errors' is deliberately omitted - it would retry a response whose body was already partially piped to s5cmd, duplicating those bytes in the uploaded object. '-L' is likewise not added: a redirect is answered by Wave's own probe in RegistryProxyController.handleDelegate0() before the blob cache branch is reached, so following redirects in the transfer job does not fix this bug and would change behaviour for registries that legitimately redirect.

2. transferCommand() built 'sh -c "curl ... | s5cmd ..."'. A shell pipeline exits with the status of its last command, so curl's failure was discarded and s5cmd's zero won. Add 'set -o pipefail', and run it under 'bash' rather than 'sh': in the s5cmd image (public.cr.seqera.io/wave/s5cmd:v2.3.0, Ubuntu 24.04) /bin/sh is dash, which does not implement pipefail and treats the unknown option as a fatal script error, so the pipeline would never run at all:

   $ docker run --rm --entrypoint sh .../s5cmd:v2.3.0
   -c 'set -o pipefail; echo DATA | cat'
   sh: 1: set: Illegal option -o pipefail
   exit=2

   Under bash the success path exits 0 and passes the data through, and an HTTP error propagates (exit 22). bash is present at /usr/bin/bash in that image; note this couples the command to an image providing it, while wave.blob-cache.s5cmd-image remains configurable.

3. onJobCompletion() trusted state.succeeded(), i.e. exitCode == 0, and nothing else. Validate the uploaded object before marking the entry COMPLETED: compare its size against the upstream Content-Length already captured in BlobEntry.contentLength, and mark the entry ERRORED on mismatch. This catches empty and truncated transfers regardless of cause, including causes not yet identified. When the upstream omits Content-Length the comparison is not possible, so the check falls back to rejecting only empty objects.

4. Marking the entry ERRORED still left the invalid object in the bucket, so the next request found it via blobExists() and served it as a valid CACHED entry - the bad transfer stayed sticky through a different path. Delete the object on any error path, not just on validation failure, since a failing pipeline can also leave a truncated object behind. S3 DeleteObject is idempotent and returns 204 for a missing key, so the common "nothing was uploaded" case is a silent no-op; the catch only trips on real problems (missing bucket, credentials, network), which are logged and swallowed as the entry is errored regardless.

blobExists() is refactored into a thin wrapper over a new blobSize(), which returns the stored object length or null. It also catches generic exceptions, so a failure inside the new validation degrades to "treat as not uploaded" rather than propagating out of job completion.

Note this prevents new 0-byte entries and cleans up the ones this code path creates, but does not remediate objects already in the bucket: blobExists() still reports true for an already-cached empty object, so buckets already affected need those objects deleted.