Skip to content

add a smoke test to ensure server images start correctly #1646

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
tmckayus wants to merge 1 commit into
base: main
Choose a base branch
from
+140 −0
Open

add a smoke test to ensure server images start correctly #1646

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 32 additions & 0 deletions .github/workflows/test_images.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -87,3 +87,35 @@ jobs:
- name: Test cuopt
run: |
bash ./ci/docker/test_image.sh

# Host-side startup smoke: exercises ENTRYPOINT/CMD (REST) and
# CUOPT_SERVER_TYPE=grpc. The jobs above run *inside* the image as a GHA
# container and never launch the servers, so they cannot catch packaging
# gaps such as UBI10's RHEL lib/lib64 NCCL path miss.
smoke:

@ramakrishnap-nv ramakrishnap-nv Aug 3, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can't we add the test script as part of test_image.sh ?

@tmckayus tmckayus Aug 3, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can't put this in test_image.sh as wired today — that job runs inside the image as a GHA container, so it never exercises ENTRYPOINT/CMD, and it already patches LD_LIBRARY_PATH for the nvidia wheels (which is what masked the UBI10 NCCL packaging bug). The smoke test has to be host-side docker run.

The bug we hit was invoking the ubi10 image with just a docker command, as a user would.

name: smoke-images (${{ inputs.ARCH }}, cuda${{ needs.prepare.outputs.CUDA_SHORT }})
runs-on: "linux-${{ inputs.ARCH }}-gpu-a100-latest-1"
needs: prepare
steps:
- name: Checkout code repo
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1
with:
fetch-depth: 0
ref: ${{ inputs.sha }}
persist-credentials: false
- name: Smoke Ubuntu image (REST + gRPC)
env:
IMAGE_TAG_PREFIX: ${{ inputs.IMAGE_TAG_PREFIX }}
CUDA_SHORT: ${{ needs.prepare.outputs.CUDA_SHORT }}
PYTHON_SHORT: ${{ needs.prepare.outputs.PYTHON_SHORT }}
run: |
bash ./ci/docker/smoke_image.sh \
"nvidia/cuopt:${IMAGE_TAG_PREFIX}-cuda${CUDA_SHORT}-py${PYTHON_SHORT}"
- name: Smoke UBI10 image (REST + gRPC)
if: ${{ startsWith(inputs.CUDA_VER, '13.') }}
env:
IMAGE_TAG_PREFIX: ${{ inputs.IMAGE_TAG_PREFIX }}
CUDA_SHORT: ${{ needs.prepare.outputs.CUDA_SHORT }}
run: |
bash ./ci/docker/smoke_image.sh \
"nvidia/cuopt:${IMAGE_TAG_PREFIX}-cuda${CUDA_SHORT}-ubi10"
14 changes: 14 additions & 0 deletions ci/docker/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,3 +26,17 @@ docker run -it --rm --gpus all -u root --volume $PWD:/repo -w /repo --entrypoint
# UBI10 image
docker run -it --rm --gpus all -u root --volume $PWD:/repo -w /repo --entrypoint "/bin/bash" nvidia/cuopt:[TAG]-ubi10 ./ci/docker/test_image.sh
```

### Startup smoke (REST + gRPC)

`test_image.sh` runs pytest inside the image and does not launch the servers.
To verify the published entrypoint starts both the default REST server and the
gRPC server (`CUOPT_SERVER_TYPE=grpc`):

```bash
./ci/docker/smoke_image.sh nvidia/cuopt:[TAG]
./ci/docker/smoke_image.sh nvidia/cuopt:[TAG]-ubi10
```

CI runs this for both variants after the multiarch manifests are published
(see `.github/workflows/test_images.yaml` job `smoke`).
Comment thread
tmckayus marked this conversation as resolved.
94 changes: 94 additions & 0 deletions ci/docker/smoke_image.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,94 @@
#!/usr/bin/env bash
# SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
#
# Smoke-test that a published cuOpt image starts its default REST server and
# the gRPC server via CUOPT_SERVER_TYPE=grpc. Runs on the host with docker so
# the real ENTRYPOINT/CMD path is exercised (unlike test_image.sh, which runs
# inside a GHA job container and never launches the servers).
#
# Usage (any published or locally built tag):
# ./ci/docker/smoke_image.sh nvidia/cuopt:[TAG]
# ./ci/docker/smoke_image.sh nvidia/cuopt:[TAG]-ubi10
#
# Env:
# SMOKE_TIMEOUT_SECS Max seconds to wait for listen (default: 90)
# SMOKE_GPU_ARGS Docker GPU flags (default: --gpus all)

set -euo pipefail

IMAGE="${1:?usage: $0 <image>}"
TIMEOUT_SECS="${SMOKE_TIMEOUT_SECS:-90}"
# shellcheck disable=SC2206
GPU_ARGS=(${SMOKE_GPU_ARGS:---gpus all})

pass() { printf 'PASS %s\n' "$*"; }
fail() { printf 'FAIL %s\n' "$*" >&2; exit 1; }
info() { printf 'INFO %s\n' "$*"; }

smoke_one() {
local label="$1"
local expect_re="$2"
shift 2
# Remaining args are extra docker run flags (e.g. -e CUOPT_SERVER_TYPE=grpc).

local name log cid i
name="cuopt-smoke-${label}-$$"
log="$(mktemp)"
cid=""

cleanup() {
if [[ -n "${cid}" ]]; then
docker rm -f "${cid}" >/dev/null 2>&1 || true
fi
rm -f "${log}"
}
trap cleanup RETURN

info "Starting ${label} server from ${IMAGE}"
# Do not use --rm: a fast crash (e.g. missing libnccl.so.2) would delete the
# container before we can collect logs.
cid="$(docker run -d --name "${name}" "${GPU_ARGS[@]}" "$@" "${IMAGE}")"

for ((i = 1; i <= TIMEOUT_SECS; i++)); do
docker logs "${cid}" >"${log}" 2>&1 || true

if grep -qiE 'error while loading shared libraries|libnccl\.so|FATAL FIPS SELFTEST|OpenSSL internal error' "${log}"; then
echo "----- ${label} logs -----"
cat "${log}"
fail "${label}: loader/crypto failure while starting"
fi

if grep -qE "${expect_re}" "${log}"; then
pass "${label}: matched /${expect_re}/"
return 0
fi

# Container exited before listen — dump logs and fail.
if ! docker inspect -f '{{.State.Running}}' "${cid}" 2>/dev/null | grep -qx true; then
echo "----- ${label} logs -----"
cat "${log}"
fail "${label}: container exited before becoming ready"
fi

sleep 1
done

echo "----- ${label} logs -----"
cat "${log}"
fail "${label}: timed out after ${TIMEOUT_SECS}s waiting for /${expect_re}/"
Comment on lines +40 to +79

@coderabbitai coderabbitai Bot Jul 31, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

Run cleanup when a smoke test fails.

trap cleanup RETURN runs only when smoke_one returns. The fail calls use exit 1, so Bash terminates before the function returns. A container that starts and then fails, and its temporary log, remain on the runner. The docker run command at Line 51 can cause the same result through set -e.

Return a nonzero status from smoke_one after each failure path, including a failed docker run, or use an EXIT cleanup design that safely retains the required container and log state.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@ci/docker/smoke_image.sh` around lines 40 - 79, Update smoke_one’s cleanup
flow so every failure path performs cleanup before terminating: handle a failed
docker run, replace fail/exit-based paths with nonzero returns followed by
cleanup, or use an EXIT trap that safely removes the container and temporary
log. Preserve log output and failure status for loader errors, early exits, and
timeouts.

}

info "Pulling ${IMAGE}"
if ! docker pull "${IMAGE}"; then
if docker image inspect "${IMAGE}" >/dev/null 2>&1; then
info "Pull failed; using local image ${IMAGE}"
else
fail "Pull failed and no local image named ${IMAGE}"
fi
fi

smoke_one rest 'Uvicorn running on'
smoke_one grpc 'Listening on' -e CUOPT_SERVER_TYPE=grpc

pass "Smoke OK for ${IMAGE}"
Loading