feat(tensorflow): add TF Serving 2.20 inference DLC on AL2023#6243
Draft
bhanutejagk wants to merge 11 commits into
Draft
feat(tensorflow): add TF Serving 2.20 inference DLC on AL2023#6243bhanutejagk wants to merge 11 commits into
bhanutejagk wants to merge 11 commits into
Conversation
First TF inference DLC on the v2 (main) branch. Mirrors master's TF 2.19 inference image but ports to AL2023 + CUDA 12.9.1 + Python 3.12, switches to uv-driven dependency management, and rebuilds nginx-mod-njs from source (no AL2023 RPM exists). SageMaker only, x86 only, single-model + MME support preserved. - Dockerfile.cuda + Dockerfile.cpu with builder-njs source-build stage - Ports SageMaker handler scripts (Falcon + nginx + njs + multi_model_utils) from master TF 2.19 build_artifacts/sagemaker/ byte-for-byte - GitHub Actions workflows mirror PR #6107 (TF 2.21 training) shape - SageMaker integration tests for single-model and MME endpoints - ECR scan allowlist overlay at tensorflow/tensorflow-2.20.json TFS 2.20.0 binary copied from tensorflow/serving:2.20.0-devel-gpu image. tensorflow-serving-api installed with --no-deps to avoid pulling 600 MB of TF framework. framework: "tensorflow" + job_type: "inference" matches the cross-team release-logic convention (training and inference share the framework slug, distinguished by job_type). Known follow-ups (not blocking this PR): - PR #6107 (TF 2.21 training) will add tensorflow/framework_allowlist.json (shared base) once it merges. CI security-test may fail until then. - Separate change against the release-logic config needed to add tensorflow entry to frameworks.yml before image can release to prod.
added 10 commits
June 15, 2026 11:21
Run pre-commit hooks (ruff-format, ruff-check, requirements-txt-fixer, trailing-whitespace) on Phase 1-6 files. Mechanical reformatting plus noqa annotations for intentional E402 imports after gevent monkey-patch in python_service.py and one F841 unused variable in serve.py (preserved as-is from master TF 2.19 vendored handler).
Re-indent RUN/ENV continuation lines from 4-space to 2-space per dockerfmt's normalization. Mechanical reformatting only — no logic changes. Closes the dockerfmt CI failure on PR #6243.
The builder-njs stage's `tar xzf` invocation needs gzip, which AL2023's base image does not include by default. CI build failed at the nginx source extraction step on PR #6243. Add `gzip xz` to the existing dnf install line in both Dockerfile.cuda and Dockerfile.cpu builder-njs stages.
nginx ./configure --with-compat with the njs dynamic module auto-enables the HTTP XSLT module, which requires libxml2 and libxslt development headers. AL2023 base lacks these. Adding to the existing dnf install line in both Dockerfile.cpu and Dockerfile.cuda builder-njs stages.
nginx + njs `make modules` with the http-only --add-dynamic-module produces ngx_http_js_module.so but not the stream variant. SageMaker HTTP-based model serving uses only the http njs module (tensorflowServing.js routes /invocations and /ping). Removing the unused cp in builder-njs, the orphan COPY in runtime-base, and the comment reference in both Dockerfile.cuda and Dockerfile.cpu.
Training PR's sagemaker-test job avoids the test/conftest.py fabric import not by installing fabric, but by having test/tensorflow/pytest.ini cap pytest rootdir at test/tensorflow/. Mirror that mechanism here. - Add test/tensorflow/pytest.ini ([pytest] stanza) - Match training's requirements.txt content exactly (boto3, pytest, sagemaker>=3.0.0) at test/tensorflow/integration/inference/ - Both inference workflows now install via -r requirements.txt
CVE-2025-23339 and CVE-2025-23308 in cuda-toolkit-config-common 12.9.79. Fix requires CUDA 13 major version bump which is tracked as a separate currency update; CUDA 12.9 is pinned to match the AL2023 base image and parallel TF 2.21 training PR.
ecr_scan.py constructs the overlay path as <framework>/<framework>-<framework_version>.json. With framework_version=2.20.0 (full semver) the lookup is tensorflow-2.20.0.json, not the short tensorflow-2.20.json we previously committed. Rename to match so the two CUDA toolkit CVEs are actually picked up.
The PyPI sagemaker package v3.x removed the top-level sagemaker.Session class along with sagemaker.tensorflow.serving.TensorFlowModel and sagemaker.multidatamodel.MultiDataModel. Our conftest used the legacy v2 API. Rewrite to use boto3 directly (sagemaker, sagemaker-runtime, s3 clients) so tests work with the >=3.0.0 pin and are SDK-version independent. boto3 maps 1:1 to the create_model / create_endpoint_config / create_endpoint / invoke_endpoint flow these tests already needed, which is simpler than adopting v3's heavier ModelBuilder abstraction for integration tests. The sagemaker dep is dropped from requirements.txt since it is no longer imported.
Replaces commit 4b99d0f which used boto3 directly. The SageMaker Python SDK v3 is the supported entrypoint for these tests; v3 removed the v2 sagemaker.Session, sagemaker.tensorflow.serving.TensorFlowModel, and sagemaker.multidatamodel.MultiDataModel classes the old fixtures relied on. For DLC integration tests we already supply image_uri and a pre-built model.tar.gz, so ModelBuilder's auto-detection adds no value. We use the v3 sagemaker-core resource layer directly (the same surface ModelBuilder calls underneath): - conftest sagemaker_session fixture uses sagemaker.core.helper.session_helper.Session (default_bucket / upload_data); cleanup_endpoint uses Endpoint.get(...).delete(), EndpointConfig.get(...).delete(), Model.get(...).delete() - single-model test uses Model.create(primary_container=ContainerDefinition(...)) + EndpointConfig.create([ProductionVariant(...)]) + Endpoint.create() with endpoint.wait_for_status("InService") and endpoint.invoke(...) - MME test expresses the multi-model contract directly: ContainerDefinition(mode="MultiModel", model_data_url=<S3 prefix>) and endpoint.invoke(target_model="modelN.tar.gz") - sagemaker>=3.0.0 retained in requirements.txt
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
main) branch — TensorFlow Serving 2.20.0, SageMaker only, x86 only, single-model + MME support preserved.docker/tensorflow/inference/2.20/tree withDockerfile.cudaandDockerfile.cpuon AL2023 + CUDA 12.9.1 + Python 3.12, uv-driven dependency management, and abuilder-njssource-build stage that compilesnginx-mod-njsagainst nginx 1.30.2 (no AL2023 RPM exists for njs 0.9.9).multi_model_utils) byte-for-byte from master TF 2.19'sbuild_artifacts/sagemaker/.tensorflow/tensorflow-2.20.json.Why this PR
This is the first TensorFlow inference image on
main(AL2023 substrate). The master branch's TF 2.19 inference image is on AL2 — porting to AL2023 surfaces a few new challenges (nonginx-mod-njsRPM in AL2023, Python 3.12 wheel availability fortensorflow-serving-api, CUDA 12.9.1 toolkit layout) that this PR addresses.Test plan
framework_allowlist.json)Related
tensorflow/framework_allowlist.jsonbase file is introduced there.Known limitations / follow-ups
tensorflow/framework_allowlist.jsonships with [TensorFlow][Training][Sagemaker] TensorFlow 2.21.0 Currency Release #6107. Until [TensorFlow][Training][Sagemaker] TensorFlow 2.21.0 Currency Release #6107 merges, thesecurity-testjob here may fail because it expects to read both the base file and this PR'stensorflow-2.20.jsonoverlay.tensorflowentry (withjob_type: inference) before this image can release to the prod ECR. That CR is not in scope for this PR.nginx-mod-njs0.9.9 + nginx 1.30.2 are source-built in abuilder-njsstage (no AL2023 RPM available). First real validation of that build is in CI; localdocker buildis not part of the pre-merge plan.