Skip to content

build(integration): pin YuanRong 237 and checkpoint sandboxd - #36

Closed
Chamberlain1998 wants to merge 50 commits into
inclusionAI:mainfrom
Chamberlain1998:codex/yuanrong-build-237-sandboxd-checkpoint
Closed

build(integration): pin YuanRong 237 and checkpoint sandboxd#36
Chamberlain1998 wants to merge 50 commits into
inclusionAI:mainfrom
Chamberlain1998:codex/yuanrong-build-237-sandboxd-checkpoint

Conversation

@Chamberlain1998

@Chamberlain1998 Chamberlain1998 commented Aug 22, 2026

Copy link
Copy Markdown

Current status

This is an integration-ready branch for joint debugging of the current
openYuanRong package and the native gVisor checkpoint implementation. It is
intentionally opened as a draft: the branch is buildable and has completed a
full image build, but the sandboxd checkpoint work still lives in a personal
fork and is not yet a release dependency.

The branch includes the YuanRong downloader and universal image pipeline work
accumulated on codex/yuanrong-downloaders, plus the exact artifacts and
component revisions validated below.

Pinned inputs

  • AKernel revision: 809b909c118a2cb707e25a1fa7f1cf10cf2f7277
  • sandboxd repository: https://github.com/Chamberlain1998/sandboxd.git
  • sandboxd branch: codex/native-gvisor-checkpoint-v1
  • sandboxd gitlink: 55971e14f533a79ac84d9e23ad455ca1e9d56c47
  • distill-fs gitlink: 5f3d6d3979a4ed76d4bb6722d7c1ce6db7126634
  • YuanRong source revision: 87cba622b491f7303415d179f3fa2fdae98e69e7
  • YuanRong Buildkite build: https://buildkite.com/openyuanrong/yuanrong-jcl/builds/237
  • core wheel: openyuanrong-core-0.7.0+87cba622b491
  • core wheel SHA256: 9eb44e1ea59153ab9a65a81fc32450c09376e835732290046d028cec2db3b200
  • RRT wheel SHA256: 3aff1b4a676ca28992a2478adab900bc7bd1e76928cc12016ae50fea412a68c4

The sandboxd gitlink was verified against the remote branch tip before opening
this PR. A recursive checkout therefore compiles the checkpoint implementation
from Chamberlain1998:codex/native-gvisor-checkpoint-v1, not the upstream
sandboxd default branch.

Integration use

The branch can be used directly for current joint debugging:

git clone --recurse-submodules \
  --branch codex/yuanrong-build-237-sandboxd-checkpoint \
  https://github.com/Chamberlain1998/AKernel.git
cd AKernel
make build

The checked-in downloader defaults select the checksum-pinned Buildkite #237
core and RRT artifacts. No private Buildkite token is required to retrieve the
published copies used by this branch.

Validation

  • YuanRong Buildkite #237 passed, including package, Build Image, and runtime
    image jobs.
  • make buildkite-check: 19 tests passed.
  • make deploy-script-check: passed.
  • Full x86_64 image build passed on ECS 47.110.253.153 using a clean recursive
    checkout of this branch.
  • Built image: akernel-e2e:build-237-809b909
  • Image ID: sha256:15260269f5b9d06d31b3ee80c9fa03950cc3deca75844f640dd14e4f2c6c6a28
  • OCI revision label: 809b909c118a2cb707e25a1fa7f1cf10cf2f7277
  • Verified executable in the image: sandboxd, sbox, sandbox-logger,
    distill_fs, runsc, and the Kata runtime-rs shim.
  • The wheel checksum passed during the image build and the build log records
    Successfully installed openyuanrong-core-0.7.0+87cba622b491.

Before release/merge

  • Move the required sandboxd checkpoint commits to the agreed long-lived
    repository/branch and update the gitlink accordingly.
  • Replace the integration artifact selection with an accepted YuanRong release
    or promoted build.
  • Run the deployment-level pause/resume and sandbox lifecycle suites in the
    target environment.

ChamberlainJI added 30 commits August 14, 2026 12:09
ChamberlainJI
docs(design): define standalone RRT pause/resume validation
2faf2bd 
Document the checksum-pinned Buildkite artifact set, conditional RRT image path, process-mode compatibility wiring, and Linux x86 end-to-end acceptance criteria. This keeps the default standalone behavior disabled while making the full SDK-to-runsc continuity proof reproducible.

Signed-off-by: ChamberlainJI <jichenglin1@huawei.com>
ChamberlainJI
docs(design): clarify native x86 validation boundary
59e36e7 
Record that Buildkite build 215 only produced amd64 RRT and runtime artifacts. Limit the arm64 macOS host to source and artifact inspection, while reserving image construction and runsc end-to-end validation for native Linux x86_64.

Signed-off-by: ChamberlainJI <jichenglin1@huawei.com>
ChamberlainJI
docs(plan): detail standalone pause/resume execution
c14949c 
Break the approved design into test-first build, compatibility wiring, sandboxd integration, standalone runtime, and native x86 verification tasks. The plan fixes artifact identities and keeps generated evidence and credentials outside Git.

Signed-off-by: ChamberlainJI <jichenglin1@huawei.com>
ChamberlainJI
build(rrt): add conditional sandbox runtime image
16fcbed 
Accept a checksum-pinned amd64 RRT wheel as an optional build input, install its native runtime into the EROFS rootfs, and advertise the RRT service only in capable images. Existing builds retain the Python-only services configuration.

Signed-off-by: ChamberlainJI <jichenglin1@huawei.com>
ChamberlainJI
build(yuanrong): wire pause resume process flags
36b673a 
Apply a checksum- and context-gated compatibility patch to Buildkite 215 core layouts. The patch exposes the feature gate, DataSystem snapshot backend, and managed checkpoint root to merged and separate FunctionAgent compositions.

Signed-off-by: ChamberlainJI <jichenglin1@huawei.com>
ChamberlainJI
build(sandboxd): select checkpoint capable revision
a3ce13d 
Advance the sandboxd gitlink to the verified native gVisor checkpoint branch. The selected revision adds wire-compatible Checkpoint, Restore, and List RPCs with managed checkpoint artifacts and runsc restore support.

Signed-off-by: ChamberlainJI <jichenglin1@huawei.com>
ChamberlainJI
feat(standalone): enable RRT pause resume validation
dcc1fc3 
Gate pause/resume startup on an RRT-capable image and persist checkpoint artifacts in the standalone data directory. Add the missing sandbox gateway routes and an acceptance runner that verifies state, filesystem, and live-process continuity.

Signed-off-by: ChamberlainJI <jichenglin1@huawei.com>
ChamberlainJI
build: allow selecting the Python package index
500a1de 
Propagate PIP_INDEX_URL through the build helper and into both image builds, including the bootstrap stage that installs uv. This keeps x86 builds usable on hosts where files.pythonhosted.org is unreachable or too slow.

Signed-off-by: ChamberlainJI <jichenglin1@huawei.com>
ChamberlainJI
build: allow mirroring managed Python downloads
fa043b4 
Expose uv's managed-Python mirror through the image build helper so x86 builders with slow GitHub release access can use a regional proxy without changing the default source.

Signed-off-by: ChamberlainJI <jichenglin1@huawei.com>
ChamberlainJI
build: support runsc-only node images
052258e 
Keep Kata enabled by default while allowing non-KVM standalone validation to omit its two-gigabyte release payload. The resulting image has no Kata shim, so sandboxd continues to advertise only available runtimes.

Signed-off-by: ChamberlainJI <jichenglin1@huawei.com>
ChamberlainJI
build: retry managed Python installation
dcadc00 
Retry the grouped uv Python installation in place so transient mirror stream failures do not discard already installed interpreter versions during the runtime image build.

Signed-off-by: ChamberlainJI <jichenglin1@huawei.com>
ChamberlainJI
build: seed runtime venvs from the selected index
0583051 
Map the configured package index to uv's seed operation so pip and setuptools do not fall back to the slow default source after managed Python installation.

Signed-off-by: ChamberlainJI <jichenglin1@huawei.com>
ChamberlainJI
build: allow CPU-only images to skip NVIDIA tools
595e2bc 
Keep NVIDIA userspace support enabled by default, but add an explicit CPU-only build switch for environments where the NVIDIA package repository is unreachable. This lets runsc pause/resume validation proceed without claiming GPU capability and keeps the public image behavior unchanged.

Signed-off-by: ChamberlainJI <jichenglin1@huawei.com>
ChamberlainJI
fix(build): install patch for YuanRong process config
2318327 
The RRT image path applies the checksum-gated Buildkite #215 process-mode configuration patch during image assembly. Install the required patch utility in the node image so the fail-closed configuration step can run.

Signed-off-by: ChamberlainJI <jichenglin1@huawei.com>
ChamberlainJI
build(yuanrong): support pause resume package 217
6711fc4 
Accept the checksum-pinned core wheel from Buildkite 217 and apply a patch generated against its updated process configuration. Enable the FunctionAgent DataSystem client whenever standalone pause/resume selects the DataSystem snapshot backend.

Signed-off-by: ChamberlainJI <jichenglin1@huawei.com>
ChamberlainJI
docs(build): design replaceable YuanRong downloaders
50b8e6f 
Define stable core-wheel and RRT-runtime downloader contracts so private pipelines can replace artifact sourcing without patching public Dockerfiles.

Signed-off-by: ChamberlainJI <jichenglin1@huawei.com>
ChamberlainJI
docs(build): plan replaceable YuanRong downloaders
6db8d90 
Define the test-first implementation sequence for extracting core-wheel and RRT-runtime acquisition from the Dockerfiles.

Signed-off-by: ChamberlainJI <jichenglin1@huawei.com>
ChamberlainJI
build(yuanrong): add replaceable artifact downloaders
9080928 
Move core-wheel and RRT-runtime source selection behind stable build-context scripts so private pipelines can replace Release acquisition with OBS logic without patching Dockerfiles.

Keep the existing checksum-pinned defaults and URL/SHA overrides, while Docker stages retain artifact-count and ELF validation.

Signed-off-by: ChamberlainJI <jichenglin1@huawei.com>
ChamberlainJI
ci(buildkite): add universal AKernel image pipeline
d5830c5 
Resolve YuanRong release or passed Buildkite artifacts into one verified manifest, then build and push a digest-confirmed RRT image. Package the same image for standalone and Helm deployments while publishing the exact sandbox SDK and provenance manifests.

Signed-off-by: ChamberlainJI <jichenglin1@huawei.com>
ChamberlainJI
perf(buildkite): avoid recursive submodule checkout
b1fd97d 
Buildkite otherwise initializes the full YuanRong submodule tree for every job, which wastes network bandwidth before the image build starts. Disable automatic submodule checkout and let the build job initialize only the two node components it needs.

Signed-off-by: ChamberlainJI <jichenglin1@huawei.com>
ChamberlainJI
ci(buildkite): route GitHub through restricted egress
932e551 
Buildkite agents in Guiyang cannot reliably reach GitHub directly. Reuse the validated WireGuard and Squid environment hook in checkout and command containers while keeping Buildkite, private networks, and Huawei Cloud traffic on direct routes.

Signed-off-by: ChamberlainJI <jichenglin1@huawei.com>
ChamberlainJI
fix(buildkite): bypass proxy for artifact storage
04686fb 
Buildkite uploads artifacts through Amazon S3 rather than a buildkite.com hostname. Keep that control-plane storage path on direct egress so the restricted Squid allowlist remains limited to source and package downloads.

Signed-off-by: ChamberlainJI <jichenglin1@huawei.com>
ChamberlainJI
fix(buildkite): bypass proxy for artifact downloads
700d181 
Buildkite serves artifact downloads from buildkiteartifacts.com even when uploads target Amazon S3. Keep both control-plane storage hosts on direct egress so restricted Squid rules do not block inter-job artifacts.

Signed-off-by: ChamberlainJI <jichenglin1@huawei.com>
ChamberlainJI
fix(buildkite): bypass proxy for artifact host
d407753 
Buildkite's Go artifact downloader treats a leading-dot NO_PROXY entry as subdomain-only. Add the apex buildkiteartifacts.com host so inter-job artifact downloads stay on direct egress instead of being rejected by the restricted Squid allowlist.

Signed-off-by: ChamberlainJI <jichenglin1@huawei.com>
ChamberlainJI
fix(buildkite): install image build driver
0325eff 
The validated Alpine checkout image does not provide make, so the image job stopped before invoking Docker. Install make with the existing restricted-egress bootstrap toolset and cover the requirement in pipeline tests.

Signed-off-by: ChamberlainJI <jichenglin1@huawei.com>
ChamberlainJI
fix(buildkite): use Alpine bash path
cf2ef8c 
The Buildkite builder provides Bash at /bin/bash while the project Makefile defaults to /usr/bin/bash. Pass the portable container path explicitly from the image build wrapper so the real Docker build can start.

Signed-off-by: ChamberlainJI <jichenglin1@huawei.com>
ChamberlainJI
fix(build): forward proxy into Docker stages
660fe73 
Docker daemon proxy settings cover base-image pulls but not commands running inside Dockerfile stages. Forward Docker's predefined upper- and lower-case proxy build arguments to both runtime and node builds so restricted GitHub and package downloads use the verified egress path without persisting proxy values in image history.

Signed-off-by: ChamberlainJI <jichenglin1@huawei.com>
ChamberlainJI
docs(buildkite): design large dependency PVC cache
4b2b892 
Keep checksum-pinned archives in a narrow persistent cache while preserving upstream URLs and repository digests as the build authority. Define atomic publication, corruption handling, local-build fallback, and two-run acceptance for the Kata archive.

Signed-off-by: ChamberlainJI <jichenglin1@huawei.com>
ChamberlainJI
docs(buildkite): plan verified Kata PVC cache
b630c16 
Translate the approved cache boundary into TDD-sized tasks covering atomic downloads, BuildKit integration, PodSpec isolation, cold-fill and warm-hit acceptance.

Signed-off-by: ChamberlainJI <jichenglin1@huawei.com>
ChamberlainJI
build(cache): add verified archive cache primitive
89dbf0b 
Publish immutable downloads only after checksum verification so jobs can safely reuse a PVC without treating it as an authority. Cover corrupt hits, interrupted transfers, and concurrent publication.

Signed-off-by: ChamberlainJI <jichenglin1@huawei.com>
ChamberlainJI added 20 commits August 14, 2026 19:17
ChamberlainJI
build(kata): consume verified dependency cache
f103672 
Prefetch the checksum-pinned static archive into an optional cache and expose it read-only to BuildKit while preserving the upstream fallback. Keep Docker storage isolated per job.

Signed-off-by: ChamberlainJI <jichenglin1@huawei.com>
ChamberlainJI
ci(buildkite): persist verified Kata downloads
63dd137 
Mount a narrow dependency cache only in the image job so repeated builds reuse immutable archives without exposing the PVC to checkout or packaging. Keep Docker storage ephemeral per job.

Signed-off-by: ChamberlainJI <jichenglin1@huawei.com>
ChamberlainJI
fix(buildkite): select topology-aware local cache
c65b665 
The Guiyang agent cluster has no default StorageClass, which left the dependency cache claim pending. Bind it through the existing csi-local-topology class so Kubernetes co-locates the local volume with the image job.

Signed-off-by: ChamberlainJI <jichenglin1@huawei.com>
ChamberlainJI
fix(buildkite): back cache with static local volume
f0a6968 
The Guiyang amd64 builders do not advertise an Everest persistent local-volume pool. Use their existing build-cache mount through a retained static PV so the dependency cache can bind without adding cloud disks.

Signed-off-by: ChamberlainJI <jichenglin1@huawei.com>
ChamberlainJI
docs(buildkite): extend verified dependency cache design
0f25221 
Define pinned gVisor and OpenTelemetry artifacts so the existing PVC remains a rebuildable performance cache rather than a source of truth.

Signed-off-by: ChamberlainJI <jichenglin1@huawei.com>
ChamberlainJI
docs(buildkite): plan gVisor and OTel cache extension
957e754 
Define TDD steps and formal cold/warm acceptance for adding both pinned artifacts to the existing PVC.

Signed-off-by: ChamberlainJI <jichenglin1@huawei.com>
ChamberlainJI
build(cache): verify SHA-512 artifacts
180fc84 
Allow the shared atomic cache primitive to validate the official gVisor digest without weakening existing SHA-256 entries.

Signed-off-by: ChamberlainJI <jichenglin1@huawei.com>
ChamberlainJI
build(cache): reuse gVisor and OTel artifacts
2401dc1 
Prefetch checksum-pinned node dependencies into the existing PVC and revalidate them from a read-only BuildKit context.

Signed-off-by: ChamberlainJI <jichenglin1@huawei.com>
ChamberlainJI
docs(buildkite): document expanded artifact cache
67eca53 
Record the pinned gVisor and OpenTelemetry entries and the operational rules for safely reusing them.

Signed-off-by: ChamberlainJI <jichenglin1@huawei.com>
ChamberlainJI
build(sandboxd): pin native gVisor checkpoint refactor
3107ecc 
Use the private sandboxd branch commit for the image validation candidate. This keeps the rewritten deterministic restore path reproducible without changing the existing pipeline source branch.

Signed-off-by: ChamberlainJI <jichenglin1@huawei.com>
ChamberlainJI
ci(buildkite): move restricted egress to UDP 443
312454d 
Cross-region UDP 51820 replies no longer reached Guiyang build pods even though the server emitted them. Rewrite only the endpoint in the temporary client configuration and retain the single-source access control so checkout can reuse the encrypted WireGuard secret without exposing Squid.

Signed-off-by: ChamberlainJI <jichenglin1@huawei.com>
ChamberlainJI
ci(buildkite): scope YuanRong API token to resolver
eaf8446 
Buildkite artifact resolution needs read_builds access for cross-pipeline metadata. Inject the encrypted credential only into the resolver job so image and packaging jobs retain their narrower secret set.

Signed-off-by: ChamberlainJI <jichenglin1@huawei.com>
ChamberlainJI
feat(deploy): enable Kubernetes RRT pause resume
c899634 
Expose an opt-in Helm value and share validated pause/resume arguments between standalone and distributed node startup. Kubernetes keeps ordinary data-plane settings unchanged while using DataSystem for snapshot transfer, allowing cross-node resume with RRT images.

Signed-off-by: ChamberlainJI <jichenglin1@huawei.com>
ChamberlainJI
fix(deploy): expose the SandboxRouter endpoint
8abeb57 
Publish the frontend SandboxRouter listener separately from the authenticated control API so public sandbox ports no longer hit the JWT middleware on port 8888. Document the explicit gateway address and cover the Helm port contract.

Signed-off-by: ChamberlainJI <jichenglin1@huawei.com>
ChamberlainJI
feat(sdk): split public SandboxRouter endpoint
4ba89bb 
Route public forwarded-port URLs through an independent SandboxRouter address while preserving the existing frontend gateway for PTY, file, and tunnel traffic. Keep the legacy shared endpoint as the fallback for existing deployments.

Signed-off-by: ChamberlainJI <jichenglin1@huawei.com>
ChamberlainJI
fix(build): retry transient distill-fs fetches
bd3ff4e 
Retry the locked distill-fs build when the pinned nydus Git fetch is interrupted by the restricted egress path. Remove only incomplete nydus Cargo Git state between attempts so transport failures do not make otherwise deterministic image builds flaky.

Signed-off-by: ChamberlainJI <jichenglin1@huawei.com>
ChamberlainJI
chore(deps): update sandboxd checkpoint cleanup
e5d2be7 
Advance the sandboxd gitlink to remove the transient gVisor restore image
while retaining deterministic physical intent when cleanup remains uncertain.
This prevents restored sandboxes from holding checkpoint.img until delete.

Signed-off-by: ChamberlainJI <jichenglin1@huawei.com>
ChamberlainJI
feat(pause-resume): add durable snapshot deployment
f4ff0ff 
Wire RRT pause and resume through Kubernetes and standalone nodes with configurable durable snapshot storage, direct SandboxRouter access, and maintained functional, stress, and memory-state E2E coverage.

Signed-off-by: ChamberlainJI <jichenglin1@huawei.com>
ChamberlainJI
chore(deps): pin squashed sandboxd checkpoint support
5bc4d45 
Advance the node image gitlink to the single reviewed sandboxd pause and resume commit so clean builds use the same physical-state reconciliation validated by the feature branch.

Signed-off-by: ChamberlainJI <jichenglin1@huawei.com>
ChamberlainJI
build(yuanrong): pin build 237 artifacts
809b909 
Default source builds to the checksum-verified core and RRT wheels produced by YuanRong Buildkite build 237. Advance the sandboxd gitlink to the native gVisor checkpoint branch so clean clones compile the matching checkpoint implementation.

Signed-off-by: ChamberlainJI <jichenglin1@huawei.com>
@Chamberlain1998

Chamberlain1998 commented Aug 22, 2026

Copy link
Copy Markdown
Author

Superseded by #37, which keeps only the YuanRong #237 OBS wheel download/integration changes and the checkpoint sandboxd gitlink. The Buildkite pipeline and PVC/cache changes were intentionally dropped.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@Chamberlain1998