Vendor base package sources in a compressed .tar.zst

[DavisVaughan]

• Rebase on top of Rework Cache to evict when untouched for 4 months #1323 after it is merged first

This is probably one of the more technically impressive PRs I've ever done 😛

Previously in oak_source::get_r("4.5.0") we would download the 100mb R 4.5.0 tarball from CRAN and store the whole thing on disk in the cache so we could have access to base package source files. We would repeat this for any other R version the user uses.

We don't really use much of that downloaded tarball right now, just src/library/{package}/R/{files}.R to have the original source files for the base R package of interest.

@lionel- suggested that maybe we could vendor the base packages' R files, starting with the full files for R 4.2.0 (our last supported version of R) and shipping git diffs for all subsequent patch versions that could be applied on the fly. Claude and I have come up with something even better 😄.

Zstd magic

Instead we inline a single 1.5 MB base-sources.tar.zst with include_bytes!() into the ark binary that we decompress and untar on the fly. This contains the complete R/ sources for every base package for every patch release since R 4.2.0, all ~20 of them. This is an incredible technical achievement, as you are about to see.

• 1.5 MB base-sources.tar.zst
• 127.3 MB base-sources.tar decompressed (by ✨ magic ✨ )
• 142.7 MB base-sources/ folder fully untarred

So how the hell did we pack 142.7 MB into 1.5 MB? Good question!

You see, a single R version's worth of src/library/{package}/R/{files}.R files weigh in at about 8 MB. Tarred and compressed, this comes in at about 1.3 MB for just that one R version (very close to our 1.5 MB!).

Between R versions, the {files}.R don't actually change all that much! We take advantage of this by leveraging a zstd feature called the compression window. Zstd maintains an adjustable "lookback window" of bytes that it uses to look for repeating patterns in the blob you are compressing. If it finds patterns, it can highly compress them.

We leverage this by placing the same file across R versions adjacent to each other in the .tar. For example, rather than tarring in this naive order:

4.5.0/base/R/a.R
4.5.0/base/R/b.R
4.5.0/methods/R/a.R

4.6.0/base/R/a.R
4.6.0/base/R/b.R
4.6.0/methods/R/a.R

we instead sort the files like this before tarring:

4.5.0/base/R/a.R
4.6.0/base/R/a.R

4.5.0/base/R/b.R
4.6.0/base/R/b.R

4.5.0/methods/R/a.R
4.6.0/methods/R/a.R

This places 4.5.0/base/R/a.R right next to 4.6.0/base/R/a.R in the tar, and they are basically the same file between R versions, so when zstd sees this it basically removes all of the duplication in the compressed result.

This means that we can store a whopping 20 versions of R sources for nearly the same price as 1, and the size is small enough that we can just vendor it in to ark itself.

Binary size

What I don't understand is what cargo build is doing. With a debug build, you can "see" the difference in the binary size:

cargo build

95,214,024 (95.2 MB) # before
97,698,232 (97.6 MB) # after

cargo build --release

24,543,568 (24.5 MB) # before
24,526,064 (24.5 MB) # after

It got...smaller?? I have no idea how that could happen, but I tried this multiple times and got the same result? I even did a cargo clean between branch switches. I'm surely doing something wrong.

But it's definitely working!

Screen.Recording.2026-07-09.at.8.59.21.AM.mov

Updating

In terms of updating, we will need to update the bundled base-sources.tar.zst after every patch R release. All we have to do is:

• Update versions.txt with the new version
• Run just vendor-r-sources, which will re-download all the R sources from CRAN and recreate an updated base-sources.tar.zst that we'd commit

Some other things

• If a user has a newer R than we have sources for, like r-devel, then we fall back to our newest version of the R sources. This is probably pretty darn good, considering that they don't change much between versions. And I think it is better than doing nothing at all.
• This all still goes through our standard Cache, so once a particular R version's sources are unpacked from the compressed tarball and stored in the cache, they are never unpacked again. That said, those sources will be subject to the same "eviction" rules as any other Cache, so if you don't use an R version's sources for 4 months, it will be evicted (freeing up around 8 MB of space).

[DavisVaughan]

I've left this as get_cran() even though there are no other getters right now in case we can figure out how to do Bioconductor in the future, so we'd have get_bioc() too

[DavisVaughan]

I was going to setup the xtask idea for this but apparently you can just create an examples/ folder and put arbitrary scripts with a main entrypoint here? They run with all of your dev-dependencies, and you run it like cargo run -p oak_vendored --example vendor, which just vendor-r-sources does

[DavisVaughan]

I thought this was rather cool