Run the test suites against managed regtest backends instead of docker compose - #171
Merged
Conversation
Introduces a workspace-internal crate that starts and tears down the services the test suites need, on ephemeral ports, so `cargo test` no longer depends on a docker-compose stack listening on fixed ports. `env()` returns one bitcoind/electrs pair per test binary, shared by every test in it. `TestEnv::new()` returns a private pair for tests that assert on contract state as blocks advance, where blocks a concurrently running sibling test mines would otherwise push a contract past a locktime early. Feature-gated helpers cover the remaining two services: an in-process nostr relay and an embedded PostgreSQL server. Two details the implementation has to work around: - libtest exits via `std::process::exit`, which never runs destructors for statics, so the shared environment registers a `libc::atexit` hook to avoid leaving its child processes behind as orphans. - Several tests mine 101 blocks in a single `generatetoaddress` while a dozen of them run at once, so the RPC clients are built by hand with a 300s transport timeout rather than jsonrpc's 15s default. The crate is `publish = false` and consumed as a path-only dev-dependency, which cargo strips when packaging the published crates.
Replaces the `ESPLORA_HOST` / `BITCOIND_*` / `DATABASE_URL` lookups and the hardcoded `127.0.0.1:30000` and `ws://127.0.0.1:8081` endpoints with backends the tests start themselves. `generate_blocks` now blocks until electrs has indexed the new tip instead of sleeping five seconds a poll, so the wallet is never asked to sync against an esplora that is behind bitcoind. The nostr and postgres tests take a relay and a server they own for the duration of the test, since both shut down when dropped.
Moves these suites off the environment variables and onto managed backends, and gives each test its own bitcoind and electrs rather than sharing one across the binary. The execution and splice tests assert on contract state as blocks advance. libtest runs tests concurrently, so on a shared chain the blocks one test mines are visible to the others and can drive a contract past its locktime before the assertion runs. `init_clients` and `create_and_fund_wallet` therefore take the environment the caller owns. The two rejection tests in `manager_tests` never mine and stay on the shared environment.
Every test job started the compose stack and polled two shell scripts waiting for bitcoind and electrs to come up. The suites now manage those services themselves, so the jobs run `cargo test` directly and the connection environment variables go away. `NB_CONFIRMATIONS` stays: the cooperative close tests depend on it. The matrix jobs restore `target/debug/build` alongside `target/debug/deps`, because the bitcoind and electrs executables live in the build scripts' `OUT_DIR` and the test binaries resolve them through a path baked in at compile time. A second cache keeps the embedded PostgreSQL binaries between runs. `docker-compose.yaml` stays for running a node locally via `just deps`, with a note that tests no longer use it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Run the test suites against managed regtest backends instead of docker compose
cargo testnow needs nothing running beforehand. Each test binary starts andtears down its own bitcoind, electrs (serving the esplora HTTP API), PostgreSQL
server, and nostr relay on ephemeral ports.
Before this, every integration test read its connection details out of the
environment (
BITCOIND_HOST,ESPLORA_HOST,DATABASE_URL, ...) and expectedthe docker-compose stack to already be up on fixed ports. CI reproduced that with
docker compose up -dplus a pair of polling scripts in front of every test job.What replaces what
bitcoin+electrsbitcoind0.41 +electrsd0.41 (esplora_a33e97e1)postgrespostgresql_embedded0.19nostr-relaynostr-relay-builder0.44 (in-process)docker-compose.yamlstays: it is still how you run addk-nodelocally(
just deps). It is simply no longer involved in testing.The new
testenvcrateddk-testenvis a workspace member withpublish = false, pulled in byddkand
ddk-manageras a path-only dev-dependency. Cargo strips such dependencieswhen packaging, so releases are unaffected — verified with
cargo package.It offers two ways to get backends, and the distinction matters:
ddk_testenv::env()— one bitcoind/electrs pair per test binary, shared byevery test in it. Cheap, and fine for tests that just need somewhere to put
coins.
TestEnv::new()— a private pair, torn down when dropped. Tests that assert oncontract state as blocks advance need this. libtest runs tests concurrently,
and on a shared chain the blocks a sibling test mines can push a contract past
a locktime before the assertion runs. The manager execution and splice tests
use it, which is why
init_clientsandcreate_and_fund_walletgained an&TestEnvparameter.Mining helpers now block on the indexer (
ElectrsD::wait_height) rather thansleeping. That replaces the
sleep(Duration::from_secs(5))polling loops the oldhelpers used, and is where most of the wall-clock saving comes from.
Details worth reviewing
Process cleanup. libtest exits via
std::process::exit, which never runsdestructors for statics, so the shared environment's child processes would
outlive the test binary as orphans.
env()registers alibc::atexithook;per-test environments clean up through
Drop. A full 50-test run leaves zerostray processes.
RPC timeouts. Several tests mine 101 blocks in a single
generatetoaddress,and a dozen of them run at once, so requests sit behind a lot of work. The test
RPC clients are built by hand with a 300s transport timeout instead of jsonrpc's
15s default, and the node runs with
-rpcthreads=32 -rpcworkqueue=1024.NB_CONFIRMATIONS=6is load-bearing. Thecooperative_closetests fail atthe default of 3. This is pre-existing — verified against the current
mastercode pointed at an external node — and CI already sets it. It is now documented
in the README so local runs match.
CI cache. The matrix jobs now restore
target/debug/buildalongsidetarget/debug/deps, because the bitcoind and electrs executables live in thebuild scripts'
OUT_DIRand the test binaries resolve them through a path bakedin at compile time.
Known constraints
electrsdis x86_64, so Apple Siliconneeds Rosetta. Linux CI is unaffected.
postgresql_embeddedis pinned to 0.19 because 0.20+ raise the MSRV to 1.94,above this workspace's toolchain. Bumping it later means bumping the toolchain.
~/.theseus/postgresql; CI caches that directory.Verification
Run with docker down and no connection environment variables set:
cargo test --all-featuresmanager_execution_tests --ignoredstateless_execution --ignoredsplice_execution_tests --ignoredcargo clippy -- -D warningsandcargo fmt --checkare clean.Follow-up
The
stateless-dlc-creationbranch addsddk/tests/stateless_utils.rs, whichstill reads
ESPLORA_HOST/BITCOIND_*. It needs the same treatment when thetwo branches meet —
ChainContext::newshould take its esplora host and RPCclient from
ddk_testenv::env(). Its 29 tests pass on the shared environment, sono per-test isolation is needed there.