Skip to content

Run the test suites against managed regtest backends instead of docker compose - #171

Merged
bennyhodl merged 4 commits into
masterfrom
managed-test-backends
Aug 4, 2026
Merged

Run the test suites against managed regtest backends instead of docker compose#171
bennyhodl merged 4 commits into
masterfrom
managed-test-backends

Conversation

@bennyhodl

Copy link
Copy Markdown
Owner

Run the test suites against managed regtest backends instead of docker compose

cargo test now needs nothing running beforehand. Each test binary starts and
tears down its own bitcoind, electrs (serving the esplora HTTP API), PostgreSQL
server, and nostr relay on ephemeral ports.

Before this, every integration test read its connection details out of the
environment (BITCOIND_HOST, ESPLORA_HOST, DATABASE_URL, ...) and expected
the docker-compose stack to already be up on fixed ports. CI reproduced that with
docker compose up -d plus a pair of polling scripts in front of every test job.

What replaces what

docker-compose service replacement
bitcoin + electrs bitcoind 0.41 + electrsd 0.41 (esplora_a33e97e1)
postgres postgresql_embedded 0.19
nostr-relay nostr-relay-builder 0.44 (in-process)

docker-compose.yaml stays: it is still how you run a ddk-node locally
(just deps). It is simply no longer involved in testing.

The new testenv crate

ddk-testenv is a workspace member with publish = false, pulled in by ddk
and ddk-manager as a path-only dev-dependency. Cargo strips such dependencies
when packaging, so releases are unaffected — verified with cargo package.

It offers two ways to get backends, and the distinction matters:

  • ddk_testenv::env() — one bitcoind/electrs pair per test binary, shared by
    every test in it. Cheap, and fine for tests that just need somewhere to put
    coins.
  • TestEnv::new() — a private pair, torn down when dropped. Tests that assert on
    contract state as blocks advance need this. libtest runs tests concurrently,
    and on a shared chain the blocks a sibling test mines can push a contract past
    a locktime before the assertion runs. The manager execution and splice tests
    use it, which is why init_clients and create_and_fund_wallet gained an
    &TestEnv parameter.

Mining helpers now block on the indexer (ElectrsD::wait_height) rather than
sleeping. That replaces the sleep(Duration::from_secs(5)) polling loops the old
helpers used, and is where most of the wall-clock saving comes from.

Details worth reviewing

Process cleanup. libtest exits via std::process::exit, which never runs
destructors for statics, so the shared environment's child processes would
outlive the test binary as orphans. env() registers a libc::atexit hook;
per-test environments clean up through Drop. A full 50-test run leaves zero
stray processes.

RPC timeouts. Several tests mine 101 blocks in a single generatetoaddress,
and a dozen of them run at once, so requests sit behind a lot of work. The test
RPC clients are built by hand with a 300s transport timeout instead of jsonrpc's
15s default, and the node runs with -rpcthreads=32 -rpcworkqueue=1024.

NB_CONFIRMATIONS=6 is load-bearing. The cooperative_close tests fail at
the default of 3. This is pre-existing — verified against the current master
code pointed at an external node — and CI already sets it. It is now documented
in the README so local runs match.

CI cache. The matrix jobs now restore target/debug/build alongside
target/debug/deps, because the bitcoind and electrs executables live in the
build scripts' OUT_DIR and the test binaries resolve them through a path baked
in at compile time.

Known constraints

  • The macOS electrs binary published by electrsd is x86_64, so Apple Silicon
    needs Rosetta. Linux CI is unaffected.
  • postgresql_embedded is pinned to 0.19 because 0.20+ raise the MSRV to 1.94,
    above this workspace's toolchain. Bumping it later means bumping the toolchain.
  • Postgres binaries are downloaded on first use and cached under
    ~/.theseus/postgresql; CI caches that directory.

Verification

Run with docker down and no connection environment variables set:

suite result
cargo test --all-features 259 passed, 0 failed
manager_execution_tests --ignored 50/50
stateless_execution --ignored 29/29
splice_execution_tests --ignored 1/1
orphaned processes afterward 0

cargo clippy -- -D warnings and cargo fmt --check are clean.

Follow-up

The stateless-dlc-creation branch adds ddk/tests/stateless_utils.rs, which
still reads ESPLORA_HOST / BITCOIND_*. It needs the same treatment when the
two branches meet — ChainContext::new should take its esplora host and RPC
client from ddk_testenv::env(). Its 29 tests pass on the shared environment, so
no per-test isolation is needed there.

Introduces a workspace-internal crate that starts and tears down the
services the test suites need, on ephemeral ports, so `cargo test` no
longer depends on a docker-compose stack listening on fixed ports.

`env()` returns one bitcoind/electrs pair per test binary, shared by
every test in it. `TestEnv::new()` returns a private pair for tests that
assert on contract state as blocks advance, where blocks a concurrently
running sibling test mines would otherwise push a contract past a
locktime early.

Feature-gated helpers cover the remaining two services: an in-process
nostr relay and an embedded PostgreSQL server.

Two details the implementation has to work around:

- libtest exits via `std::process::exit`, which never runs destructors
  for statics, so the shared environment registers a `libc::atexit` hook
  to avoid leaving its child processes behind as orphans.
- Several tests mine 101 blocks in a single `generatetoaddress` while a
  dozen of them run at once, so the RPC clients are built by hand with a
  300s transport timeout rather than jsonrpc's 15s default.

The crate is `publish = false` and consumed as a path-only
dev-dependency, which cargo strips when packaging the published crates.
Replaces the `ESPLORA_HOST` / `BITCOIND_*` / `DATABASE_URL` lookups and
the hardcoded `127.0.0.1:30000` and `ws://127.0.0.1:8081` endpoints with
backends the tests start themselves.

`generate_blocks` now blocks until electrs has indexed the new tip
instead of sleeping five seconds a poll, so the wallet is never asked to
sync against an esplora that is behind bitcoind.

The nostr and postgres tests take a relay and a server they own for the
duration of the test, since both shut down when dropped.
Moves these suites off the environment variables and onto managed
backends, and gives each test its own bitcoind and electrs rather than
sharing one across the binary.

The execution and splice tests assert on contract state as blocks
advance. libtest runs tests concurrently, so on a shared chain the
blocks one test mines are visible to the others and can drive a contract
past its locktime before the assertion runs. `init_clients` and
`create_and_fund_wallet` therefore take the environment the caller owns.

The two rejection tests in `manager_tests` never mine and stay on the
shared environment.
Every test job started the compose stack and polled two shell scripts
waiting for bitcoind and electrs to come up. The suites now manage those
services themselves, so the jobs run `cargo test` directly and the
connection environment variables go away. `NB_CONFIRMATIONS` stays: the
cooperative close tests depend on it.

The matrix jobs restore `target/debug/build` alongside
`target/debug/deps`, because the bitcoind and electrs executables live in
the build scripts' `OUT_DIR` and the test binaries resolve them through a
path baked in at compile time. A second cache keeps the embedded
PostgreSQL binaries between runs.

`docker-compose.yaml` stays for running a node locally via `just deps`,
with a note that tests no longer use it.
@bennyhodl
bennyhodl merged commit 77854d3 into master Aug 4, 2026
59 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant