Add Async OHTTP client

[arminsabouri]

OHTTP lets a client send encrypted requests through a relay so the server can’t see who sent them and the relay can’t see what they contain. The following commit adds optional configurations to enable clients to proxy their requests through an OHTTP relay and gateway.

OHTTP functionality is feature flagged off behind async-ohttp. If a client provided OHTTP config it will attempt to use the relay instead of the target resource directly.

---

This work was first started by @alexlwn123
Tested against a blockstream's electrs and running an OHTTP relay and OHTTP Gateway locally. Automated tests are still TODO.

Seeking concept Ack/Nack

[luisschwab]

Concept ACK, this is a cool ideia. We are, however, in the process of completely dropping reqwest/minreq in favor of bitreq. Can you change the PR to use that?

[arminsabouri]

Concept ACK, this is a cool ideia. We are, however, in the process of completely dropping reqwest/minreq in favor of bitreq. Can you change the PR to use that?

Sure. Looks like the bitreq migration is a WIP #137. I may wait till thats in good shape before rebasing my commits on top of that.

[pseudozach]

I’ve also got a branch that ports this to the current bitreq client and gets the CI passing:
https://github.com/pseudozach/rust-esplora-client/tree/ohttp-bitreq

I also ended up building a compatible Bitcoin-HPKE OHTTP gateway (obitcoin) so I could actually do end-to-end testing with the client.

If any of that is useful, I’m happy to fold the gateway-backed smoke test, CI workflow, or related pieces into this PR, or you can cherry-pick whatever is helpful.

[arminsabouri]

I’ve also got a branch that ports this to the current bitreq client and gets the CI passing: https://github.com/pseudozach/rust-esplora-client/tree/ohttp-bitreq

This was useful ref for marking certain deps as duplicates. Thank you.

I also ended up building a compatible Bitcoin-HPKE OHTTP gateway (obitcoin) so I could actually do end-to-end testing with the client.
If any of that is useful, I’m happy to fold the gateway-backed smoke test, CI workflow, or related pieces into this PR, or you can cherry-pick whatever is helpful.

I suppose it could reduce the test boilerplate in this PR. i.e removing my dummy gateway. Do you already have a int. . e2e test that uses your gateway?

[pseudozach]

I’ve also got a branch that ports this to the current bitreq client and gets the CI passing: https://github.com/pseudozach/rust-esplora-client/tree/ohttp-bitreq

This was useful ref for marking certain deps as duplicates. Thank you.

happy to hear that!

I suppose it could reduce the test boilerplate in this PR. i.e removing my dummy gateway. Do you already have a int. . e2e test that uses your gateway?

e2e test is available here: https://github.com/pseudozach/rust-esplora-client/blob/ohttp-bitreq/tests/ohttp_smoke.rs
and runs here: https://github.com/pseudozach/rust-esplora-client/blob/ohttp-bitreq/.github/workflows/ohttp-smoke.yml

[arminsabouri]

Cleaned up the commits. Couple things are bothering me.

1. There is alot of boiler plate that could and should be abstracted away. things like bhttp payload construction. Assuming ohttp gets integrated into other clients in this space we could deduplicate this code. I already started putting together a ohttp-client crate. I'll share it later. Probably not a blocker for this PR -- something that could be refactored in a follow up. Maintainers please lmk if you disagree.
2. just audit fails on the second commit bc of the ohttp-relay dep. I could go the route @pseudozach went with (above) however, I would want the test harness to create and clean up the relay/gateway and electrum server instead of CI doing all of that.

The other thing worth revisiting is depending on ohttp vs bitcoin-ohttp. I dont see a compeling reason to use the secp256k1 curve for hpke. I think the main point that justified bitcoin-ohttp was clients already are using the same crypto primitives (secp256k1, sha256, polychaha). The tradeoff is clients that use the bitcoin-ohttp dep cannot use any of the existing ohttp libraries and/or infra. And this becomes a maintaince headache -- esp as new features comes up such as, PQ support and ohttp-streaming

[DanGould]

the main point that justified bitcoin-ohttp was clients already are using the same crypto primitives

Yes the same primitives AND the same exact dependencies. So no RustCrypto secp256k1, just the same exact deps rust-bitcoin already has everywhere. This does make it incompatible with existing ohttp libraries unless those were PR'd. Even if there was support for other crypto in this client.

Regarding infra, I'm not sure what running OHTTP Infra if any is open for the public to use. My thinking was to use the payjoin infra to bootstrap, along with a new allowed_purposes string based on the Directory Interactions spec (https://github.com/bitcoin/bips/blob/master/bip-0077.md#relaydirectory-interactions).

I have not though through post-quantum support or ohttp-streaming story yet because they have not (until now) been demanded by integrators. Lack of demand is a big reason this has been sidelined in the first place. Cake and Bull Bitcoin mobile already sync over Tor by default, so getting OHTTP esplora sync isn't a huge win. It's a bigger win for Cove, though.

[arminsabouri]

Yes the same primitives AND the same exact dependencies. So no RustCrypto secp256k1, just the same exact deps rust-bitcoin already has everywhere. This does make it incompatible with existing ohttp libraries unless those were PR'd. Even if there was support for other crypto in this client.

Why can't AEAD and hash primitives use rust-bitcoin deps? IIUC bitcoin-hpke depends directly on RustCrypto's (or some other crate's) chacha20poly1305, sha2, hkdf, and hmac.

The tradeoff here to consider: is it worth the maintance overhead of maintaining both bitcoin-hpke and bitcoin-ohttp vs clients will need to use new crypto primitives (namely X25119) to use ohttp. And does the latter then exclude some subset of clients from using ohttp for electrum calls (and why? I understand bitcoin-core wallet does not want to depend on X25119 but who else has this constraint?).

Separately: bitcoin-ohttp's rust-hpke feature currently bundles a legacy RustCrypto-based backend as dead code alongside the bitcoin-hpke backend actually in use. That's the source of the duplicate-crate entries in this diff: https://github.com/bitcoindevkit/rust-esplora-client/pull/145/changes#diff-2e9d962a08321605940b5a657135052fbcef87b5e360662bb527c96d9a615542R83-R97.

Regarding infra, I'm not sure what running OHTTP Infra if any is open for the public to use. My thinking was to use the payjoin infra to bootstrap, along with a new allowed_purposes string based on the Directory Interactions spec (https://github.com/bitcoin/bips/blob/master/bip-0077.md#relaydirectory-interactions).

Can the directory proxy requersts to electrum servers today? This may be problematic if the payjoin users are also using the same directory for the electrum gateway.

[DanGould]

Why can't AEAD and hash primitives use rust-bitcoin deps?

They can and should. chacha20poly1305 crate was a dependency because the rust-bitcoin https://crates.io/crates/chacha20-poly1305 was not yet released. The others were just not demanded until now and it is another hygiene item to do.

clients will need to use new crypto primitives (namely [X25519]) to use ohttp

Where is this demand coming from? My inclination would be to wait until p2p adopts a post quantum scheme and then use that (edit: assuming you're asking about because you want X25519+MLKEM768? ).

I understand bitcoin-core wallet does not want to depend on X25119 but who else has this constraint?

Not sure yet. The idea of using the lowest common denominator primitives to remove the entire class of questions / concerns from implementers.

bitcoin-ohttp's rust-hpke feature currently bundles a legacy RustCrypto-based backend as dead code

whoops! siccing bots on it overnight. Been a while since either crate has gotten any love

Can the directory proxy requersts to electrum servers today?

No, directories don't proxy requests, OHTTP Relays do, and Relays cannot proxy requests to esplora servers until they adopt an allowed_purposes scheme, but the payjoin-mailroom is willing & ready to accept that change and promote it to infrastructure operators in the next update.

edit: Alternatively, an esplora server could, (though perhaps shouldn't) be hosted behind an OHTTP Gateway with the payjoin directory BIP 77 specified allowed_purpose to opt in to receive requests from any relay willing to forward requests to payjoin directories.

Electrum is off the table because it's direct streaming over TCP socket, not HTTP requests like esplora which can be encapsulated in OHTTP afaiu.

This may be problematic if the payjoin users are also using the same directory for the electrum gateway.

What problem do you foresee specifically? The relay should see padded-length encrypted payloads (perhaps 1kb) and forward them to esplora servers who have opted in with the allowed_purposes advertisement. The directory should not be involved, and if a mailroom serves as both directory and OHTTP Relay, assuming directory requests reach them from OHTTP they won't know the source origin IPs that may also be interested in making esplora requests.