Skip to content

dtls13: add an opt-in stateful server and extend the NSS interop tests - #33

Open
Frauschi wants to merge 1 commit into
masterfrom
dtls_interop
Open

dtls13: add an opt-in stateful server and extend the NSS interop tests#33
Frauschi wants to merge 1 commit into
masterfrom
dtls_interop

Conversation

@Frauschi

Copy link
Copy Markdown
Owner

Motivation

NSS fragments its ML-KEM ClientHello and orders the large key_share ahead of the cookie extension, so the cookie does not land in the first fragment. A stateless wolfSSL DTLS 1.3 server drops such a ClientHello and the handshake stalls on HelloRetryRequest retransmits.

Measured against a locally built NSS 3.123.1, with a default wolfSSL DTLS 1.3 server:

NSS client group Result ClientHellos processed
mlkem768x25519 timeout 0
secp384r1mlkem1024 (fragmented first CH) timeout 0
HRR case (fragmented CH2 carrying the cookie) timeout 1, then stall

So wolfSSL-server / NSS-client DTLS 1.3 with a post-quantum group does not work today. The other direction (NSS server, wolfSSL client) already works, because our client sends an empty key share and gets an HRR.

What this adds

WOLFSSL_DTLS13_STATEFUL_SERVER, via --enable-dtls13-stateful-server (and a CMake option of the same name), plus the per-connection setter wolfSSL_dtls13_use_stateful_server(). Server side of a DTLS connection only, off by default.

When enabled the server reassembles a fragmented ClientHello including the very first one, and completes in 1-RTT with no cookie when the peer already offers a usable key share. All three cases above then pass.

Crucially, the cookie is still sent whenever a HelloRetryRequest is actually needed, and the second ClientHello is validated by the normal cookie path. Clients that omit their key share from the first ClientHello -- wolfSSL's own DTLS client does this for PQC groups -- therefore keep working unchanged.

Security trade-off

This waives the HelloRetryRequest cookie's return-routability check (RFC 9147 section 5.1) for those handshakes. The server keeps state for, and completes a handshake with, a peer whose source address has not been validated, and answers a ClientHello that may be a few hundred bytes with its full flight of several kB -- i.e. it can be used as a traffic amplifier.

That is why it is opt-in and off by default. The trade-off is stated in the configure help, the CMake option comment, the wolfSSL_dtls13_use_stateful_server() documentation, the src/dtls.c macro block and the example server usage, so it is visible at the point of decision.

Default builds are unaffected: every behavioural change is inside #ifdef WOLFSSL_DTLS13_STATEFUL_SERVER. The one hunk outside the guard is a refactor of the existing cookie check in DoTls13ClientHello() into a cookieOptional local, which is truth-table identical without the macro.

NSS interop tests

.github/workflows/nss.{sh,yml} previously covered only wolfSSL-server / NSS-client against an NSS (3.107) that predated ML-KEM. Now:

  • full matrix of {wolfSSL, NSS} as server x {TLS 1.3, DTLS 1.3} x {classical, ML-KEM hybrid}
  • DTLS cases for SecP384r1MLKEM1024, whose key share alone overflows a datagram, so the first ClientHello is already fragmented
  • a case that forces a HelloRetryRequest (NSS offered x25519,secp384r1mlkem1024 against a server that accepts only the hybrid) so the fragmented second ClientHello carrying the cookie is exercised
  • NSS bumped to NSS_3_123_1_RTM; NSS server roles added via selfserv (TLS) and tstclnt -P server (DTLS), with its identity imported from the existing certs/test-servercert.p12

The negotiated-group assertion anchors on the post-handshake output (SSL curve name is ... / Key Exchange Group: ...) rather than the bare group name, because the example server echoes Using Post-Quantum KEM: <group> while merely setting up its key share, before any handshake -- matching that would pass even on a downgrade.

Testing

  • ./configure --enable-all --enable-dtls13-stateful-server && make && ./tests/unit.test: passes, no new failures, zero compiler warnings
  • new tests/test-dtls13-stateful-server.conf (classical, 1-RTT no cookie) and tests/test-dtls13-stateful-server-pq.conf (PQC hybrid, exercises the HRR + cookie + fragmented CH2 path); both wired into tests/suites.c
  • new API test test_wolfSSL_dtls13_use_stateful_server covering NULL, client-side and non-DTLS rejection
  • --enable-dtls13-stateful-server without DTLS 1.3 is rejected by configure, by CMake, and by a settings.h #error for user_settings.h builds
  • the option is added to the all-pq-quic job in pq-all.yml so the new conf tests run in CI
  • live NSS 3.123.1 interop: all 11 cases pass

Note

NSS has no DTLS Connection ID (RFC 9146/9147) as of 3.123.1, so WOLFSSL_DTLS_CID cannot be interop-tested against it; the handshake cleanly falls back to no-CID.

NSS fragments its ML-KEM ClientHello and orders the large key_share ahead
of the cookie extension, so the cookie is not in the first fragment. A
stateless wolfSSL DTLS 1.3 server drops such a ClientHello and the
handshake stalls on HelloRetryRequest retransmits, which makes a wolfSSL
server talking DTLS 1.3 to an NSS client with a post-quantum group fail
outright.

Add WOLFSSL_DTLS13_STATEFUL_SERVER, exposed as
--enable-dtls13-stateful-server and as the CMake option of the same name,
together with the per-connection setter
wolfSSL_dtls13_use_stateful_server(). When enabled, the server
reassembles a fragmented ClientHello including the very first one, and
completes in 1-RTT with no cookie when the peer already offers a usable
key share. A cookie is still sent whenever a HelloRetryRequest is
actually needed, so clients that omit their key share from the first
ClientHello, as wolfSSL's own DTLS client does for PQC groups, keep
working over the normal cookie path.

This waives the HelloRetryRequest cookie's return-routability check from
RFC 9147 section 5.1 for those handshakes: the server keeps state for,
and completes handshakes with, unvalidated source addresses, and answers
a small ClientHello with its full flight, so it can be used as a traffic
amplifier. It is therefore opt-in and off by default, and the trade-off
is spelled out in the configure help, the API documentation and the
example server usage.

Extend the NSS interop workflow to the full matrix of {wolfSSL, NSS} as
server x {TLS 1.3, DTLS 1.3} x {classical, ML-KEM hybrid}. Add DTLS cases
for SecP384r1MLKEM1024, whose key share alone overflows a datagram so the
first ClientHello is already fragmented, and a case that forces a
HelloRetryRequest so the fragmented second ClientHello carrying the
cookie is exercised as well. NSS is bumped to NSS_3_123_1_RTM for ML-KEM
support.

Verified against a local NSS 3.123.1 build: all 11 interop cases pass,
and the default (stateless) server still drops the fragmented NSS
ClientHello, so the new behaviour is confined to the opt-in path.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant