dtls13: add an opt-in stateful server and extend the NSS interop tests - #33
Open
Frauschi wants to merge 1 commit into
Open
dtls13: add an opt-in stateful server and extend the NSS interop tests#33Frauschi wants to merge 1 commit into
Frauschi wants to merge 1 commit into
Conversation
NSS fragments its ML-KEM ClientHello and orders the large key_share ahead
of the cookie extension, so the cookie is not in the first fragment. A
stateless wolfSSL DTLS 1.3 server drops such a ClientHello and the
handshake stalls on HelloRetryRequest retransmits, which makes a wolfSSL
server talking DTLS 1.3 to an NSS client with a post-quantum group fail
outright.
Add WOLFSSL_DTLS13_STATEFUL_SERVER, exposed as
--enable-dtls13-stateful-server and as the CMake option of the same name,
together with the per-connection setter
wolfSSL_dtls13_use_stateful_server(). When enabled, the server
reassembles a fragmented ClientHello including the very first one, and
completes in 1-RTT with no cookie when the peer already offers a usable
key share. A cookie is still sent whenever a HelloRetryRequest is
actually needed, so clients that omit their key share from the first
ClientHello, as wolfSSL's own DTLS client does for PQC groups, keep
working over the normal cookie path.
This waives the HelloRetryRequest cookie's return-routability check from
RFC 9147 section 5.1 for those handshakes: the server keeps state for,
and completes handshakes with, unvalidated source addresses, and answers
a small ClientHello with its full flight, so it can be used as a traffic
amplifier. It is therefore opt-in and off by default, and the trade-off
is spelled out in the configure help, the API documentation and the
example server usage.
Extend the NSS interop workflow to the full matrix of {wolfSSL, NSS} as
server x {TLS 1.3, DTLS 1.3} x {classical, ML-KEM hybrid}. Add DTLS cases
for SecP384r1MLKEM1024, whose key share alone overflows a datagram so the
first ClientHello is already fragmented, and a case that forces a
HelloRetryRequest so the fragmented second ClientHello carrying the
cookie is exercised as well. NSS is bumped to NSS_3_123_1_RTM for ML-KEM
support.
Verified against a local NSS 3.123.1 build: all 11 interop cases pass,
and the default (stateless) server still drops the fragmented NSS
ClientHello, so the new behaviour is confined to the opt-in path.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
NSS fragments its ML-KEM ClientHello and orders the large
key_shareahead of the cookie extension, so the cookie does not land in the first fragment. A stateless wolfSSL DTLS 1.3 server drops such a ClientHello and the handshake stalls on HelloRetryRequest retransmits.Measured against a locally built NSS 3.123.1, with a default wolfSSL DTLS 1.3 server:
mlkem768x25519secp384r1mlkem1024(fragmented first CH)So wolfSSL-server / NSS-client DTLS 1.3 with a post-quantum group does not work today. The other direction (NSS server, wolfSSL client) already works, because our client sends an empty key share and gets an HRR.
What this adds
WOLFSSL_DTLS13_STATEFUL_SERVER, via--enable-dtls13-stateful-server(and a CMake option of the same name), plus the per-connection setterwolfSSL_dtls13_use_stateful_server(). Server side of a DTLS connection only, off by default.When enabled the server reassembles a fragmented ClientHello including the very first one, and completes in 1-RTT with no cookie when the peer already offers a usable key share. All three cases above then pass.
Crucially, the cookie is still sent whenever a HelloRetryRequest is actually needed, and the second ClientHello is validated by the normal cookie path. Clients that omit their key share from the first ClientHello -- wolfSSL's own DTLS client does this for PQC groups -- therefore keep working unchanged.
Security trade-off
This waives the HelloRetryRequest cookie's return-routability check (RFC 9147 section 5.1) for those handshakes. The server keeps state for, and completes a handshake with, a peer whose source address has not been validated, and answers a ClientHello that may be a few hundred bytes with its full flight of several kB -- i.e. it can be used as a traffic amplifier.
That is why it is opt-in and off by default. The trade-off is stated in the configure help, the CMake option comment, the
wolfSSL_dtls13_use_stateful_server()documentation, thesrc/dtls.cmacro block and the example server usage, so it is visible at the point of decision.Default builds are unaffected: every behavioural change is inside
#ifdef WOLFSSL_DTLS13_STATEFUL_SERVER. The one hunk outside the guard is a refactor of the existing cookie check inDoTls13ClientHello()into acookieOptionallocal, which is truth-table identical without the macro.NSS interop tests
.github/workflows/nss.{sh,yml}previously covered only wolfSSL-server / NSS-client against an NSS (3.107) that predated ML-KEM. Now:SecP384r1MLKEM1024, whose key share alone overflows a datagram, so the first ClientHello is already fragmentedx25519,secp384r1mlkem1024against a server that accepts only the hybrid) so the fragmented second ClientHello carrying the cookie is exercisedNSS_3_123_1_RTM; NSS server roles added viaselfserv(TLS) andtstclnt -P server(DTLS), with its identity imported from the existingcerts/test-servercert.p12The negotiated-group assertion anchors on the post-handshake output (
SSL curve name is .../Key Exchange Group: ...) rather than the bare group name, because the example server echoesUsing Post-Quantum KEM: <group>while merely setting up its key share, before any handshake -- matching that would pass even on a downgrade.Testing
./configure --enable-all --enable-dtls13-stateful-server && make && ./tests/unit.test: passes, no new failures, zero compiler warningstests/test-dtls13-stateful-server.conf(classical, 1-RTT no cookie) andtests/test-dtls13-stateful-server-pq.conf(PQC hybrid, exercises the HRR + cookie + fragmented CH2 path); both wired intotests/suites.ctest_wolfSSL_dtls13_use_stateful_servercovering NULL, client-side and non-DTLS rejection--enable-dtls13-stateful-serverwithout DTLS 1.3 is rejected by configure, by CMake, and by asettings.h#errorforuser_settings.hbuildsall-pq-quicjob inpq-all.ymlso the new conf tests run in CINote
NSS has no DTLS Connection ID (RFC 9146/9147) as of 3.123.1, so
WOLFSSL_DTLS_CIDcannot be interop-tested against it; the handshake cleanly falls back to no-CID.