Realtime SDK: from-scratch rewrite (REST + Realtime)#37
Draft
paddybyers wants to merge 68 commits into
Draft
Conversation
DESIGN.md defines the complete public API surface for the SDK rewrite: unified ErrorInfo type, Transport/HttpClient traits, builder-pattern publish, REST state management with token caching and fallback hosts. PROGRESS.md tracks phase completion across sessions. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Restructure the SDK from a monolithic lib.rs into focused modules: - protocol.rs: wire types (pub(crate)) and state enums (pub) - transport.rs: Transport/TransportConnection traits (pub(crate)) - channel.rs: RealtimeChannel, Channels, RealtimePresence stubs - realtime.rs: Realtime, Connection, RealtimeAuth stubs - http_client.rs: HttpClient trait (pub(crate)) - mock_ws.rs: MockWebSocket/MockTransport test infrastructure - presence.rs: PresenceMap, LocalPresenceMap Port all 1,157 tests from the monolithic unit_tests module into 12 thematic test files split by spec prefix (RSC, RSA, RTL, RTP, etc.). Tests compile but Realtime implementations are stubs returning todo!(). Remove old examples that referenced the previous API. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Full REST implementation with token auth, HMAC-SHA256 signing, fallback host caching (RSC15f), 401 token renewal, pagination, message encoding/decoding, push admin, and batch operations. 662 tests pass (all REST); 440 remaining failures are Realtime stubs. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- tests_rest_integration.rs: 83 tests (47 active vs sandbox, 36 ignored stubs) - Reorganize unit tests into tests_rest_unit_* / tests_realtime_unit_* by UTS structure - Add TokenAuthCannotRevokeTokens (40162) error code, use it in revoke_tokens (RSA17d) - Add CLAUDE.md with test baseline and conventions Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…batch/revoke envelopes) - MessageAction values now match TM5 (update no longer sends DELETE on the wire) - update/delete/append send full RSL4-encoded message fields; 40003 on missing serial; optional MessageOperation; UpdateDeleteResult preserves null versionSerial (UDR2a) - batch_presence: comma-joined channels param + BatchResult envelope response (BAR2) - batch publish/presence results discriminate Success/Failure on error key - batch_publish validates empty specs/channels/messages (RSC22), encodes per RSL4 (RSC22c6) - revoke_tokens parses v3 BatchResult envelope with legacy array fallback (RSA17c) - binary payloads: native bin under msgpack, base64 under JSON (RSL4c) - AuthOptions::default().method is GET (AO2d); stale tests fixed (rsa17d, rsh1b3, tm3) - Tests rewritten UTS-faithful from uts/rest/unit specs; legacy duplicates removed Unit: 724 pass / 439 fail (realtime stubs) / 92 ignored. Integration: 47/47 vs sandbox. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- authUrl (RSA8c) implemented: GET/POST, headers/params, TokenRequest/TokenDetails/JWT responses; Credential::TokenRequest exchangeable; AuthToken::Token for JWT callbacks - key+clientId keeps basic auth with X-Ably-ClientId (RSA7e2); clientId no longer forces token auth (RSA4); header sent only under basic auth - create_token_request omits ttl/capability when unspecified (RSA5/RSA6) — fixes 40160 for restricted keys; effective params merge defaultTokenParams + clientId (RSA7d) - authorize() forces token auth (RSA10a), replaces stored params/options (RSA10g/h, key preserved RSA10i), never stores timestamp; request_token is stateless (RSA8f) - queryTime honored with cached clock offset (RSA9d/RSA10k); time() unauthenticated (RSC16) - pre-emptive renewal of expired tokens (RSA4b1); 40171 when unrenewable (RSA4a2) - RSA15 clientId compatibility at construction and runtime (40102) - RSA17d_2 key+useTokenAuth revoke rejection; Key Debug/Display redact the secret - AuthOptions is full AO2; auth signatures take Option<&TokenParams>/Option<&AuthOptions> - vacuous auth tests replaced with 18 UTS-derived behavior tests Unit: 731 pass / 439 fail (realtime stubs) / 92 ignored. Integration: 47/47 vs sandbox. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…e, PublishResult) - RSL1k: idempotent publishing default true; base64url(9B):index ids, one base per publish, client ids preserved; RSC22d per-spec ids in batch publish - RSL5/RSL6: encryption wired end-to-end — shared encode/decode codec, cipher from builder or channel, decrypting history/get_message/versions/presence/pagination; cipher() is no longer a silent no-op - RSL6b: failed/unknown decode steps leave the unprocessed chain prefix - RSL1c/RSL1n: multi-message publish via messages(); send() returns PublishResult with per-message serials (null = conflated, PBR2a) - RSL1i: TM6 size calculation; RSL4a: top-level JSON scalars rejected (40013) - Tests: strict UTS-derived idempotency/serials/encryption tests incl. canonical ably-common crypto fixtures (128/256-bit, all items) Unit: 742 pass / 439 fail (realtime stubs) / 91 ignored. Integration: 47/47 vs sandbox. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…esponse, logging - endpoint option with hostname/routing-policy/nonprod resolution; new default domains main.realtime.ably.net + main.[a-e].fallback.ably-realtime.com - request_id stable across retries and attached to ErrorInfo (RSC7c) - httpMaxRetryDuration enforced (TO3l6); retriable bounded to 500-504 (RSC15l3); http_open_timeout wired as connect timeout; primary no longer cached as fallback - Rest::request() returns HttpPaginatedResponse (HP1-8): normalised items, success/errorCode/errorMessage, pagination, per-request version (RSC19f1); HTTP errors inspectable rather than Err; raw mode still token-renews on 401 - RSC2 logging: LogLevel filtering + log_handler with request/retry/error events Unit: 751 pass / 439 fail (realtime stubs) / 91 ignored. Integration: 47/47 vs sandbox. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…RSAN1c4 - Channel::status() + ChannelDetails/ChannelStatus/ChannelOccupancy/ChannelMetrics with UTS tests; set_options() applies cipher (RSL7) - annotation publish generates idempotent ids (RSAN1c4) - realtime-dependent tests re-homed to realtime files: tests_rest_* is fully green - remaining vacuous/conditional tests made strict or deleted; duplicates removed - DESIGN.md amended for all Phase R API changes + RSN ephemeral-channel decision Unit: 746 pass / 439 fail (all realtime stubs) / 91 ignored. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…load asserts, 5 stubs)
- sandbox moved to endpoint("nonprod:sandbox") per UTS; provisioning updated
- suite now runs over MessagePack (SDK default) — first live binary-protocol
coverage; explicit JSON + msgpack round-trip variant tests added
- payload assertions: rsl2a typed data, rsp3a2 raw-string, rsl1k5 first-write-wins
with stable-poll dedup check
- implemented stubs: callback TokenRequest exchange, history time range,
capability restriction (native token, 40160), publish serials, live encrypted
publish/history round-trip
Unit: 753 pass / 439 fail (realtime stubs) / 86 ignored. Integration: 54 pass / 31 ignored.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The serials half of the reason was fixed in R3; the fixture namespace is the only remaining blocker. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…voke integration - implement all 8 UTS proxy tests via the uts-proxy harness (auto-download); fallback classification, error parsing, and idempotent retry dedup verified end-to-end against the live sandbox through fault injection - fallback retry no longer drops fallback hosts equal to the primary - ably-common submodule updated (keys[4] revocableTokens, mutable namespace); 8 newly-unblocked integration tests implemented and passing live - fix Annotation wire field: messageSerial per TAN2j (was msgSerial); RSAN1c2 sets it on publish/delete — caught by the live annotation lifecycle test Unit: 768/440(realtime stubs)/70. Integration: 62 pass / 15 ignored. Proxy: 8/8. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- DESIGN.md 'Realtime State & Concurrency': single-writer connection loop owns all protocol state (zero locks on it); LoopInput/Command protocol; watch + broadcast observation contract; in-loop deadline timers; generation-guarded transport tasks; all-channels-in-loop decision; presence ownership; explicit invariants; alternatives analysis; test sourcing DECIDED as regenerate-from-UTS with ported tests as cross-check/quarry (option 2, user-approved); §14 enforcement mechanisms - tests_design_conformance.rs: lock-inventory ratchet (fails the build on any sync primitive beyond the documented whitelist) + no-pub-fields check on loop-owned structs; immediately caught and quarantined the two pre-design stub presence-map mutexes (temporary whitelist, removed in stage 5.7) - CLAUDE.md: binding realtime design contract loaded into every session Awaiting human review of the design before any Phase 5 implementation. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… UTS tests - connection.rs: the DESIGN.md loop (ConnectionCtx owns all state, no locks; generation-guarded connect/reader/writer tasks; watch-before-broadcast) - realtime.rs: thin Connection/Realtime/RealtimeAuth handles; RTN26 whenState - ws_transport.rs: tokio-tungstenite production transport (json/msgpack frames) - mock_ws.rs: full UTS mock_websocket.md implementation - behaviors: RTN2 URL, RTN3, RTN4(+h), RTN8/9(+c), RTN11, RTN12a/d/f, RTN25, RTN26 - 20 UTS-derived tests (all first-run green) + live sandbox connect/close test; 17 superseded ported tests deleted per DESIGN §12; conformance ratchet green with the two new modules added at zero allowance 851 pass / 363 fail (realtime stubs) / 70 ignored. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- in-loop deadline timers (DESIGN §5): connect timeout, close timeout (RTN12b), RTB1 backoff+jitter retries (RTN14d), connectionStateTtl suspension (RTN14e/f), idle timeout with traffic reset (RTN23a), ping deadlines (RTN13c) - RTN15 resume: immediate reconnect with resume=key, c6/c7 id comparison, key refresh, TTL-cleared state; RTN15h/RTN14b token-error renewal in the spawned connect task; RSA4a/RTN15h1 unrenewable → FAILED - Connection::ping (RTN13a/b/c/e); echo=false param (RTN2b) - 24 new UTS-derived tests (45 total, all green; paused-clock timer tests); live test now pings over the real connection; 11 implementation-pinned/racy ported tests superseded; conformance ratchet green 891 pass / 336 fail (remaining stubs) / 70 ignored. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…/RTC8) - loop-driven host cycling: primary first, shuffled REC2 fallbacks, qualifying failures advance within the CONNECTING phase; Connection::host(); RTN17e REST fallback affinity - AUTH message → off-loop token renewal → AUTH reply with accessToken; stays CONNECTED with UPDATE on the server ack; RealtimeAuth::authorize applies tokens in place (RTC8) - realtime ported tests migrated to REC domains; 7 rtn17 tests adopted; 2 racy rtn22a tests superseded; RTN14 tests isolated from cycling via empty fallback sets; connectivity check deferred (recorded) 901 pass / 327 fail (remaining stubs) / 70 ignored. Conformance + live green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ttach/detach, RTL3 effects ChannelCtx (state machine, serials, modes, op deadlines, pending repliers) is loop-owned plain data; the Channels handle registry is the design's one sanctioned realtime lock. Attach/detach per RTL4/RTL5 incl. pending-state continuations, op timeouts, queueing while not CONNECTED; RTL3 connection effects incl. reattach-on-CONNECTED with serial + ATTACH_RESUME; RTL2 events with UPDATE/resumed/hasBacklog semantics (RESUMED suppresses UPDATE, RTL12); RTL15b1 serial clearing; RTL25 when_state. Fixes found deriving tests from UTS: RTL5l detach of a queued attach while not CONNECTED previously waited forever; UPDATE was emitted even when RESUMED was set. 37 UTS-derived tests (incl. live sandbox attach/detach), all green. Ported cross-check: 68 adopted (13 via mechanical tokio conversion), 12 superseded and deleted. Suite: 1029 pass / 224 fail (stubs) / 70 ignored, no hangs. Conformance ratchet green (channel.rs allowance unchanged at 3). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…s it found uts_coverage.txt maps all 963 UTS Test IDs to passing Rust tests (686) or excludes them with a stage/deferral reason (277); tests_uts_coverage.rs enforces the matrix on every cargo test (DESIGN.md §14.5). Bootstrap generator in tools/uts_coverage_generate.py. Behavior fixes the audit surfaced: - RTN13d: pings while CONNECTING/DISCONNECTED are now deferred and executed on CONNECTED (timeout from send, RTN13c; terminal states fail them, RTN13b) — previously errored immediately, pinned by a wrong test. - RTC8: authorize() now resolves only on the server's response (RTC8a3), halts+restarts an in-flight attempt with the new token (RTC8b), and initiates a connection from non-connected states (RTC8c/RTC8b1). - RSA4c3: failed RTN22 renewal while CONNECTED is silently swallowed. - RTC1a/RTC1f1: echo param explicit; transportParams override URL defaults. - RSA4f: >128KiB callback tokens rejected (80019); RSA4a1: 40171 warning for non-renewable literal tokens; RSL4a: bare-scalar JSON payloads rejected (40013). New option: fallback_retry_timeout (TO3l10). Tests: +19 realtime UTS (RTN13d x5, RTC8 x10, RTF1/RTN22a/RSA4f/RSA4a1), +18 REST backfill; 7 ported client tests converted sync→tokio, 8 superseded deleted. Suite: 1079 pass / 202 fail (stubs) / 66 ignored; REST unit all green; both ratchets green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… mutations Loop-owned msgSerial + pending-ACK queue + connection-wide message queue; RTL6 publish state table; RTN7d/e pending-publish outcomes; RTN19a/a2/b resend on new transports (serials kept on resume, renumbered on failed resume; DETACH resend was a real gap). Subscriber registry per DESIGN §8 (all/name/RTL22 MessageFilter), RTL7g implicit attach, RTL17 gating, RTL8 unsubscribe. Inbound TM2 field population + RSL6 decode + RTL15b serials + RTF1/RSF1 tolerance. RTL32 update/delete/append as wire operations resolving via ACK (40003 serial validation, versionSerial). LIVE-CAUGHT: the service emits duplicate map keys in msgpack frames; serde rejects them, so every inbound MESSAGE over msgpack was dropped. Tolerant decode (dedup, re-encode) fixes it; flagged upstream. RSL4a now treats JSON scalar strings as string payloads. 25 UTS tests (incl. live msgpack publish/subscribe echo), ~60 ported adopted, 14 superseded. Matrix: 5.5 exclusions converted (766 mapped). Suite: 1161 / 132 (5.6+ stubs) / 66; both ratchets green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…annels Authoritative channel options move into the loop (ChannelCtx), observable via ChannelSnapshot; get_with_options is now fallible (approved amendment): Err 40000 when params/modes change on a live channel (RTS3c1), safe updates via Command::SetOptions (RTS3c). RTL16/16a set_options reattaches when needed and resolves on re-ATTACHED. RTL13a/b/c server-initiated DETACHED: immediate reattach, SUSPENDED + RTB1-jittered channelRetryTimeout retries (ChannelStateChange.retry_in), cancelled off-CONNECTED; attach timeouts join the retry cycle. RTS5 derived channels ([filter=b64?params]name). RTN25: clean CONNECTED clears errorReason. 10 UTS tests; ported rtl13/rtl16/rts3c/rts5 groups adopted (2 broken rts5 ports fixed per UTS), 9 superseded. Matrix: 791 mapped / 174 excluded. Suite: 1185 / 112 (presence+annotations stubs) / 63; ratchets green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…eleted PresenceMap/LocalPresenceMap implement the full RTP2/RTP17/RTP18/RTP19 semantics as pure data; PresenceCtx joins ChannelCtx with the inbound PRESENCE/SYNC engine, RTP1/RTP19a attach semantics, RTP5 state effects, RTP17 internal map + automatic re-entry (RTP17e 91004 UPDATE on failure). Ops follow the RTP16 table over the PRESENCE+ACK pipeline (RTP8c implicit identity, RTP8j/RTP15f validation); RTP11 get defers on waitForSync and fails 91005 on SUSPENDED (a deferred-get hang the tests caught); RTP6/7 subscribers with per-action narrowing. The two temporary stub mutexes are deleted; the conformance allowance drops 3->1 — the lock inventory is at its designed steady state. Upstream UTS conflict flagged: RTP8j wildcard rejection vs wildcard-identity setups in RTP14a/15a/15c/RTP4. 17 UTS tests (incl. live sandbox presence round trip); ported sweep: 93 adopted, 26 superseded. Matrix: 895 mapped / 70 excluded. Suite: 1274 / 14 (annotations) / 63, no hangs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…REEN ANNOTATION ProtocolMessages through the shared ACK/NACK pipeline (CREATE/DELETE actions, messageSerial per TAN2j, type validation), gated by the message-publish state table (RTAN1b). Inbound dispatch to type-filtered subscribers (RTAN4), implicit attach, missing-mode warning (RTAN4e). ChannelMode::AnnotationPublish/AnnotationSubscribe with flag mappings. 4 UTS tests; ported sweep 10 adopted / 7 superseded (5 were compile-shells that hung awaiting ACKs). Matrix: 909 mapped / 56 excluded (all recorded deferrals). Suite: 1287 pass / 0 fail / 61 ignored — every realtime stage (5.1-5.8) complete. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ive fixtures Clippy clean across all targets (documented policy allows in lib.rs; num-derive 0.4). The coverage ratchet now tracks brace depth after an unbalanced edit was found to have hidden 4 compiled-but-never-run tests (one carried a latent bug). 9 previously realtime-blocked integration tests implemented live: RSP4* presence history via realtime fixtures, RSC24 batch presence, RSA17g revocation observed as a live 4014x forced disconnect, RSA17c mixed revocation; 10 stale shells deleted. Serial: integration 68/5-ignored, proxy 8/8. Matrix: 909 mapped / 54 excluded (recorded deferrals only). FINAL: 1297 pass / 0 fail / 41 ignored. The clean rewrite is complete. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
backlog init (agent instructions in CLAUDE.md) + the deferred-work backlog loaded as TASK-1..10: JWT test enablement, PresenceMessage::size, TM2s1 version defaulting, RTN16 recovery, RTN17j (behind dual-mock test infra), push LocalDevice project, delta/vcdiff plugin, RTN20 network events, upstream issue filing, and test housekeeping. Repo .tool-versions pins nodejs so the backlog shim resolves in-repo. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
TASK-11 integration-spec traceability + missing realtime-integration tests (157 untraced IDs, realtime/integration largely uncovered); TASK-12 fix the 18 unverified claim-set matrix mappings (RSA16a capability proven false); TASK-13 systematic logging pass (6 call sites today, silent discard paths, no default sink); TASK-14 connection.rs refactor (3k lines, duplicated attach-time presence/re-entry logic). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
DESIGN.md gains a NORMATIVE observability policy (levels, no-silent-discard rule, trace-at-entry, secrets exclusion); CLAUDE.md gains three binding engineering policies: instrumentation is part of done, the whole UTS tree is dispositioned, matrix mappings are verified claims (generators draft, humans disposition). Mechanical half: tests_uts_coverage.rs now enumerates every uts/ area containing Test IDs and fails unless the area is traced or carries an explicit '!area <name> -- <reason>' line; negative-tested. The integration areas are dispositioned as visible IOUs pending TASK-11; objects/ and docs/ as out-of-scope with reasons. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ards Logger handle in options.rs (level-gated, lazy formatting, cloneable) with an optional 'tracing' cargo feature bridging to the tracing crate when no handler is installed. Instrumented per the DESIGN.md observability policy: Major for connection/channel transitions and UPDATEs; Minor for resume outcomes, retry scheduling, queue flush/resend, SYNC lifecycle; Micro for every realtime API entry and the wire; Error for every discard path — undecodable frames (incl. tolerant-msgpack failures that previously vanished), undecodable entries, unknown-serial ACK/NACK, transport write failures. Three policy tests assert discards/transitions/traces log. Suite: 1300 / 0 / 41; clippy clean on default and tracing features. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…proxy tests
Matrix now spans rest+realtime, unit+integration: 1120 IDs — 1056 mapped,
66 excluded with reasons, 0 unresolved; objects/ and docs/ dispositioned
via !area lines. Ratchet (tests_uts_coverage.rs) scans all four areas.
New: tests_realtime_integration.rs (13 live-sandbox tests) and
tests_proxy_realtime.rs (28 uts-proxy fault-injection tests over the real
WebSocket transport). Full serial suite: 1342 passed / 0 failed / 41 ignored.
SDK fixes the new tests forced:
- RTL15b: SYNC no longer updates the channel serial — the sync cursor is not
a channel serial, and sending it back in a reattach ATTACH (RTL4c1) was
rejected by the server ("Unable to parse channel params").
- RTN15h1: DISCONNECTED token error with a non-renewable token now fails the
connection with 40171 (was pass-through 40142), server error kept as cause
(TI1). Unit-spec conflict recorded in TASK-9 (item 6).
- RTN19a: typed PendingPayload — resends reconstruct the same ProtocolMessage
kind (MESSAGE/PRESENCE/ANNOTATION) instead of pre-serialized JSON.
Test-infra fixes:
- uts-proxy spawns with null stdio (inherited stdout held test pipes open).
- Randomized proxy port base + session-create retry (sessions orphaned by
panicked tests hold their port on the long-lived daemon).
- await_state reads a coalescing watch and misses ms-fast DISCONNECTED
transients; fast reconnect cycles are asserted via a broadcast recorder
(await_states_in_order) or the proxy event log.
- Coverage generator: bare fn names deduplicate across modules — integration
eligibility now considers every module a name passes in.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The generator's score-0 fallback claimed spec variants by listing every same-token test without verifying any of them covered the variant. The class had grown to 31 IDs. Each was dispositioned by reading the spec variant and the candidate test: 17 tightened to the single verified covering test, 10 new tests written, 4 excluded with reasons (fallbackHostsUseDefault deliberately not exposed; connectivity check is TASK-5 scope). The fallback now emits `?? UNRESOLVED` with the candidate list, so the class cannot reappear. Matrix: 1052 mapped / 70 excluded / 0 unresolved; ratchet green. SDK bugs the tightened tests forced out: - Annotations skipped RSL4 data encoding on publish (REST and realtime) and RSL6 decoding on receipt/list — a JSON payload went out as a raw object instead of a string with encoding "json" (RTAN1a/RSAN1c3/RTAN4b1). - A connect-time 40102 IncompatibleCredentials (token clientId vs configured clientId) was retried forever; per RSA15c it is terminal — the connection now transitions to FAILED. Notable new tests: rtl10b_until_attach_bounded_by_attach_point proves the fromSerial attach bound behaviorally against the live sandbox (the unit mock cannot see the HTTP layer until TASK-5, and the uts-proxy strips query strings from its http_request log); rtp5f_suspended_maintains_presence_map drives a real connection into SUSPENDED via a 1ms connectionStateTtl. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
generate_jwt() mints Ably-shaped HS256 JWTs (kid=keyName, x-ably-clientId). rsa8_jwt_token_auth, rsa8_auth_callback_jwt and rsc10_token_renewal_with_expired_jwt replace their ignored stubs; the matrix maps their IDs to the real tests (they were auto-matched to plausible-but-wrong ones). Suite: 1354 / 0 / 38. An "expired" JWT needs iat in the past too: with iat=now and exp<now the server computes a negative ttl and rejects the JWT as malformed (400/40003) rather than expired (401/40142), which never exercises RSC10 renewal. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
connection.rs (3,417 lines) becomes connection/{mod,channel_arm,
presence_arm,publish_arm}.rs. The ownership model is untouched: every state
type stays in mod.rs, owned by the loop; the arms hold only behaviour
(pub(super)/pub(crate) methods on the same structs). The conformance
ratchet now scans all four files at allowance 0.
The duplicated RTP17i re-entry block and HAS_PRESENCE attach handling
(each pasted twice in handle_attached) are now single helpers:
ConnectionCtx::reenter_internal_members and ChannelCtx::apply_has_presence.
A PendingReply enum (Publish/Op) on PendingPublish replaces the per-op
spawned oneshot-bridge tasks for presence and annotation ops; raw
91004/91005 codes replaced with ErrorCode variants.
No behavior change: suite identical before and after — unit 1246/0/26,
live integration + proxy serial 125/0/2, both ratchets green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
get_sandbox() registers a libc::atexit handler on first provision; the
handler DELETEs /apps/{appId} with basic key auth as a raw HTTP/1.1
request over std TcpStream + native-tls. An async runtime cannot be
started inside atexit — reqwest's blocking client aborts the process
there — so the request is hand-rolled with explicit error handling and
graceful degradation (sandbox apps are autodelete-labelled anyway).
Verified live: DELETE returns 204 and the handler logs one line at exit.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ureq is purely blocking (threads and sockets, no async runtime), so it is safe inside the atexit handler — the constraint was never "no HTTP library", only "no reqwest::blocking", which starts a tokio runtime and aborts the process at exit. Drops 45 lines of bespoke HTTP parsing and the direct native-tls dev-dependency. Verified live: DELETE 204 at exit. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The identical mock_client/get_mock/mock_client_json trio was pasted into 12 unit-test files (verified hash-identical); one definition now lives in test_support.rs. Suite unchanged: 1246/0/26. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Each test's body was matched against features.md to identify the spec item it actually asserts (TG3, TI1/2/3/5, TP2, TD1/2/5/7, RSC19f, HP4, RSA11/ RSA3b/RSA17c/d, RSC1a, RSN3a, RSL1b/1e/2a/4d1-3, RSP3a/4a, TM2/TM2d, TO3d/n/k1, RSC16, RSC7d2). 65 got real IDs; the 6 that assert pure Rust mechanics (ErrorCode enum conversions, Display/Debug/std::error impls) are named plainly rather than given fake IDs. Bodies untouched; the names were unreferenced by the curated matrix. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The regen diff caught 6 regressions: tasks 2/3/5 converted matrix exclusions without updating the generator's OVERRIDES dispositions (REC3/ REC3a/REC3b, TM2s1, TP5 reverted to stale exclusions), and the TASK-10 renames caused one auto-match drift (RSP4a/history-returns-paginated-1 re-pointed to a URL-only test; pinned back to the verified item-returning test). OVERRIDES updated — the generator now reproduces the curated matrix byte-identically from a fresh full serial run (1371/0/28; 1120 IDs, 0 unresolved). Task-10 complete. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Decoder strategy is settled: own pure-Rust decoder at ably/vcdiff-rust, publishing as vcdiff-decode (import path vcdiff). ably must depend on it from crates.io (git deps forbidden in published crates), so the crate must be cargo-published before ably ships delta support. Captured the API, RTL18/19 discontinuity path, untrusted-input panic risk, and the VCD_TARGET gap for the eventual implementation. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The `ably` crate is owned by two personal accounts (lmars, Morganamilo) with no org team owner — a release blocker for the rewrite. vcdiff-decode is in the same state. Task covers confirming a current owner can still publish, adding an ably org team as owner of both crates, and deciding the personal-owner disposition. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Bundle the vcdiff-decode crate as a normal dependency (no user plugin). An internal DeltaDecoder seam wraps vcdiff::decode in production and is overridable in #[cfg(test)] so the RTL18/19/20/21 bookkeeping is tested with an injected mock — pass-through, recording, and failing — exactly as ably-js tests the same UTS IDs. No real deltas are embedded; real decoding is covered by vcdiff-decode's own conformance suite. - ChannelCtx::decode_message: RTL19a base64-first, RTL19b/19c base-payload storage (wire form, before json/utf-8), RTL20 delta.from vs stored last id, PC3a string-base -> utf8, then the RSL6 chain for residual steps. Base/last-id cleared on any transition out of ATTACHED. - handle_message_action: RTL21 in-array order; on decode failure or id mismatch, RTL18a log 40018 / RTL18b discard+stop / RTL18c re-attach from the previous message's channelSerial into ATTACHING (reason 40018). RTL18 single-recovery falls out of the RTL17 not-ATTACHED drop. - ErrorCode::VcdiffDecodeFailure = 40018. Negotiation reuses channel-option params (delta=vcdiff), no new API. 12 new tests; 11 unit matrix IDs mapped; PC3/no-plugin (40019) dispositioned N/A (decoder bundled, never absent); generator OVERRIDES updated so regen is byte-identical. Suite: 1383/0/17 full serial (incl. live sandbox + proxy). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The earlier "live sandbox -> exclude" was unjustified: the suite runs live tests routinely and the decoder seam makes counting/failing injection easy. These are the highest-value delta tests — real server, real decoder, real negotiation and recovery — and the end-to-end one empirically confirms the sandbox emits deltas our crate decodes (also answering the VCD_TARGET concern for real payloads). Implemented against the live sandbox: pc3_delta_decode_end_to_end (real deltas + the real bundled decoder, decode_count == n-1), pc3_no_deltas_without_param, rtl18_recovery_after_decode_failure (full recovery loop), rtl19b_dissimilar_payloads. Excluded with specific reasons (not blanket): the RTL18 live id-mismatch (covered by the RTL20 unit test; a live mismatch needs an internal test hook) and PC3 no-plugin 40019 (N/A, bundled). Generator OVERRIDES updated; regen byte-identical. Suite: 1387/0/17 full serial (unit 1258/15, live integration + proxy 129/2). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Delete top-level PROGRESS.md: its task-era half duplicated the backlog task files (the source of truth) and its phase-era half is covered by the detailed commit history — a purely-historical top-level artifact has no place in a library advertised as current. - Delete REVIEW-2026-06-10.md: a point-in-time audit whose findings are all resolved (they became the R-phases and the binding engineering policies); git history preserves it. - README.md: rewritten — it claimed "REST only, no realtime features" and used the pre-rewrite API. Now covers REST + realtime (connect, subscribe, publish, presence, delta) against the current API. - DESIGN.md: module layout now includes the connection/ event-loop submodules; new §9a documents the bundled vcdiff delta decoder and RTL18 recovery; the per-stage conformance rule points at commits/backlog notes instead of PROGRESS.md. - CLAUDE.md: drop the dead .claude/plans reference and the PROGRESS.md pointers. - Repoint the two remaining PROGRESS.md references (task-9 text, a presence test comment) to the backlog / spec PR. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Restructure the README to match ably-js/ably-java/ably-python: title "Ably Pub/Sub Rust SDK", intro + Find out more, Getting started, Supported platforms, Installation, a single canonical realtime Usage example (connect/subscribe/publish), then Contribute/Releases/Support. The developer-preview status and not-yet-implemented features (push LocalDevice, network events) are called out per the template's convention. Add CONTRIBUTING.md in the same house style, adapted to this repo: fork/PR flow with submodule init, build, the unit vs live-sandbox (serial) test split, fmt/clippy, the UTS-derived test + traceability-ratchet conventions (pointing at CLAUDE.md/DESIGN.md), and the crates.io release process. The README Contribute section links to it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
DESIGN.md was an implementation-steering artifact — an exhaustive catalogue of every type plus a running phase/prior-design narrative. Rewrite it as a doc for someone reading the repo now: design principles, module layout, the REST client, and the realtime state/synchronisation model (one event loop owns all state, no locks; the four handle primitives; generation guard; timers; transport/resume; presence; delta; invariants), plus the observability policy and the testing/enforcement mechanics. The per-type definitions are dropped (the code and rustdoc are the source of truth) and all "differences vs the prior design" narrative is removed. That prior-vs-current material now lives in MIGRATION.md, which documents how the public API differs from the last published (REST-only) 0.2.0 release: the Error -> ErrorInfo rename, presence field -> method, synchronous slice-returning pagination, PublishResult from send(), optional token args + new authorize(), Message.encoding as Option<String>, the crypto builder (generate_random_key removed), construction via ClientOptions, and the new realtime API. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The pinned uts-proxy release was v0.1.0; the current latest is v0.3.0. The asset naming also changed in v0.2.0 (the version is now embedded in the filename), so asset_name() now builds uts-proxy_<ver>_<platform>_<arch>.tar.gz and the checksum table carries the v0.3.0 hashes (from the release checksums.txt; verified against the downloaded darwin/arm64 tarball). The cache path is version-namespaced, so no stale binary is reused. Verified: all 38 proxy tests pass against v0.3.0 (download + checksum + extract + live-sandbox fault injection). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…anism Auth logic was split awkwardly — the auth types and the Auth facade lived in auth.rs, but ~250 lines of the actual machinery (AuthState/AuthHeader and the impl Rest auth methods) sat in rest.rs. Move it so auth.rs owns the durable auth core and rest.rs keeps only the client-owned state field and the generic request pipeline. Behaviour-preserving verbatim move (no logic/signature changes). Moved into auth.rs (as impl Rest blocks): AuthState/AuthHeader, acquire_token, get_auth_header, fetch_token_from_url (the authURL path — raw HttpClient, no Ably pipeline), auth_config[_with], effective_token_params, check_client_id_compat, token_auth_in_effect, adjusted_now_ms, invalidate_cached_token. The Key TYPE, the Auth facade, and revoke_tokens (same for all token types) stay in auth.rs. New token_request.rs isolates the deprecatable Ably-native-token mechanism: exchange_token_request + token_request_timestamp (impl Rest) and Key::sign/ sign_with_timestamp (impl Key). When native tokens give way to JWT, this file and the two acquire_token branches that call it are a bounded excision; basic auth, authURL, callbacks, revocation, and the state/header machinery are untouched. rest.rs keeps Mutex<auth::AuthState> (client-owned state). Only pub(crate) visibility widening (AuthHeader::value, do_request_internal); nothing widened beyond crate. Suite unchanged: unit 1258/0/15, live integration 91/0, clippy + fmt clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
`extras` is always a JSON object per the spec, but the wire types typed it as `Option<serde_json::Value>` — looser than the contract, and looser than the REST publish builder, which already took a `Map` and converted to `Value::Object` only when assembling the message. (The previously published API also used `Option<Map>`; the rewrite had loosened it.) Introduce `pub type Extras = serde_json::Map<String, serde_json::Value>` in rest.rs and use it for the `extras` field of Message/PresenceMessage/ Annotation. This is exactly as open as `Value` (arbitrary/unknown keys and value shapes) but rules out the nonsensical non-object cases — no typed schema, so it stays forward-compatible. The REST builder now stores the map directly; the realtime builder keeps its `Value` input (so `json!(...)` stays ergonomic) and stores the contained object. Removes json.rs, whose only remaining content was a `serde_json::Value` re-export (unused) and a `Map` alias referenced by three tests to build extras. `Extras` is re-exported at the crate root. Suite green: unit 1258/0/15, clippy + fmt clean, live extras round-trip (rtl6, over msgpack) passes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…nested types As of spec version 2.2 the deep per-type Stats structure (TS12d-o) is deleted in favour of a flat model. Rewrite stats.rs to the current TS12 shape: intervalId (TS12a), unit as StatsIntervalGranularity (TS12c), inProgress (TS12q), a flat `entries: HashMap<String, f64>` map (TS12r; f64 since rate entries are fractional though the spec types them as int), schema (TS12s), and appId (TS12t). intervalTime (TS12p) is a method that parses intervalId (month/day/hour/minute granularities). All the old nested structs (MessageTypes, MessageTraffic, ConnectionTypes, Push, XchgMessages, Rates, …) are removed; nothing outside stats.rs referenced them. Updated the rsc6a unit test to the flattened fixture and to assert entries, unit, schema/appId, and interval_time. Added a unit test for interval-id parsing. Suite: 1259/0/15; live rsc6 + rtc5 pass. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Injects known datapoints into the sandbox via the authenticated POST /stats endpoint (G3), reads them back, and asserts the flattened TS12 round-trip: ingested `inbound.realtime.messages.*` metrics surface under the `messages.inbound.realtime.messages.*` entries keys, alongside intervalId/unit/schema/appId. This guards the deep-vs-flattened regression that previously went undetected: the flattened `entries` API is gated on `X-Ably-Version: 6`, and a client that falls back to the deprecated deep structure would deserialize into empty `entries` and fail here. Maps UTS rest/integration/RSC6/stats-flattened-entries-2. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
uts-proxy (>= v0.2.0) auto-assigns a free port when `port` is omitted from POST /sessions and reports it in the response. Switch to that: drop the client-side port allocator (allocate_port/NEXT_PORT) and the port-reuse retry loop, omit `port` in the create body, and read the bound port/host from the response `proxy` object. This removes the TOCTOU race of the caller guessing a free port and the 409 conflicts from sessions orphaned by panicked tests. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
protocol.rs mixed the public realtime state model with the internal wire protocol. Move the public types to their owning domain modules and leave protocol.rs wire-only: - ConnectionState/ConnectionEvent/ConnectionStateChange -> connection/mod.rs - ChannelState/ChannelEvent/ChannelStateChange/ChannelMode -> channel.rs protocol.rs now holds only ProtocolMessage and its supporting wire types, and is demoted to `pub(crate) mod` (it exposed no public items of its own). The crate-root re-exports are unchanged, so the public API (`ably::ConnectionState`, etc.) is byte-for-byte identical; only internal paths moved. Import sites now take the state types via the crate-root re-export and keep wire types on `crate::protocol::`. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The ably-common compliance/capabilities manifest is an obsolete feature-tracking mechanism, unreferenced anywhere in the repo. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replace the misnamed empty `rsl6a3_vcdiff_decode` placeholder with the real RSL6a3 tests. RSL6a3 is about decoding binary (msgpack) ProtocolMessages against the shared cross-SDK interop fixtures — not deltas: vcdiff decoding is realtime-only (RSL6a applies deltas "in the case of a realtime client"), so there is no REST vcdiff path. Drives `ably-common/test-resources/msgpack_test_fixtures.json` through the existing decode pipeline (decode + round-trip); all 8 fixtures pass, confirming the REST decode path is complete. No production code change was required. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The only pin was nodejs, added with the Backlog.md integration for the local `backlog` CLI. The crate itself uses no Node — no package.json, no JS, and CI is pure Rust — so a Node toolchain pin at the repo root is misleading. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
check.yml: replace the archived actions-rs/* actions (unmaintained since ~2020, Node 12) and checkout@v2 with dtolnay/rust-toolchain@stable, Swatinem/rust-cache, and direct cargo invocations. Scope the test step to the deterministic unit suite plus the conformance ratchets; the live sandbox and uts-proxy integration tests need network access and are run manually, not on every PR. features.yml: remove. It invoked the Ably SDK-features/compliance workflow, which consumes the .ably/capabilities.yaml manifest removed earlier — an obsolete feature-tracking mechanism. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The uts_coverage ratchet reads the ably/specification tree at ../specification and couples the matrix to an exact spec commit, so it can't run in the SDK's own CI (no spec checkout; and the matrix currently references unmerged #507/#508 Test IDs). Keep the self-contained include_str!-based design_conformance ratchet; run uts_coverage manually alongside a spec checkout. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The previously published
ablycrate (0.2.0) was REST-only. This is afrom-scratch rewrite that retains a REST API (with breaking changes) and adds a
full Realtime API. Built and verified against the Ably Universal Test
Specification (UTS) and the features spec, with every spec point traced to a
test or an explicit, reasoned exclusion.
In scope
REST
PublishResult, extras.get.entriesAPI (TS12).HttpPaginatedResponse, structured logging.Realtime
recover=,createRecoveryKey), heartbeat/ping, fallback hosts (RTN17),server-initiated reauth (RTN22/RTC8).
idempotency, derived channels, RTL13 retries.
get, sync.vcdiff-decodecrate.Deliberately out of scope
These are excluded functionality, not unfinished work. Each corresponds to an
explicit UTS exclusion (
uts_coverage.txt) and a set of#[ignore]d tests:objects/UTS area is excluded.LocalDevice— device activation & registration,push-channel device/client subscriptions (RSH7), and
Realtime.pushdelegation (RTC13). (Push admin publish is in scope.)
connection state.
A handful of further UTS entries are excluded as test-methodology N/A rather
than missing features — cases unrepresentable in Rust's type system
(e.g. a wrong-typed
AuthCallbacktoken), the "no vcdiff plugin" states(the decoder is bundled, never absent), deprecated options deliberately not
exposed (
fallbackHostsUseDefault), and the protocol-heartbeat vstransport-ping design choice (RTN23b). All are recorded with reasons in
uts_coverage.txt.Architecture
The realtime core is a single connection loop that owns all mutable
protocol state — no locks on protocol state (the prior approach had accumulated
43 mutexes). See DESIGN.md. A design-conformance ratchet fails
the build if a forbidden sync primitive is introduced.
Testing & conformance
out-of-scope areas above).
uts_coverage.txt) maps every UTS Test ID to apassing Rust test or an explicit exclusion; a ratchet fails the build when the
matrix and the spec tree diverge.
and JSON.
errors, resume/recovery).
Related PRs & issues
wildcard-clientId presence setups, corrupt RSP5g fixture).
stats-injection integration test.
vcdiff-decodepublished to crates.io (from ably/vcdiff-rust).Docs
README (standard Ably SDK layout), DESIGN.md (architecture), MIGRATION.md
(REST breaking changes), CONTRIBUTING.md.
Reviewing
Large PR — the commit history is structured and self-contained (phases R1–R6,
5.1–5.8, then numbered TASKs), so reviewing commit-by-commit is likely easiest.
🤖 Generated with Claude Code