feat(net)!: network modules — HTTP client/server + WebSocket client, C/Rust cores, ESP-IDF host, TLS - #299
Open
HalfSweet wants to merge 25 commits into
Open
feat(net)!: network modules — HTTP client/server + WebSocket client, C/Rust cores, ESP-IDF host, TLS#299HalfSweet wants to merge 25 commits into
HalfSweet wants to merge 25 commits into
Conversation
HalfSweet
force-pushed
the
feat/network-v2
branch
3 times, most recently
from
August 18, 2026 16:09
2fa7f8e to
62e5843
Compare
HalfSweet
marked this pull request as ready for review
August 19, 2026 02:21
…odule specs - net.ts: ops 6-9 (readInto/limits/write/endBody), retire op 2, streaming headers/readable/end/error events, portable ceilings, the shared error vocabulary with the four tls_* codes and NetworkError categories - ws.ts: WebSocket Client ops/events/limits (globalThis.ws) - httpd.ts: HTTP Server ops/events/limits (globalThis.httpd) - gen-rust.ts emits net/ws/httpd modules; new gen-c.ts emits the C mirror engine/net/include/pocketjs/net/spec.h, both under the drift guard
@pocketjs/framework/net is the support module (NetworkError, AbortController, AbortSignal, URL, getNetworkLimits, shared types); net/http provides fetch, Headers, Request, Response, BodyStream and serve over the v2 net/httpd namespaces; net/websocket provides connect over ws. A Network Guest Binding per module drains one poll per tick from the service pump, settles Promises and calls handlers inside that pump, and copies bodies through readInto. hosts/sim gains deterministic net/httpd/ws hosts; tests cover delivery order, streaming, backpressure, cancellation, error mapping and refusals.
mountHeadless() installs a frame handler that runs the fixed prefix of the frame transaction — virtual clock, service pumps, effect delivery, app hook — without a UI root, for display-less hosts and the network smoke firmware.
Browser fetch stays the transport; the adapter now speaks the v2 ops (readInto/limits), keeps a bounded per-handle receive queue with reader backpressure, freezes readable watermarks at beginFrame(), and maps hidden redirects to unsupported per the browser profile.
engine/net is the native implementation of the v2 module boundaries in portable C99: platform/driver interfaces, the shared async runtime (bounded allocator accounting, tick queues with readable-before-terminal ordering and per-tick budgets, connections, resolve+connect dialer with per-address policy checks), strict HTTP/1.1 head/body parsing, URL and policy handling, and the HTTP Client core behind pnet_http_* (redirect policy, timeouts, backpressure through the receive queue, readInto). Includes a BSD-socket driver shared by POSIX hosts and lwIP, and a unit-test binary under ASan/UBSan.
pnet_httpd_* implements listen/stop, bounded accept, strict request parsing (400/413/414/431/503 refusals), request-body streaming into a bounded queue, respond/write/endBody with keep-alive, chunked streaming, HEAD discard, Expect: 100-continue, handler/idle/keep-alive deadlines, peer-disconnect and abort reporting, and graceful stop. The host test drives both cores over real loopback sockets against scripted peers.
pnet_ws_* implements the RFC 6455 client: upgrade handshake with accept-key, subprotocol and extension checks, masked client frames, unmasked-only server frames, fragment reassembly with incremental UTF-8 validation, native pong replies, bounded receive/send queues with drain, 1002/1007/1009/1013 local closes reported as error → close, close handshake with a deadline and terminate. The host harness gains a scripted WebSocket peer covering echo, fragmentation, control frames, limit and protocol violations, transport loss, handshake refusals, terminate and backpressure.
…nd Tab5 - pocketjs_net_core: engine/net + the lwIP/BSD driver as an IDF component - pocketjs_esp_host: guest owner task (PSRAM heap, fixed-rate frame ticks with begin_tick, job drain), globalThis.net/ws/httpd bindings, network task under one runtime lock, wind-down frames on stop - pocketjs_board: STA + DHCP for the AtomS3R (native) and Tab5 (C6 over SDIO via esp_hosted/esp_wifi_remote, WLAN rail on the IO expander) - examples/net-smoke: headless smoke app + firmware template (rev 1.3 and hosted sdkconfig), tools/net-peer.ts as the independent workstation peer - core fixes found on hardware: httpd end is a readable barrier, connect failures map to connect, WebSocket keeps reading after a local close and ignores data after its Close frame, deferred write shutdown Both boards pass 20/20 against the peer and each other.
NetCore now speaks spec v2: HttpClientBackend reports streaming Headers/Body/End/Error completions, the core keeps per-handle bounded receive queues with backpressure hints, freezes readable watermarks at begin_tick (inserted before the handle's barrier event), applies the per-tick event/byte budget, serves readInto and limits(), enforces the connect policy and the shared error vocabulary; the mount feature installs the six v2 ops through rquickjs. Fixture-backed tests cover ordering, budget truncation, cancel/late completions, refusals and the mount.
…bilities docs/NET.md and site/content/docs/net.md now document the v2 modules (net/http, net/websocket, the support module), delivery at frame boundaries, streaming bodies, errors, limits, ownership and testing; RUNTIMES.md and concepts.md follow. contracts/spec/platforms.ts replaces net.http with the role-split network.* capability ids (no stock target advertises them).
globalThis.net comes with the policy; ws and httpd are mounted only when the host admits those roles (both default on for the smoke).
The core gains a TlsProvider interface (pnet_tls_ops) and pnet_runtime_create_tls: https:/wss: are accepted and the "tls" feature is advertised only when a provider is present. The connection layer runs a non-blocking client handshake between the plain connect and reporting open, routes application I/O through the session, owns the connect deadline and cancellation, and fails closed with tls_clock_untrusted before any I/O when the platform reports the wall clock untrusted. SNI equals the authorized hostname and is the DNS-ID/IP-ID the certificate must match; a plaintext fallback never happens. engine/net/drivers/openssl implements the reference NativeTlsProvider (TLS 1.2 min, renegotiation/tickets off, verify failures mapped to the four stable tls_* codes). A new tls conformance harness stands up an in-process OpenSSL PKI and HTTPS/WSS peers covering a valid chain, unknown CA, expired cert, hostname mismatch, no plaintext fallback, an untrusted clock, the development-insecure refusal, and a WSS echo.
pocketjs_net_esptls wraps ESP-TLS + the IDF certificate bundle as a pnet_tls_ops over the driver's connected lwIP socket (non_block=false so the Mbed TLS handshake progresses asynchronously without ESP-TLS owning the connect). The host creates the runtime with pnet_runtime_create_tls when network_tls is set and exposes a wall-clock-trusted hook backed by time(); the board syncs SNTP before the guest starts. The smoke gains a TLS suite against a public host plus badssl.com's expired/wrong-host/self-signed/ untrusted-root endpoints. Both boards: 26/26 (20 plaintext + 6 TLS). HTTPS/1.1 to a public host with a valid chain succeeds; every bad certificate fails closed. docs/NET.md and the esp-idf README document the TLS boundary and the gate.
The network architecture and guest-boundary v2 design notes are internal artifacts and are no longer committed. Point the API docs (docs/NET.md, site/content/docs/net.md) at the in-repo pinned specs (contracts/spec/*), and rewrite the source-comment citations to be self-describing: each now states the rule it used to reference by section number, with no dead repo-relative path. Also correct the ESP-TLS provider header, which described a socket-fd duplicate the implementation does not make.
The Phase 1A/1B/1C staged-gate terminology came from the internal design notes that are no longer committed. State what each module actually delivers and the plain rule that a target advertises a capability only when its native host ships and tests the module, instead of naming rollout phases. No stock target advertises the network capabilities yet.
… 3, ResolvedNetworkPolicy, host inputs contracts/spec/network-policy.ts fixes the typed policy contract (connect / listen rules, credentials, localNetwork, insecureTransport, dev-TLS switch), its normalization into a canonical ResolvedNetworkPolicy (version 1), the reference matcher (connect / address classification / listen) and the canonical JSON a host hands to its core. Manifest format 3 adds the top-level permissions.network block (pocket-3.json; format 2 stays valid and resolves to the deny-all policy). The resolver writes the policy into ResolvedBuildPlan.network so planHash covers it; extractHostBuildInputs projects planHash, features and the canonical policy JSON (POCKETJS_NETWORK_POLICY) to custom hosts. The C parser now accepts exactly the shapes the reference produces (version check, A-label hostnames, no bare wildcard, no wildcard over an IP literal, no leading-zero IPv4 octets) and both run the shared vectors in contracts/spec/vectors/network-policy.json. pnet_json records container source spans so a sub-document can be handed to another parser. Also: framework/src/net/url.ts carried raw NUL/DEL bytes inside a regex (git treated the file as binary); they are escapes now.
…ed vectors on every implementation NET_METHODS_FORBIDDEN now carries TRACK; HTTP_CORE_OWNED_REQUEST_HEADERS, HTTP_BODYLESS_STATUS (RFC 9112 framing), HTTP_NULL_BODY_STATUS (Fetch null body), HTTP_REDIRECT_STATUS and the redirect rewrite table join it. gen-c / gen-rust emit them; the new gen-web emits hosts/web/net-spec.js so the browser dev host (plain ESM) stops re-typing the ceilings and rules, under the drift guard. The SDK, the sim, the browser host, the C client/server (pnet_http_redirect_plan, pnet_status_is_bodyless/null_body) and the Rust core drop their private copies (TRACK hardcodes, owned-header lists, the 205 server drift). contracts/spec/vectors/http-semantics.json pins method acceptance, owned headers, status classes and redirect rewrites; the TS test, the sim, the browser host and pnet_unit_test run it. The sim method check is case-insensitive like the cores; the sim hosts also enforce an optional Build Plan policy (connect/listen/redirect re-check) through the contract matcher.
…for the breaking net migration teeBody sizes every pull to the remaining room (16 KiB at most, never past the aggregate limit), so a clone's lagging branch is a hard bound, not 'stop after crossing'; a test drives 3 KiB through a 2 KiB window. The browser dev host reads through a BYOB reader sized to the queue's free space where the body is a byte stream (the receive queue is then a hard queueBytes cap like a native core's) and documents the one-chunk overshoot of the default-reader fallback. runFramePrelude (frame-prelude.ts) is the single definition of the frame prefix — virtual clock → input latches → service pumps → effects — used by the Solid, Vue Vapor, Octane and headless entries instead of four hand-typed copies. The changelog gains an Unreleased entry that names the net module change a breaking migration (net → net/http, NetError → NetworkError, net.http → network.*) and introduces manifest format 3.
…nsumes nothing pnet_queue_poll used to dequeue and free each visible event while appending it to the batch buffer and only then notice the string builder had failed, returning NULL: under the heap cap (1 MiB on the ESP host) a tick's events — including a handle's terminal end/error — vanished, the core retired the handle, and the guest's Promise or BodyStream waited forever. The poll is now render + consume: the batch size is computed from the visible events, reserved in one allocation (the stale buffer is released first rather than held beside its replacement), and only a rendered batch is consumed. An allocation failure leaves the whole visible set in place and the next poll retries; the episode is logged once. pnet_*_poll_render/consume expose the two phases so a host that marshals the batch into a guest value (the ESP bindings, JS_NewStringLen) consumes only after the guest holds its copy. Unit tests pin the deferred batch, idempotent render, consume-exactly-the- rendered-events with a freeze in between.
…task The POSIX driver resolved names inside pnet_posix_driver_wait(), i.e. on the host's single network task: one slow or unanswered lookup froze every socket, the HTTP server accept loop, the WebSocket traffic and the core's deadline machinery — so the spec's promise that connectMs covers DNS could not hold. Lookups now run on the driver's own worker (a pthread on desktop, the 'pnet-dns' task on ESP-IDF) fed by a signal; the worker wakes the network task when results are ready and dispatch() hands them to the runtime as before. resolve() and resolve_cancel keep their contract; if the worker cannot be started the driver falls back to inline resolution. The host test drives a hostname through the worker, checks NXDOMAIN reports dns, and shows a literal-address exchange completing while a lookup is in flight.
… real clock trust, plan-derived policy Cadence: tick k is scheduled at t0 + k/tick_hz on the microsecond timer instead of pdMS_TO_TICKS(1000/60) = 16 ms (62.5 Hz against a 60 Hz __simHz); late ticks still get their one turn (Law 3), and only a host more than 0.5 s behind drops ticks (stats.frames_skipped). On the AtomS3R the host now reports 1800 frames per 30.0 s. Lifecycle: start() unwinds everything it created on any failure (the TLS provider included, the network task joined when the guest task fails); stop() publishes a waiter, sets stopping, waits for both tasks' exit flags (notification + poll) and frees only then — a wedged task is leaked with an error, never freed under; a QuickJS interrupt handler bounds a guest turn in progress while stopping (stop_turn_budget_ms); NULL ctx/rt guards on the failed-boot path; guest_boot_failed and the plan hash appear in stats. Clock trust: plat_clock_trusted delegates to cfg.wall_clock_trusted (none = never trusted = TLS fails closed); the board layer latches the state from a completed SNTP sync (and every re-sync) or an explicit setter, instead of time(NULL) > 2024. Policy: the smoke firmware embeds the canonical ResolvedNetworkPolicy of its Build Plan. hosts/esp-idf/examples/net-smoke/pocket.json is a format-3 manifest; tools/esp-idf.ts (run by main/CMakeLists.txt) merges the rig's Kconfig endpoints, resolves the plan against the private atoms3r-dev / tab5-dev profiles (tools/esp-idf-profile.ts), and writes network-policy.json, host-inputs.h (plan hash, features → mounted roles) and app.js; main.c authors nothing. The ESP bindings use the two-phase poll. AtomS3R: rebuilt, flashed, SMOKE PASS 26/26 with the plan-derived policy and TLS under the SNTP-latched clock trust.
…backends, shared vectors NetPolicy moves to policy.rs and parses exactly the canonical ResolvedNetworkPolicy (version, connect + listen rules, credentials, the switches; A-label hostnames, no bare wildcard, no wildcard over an IP literal, no leading-zero octets) with the reference matcher: allows_connect, allows_address (the public/private/multicast classification the C core and the TypeScript reference use) and allows_listen; localNetwork is enforced. permissive() is an explicit loopback/*.test policy, no '*'. The backend no longer decides anything on the wire side: start() hands it a PolicyGate (clone-cheap, Send + Sync) whose authorize_address must gate every resolved candidate, whose authorize_redirect applies the spec's followed statuses, method/body rewrite table, hop budget, scheme/TLS availability and the endpoint policy of the target (recording each hop), and whose tls_verification fixes the verification mode. The core classifies literal addresses itself (asynchronous permission_denied, like the C dialer) and rejects a response whose URL is not the last hop the gate authorized for that handle (or whose redirected flag disagrees). Tests run contracts/spec/vectors/network-policy.json and http-semantics.json — the same files pnet_unit_test and the TS suite run — plus the gate and the URL check.
…oll, hard bounds, clock trust; publish pocket-3.json
HalfSweet
force-pushed
the
feat/network-v2
branch
from
August 19, 2026 15:38
ebef603 to
cbd77ba
Compare
HalfSweet
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.