Imfs contiguous for mmap#166
Open
rennergade wants to merge 61 commits into
Open
Conversation
…-mmap'd files Precondition for the mmap(MAP_SHARED) plumbing: imfs needs a regular-file storage shape whose backing is a single contiguous host page range (SYS_MMAP'd, ANON|SHARED), rather than the existing chunk-chain. That's the only shape RawPOSIX can alias into a cage's vmmap when the cage mmaps the file, and it's what makes multi-cage MAP_SHARED 'work for free' — every cage that maps the file ends up aliased onto the same host pages. Adding it as a sibling variant rather than morphing Reg keeps the existing chunk-based code path untouched, so this change is zero-impact for current callers. Follow-ups will (1) wire the mmap handler to recognize RegMapped fds and route the cage's mmap call back through SYS_MMAP with MAP_FIXED at the stored host_addr, (2) implement read/write/ftruncate against the host mapping, (3) decide where the RegMapped vs Reg choice gets made at create time (path-based config in the routing layer is the leading candidate).
…Mapped nodes
Two private helpers on Filesystem so RegMapped nodes can grow / release
their host page range without callers needing to know how to invoke
threei.
ensure_mapped_backing(node_idx, min_capacity):
- No-op if current capacity already covers the request.
- Refuses with EBUSY if mmap_refs > 0 and a grow is needed (moving
the region would silently invalidate cages that still have it
mapped — better to fail than to corrupt).
- Otherwise rounds min_capacity up to a page, allocates a fresh
MAP_ANONYMOUS | MAP_SHARED region via make_threei_call(SYS_MMAP),
copies the live bytes from the old region (anonymous mappings
start zeroed so no tail-fill is needed), and releases the old
region via SYS_MUNMAP.
free_mapped_backing(node_idx):
- Releases the host mapping via SYS_MUNMAP when the RegMapped node
is being reclaimed. Debug-asserts mmap_refs == 0 to surface
refcount bugs early rather than silently leaking host pages.
Both go through make_threei_call so RawPOSIX is the one allocating /
freeing — gives us real host kernel pages, which is what later
RegMapped mmap-routing will alias into the calling cage's vmmap with
MAP_FIXED. Nothing calls these yet; wiring read/write/ftruncate and
the mmap handler are the next commits.
…directly read_from_node, write_to_node, and truncate_node now have a RegMapped fast path that runs before the chunk-walking code. - read: single copy_nonoverlapping out of `host_addr + offset` for `min(buf.len(), total_size - offset)` bytes. No chunk walk. - write: ensure_mapped_backing(end_offset), then a single copy_nonoverlapping into the mapping. Anonymous backing means any hole between the old total_size and the write offset is already zeroed. If the grow fails (EBUSY because cages still have the region mapped, or ENOMEM), return 0 — caller surfaces as short write / ENOSPC. - truncate: grow → ensure_mapped_backing then update total_size. Shrink → just update total_size; we deliberately do not munmap the tail so any live cage mapping past the new EOF stays valid. The chunk-based paths for plain `Reg` are untouched. RegMapped nodes still aren't created anywhere (that's the next decision — path-based config in the routing layer, or an explicit choice in the imfs grate's config), so this commit is a no-op at runtime; it just makes the storage code ready for a RegMapped node to actually exist.
…o cage vmmap
This is the wire-up that makes `mmap(RegMapped_fd, MAP_SHARED, …)`
actually map the file's host-backed pages into the calling cage.
ImfsState::mmap(cage_id, addr, len, prot, flags, fd, off):
- Refuses anonymous mappings (MAP_ANONYMOUS / fd == -1) — handler
forwards those to RawPOSIX directly.
- Looks up fd → node. Only RegMapped is handled; plain Reg gets
-ENODEV (its chunk-chain backing isn't contiguous and isn't
suitable for being aliased into a cage vmmap).
- Ensures the host region covers off+len via ensure_mapped_backing.
- Issues make_threei_call(SYS_MMAP) targeted at the calling cage
with MAP_FIXED | MAP_ANONYMOUS | MAP_SHARED at `host_addr + off`.
MAP_FIXED tells RawPOSIX to alias those exact host pages into
the cage's vmmap — meaning every cage that maps the same file
shares physical pages, which is what gives MAP_SHARED its
cross-cage write-visibility semantics (postgres' dyn-shm path
needs this).
- Bumps mmap_refs and records (cage, mapped_vaddr) → (node_idx,
len) so munmap can find it.
ImfsState::munmap(cage_id, addr, len):
- Always forwards to RawPOSIX via SYS_MUNMAP so the cage's vmmap
entry is torn down, regardless of whether we know about it.
- If we tracked the address, decrements the node's mmap_refs so
later grows / frees aren't pinned by a stale refcount.
handlers.rs gains mmap_handler / munmap_handler, both wired into
GrateBuilder in main.rs. The mmap handler also handles the
forward-to-RawPOSIX case (when imfs returns -ENOSYS to mean 'not
mine').
Also broadens NodeType::Reg checks in dirent type, open/O_TRUNC,
truncate, and ftruncate to accept NodeType::RegMapped — same
semantics, different storage variant.
Still no RegMapped nodes get created anywhere; that's the next
commit (path-based config / startup-flag decision).
No upfront decision needed about which files will be mapped; every regular file starts cheap (chunk-chain) and is promoted in-place on the first mmap call that targets it. Adds promote_to_mapped(node_idx): allocates a host MAP_ANON|MAP_SHARED region sized to the current total_size, drains the chunk chain into it in file order, reclaims the chunks, and flips both the NodeInfo variant and NodeType to the mapped form. After promotion the existing host-mapped fast paths in read/write/ftruncate take over — no caller-visible change other than 'this file is now mappable'. Wired into ImfsState::mmap before the existing 'must be RegMapped' check, so the first mmap() on a file is the trigger. Files that never get mapped never pay the promotion cost. This obviates needing a path-based config or an 'all files mapped' startup flag — the right decision is made at the moment we actually know it's needed. Cost: one extra full-file memcpy on the first mmap of each file (the drain), trivial vs the typical workload that follows. The Node::new arm for NodeType::RegMapped is now technically dead (nothing creates RegMapped at construction), but keeping it lets the variant exist for any future paths that want to allocate a node already-mapped.
Three follow-ups to the RegMapped lazy-promotion work, plus tests:
(1) Cage exit cleanup. Adds ImfsState::cage_exit(cage_id) which
walks mmap_tracking and decrements the corresponding mmap_refs
counters for every live RegMapped mapping owned by that cage,
then removes the per-cage cwd and fd-offset entries. Hooked via
new exit_handler / exit_group_handler in handlers.rs (forwards
SYS_EXIT / SYS_EXIT_GROUP through threei after the cleanup),
registered in main.rs. Without this, a cage that exited without
explicit munmap'ing would leak mmap_refs and permanently pin
that file's host region against grows.
(2) reclaim_node frees the host mapping. When a RegMapped node is
being reclaimed (final close after unlink), call
free_mapped_backing first. Without this the SYS_MMAP'd region
leaks every time a mapped file is deleted.
(3) truncate_node shrink path. On RegMapped shrink, zero the bytes
between the new EOF and the old EOF in the host mapping so a
cage with a live mapping past the new EOF doesn't observe stale
data. We can't reach into the cage's vmmap to actually shorten
its mapping (real Linux would SIGBUS post-shrink), so degrade
to zeros — strictly safer than leaving stale contents visible.
Tests added in test/imfs_test.c:
- test_mmap_basic: open → ftruncate → mmap → write through
mapping → read via fd round-trips correctly (also exercises
the lazy Reg→RegMapped promotion path).
- test_mmap_truncate_shrink_zeros: bytes past the new EOF in a
live mapping read as zero, head bytes untouched.
- test_mmap_shared_fork: parent maps with MAP_SHARED, forks, child
sees parent's writes and overwrites, parent sees child's writes
— covers the cross-cage MAP_SHARED-via-aliased-host-pages path
that motivated the whole design.
- test_mmap_refcount_release: mmap → munmap → ftruncate grow
succeeds, indirectly checking that munmap properly decrements
mmap_refs (a leak would surface as EBUSY here).
write_to_node panicked at `data[used..local_offset].fill(0)` when the write offset was past a freshly-allocated chunk's end. Triggered by truncate_node's grow path on a fresh file: ftruncate(fd, 4096) on an empty Reg file calls write_to_node(.., 4095, &[0]), which allocates one chunk then tries to zero-fill bytes 0..4095 in a 1024-byte chunk. Fix: after allocating (or stepping to) a chunk, if local_offset still exceeds CHUNK_SIZE, mark the whole chunk as zero-used and advance to the next slot. Pre-existing bug, exposed by the new test_mmap_basic which is the first test to grow a fresh file past one chunk.
mmap_handler and munmap_handler were reading the calling cage from arg1cage, but arg1 is the addr pointer — its cage tag can be a transient grate's cage (see open_handler's note), not the caller's. Switch to arg5cage (fd, integer) for mmap and arg2cage (len, integer) for munmap. Also stop discarding the integer args' cage tags in the ENOSYS-forwarding path so the forward replays the real per-arg cages RawPOSIX expects.
The cage-side mmap that aliases imfs's host pages into the calling cage's vmmap was forwarding the caller's flags with just MAP_FIXED added. The caller's flags describe a file-backed mapping (no MAP_ANON, fd = imfs fd), but we're calling RawPOSIX with fd=-1 since the file-backing role is now ours. RawPOSIX/libc rejects fd=-1 without MAP_ANON with EINVAL — which is what the test saw as errno=12 (ENOMEM, our default-to-ENOMEM error path swallowing the real EINVAL). Hardcode MAP_ANON | MAP_SHARED | MAP_FIXED for the cage-side call; the caller's flags are intentionally not propagated to RawPOSIX.
The cage-side SYS_MMAP needs to MAP_FIXED at the *host* page our first SYS_MMAP gave us, not at that offset reinterpreted against the calling cage's base. Mark arg1cage with GRATE_MEMORY_FLAG so glibc's make_threei_call wrapper translates target_addr from the grate's wasm offset to a real host pointer (`__lind_base + target_addr`) before RawPOSIX sees it.
Distinguishes which mmap step is failing in the test: -74 EBADMSG = promote_to_mapped failed (first SYS_MMAP, target=grate) -75 ENOTUNIQ = ensure_mapped_backing failed -76 EBADFD = host_addr is still 0 after promote -77 EREMCHG = cage-side SYS_MMAP returned non-syscall threei error Will revert once we know which path is hit.
mmap returns a wasm32 user-space address. find_map_space picks the END of the gap (so on an empty cage it returns ~0xFFFFF000 — the top of the 4GB vmmap). Those addresses look negative as i32, and grate-rs's make_threei_call collapses every negative i32 into Err(MakeSyscallError(n)). We were treating any such Err as ENOMEM, turning a successful high-address mmap into a -ENOMEM failure on the caller side. glibc handles this by only treating values in (-256, 0) — the actual errno range — as errors. Apply the same convention in promote_to_mapped, ensure_mapped_backing, and the cage-side mmap return handling: large-negative i32s are valid user_addrs.
Only the alias / promote-to-mapped flow makes sense for imfs-owned files. An mmap call landing here with a non-imfs fd (kernel fd inherited through a layered grate, IPC fd, etc.) was being treated as an imfs fd: `entry.underfd` would get used as a node_idx and we would silently mangle the call. Gate the imfs path on `fdkind == IMFS_FDKIND`. Anything else (no fdtable entry, or a different fdkind) bounces back as -ENOSYS so the handler forwards the original SYS_MMAP unchanged to the next layer.
stderr-flushed eprintlns at entry / exit / decision points for: - read_handler, write_handler, lseek_handler - mmap_handler, munmap_handler - exit_handler, exit_group_handler, fork_handler - imfs::mmap, imfs::promote_to_mapped, imfs::ensure_mapped_backing So we can see which call lands when the runtime starts truncating output. Drop before merge.
This reverts commit 2955265.
…fd I/O
Path A from PR 1185's design discussion: stop trying to alias the
grate's host pages into the calling cage's vmmap via MAP_FIXED +
GRATE_MEMORY_FLAG. That approach collided with whatever the cage
already had at the derived uaddr (stack, libc data, etc.), silently
corrupting state.
New model:
- mmap forwards to RawPOSIX as plain MAP_ANON | MAP_SHARED (no
MAP_FIXED, no GRATE_MEMORY_FLAG). Runtime picks a free uaddr in
the calling cage's vmmap.
- Seed the new region by copying the file's current chunk bytes
through copy_data_between_cages.
- read_from_node / write_to_node route through the owner cage's
mapping via copy_data_between_cages when any cage has the node
mapped.
- munmap (or cage_exit on an unclosed mapping) syncs the cage's
mapping back to chunks so future fd reads see the latest bytes.
- truncate-shrink on an mapped file zeros the cage's mapping bytes
past the new EOF.
Cross-cage sharing comes from kernel fork-inheritance of MAP_ANON |
MAP_SHARED — parent and child see the same physical pages. Unrelated
cages mapping the same file each get their own region, which is the
explicit tradeoff we took for now (postgres's typical access pattern
is fork from the postmaster, which works).
Leaves promote_to_mapped / ensure_mapped_backing / free_mapped_backing
and the RegMapped variants in place as dead code; will clean up after
the test run confirms the approach.
cage_exit fires from the SYS_EXIT handler, but by that point the cage's address space may already be gone — a copy_data_between_cages read against the cage's mapping uaddrs hits the runtime's range- validation and prints a noisy '[3i|copy] range invalid' to stderr even though the operation succeeds overall (we ignored the error). The sync was best-effort anyway: chunks are stale during the mapped period, and any consumer that wants the post-mmap bytes to persist needs an explicit munmap (which we do sync, before the cage's vmmap entry is torn down). Drop the cage_exit sync and just clear the tracking entries.
Composition: fs-routing-clamp routes /tmp to imfs-grate (no fs-view-
grate, so all cages share the same imfs). Demonstrates the postgres
DSM pattern that single-fork shared mmap doesn't cover:
- parent opens /tmp/dsm_segment, ftruncate to one page, mmap
MAP_SHARED, writes 'parent_init' at offset 0
- parent forks two children (N=3 cooperating cages total)
- each child independently opens the same path and mmaps it again
(does not reuse the inherited mapping — that's how postgres
workers attach to a DSM segment by name)
- each child stamps a child-specific marker at a distinct offset
- parent waits for both and verifies its mapping shows all three
markers — proves all three cages share the same imfs pages
Goal: catch regressions in shared mmap when more than 2 cooperating
cages touch the same imfs-backed page.
Path A handles single-fork shared mmap correctly via kernel-level
inheritance of MAP_ANON | MAP_SHARED: parent and forked child see
each other's writes through the inherited mapping at the parent's
uaddr. The postgres-DSM pattern adds a wrinkle: each child reopens
the file and calls mmap() *itself* instead of reusing the inherited
address, expecting to attach to the existing segment. Path A's
mmap forwards a fresh anonymous mmap on each call, so the child gets
its own region — no sharing, parent never sees the child's writes.
Two small changes recover the demo:
- imfs::fork now also copies the parent's mmap_tracking entries
over to the child (with the same uaddr). That uaddr is valid in
the child too, because Linux fork inherits the parent's vmmap.
- imfs::mmap checks mmap_tracking before forwarding SYS_MMAP. If
the calling cage already has an inherited entry for the node,
hand back the existing uaddr. The cage's memcpy through that
uaddr lands on the inherited shared kernel pages, which is what
the demo (and postgres DSM) needs.
Cages outside the fork lineage still get a fresh anonymous region
(no sharing — same explicit tradeoff path A makes). imfs_test.c
keeps passing 67/67.
Same behavior, less nested pattern-matching.
A/B probe to localize the test_mmap_basic regression seen after 7e46a35. Two changes landed in that commit: 1. imfs::fork copies mmap_tracking from parent to child 2. imfs::mmap returns the inherited uaddr early if cage+node match For test_mmap_basic the parent cage's mmap_tracking is empty (no mmap ran before this test) so both changes should be inert — yet the test segfaults. This commit removes (2) only; if test_mmap_basic now passes, (2) is the culprit; if it still fails, (1) is. The DSM demo will regress for now until we re-add the working piece.
Re-adds the imfs::mmap inherited-uaddr short-circuit and instruments
every mmap-related decision point with eprintln! so we can correlate
test output with what the grate is actually doing.
Traces (all go to stderr, prefixed with [TRACE imfs::...]):
- imfs::open: cage/path/norm/flags -> fd
- imfs::read / imfs::write: cage/fd/buflen
- imfs::read_from_node / write_to_node: when find_active_mapping
routes through an existing mapping (owner cage + uaddr)
- imfs::mmap: ENTER args; fd->node lookup; short-circuit check
(table_size + inherited match); SYS_MMAP forward + ret; mapped
uaddr; seed copy_len + copy_data_between_cages ret; INSERT
- imfs::munmap: cage/addr/len; tracked entry removed
- imfs::fork: parent/child; mmap_tracking inheritance count + each
copied entry
- imfs::cage_exit: cage; each dropped entry; before/after size
Use stderr (eprintln!) so output flushes immediately on segfault —
previous "test_lseek PAS<segfault>" was a buffered-stdout artifact
that hid where the crash actually happens.
Diagnostic only — to be reverted once the cause is identified.
ROOT CAUSE: imfs::mmap stores mmap_tracking entries keyed by the
wasm uaddr returned from SYS_MMAP (e.g. 0xfffb7000), but by the
time the cage's munmap call reaches munmap_handler the runtime has
already translated the pointer arg from the cage's uaddr to the
host sysaddr (e.g. 0x724bbfdb7000). The (cage, sysaddr) key never
matched any stored (cage, uaddr) entry, so munmap was a no-op for
the table.
Visible symptom: a subsequent imfs::mmap on a different file that
happened to be assigned the same reused node slot would trigger the
inherited-uaddr short-circuit, hand back the previous (now-unmapped)
uaddr, and the cage segfaulted on first memcpy through it.
This was confirmed by tracing — after test_mmap_basic:
INSERT (cage=2,uaddr=0xfffb7000) -> (node=16,len=4096). table_size=1
munmap cage=2 addr=0x724bbfdb7000 len=4096
tracked entry removed: None <-- LEAK
then in test_mmap_truncate_shrink_zeros:
short-circuit check: mmap_tracking_size=1 inherited=Some("0xfffb7000")
SHORT-CIRCUIT FIRE -> return uaddr=0xfffb7000
Segmentation fault
FIX: in imfs::munmap, when the direct (cage, addr) lookup misses,
fall back to iterating cage's entries and matching by len. Each
cage has at most one live mapping per node in our use cases, so
(cage, len) is unique enough. Long-term: either disable the
runtime's pointer translation for this arg, or store host_addr
alongside uaddr at mmap time so direct lookup works.
Also adds an arg1cage trace in munmap_handler so we can see whether
the runtime tags the translated pointer with GRATE_MEMORY_FLAG.
Drops the eprintln traces added in 0598360 and the arg1cage trace added in 48ed877. The munmap fallback lookup that resolves the sysaddr-vs-uaddr mismatch stays in place. Net change over the previous passing path A baseline (6f856e8) is three additions: fork-copy of mmap_tracking, the inherited-uaddr short-circuit in imfs::mmap, and the (cage, len) fallback in imfs::munmap. imfs_test passes 67/67 and the DSM demo passes 9/9.
# Conflicts: # rust-grates/imfs-grate/src/imfs/mod.rs # rust-grates/imfs-grate/src/main.rs # rust-grates/imfs-grate/test/imfs_test.c
# Conflicts: # rust-grates/imfs-grate/src/handlers.rs # rust-grates/imfs-grate/src/imfs/node.rs # rust-grates/imfs-grate/test/imfs_test.c
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.