Pod quota counter - #139
Open
bourgeoa wants to merge 16 commits into
Open
Conversation
added 16 commits
August 15, 2026 19:11
Per-write pod quota on the CSS file backend recursively walks the whole pod per stream chunk (chunks × O(N)), causing extreme slowness and memory exhaustion on large inboxes. This adds two pivot components wired via config Overrides (no CSS fork): - DuSizeReporter: SizeReporter measuring apparent bytes (du -sb / BSD -s -A -B 1) with a per-path TTL cache, Node-walk fallback when du is unavailable (Windows), and invalidate() to drop resource + ancestor (pod root) cache entries after writes. - FastQuotaStrategy: extends PodQuotaStrategy; computes available space ONCE per write and only tracks the write's own byte delta per chunk (was: full pod walk per chunk). Config: config/storage/backend/quota-fast-file.json overrides urn:solid-server:default:SizeReporter + QuotaStrategy (70 MB, apparent bytes, ignoreFolders ^/\.internal$, ttl 5000); imported from customise-me.json. Benchmark (5000 files/1KB pod, 4MB write in 64KB chunks): guard 36 279 ms -> 3.3 ms (~11 000x); cached getSize 0.13 ms. Equivalence proof: 12/12 random trees byte-identical to FileSizeReporter (du path + Node fallback). Includes unit tests and scripts/benchmark-quota.js + scripts/verify-size-equivalence.js.
Moved from docs/ (which is gitignored) to the repo root so it is tracked. Documents the problem (per-chunk pod walks), the A+B design (DuSizeReporter + FastQuotaStrategy), the decisions (apparent bytes, persist-per-write, A+B first), staleness/recovery (SS4.8), and the verification section SS7 (benchmark ~11 000x guard speedup + 12/12 byte-identical equivalence).
Steady-state quota writes become O(1): a per-pod counter is updated by a delta hook and read by the strategy; a full du/Node walk happens only once per pod (bootstrap/recovery). - QuotaCounter: in-memory Map + per-pod mutex, sidecar <podRoot>/.internal/pivot-quota.json (atomic temp+rename per delta), lazy recount via uncached DuSizeReporter, pod-root mtime staleness check. - IncrementalSizeReporter: replaces SizeReporter — pod root = O(1) counter read, other resources = single stat. - QuotaDeltaDataAccessor: PassthroughDataAccessor wrapping the top of the accessor chain; before/after size on writeDocument/writeContainer/ writeMetadata/deleteResource → counter.add(pod, Δ); pim:Storage pod discovery (mirrors PodQuotaStrategy) cached per path; pod-root delete drops the counter. - Config quota-counter-file.json: QuotaCounter instance + Overrides for SizeReporter, FileDataAccessor (preserving the content-length filter) and QuotaStrategy (FastQuotaStrategy, 70MB). Imported by customise-me.json. No CSS fork. - Tests: QuotaCounter / IncrementalSizeReporter / QuotaDeltaDataAccessor (delta test asserts counter == real walk through create/overwrite/meta/ delete). All green. - Benchmark (scripts/benchmark-quota-c.js), cold vs warm, 5k files: old guard 36 660 ms -> A+B warm 32.8 ms -> C warm 3.38 ms (O(1), flat; at 10k: old 108 670 ms vs C warm 10.6 ms on WSL1). smoke-design-c.js verifies the compiled dist without jest. - POD-STORAGE-QUOTA.md: new section 8 (design C implementation + verification).
ResourceIdentifier.path is the full canonical URL (e.g. https://pivot-test.solidproject.org:3000/.internal/...), not a bare path - identifier strategies test it against URL regexes. The previous startsWith('/.internal/') check never matched, so quota hooks still ran the pod-discovery + size walk on internal writes (IDP AuthorizationCode store), blowing the WrappedExpiringReadWriteLocker 6s expiry on pivot-test. Extract the URL pathname before comparing; works in both suffix and subdomain modes.
…as never active on pivot-test prod.json imported css:config/storage/backend/pod-quota-file.json (the standard CSS quota: per-chunk pod walks, no /.internal exemption), so all design-C fixes (QuotaDeltaDataAccessor + FastQuotaStrategy + InternalPath) were correct but never loaded in production. Only the dev scripts passed customise-me.json (which imports quota-counter-file.json) as a second config. This is why the IDP lock-expiry appeared subdomain-only: pivot-test (prod.json, big pods, standard quota) vs local (dev + customise-me, design C, small pods). Add pivot:config/storage/backend/quota-counter-file.json to prod.json imports so production runs design C.
…fix as prod.json)
…config was never active on pivot-test" This reverts commit bf46bce.
…-expiry fix as prod.json)" This reverts commit 13c1882.
…in customise-me.json) The IDP AuthorizationCode container accumulated ~3900 stale codes because the hourly WrappedExpiringStorage cleanup lists the container under a 6s WrappedExpiringReadWriteLocker (prod uses FileSystemResourceLocker on a busy disk); the listing exceeded 6s, cleanup aborted, codes accumulated, listing got slower — self-reinforcing loop (2026-08-16 incident). long-expiry.json keeps the same file-based locker but raises expiration to 30s so a slow disk / large internal container can't abort cleanup. Wired via customise-me.json (single source, same as quota-counter-file.json). NOTE: clean the existing stale codes on the server (find ... -mtime +1 -delete) and restart; the override prevents recurrence.
…est) CSS's PodQuotaStrategy.searchPimStorage tests isRootContainer() BEFORE reading metadata. In subdomain mode every pod root IS a root container, so discovery bailed with 'no pod' — no counter sidecar, no delta updates, and quota unlimited (an upstream CSS limitation too). Shared PodDiscovery helper reads metadata first, returns the pod when it has pim:Storage, then falls back to the root-container stop. Used by QuotaDeltaDataAccessor and FastQuotaStrategy.getAvailableSpace (replicates QuotaStrategy semantics: pod total minus overwritten resource size). Verified: subdomain smoke — pod registered true, sidecar created, getAvailableSpace = limit - used; suffix smoke-design-c ALL CHECKS PASSED.
…uotaStrategy mocks QuotaDeltaDataAccessorSubdomain.test.ts verifies the 66f0485 fix: in subdomain mode the pod root IS a root container, and discovery must read its pim:Storage metadata before the root-container stop (CSS's searchPimStorage bails early, so pods were never found / counters never created). Asserts pod registration, counter == real walk, and the sidecar file. FastQuotaStrategy.test.ts: getAvailableSpace no longer uses getTotalSpaceUsed (it now discovers the pod directly), so the old FixedTotalStrategy subclass was dead code and the mocks crashed ts-jest. Rewrote as a createStrategy helper with mocked discovery + reporter (pod vs resource sizes), and the no-pod case for unlimited space. All 4 quota suites pass in WSL (QuotaCounter staleness test is flaky on /mnt/d directory mtimes — passed on rerun, unrelated to this change).
…hout @types/jest On the test servers ts-jest type-checked test files against tsconfig.json (which includes only src) and failed every suite with 'Cannot find name describe/beforeEach' when @types/jest (a devDependency) wasn't installed. isolatedModules: true makes ts-jest transpile-only — no hard type-check of test globals — and also silences the TS151002 hybrid-module warning.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.