feat(fs): add pipelined multipart upload#2723
Open
thetapilla wants to merge 4 commits into
Open
Conversation
- Reassemble concurrently uploaded chunks into a sequential stream through a ring file, bounding disk usage to slots*chunkSize per session - Park writers up to a deadline when their slot is busy instead of rejecting instantly, so flow control does not surface as connection errors in browsers - Record a per-chunk CRC32 table for re-fill verification and keep it readable after close - Propagate cancellation to blocked readers via CloseWithError so drivers treat aborts like canceled requests - Cover ordering, backpressure, idempotent resends and close/abort wake-ups with race-enabled tests
- Start the driver upload at session init over a sequential stream backed by the window, so client-to-server and server-to-storage transfers run concurrently - Attach client-provided hashes to the stream so drivers can attempt rapid upload before any chunk arrives, and absorb chunks racing pipeline completion idempotently - Keep only metadata and chunk CRCs after a failed attempt: re-sending chunk 0 re-fills a fresh window, and content changes between attempts are rejected - Resume receiving sessions only when client hashes prove the same file; failed_retriable sessions resume unconditionally - Reclaim sessions with a sliding-TTL GC and sweep orphaned ring files at startup - Cover the state machine with race-enabled tests over a stubbed storage layer
- Add multipart_enabled and multipart_chunk_size (MB) as public traffic settings - Validate the chunk size on save (integer within 1-90) via the setting item hook
- Add /api/fs/multipart init/chunk/complete/status/abort endpoints with headers aligned with /fs/put - Gate init behind the FsUp permission checks and reuse the client upload rate limiter for chunk uploads - Drain the request body before answering chunk requests on every path, so browsers do not see early responses as network errors - Start the multipart session GC when the router is initialized to reclaim ring files orphaned by a previous run
25 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary / 摘要
实现网页端分块上传（multipart upload），解决经 CDN 反代（如 Cloudflare 免费计划 100MB 请求体上限）时无法上传大文件的问题，并为上传链路补上断点续传与失败重试能力。
用户可感知的变化：
Multipart：客户端按分片并发上传，单个请求体可控制在 CDN 限制之内（分片大小可配，默认 10MB）multipart_enabled（功能开关）、multipart_chunk_size（分片大小 MB，保存时校验 1–90）实现要点：
op.Put启动驱动上传，客户端→服务端与服务端→存储两段并行，分别占用服务器下行/上行带宽FileStreamer，与现有/fs/put直传对驱动完全同构，87 个驱动无需任何适配（已在本地/S3/WebDAV/百度/123 五种驱动形态上实测）关于与 As-Task 的关系：本 PR 中 multipart 有意不支持"添加为任务"。As-Task 的语义是"先收完、入队、稍后传"，必然要求文件全量落盘，与流水线形态（收片即传、磁盘占用窗口封顶、秒传免传后续分片）在根本上互斥；且 multipart 会话自身已具备任务队列的核心价值（进度可查、失败可断点重试）。若社区确有"multipart + As-Task"的需求（全量落盘后再推远端，形态接近 #1877 的 store-and-forward 方案），可以作为后续独立 PR 在本 PR 的会话框架上补充，实现难度低于本 PR。
兼容性说明：不改动任何既有端点与驱动接口；新增
/api/fs/multipart/{init,chunk,complete,status,abort}五个端点（header 风格与/fs/put对齐）；新增两个 PUBLIC 设置项；会话仅存内存（与 Upload 任务不持久化的既有决策对齐），重启后未完成会话失效、暂存自动清扫。/ 此 PR 包含破坏性变更。
/ 此 PR 修改了公开 API、配置、存储格式或迁移行为。
/ 此 PR 需要关联仓库同步修改。
Related repository PRs / 关联仓库 PR:
Related Issues / 关联 Issue
Relates to #460
说明：#460 同时提出了网页端与 WebDAV 两个方向的分块上传，本 PR 覆盖网页端；WebDAV 分块（如 Nextcloud 兼容协议）不在本 PR 范围内，但服务端重组器与协议无关，后续可将 WebDAV/Nextcloud 入口作为它的另一个前端接入。
致谢：感谢 #1877（@dnslin）对该方向的探索，以及 @KirCute 在该 PR 评审中提出的两点要求——"开始上传时即构建 FileStream 调用驱动（两段带宽并行）"与"首个请求携带文件哈希触发秒传、命中后免传后续分片"。本 PR 为独立的从零实现，上述两点均为本设计的出发点并已实现（哈希放在 init 请求而非首个分片，语义等价且更早：秒传判定可在任何分片到达前完成）。
Testing / 测试
测试平台：macOS arm64，Go 1.26.4。
go test ./...本机环境因缺少 macFUSE 头文件无法构建 internal/fuse（存量平台问题，与本 PR 无关），故未跑全仓测试；实际执行为下述包级测试与主二进制构建。
自动化测试：
go test -race -count=1 ./internal/multipart/：32 个用例全部通过，覆盖乱序/并发写入、背压停靠与唤醒、幂等重发、秒传与在途分片的竞态（含确定性竞态构造与变异验证）、窗口关闭与完成裁决之间的间隙、CRC 重灌一致性、会话状态机全迁移、GC 过期与阻塞读者唤醒gofmt -l对全部改动文件零违规；go build .主二进制构建通过跨驱动实测：
浏览器手动验证（配合前端配套分支联调）：覆盖已有文件 × 尝试秒传四种组合、刷新页面后重拖续传、3×3 大文件并发上传、失败行重试按钮断点续传、管理端分片大小非法值拒绝提示。
Checklist / 检查清单
/ 我已阅读 CONTRIBUTING。
/ 我确认此贡献符合仓库许可证、贡献规范和行为准则。
gofmt,go fmt, orprettierwhere applicable./ 我已按适用情况使用
gofmt、go fmt或prettier格式化变更代码。/ 我已在适用情况下请求相关维护者或代码所有者审查。
AI Disclosure / AI 使用声明
/ 此 PR 包含 AI 辅助内容。
Tools used / 使用工具:
Usage scope / 使用范围:
Code generation / 代码生成
Refactoring / 重构
Documentation / 文档
Tests / 测试
Translation / 翻译
Review assistance / 审查辅助
I have reviewed and validated all AI-assisted content included in this PR.
/ 我已审核并验证此 PR 中的所有 AI 辅助内容。
I have ensured that all AI-assisted commits include
Co-Authored-Byattribution./ 我已确保所有 AI 辅助提交都包含
Co-Authored-By归属信息。I can reproduce all AI-assisted content included in this PR without any AI tools.
/ 我可以在没有任何 AI 工具的情况下重现此 PR 中包含的所有 AI 辅助内容。