Multi-wacz upload processing - #3323
Open
emma-sg wants to merge 54 commits into
Open
Conversation
emma-sg
force-pushed
the
issue-2814-multi-wacz-upload
branch
3 times, most recently
from
June 17, 2026 14:31
417e564 to
8822375
Compare
emma-sg
force-pushed
the
issue-2814-multi-wacz-upload
branch
4 times, most recently
from
June 23, 2026 19:34
89e579b to
a88251b
Compare
emma-sg
marked this pull request as ready for review
June 23, 2026 20:29
emma-sg
force-pushed
the
issue-2814-multi-wacz-upload
branch
from
June 24, 2026 16:45
a88251b to
548e84d
Compare
emma-sg
force-pushed
the
issue-2814-multi-wacz-upload
branch
2 times, most recently
from
July 6, 2026 18:07
b680d69 to
d952a61
Compare
mistydemeo
approved these changes
Jul 6, 2026
mistydemeo
left a comment
mistydemeo
left a comment
Contributor
There was a problem hiding this comment.
Looking good! Gave it a test and it's working well. UI works nicely too.
tw4l
reviewed
Jul 7, 2026
tw4l
reviewed
Jul 7, 2026
mistydemeo
reviewed
Jul 8, 2026
emma-sg
force-pushed
the
issue-2814-multi-wacz-upload
branch
from
July 8, 2026 20:29
69e5314 to
0bee994
Compare
tw4l
reviewed
Jul 9, 2026
tw4l
left a comment
tw4l
left a comment
Member
There was a problem hiding this comment.
Still testing (especially with large multi-waczs) on dev, but looking great so far! Just left a few comments here.
emma-sg
marked this pull request as draft
July 9, 2026 19:27
tw4l
reviewed
Jul 9, 2026
Member
I think this is the right choice, and that the backend API will already reject attempting to add these items to a collection anyway since they are not yet in a successful state until postprocessing completes anyway. |
emma-sg
marked this pull request as ready for review
July 13, 2026 18:03
emma-sg
force-pushed
the
issue-2814-multi-wacz-upload
branch
from
July 13, 2026 18:10
1bd6cda to
b28bba3
Compare
emma-sg
marked this pull request as draft
July 13, 2026 18:43
Member
Author
|
I think maybe it makes sense to review this separately from the upload speedup changes that are in progress, since they're not really related changes. Should definitely get both in for the same release though! Marking as ready for review. |
emma-sg
marked this pull request as ready for review
July 13, 2026 21:00
- bare `raise` re-raises caught exception, so no need for `raise e` - variable casing fix Co-authored-by: Misty De Meo <mistydemeo@gmail.com>
Co-authored-by: Tessa Walsh <tessa@bitarchivist.net>
this ended up being a bit more in-depth than i first thought, since we need to keep track of the particular file being processed in a list of pre-existing files and replace only the processed file in that list, rather than the whole file
- adds concurrency limit for multi-WACZ file splitting - refactors the database updates to use atomic $pull/$push/$inc operations - makes cleanup best-effort to avoid blocking uploads
mongodb throws an error (`ConflictingUpdateOperators`) if you have both a push and a pull in the same operation
deleting when an upload is deleted during post-processing, it gets marked as deleted in the db and aborts further processing also removes unnecessary `upsert=True` on status updates where the document should already exist
updates `ensure_cron_cleanup_jobs_exist` to instead be generic for various cron jobs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #2814
Changes
This adds an upload post-processing stage that runs after an item is uploaded, either in the same request as the upload if the file is <50MiB or in a background job if the file is larger.
This step inspects the WACZ file and if it find it's a nested file (with multiple WACZ files inside it), it expands the nested WACZs into separate files that are attached to the archived item. This also allows for pages to be correctly ingested for nested uploaded items.
The upload completion webhook is sent after this processing completes, so for larger uploads webhooks may send later than they previously would have.
Also adds a new "upload-processing" archived item state, which displays in most places on the frontend the same way as "uploaded" does, but with a pulsing dot icon & a loading indicator for the page count.
I made the deliberate choice here not to include processing uploads in the archived item selector dialog window for collections, since I believe the page calculation there relies on the archived items being correctly split/having pages in the DB by then.
Testing
The backend changes here are pretty thoroughly integration-tested.
To test manually: