Implement custom storage for orgs - #2093
Open
tw4l wants to merge 69 commits into
Open
Conversation
tw4l
force-pushed
the
issue-578-custom-storage
branch
from
October 1, 2024 15:24
df5b6e9 to
10ab1b6
Compare
tw4l
force-pushed
the
issue-578-custom-storage
branch
9 times, most recently
from
October 17, 2024 15:40
f226271 to
eb065b6
Compare
This was referenced Oct 17, 2024
tw4l
marked this pull request as ready for review
October 17, 2024 16:34
tw4l
force-pushed
the
issue-578-custom-storage
branch
from
December 3, 2024 21:54
c5e88e3 to
a867411
Compare
tw4l
force-pushed
the
issue-578-custom-storage
branch
from
December 30, 2024 15:41
ee2b74d to
c889664
Compare
tw4l
force-pushed
the
issue-578-custom-storage
branch
from
January 24, 2025 14:57
c889664 to
74c8fd6
Compare
Member
Author
|
Rebased on latest main, will need testing to ensure the several rounds of rebases didn't cause any unintended issues |
tw4l
force-pushed
the
issue-578-custom-storage
branch
from
February 24, 2025 19:35
8fbc995 to
92ecc3f
Compare
Member
Author
|
@ikreymer Will need testing after multiple rounds of rebases but this is now rebased on latest main |
tw4l
force-pushed
the
issue-578-custom-storage
branch
from
April 22, 2025 16:15
0b6fc9d to
0b451ab
Compare
tw4l
commented
Apr 29, 2025
- Set access_endpoint_url to the endpoint url with bucket so that we can generate a presigned URL as expected - Make adding bucket in verify_storage_upload a backup routine after first exception is raised
tw4l
force-pushed
the
issue-578-custom-storage
branch
from
April 29, 2025 20:54
a0c20fe to
3a65fee
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #578
Adds
Notes
Currently, no delete operations happen for a a bucket previously used as a primary or replica location that is unset. Files are copied to the new bucket to ensure there are no usability issues moving forward in the app, but the files are not automatically deleted from the source after the copy job. We could add that but I wonder if it's safer, especially in the early days of testing, to perform that cleanup manually as desired.
Once we're comfortable, we can change the rclone command in the
copy_job.yamlbackground job template fromcopytomoveif we want it to automatically clean up files from the source location on completion.Since the same template is used for copying files from an old primary storage to a new primary storage as well as to replicate from primary storage to a new replica location, we'd want to make sure the latter still uses
copyso as not to delete files from the primary storage location, perhaps through use of a conditional.