Raise work_mem for the tag swap transaction - #284
Merged
Conversation
Both statements in swap_docker_tags/2 process the whole staged set, which is about 1M rows for hexpm/elixir, and both spill at the server's 4MB work_mem. From one night's Cloud SQL temp file logs, a single swap wrote 391MB: the INSERT ... DISTINCT ON sorts into a 229MB external merge file, and the DELETE's anti-join runs at 64 batches, writing 63 files of 2.58MB. That is most of the 395MB bob wrote in 24 hours. 256MB holds the hash side in one batch and cuts the sort to a single merge pass. Reconcile walks its repos with Enum.each, so only one of these transactions is open at a time on one connection.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Both statements in
swap_docker_tags/2process the whole staged set, about 1M rows forhexpm/elixir, and both spill at the server's 4MBwork_mem.From one night's Cloud SQL temp file logs, a single swap wrote 391MB. The
INSERT ... DISTINCT ONsorts into one 229MB external merge file, and theDELETE ... NOT EXISTSanti-join runs at 64 batches, writing 63 files of 2.58MB. That is most of the 395MB bob wrote in 24 hours.256MB holds the hash side in one batch and cuts the sort to a single merge pass.
Reconcilewalks its repos withEnum.each, so one of these transactions is open at a time on one connection.Same shape as the
SET LOCAL work_memthe hexpm stats job already uses for its matview refresh. hexpm/hexpm-ops#240 raises the instance-wide default to 16MB, which is sized for ordinary web queries and nowhere near enough for this one.The test asserts the statement is issued during a swap, filtered to the test process because telemetry handlers are global. I checked it discriminates by removing the
SET LOCALand confirming it fails.