Skip to content

Commit fda2656

Browse files
x4mhackorum
authored andcommitted
Add whole-record WAL compression alongside FPI compression
When a record is larger than wal_compression_threshold, compress it as a single unit instead of compressing each full-page image separately. This wins whenever the images in one record share content, which is typical of B-tree index builds: CREATE INDEX over 10M random floats drops from 160MB of WAL to 132MB with lz4, and from 125MB to 97MB with zstd. Only lz4 and zstd take part. pglz refuses input it cannot shrink by 25%, which a record made of full-page images rarely clears, so it saved nothing while costing an extra compression attempt. It still compresses full-page images as before. Assembly skips per-FPI compression while whole-record compression is attempted, and reassembles the record if that does not pay off. Setting wal_compression_threshold above the largest possible record restores the previous behaviour. The per-block compressed_page arrays make way for two buffers of the same maximum size, one staging the record and one taking the compressor output. XLR_COMPRESSED in xl_info marks a compressed record, which begins with an XLogCompressionHeader giving the method and the decompressed length. Author: Andrey Borodin Discussion: https://postgr.es/m/4DC38068-976E-4A84-8EE6-4EFACBBD927A@yandex-team.ru
1 parent 14ae09c commit fda2656

16 files changed

Lines changed: 804 additions & 71 deletions

File tree

src/backend/access/transam/xlog.c

Lines changed: 18 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -142,6 +142,7 @@ int wal_retrieve_retry_interval = 5000;
142142
int max_slot_wal_keep_size_mb = -1;
143143
int wal_decode_buffer_size = 512 * 1024;
144144
bool track_wal_io_timing = false;
145+
int wal_compression_threshold = 512;
145146

146147
#ifdef WAL_DEBUG
147148
bool XLOG_DEBUG = false;
@@ -751,6 +752,22 @@ static void WALInsertLockUpdateInsertingAt(XLogRecPtr insertingAt);
751752

752753
static void XLogChecksums(uint32 new_type);
753754

755+
#ifdef WAL_DEBUG
756+
/* Read length of a record, accounting for possible compression */
757+
static uint32
758+
XLogGetRecordTotalLen(XLogRecord *record)
759+
{
760+
if (record->xl_info & XLR_COMPRESSED)
761+
{
762+
XLogCompressionHeader *c = (XLogCompressionHeader *) record;
763+
764+
Assert(c->decompressed_length > 0);
765+
return c->decompressed_length;
766+
}
767+
return record->xl_tot_len;
768+
}
769+
#endif
770+
754771
/*
755772
* Insert an XLOG record represented by an already-constructed chain of data
756773
* chunks. This is a low-level routine; to construct the WAL record header
@@ -1068,7 +1085,7 @@ XLogInsertRecord(XLogRecData *rdata,
10681085
/* We also need temporary space to decode the record. */
10691086
record = (XLogRecord *) recordBuf.data;
10701087
decoded = (DecodedXLogRecord *)
1071-
palloc(DecodeXLogRecordRequiredSpace(record->xl_tot_len));
1088+
palloc(DecodeXLogRecordRequiredSpace(XLogGetRecordTotalLen(record)));
10721089

10731090
if (!debug_reader)
10741091
debug_reader = XLogReaderAllocate(wal_segment_size, NULL,

0 commit comments

Comments
 (0)