Commit fda2656
Add whole-record WAL compression alongside FPI compression
When a record is larger than wal_compression_threshold, compress it as a
single unit instead of compressing each full-page image separately. This
wins whenever the images in one record share content, which is typical of
B-tree index builds: CREATE INDEX over 10M random floats drops from 160MB
of WAL to 132MB with lz4, and from 125MB to 97MB with zstd.
Only lz4 and zstd take part. pglz refuses input it cannot shrink by 25%,
which a record made of full-page images rarely clears, so it saved
nothing while costing an extra compression attempt. It still compresses
full-page images as before.
Assembly skips per-FPI compression while whole-record compression is
attempted, and reassembles the record if that does not pay off. Setting
wal_compression_threshold above the largest possible record restores the
previous behaviour.
The per-block compressed_page arrays make way for two buffers of the same
maximum size, one staging the record and one taking the compressor
output. XLR_COMPRESSED in xl_info marks a compressed record, which
begins with an XLogCompressionHeader giving the method and the
decompressed length.
Author: Andrey Borodin
Discussion: https://postgr.es/m/4DC38068-976E-4A84-8EE6-4EFACBBD927A@yandex-team.ru1 parent 14ae09c commit fda2656
16 files changed
Lines changed: 804 additions & 71 deletions
File tree
- src
- backend
- access/transam
- utils/misc
- include
- access
- utils
- test
- perl/PostgreSQL/Test
- recovery
- t
- tools/pgindent
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
142 | 142 | | |
143 | 143 | | |
144 | 144 | | |
| 145 | + | |
145 | 146 | | |
146 | 147 | | |
147 | 148 | | |
| |||
751 | 752 | | |
752 | 753 | | |
753 | 754 | | |
| 755 | + | |
| 756 | + | |
| 757 | + | |
| 758 | + | |
| 759 | + | |
| 760 | + | |
| 761 | + | |
| 762 | + | |
| 763 | + | |
| 764 | + | |
| 765 | + | |
| 766 | + | |
| 767 | + | |
| 768 | + | |
| 769 | + | |
| 770 | + | |
754 | 771 | | |
755 | 772 | | |
756 | 773 | | |
| |||
1068 | 1085 | | |
1069 | 1086 | | |
1070 | 1087 | | |
1071 | | - | |
| 1088 | + | |
1072 | 1089 | | |
1073 | 1090 | | |
1074 | 1091 | | |
| |||
0 commit comments