Skip to content

Add update_stats query for HOT and same-page update analysis - #39

Merged
pawurb merged 5 commits into
pawurb:mainfrom
chaadow:add_update_stats
Jul 24, 2026
Merged

Add update_stats query for HOT and same-page update analysis#39
pawurb merged 5 commits into
pawurb:mainfrom
chaadow:add_update_stats

Conversation

@chaadow

@chaadow chaadow commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Summary

This PR adds a new update_stats.sql query that exposes how updates are physically handled for each table in a schema.

It builds on the update counters available in pg_stat_user_tables to distinguish between:

  • HOT updates
  • Same-page non-HOT updates
  • Updates where the new tuple version was written to another heap page

Motivation

Looking only at n_tup_hot_upd does not explain why an update was not HOT.

A low HOT ratio can have very different causes:

  • The update modifies columns referenced by indexes, making it ineligible for HOT.
  • The update is HOT-eligible, but the current heap page does not have enough free space for the new tuple version.
  • Both conditions apply.

Using n_tup_newpage_upd, the query provides more context around page placement and helps evaluate:

  • Table fillfactor settings
  • Frequently updated indexed columns
  • HOT update effectiveness
  • Page-space pressure
  • The potential write amplification caused by non-HOT updates

Reported metrics

The query includes:

  • The table's configured fillfactor
  • Total update count
  • HOT update count and percentage
  • Same-page non-HOT update count and percentage
  • New-page update count and percentage
  • Percentage of all updates that remained on the same page
  • Percentage of same-page updates that were HOT

This makes it possible to interpret update behavior more precisely. For example, a table may have a low overall HOT ratio while still keeping most updates on the same page, indicating that indexed-column changes—not page space—are the main limiting factor.

Compatibility and usage notes

n_tup_newpage_upd is available starting with PostgreSQL 16, so this query requires PostgreSQL 16 or newer.

The underlying statistics are cumulative. For before-and-after comparisons, such as evaluating a fillfactor change, the metrics should ideally be captured at two points in time and compared using their deltas.

Reference :

@chaadow
feat: add update_stats for HOT and new-page update breakdown
e3b24be 
## Summary
Add `update_stats` to report per-table HOT, same-page non-HOT, and new-page update metrics from `pg_stat_user_tables`, including fillfactor and percentages.

## Why
Update-heavy workloads benefit from HOT updates, but diagnosing low HOT rates needs separating space pressure (`n_tup_newpage_upd`) from indexed-column updates. PostgreSQL 16 exposes that split; older versions still get a HOT vs non-HOT fallback.
@chaadow

chaadow commented Jul 16, 2026

Copy link
Copy Markdown
Contributor Author

@pawurb once you review and approve this, my end goal is to change rails-pg-extras's diagnose page, in order to display tables that have high percentages of New-page update count and percentage indicating that tweaking fillfactor can be beneficial for heavily updated tables!

It would be akin to MissingFkIndexes ( so something like HighNewPageUpdatePercentage )
I will of course make this ignorable using our recently added IgnoreList helper so users can ignore these warnings.

But yeah it is quite insane, how adding an innocent-looking index on a column, or changing an UPDATE statement to update an indexed column can make a hot-path update NON-HOT, and drastically decrease an application's performance ( index bloating, write amplification, more WAL, decreased read performance, slower vacuums! etc... )

Overall, I think it's beneficial for ruby/rails application developers to be aware of which tables are "update-heavy" and then act accordingly.

this photo from the Adyen video summarizes the gotchas for application developers

image

@chaadow
feat: add new_page_updates check for diagnosing update performance
d739417 
## Summary
Introduce a new `new_page_updates` check in the `diagnose` module to identify tables in PostgreSQL 16 and newer that experience a high ratio of new-page updates. This check helps diagnose potential page-space pressure and the effectiveness of the table's `fillfactor`.

## Details
- The check reports tables with at least 10,000 cumulative updates and 20% or more of updates resulting in new-page placements.
- Users can override default thresholds using environment variables.
- The implementation includes detailed reporting on new-page ratios, fillfactor, and HOT update percentages.

## Why
This feature enhances the ability to diagnose performance issues related to update-heavy workloads, providing insights into how table configurations may impact update efficiency.
@chaadow
chaadow force-pushed the add_update_stats branch from 205cc79 to d739417 Compare July 17, 2026 23:17
chaadow added 3 commits July 18, 2026 01:04
@chaadow
feat: add avg_row_bytes calculation to update_stats queries
3c8293b 
## Summary
Enhance the `update_stats` and `update_stats_legacy` SQL queries to include the calculation of average row size in bytes (`avg_row_bytes`). This metric is derived from the relation size divided by the number of tuples, providing better insights into table storage efficiency.

## Details
- Added `reltuples` to the selected columns to facilitate the average row size calculation.
- Implemented the `avg_row_bytes` calculation in both queries.
- Updated smoke tests to verify the correctness of the new metric after running `ANALYZE`.

## Why
This addition improves the diagnostic capabilities of the tool by allowing users to assess the average size of rows in their tables, which can be crucial for performance tuning and storage optimization.
@chaadow
More polishes
079ad67
@chaadow
feat: add low_hot_same_page check for diagnosing update performance
7888bd1 
## Summary
Introduce a new `low_hot_same_page` check in the `diagnose` module to identify tables in PostgreSQL 16 and newer that have a low ratio of HOT updates among same-page updates. This check helps diagnose potential issues with indexed column updates.

## Details
- The check reports tables with at least 10,000 cumulative updates and fewer than 10% of updates being HOT.
- Users can override default thresholds using environment variables.
- The implementation includes detailed reporting on HOT ratios, same-page and new-page update percentages, and current fillfactor.

## Why
This feature enhances the ability to diagnose performance issues related to update-heavy workloads, providing insights into how table configurations may impact update efficiency.
@pawurb

pawurb commented Jul 23, 2026

Copy link
Copy Markdown
Owner

@codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Jul 23, 2026

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Already looking forward to the next diff.

Reviewed commit: 7888bd18bb

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chaadow

chaadow commented Jul 24, 2026

Copy link
Copy Markdown
Contributor Author

@pawurb I can show the results :

Before :
image

After

image

after :

  • setting fillfactor from 100 to 90
  • removing the updated_at index ( which completely disabled hot updates since, rails always sets updated_at on each update
  • replacing it with created_at index (which is set only at insert time, so not an update ) and avoid session fixation
  • went from 0% HOT pct, to 96,01% ( I suspect it's not 100% because probably sometime we store huge data jsons that cant fit on a 8Kb page. it's still a theory, because i've been tracking update statements, and it's the only update that might cause that )

After this gets merged, I'm going to open a PR on rails-pg-extras to improve the display on the diagnose page. maybe using simple_format or something similar. I've always wanted to improve the display there, for instance when you have a lot of unused indexes it's impossible at a glance to read whats going on without exerting a lot of effort. ( Thankfully now we have AI but that'd still be better if we can just quickly analyze without resorting to AI every single time haha )

@pawurb
pawurb merged commit ae8632b into pawurb:main Jul 24, 2026
6 checks passed
@pawurb

pawurb commented Jul 24, 2026

Copy link
Copy Markdown
Owner

@chaadow thanks! I'll go ahead and release so that you can start the rails-pg-extras pr.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@chaadow @pawurb