Skip to content

Add namespace-scoped performance indexes on projects#161

Merged
khoroshevskyi merged 1 commit into
query_improvementsfrom
perf/namespace-indexes
Jul 8, 2026
Merged

Add namespace-scoped performance indexes on projects#161
khoroshevskyi merged 1 commit into
query_improvementsfrom
perf/namespace-indexes

Conversation

@nsheff

@nsheff nsheff commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Builds on query_improvements. Adds the indexes for the two namespace-scoped hot paths, complementing this branch's search / ORDER-BY indexes and N+1 fix. Added as model-level Index() to match the convention here.

Indexes added (Projects.__table_args__)

Index Purpose
ix_projects_namespace_update_date on (namespace, last_update_date) Namespace project listing (WHERE namespace = ? ORDER BY last_update_date DESC LIMIT n) → bounded index range scan instead of walking the global last_update_date index and filtering by namespace (the plan flip that scans most of the table for a namespace whose rows are not near the date tip), and no full sort.
ix_projects_ns_agg on (namespace) INCLUDE (number_of_samples, name) WHERE private IS FALSE Namespace aggregation (SELECT namespace, count(name), sum(number_of_samples) ... GROUP BY namespace) → index-only scan (~25 MB) instead of a ~250 MB heap seq scan. Needs a fresh visibility map, so run VACUUM ANALYZE projects after.
trgm_index_namespace GIN on namespace Namespace ILIKE search.

Why

The two namespace paths were full-scanning the ~250 MB projects heap on every request. On a RAM-starved host that means seconds-to-minutes of cold reads. These make both paths read a bounded, cacheable number of pages.

Applying to production (existing DB)

create_all (checkfirst) covers fresh installs. For the existing prod table, create concurrently so no blocking lock, then vacuum for the index-only scan:

CREATE INDEX CONCURRENTLY IF NOT EXISTS ix_projects_namespace_update_date
    ON projects (namespace, last_update_date);
CREATE INDEX CONCURRENTLY IF NOT EXISTS ix_projects_ns_agg
    ON projects (namespace) INCLUDE (number_of_samples, name) WHERE private IS FALSE;
CREATE INDEX CONCURRENTLY IF NOT EXISTS trgm_index_namespace
    ON projects USING gin (namespace gin_trgm_ops);
VACUUM ANALYZE projects;

Recommended cleanup while applying (not expressible via create_all), to cut write amplification:

DROP INDEX CONCURRENTLY IF EXISTS registry_index;              -- exact dup of projects_namespace_name_tag_key
DROP INDEX CONCURRENTLY IF EXISTS project_id_samp_index_ccnew; -- leftover invalid index from an interrupted REINDEX

Notes

  • The new Index() DDL was verified to compile cleanly against the PostgreSQL dialect.
  • Index changes alone are mitigation; they do not cure the underlying host RAM/EBS-bandwidth constraint (tracked separately).

@khoroshevskyi khoroshevskyi merged commit bdc3c39 into query_improvements Jul 8, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants