Skip to content

Switchover mechanism, for use when primary itself is doing switchover - #83

Open
serprex wants to merge 6 commits into
mainfrom
switchover
Open

Switchover mechanism, for use when primary itself is doing switchover#83
serprex wants to merge 6 commits into
mainfrom
switchover

Conversation

@serprex

@serprex serprex commented Aug 7, 2026

Copy link
Copy Markdown
Member

Includes support for changing source/destination config without restart

@serprex

serprex commented Aug 17, 2026

Copy link
Copy Markdown
Member Author

depends on ClickHouse/wal-rus/pull/24

@serprex serprex changed the title Start on plans/future/failover.md Switchover mechanism, for use when primary itself is doing switchover Aug 17, 2026
@serprex
serprex marked this pull request as ready for review August 17, 2026 20:43
Comment thread plans/failover.md
configured `[source] slot` name, when slot mode is on. Physical slots
are never synchronized to a standby at any PostgreSQL version; PG 17
slot sync covers logical slots created with `failover = true`
(`src/backend/replication/logical/slotsync.c`)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we create the slot on the standby target before hand? We could make the slot on the standby all the time. We can keep the wal available on primary and standby in sync. This could help avoid wal hole problem.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes but then you need to report slot progress to both primary & standbys

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, would that be a big issue?

Comment thread src/backfill/backfill_types.rs Outdated
Comment thread src/config.rs Outdated
Comment thread src/source/transition.rs
Implements support for changing source/destination config without restart

Some issues being avoided:
- boundary aligned fork needs zero crc
- pre-commit restart skipping ancestor crossing, ie
1. manifest still says timeline 1, crossing commit has not happened
2. floor is start of segment containing fork, common when `C` and `F` share a segment
3. target is promoted to timeline 2
4. daemon dies after promotion while paused or waiting inside barrier
5. boot reads live history, `tli_of_segment(floor)` returns timeline 2 because descendant owns fork segment filename
6. condition accepts timeline 2 despite manifest still naming timeline 1
7. pump starts directly on timeline 2, so it never performs timeline 1 to 2
Block on erroneous scenarios so daemon still accessible,
pause/resume to retry. Improve handling stop/promote/start

Protocol extends status with necessary information for controlplane doing switchover

Add more validation. Test more restart scenarios
Comment thread src/config.rs Outdated
Comment thread src/config.rs Outdated
Comment thread src/config.rs Outdated
Comment thread src/source/wal_stream.rs
Comment thread src/bin/stream.rs
Comment thread src/source/shadow_stream.rs
Comment thread src/ops/metrics.rs
Comment thread src/source/source_feed.rs Outdated
Comment thread src/bin/stream.rs Outdated
Comment thread src/bin/stream.rs
)
.await?;
}
if ancestor_ended {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: could be over engineering, but there's a lot of a of these state parameters, can we do something about it? like some state machine?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added CrossingState to try clean it up a bit, but finite state machines might be a direction to explore further to get logic out of the muck. would like some more reliable barrier/invalidation management somehow

@serprex
serprex force-pushed the switchover branch 2 times, most recently from 9c6311e to f970caf Compare August 21, 2026 02:42
* slot in SouceConn
* move code to timeline.rs/transition.rs
* monitor source_system_id
* add switchover info in grafana
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants