Skip to content

Exit the supervisor loop on SIGTERM instead of spinning until SIGKILL #67

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
tamalsaha wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Exit the supervisor loop on SIGTERM instead of spinning until SIGKILL #67

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 20 additions & 3 deletions scripts/run.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,14 +2,31 @@
RECOVERY_DONE_FILE="/var/pv/"$PITR_UNIX_TIME"_recovery.done"
PITR_RS=${PITR_REPLICATION_STRATEGY:-none}
STOP=false
# don't restart postgres on SIGTERM (eg, pod deleted)
# don't restart postgres on SIGTERM (eg, pod deleted), and stop supervising once
# the signal arrives, so this script (the container's main process) EXITS.
#
# The trap alone is not enough: it only tells the loop below not to start postgres
# again. With `while true` the loop kept spinning forever after that, the container
# never exited on its own, and kubelet had no choice but to wait out the whole
# terminationGracePeriodSeconds and SIGKILL. Two consequences, both observed live:
# pod deletions blocked for the entire grace period (5 minutes on a DB that sets
# 300), and on the default 30s grace a postgres shutdown checkpoint slower than
# that got SIGKILLed midway, leaving a data directory that needs crash recovery.
#
# STOP is only ever set by a signal delivered to THIS bash process, which in
# practice means kubelet stopping the container. The coordinator stops postgres
# with `pg_ctl stop` exec'd into the container (and kills nothing else by name or
# process group), so its shutdowns do not reach this process: the loop keeps
# retrying and re-runs the role script when the coordinator writes the next one,
# exactly as before. Verified live: a single container instance served 17 role
# script starts with no restart.
# ref: https://opensource.com/article/20/6/bash-trap
trap \
"{ STOP=true; }" \
SIGINT SIGTERM EXIT

if [[ "$PITR_RESTORE" == "true" ]]; then
while true; do
while [[ "$STOP" = false ]]; do
sleep 2
echo "Point In Time Recovery In Progress. Waiting for $RECOVERY_DONE_FILE file"
if [[ -e "$RECOVERY_DONE_FILE" ]]; then
Expand All @@ -22,7 +39,7 @@ fi
#going to change this with the check of process id
rm -f "$PGDATA"/postmaster.pid
echo "waiting for the role to be decided ..."
while true; do
while [[ "$STOP" = false ]]; do
# Robust /var/pv mount availability check before any destructive operation or basebackup
if [[ -e /var/pv/BOOTSTRAP_INITIALIZATION_STARTED ]]; then
rm /var/pv/BOOTSTRAP_INITIALIZATION_STARTED
Expand Down
Loading