Exit the supervisor loop on SIGTERM instead of spinning until SIGKILL

[tamalsaha]

The bug

scripts/run.sh traps SIGTERM and sets STOP=true, but the loop condition is the
literal true; STOP is only consulted to decide whether to start postgres again:

trap "{ STOP=true; }" SIGINT SIGTERM EXIT
while true; do                                   # never re-evaluates STOP
    if [[ -e /run_scripts/role/run.sh ]] && [[ "$STOP" = false ]]; then ... fi
    sleep 1
done

So on SIGTERM the script does exactly what commit 06ba93a promised (stop restarting
postgres) and then spins forever. This script is the container's main process
(tini -> bash run.sh -> postgres), so the container can never exit on its own:
every termination ends in a SIGKILL at the end of terminationGracePeriodSeconds.

Impact, observed live on a DC-DR pair

1. Pod deletions block for the entire grace period. A database with
   terminationGracePeriodSeconds: 300 took 5 minutes to delete, idle almost the
   whole time. Its own logs show postgres finished shutting down at 09:03:35 and
   run.sh immediately printed removing the initial scripts as server is not running ... (the loop regaining control) — then nothing until the SIGKILL at
   09:07:34, exactly the deletion deadline.
2. On the default 30s grace, postgres gets SIGKILLed mid-checkpoint. A measured
   shutdown on a moderately busy instance took 42s. Past the ceiling the postmaster
   is killed part-way through its shutdown checkpoint, leaving a data directory that
   needs crash recovery — for a DC-DR primary, precisely the torn-checkpoint shape
   behind timeline divergence incidents.

The fix

while [[ "$STOP" = false ]]; do (same for the PITR wait loop, which had the same
problem). After SIGTERM the current iteration finishes and the script exits, so the
container stops in about a second once postgres is down, and the grace period goes
back to being a ceiling rather than a fixed cost.

Why this does not break the restart-on-role-change behavior

The concern is real and was checked before changing the line: pg-coordinator stops
postgres on its own (demote, fence, rewind), and the loop must keep retrying so the
next role script gets picked up. It still does, because a bash trap only fires for
signals delivered to that bash process, and no coordinator path signals it:

coordinator path	what it runs	target
TerminatePostgres (gRPC + fallback)	pg_ctl -m fast|immediate -w stop exec'd into the postgres container	postmaster, by its own pidfile
ShutDownCleanly	same pg_ctl path	postmaster
KillBaseBackupProcess	scans /proc/*/comm, kills only where comm == pg_basebackup	pg_basebackup
reload / walreceiver stop	kill -HUP $(head -1 postmaster.pid), kill -TERM $walreceiver_pid	postmaster / one backend

No pkill, no killall, no process-group or pid-1 kill anywhere. Killing a child
only makes bash's wait return, which is the existing retry path.

Live proof: a single postgres container instance (restartCount unchanged) served
17 running the initial script ... starts across many coordinator-driven role
changes. That is only possible if STOP stayed false the whole time — if any of
those kills had reached bash, the [[ "$STOP" = false ]] guard would have blocked
every subsequent start and postgres would never have come back.

Caveat for future readers: this holds because tini runs without -g. With group
mode, a container SIGTERM would also hit postgres directly and bypass the pg_ctl
fast-stop ordering.

Testing

• bash -n clean.
• Sandbox reproduction of the old shape: still alive after SIGTERM, needed SIGKILL.
• Sandbox of the fixed shape: restarted the child twice (retry preserved), then
  exited on its own after SIGTERM.
• Live verification on a real cluster with a database built on this image: pending
  in the PR discussion (bootstrap, role transitions, and deletion timing).

[tamalsaha]

Live verification: this change is correct but NOT sufficient on its own, and must not merge alone

Verified on a real cluster (3-replica Postgres, terminationGracePeriodSeconds: 300,
built on this branch's image). Results, and a second defect this uncovered.

The change works when the signal reaches bash

Sending SIGTERM directly to the run.sh bash process, then stopping postgres, produced
exactly the intended behavior: the deferred trap fired, STOP=true, the loop exited and
bash exited on its own. Confirmed by ps in the container: PID 7 gone.

(Note for reviewers: bash defers a trap handler while a foreground child is running, so
the loop only exits after postgres has stopped. That ordering is correct here, and it is
also why sending SIGTERM while postgres is still up appears to do nothing.)

But pod deletion still takes the full grace period

Three separate deletions on the fixed image: 301s, 300s, 303s. The instrumented run
shows why the fix is invisible: postgres shut down at t+10s via the preStop hook
(pg_ctl -m fast -w stop, measured at 0.10s, exit 0 — the hook does not hang), run.sh
printed removing the initial scripts as server is not running ..., and then the
container stayed running until the SIGKILL at the deadline.

The container is held open by PID 1, not by this script. /scripts/tini comes from
the fork github.com/kubedb/tini, which carries a deliberate commit,
"Don't exit if the original child exits" (2020-12-24), that comments out tini's
child-exit handling entirely:

-int reap_zombies(const pid_t child_pid, int* const child_exitcode_ptr) {
+int reap_zombies(const pid_t child_pid) {
     if (current_pid == child_pid) {
-        ... set *child_exitcode_ptr; tini then exits ...
+        // ... commented out ...

So tini never exits with its child. Observed live in-cluster: children of pid1: [],
tini parked in do_sigtimedwait, container still running. Reproduced locally, away
from Kubernetes, with this branch's image:

• child exits → tini keeps running (container had to be killed by timeout)
• docker stop → the child's SIGTERM trap fires (forwarding works), but the stop
  still takes the full timeout because tini stays alive

The failure mode this change introduces without a tini fix

With the fork's tini, a run.sh that exits leaves a zombie container: PID 1 idle, no
bash, no postgres — and because the postgres container has no liveness or readiness
probe, kubelet reports it Ready with restartCount: 0. Observed exactly that on a test
pod, while the database went Critical. Today's infinite loop at least keeps the
supervisor alive, so this change is a regression until PID 1 can exit.

Proposed complete fix

1. This PR (run.sh exits on SIGTERM) — necessary, verified.
2. kubedb/tini: make the child-exit path shutdown-aware instead of removed. Set a
   flag when tini forwards a termination signal, and exit with the child's status if the
   child exits while that flag is set. That keeps the fork's intent (a child exiting on
   its own does not kill the container) while restoring graceful termination:
   SIGTERM → forwarded → run.sh exits → tini exits → container gone in ~1s instead of
   the full grace period.

Until item 2 lands, this PR should stay open. Holding it also has a cost worth stating:
today every DB pod termination is a SIGKILL at the ceiling, so on the default 30s grace a
shutdown checkpoint slower than 30s (42s measured) is killed midway.