Conversation
📝 WalkthroughWalkthroughChangesFailover and Disaster Recovery Guides
Estimated code review effort: 3 (Moderate) | ~20 minutes Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 15
Note
Due to the large number of review comments, Critical, Major severity comments were prioritized as inline comments.
🟡 Minor comments (10)
docs/guides/elasticsearch/failover/index.md-377-379 (1)
377-379: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick winFix the cleanup heading and wording.
Use
## Cleanup. Rewrite the sentence as “To clean up the resources created in this tutorial, run:”. Rewrite Line 394 as “Learn about backing up and restoring an Elasticsearch database using Stash.”Also applies to: 394-394
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/guides/elasticsearch/failover/index.md` around lines 377 - 379, Update the cleanup section heading to “## Cleanup”, replace its introductory sentence with “To clean up the resources created in this tutorial, run:”, and revise the sentence at the referenced later line to “Learn about backing up and restoring an Elasticsearch database using Stash.”Source: Linters/SAST tools
docs/guides/elasticsearch/failover/index.md-13-13 (1)
13-13: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick winUse descriptive link text.
Replace
[here]with labels that identify the destination, such as[KubeDB documentation overview]and[KubeDB setup guide].Also applies to: 62-62
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/guides/elasticsearch/failover/index.md` at line 13, Update the introductory links in the Elasticsearch failover guide to use descriptive destination-specific text instead of “here,” including the corresponding occurrence noted in the comment. Preserve the existing link targets while labeling them clearly, such as “KubeDB documentation overview” and “KubeDB setup guide.”Source: Linters/SAST tools
docs/guides/kafka/failover/index.md-297-297 (1)
297-297: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick winFix the visible wording issues.
Use
Cleanupfor the heading andKubeDB-managed Kafkain the Next Steps list.Also applies to: 317-317
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/guides/kafka/failover/index.md` at line 297, Update the visible wording in the Kafka failover guide: change the “CleanUp” heading to “Cleanup” and replace the affected Next Steps wording with “KubeDB-managed Kafka” at both referenced locations.Source: Linters/SAST tools
docs/guides/kafka/failover/index.md-207-210 (1)
207-210: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winDo not identify the active controller without checking
LeaderId.The preceding output reports
LeaderId: 0, which identifieskafka-multinode-0as the active controller. It identifieskafka-multinode-1only as the partition leader. Replace the parenthetical with an instruction to check the currentLeaderIdbefore deleting the pod.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/guides/kafka/failover/index.md` around lines 207 - 210, Update the Case 1 description to remove the claim that kafka-multinode-1 may be the active controller, and replace the parenthetical with an instruction to check the current LeaderId before deleting the pod.docs/guides/kafka/failover/index.md-13-13 (1)
13-13: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick winUse descriptive link text.
Replace
[here]with labels such as[KubeDB overview]and[KubeDB installation guide]. This improves accessibility and makes the destination clear in link lists.Also applies to: 58-58
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/guides/kafka/failover/index.md` at line 13, Update the introductory link in the Kafka failover guide to replace the vague “here” label with descriptive text that identifies the KubeDB overview or getting-started destination, while preserving the existing target path; apply the same link-text improvement to the additional occurrence noted in the comment.Source: Linters/SAST tools
docs/guides/redis/failover/guide.md-384-386 (1)
384-386: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick winFix the cleanup heading and sentence.
Use
## Cleanup. Rewrite the final link as “Learn how to back up and restore a Redis database using Stash.”Also applies to: 401-401
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/guides/redis/failover/guide.md` around lines 384 - 386, Update the cleanup section in the Redis failover guide to use the heading “## Cleanup” and revise the final link text to “Learn how to back up and restore a Redis database using Stash.” Apply both wording fixes wherever this cleanup content appears.Source: Linters/SAST tools
docs/guides/redis/failover/guide.md-13-13 (1)
13-13: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick winUse descriptive link text.
Replace “here” with the destination name, such as “the KubeDB installation guide” and “the Redis Cluster concept guide”.
Also applies to: 53-53
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/guides/redis/failover/guide.md` at line 13, Update the links in the Redis failover guide, including the corresponding link at the other referenced location, to replace vague “here” anchor text with descriptive destination names such as “the KubeDB installation guide” and “the Redis Cluster concept guide,” while preserving the existing destinations.Source: Linters/SAST tools
docs/guides/redis/failover/guide.md-117-132 (1)
117-132: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winAlign the sample output with the displayed
kubectl getcommand.
$ kubectl get redis,petset,pods -n redisuses the default table format, so theNAMEcolumn should use the object names used by the command:redis,redis-shard0,redis-shard0-0, etc. The current names match a--show-kind/full-type output format, not the command shown above.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/guides/redis/failover/guide.md` around lines 117 - 132, The sample output for the kubectl get command should use default table-format names rather than kind-qualified identifiers. Update the NAME values in the redis, petset, and pod sections to redis, redis-shard0 through redis-shard2, and redis-shard0-0 through redis-shard2-1, while preserving the other columns and values.docs/guides/redis/failover/guide.md-233-234 (1)
233-234: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winRemove
MaxScalefrom the Redis comparison.
MaxScaleis a MariaDB database proxy, so it does not fit this Redis HA/Sentinel comparison. Change this to “Sentinel or a generic Redis proxy.”🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/guides/redis/failover/guide.md` around lines 233 - 234, Update the Redis Cluster comparison text to replace the MariaDB-specific “MaxScale” reference with a generic Redis proxy, yielding “Sentinel or a generic Redis proxy” while preserving the surrounding failover explanation.docs/guides/redis/failover/guide.md-43-43 (1)
43-43: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winDo not claim that
redis-cli -csendsREADONLY.
-cfollowsMOVEDandASKredirects, but replica direct reads still require the connection to sendREADONLY. Update the note at line 43 and the section spanning 189–205 to separate redirect handling from replica read mode, and show an explicitREADONLYstep if the guide tests reads directly from a replica.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/guides/redis/failover/guide.md` at line 43, Correct the Redis failover guide’s note and the replica-read instructions to remove the claim that redis-cli -c issues READONLY. Explain that -c only follows MOVED and ASK redirects, and add an explicit READONLY command or connection step wherever the guide tests direct reads from a replica.
🧹 Nitpick comments (1)
docs/guides/kafka/failover/index.md (1)
173-179: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winUse
--command-configin the Kafka 4.2 commands.Kafka 4.2 deprecates
--producer.configand--consumer.configin favor of--command-config. Replace both flags to avoid deprecation warnings and future incompatibility. (kafka.apache.org)Suggested replacement
- --producer.config config/clientauth.properties + --command-config config/clientauth.properties - --consumer.config config/clientauth.properties + --command-config config/clientauth.properties🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/guides/kafka/failover/index.md` around lines 173 - 179, Update the Kafka producer and consumer commands in the failover guide to replace the deprecated --producer.config and --consumer.config flags with --command-config, preserving the existing config/clientauth.properties value and all other command arguments.Source: MCP tools
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/guides/elasticsearch/failover/index.md`:
- Around line 324-337: Update the Case 2 demonstration command to delete only
the selected master-eligible pod, matching the single-node failure scenario
described by the section and preserving the remaining master node for failover
behavior.
- Around line 307-312: Correct the post-failure _cat shards output after
deleting es-topology-data-1: show the surviving data node as the primary (p),
mark the deleted or recovering node as unavailable rather than listing it as
STARTED, and preserve the document count of 1 consistently across the affected
shard rows.
- Around line 38-42: Update the “Data node (shard) failover” section to
distinguish shard-level independence from master coordination: state that an
elected master is required to promote in-sync replicas and schedule replacement
replicas, while the per-shard process does not require a specific
master-eligible node quorum.
- Around line 179-186: The Elasticsearch example must not pass ES_PASS through
the curl command-line arguments. Update the curl invocation in the node-listing
example to use a restrictive authentication/config file or stdin-based
credential mechanism, while preserving the existing ES_USER and ES_PASS values
and request endpoint.
- Around line 172-186: The Elasticsearch API request in the node-listing example
should use HTTPS instead of HTTP. Update the curl command under the port-forward
instructions to use https://localhost:9200, authenticate with the generated TLS
credentials, and include -k only as a clearly labeled insecure demo fallback.
- Around line 90-117: Update the failover guide around the topology manifest to
explicitly require and validate the local-path StorageClass before applying it,
and state that Local Path Provisioner uses node-local volumes without storage
failover after node failure. Clarify that the demo covers pod deletion only, or
replace local-path with an available shared-storage class while preserving the
topology configuration.
In `@docs/guides/kafka/failover/index.md`:
- Around line 252-254: Update the Case 2 verification commands to execute into
the surviving kafka-multinode-1 pod instead of kafka-multinode-0, and change the
sample shell prompt accordingly while leaving the metadata quorum check
unchanged.
- Around line 84-91: Update the Kafka failover guide’s storage setup around the
storageClassName and storageType settings to explicitly require a working
StorageClass. Add a prerequisite/check such as kubectl get storageclass, and
instruct readers to replace local-path with a StorageClass available in their
cluster before deploying.
- Around line 162-168: Update the default topic setup and related failover
documentation to use min.insync.replicas=2 with replication-factor 3, removing
any misleading lossless-failover implication. Ensure the displayed topic
configuration and subsequent explanation consistently reflect the configured
threshold and acknowledge that a single acknowledged replica is not sufficient.
- Around line 293-295: Update the polling command in the Kafka failover guide to
remove the interactive TTY flags, add the container selector -c kafka, and
ensure it waits until kafka-multinode-0 is Ready before executing the metadata
quorum status command.
In `@docs/guides/redis/failover/guide.md`:
- Around line 174-176: Update the Redis CLI commands throughout the guide,
including the repeated "$REDIS_PASSWORD" usages, to stop passing the password
via the -a argument; use redis-cli --askpass with stdin for interactive
operations or the guide’s protected Secret/environment mechanism, while
preserving each command’s existing operation.
- Around line 77-81: Update the Redis Cluster manifest’s cluster.replicas value
from 1 to a value conforming to the CRD contract, using the documented default
or minimum greater-than-one setting. Also revise any nearby
production-recommendation text in the failover guide so it consistently
recommends the valid replica count.
- Around line 249-251: Redesign Case 2 in the Redis failover guide so it does
not claim that a three-master cluster can automatically fail over two deleted
masters. Either remove the two-master deletion scenario or update it to delete
only one master; if retaining simultaneous failures, use a topology with enough
reachable masters and replicas to satisfy quorum.
- Around line 360-382: The Case 4 documentation currently implies that deleting
both pods permanently destroys the shard. Update the opening explanation to
state that no live failover copy remains and the shard becomes unavailable until
a PVC-backed pod recovers, while clarifying that permanent deletion requires
removing the Redis object with deletionPolicy: WipeOut; preserve the existing
recovery instructions.
- Around line 289-298: Update the failover verification section around the
“Confirm the data survived the promotion” instructions to state that Redis
replication is asynchronous and the surviving key only demonstrates this run,
not guaranteed zero data loss; add an explicit RPO warning. Insert the `hello`
write command directly before the pod-deletion step so the documented test
sequence clearly establishes when the data is created.
---
Minor comments:
In `@docs/guides/elasticsearch/failover/index.md`:
- Around line 377-379: Update the cleanup section heading to “## Cleanup”,
replace its introductory sentence with “To clean up the resources created in
this tutorial, run:”, and revise the sentence at the referenced later line to
“Learn about backing up and restoring an Elasticsearch database using Stash.”
- Line 13: Update the introductory links in the Elasticsearch failover guide to
use descriptive destination-specific text instead of “here,” including the
corresponding occurrence noted in the comment. Preserve the existing link
targets while labeling them clearly, such as “KubeDB documentation overview” and
“KubeDB setup guide.”
In `@docs/guides/kafka/failover/index.md`:
- Line 297: Update the visible wording in the Kafka failover guide: change the
“CleanUp” heading to “Cleanup” and replace the affected Next Steps wording with
“KubeDB-managed Kafka” at both referenced locations.
- Around line 207-210: Update the Case 1 description to remove the claim that
kafka-multinode-1 may be the active controller, and replace the parenthetical
with an instruction to check the current LeaderId before deleting the pod.
- Line 13: Update the introductory link in the Kafka failover guide to replace
the vague “here” label with descriptive text that identifies the KubeDB overview
or getting-started destination, while preserving the existing target path; apply
the same link-text improvement to the additional occurrence noted in the
comment.
In `@docs/guides/redis/failover/guide.md`:
- Around line 384-386: Update the cleanup section in the Redis failover guide to
use the heading “## Cleanup” and revise the final link text to “Learn how to
back up and restore a Redis database using Stash.” Apply both wording fixes
wherever this cleanup content appears.
- Line 13: Update the links in the Redis failover guide, including the
corresponding link at the other referenced location, to replace vague “here”
anchor text with descriptive destination names such as “the KubeDB installation
guide” and “the Redis Cluster concept guide,” while preserving the existing
destinations.
- Around line 117-132: The sample output for the kubectl get command should use
default table-format names rather than kind-qualified identifiers. Update the
NAME values in the redis, petset, and pod sections to redis, redis-shard0
through redis-shard2, and redis-shard0-0 through redis-shard2-1, while
preserving the other columns and values.
- Around line 233-234: Update the Redis Cluster comparison text to replace the
MariaDB-specific “MaxScale” reference with a generic Redis proxy, yielding
“Sentinel or a generic Redis proxy” while preserving the surrounding failover
explanation.
- Line 43: Correct the Redis failover guide’s note and the replica-read
instructions to remove the claim that redis-cli -c issues READONLY. Explain that
-c only follows MOVED and ASK redirects, and add an explicit READONLY command or
connection step wherever the guide tests direct reads from a replica.
---
Nitpick comments:
In `@docs/guides/kafka/failover/index.md`:
- Around line 173-179: Update the Kafka producer and consumer commands in the
failover guide to replace the deprecated --producer.config and --consumer.config
flags with --command-config, preserving the existing
config/clientauth.properties value and all other command arguments.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: ace02b76-0f88-4aaf-a877-dc2a2c6404f3
📒 Files selected for processing (6)
docs/guides/elasticsearch/failover/_index.mddocs/guides/elasticsearch/failover/index.mddocs/guides/kafka/failover/_index.mddocs/guides/kafka/failover/index.mddocs/guides/redis/failover/_index.mddocs/guides/redis/failover/guide.md
| - **Data node (shard) failover:** | ||
| Every index's data is split into shards, and each shard can have one or more replica copies stored on | ||
| different `data` nodes. If a data node holding a primary shard goes down, the elected master promotes an | ||
| in-sync replica of that shard to primary on a surviving node, and schedules a new replica to be built once | ||
| capacity is available. This happens independently, per shard, and doesn't depend on master-node count. |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
file="docs/guides/elasticsearch/failover/index.md"
if [ -f "$file" ]; then
echo "== target excerpt =="
sed -n '1,80p' "$file" | cat -n
echo
echo "== relevant dependency mention =="
sed -n '250,285p' "$file" | cat -n
echo
echo "== occurrences of master-node count/elected master =="
rg -n "master-node count|master count|elected master|master-eligible|data node|replica|shard" "$file"
else
echo "missing file: $file"
fiRepository: kubedb/docs
Length of output: 12334
🌐 Web query:
Elasticsearch node roles overview master eligible nodes data node shard allocation replica promotion elected master requirement
💡 Result:
In Elasticsearch, nodes are assigned specific roles via the node.roles setting [1][2]. If not explicitly defined, a node defaults to multiple roles, including master and data [1][2]. Master-Eligible Nodes A master-eligible node (role: master) is a node that can be elected to act as the cluster's master [3][4]. The elected master is responsible for lightweight, cluster-wide tasks such as creating or deleting indices, tracking cluster membership, and determining shard allocation [3][4]. For cluster stability, it is recommended to use dedicated master-eligible nodes that do not handle data or other resource-intensive tasks [4][1]. Elected Master Requirement Elasticsearch uses a quorum-based election process to select a master node [5][6]. Only master-eligible nodes participate in this process and the voting configuration—the set of nodes whose votes count toward electing a master or committing cluster state changes [5][7]. A decision is made only after a majority (more than half) of the voting configuration responds [7]. Without an elected master, the cluster cannot perform administrative tasks or maintain cluster state, effectively rendering it non-functional [8][9]. Data Nodes and Shard Allocation Data nodes (roles: data, data_content, etc.) hold the actual shard data and perform data-related operations like CRUD, search, and aggregations [3][1]. The master node is responsible for deciding where shards are allocated and when they should be moved to rebalance the cluster [3][10]. Shard allocation is the process of assigning shard copies to specific nodes, which is controlled by the master using cluster-level allocation settings and shard awareness constraints [10][11][12]. Replica Promotion Each shard has one primary copy and zero or more replicas [11][12]. The primary shard processes indexing operations and forwards them to in-sync replicas [13][14]. If a primary shard fails, the master node promotes one of the existing, in-sync replicas to be the new primary [13][11]. This promotion ensures that the cluster maintains write availability and data integrity, provided the promoted replica is part of the in-sync set—the group of copies guaranteed to contain all acknowledged writes [13][15].
Citations:
- 1: https://www.elastic.co/guide/en/elasticsearch/reference/8.19/modules-node.html
- 2: https://www.elastic.co/docs/reference/elasticsearch/configuration-reference/node-settings
- 3: https://www.elastic.co/guide/en/elasticsearch/reference/8.19/node-roles-overview.html
- 4: https://www.elastic.co/docs/deploy-manage/distributed-architecture/clusters-nodes-shards/node-roles
- 5: https://www.elastic.co/guide/en/elasticsearch/reference/8.19/modules-discovery-quorums.html
- 6: https://www.elastic.co/docs/deploy-manage/distributed-architecture/discovery-cluster-formation/modules-discovery-quorums
- 7: https://www.elastic.co/guide/en/elasticsearch/reference/8.19/modules-discovery-voting.html
- 8: https://www.elastic.co/docs/troubleshoot/elasticsearch/discovery-troubleshooting
- 9: https://www.elastic.co/guide/en/elasticsearch/reference/8.19/discovery-troubleshooting.html
- 10: https://www.elastic.co/guide/en/elasticsearch/reference/8.19/modules-cluster.html
- 11: https://www.elastic.co/guide/en/elasticsearch/reference/8.19/shard-allocation-relocation-recovery.html
- 12: https://www.elastic.co/docs/deploy-manage/distributed-architecture/shard-allocation-relocation-recovery
- 13: https://www.elastic.co/guide/en/elasticsearch/reference/8.19/docs-replication.html
- 14: https://github.com/elastic/elasticsearch/blob/master/docs/reference/docs/data-replication.asciidoc
- 15: https://www.elastic.co/blog/tracking-in-sync-shard-copies
🌐 Web query:
Elasticsearch failover elected master primary shard replica promotion shard allocation coordination cluster state
💡 Result:
Elasticsearch ensures high availability through a distributed architecture that manages failure, coordination, and shard lifecycle. Cluster Coordination and Master Election Elasticsearch uses a quorum-based system for cluster coordination to prevent split-brain scenarios [1][2]. A cluster requires a set of master-eligible nodes, and a majority (quorum) of the voting configuration must agree on decisions, such as electing a new master or committing changes to the cluster state [1][2]. If the elected master node fails, remaining master-eligible nodes initiate an election to select a new master [1][3]. Cluster State Management The elected master node is the sole authority for making changes to the cluster state, which is a global data structure containing metadata, index settings, and shard allocation information [4][5]. The master computes required updates and publishes them to all nodes [5]. A change is only committed once a quorum of master-eligible nodes has acknowledged it, after which the master instructs all nodes to apply the new state [5][2]. Shard Allocation and Failover Elasticsearch maintains data redundancy through primary and replica shards [6]. The master node is responsible for shard allocation—deciding which nodes host which shards—and this information is maintained in the cluster state [4]. When a failure occurs: 1. Master Failure: The remaining master-eligible nodes detect the failure and elect a new master [3][7]. The new master ensures the cluster state is consistent across the cluster [5][2]. 2. Primary Shard Failure: If a node holding a primary shard fails, the master promotes one of the available in-sync replica shards to primary to maintain write availability [6][4][7]. The master then instructs another node to create a new replica shard to restore the desired redundancy level [8][9]. 3. Replica Promotion: Promotion is managed by the master, which tracks in-sync shard copies via allocation IDs to ensure that only shards containing the most recent data are promoted to primary [4]. The entire process is designed to be self-healing, automatically redistributing shards to nodes to restore health after failures [6][9]. Nodes also perform proactive fault detection, checking the connectivity of the elected master and other nodes to trigger these recovery mechanisms when necessary [3].
Citations:
- 1: https://www.elastic.co/guide/en/elasticsearch/reference/8.19/modules-discovery-quorums.html
- 2: https://www.elastic.co/blog/a-new-era-for-cluster-coordination-in-elasticsearch
- 3: https://www.elastic.co/guide/en/elasticsearch/reference/8.19/cluster-fault-detection.html
- 4: https://www.elastic.co/blog/tracking-in-sync-shard-copies
- 5: https://www.elastic.co/guide/en/elasticsearch/reference/8.19/cluster-state-overview.html
- 6: https://www.elastic.co/guide/en/elasticsearch/reference/8.19/shard-allocation-relocation-recovery.html
- 7: https://github.com/elastic/elasticsearch-definitive-guide/blob/a10d6678/020_Distributed_Cluster/35_Coping_with_failure.asciidoc
- 8: https://www.elastic.co/guide/en/elasticsearch/reference/8.19/docs-replication.html
- 9: https://www.elastic.co/guide/en/elasticsearch/reference/8.19/high-availability-cluster-design.html
Clarify master-eligible count for shard failover.
The shard failover flow does require an elected master to promote in-sync replicas and schedule new shards, so separate that coordination requirement from the independent quorum requirement for master-eligible nodes. Rephrase “doesn't depend on master-node count” to avoid implying data failover can proceed without master coordination.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/guides/elasticsearch/failover/index.md` around lines 38 - 42, Update the
“Data node (shard) failover” section to distinguish shard-level independence
from master coordination: state that an elected master is required to promote
in-sync replicas and schedule replacement replicas, while the per-shard process
does not require a specific master-eligible node quorum.
| topology: | ||
| master: | ||
| replicas: 2 | ||
| storage: | ||
| accessModes: | ||
| - ReadWriteOnce | ||
| resources: | ||
| requests: | ||
| storage: 1Gi | ||
| storageClassName: local-path | ||
| data: | ||
| replicas: 3 | ||
| storage: | ||
| accessModes: | ||
| - ReadWriteOnce | ||
| resources: | ||
| requests: | ||
| storage: 1Gi | ||
| storageClassName: local-path | ||
| ingest: | ||
| replicas: 2 | ||
| storage: | ||
| accessModes: | ||
| - ReadWriteOnce | ||
| resources: | ||
| requests: | ||
| storage: 1Gi | ||
| storageClassName: local-path |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "Track current file"
git ls-files | grep -F 'docs/guides/elasticsearch/failover/index.md' || true
echo
echo "Search relevant StorageClass/topology references"
rg -n "storageClassName|StorageClass|local-path|topology-cluster|Topology|storage failover|storage fail" docs/guides/elasticsearch -S || true
echo
echo "File excerpt"
sed -n '1,180p' docs/guides/elasticsearch/failover/index.md
echo
echo "Target topology guide if present in repo"
fd -a 'simple-dedicated-cluster|topology-cluster' docs 2>/dev/null | sed 's#^\./##' | head -20
for f in $(fd 'simple-dedicated-cluster/index.md|topology-cluster/index.md' docs 2>/dev/null | head -5); do
echo "===== $f ====="
sed -n '1,160p' "$f"
doneRepository: kubedb/docs
Length of output: 50367
🌐 Web query:
Local Path Provisioner Kubernetes StorageClass node-local volumes failure domain node failure
💡 Result:
The Local Path Provisioner is designed to provide dynamic, node-local persistent storage in Kubernetes by utilizing the storage available directly on the node where a pod is scheduled [1][2]. Because it relies on physical storage tied to a specific node, it has critical implications regarding failure domains and node failure [1][3]. Key Characteristics and Failure Behavior: 1. No Data Replication: The provisioner does not perform data replication across nodes [3]. If a node fails, the data stored on that node is inaccessible until the node recovers [2][4]. 2. Pod Rescheduling: Because a PersistentVolume (PV) created by this provisioner has strict node affinity (typically using the kubernetes.io/hostname label), a pod requiring that volume will only be scheduled onto the node where the volume resides [5][6][2]. If that node is down, the pod will remain in a pending state, unschedulable until the node is restored [1][2][3]. 3. Node Deletion and Orphaned Volumes: If a node is permanently removed from the cluster (e.g., via cluster autoscaler or cloud instance termination), the PVs associated with that node become orphaned [7][8]. The provisioner may encounter errors during cleanup attempts because it cannot reach the deleted node to delete the underlying storage path [9][7][8]. In such cases, administrators often need to manually intervene to delete the stale PVCs and PVs [9][7][8]. Best Practices for Resilience: - Application-Level Replication: Because the storage layer itself is not highly available, it is essential to use workloads that handle replication at the application level (e.g., distributed databases like Cassandra or Elasticsearch) [2][3]. - Proper Scheduling: Always use volumeBindingMode: WaitForFirstConsumer in the StorageClass [2]. This ensures that the PV is not bound to a node until the scheduler has determined exactly which node the pod will run on, preventing binding mismatches [2]. - Stable Node Identifiers: In environments where node hostnames might change (e.g., dynamic cloud instances), you can use the nodeAffinityKey parameter in the StorageClass to bind volumes to a stable node label rather than the default hostname [5][10]. - Use Alternative Solutions for HA: If the workload requires storage-level high availability or automatic failover, the Local Path Provisioner is not suitable [1]. For such requirements, consider using distributed block storage solutions like Longhorn or LINSTOR [9][3][11].
Citations:
- 1: https://oneuptime.com/blog/post/2026-03-19-rancher-local-path-storage/view
- 2: https://kubernetes.recipes/recipes/storage/kubernetes-local-persistent-volumes/
- 3: https://cnap.mintlify.app/clusters/storage/local-path
- 4: https://docs.okd.io/4.17/storage/persistent_storage_local/ways-to-provision-local-storage.html
- 5: https://github.com/rancher/local-path-provisioner?tab=readme-ov-file
- 6: pv alway store data in one node rancher/local-path-provisioner#63
- 7: retry blocking reconcile loop when pv/pvc stuck with nodes are deleted rancher/local-path-provisioner#533
- 8: Bad behavior when k8s nodes are removed by the cluster auto-scaler rancher/local-path-provisioner#215
- 9: provisioner doesn't like when nodes go away, VolumeFailedDelete rancher/local-path-provisioner#31
- 10: https://github.com/rancher/local-path-provisioner
- 11: https://philprime.dev/guides/migrating-k3s-to-rke2/lesson-7.html
Make the storage prerequisite and failure domain explicit.
This manifest hard-codes storageClassName: local-path, but the guide does not check for that StorageClass before applying the topology. Without local-path, the Elasticsearch PVCs can remain Pending. Local Path Provisioner also creates node-local volumes, so this setup does not provide storage failover after a node failure; use a shared-storage class, or state clearly that this demo covers pod deletion only.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/guides/elasticsearch/failover/index.md` around lines 90 - 117, Update
the failover guide around the topology manifest to explicitly require and
validate the local-path StorageClass before applying it, and state that Local
Path Provisioner uses node-local volumes without storage failover after node
failure. Clarify that the demo covers pod deletion only, or replace local-path
with an available shared-storage class while preserving the topology
configuration.
| Port-forward the client service and export credentials as shown in the | ||
| [topology cluster guide](/docs/guides/elasticsearch/clustering/topology-cluster/simple-dedicated-cluster/index.md#connect-with-elasticsearch-database). | ||
|
|
||
| ```bash | ||
| $ kubectl port-forward -n es-demo svc/es-topology 9200 # in one terminal | ||
|
|
||
| # in another terminal | ||
| $ export ES_USER=$(kubectl get secret -n es-demo es-topology-auth -o jsonpath='{.data.username}' | base64 -d) | ||
| $ export ES_PASS=$(kubectl get secret -n es-demo es-topology-auth -o jsonpath='{.data.password}' | base64 -d) | ||
| ``` | ||
|
|
||
| List every node along with its roles, and see which one is currently elected master (marked with `*`): | ||
|
|
||
| ```bash | ||
| $ curl -s -u "$ES_USER:$ES_PASS" "http://localhost:9200/_cat/nodes?v&h=name,node.role,master" |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== target excerpt =="
sed -n '140,195p' docs/guides/elasticsearch/failover/index.md || true
echo
echo "== sibling topology excerpt around connect section, if present =="
fd -a 'index.md' . | sed 's#^\./##' | grep 'docs/guides/elasticsearch/clustering/topology-cluster/simple-dedicated-cluster/index.md' || true
topo=$(fd 'index.md' docs/guides/elasticsearch/clustering/topology-cluster/simple-dedicated-cluster/ | head -n1 || true)
if [ -n "${topo:-}" ]; then
rg -n -C 8 'connect with|_cat/nodes|curl|https://localhost|enableSSL|HTTPBasic|password' "$topo" || true
fi
echo
echo "== search failover guide for curl/http/es-pass/enableSSL =="
rg -n 'curl|http://localhost|https://localhost|ES_PASS|enableSSL|kubectl port-forward|security\.enableSSL|security\.ssl' docs/guides/elasticsearch/failover/docs/guides/elasticsearch/failover/index.md || trueRepository: kubedb/docs
Length of output: 10237
Sensitive Data Exposure (CWE-319): Cleartext Transmission of Sensitive Information
Reachability: External
Use HTTPS for Elasticsearch API requests in this step.
ES_PASS is decoded from an Elasticsearch secret and sent with Basic auth to http://localhost:9200, exposing credentials over loopback HTTP. Use the generated TLS credentials with https://localhost:9200, and only include -k as an insecure demo fallback if it is clearly marked.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/guides/elasticsearch/failover/index.md` around lines 172 - 186, The
Elasticsearch API request in the node-listing example should use HTTPS instead
of HTTP. Update the curl command under the port-forward instructions to use
https://localhost:9200, authenticate with the generated TLS credentials, and
include -k only as a clearly labeled insecure demo fallback.
| $ export ES_USER=$(kubectl get secret -n es-demo es-topology-auth -o jsonpath='{.data.username}' | base64 -d) | ||
| $ export ES_PASS=$(kubectl get secret -n es-demo es-topology-auth -o jsonpath='{.data.password}' | base64 -d) | ||
| ``` | ||
|
|
||
| List every node along with its roles, and see which one is currently elected master (marked with `*`): | ||
|
|
||
| ```bash | ||
| $ curl -s -u "$ES_USER:$ES_PASS" "http://localhost:9200/_cat/nodes?v&h=name,node.role,master" |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf 'File excerpt:\n'
sed -n '170,190p' docs/guides/elasticsearch/failover/index.md | cat -n
printf '\nShell variable expansion and curl argv effect probe:\n'
cat <<'PY' > /tmp/curl_argv_probe.py
# This small Python simulation shows the expanded command before curl is started.
ES_USER = "user1"
ES_PASS = "secretpassword"
cmd = ["curl", "-s", "-u", f"{ES_USER}:{ES_PASS}", "http://localhost:9200/_cat/nodes?v&h=name,node.role,master"]
print(" ".join(cmd))
print("username_password_arg=", cmd[3])
PY
python3 /tmp/curl_argv_probe.pyRepository: kubedb/docs
Length of output: 1446
Sensitive Data Exposure (CWE-214)
Reachability: External
Do not pass ES_PASS in the curl argument list.
The shell expands "$ES_USER:$ES_PASS" into a single command-line argument before curl starts, so credentials are visible in the process arguments while the request runs. Use a restrictive auth/config file or stdin-based credential setup instead.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/guides/elasticsearch/failover/index.md` around lines 179 - 186, The
Elasticsearch example must not pass ES_PASS through the curl command-line
arguments. Update the curl invocation in the node-listing example to use a
restrictive authentication/config file or stdin-based credential mechanism,
while preserving the existing ES_USER and ES_PASS values and request endpoint.
| $ curl -s -u "$ES_USER:$ES_PASS" "http://localhost:9200/_cat/shards/info?v" | ||
| index shard prirep state docs store dataset ip node | ||
| info 0 r STARTED 2 11kb 11kb 10.42.0.214 es-topology-data-2 | ||
| info 0 p STARTED 2 11kb 11kb 10.42.0.211 es-topology-data-1 | ||
| info 0 r UNASSIGNED | ||
| ``` |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Correct the post-failure shard output.
After Line 297 deletes es-topology-data-1, Lines 309-310 still list that node as the primary and change the document count from 1 to 2. Show a surviving data node as p, mark the deleted or recovering node as unavailable, and keep the document count consistent.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/guides/elasticsearch/failover/index.md` around lines 307 - 312, Correct
the post-failure _cat shards output after deleting es-topology-data-1: show the
surviving data node as the primary (p), mark the deleted or recovering node as
unavailable rather than listing it as STARTED, and preserve the document count
of 1 consistently across the affected shard rows.
| version: 8.2.2 | ||
| mode: Cluster | ||
| cluster: | ||
| shards: 3 | ||
| replicas: 1 |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== locate file =='
fd -a 'guide.md$' . | sed 's#^\./##' | rg 'docs/guides/redis/failover/guide.md|guide.md' || true
echo "== file excerpt =="
sed -n '60,110p' docs/guides/redis/failover/guide.md || true
echo "== later excerpts =="
sed -n '370,390p' docs/guides/redis/failover/guide.md || true
echo "== search replica values in redis docs =="
rg -n "replicas:\s*(1|2)|spec\.cluster\.replicas|redis-shard-\*-[0-9]" docs/guides docs/examples/redis -S || true
echo "== related cluster examples =="
if [ -f docs/examples/redis/update-version/rd-cluster.yaml ]; then
cat -n docs/examples/redis/update-version/rd-cluster.yaml
fi
echo "== repo search for concepts redis replicas =="
rg -n "more than one|greater than one|ReplicasPerShard|replicasPerShard|replicas:" . -g '*.md' -g '*.yaml' -g '*.yml' | head -n 120Repository: kubedb/docs
Length of output: 203
🌐 Web query:
KubeDB Redis CRD spec.cluster.replicas per shard replicas greater than 1 docs
💡 Result:
In KubeDB, when configuring a Redis cluster (using spec.mode: Cluster), the spec.cluster.replicas field defines the number of replica nodes per shard [1]. According to the current KubeDB documentation, this value must be greater than or equal to 1 [1]. While some versions of the documentation or older code comments may state that this value must be greater than 1 (referring to a minimum of 2 replicas total per shard), the operational requirement for a functional Redis cluster is at least 1 replica per shard [1][2]. If the field is not explicitly set, the KubeDB operator defaults to 2 replicas per shard [1][2]. Key configuration details: - spec.cluster.shards: Defines the number of shard nodes (must be >= 3) [1]. - spec.cluster.replicas: Defines the number of replica nodes per shard (must be >= 1) [1]. If you set replicas to 1, each shard will consist of one master and one replica node. KubeDB manages the cluster topology and automatically configures the appropriate master-replica relationships [3].
Citations:
- 1: https://kubedb.com/docs/v2025.6.30/guides/redis/concepts/redis/
- 2: kubedb/apimachinery@f8063159a
- 3: https://appscode.com/blog/post/provisioning-redis-cluster-in-k8s/
🌐 Web query:
KubeDB Redis Cluster replicas PerShard greater than 1 Cluster mode
💡 Result:
In KubeDB, when running Redis in Cluster mode, you can configure the number of replica nodes per shard using the spec.cluster.replicas field [1][2]. According to official KubeDB documentation and repository specifications, the field spec.cluster.replicas allows you to define the number of replicas per shard, and it supports values greater than 1 [1]. If this value is not explicitly set, the KubeDB operator defaults to 2 replicas per shard [1][2]. Key configuration details for Redis Cluster in KubeDB: - spec.mode: Must be set to "Cluster" [1][3]. - spec.cluster.shards: Specifies the number of shards (must be >= 3) [1][2]. - spec.cluster.replicas: Specifies the number of replica nodes per shard [1][2]. Note that while some other Kubernetes operators for Redis (such as the OT-CONTAINER-KIT/redis-operator) use an explicit field named replicasPerShard [4][5], the KubeDB API uses spec.cluster.replicas to achieve the same result [1][2]. Ensure you are referencing the correct API specification for your installed version of KubeDB [1].
Citations:
- 1: https://kubedb.com/docs/v2025.6.30/guides/redis/concepts/redis/
- 2: kubedb/apimachinery@f8063159a
- 3: https://appscode.com/blog/post/provisioning-redis-cluster-in-k8s/
- 4: fix: multiple replicas per shard in RedisCluster OT-CONTAINER-KIT/redis-operator#1835
- 5: OT-CONTAINER-KIT/redis-operator@26223ae
🏁 Script executed:
#!/bin/bash
set -u
echo "== locate failover guide =="
fd -a 'guide.md$' . 2>/dev/null | sed 's#^\./##' | rg 'docs/guides/redis/failover/guide.md|guide.md' || true
echo "== file excerpt lines 70-110 =="
sed -n '70,110p' docs/guides/redis/failover/guide.md || true
echo "== file excerpt lines 370-390 =="
sed -n '370,390p' docs/guides/redis/failover/guide.md || true
echo "== search relevant values in docs =="
rg -n "replicas:\s*[0-9]+|spec\.cluster\.replicas|redis-shard-[0-9]+-[0-9]+" docs/guides docs/examples/redis docs 2>/dev/null | head -n 200 || true
echo "== related cluster example =="
if [ -f docs/examples/redis/update-version/rd-cluster.yaml ]; then
sed -n '1,140p' docs/examples/redis/update-version/rd-cluster.yaml
fi
echo "== search validation text in repo =="
rg -n "greater than one|must be greater|replicasPerShard|PerShard" . 2>/dev/null | head -n 160 || trueRepository: kubedb/docs
Length of output: 28513
🏁 Script executed:
#!/bin/bash
set -u
python3 - <<'PY'
from pathlib import Path
p = Path("docs/guides/redis/failover/guide.md")
text = p.read_text()
checks = {
"manifest_has_replicas_1": "replicas: 1" in text.split("manifest")[0] if "manifest" in text else "replicas: 1" in text,
"cluster_explains_replicas_1_has_master_plus_one_replica": "With `1`, each shard has exactly one master and one replica" in text,
"production_recommendation_says_at_least_one": "spec.cluster.replicas` of at least `1`" in text,
"pod_refs_use_shard0_index_0": "redis-shard0-0" in text,
"other_pod_refs_exist": "redis-shard" in text and ("-1" not in text or "redis-shard*-1" in text),
}
for name, value in checks.items():
print(f"{name}={value}")
for i,line in enumerate(text.splitlines(), 1):
if "redis-shard" in line and ("-1" not in line):
print(f"POD_REF {i}: {line.strip()}")
PY
echo "== concept field doc =="
sed -n '155,168p' docs/guides/redis/concepts/redis.md
echo "== concepts lines in failover guide around replicas =="
rg -n "spec\.cluster\.replicas|replicas:" docs/guides/redis/failover/guide.md docs/guides/redis/concepts/redis.md docs/examples/redis/update-version/rd-cluster.yamlRepository: kubedb/docs
Length of output: 4830
Conform spec.cluster.replicas to the Redis Cluster CRD contract.
docs/guides/redis/concepts/redis.md says spec.cluster.replicas must be greater than 1 and defaults to 2, while this manifest sets replicas: 1. Update the manifest and any production-recommendation text so guide users do not create a Redis Cluster object that conflicts with the documented validation.
[low_effort_and_high reward]
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/guides/redis/failover/guide.md` around lines 77 - 81, Update the Redis
Cluster manifest’s cluster.replicas value from 1 to a value conforming to the
CRD contract, using the documented default or minimum greater-than-one setting.
Also revise any nearby production-recommendation text in the failover guide so
it consistently recommends the valid replica count.
| $ export REDIS_PASSWORD=$(kubectl get secrets -n redis redis-auth -o jsonpath='{.data.password}' | base64 -d) | ||
|
|
||
| $ kubectl exec -it -n redis redis-shard0-0 -c redis -- redis-cli -a "$REDIS_PASSWORD" -c cluster nodes |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== file and occurrences =="
if [ -f docs/guides/redis/failover/guide.md ]; then
sed -n '150,195p' docs/guides/redis/failover/guide.md
echo "== all -a REDIS_PASSWORD occurrences in guide =="
rg -n 'REDIS_PASSWORD|redis-cli -a' docs/guides/redis/failover/guide.md || true
else
echo "guide not found"
fi
echo "== installed redis-cli --help? =="
if command -v redis-cli >/dev/null 2>&1; then
redis-cli --help | sed -n '1,120p' | rg -n 'auth|askpass|REDISCLI|environment' || true
else
echo "redis-cli not in PATH"
fiRepository: kubedb/docs
Length of output: 4342
Sensitive Data Exposure (CWE-214)
Reachability: Internal
Avoid passing the Redis password in redis-cli arguments.
The guide decodes the Kubernetes Secret and passes it with redis-cli -a, which exposes the password in process arguments to anyone who can inspect the host or pod. Use --askpass and stdin for interactive operations, or provide the credential through a protected environment variable or mounted Secret. Update the repeated "$REDIS_PASSWORD" commands on lines 176, 197, 200, 208, 211, 219, 227, 293, 296, and 375.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/guides/redis/failover/guide.md` around lines 174 - 176, Update the Redis
CLI commands throughout the guide, including the repeated "$REDIS_PASSWORD"
usages, to stop passing the password via the -a argument; use redis-cli
--askpass with stdin for interactive operations or the guide’s protected
Secret/environment mechanism, while preserving each command’s existing
operation.
| Because voting requires a majority of the cluster's masters, a single shard's failover doesn't depend on the | ||
| health of the other shards — but the cluster as a whole needs a majority of master nodes reachable for any | ||
| new election to succeed at all. |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "Repo files around doc:"
git ls-files | rg 'docs/guides/redis/failover/guide\.md|docs/guides/redis/failover|failover' || true
echo
echo "Relevant lines from guide:"
sed -n '230,340p' docs/guides/redis/failover/guide.md | nl -ba -v 230
echo
echo "Search for redis-cluster majority/quorum mentions:"
rg -n "majority|quorum|Cluster failover|delete both|two masters|3 master|masters" docs/guides/redis/failover/guide.mdRepository: kubedb/docs
Length of output: 1100
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "Relevant lines from docs/guides/redis/failover/guide.md:"
sed -n '230,350p' docs/guides/redis/failover/guide.md
echo
echo "Redis failover mentions in docs/guides/redis/failover/guide.md:"
sed -n '1,450p' docs/guides/redis/failover/guide.md | grep -nEi "major|quorum|cluster fail|failover|delete|master|replica|auto|case 2|two masters" || true
echo
echo "Search redis Cluster quorum docs:"
python3 - <<'PY'
import urllib.request
urls = [
"https://redis.io/docs/latest/operate/oss_and_stack/reference/cluster-spec/",
"https://redis.io/docs/latest/operate/oss_and_stack/management/failover/",
]
for url in urls:
print(f"\n--- {url} ---")
try:
req = urllib.request.Request(url, headers={"User-Agent":"Mozilla/5.0"})
with urllib.request.urlopen(req, timeout=20) as r:
data = r.read().decode("utf-8", "replace")
for needle in ["majority of masters", "quorum", "failover", "minimum"]:
idx = data.find(needle)
if idx != -1:
start = max(0, idx-800)
end = min(len(data), idx+800)
print(f"\n>>> {needle} at {idx}:")
print(data[start:end].replace("\n", " ")[:2000])
except Exception as e:
print(f"ERROR: {e}")
PYRepository: kubedb/docs
Length of output: 12388
🌐 Web query:
Redis Cluster failover cluster node majority of masters quorum require majority replica reachable
💡 Result:
Redis Cluster requires a majority of master nodes to be reachable to perform an automatic failover and to maintain cluster availability [1][2][3]. Key aspects of the quorum requirement: 1. Automatic Failover: When a master node fails, its replica attempts to initiate a failover [4]. To succeed, the replica must be elected as the new master. This election requires the replica to obtain authorization from a majority of the existing master nodes in the cluster [5][6][7]. A quorum is defined as floor(total_masters / 2) + 1 [4]. 2. Failure Detection: The cluster uses a gossip protocol to detect failures [1][4]. For a master to be officially marked as failed (FAIL state), a majority of the masters must report it as unreachable (PFAIL state) [8][4]. If this majority is not reached, the FAIL flag is not set, and no automatic failover is triggered [8][1]. 3. Cluster Availability: If a node cannot reach a majority of the master nodes for the duration of the cluster-node-timeout, it will stop accepting queries [6]. Furthermore, if the cluster cannot maintain full coverage of its hash slots—which occurs if a primary node is unreachable and cannot be failed over—the cluster enters a CLUSTERDOWN state and rejects all commands [2][6]. 4. Manual Failover: Standard manual failover (CLUSTER FAILOVER) also requires the majority of masters to be available to authorize the process [5][7]. The TAKEOVER option is the only mechanism that allows a replica to be promoted to a master without cluster-wide consensus, typically used during significant partitions or data center switches when a majority of masters are unavailable [5][7]. Adding replicas does not change the quorum requirement; it remains dependent on the number of master nodes [2]. Consequently, a cluster with 3 masters requires at least 2 reachable masters to remain operational and perform failovers [2][3].
Citations:
- 1: https://redis.io/docs/latest/operate/oss_and_stack/reference/cluster-spec/
- 2: https://oneuptime.com/blog/post/2026-03-31-redis-what-does-clusterdown-the-cluster-is-down-mean-in-redis/view
- 3: https://oneuptime.com/blog/post/2026-03-31-redis-troubleshoot-redis-cluster-partition-tolerance/view
- 4: https://tools.yiteai.com/en/books/redis/ch20
- 5: https://redis.io/docs/latest/commands/cluster-failover/
- 6: https://redis.io/docs/latest/operate/oss_and_stack/management/scaling/
- 7: https://redis.antirez.com/commands/content/commands/cluster-failover.md
- 8: https://github.com/redis/redis/blob/6.0/src/cluster.c
Remove or redesign Case 2.
A three-master Redis Cluster cannot auto-fail over both deleted masters at once. Failover requires a majority of masters (>3/2) to approve the election; deleting redis-shard1-0 and redis-shard2-0 leaves only one master, which is below the quorum. Delete one master for this test, or use a topology with enough reachable masters plus reachable replicas.
🧰 Tools
🪛 LanguageTool
[style] ~249-~249: ‘a majority of the’ might be wordy. Consider a shorter alternative.
Context: ...he new master. Because voting requires a majority of the cluster's masters, a single shard's fai...
(EN_WORDINESS_PREMIUM_A_MAJORITY_OF_THE)
[style] ~250-~250: ‘a majority of’ might be wordy. Consider a shorter alternative.
Context: ...ards — but the cluster as a whole needs a majority of master nodes reachable for any new elec...
(EN_WORDINESS_PREMIUM_A_MAJORITY_OF)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/guides/redis/failover/guide.md` around lines 249 - 251, Redesign Case 2
in the Redis failover guide so it does not claim that a three-master cluster can
automatically fail over two deleted masters. Either remove the two-master
deletion scenario or update it to delete only one master; if retaining
simultaneous failures, use a topology with enough reachable masters and replicas
to satisfy quorum.
| Confirm the data survived the promotion — both the new master and the recovered old master (now a replica) | ||
| serve it correctly with `-c`: | ||
|
|
||
| ```bash | ||
| $ kubectl exec -it -n redis redis-shard0-1 -c redis -- redis-cli -a "$REDIS_PASSWORD" -c get hello | ||
| "world" | ||
|
|
||
| $ kubectl exec -it -n redis redis-shard0-0 -c redis -- redis-cli -a "$REDIS_PASSWORD" -c get hello | ||
| "world" | ||
| ``` |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Check whether the referenced file exists and inspect the surrounding context.
if [ -f docs/guides/redis/failover/guide.md ]; then
echo "== file exists =="
wc -l docs/guides/redis/failover/guide.md
echo "== lines 230-320 =="
sed -n '230,320p' docs/guides/redis/failover/guide.md | nl -ba -v230
else
echo "file not found: docs/guides/redis/failover/guide.md"
fi
echo "== nearby 'data survived' / failover instructions =="
rg -n "data survived|acknowledged|replica|master|failover|delete|pod|writes|RPO|rpo|redis-cli get hello|get hello" docs/guides/redis/failover/guide.md || true
echo "== failover guide files =="
fd -a 'guide\.md|README\.md|.*failover.*|.*redis.*' docs/guides/redis 2>/dev/null | sed 's#^\./##' | head -80Repository: kubedb/docs
Length of output: 267
🌐 Web query:
Redis Cluster asynchronous replication failover can lose acknowledged writes documentation
💡 Result:
Yes, Redis Cluster can lose acknowledged writes during a failover because it uses asynchronous replication [1][2]. Because Redis Cluster does not guarantee strong consistency, a write acknowledged by a primary node is not guaranteed to have been persisted to any of its replicas at the time of acknowledgement [2][3]. If a primary node fails and is unreachable, a replica may be promoted to primary [4][5]. If that promoted replica did not receive the specific write before the original primary crashed, that write is lost permanently [1][2]. Key details regarding this behavior include: Asynchronous Replication Window: Redis Cluster prioritizes performance by not waiting for replicas to acknowledge writes before confirming success to the client [1][2]. While Redis typically attempts to propagate writes to replicas at nearly the same time it acknowledges them to the client, a small window of vulnerability always exists [1][2]. Mitigation with WAIT: You can reduce the probability of losing acknowledged writes by using the WAIT command [2][3]. This command forces the primary to wait for a specified number of replicas to acknowledge the write before returning success [1][2]. However, even with WAIT, Redis Cluster does not become a CP (consistent and partition-tolerant) system with strong consistency [3]. In complex failure scenarios, it remains possible for a replica that did not receive the write to be elected as the new primary [2][3]. Failover Mechanisms: - Automatic Failover: Triggered by cluster failure detection (Gossip protocol, PFAIL/FAIL states), this process involves electing the replica with the most up-to-date replication offset to minimize data loss [4][5]. - Manual Failover: The CLUSTER FAILOVER command (without the TAKEOVER option) coordinates a failover by pausing the primary and waiting for a replica to fully consume the replication stream before promotion, which is designed to prevent data loss [6][4][7]. This is safer than an automatic, emergency failover but requires the primary to be reachable for the coordination process [6][4]. In summary, because Redis Cluster is designed for high availability and performance, acknowledged writes can be lost if a primary failure occurs before asynchronous replication completes [1][4][2]. The WAIT command helps reduce this risk but does not eliminate it entirely [2][3].
Citations:
- 1: https://redis.io/docs/latest/operate/oss_and_stack/reference/cluster-spec/
- 2: https://redis.io/docs/latest/operate/oss_and_stack/management/scaling/
- 3: https://redis.io/docs/latest/operate/oss_and_stack/management/replication/
- 4: https://www.jusdb.com/blog/redis-cluster-sharding-replication-ha
- 5: https://www.besthub.dev/articles/redis-cluster-deep-dive-sharding-replication-failover-smart-client-3ee71ec42d31
- 6: https://redis.io/docs/latest/commands/cluster-failover/
- 7: https://redis.io/docs/latest/commands/failover/
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== lines 230-320 of docs/guides/redis/failover/guide.md =="
python3 - <<'PY'
from pathlib import Path
p = Path("docs/guides/redis/failover/guide.md")
lines = p.read_text().splitlines()
for i in range(229, min(len(lines), 320)):
print(f"{i+1:4d}: {lines[i]}")
PY
echo "== nearby RPO/cluster failover/write wording =="
python3 - <<'PY'
from pathlib import Path
text = Path("docs/guides/redis/failover/guide.md").read_text()
terms = ["RPO", "RPL", "rpo", "async", "asynchronous", "data survived", "failover", "primary", "replica", "deleted", "pod ", "set hello", "get hello", "cluster failover", "WAIT"]
for term in terms:
idx = text.find(term)
if idx != -1:
start = max(0, text.rfind("\n", 0, idx-200))
end = text.find("\n", idx+1000)
snippet = text[start:end]
print(f"\n--- first occurrence of {term!r} at {idx+1} ---")
print(snippet[:1200])
PY
echo "== all set/get hello occurrences =="
python3 - <<'PY'
from pathlib import Path
for i, line in enumerate(Path("docs/guides/redis/failover/guide.md").read_text().splitlines(), 1):
if "hello" in line or "redis-cli" in line:
print(f"{i}: {line}")
PYRepository: kubedb/docs
Length of output: 14487
🌐 Web query:
Redis Cluster reference asynchronous replication failover can lose committed writes data integrity
💡 Result:
Yes, it is possible for Redis Cluster to lose committed writes during a failover because it uses asynchronous replication by default [1][2]. In Redis Cluster, a master node acknowledges a write to the client as soon as it is processed locally, before the write has been propagated to its replicas [2][3]. Because replication is asynchronous, there is a time window where a master may fail after acknowledging a write to a client but before that write reaches its replicas [1][4][5]. If the master remains unreachable and a replica—which has not yet received the most recent data—is promoted to master, those writes are lost [1][2][5]. Key factors and mitigations regarding this behavior include: 1. Nature of the System: Redis Cluster is designed for high performance and availability rather than strict, system-wide strong consistency [2][6]. It does not guarantee that every acknowledged write is durably stored across all nodes before returning success to the client [2][5]. 2. Synchronous Write Options: To reduce the likelihood of data loss, Redis provides the WAIT command [2][3]. This allows clients to wait for a specified number of replicas to acknowledge receipt of the writes [3]. However, even with the use of WAIT, Redis Cluster does not become a CP (consistent and partition-tolerant) system with strong consistency; under complex failure scenarios (e.g., if a replica that did not receive the write is elected as master), data loss can still occur [2][3]. 3. Failover Scenarios: - Automatic Failover: Triggered when a master is unreachable for a duration exceeding cluster-node-timeout. During this, a replica is elected based on the most up-to-date replication offset, which helps minimize, though does not eliminate, the risk of data loss [4][7]. - Manual Failover: The CLUSTER FAILOVER command (without the TAKEOVER option) is designed to be safe [8][9]. It pauses the master and waits for the replica to fully consume the replication stream before promoting it, thereby avoiding data loss during the transition [8][4][9]. - Force/Takeover Failover: Commands like CLUSTER FAILOVER TAKEOVER bypass the safety checks and handshakes typically used during a graceful failover. These can result in data loss or configuration inconsistencies and should generally be used only when standard recovery is impossible [8][4][9].
Citations:
- 1: https://redis.io/docs/latest/operate/oss_and_stack/reference/cluster-spec/
- 2: https://redis.io/docs/latest/operate/oss_and_stack/management/scaling/
- 3: https://redis.io/docs/latest/operate/oss%5Fand%5Fstack/management/replication/
- 4: https://www.jusdb.com/blog/redis-cluster-sharding-replication-ha
- 5: https://www.c-sharpcorner.com/article/redis-cluster-failure-modes-and-recovery-behavior-in-production/
- 6: https://www.antirez.com/news/62
- 7: https://www.besthub.dev/articles/redis-cluster-deep-dive-sharding-replication-failover-smart-client-3ee71ec42d31
- 8: https://redis.io/docs/latest/commands/cluster-failover/
- 9: https://redis.antirez.com/commands/content/commands/cluster-failover.md
🌐 Web query:
site:redis.io/cluster-spec asynchronous replication acknowledged writes lost failover
💡 Result:
Redis Cluster uses asynchronous replication, so a master may acknowledge a write before replicas receive it. If the master fails and a replica that missed the write is promoted, the acknowledged write can be lost permanently. (redis.io)
WAIT can require acknowledgment from replicas, but it does not provide strong consistency or guarantee that acknowledged writes survive failover. (redis.io)
Citations:
- 1: https://redis.io/docs/latest/operate/oss_and_stack/management/scaling/?utm_source=openai
- 2: https://redis.io/docs/latest/operate/oss_and_stack/management/replication/?utm_source=openai
Qualify the data-survival statement.
Redis Cluster uses asynchronous replication, so an acknowledged write can be lost if the master fails before the replica receives it. Reading hello after promotion only proves that key survived this run. Add an RPO warning and write directly before pod deletion.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/guides/redis/failover/guide.md` around lines 289 - 298, Update the
failover verification section around the “Confirm the data survived the
promotion” instructions to state that Redis replication is asynchronous and the
surviving key only demonstrates this run, not guaranteed zero data loss; add an
explicit RPO warning. Insert the `hello` write command directly before the
pod-deletion step so the documented test sequence clearly establishes when the
data is created.
| #### Case 4: Delete both pods of a shard at once | ||
|
|
||
| Since this deployment has only 1 replica per shard, deleting both wipes out the shard entirely — there's no | ||
| surviving copy of that shard's data anywhere in the cluster to fail over to. | ||
|
|
||
| ```bash | ||
| $ kubectl delete pod -n redis redis-shard1-0 redis-shard1-1 | ||
| pod "redis-shard1-0" deleted | ||
| pod "redis-shard1-1" deleted | ||
| ``` | ||
|
|
||
| Shard 1's hash slots become unowned until at least one of the two pods comes back and recovers its data | ||
| from its PVC (check `cluster nodes` to see exactly which range that shard was assigned): | ||
|
|
||
| ```bash | ||
| $ kubectl exec -it -n redis redis-shard0-0 -c redis -- redis-cli -a "$REDIS_PASSWORD" -c cluster info | grep cluster_state | ||
| cluster_state:fail | ||
| ``` | ||
|
|
||
| Once the PetSet brings a pod back with its PVC reattached, it resumes ownership of its previously assigned | ||
| slots and `cluster_state` returns to `ok`. This case is why production clusters should run with | ||
| `spec.cluster.replicas` of at least `1` spread across failure domains (e.g. different nodes/zones via | ||
| pod anti-affinity), so a single node loss can never take out every copy of a shard. |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== locate file and relevant docs =="
git ls-files | grep -F 'docs/guides/redis/failover/guide.md' || true
if [ -f docs/guides/redis/failover/guide.md ]; then
wc -l docs/guides/redis/failover/guide.md
sed -n '330,405p' docs/guides/redis/failover/guide.md
fi
echo "== search for WipeOut and PVC references in docs =="
if [ -f docs/guides/redis/failover/guide.md ]; then
rg -n "WipeOut|deletionPolicy|PVC|persist" docs/guides/redis/failover/guide.md
fi
echo "== search code/docs for RedisObject deletion or failover PVC semantics =="
rg -n "deletionPolicy|WipeOut|cluster replicas|RedisObject|spec\.cluster\.replicas|failover|PVC" docs . --glob '!node_modules' --glob '!dist' --glob '!build' 2>/dev/null | head -200Repository: kubedb/docs
Length of output: 50368
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== guide section =="
wc -l docs/guides/redis/failover/guide.md
sed -n '340,390p' docs/guides/redis/failover/guide.md
echo "== targeted WipeOut/deletionPolicy references in guided docs =="
rg -n "WipeOut|deletionPolicy|deletion policy|delete Redis|delete pod|PVC|pod anti-affinity|replicas" docs --glob '*.md' --max-count 200
echo "== Kubernetes/PVC behavior probe from documentation text only =="
python3 - <<'PY'
from pathlib import Path
p = Path('docs/guides/redis/failover/guide.md')
text = p.read_text()
lines = text.splitlines()
for needle in ["wipes out the shard entirely", "PVC", "deletionPolicy"]:
hits = [(i, line) for i,line in enumerate(lines,1) if needle in line]
print(f"{needle}: {len(hits)}")
for i,line in hits[:10]:
print(f"{i}: {line}")
PYRepository: kubedb/docs
Length of output: 50368
🏁 Script executed:
#!/bin/bash
set -euo pipefail
python3 - <<'PY'
from pathlib import Path
p = Path('docs/guides/redis/failover/guide.md')
lines = p.read_text().splitlines()
for needle in ["wipes out the shard entirely", "PVC", "deletionPolicy", "WipeOut"]:
print(f"\n{needle}:")
hits = [(i, line) for i, line in enumerate(lines, 1) if needle in line]
print(f"count={len(hits)}")
for i, line in hits[:20]:
print(f"{i}: {line}")
print("\nlines 360-382:")
for i in range(360, 383):
if i <= len(lines):
print(f"{i}: {lines[i-1]}")
PYRepository: kubedb/docs
Length of output: 2257
Distinguish shard unavailability from permanent data loss.
Deleting both pods removes the live master and replica, but it does not delete their persistent PVCs. Replace “wipes out the shard entirely” with wording that states the shard has no live failover copy and becomes unavailable until a PVC-backed pod recovers. A shard is permanently wiped out only when the Redis object is deleted with deletionPolicy: WipeOut, not when pods are deleted.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/guides/redis/failover/guide.md` around lines 360 - 382, The Case 4
documentation currently implies that deleting both pods permanently destroys the
shard. Update the opening explanation to state that no live failover copy
remains and the shard becomes unavailable until a PVC-backed pod recovers, while
clarifying that permanent deletion requires removing the Redis object with
deletionPolicy: WipeOut; preserve the existing recovery instructions.
|
Visit the preview URL for this PR (updated for commit 860444b): https://kubedb-v2-hugo--pr1037-failover2-y11c2qmv.web.app (expires Thu, 20 Aug 2026 06:05:22 GMT) 🔥 via Firebase Hosting GitHub Action 🌎 Sign: 0f29ae8ae0bd54a99bf2b223b6833be47acd5943 |
1 participant
Summary by CodeRabbit