Mongo ReplicaSet instance support by shivaprasadmb · Pull Request #244 · nutanix-cloud-native/ndb-operator

shivaprasadmb · 2026-06-19T08:57:00Z

What this PR does / why we need it:
This PR is for adding support for Mongo HA (ReplicaSet type) via NDB Operator.

How Has This Been Tested?:
A mongo HA replicaset instance would be provisioned and one sample mongo-app pod would be run inside k8s cluster by using the secret connection string generated by operator.
There are three test scenarios conducted:

Provision mongo HA cluster with arbiter and check the headless services and secrets it created.
it excluded arbiter nodes and utilized other nodes.
database CR yaml manifest

apiVersion: ndb.nutanix.com/v1alpha1
kind: Database
metadata:
  name: k8s-mongo-ha-arb
  namespace: default
spec:
  ndbRef: ndb

  databaseInstance:
    name: "K8s_MONGO_HA_ARB"
    description: "[NDB-Operator] MongoDB HA Replica Set with Arbiter"
    type: mongodb

    credentialSecret: mongo-ha-secret
    timezone: "UTC"
    size: 100                           # disk size in GiB (applied to data nodes only)

    # Top-level cluster is required by the webhook even for HA (individual node placement is in haConfig.nodes[]).
    clusterName: "auto_cluster_nested_6a4b404a360236a2720ac564"

    databaseNames:
      - mongo-ha-db-arb

    profiles:
      software:
        name: "MONGODBHA_PROFILE"       # replace with actual NDB MongoDB software profile name
      compute:
        name: "DEFAULT_HA_COMPUTE"             # replace with actual compute profile name
      network:
        name: "MONGODBHA_VLAN"    # replace with actual network profile name
      dbParam:
        name: "DEFAULT_MONGODB_PARAMS"            # replace with actual db parameter profile name

    timeMachine:
      name: "K8s_MONGO_HA_ARB_TM"
      description: "[NDB-Operator] TM for MongoDB HA with Arbiter"
      sla: "DEFAULT_OOB_BRASS_SLA"
      dailySnapshotTime: "10:00:00"
      snapshotsPerDay: 1
      logCatchUpFrequency: 30
      weeklySnapshotDay: "THURSDAY"
      monthlySnapshotDay: 11

    haConfig:
      clusterName: "K8s_MONGO_HA_ARB_cluster"
      mongodb:
        replicaSetName: "rs0"
        replicaSetDescription: "K8s MongoDB HA Replica Set with Arbiter"
        deployArbiter: true
        # arbiterComputeProfileId: "..."   # optional: smaller profile for the arbiter VM
        listenerPort: 27017
      nodes:
        - vmName: "mongo-rs1"
          nodeType: "database"
          role: "primary"
          clusterName: "auto_cluster_nested_6a4b404a360236a2720ac564"   # PE cluster 1
        - vmName: "mongo-rs2"
          nodeType: "database"
          role: "secondary"
          clusterName: "auto_cluster_nested_6a171033360236320a01a757"   # PE cluster 2
        - vmName: "mongo-rs-arb"
          nodeType: "arbiter"
          clusterName: "auto_cluster_nested_6a171033360236320a01a757"   # PE cluster 1

2) Provision mongo HA cluster without arbiter node, it should create headless svc for all nodes. database cr yaml manifest

apiVersion: ndb.nutanix.com/v1alpha1
kind: Database
metadata:
  name: k8s-mongo-ha
  namespace: default
spec:
  ndbRef: ndb

  databaseInstance:
    name: "K8s_MONGO_HA"
    description: "[NDB-Operator] MongoDB HA Replica Set"
    type: mongodb

    credentialSecret: mongo-ha-secret
    timezone: "UTC"
    size: 100                           # disk size in GiB

    # Top-level cluster is required by the webhook even for HA (individual node placement is in haConfig.nodes[]).
    clusterName: "auto_cluster_nested_6a4b404a360236a2720ac564"

    databaseNames:
      - mongo-ha-db

    profiles:
      software:
        name: "MONGODBHA_PROFILE"       # replace with actual NDB MongoDB software profile name
      compute:
        name: "DEFAULT_HA_COMPUTE"             # replace with actual compute profile name
      network:
        name: "MONGODBHA_VLAN"    # replace with actual network profile name
      dbParam:
        name: "DEFAULT_MONGODB_PARAMS"           # replace with actual db parameter profile name

    timeMachine:
      name: "K8s_MONGO_HA_TM"
      description: "[NDB-Operator] TM for MongoDB HA"
      sla: "DEFAULT_OOB_BRASS_SLA"
      dailySnapshotTime: "10:00:00"
      snapshotsPerDay: 1
      logCatchUpFrequency: 30
      weeklySnapshotDay: "THURSDAY"
      monthlySnapshotDay: 11

    haConfig:
      clusterName: "K8s_MONGO_HA_cluster"
      mongodb:
        replicaSetName: "rs0"
        replicaSetDescription: "K8s MongoDB HA Replica Set"
        deployArbiter: false
        listenerPort: 27017
      nodes:
        - vmName: "mongo-rs1"
          nodeType: "database"
          role: "primary"
          clusterName: "auto_cluster_nested_6a4b404a360236a2720ac564"   # PE cluster 1
        - vmName: "mongo-rs2"
          nodeType: "database"
          role: "secondary"
          clusterName: "auto_cluster_nested_6a171033360236320a01a757"   # PE cluster 2
        - vmName: "mongo-rs3"
          nodeType: "database"
          role: "secondary"
          clusterName: "auto_cluster_nested_6a171033360236320a01a757"   # PE cluster 2
        - vmName: "mongo-rs4"
          nodeType: "database"
          role: "secondary"
          clusterName: "auto_cluster_nested_6a4b404a360236a2720ac564"   # PE cluster 1

3) Provision mongo HA without arbiter node, simulate re-election and check if mongo-app is able to recognize which node is primary. This screenshot shows logging into primary node and performing re-election

This screenshot shows mongo-app pod logs before changing and after changing primary node.

Special notes for your reviewer:
Mongo ReplicaSet instance contains of primary, secondary nodes and optional arbiter node. Primary node would receive write/read traffic and secondary nodes would be used for read only traffic. There is asynchronous replication happening from primary to secondary nodes. Arbiter node is optional as it doesn't hold any actual data but its helps in leader election incase primary node goes down. Use arbiter node for smaller workloads (to save infra costs) and not for production workload.

Mongo driver is used in application code to connect to its DBs, this driver contains SDAM (service discovery and monitoring) which monitors health of nodes in the cluster and helps in traffic routing and automation failover scenario.
Since SDAM handles which requests should go to primary and secondary nodes, we don't need to create separate read and write endpoints. We will have a single connection URI string used in application code.
Users would require a ready-to-use connection string when DB is provisioned, they don't need to make much changes in code to use this string. This should be stored securely and be accessible to code.
NDB-Operator can generate connection URI string which consists of user, password, node IPs and replicaSet name. This would be stored in K8s secret object and become accessible to users.
SDAM component uses hostnames to resolve and route traffic, since we need to resolve the hostnames we need to create headless service for each node so that it becomes a raw DNS record in coreDNS.
we create headless service instead of normal service object because when normal service object is used then it gets a clusterIP which is different from actual node IPs. kube-rbac-proxy intercepts the traffic to resolve service name to actual IPs which messes the flow, hence headless service would be used.

sasikanthmasini · 2026-07-09T08:38:52Z

+	for _, n := range dataNodes {
+		hostParts = append(hostParts, fmt.Sprintf("%s:%d", n.hostname, port))
+	}
+	uri := fmt.Sprintf("mongodb://%s:%s@%s/%s?replicaSet=%s&authSource=admin",


Can we URL-encode username/password in Mongo URI. Special chars like @ or : can break authentication.

sasikanthmasini · 2026-07-09T08:39:23Z

+	haConfig := database.Spec.Instance.HAConfig
+	port := haConnectivityManagers[common.DATABASE_TYPE_MONGODB].PrimaryPort(haConfig)
+	rsName := haConfig.MongoDB.ReplicaSetName
+	dbName := strings.Join(database.Spec.Instance.DatabaseNames, ",")


Mongo URI path should have only one DB name. Comma-joined names can cause auth failure.

sasikanthmasini · 2026-07-09T08:39:53Z

+	// Query NDB for the current set of database nodes (hostname + IP).
+	// This is a targeted GET on the cluster ID — the same ID that was set when
+	// the provisioning operation completed and stored in database.Status.Id.
+	dbResponse, err := ndb_api.GetDatabaseById(ctx, ndbClient, database.Status.Id)


This NDB API call runs every reconcile for READY DBs. Please avoid repeated calls by reading from status/cache.

sasikanthmasini · 2026-07-09T08:40:17Z

+
+	// Create or reconcile a headless Service + Endpoints for each data node.
+	for _, n := range dataNodes {
+		nn := types.NamespacedName{Name: n.hostname, Namespace: database.Namespace}


Service name uses only hostname; this can collide across DBs in same namespace. Please add DB scoping or ownerRef check.

the headless services must match the VM hostname used in mongoDB's rs.conf().
We can enforce a rule saying that each mongo HA DB resource should be created in its own namespace.

We can't add DB scoping here as VM names should be just the node names and not like ..svc.local
What we can do is enforce a rule saying that each mongo HA resource should be having its own namespace.

sasikanthmasini · 2026-07-09T08:41:36Z

 		"restart_mongod": "true",
 		"working_dir":    "/tmp",
-		"db_user":        "admin",
+		"db_user":        "mongod",


This may change default Mongo user globally. Please keep shared default as "admin" and override only for Mongo HA.

sasikanthmasini · 2026-07-09T08:42:17Z

+	// create headless services for MongoDB HA replicaset type
+	if database.Spec.Instance != nil &&
+		database.Spec.Instance.HAConfig != nil &&
+		database.Spec.Instance.HAConfig.MongoDB != nil {


Please use database.IsMongoHA() here for proper type check and consistency.

sasikanthmasini · 2026-07-09T08:42:44Z

+		for _, n := range haConfig.Nodes {
+			node := Node{
+				VmName:           n.VmName,
+				NxClusterId:      n.ClusterId,


clusterName-only configs may fail since only ClusterId is used. Please resolve clusterName - clusterId or validate strictly.

This is already handled — ResolveNamesToUUIDs in controller_adapters/name_resolution.go
this resolves clusterName to clusterId before reaching this state.

sasikanthmasini · 2026-07-09T08:43:15Z

+		}
+	}
+
+	if len(haConfig.Nodes) > 0 && primaryCount != 1 {


Please enforce minimum Mongo HA node count.

sasikanthmasini · 2026-07-09T08:43:59Z

+			"cluster_name":        haConfig.ReplicaSetName,
+			"cluster_description": haConfig.ReplicaSetDescription,
+			// listener_port: use the configured port (may differ from 27017 if overridden in CRD)
+			"listener_port":  strconv.Itoa(int(haConfig.MongoListenerPort)),


Please use PrimaryPort() fallback here too, so listener_port is never sent as 0.

sasikanthmasini · 2026-07-09T08:44:49Z

+	// inside the driver matches the names in rs.conf() and CoreDNS can resolve them.
+	// e.g.: mongodb://user:pass@mongo-rs1:27017,mongo-rs2:27017/dbname?replicaSet=rs0&authSource=admin
+	var hostParts []string
+	for _, n := range dataNodes {


Services/endpoints look create-only; please clean up stale ones when nodes are removed.

Yes we ought to add the logic for this. Shall we have another PR for this change as the current PR is big.

sasikanthmasini

Added Comments

mazin-s · 2026-08-04T21:09:37Z

+	// When true, exactly one node with nodeType "arbiter" must be present in nodes[].
+	// When false (default), no arbiter nodes may be present — all nodes are data-bearing.
+	// +optional
+	DeployArbiter bool `json:"deployArbiter,omitempty"`


Is deployArbiter, and listener port defaulted in web-hooks or run time code?

Fixed, this is defaulted in webhook!

mazin-s · 2026-08-04T21:10:45Z

@@ -109,6 +112,74 @@ func (v *MysqlHAParamsValidator) Validate(haConfig *InstanceHAConfig, haPath *fi
 	}
 }



What do you think about breaking down these validators into corresponding database files? e.g monogdbvalidaotr, postgres, etc. Its fine you dont think its needed, just wanted to ask.

Oh, i need this only for HA type of database instance. Keeping them here would be fine.

mazin-s · 2026-08-04T21:11:36Z

+
+	HA_NODE_TYPE_ARBITER = "arbiter"
+
+	HA_NODE_ROLE_MONGO_PRIMARY   = "primary"


we do have an existing "primary or "secondary constant we can just use,

Actually other DBs refer them as "Primary/Secndary" and not "primary/secondary". Capital letter difference 😄

mazin-s · 2026-08-04T21:12:02Z

+	// A replica set needs at least 3 members to form a quorum (e.g. PSS or PSA topology).
+	HA_MONGO_MIN_NODE_COUNT = 3
+
+	HA_NODE_TYPE_ARBITER = "arbiter"


why do you have HA_NODE_TYPE_ARBITER = "arbiter" and HA_NODE_ROLE_MONGO_ARBITER = "arbiter

HA_NODE_TYPE_ARBITER variable is used for CRD spec validation when user defines mongo HA CR.
HA_NODE_ROLE_MONGO_ARBITER is used to populate field in provision API call.
keeping them distinct is better because both serve different purpose.

mazin-s · 2026-08-04T21:13:03Z


+	HA_MONGO_DEFAULT_LISTENER_PORT = int32(27017)
+	// A replica set needs at least 3 members to form a quorum (e.g. PSS or PSA topology).
+	HA_MONGO_MIN_NODE_COUNT = 3


can you link a doc explaining where you got this please

i searched through online resources, we keep odd number for leader elections.
For HA cluster min odd number would be 3.

mazin-s · 2026-08-04T21:17:46Z

+}
+
 // Database related info to be stored in the status field of the NDB CR
 type NDBServerDatabaseInfo struct {


This class is database agnostic, we should avoid adding in database specific info here, is there another way we can do this?

Yeah I'm storing here because most of the status fields in DB CR is fetched from NDB CR status fields.

If we want to store in DB CR then we should have a separate NDB API call every reconcile which would make the operator feel loaded.

We are anyways fetching IP address from NDB CR reconcile and populating it in DB CR status field. Witht eh same API call we can fetch hostnames and store here for DB reconcile logic. This field is specific for mongo HA alone and its optional.

shivaprasadmb requested review from krunal-jhaveri, manavrajvanshi and sasikanthmasini June 19, 2026 08:57

shivaprasadmb marked this pull request as draft June 19, 2026 08:58

shivaprasadmb force-pushed the ERA-66338 branch from 494abc7 to 78e0998 Compare July 7, 2026 12:23

shivaprasadmb marked this pull request as ready for review July 7, 2026 12:23

shivaprasadmb changed the title ~~Draft: Mongo ReplicaSet instance support~~ Mongo ReplicaSet instance support Jul 8, 2026

sasikanthmasini mentioned this pull request Jul 9, 2026

ERA-66338: Add MongoDB HA (ReplicaSet) instance support #246

Draft

nutanix-cloud-native deleted a comment from cursor Bot Jul 9, 2026

sasikanthmasini reviewed Jul 9, 2026

View reviewed changes

shivaprasadmb force-pushed the ERA-66338 branch from e7261f7 to 7d400e2 Compare July 11, 2026 10:18

shivaprasadmb force-pushed the ERA-66338 branch from 7d400e2 to bf455c9 Compare July 14, 2026 02:59

mazin-s reviewed Aug 4, 2026

View reviewed changes

shivaprasadmb added 4 commits August 6, 2026 12:59

initial changes: only k8s secret

d2027f4

changes 2

70154ea

honoured PR comments

1195053

honored PR comments 2

fc0e28f

shivaprasadmb force-pushed the ERA-66338 branch from abbfb11 to 7b6e0b2 Compare August 6, 2026 07:31

vulnfix

ff86707

shivaprasadmb force-pushed the ERA-66338 branch from 7b6e0b2 to ff86707 Compare August 6, 2026 07:46

		@@ -109,6 +112,74 @@ func (v MysqlHAParamsValidator) Validate(haConfig InstanceHAConfig, haPath *fi
		}
		}


		HA_NODE_TYPE_ARBITER = "arbiter"

		HA_NODE_ROLE_MONGO_PRIMARY = "primary"

Uh oh!

Conversation

shivaprasadmb commented Jun 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shivaprasadmb Jul 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sasikanthmasini left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

shivaprasadmb commented Jun 19, 2026 •

edited

Loading

shivaprasadmb Jul 11, 2026 •

edited

Loading