Add and remove hosts

This section describes dynamic host scaling and host management for MarkLogic Operator clusters.

Dynamic host scaling (Operator)

Dynamic host scaling for the Operator is controlled by changing markLogicGroups[].replicas in your MarklogicCluster custom resource and applying the updated resource.

This feature is available with MarkLogic Operator 1.3 and requires MarkLogic Server 12.0 or later.

Dynamic groups vs. standard groups

A dynamic group is an evaluator-only group declared with isDynamic: true in the MarklogicCluster custom resource. Dynamic groups differ from standard groups in the following ways:

• No datadir persistence: Standard MarkLogic datadir persistence is disabled by default. Dynamic hosts do not manage forests directly.
• Token-based join/remove: The operator uses dynamic host tokens to join pods to the MarkLogic cluster and remove them, rather than relying on the standard host join mechanism.
• Evaluator-only: Dynamic hosts serve as evaluators (query processors) and do not host forests.
• Automatic lifecycle management: The operator fully manages dynamic host membership, including rejoining after restarts.

Define a dynamic group

To declare a dynamic group, set isDynamic: true on a non-bootstrap group in your manifest. Dynamic groups must not be bootstrap groups:

ApiVersion: marklogic.progress.com/v1
kind: MarklogicCluster
metadata:
   name: my-cluster
   namespace: marklogic
spec:
   image: "progressofficial/marklogic-db:12.0.3-ubi9-rootless-2.2.6"
   auth:
     secretName: my-super-secret
   markLogicGroups:
   - replicas: 2
     name: dnode
     groupConfig:
         name: dnode
         enableXdqpSsl: true
     persistence:
         enabled: true
         size: 100Gi
     isBootstrap: true
   - replicas: 2
     name: enode
     groupConfig:
         name: enode
         enableXdqpSsl: true
     isDynamic: true

When you apply this manifest, the operator:

1. Creates a child MarkLogicGroup resource for the dynamic group.
2. Creates a dynamic StatefulSet and associated Service resources.
3. Configures the target MarkLogic group for dynamic hosts.

Prerequisites

Before you scale:

1. Confirm your target group and namespace.
2. Confirm forest placement and failover readiness for any hosts being removed.
3. Plan persistent volume cleanup only if you are permanently removing hosts.
4. Ensure at least one bootstrap group (with isBootstrap: true) exists in your cluster.
5. Dynamic groups require MarkLogic Server 12.0 or later.

API and configuration reference for dynamic groups

Dynamic group spec fields

When declaring a dynamic group, use these spec fields:

Validation rules and constraints

The operator enforces the following rules for dynamic groups:

1. isDynamic immutability: Once a group is created with isDynamic: true or false, you cannot change this value.
2. Bootstrap exclusion: isDynamic: true requires isBootstrap: false. Dynamic groups cannot be bootstrap groups.
3. No forests: Dynamic groups must not be assigned to or reused from MarkLogic groups that already host forests.
4. Bootstrap requirement: At least one bootstrap group must exist in the cluster spec when any dynamic group is defined.
5. Unique group names: Group names must be unique across both static and dynamic groups within a cluster.

Storage and persistence options for dynamic groups

Dynamic groups support two storage models for the main MarkLogic datadir, each with different lifecycle semantics:

EmptyDir (default)

Configuration:

• persistence.enabled: false or omit the entire persistence block

Characteristics:

• Storage is ephemeral and local to each Kubernetes node.
• Lost when a pod is deleted or evicted.
• No PVC dependencies; works on any Kubernetes cluster without storage classes.
• Suitable for stateless, burst query capacity.

Scale-down behavior: When you reduce replicas, the operator deregisters hosts from MarkLogic before allowing pod deletion.

Restart recovery: If MarkLogic restarts and clears dynamic membership, pods are rejoined with new tokens. Lost EmptyDir state is expected.

PVC-backed persistence (opt-in)

Configuration:

isDynamic: true
persistence:
  enabled: true
  size: 50Gi

Characteristics:

• Main MarkLogic datadir is backed by a persistent volume claim.
• Local host state on the PVC survives pod recreation during ordinary scale-down.
• Requires a storage class and CSI driver.
• Still evaluator-only; cannot host forests.

Scale-down behavior: During ordinary replica scale-down, the operator retains registered hosts in MarkLogic rather than deregistering them. Pods are deleted, but the PVC and host registration remain.

Restart recovery: For PVC-backed hosts, the operator clears the retained dynamic-host datadir from the PVC before issuing a new token and rejoining. Log data on the PVC is preserved if possible, but correctness of rejoin takes precedence.

Scale up hosts

The MarkLogic Operator creates one MarkLogic "host" per Kubernetes pod in a StatefulSet. To add a new MarkLogic host to an existing cluster, increase the number of replicas in the MarkLogic group configuration.

How scale-up works for dynamic groups

When you increase replicas on a dynamic group (isDynamic: true), the operator performs the following steps:

1. Configures the target MarkLogic group for dynamic hosts on the bootstrap host.
2. Waits for new pods to become locally ready (the pod passes its readiness probe).
3. Requests a dynamic host token from the bootstrap host for each new pod.
4. Joins each pod to the MarkLogic cluster using the token.
5. Records per-host and group status in MarklogicGroup.status.dynamic.

You can monitor progress through Kubernetes events emitted by the operator and the MarklogicGroup status fields.

Scale-up procedure

For example, to change the host count of an existing MarkLogic group from 2 to 3:

1. Modify the replicas count in sample.yaml:

   Copy

    apiVersion: marklogic.progress.com/v1
    kind: MarklogicCluster
    metadata:
        name: my-cluster
        namespace: marklogic
    spec:
        image: "progressofficial/marklogic-db:12.0.3-ubi9-rootless-2.2.6"
        auth:
        secretName: my-super-secret
        markLogicGroups:
        - replicas: 3 # Previously it is 2 and we are adding new static host
        name: dnode
        groupConfig:
            name: dnode
            enableXdqpSsl: true
        persistence:
            enabled: true
            size: 100Gi
        isBootstrap: true
        - replicas: 3 # Previously it is 2 and we are adding new dynamic host
        name: dynamic-enode
        groupConfig:
            name: dynamic-enode
            enableXdqpSsl: true
        isDynamic: true
   

2. Apply the sample.yaml changes:

   Copy

      kubectl apply -f <path/to/sample/file> -n <namespace> 
   

3. Once this deployment is complete, the new MarkLogic host joins the existing cluster.

4. To track deployment status, use the kubectl get pods command.

Remove hosts

When scaling down a StatefulSet, Kubernetes attempts to stop one or more pods in the set to achieve the desired number of pods. However, the storage attached to the pod remains until the persistent volume claims are deleted.

How scale-down works for dynamic groups

For dynamic groups (isDynamic: true), the operator handles host removal automatically:

1. The operator identifies the highest-ordinal dynamic hosts for removal.
2. For EmptyDir-backed dynamic groups, the operator removes the dynamic hosts from MarkLogic before allowing pod deletion.
3. For PVC-backed dynamic groups, the operator retains the host registration in MarkLogic during ordinary scale-down, then allows Kubernetes to terminate the pods.
4. Kubernetes terminates the pods after the required cleanup decision for that storage mode is complete.

Scaling to zero: Setting replicas: 0 on a dynamic group scales the pods down to zero. EmptyDir-backed dynamic groups are deregistered from MarkLogic before pod deletion, while PVC-backed dynamic groups retain their MarkLogic host registrations during ordinary scale-to-zero.

Group deletion: Removing a dynamic group entirely from the markLogicGroups array in the custom resource triggers cleanup of all registered dynamic hosts in that group, including PVC-backed retained hosts.

Standard (non-dynamic) groups

Shutting down a pod from Kubernetes does not modify the MarkLogic Cluster configuration; it merely stops the pod. Stopping the pod causes the MarkLogic host to go offline. If there are forests assigned to the stopped hosts, the associated forests will go offline.

Scale down the MarkLogic hosts (standard groups)

The procedure to scale down the number of MarkLogic hosts in a standard (non-dynamic) cluster varies depending on whether forests are assigned to the hosts and whether the hosts will be permanently removed from the MarkLogic Cluster.

For example, after migrating forest data from the third MarkLogic host, change the host count on an existing MarkLogic Cluster from 3 to 2 using sample.yaml file:

1. Modify the replicas count in sample.yaml:

   Copy

   apiVersion: marklogic.progress.com/v1
   kind: MarklogicCluster
   metadata:
       name: single-group
       namespace: marklogic
   spec:
       image: "progressofficial/marklogic-db:12.0.3-ubi9-rootless-2.2.6"
       imagePullSecrets: []
       auth:
       secretName: my-super-secret
       markLogicGroups:
       - replicas: 2 # Previously it is 3 and we are removing host
       name: node
       groupConfig:
           name: node
           enableXdqpSsl: true
       persistence:
           enabled: true
           size: 100Gi
       resources:
           requests:
           memory: "32Gi"
           cpu: "4"
           limits:
           memory: "32Gi"
           cpu: "4"
       isBootstrap: true 
   

2. Apply the sample.yaml changes:

   Copy

    kubectl apply -f <path/to/sample/file> -n <namespace> 
   

Before Kubernetes stops the pod, it makes a call to the MarkLogic host to shut down with the fastFailOver flag set to true. This tells the remaining hosts in the cluster that this host is shutting down. It also triggers failover for any replica forests available on this host. There is a two-minute grace period to allow MarkLogic to shut down cleanly before Kubernetes kills the pod.

Track shutdown progress

To track the host shutdown progress, run this command:

kubectl logs pod/terminated-host-pod-name -n <namespace>

Verify scaling results

After scaling up or down:

1. Verify pod state with kubectl get pods -n <namespace>.
2. Confirm expected host count in MarkLogic.
3. Verify forest status and failover state before additional scaling actions.

Permanently remove the host

If the host should be permanently removed from the MarkLogic Cluster, once the pod is terminated, follow the procedure in "Recovery - Step 3: Remove dead host configuration" in the MarkLogic Knowledgebase article Replacing a failed MarkLogic node in a cluster: a step by step walkthrough.

To delete the persistent volumes and persistent volume claims of the terminated host:

1. Get the persistent volume claims:

   Copy

   kubectl get pvc datadir-<terminated-host-pod-name> -n <namespace>
   

2. Delete the persistent volume:

   Copy

   kubectl delete pv <volume name from get pvc command>
   

3. Delete the persistent volume claims:

   Copy

   kubectl delete pvc datadir-<terminated-host-pod-name> -n <namespace>
   

HAProxy integration for dynamic groups

Dynamic groups can participate in HAProxy traffic distribution when HAProxy is enabled on your cluster.

Enable HAProxy for a dynamic group

Add or update the haproxy field in your dynamic group spec:

markLogicGroups:
  - name: enode-burst
    isDynamic: true
    haproxy:
      enabled: true
      appServers:
        - App-Services

Traffic routing behavior

When HAProxy is enabled for a dynamic group:

• Evaluator/query traffic: Dynamic hosts can receive traffic on configured app servers for query and evaluation work.
• Control-plane endpoints: Dynamic hosts do NOT participate in Admin or Manage traffic by default. These endpoints remain on static bootstrap groups.

Restart recovery for dynamic groups

If a MarkLogic Server restart clears dynamic host membership (for example, due to a pod restart, node failure, or cluster restart), the operator automatically detects the mismatch between the desired state in the custom resource and the actual MarkLogic cluster membership.

Automatic restart recovery

When a mismatch is detected, the operator:

1. Identifies pods that are running but no longer registered as dynamic hosts in MarkLogic.
2. For EmptyDir-backed dynamic groups, requests a new dynamic host token and rejoins the affected pod.
3. For PVC-backed dynamic groups, clears the retained local dynamic-host state, then requests a new dynamic host token and rejoins the affected pod.

This process is automatic and does not require you to recreate the group or manually intervene. The operator emits Kubernetes events to track rejoin progress.

Storage-mode-specific recovery behavior

For EmptyDir-backed hosts:

• Lost EmptyDir state is expected and normal.
• Pods rejoin with fresh local state.
• Previous host ID is replaced with a new ID.

For PVC-backed hosts:

1. The operator detects that the PVC still holds retained dynamic-host state.
2. Before issuing a new token and rejoining, the operator clears the retained dynamic-host datadir on the PVC to prevent stale MarkLogic state from interfering with rejoin.
3. Log folders are preserved on the PVC if possible, but successful rejoin takes precedence.
4. Pods then rejoin using the fresh datadir state and a new token.

Dynamic group status and observability

The operator exposes dynamic scaling progress through multiple channels:

MarklogicGroup status

The MarklogicGroup.status.dynamic field reports per-group dynamic scaling state:

kubectl get marklogicgroup <group-name> -n <namespace> -o jsonpath='{.status.dynamic}'

Status phase values

Reason codes

Detailed status fields

The status.dynamic object includes:

status:
  dynamic:
    phase: Reconciling
    desiredReplicas: 5
    localReadyReplicas: 5
    readyReplicas: 3
    bootstrapReady: true
    configured: true
    dynamicHostsEnabled: true
    message: "Joining host 4 of 5"
    hosts:
      - podName: enode-burst-0
        hostname: enode-burst-0.enode-burst.marklogic.svc.cluster.local
        hostId: "14271932171933168288"
        state: joined
        lastUpdated: "2026-04-06T10:01:00Z"
    lastTransitionTime: "2026-04-06T10:00:00Z"

Field descriptions:

• phase: Current lifecycle phase (Pending, Reconciling, Idle, Degraded, Failed)
• desiredReplicas: Desired number of dynamic hosts from spec
• localReadyReplicas: Pods passing readiness probe (locally ready, may not be joined)
• readyReplicas: Pods both locally ready AND registered in MarkLogic
• dynamicHostsEnabled: Whether allow-dynamic-hosts is enabled on the MarkLogic group
• hosts[].state: pending | joining | joined | retained | removing | removed | failed

Viewing status

Check entire status:

kubectl get marklogicgroup <group-name> -n <namespace> -o yaml

Check just the dynamic block:

kubectl get marklogicgroup <group-name> -n <namespace> -o jsonpath='{.status.dynamic}' | jq .

Watch status changes:

kubectl get marklogicgroup <group-name> -n <namespace> --watch

In addition to status.dynamic, the MarklogicGroup.status.conditions array includes a DynamicHostsReady condition. DynamicHostsReady becomes True when the dynamic group has reached its desired joined replica count.

Kubernetes events

The operator emits Kubernetes events on the MarklogicGroup resource during dynamic scaling operations. View events with:

kubectl describe marklogicgroup <group-name> -n <namespace>

Common events:

• Normal DynamicHostsConfigured — The MarkLogic group was configured for dynamic hosts.
• Normal DynamicHostScaleStarted — Scale-up or scale-down began.
• Normal DynamicHostRestartRecoveryStarted — Restart recovery began.
• Normal DynamicHostScaleComplete — Scale-up or scale-down finished.
• Normal DynamicHostRestartRecoveryComplete — Restart recovery completed.
• Normal DynamicDeleting — Dynamic group deletion is in progress.
• Warning BootstrapNotReady — Bootstrap cluster is unreachable or unhealthy.
• Warning GroupConfigFailed — Group configuration failed.
• Warning JoinFailed — Host join failed.
• Warning RemoveFailed — Host removal failed.
• Warning InvalidConfiguration — Configuration is invalid.
• Warning RetryBudgetExhausted — Retry limit exceeded.

Structured logs

The operator emits structured log entries for dynamic scaling operations. Filter operator logs for dynamic scaling messages:

kubectl logs deployment/<operator-deployment> -n <operator-namespace> | grep "dynamic"

Security and authentication for dynamic groups

Least-privilege credential model

Dynamic host operations use a separate, least-privilege credential distinct from the bootstrap admin credential:

1. Bootstrap operations (static group join, admin setup): Use the admin user.
2. Dynamic host operations (token issuance, host removal): Use a manage-admin role user.

The operator creates or uses a dedicated MarkLogic user with the manage-admin role for dynamic-host reconciliation, following the principle of least privilege.

TLS configuration

When TLS is enabled on your cluster (tls.enableOnDefaultAppServers: true):

1. The operator uses HTTPS for all Management API calls to the bootstrap host (port 8002).
2. The operator uses HTTPS for the token join POST to dynamic pods (port 8001).
3. Server name verification uses the pod FQDN.
4. Plain-text fallback is not permitted when TLS is enabled.

Troubleshooting dynamic host scaling

Common error conditions

Diagnosing issues

1. Check group status and phase:

   Copy

   kubectl get marklogicgroup <group-name> -n <namespace> -o yaml
   

2. Check detailed status with jq:

   Copy

   kubectl get marklogicgroup <group-name> -n <namespace> -o jsonpath='{.status.dynamic}' | jq .
   

3. Check operator events:

   Copy

   kubectl describe marklogicgroup <group-name> -n <namespace>
   

4. Check operator logs:

   Copy

   kubectl logs deployment/<operator-deployment> -n <operator-namespace> --tail=200 | grep -E '(dynamic|<group-name>)'
   

5. Check MarkLogic admin logs on the bootstrap host:

   Copy

   kubectl logs pod/<bootstrap-pod-name> -n <namespace> --tail=100
   

Transient vs. permanent failures

Transient failures (operator will retry with backoff):

• Network timeouts
• 5xx server errors
• Temporary bootstrap unavailability
• Token expiration before join completion

Permanent failures (require manual intervention):

• 401/403 authentication or authorization errors
• Invalid MarkLogic configuration
• Unsupported MarkLogic version
• Configuration violations (isDynamic on bootstrap, forest reuse, etc.)

TLS for dynamic host operations

Dynamic host operations use the MarkLogic Management and Admin APIs for group configuration, dynamic host token issuance, host removal, and pod join.

When TLS is enabled for the group, the operator uses HTTPS for:

1. Bootstrap-cluster management API calls.
2. The per-pod join call to the Admin API.

If dynamic host token generation or host join fails in a TLS-enabled deployment, verify the group TLS configuration and the certificates used by the bootstrap and pod Admin/Manage endpoints.