To scale a Hybrid Data Pipeline cluster, best practices recommend that you update the replicaCount parameter in the values.yaml file and then run the helm upgrade command. This procedure ensures the correct routing of requests to and from the On-Premises Connector by providing an on-premises accessor service for each pod in the cluster.

Scale with a Helm upgrade

Take the following steps to scale a cluster that uses the On-Premises Connector.

  1. Execute the update command:

    helm repo update

  2. Update the value of the replicaCount in the values.yaml file to specify the number of pods you want in the cluster.
  3. Execute the upgrade command:

    helm upgrade release-name chart-name --version version-name --values values.yaml --namespace namespace-value

Scale with the Kubernetes command line

The kubectl scale command may be used for deployments that are not using the On-Premises Connector. In the following example, the hdp-deployment is scaled to 6 replicas or pods.

kubectl scale deployment hdp-deployment --replicas=6 --namespace namespace-value