The following best practices help you prepare a production-ready deployment of the Hybrid Data Pipeline Helm chart. Each practice addresses a specific area of operational governance and provides guidance for securing, sizing, and managing your Kubernetes deployment.

Image governance

Use approved private registries and immutable image references for all container images in your deployment.
  • Supply hdp.image.repository and hdp.image.tag from a customer-approved registry.
  • Mirror and govern publicly sourced defaults (for example Fluent Bit and PostgreSQL images).
  • Prefer digest pinning for production releases.
Important: Do not deploy with default public image references in production. Mirror all images to a private registry and control updates through your change management process.

Secret management

Keep credentials exclusively in Kubernetes Secrets and enforce strict access governance.
  • Do not duplicate credentials in rendered artifacts or configuration files.
  • Treat Base64 as encoding only. Enforce encryption at rest (etcd/KMS) and least-privilege Secret access.
  • Restrict who can read or update hdp-secrets and related TLS Secrets.

For details on creating required Secrets, see Creating Kubernetes Secrets for required credentials.

Shared storage hardening

Protect the shared storage paths (/hdpshare and /logs) with strict access controls.
  • Allow write access only to trusted operational identities.
  • Protect TLS key material and plugin/driver paths with strict access controls, audit, and backup governance.
  • Use storage classes and policies that support your security and retention requirements.

For details on the shared file location, see Obtaining configuration and certificate files to install components

Logging strategy

Choose one logging mode and configure it intentionally. The Helm chart supports two mutually exclusive approaches:
  • Persistent log PVC Enable with hdp.persistence.logs.enabled=true. Enforce retention policies, access restrictions, and backup controls.
  • Fluent Bit sidecar collection Enable with hdp.logCollection.enabled=true. Define explicit outputs, avoid stdout in production, and route logs to approved destinations.
Note: Do not enable both logging modes simultaneously. Choose the mode that aligns with your organization's log management and compliance requirements.

For details on configuring logs, see Managing and accessing system logs.

Cluster governance and identity boundaries

Enforce strong cluster governance and identity boundaries as part of your platform policy.
  • Use namespace isolation, strict service account usage standards, and controlled token exposure.
  • Disable unnecessary token mounting where feasible in your cluster baseline.
  • Enforce admission policies, privileged-role separation, and centralized audit controls at cluster level.

Network exposure

Use exactly one external topology at a time for your Hybrid Data Pipeline deployment.
  • Ingress mode Use an approved ingress class, managed certificates, and strict ingress annotations and policies. Ensure AGIC is properly configured with TLS enabled.
  • Headless external-gateway mode Ensure upstream controls for TLS termination, DNS, WAF, and sticky-session behavior are in place.
Note: Do not enable both ingress and headless external-gateway modes simultaneously. Verify that only one hdp.hdpingressconfiguration mode is set to true.

PostgreSQL resilience

The Helm chart defaults prioritize first-deploy simplicity with postgresql.architecture: standalone. For production deployments, evaluate and configure the following:
  • Replication mode and read replica settings
  • PodDisruptionBudget values
  • Anti-affinity and topology spread constraints
  • Failover posture
Keep PostgreSQL security context hardening enabled for both primary and read replica instances.
postgresql:
  architecture: replication
  primary:
    podAntiAffinityPreset: "soft"
    topologySpreadConstraints:
    - maxSkew: 1
      topologyKey: topology.kubernetes.io/zone
      whenUnsatisfiable: DoNotSchedule
      labelSelector:
        matchLabels:
          app.kubernetes.io/component: primary

For details on PostgreSQL configuration, see PostgreSQL replication. For scheduling options, see Advanced pod scheduling.

Compute and storage sizing

Set resource allocations, JVM heap size, and PVC sizes based on production-like load and growth forecasts before go-live.
  • Configure hdp.resources, hdp.jvmHeap, and PVC sizes from validated capacity planning.
  • Keep JVM heap below memory limits with safe headroom to reduce OOM risk.
  • Add monitoring and alerting for memory pressure, PVC capacity, and restart anomalies.
hdp:
  resources:
    requests:
      memory: 4096Mi
      cpu: 2000m
    limits:
      memory: 4096Mi
      cpu: 2000m
  jvmHeap: 3072m
Note: The default resource settings are minimum allocations to facilitate deployment. The recommended allocation for running the Hybrid Data Pipeline service is 100 Gi. Adjust memory, CPU, and storage allocations based on your workload requirements.

For details on resource allocation, see Allocating resources.

Plugin and initialization governance

Treat plugin and driver updates and initialization-time SQL and input values as controlled operational changes.
  • Require pre-production validation for all initialization-related changes.
  • Maintain auditable approval and rollback procedures for plugin deployments.
  • Version-control custom plugins and drivers alongside your manifest files.

For details on deploying plugins, see Integrating data store connectors and authentication plugins.

Pod security baseline

Keep the following security settings enabled for all Hybrid Data Pipeline pods in production.
  • Non-root execution
  • Read-only root filesystem
  • Dropped capabilities
  • RuntimeDefault seccomp profile

Document any required exceptions centrally and review them periodically. For details on read-only root filesystem configuration, see Mounting the root filesystem as read-only.

Minimum release gate

Verify the following items before deploying to production.
  • Image sources approved and pinned Confirm all runtime images come from approved private registries and are pinned by tag or digest per policy.
  • Secrets encrypted at rest and access reviewed Confirm all chart Secrets are in Kubernetes Secrets with encryption at rest enabled and least-privilege access reviewed.
  • Shared storage permissions validated Confirm /hdpshare and /logs access controls enforce least-privilege write access and auditable ownership.
  • Logging mode selected and verified Confirm exactly one log strategy is enabled (PVC or Fluent Bit) and the selected path is validated end-to-end.
  • External exposure model validated Confirm only one external access model is active with required TLS, DNS, and WAF controls in place.
  • PostgreSQL resilience profile approved Confirm PostgreSQL architecture and disruption settings (PDB, replication posture, scheduling spread) are approved for production resilience goals.
  • Capacity and PVC sizing load-tested Confirm Hybrid Data Pipeline resources, JVM heap, and PVC sizes are validated by production-like load and growth testing.
  • Cluster governance controls verified Confirm cluster-level admission policy, identity governance, privileged-role separation, and audit controls are enforced for the target namespace.