The Hybrid Data Pipeline Helm chart includes an optional Fluent Bit sidecar that streams application logs from each pod to external observability backends. When enabled, the sidecar runs alongside the Hybrid Data Pipeline server container and forwards log data through a configurable pipeline. The chart provides a default pipeline and allows controlled customization through the values.yaml file. Fluent Bit log collection is disabled by default and activates only when hdp.logCollection.enabled is set to true.

Important: Log collection (hdp.logCollection.enabled: true) requires hdp.persistence.logs.enabled: false. The chart fails at render time if both are set to true. See Managing and accessing system logs for details on choosing a logging mode.
The following sections describe the extension points available for customizing the Fluent Bit sidecar.

Enable log collection

To enable the Fluent Bit sidecar, update the following parameters in the values.yaml file.

hdp:
  logCollection:
    enabled: true
  persistence:
    logs:
      enabled: false
Note: When log collection is enabled, the StatefulSet terminationGracePeriodSeconds is set to 90 seconds to allow the sidecar to flush remaining log data before the pod terminates.

Inject environment variables

Use the extraEnvVars parameter to append or override environment variables in the Fluent Bit sidecar container. Entries are placed after the static environment block. Because Kubernetes resolves duplicate environment variable names with a last-definition-wins rule, any static variable can be overridden.

hdp:
  logCollection:
    extraEnvVars:
      - name: DAS_EXCLUDE_PATH
        value: "$(HDP_LOG_BASE)/das/debug.log"
      - name: MY_SECRET
        valueFrom:
          secretKeyRef:
            name: my-secret
            key: my-key

Inject bulk environment variables

Use the extraEnvFrom parameter to bulk-inject environment variables from an existing Kubernetes Secret or ConfigMap into the Fluent Bit sidecar. This parameter is rendered directly as the container envFrom: block. Use this approach to inject OTel collector authentication tokens or other bulk key sets without enumerating each key individually.
Note: The referenced Secret or ConfigMap must be created before running helm install or helm upgrade.
hdp:
  logCollection:
    extraEnvFrom:
      - secretRef:
          name: otel-collector-auth
      - configMapRef:
          name: fluent-bit-runtime-config

Add metadata fields to log records

Use the extraRecordFields parameter to stamp static key-value metadata on every Fluent Bit log record. This map is rendered into a Fluent Bit modify filter that runs before Hybrid Data Pipeline platform field enrichment. Keys must follow OpenTelemetry naming conventions: lowercase, dot-separated, and owner-namespaced.
Note: Avoid the hdp.* namespace. Platform fields such as hdp.cluster and hdp.image_tag are injected after extraRecordFields and overwrite any matching keys. The service.name key is the only exception: the enrichment layer sets it only when absent, so a customer-supplied service.name is preserved.
Values support ${ENV_VAR} references that are resolved from the sidecar environment at runtime. Variables injected via extraEnvVars or extraEnvFrom are eligible for resolution.
hdp:
  logCollection:
    extraRecordFields:
      org.team: "platform-sre"
      org.cost_center: "cc-1042"
      deployment.region: "eastus"
      customer.account_id: "${CUSTOMER_ACCOUNT_ID}"

Add custom pipeline filters

Use the extraFilters parameter to inject custom Fluent Bit filter blocks into the log pipeline. Filters are injected verbatim after all Hybrid Data Pipeline redaction filters and support any native Fluent Bit filter plugin.
Important: A mandatory redaction pass runs after custom filters. Any field added or rewritten by a custom filter is scrubbed before egress. Do not rely on custom filters to pass cleartext secrets to output destinations.
hdp:
  logCollection:
    extraFilters:
      - name: modify
        match: "hdp.das"
        set:
          - team.owner payments-platform
      - name: grep
        match: "hdp.tomcat"
        exclude: level DEBUG
      - name: lua
        match: "*"
        script: /fluent-bit/etc/custom.lua
        call: add_metadata

Mount custom volumes

Use the extraVolumes and extraVolumeMounts parameters to mount custom Lua scripts, TLS CA bundles, or other files into the Fluent Bit sidecar container. Mount paths under /fluent-bit/etc/ are recommended for Lua scripts so they can be referenced in extraFilters.
Note: The referenced ConfigMap or Secret must be created before running helm install or helm upgrade.
Take the following steps to add a custom Lua script to the Fluent Bit sidecar.
  1. Create a ConfigMap containing the Lua script.
  2. Add the ConfigMap as a volume using extraVolumes and mount it into the Fluent Bit container using extraVolumeMounts.
  3. Add a Lua filter in extraFilters that references the script path and call function.
  4. Run helm upgrade with the updated values and verify the Fluent Bit container logs.
hdp:
  logCollection:
    extraVolumes:
      - name: custom-lua
        configMap:
          name: my-custom-lua-scripts
      - name: otel-ca-bundle
        secret:
          secretName: otel-ca-bundle

    extraVolumeMounts:
      - name: custom-lua
        mountPath: /fluent-bit/etc/custom.lua
        subPath: custom.lua
        readOnly: true
      - name: otel-ca-bundle
        mountPath: /etc/ssl/otel-ca.crt
        subPath: ca.crt
        readOnly: true

Configure output destinations

Use the `outputs` parameter to configure one or more Fluent Bit output destinations. At least one output must be configured when hdp.logCollection.enabled is set to true. The parameter supports any Fluent Bit output plugin. For a complete list of available output plugins, refer to the Fluent Bit output plugin documentation.

The following examples show common output configurations.

OpenTelemetry output
hdp:
  logCollection:
    outputs:
      - name: opentelemetry
        match: "*"
        host: otel-collector-opentelemetry-collector
        port: 4318
        tls: off
        tls.verify: off
        logs_body_key: message
        retry_limit: false
        storage.total_limit_size: 200MB
Loki output
hdp:
  logCollection:
    outputs:
      - name: loki
        match: "*"
        host: loki.default.svc.cluster.local
        port: 3100
        labels: job=fluent-bit
Elasticsearch output
hdp:
  logCollection:
    outputs:
      - name: es
        match: "*"
        host: elasticsearch.default.svc.cluster.local
        port: 9200
        index: hdp-logs
Splunk HEC output
hdp:
  logCollection:
    outputs:
      - name: splunk
        match: "*"
        host: splunk.default.svc.cluster.local
        port: 8088
        splunk_token: "<your-hec-token>"
Important: Do not use stdout output in production. The stdout output doubles log volume by emitting every record to pod stdout in addition to the primary output. Use stdout only for local debugging with kubectl logs <pod> -c fluent-bit -n <namespace>.

For descriptions of supported parameters, see Helm chart parameters or refer to the values.yaml file in the Hybrid Data Pipeline Helm chart GitHub repository.