Log collection

MarkLogic Kubernetes deployments use Fluent Bit for log collection in both Helm Chart and Operator deployments. Log collection is disabled by default and must be enabled by updating the Kubernetes manifest for the deployment type you are using.

Operator deployments

• Starting with Operator 1.1.1, Fluent Bit supports hot reload, allowing you to update log collection settings without restarting pods.
• Starting with Operator 1.3.1, the Fluent Bit sidecar can consume environment variables from Kubernetes Secrets. This allows logging outputs to authenticate with external logging backends without exposing credentials in the Fluent Bit ConfigMap.

Helm Chart deployments

• Fluent Bit hot reload is not supported.
• Consuming environment variables from Kubernetes Secrets for Fluent Bit authentication is not supported.

Prerequisites

Before configuring log collection, ensure you have the following:

• Logging backend deployed: A logging system such as Loki, Elasticsearch, Splunk, or AWS CloudWatch must be deployed and accessible from your MarkLogic cluster.
• Network connectivity: MarkLogic pods must be able to connect to your logging backend over the network.
• Connection details: Obtain the hostname/IP address, port, and any required authentication credentials for your logging backend.
• Storage capacity: Ensure your logging backend has sufficient storage capacity for the expected log volume.
• Access permissions: Verify that the MarkLogic service account has the necessary permissions to send logs to your logging backend (if applicable).

Configure log collection for Helm Chart deployments

To configure log collection, update the Helm Chart manifest file (values.yaml) according to the following steps:

1. Set the logCollection.enabled parameter to true.

2. Set logCollection.files parameters:

   • Set to true if you want to collect the given file.
   • Set to false if you do not want to collect the given log file.

   Note:

   See Helm Chart parameter reference for Helm Chart parameter descriptions.

3. Configure the logCollection.outputs parameter to specify where Fluent Bit should send the collected logs. This defines the logging backend destination such as Loki, Elasticsearch, Splunk, or CloudWatch. Include connection details like hostname, port, and authentication as needed for your specific logging backend.

4. Upgrade the Helm Chart deployment. For example:

   Copy

   helm upgrade marklogic-cluster marklogic/marklogic -f values.yaml
   

After upgrading the Helm Chart deployment, you may use Fluent Bit to parse and output the log files from each pod. See Fluent Bit's output documentation for more information on configuring Fluent Bit output with a logging backend.

Sample Helm Chart log collection configuration

logCollection:
  enabled: true
  image: fluent/fluent-bit:5.1.0
  files:
    errorLogs: true
    accessLogs: true
    requestLogs: true
    crashLogs: true
    auditLogs: true
  outputs: |-
    - name: loki
      match: "*"
      host: loki.loki.svc.cluster.local
      port: 3100
      labels: job=fluent-bit

Configure log collection for Operator deployments

To configure log collection, update the Operator manifest file (sample.yaml) according to the following steps.

1. Set the logCollection.enabled parameter to true.

2. Set logCollection.files parameters:

   • Set to true if you want to collect the given file.
   • Set to false if you do not want to collect the given log file.

   Note:

   See Operator API reference for Operator parameter descriptions.

3. Configure the logCollection.outputs parameter to specify where Fluent Bit should send the collected logs. This defines the logging backend destination such as Loki, Elasticsearch, Splunk, or CloudWatch. Include connection details like hostname, port, and authentication as needed for your specific logging backend.

4. Apply the log collection configuration to the Operator deployment. For example:

   Copy

   kubectl apply -f sample.yaml
   

After applying the configuration to your Operator deployment, you may use Fluent Bit to parse and output the log files from each pod. See Fluent Bit's output documentation for more information on configuring Fluent Bit output with a logging backend.

Sample Operator log collection configuration

The Operator log collection configuration must be located in the spec section of the manifest.

  logCollection:
    enabled: true
    image: fluent/fluent-bit:5.1.0
    files:
      errorLogs: true
      accessLogs: true
      requestLogs: true
      crashLogs: true
      auditLogs: true
    outputs: |-
      - name: loki
        match: "*"
        host: loki.loki.svc.cluster.local
        port: 3100
        labels: job=fluent-bit

Configure secret-backed environment variables for Fluent Bit

Beginning with Operator 1.3.1, the Fluent Bit sidecar can consume environment variables defined in logCollection.env using the standard Kubernetes EnvVar schema. This lets Fluent Bit reference values from Kubernetes Secrets, such as bearer tokens or API keys, so outputs can send authenticated logs to a backend like an OpenTelemetry Collector without exposing credentials in the Fluent Bit ConfigMap. This capability applies only to Operator deployments; Helm Chart deployments do not support it.

Use these steps to reference a Secret-backed value in your log collection configuration.

1. Create a Kubernetes Secret in the same namespace as your MarkLogic deployment. For example:

   Copy

   apiVersion: v1
   kind: Secret
   metadata:
     name: otel-auth
   stringData:
     token: replace-with-your-token
   

2. Add an entry in logCollection.env in your Operator manifest file (sample.yaml) that reads a key from the Secret by using valueFrom.secretKeyRef.

3. Use that environment variable in logCollection.outputs, for example in an authorization header.

4. Apply the configuration to your Operator deployment:

   Copy

   kubectl apply -f sample.yaml
   

Only the Secret reference, including its name and key, is passed to the Fluent Bit container; the Operator does not resolve or copy the Secret value. Kubernetes resolves the reference when the pod starts, so the value never appears in the Fluent Bit ConfigMap, the MarklogicCluster or MarklogicGroup status, or the Operator logs.

Sample secret-backed environment variable configuration

The Operator log collection configuration must be located in the spec section of the manifest.

  logCollection:
    enabled: true
    env:
      - name: OTEL_AUTH_TOKEN
        valueFrom:
          secretKeyRef:
            name: otel-auth
            key: token
    outputs: |-
      - name: opentelemetry
        match: kube.marklogic.logs.*
        host: otel-collector.observability
        port: 4317
        grpc: 'on'
        header:
          - Authorization Bearer ${OTEL_AUTH_TOKEN}

Apply configuration changes using hot reload

Beginning with Operator 1.1.1, Fluent Bit hot reload is supported for Operator deployments. Hot reload allows you to update logging filters, parsers, and outputs without restarting MarkLogic pods, ensuring uninterrupted log collection and zero downtime for your monitoring pipeline.

Take the following steps to set up hot reload for your Operator deployment.

1. Update your log collection configuration according to your specifications. For example, you might change the output destination or update the Fluent Bit image.

2. Apply the updated configuration:

   Copy

   kubectl apply -f sample.yaml
   

   Important:

   Wait at least 60 seconds before proceeding with the next step. This will allow the Kubernetes ConfigMap to synchronize with the volumes mounted inside the Fluent Bit sidecar containers.

3. Make a POST request to the Fluent Bit local API to refresh the configuration. For example:

   Replace <namespace> with your namespace and <marklogic-pod-name> with your pod name.

   Copy

   kubectl exec -n <namespace> <marklogic-pod-name> -- curl -s -X POST -d '{}' http://localhost:2020/api/v2/reload
   

   You should see a response indicating that the configuration has been reloaded successfully like this:

   Copy

   {"reload":"done","status":0}
   

   Note:

   The reload command must be run against a specific pod. If you have a multi-pod cluster, you should trigger the reload for each pod to ensure consistency across the deployment.

4. Confirm the reload was successful by inspecting the Fluent Bit container logs. For example:

   Replace <namespace> with your namespace and <marklogic-pod-name> with your pod name.

   Copy

   kubectl logs -n <namespace> <marklogic-pod-name> -c fluent-bit --tail=50 | grep -i reload
   

   You should see log entries indicating that the configuration has been reloaded:

   Copy

   [info] [reload] successful reload done.
   

   Note:

   If the reload fails, Fluent Bit will typically continue running with the previous valid configuration. Check the logs for syntax errors in your Operator manifest file if the "successful" message does not appear.