Log collection

MarkLogic Helm Chart and Operator Kubernetes deployments use Fluent Bit for log collection. By default, log collection is not enabled, but you may enable it by updating the Kubernetes manifest file for each deployment type. Beginning with Operator 1.1.1, Fluent Bit hot reload is supported for Operator deployments. Hot reload allows you to update your log configuration without restarting pods. Fluent Bit hot reload is not supported for Helm Chart deployments.

Prerequisites

Before configuring log collection, ensure you have the following:

• Logging backend deployed: A logging system such as Loki, Elasticsearch, Splunk, or AWS CloudWatch must be deployed and accessible from your MarkLogic cluster.
• Network connectivity: MarkLogic pods must be able to connect to your logging backend over the network.
• Connection details: Obtain the hostname/IP address, port, and any required authentication credentials for your logging backend.
• Storage capacity: Ensure your logging backend has sufficient storage capacity for the expected log volume.
• Access permissions: Verify that the MarkLogic service account has the necessary permissions to send logs to your logging backend (if applicable).

Configure log collection for Helm Chart deployments

To configure log collection, update the Helm Chart manifest file (values.yaml) according to the following steps.

1. Set the logCollection.enabled parameter to true.

2. Set logCollection.files parameters:

   • Set to true if you want to collect the given file.
   • Set to false if you do not want to collect the given log file.

   Note:

   See Helm Chart parameter reference for Helm Chart parameter descriptions.

3. Configure the logCollection.outputs parameter to specify where Fluent Bit should send the collected logs. This defines the logging backend destination such as Loki, Elasticsearch, Splunk, or CloudWatch. Include connection details like hostname, port, and authentication as needed for your specific logging backend.

4. Upgrade the Helm Chart deployment. For example:

   Copy

   helm upgrade marklogic-cluster marklogic/marklogic -f values.yaml
   

After upgrading the Helm Chart deployment, you may use Fluent Bit to parse and output the log files from each pod. See Fluent Bit's output documentation for more information on configuring Fluent Bit output with a logging backend.

Sample Helm Chart log collection configuration

logCollection:
  enabled: true
  image: fluent/fluent-bit:4.1.1
  files:
    errorLogs: true
    accessLogs: true
    requestLogs: true
    crashLogs: true
    auditLogs: true
  outputs: |-
    - name: loki
      match: "*"
      host: loki.loki.svc.cluster.local
      port: 3100
      labels: job=fluent-bit

Configure log collection for Operator deployments

To configure log collection, update the Operator manifest file (sample.yaml) according to the following steps.

1. Set the logCollection.enabled parameter to true.

2. Set logCollection.files parameters:

   • Set to true if you want to collect the given file.
   • Set to false if you do not want to collect the given log file.

   Note:

   See Operator API reference for Operator parameter descriptions.

3. Configure the logCollection.outputs parameter to specify where Fluent Bit should send the collected logs. This defines the logging backend destination such as Loki, Elasticsearch, Splunk, or CloudWatch. Include connection details like hostname, port, and authentication as needed for your specific logging backend.

4. Apply the log collection configuration to the Operator deployment. For example:

   Copy

   kubectl apply -f sample.yaml
   

After applying the configuration to your Operator deployment, you may use Fluent Bit to parse and output the log files from each pod. See Fluent Bit's output documentation for more information on configuring Fluent Bit output with a logging backend.

Sample Operator log collection configuration

The Operator log collection configuration must be located in the spec section of the manifest.

  logCollection:
    enabled: true
    image: fluent/fluent-bit:4.1.1
    files:
      errorLogs: true
      accessLogs: true
      requestLogs: true
      crashLogs: true
      auditLogs: true
    outputs: |-
      - name: loki
        match: "*"
        host: loki.loki.svc.cluster.local
        port: 3100
        labels: job=fluent-bit

Apply configuration changes using hot reload

Beginning with Operator 1.1.1, Fluent Bit hot reload is supported for Operator deployments. Hot reload allows you to update logging filters, parsers, and outputs without restarting MarkLogic pods, ensuring uninterrupted log collection and zero downtime for your monitoring pipeline.

Take the following steps to set up hot reload for your Operator deployment.

1. Update your log collection configuration according to your specifications. For example, you might change the output destination or update the Fluent Bit image.

2. Apply the updated configuration:

   Copy

   kubectl apply -f sample.yaml
   

   Important:

   Wait at least 60 seconds before proceeding with the next step. This will allow the Kubernetes ConfigMap to synchronize with the volumes mounted inside the Fluent Bit sidecar containers.

3. Make a POST request to the Fluent Bit local API to refresh the configuration. For example:

   Replace <namespace> with your namespace and <marklogic-pod-name> with your pod name.

   Copy

   kubectl exec -n <namespace> <marklogic-pod-name> -- curl -s -X POST -d '{}' http://localhost:2020/api/v2/reload
   

   You should see a response indicating that the configuration has been reloaded successfully like this:

   Copy

   {"reload":"done","status":0}
   

   Note:

   The reload command must be run against a specific pod. If you have a multi-pod cluster, you should trigger the reload for each pod to ensure consistency across the deployment.

4. Confirm the reload was successful by inspecting the Fluent Bit container logs. For example:

   Replace <namespace> with your namespace and <marklogic-pod-name> with your pod name.

   Copy

   kubectl logs -n <namespace> <marklogic-pod-name> -c fluent-bit --tail=50 | grep -i reload
   

   You should see log entries indicating that the configuration has been reloaded:

   Copy

   [info] [reload] successful reload done.
   

   Note:

   If the reload fails, Fluent Bit will typically continue running with the previous valid configuration. Check the logs for syntax errors in your Operator manifest file if the "successful" message does not appear.