When structured logging is enabled, each log entry is emitted as a single-line JSON object. This format ensures compatibility with most log aggregation systems and makes ingestion into tools such as Elasticsearch, Loki, Splunk, or OpenSearch straightforward and reliable.

Note: Adopt structured logging early. Retrofitting structured logging after deployment results in the loss of valuable historical comparability.
The following steps summarize the recommended pipeline for log aggregation:
  1. Configure log forwarding—Log forwarding ensures that logs generated by containers or services are collected and sent to a central location for processing. Use one of the following methods to configure log forwarding:
    Method Description
    Docker logging driver Configure the Docker logging driver with the json-file format to capture logs directly from containers.
    Sidecar forwarder Deploy a sidecar container that forwards logs to the central log aggregator.
  2. Create a central index—A central index organizes log data for efficient querying and visualization. Proper indexing improves performance when analyzing metrics such as latency and request rates. Define an index template that maps numeric fields for optimal performance:
    Field Description
    duration_ms Represents request duration in milliseconds for latency analysis.
    rps Indicates requests per second for throughput monitoring.
  3. Build dashboards—Dashboards provide a visual representation of key metrics. They help you monitor system health and identify performance bottlenecks quickly. Create dashboards to track the following metrics:
    Metric Description
    Request rate Displays the number of requests processed per second.
    Median latency Shows the median response time for requests.
    Deny counts Tracks the number of denied requests.
    Top tools by volume Lists tools generating the highest log volume.