Docker Logging - Types, Configuring Drivers, Logging Strategies [Complete Guide]

Log analysis is a very powerful feature for an application when it comes to debugging and finding out which flow is working properly in the application and which is not. In a world of containerization and cloud computing, it is essential to understand logs generated by a Docker environment to maintain healthy performing applications.

In this article, we will discuss log analysis in Docker and how logging in Docker containers is different than in other applications. These logs are specific to Docker and are stored on the Docker host. We’ll thoroughly discuss the docker logs command and how we can configure a logging driver for containers.

Why is Docker logging different?

Life would be much simpler if applications running inside containers always behaved correctly. But unfortunately, as every developer knows, that is never the case.

With other systems, recording application log messages can be done explicitly by writing those messages to the system logger. This is done using syslog() system call from our application. But this doesn’t work for containerized logging. Here’s why:

• Containers are Multi-leveled
  

  Containers are just like boxes inside a big box. No matter how simple the Docker installation is, we’ll have to deal with two levels of aggregation. One level is where you see logs inside the container in your Dockerized application, known as Docker Container Logs.

  The second level is where you see the logs from the host servers (that is, system logs or Docker daemon logs). These are generally located in /var/log.

  A log aggregator that has access to the host application can not pull log files from the Dockerized application as if they are the host log files. In these scenarios, we will have to find a way to correlate the logs.

• Containers are ephemeral
  

  Container-based environments change very often, but they don’t serve the monitor well. Docker containers emit logs to stdout and stderr output streams.

  Logs are often stored on the Docker host because containers are stateless (failing to remember or save data from previous actions).

  It is dangerous to store logs in a Docker host because Docker doesn’t impose any size limit on log files, and they can build up over time and eat into your disk space. It is advised to store logs in a centralized location and enable log rotation for all the Docker containers.

Types of Docker Logs

There are two primary types of logs in the Docker ecosystem: Docker Container logs and Docker Daemon logs. These logs serve different purposes and are useful in different contexts.

Docker Container Logs

Docker Container Logs refer to the output generated by applications and processes running inside Docker containers. These logs are crucial for understanding the behavior and state of the containerized applications.

Container logs primarily consist of the standard output (stdout) and standard error (stderr) streams of the processes running inside the container.

These logs can be accessed using the docker logs [CONTAINER_ID or CONTAINER_NAME] command, which retrieves the logs for a specific container.

Docker Daemon Logs

Docker Daemon Logs are generated by the Docker daemon (dockerd), the background service that manages Docker containers and handles Docker engine tasks.

These logs provide information about the operations performed by the Docker daemon, including details on container lifecycle events, system messages, errors, and warnings related to Docker engine operations.

Accessing Docker daemon logs depends on the host system and the way Docker is installed and configured.

Using docker logs command

Container logs can be accessed using the docker logs command, followed by the container’s ID or name.

It is used to get all the information logged by a running container. The example below shows JSON logs created by the hello-world Docker image using json-file driver:

{"log":"Hello there!\n","stream":"stdout","time":"2022-07-28T22:51:31.549390877Z"}
{"log":"This message shows that everything seems to be working correctly.\n","stream":"stdout","time":"2022-07-28T22:51:31.549396749Z"}

The log follows a pattern of printing:

• Log’s origin
• Either stdout or stderr
• A timestamp

In order to review a container’s logs from the command line, you can use the docker logs <container-id> command. Using this command, the logs shown above are displayed this way:

Hello there!
This message shows that everything seems to be working correctly.

Here are a few options in the command that you can use to modify the output of your log:

docker logs [OPTIONS] <container-id>

• Using -f or --follow option, if you want to follow the logs:

  docker logs <container_id> --follow
• If you want to see the last N log lines:

  docker logs <container-id> --tail N
• If you want to see the specific logs, use the grep command:

  docker logs <container_id> | grep pattern
• If you want to show errors:

  docker logs <container_id> | grep -i error

Once an application starts growing, you tend to start using Docker Compose. docker compose logs command shows logs from all the services running in the containerized application.

One other command worth learning about is docker service logs command that is used in the context of Docker Swarm. This command retrieves logs from all the containers that are part of a given service in a Docker Swarm.

You can run the command as follows:

docker service logs [SERVICE_ID or SERVICE_NAME]

It aggregates and outputs the logs from all containers that are part of the specified service. This is particularly useful for getting a unified view of what's happening across multiple containers that are part of the same service, potentially spread across different nodes in the cluster.

Note that the offering from the docker logs command may vary based on the Docker version you are using. In case of Docker Community, docker logs can only read logs created by the json-file, local, and journald drivers whereas in case of Docker Enterprise, docker logs can read logs created by any logging driver.

Configuring Logging Drivers in Docker

A logging driver can forward logs to a centralized log system and maintain a consistent logging approach across all containers. Configuring logging drivers in Docker involves specifying which driver Docker should use for handling and forwarding container logs.

Docker supports various logging drivers, each of which allows logs to be routed to different destinations or handled in different ways.

Understanding Available Logging Drivers

Some common logging drivers include:

• json-file: The default driver, storing logs in JSON files.
• syslog: Forwards logs to a syslog server.
• journald: Sends logs to systemd journal.

For configuring the logging driver, you can either set it up globally for the entire Docker daemon or individually for each container.

Configuring the Logging Driver for a Single Container

To configure the logging driver for a single container at runtime, use the --log-driver option when running the docker run command. Additionally, you can specify driver-specific options with the --log-opt flag.

Example:

docker run --log-driver=syslog --log-opt syslog-address=udp://192.168.0.1:514 --log-opt syslog-facility=daemon my-app:latest

In this example, the container's logs are sent to a syslog server at 192.168.0.1 on port 514.

Configuring the Default Logging Driver for the Docker Daemon

To change the default logging driver for all containers managed by the Docker daemon:

Step 1. Edit the Docker Daemon Configuration:

• Locate the Docker daemon configuration file, typically found at /etc/docker/daemon.json on Linux systems.
• If the file doesn’t exist, create it.

Step 2. Specify the Logging Driver:

• Add or modify the log-driver and log-opts keys.

Example daemon.json with the syslog driver:

{
  "log-driver": "syslog",
  "log-opts": {
    "syslog-address": "udp://192.168.0.1:514",
    "syslog-facility": "daemon"
  }
}

Step 3. Restart the Docker Daemon:

• After making changes, restart the Docker daemon to apply them.
• On systems using systemd, this can typically be done with sudo systemctl restart docker.

To find the current logging driver for the Docker daemon:

$ docker info --format '{{.LoggingDriver}}'

syslog

Deciding the delivery mode of log messages from container to log driver

Docker provides two types of delivery modes for log messages.

• Blocking(default mode)
  As the name suggests, this mode blocks the main process inside a container to deliver log messages. And this will add latency to the performance of the application. But, it ensures that all the log messages will be successfully delivered to the log driver.

  The default log driver (json-files) logs messages very quickly since it writes to the local file system. Therefore it’s unlikely to cause latency. But drivers like gcplogs and awslogs open a connection to a remote server and are more likely to block and cause latency.

• Non-blocking
  In this mode, the container writes logs to an in-memory ring buffer. This in-memory ring buffer works like a mediator between logging-driver and the container. When the logging-driver isn’t busy processing the logs, the container shares the logs to the driver immediately. But when the driver is busy, these logs are put into the ring-buffer.

  This provides you a safety check that a high volume of logging activity won’t affect the application’s performance running inside the container. But there is a downside. It doesn’t guarantee that all the log messages will be delivered to the logging driver. In cases where log broadcasts are faster than the driver processor, the ring buffer will soon run out of space. As a result, buffered logs are deleted to make space for the next set of incoming logs. The default value for max-buffer-size is 1 MB.

  To change the mode, edit the daemon.json file at /etc/docker/daemon.json:

  {
      "log-driver": "json-file",
      "log-opts": {
          "max-size": "20m",
              "max-file": "10",
          "mode": "non-blocking"
      }
  }

  Alternatively, you can set the non-blocking mode on an individual container by using the --log-opt option in the command that creates the container:

  docker run --log-opt mode=non-blocking alpine echo hello world

The default log driver stores data in a local file, but if you want more features, then you can opt for other log drivers as well, such as logagent, syslog, journald, elf, awslogs, etc.

Logging strategies

Docker logging means logging events of the dockerized application, host OS and the docker service. There are various ways to log events for a docker container.

Some of them are:

• Application logging: In this strategy, the application inside the container can have its own logging framework. The logs can either be stored locally or sent to a centralized location using a log management tool.

• Data volumes: Because containers are stateless, and to avoid losing logs data, you can bind the container’s directory to the host OS directory. Containers can now be terminated or shut down, and access logs from multiple containers. You can run a regular backup in order to prevent data corruption or loss in case of failure.

• Docker logging driver: This type has already been discussed in detail. The configured driver reads the data broadcast by the container’s stdout or stderr streams and writes it to a file on the host machine. You can then send this log data anywhere you want to.

• Sidecar pattern: The sidecar pattern involves running a separate container alongside the main application container. This container performs a specific function, such as logging, monitoring, or caching. This approach allows for better separation of concerns, where each container has a single responsibility.

• Dedicated logging containers: This strategy is similar to the sidecar pattern, but instead of having a dedicated logging container for each application, you have a pool of logging containers serving multiple applications. This approach can be more resource-efficient than the single sidecar per application approach.

• Centralized Logging: This involves aggregating logs from multiple sources (containers, hosts, services) into a single, unified logging system. This makes it easier to search, analyze, and monitor logs.

Choosing the Right Logging Strategy

Selecting the appropriate logging strategy for your Dockerized applications is necessary for effective monitoring, debugging, and maintenance. The best logging strategy will depend on several factors, including the scale of your deployment, the complexity of your applications, the infrastructure, and your specific logging requirements. Here's a general guide to help you choose:

• If isolating log data from your application is your concern, the sidecar approach is well-suited.
• Use dedicated logging containers for scenarios where portability, resource efficiency, and scalability are priorities.
• If you need to integrate your log data with existing logging solutions or send logs to remote storage or analysis services, using a logging driver can streamline this process.
• For large and dynamic environments with numerous containers and services, a centralized logging solution is the best option.
• To ensure log data persistence across container restarts and failures, use data volumes.
• For applications with specific, complex logging requirements, use the application logging strategy.
• If you require advanced log management features, analysis tools, and alerting capabilities, a centralized logging solution is an excellent choice.

Final Thoughts

Containerization surely provides an easy way to deal with application portability and scalability issues but it does requires maintenance from time to time. Container environments are just like a box inside a box, with multiple layers of abstraction. So, it becomes hard to debug in such environments and if performed correctly, log-analysis can be your go-to friend to find out performance related issues.

In this guide, you learned how to configure the Docker logging driver for log analysis in containerized applications, how Docker logging is different from application logging hosted on a physical machine or virtual host, and in detail study of the docker logs command.

There are various logging strategies that you can follow for log analysis. This blog thoroughly discussed the default logging strategy - json-file and the two delivery modes of log messages. Containers being stateless, doesn’t ensure data persistence, hence to prevent data loss, you can use a log management tool. SigNoz - an open source APM and observability tool provides an efficient log management solution.

SigNoz uses a columnar database - ClickHouse, for storing logs efficiently. Big companies like Uber and Cloudflare have shifted from Elasticsearch to ClickHouse for storing their log data.

Logs management in SigNoz

Getting started with SigNoz

SigNoz cloud is the easiest way to run SigNoz. Sign up for a free account and get 30 days of unlimited access to all features.

You can also install and self-host SigNoz yourself since it is open-source. With 16,000+ GitHub stars, open-source SigNoz is loved by developers. Find the instructions to self-host SigNoz.

But logs are just one aspect of getting insights from your software systems. Modern applications are complex distributed systems. For debugging performance issues, you need to make your systems observable. Logs, when combined with metrics and traces form an observability dataset that can help you debug performance issues quickly.

SigNoz can help you monitor your application by collecting all types of telemetry data. It correlates all your telemetry data(logs, metrics, and traces) into a single suite of monitoring. It is built to support OpenTelemetry natively. OpenTelemetry is becoming the world standard for instrumenting cloud-native applications.

You can check out SigNoz GitHub repo:

---

If you want to read more about SigNoz, check out the following blog:

SigNoz - an open source alternative to DataDog

SigNoz uses OpenTelemetry to generate and collect telemetry data like logs. Read more about OpenTelemetry logs here.