nginx in Docker
When setting up app hosting on Slinky Dog, we decided to follow our
current approach of dockerizing any internal services. Therefore,
we decided to use nginx Docker image as the resource server.
When you run nginx in docker, you can use the docker logs command
to see both the access and error logs. This is setup in the Dockerfile
for the image, by creating an access.log and an error.log symlink
in the standard nginx log directory /var/log/nginx.
These symlinks, go to /dev/stdout and /dev/stderr, which in turn
are symlinks to system level file descriptors.
This means when nginx logs something, rather than sending it to a file
on disk, it is sent to stdout (access) and stderr (error). These
file descriptors are the source for output in docker logs.
Portainer container logs
When we open the container in Portainer, there is an option to view
the logs. This output is the same docker logs. This is a useful
way to stream the logs to see the current status.
Persisting logs
It would be nice to persist logs though. If we can only access the logs
via docker logs we don’t have an easy way to go back in time to see
previous logs.
One way to achieve this is to bind mount a local directory to the
/var/log/nginx directory e.g. -v $PWD/logs:/var/log/nginx. This bind
mount overrides the work of the Dockerfile, and now all logs go to text
files in $PWD/logs on the host computer.
However, a downside to this is that the logs no longer show up in
docker logs, thus eliminating the easy access via Portainer.
A second downside to this approach is SSH access to the host computer is required to view the logs.
Docker volumes
Let’s remedy both of these issues with the help of docker volumes. Rather, than creating a bind mount, we can create and mount the volume.
The volume nginx_logs is created, and it can be mounted with -v nginx_logs:/var/log/nginx_logs. Note the path on the container has been changed to being non-standard.
However, now we need to write nginx logs to that new location. This can be achieved
by adding more access_log and error_log directives to nginx.conf:
error_log /var/log/nginx_logs/error.log notice;
error_log /var/log/nginx/error.log notice;
...
access_log /var/log/nginx_logs/access.log main;
access_log /var/log/nginx/access.log main;
Nginx will log to all of these locations, and docker logs is useful again.
That solves the first downside, what about removing the SSH access constraint? This has also been resolved, by using a docker volume.
Browse volume in Portainer
With nginx now writing files to the docker volume, these files can be accessed
via Portainer and inspected on a local machine without needing SSH access.
Rotating logs
Given log files are being written to disk, it’s important to rotate them to avoid taking up unnecessary disk space. This can be achieved with the logrotate utility.
A new Dockerfile was created and installed logrotate into the image. When
running this image, we mount the nginx_logs volume, and rotate the logs in
that volume.
This was where we bumped into another problem. After calling logrotate we need
to notify nginx so it can reopen the logs. The standard way to do this is by
sending a signal to the nginx process nginx -s reopen. This worked for stdout
but did not work for stderr.
Reopening logs
This meant we could see latest error logs in the volume, and latest access logs in
the volume and in docker logs. However, we could not see the latest error logs in
docker logs.
The easiest solution for now was to go with a brute force approach. The rotate.sh
script that launches logrotate restarts the nginx container once the log rotation
has completed.
Now we can see both the access and error logs in docker logs for the nginx container.
References
This is a word heavy post. Checkout the server-ops repository, where all the files needed for this setup can be found.