OpenMetal private clouds generate a large quantity of log files. It can be daunting knowing which log and on which host to look in to diagnose an issue. In this guide, we present strategies for extracting information from a private cloud's log files to determine the root cause of a failure.
Root Access to OpenStack Control Plane
Root access to your cloud's control plane nodes is required.
Elasticsearch and Kibana
This guide focuses on how to manually look through logs, however Elasticsearch and Kibana, commonly referred to as an ELK stack, are often used to aggregate and view logs in a visual manner. With a properly configured ELK stack, you can view all of your cloud's logs from a single location, visually.
For more information about enabling an ELK stack in your cloud, see How to Enable Elasticsearch and Kibana using Kolla Ansible.
Kolla Ansible Log Locations
Private Clouds are deployed using Kolla Ansible, an OpenStack deployment
system. This system deploys all OpenStack services into Docker
containers. Each service's log file is then stored as
<service-name> is an OpenStack
service, like Neutron for example.
For example, to see all logs associated with the Neutron service,
# ls /var/log/kolla/neutron/*.log
Determining the Correct Log
How do you know which log to look in for an issue? Which host should you be in? It is generally useful to know which OpenStack services are running and what their purposes are before determining which log file to examine to troubleshoot an issue. For a list of OpenStack services and their purpose, see the OpenStack Components page.
When diagnosing issues, consider the services associated with the action that may be failing. For example, if looking into an issue with creating a volume, consider looking at Cinder's various logs.
Determining the Correct Host
By default, your cloud has three control plane nodes. Each of these nodes has very similar logs, and typically, only one of the nodes is recording log events for a specific service. Due to this, you may need to examine all three host's logs in real time, replicate the issue, then see if any of those logs recorded any events.
To view the logs for all hosts at the same time, consider using a
terminal multiplexer, especially one where you can issue the same
commands in multiple SSH connections. The application
tmux is an
example of a terminal multiplexer.
There are several native ways to view the contents of a log file.
vim will suffice. For more
colorful output, consider using an application like
lnav, which has
proven especially useful for examining unfamiliar logs.