Installed packages

On Debian and Ubuntu systems, we can leverage the advanced package tool by running the apt command. This tool can be used to provide insight into which packages are installed on our system. This knowledge can be useful when troubleshooting OpenStack problems related to packages:

apt search openstack | less

The preceding command will list the packages that have the word openstack in their description and paginate the output. Running this command will give you a sense of some of the packages that come together to create an OpenStack cloud. Not all the packages listed are required, but this will give you an idea of which packages are available:

apt list | grep nova

The preceding apt list command will list the packages on the system. For example, we can pipe apt list to grep for nova to see a list of packages with nova in the name. We can take any of these packages and run them through apt show to get the details about the package. Take this line of command, for example:

apt show nova-api

General tools

We will make use of several Linux utilities throughout this book. Some of these tools are outlined in the following section.

The watch command

One of the many commands you may find useful when troubleshooting is the watch command. This command provides a convenient way to execute a command on a given time interval. I often use it to keep an eye on my processes when I'm trying to get them to restart. I've also leveraged this command when troubleshooting instance creation, as it allows me to check whether and when the instance becomes active:

watch pgrep -l nova 

The preceding command will run the pgrep -l nova command every two seconds by default. You can adjust the interval at which the command is run by passing the -n option:

watch -n 3 nova list

This command will run the nova list command every 3 seconds.

File tools

We will leverage some commonly used Linux tools when troubleshooting. These tools include the following:

• cat: This is used to print files or input to the standard output.
• less: This is used to view files and allows you to page through those files.
• find: This allows you to search through files in a hierarchy of directories.
• grep: This allows you to search through files for lines that match a given pattern. We will use the grep command quite a bit as we are searching through logs for different types of messages.
• tail: This allows you to output the last part of a file. We will leverage the tail command often with the -f argument, which will allow us to follow a file as it is updated. This is used to watch logs live as we run different services or commands.

Message broker tools

One of the central components of any OpenStack cluster is the messaging broker. OpenStack uses a message broker to pass information back and forth between its components. The message queue facilitates intra-component communication, and as a result, it can often be a useful place to search for troubleshooting clues in an OpenStack cluster. While the default message broker installed with OpenStack is RabbitMQ, deployers have the ability to select from several other messaging technologies including ZeroMQ or QPid. We will explore some high-level troubleshooting tools for RabbitMQ in the following sections.

RabbitMQ

The RabbitMQ message system comes with a handy utility named rabbitmqctl. This tool allows operators to complete several useful tasks, but here we will highlight a few that are particularly helpful to troubleshoot OpenStack.

The preceding command will return the status of your RabbitMQ message broker. It can be helpful to check the output of this command for any errors. For example, if you see an error that starts with Error: unable to connect to node, then this means that RabbitMQ is likely not running. You can run the following command on Ubuntu to try and start it:

sudo service rabbitmq-server start
rabbitmqctl stop

The preceding command will stop your RabbitMQ message broker. You can use the same service command from the one we used here to restart it.

The list_queues command will list the queues in your message broker. When you run this on an OpenStack cluster, you will be able to see the decent number of queues used by the software to pass messages back and forth between OpenStack components. In addition to the name of the queues, this command can show you several attributes for each queue. Select the attributes you want to see by passing them after your list_queues command.

In this command, we are requesting several columns of data directly after the list_queues command. Running the command this way will return a tab-delimited list including the name of the queue, whether or not the queue is durable, the number of messages ready to be read from the queue, the number of consumers listening to the queue, and the current queue state.

Like the list_queues command, there is also a list_exchanges command, which allows you to see the exchanges in your message broker. Running this command on an OpenStack cluster will allow you to see the exchanges that OpenStack leverages. Exchanges sit between message producers and the queues where those messages will eventually reside. The exchange is responsible for taking the message from the message producer and delivering that message to the appropriate queue if there is any queue at all:

rabbitmqctl list_exchanges name type durable policy

Running the list_exchanges command with the name, type, durable, and policy column headers, as demonstrated in the preceding line of code, will output the exchanges and their respective values for each column. Specifically, the name of the exchange; the exchange type which is either direct, topic, headers, or fanout; whether or not the exchange is durable, meaning it will survive a server restart; and finally, the exchange's policy. Policies are a method by which administrators can control the behavior of queues and exchanges across the entire RabbitMQ cluster. You can see which policies, if any, are configured by running this:

rabbitmqctl list_policies

In RabbitMQ, exchanges are related to queues through bindings. Queues use bindings to tell exchanges that they are interested in messages flowing through that exchange. You can list the bindings by running this code:

The list bindings command will display each of the bindings between exchanges and queues. By default, it will list the name of the source, the type of the source, the name of the destination, the type of the destination, a routing key, and any binding arguments.

To view a list of RabbitMQ clients with connections to the RabbitMQ server, run the list_connections command:

rabbitmqctl list_connections

This command will list the username associated with the connection, the hostname of the connected peer, the port of the peer, and the state of the connection.