Troubleshooting OpenStack

A Quick Study Guide for the COA Exam

A quick reference study guide to use for troubleshooting OpenStack components and to prepare students for the Certified OpenStack Administrator (COA) exam. If you’d like to learn more, and practice hands-on exercises to prepare you for the exam, check out our online OpenStack training courses.

* Please note that some commands listed in this guide are for use in CentOS and RedHat only. As such, service names may change in other operating systems such as Ubuntu.

OpenStack Troubleshooting - A Quick Study Guide for the COA Exam: OpenStack Project Mascots

Keystone Identity System

About:

Provides API client authentication, service discovery, and distributed multi-tenant authorization. Administrator level access that bypasses users, tenants, roles, services and endpoints.

Logs:

Keystone logs: /var/log/keystone and /var/log/httpd/keystone_wsgi_*.log (if Keystone is running as an Apache wsgi application)

Troubleshooting:

Show detailed output: set debug=True in /etc/keystone/keystone.conf

OpenStack Troubleshooting - A Quick Study Guide for the COA Exam: KeyStone Turtle

Nova Compute Service

About:

Massively scalable, on demand, self service access to compute resources. Nova has multiple components: api, cert, conductor, consoleauth, scheduler, novncproxy.

Logs:

  • Components log:
    ‣ /var/log/nova/nova-­api.log
    ‣ /var/log/nova/nova-­cert.log

    ‣ /var/log/nova/nova-­conductor.log
    ‣ /var/log/nova/nova-­consoleauth.log
    ‣ /var/log/nova/nova-­scheduler.log
    ‣ /var/log/nova/nova-­novncproxy.log
  • Instance specification: /var/lib/nova/instances/INSTANCE_UUID/libvirt.xml
  • Instance dmesg output: /var/lib/nova/instances/INSTANCE_UUID/console.log
  • Libvirt instance log: /var/log/libvirt/qemu/instance-INSTANCE_UUID.log
Troubleshooting:

  • Show detailed output: set debug=True in /etc/nova/nova.conf
  • See state of nova components in the control and compute planes: nova service-list (as user with admin role)
  • Verify nova can correctly proxy image API calls to glance: nova image-­list
  • Verify nova can correctly proxy network API calls to neutron: nova secgroup­-list or nova net-­list
  • Verify nova can correctly proxy storage API calls to cinder: nova volume­-list
  • Check VM configuration and status with libvirt tools: virsh
OpenStack Troubleshooting - A Quick Study Guide for the COA Exam: Nova Supernova

Neutron Network

About:

Provides network connectivity as a service between interface devices.

Logs:

  • Neutron-server logs: /var/log/neutron/server.log
  • Neutron plugins & agents logs: /var/log/neutron/
Troubleshooting:

  • Show detailed output: set debug=True in /etc/neutron/neutron.conf
  • See state of neutron agents running in the network and compute plane: run neutron agent-­list (as user with admin role)
  • Show detailed output for Neutron Plugins & Agent logs: set debug=True in the appropriate agent or plugin INI file
  • Check network configuration and ip namespace: ip -o addr
  • Check ip namespace: ip netns
  • Check OpenVSwitch bridges and ports: ovs-vsctl show
  • Check OpenVSwitch datapath and packets: ovs-dpctl
  • Packet analysis: tcpdump
OpenStack Troubleshooting - A Quick Study Guide for the COA Exam: Neutron Spider Web

Glance Image Service

About:

Glance provides discovery, registration and delivery services for disk and server images. Glance has three components: api, glare and registry.

Logs:

  • Glance-api logs: /var/log/glance/api.log
  • Glance-glare logs: /var/log/glance/glare.log
  • Glance-registry logs: /var/log/glance/registry.log
Troubleshooting:

  • Show detailed output: set debug=True in:
    Glance-api logs: /var/log/glance/api.log
    Glance-glare logs: /var/log/glance/glare.log
    Glance-registry logs: /var/log/glance/registry.log
OpenStack Troubleshooting - A Quick Study Guide for the COA Exam: Glance Chipmunk

Cinder Block Storage

About:

Cinder is designed to present storage resources to end users that can be consumed by the OpenStack Compute Project (Nova). Cinder has four components: api, backup, scheduler and volume.

Logs:

  • Cinder-api logs: /var/log/cinder/cinder-­api.log
  • Cinder-backup logs: /var/log/cinder/cinder-­backup.log
  • Cinder-scheduler logs: /var/log/cinder/cinder­-scheduler.log
  • Cinder-volume logs: /var/log/cinder/cinder­-volume.log
Troubleshooting:

  • Show detailed output: set debug=True in /etc/cinder/cinder.conf
  • View state of cinder components in control and storage planes: cinder service-­list (as user with admin role)
  • LVM backend, use LVM commands to check volumes: lvs
OpenStack Troubleshooting - A Quick Study Guide for the COA Exam: Cinder Horse

Telemetry Data Collection

About:

Telemetry is a project designed to reliably collect data on the utilization of the physical and virtual resources comprising deployed clouds, persist these data for subsequent retrieval and analysis, and trigger actions when defined criteria are met. It consists of four projects: Aodh, Ceilometer, Gnocchi and Panko.

Logs:

  • Aodh log: /var/log/aodh/*.log
  • Ceilometer log: /var/log/ceilometer/*.log
  • Gnocchi log: /var/log/gnocchi/*.log
  • Panko log: /var/log/panko/*.log
Troubleshooting:

  • Show detailed output
    set debug=True in /etc/aodh/aodh.conf
    set debug=True in /etc/aodh/ceilometer.conf
    set debug=True in /etc/aodh/gnocchi.conf
    set debug=True in /etc/aodh/panko.conf
  • Check if processes are running:
    ‣ systemctl status openstack-aodh-*
    ‣ systemctl status openstack-ceilometer-*
    ‣ systemctl status openstack-gnocchi-*
    ‣ systemctl status openstack-panko-*
    Note that their api services can run as Apache wsgi applications. In that case you need to check if apache is running.
  • List metrics: openstack metric metric list
OpenStack Troubleshooting - A Quick Study Guide for the COA Exam: Telemetry Ceilometer Meerkat

Swift Object Store

About:

Cloud storage software to store and retrieve lots of data with a simple API. Ideal for storing unstructured data that can grow without bound.

Logs:

• Swift log: /var/log/swift/*.log

Troubleshooting:

  • List objects: swift list
  • Create or download objects: swift upload/download
  • Show detailed output: change log_level in /etc/swift/*-server.conf and restart the corresponding service
  • Get swift object info: swift-object-info
OpenStack Troubleshooting - A Quick Study Guide for the COA Exam: KeyStone Turtle

Heat Orchestration

About:

An orchestration engine to launch multiple composite cloud applications based on templates in the form of text files that can be treated like code. Heat has two main components heat-api and heat-engine.

Logs:

  • Heat-api log: /var/log/heat/heat-api.log
  • Heat-engine log: /var/log/heat/heat-engine.log
Troubleshooting:

  • List heat stacks: openstack stack list
  • Show details of a stack: openstack stack show STACK_UUID
  • List heat services running: openstack orchestration service list
  • Check if heat services are running: systemctl status openstack-heat-*
OpenStack Troubleshooting - A Quick Study Guide for the COA Exam: Heat Flame Fire

Start Learning

Learn from Instructors with Real World Expertise.