Tempest is the official OpenStack test suite that runs integration tests against an OpenStack cloud to validate its healthiness and find out its (potential) problems. It contains a list of test classes, in three categories, being API, scenario, and stress. API tests validate API functionalities; scenario tests simulate complex multi-step operations; stress tests run several jobs in parallel to see if the service can sustain high workload. Tempest uses its own client implementation rather than existing Python clients so that it can send fake or invalid requests to check if APIs are implemented correctly.
Setting up Tempest is a challenge as the documentation is not very clear. Some problems may not be caused by the configuration, but the setup of your cloud. Fortunately, there are several tools that can help users take advantage of Tempest without going through too much pain.
In this series, we will show you how to run Tempest, standalone or with the help of another tool.
In Part 1 we described how to use the RefStack client to run tests. RefStack is a tool developed to assist DefCore capability testing. In Part 2, we will show you how to run tests with Rally.
Rally is a benchmarking tool that tests the scalability of an OpenStack cloud. Its main job is to run workloads in parallel in order to detect performance issues of a cloud, or find out the best configuration model to achieve best performance. Furthermore, it provides a feature to call Tempest and run tests against a cloud to ‘verify’ its functionalities. That is the feature we will explore in this blog post.
Rally’s documentation is very clear on installation and set up. I will list key steps here. We assume you are running Rally against an All-in-One OpenStack that is installed by following Part 1’s instructions.
Firstly, we install Rally. The Rally team has provided a script to automate the download and installation, which makes things a lot easier.
wget -q -O- https://raw.githubusercontent.com/openstack/rally/master/install_rally.sh | bash
Then we need to add a deployment to Rally. The easiest way is to do it via environment variables.
source openrc
rally deployment create --fromenv --name=mycloud
Next we can install Tempest in Rally.
rally verify install
The above command does a default installation and Tempest will be put into a virtual environment. But you can also specify a particular version of Tempest to install, and/or install Tempest into system library path.
rally verify genconfig
This command will generate a default config file. To view it, run the following command. It will also give you the path. So you can tweak it according to Part 1.
rally verify showconfig
Then we can start running Tempest. If nothing is specified, it will run all tests.
rally verify start
If we just want to run DefCore tests, like we did in Part 1, we can download the test list, e.g.
wget https://refstack.openstack.org/api/v1/guidelines/2016.01/tests?type=required -O defcore.txt
and update these variables in the auto-generated config file (like what we did in Part 1):
[compute]
image_ref = [your cirros image uuid]
image_ref_alt = [your cirros-alt image uuid]
flavor_ref = [your flavor uuid]
flavor_ref_alt = [your flavor uuid]
fixed_network_name = [your fixed network name]
[validation]
image_ssh_password = cubswin:)
The good thing is Rally allows you to use auto-generated accounts so you don’t need to create any for the test.
And then run rally with this list.
rally verify start --tests-file defcore.txt
Here is an example output.
...
======
Totals
======
Ran: 76 tests in 399.0000 sec.
- Passed: 75
- Skipped: 1
- Expected Fail: 0
- Unexpected Success: 0
- Failed: 0
Sum of execute time for each test: 593.3815 sec.
==============
Worker Balance
==============
- Worker 0 (14 tests) => 0:05:59.292310
- Worker 1 (23 tests) => 0:02:39.609214
- Worker 2 (22 tests) => 0:05:24.914187
- Worker 3 (17 tests) => 0:00:16.433136
2016-09-20 02:39:58.942 15238 INFO rally.verification.tempest.tempest [-] Verification ef55832c-d61c-4129-a052-2b7beacbc4d4 | Completed: Run verification.
2016-09-20 02:39:58.943 15238 INFO rally.verification.tempest.tempest [-] Verification ef55832c-d61c-4129-a052-2b7beacbc4d4 | Starting: Saving verification results.
2016-09-20 02:39:59.126 15238 INFO rally.verification.tempest.tempest [-] Verification ef55832c-d61c-4129-a052-2b7beacbc4d4 | Completed: Saving verification results.
Verification UUID: ef55832c-d61c-4129-a052-2b7beacbc4d4
Rally uses multiple threads to run tests so it is quite fast.
Then you can view the result in a table or in HTML format.
# rally verify show
Total results of verification:
+--------------------------------------+--------------------------------------+----------+-------+----------+----------------------------+----------+
| UUID | Deployment UUID | Set name | Tests | Failures | Created at | Status |
+--------------------------------------+--------------------------------------+----------+-------+----------+----------------------------+----------+
| ef55832c-d61c-4129-a052-2b7beacbc4d4 | 71fc6dc5-ca05-49d0-badc-0ef5b3f58225 | | 76 | 0 | 2016-09-20 02:33:16.006425 | finished |
+--------------------------------------+--------------------------------------+----------+-------+----------+----------------------------+----------+
Tests:
+-----------------------------------------------------------------------------------------------------------------------------------+---------+---------+
| name | time | status |
+-----------------------------------------------------------------------------------------------------------------------------------+---------+---------+
| tempest.api.compute.images.test_images.ImagesTestJSON.test_delete_saving_image | 10.859 | success |
| tempest.api.compute.images.test_images_oneserver.ImagesOneServerTestJSON.test_create_delete_image | 7.689 | success |
| tempest.api.compute.images.test_images_oneserver.ImagesOneServerTestJSON.test_create_image_specify_multibyte_character_image_name | 7.144 | success |
| tempest.api.compute.images.test_list_image_filters.ListImageFiltersTestJSON.test_list_images_filter_by_changes_since | 0.106 | success |
| tempest.api.compute.images.test_list_image_filters.ListImageFiltersTestJSON.test_list_images_filter_by_name | 0.097 | success |
| tempest.api.compute.images.test_list_image_filters.ListImageFiltersTestJSON.test_list_images_filter_by_server_id | 0.110 | success |
| tempest.api.compute.images.test_list_image_filters.ListImageFiltersTestJSON.test_list_images_filter_by_server_ref | 0.201 | success |
| tempest.api.compute.images.test_list_image_filters.ListImageFiltersTestJSON.test_list_images_filter_by_status | 0.164 | success |
| tempest.api.compute.images.test_list_image_filters.ListImageFiltersTestJSON.test_list_images_filter_by_type | 0.172 | success |
| tempest.api.compute.images.test_list_image_filters.ListImageFiltersTestJSON.test_list_images_limit_results | 0.260 | success |
| tempest.api.compute.images.test_list_image_filters.ListImageFiltersTestJSON.test_list_images_with_detail_filter_by_changes_since | 0.162 | success |
| tempest.api.compute.images.test_list_image_filters.ListImageFiltersTestJSON.test_list_images_with_detail_filter_by_name | 0.127 | success |
| tempest.api.compute.images.test_list_image_filters.ListImageFiltersTestJSON.test_list_images_with_detail_filter_by_server_ref | 0.289 | success |
| tempest.api.compute.images.test_list_image_filters.ListImageFiltersTestJSON.test_list_images_with_detail_filter_by_status | 0.251 | success |
| tempest.api.compute.images.test_list_image_filters.ListImageFiltersTestJSON.test_list_images_with_detail_filter_by_type | 0.180 | success |
| tempest.api.compute.images.test_list_image_filters.ListImageFiltersTestJSON.test_list_images_with_detail_limit_results | 0.103 | success |
| tempest.api.compute.images.test_list_images.ListImagesTestJSON.test_get_image | 0.842 | success |
| tempest.api.compute.images.test_list_images.ListImagesTestJSON.test_list_images | 0.707 | success |
| tempest.api.compute.images.test_list_images.ListImagesTestJSON.test_list_images_with_detail | 0.147 | success |
| tempest.api.compute.servers.test_create_server.ServersTestJSON.test_host_name_is_same_as_server_name | 32.323 | success |
| tempest.api.compute.servers.test_create_server.ServersTestJSON.test_list_servers_with_detail | 0.298 | success |
| tempest.api.compute.servers.test_create_server.ServersTestJSON.test_verify_created_server_vcpus | 0.821 | success |
| tempest.api.compute.servers.test_create_server.ServersTestManualDisk.test_host_name_is_same_as_server_name | 29.806 | success |
| tempest.api.compute.servers.test_create_server.ServersTestManualDisk.test_list_servers_with_detail | 0.258 | success |
| tempest.api.compute.servers.test_create_server.ServersTestManualDisk.test_verify_created_server_vcpus | 0.426 | success |
| tempest.api.compute.servers.test_instance_actions.InstanceActionsTestJSON.test_get_instance_action | 0.332 | success |
| tempest.api.compute.servers.test_instance_actions.InstanceActionsTestJSON.test_list_instance_actions | 3.442 | success |
| tempest.api.compute.servers.test_list_server_filters.ListServerFiltersTestJSON.test_list_servers_detailed_filter_by_flavor | 0.441 | success |
| tempest.api.compute.servers.test_list_server_filters.ListServerFiltersTestJSON.test_list_servers_detailed_filter_by_image | 0.341 | success |
| tempest.api.compute.servers.test_list_server_filters.ListServerFiltersTestJSON.test_list_servers_detailed_filter_by_server_name | 0.432 | success |
| tempest.api.compute.servers.test_list_server_filters.ListServerFiltersTestJSON.test_list_servers_detailed_filter_by_server_status | 0.359 | success |
| tempest.api.compute.servers.test_list_server_filters.ListServerFiltersTestJSON.test_list_servers_detailed_limit_results | 0.519 | success |
| tempest.api.compute.servers.test_list_server_filters.ListServerFiltersTestJSON.test_list_servers_filter_by_flavor | 0.298 | success |
| tempest.api.compute.servers.test_list_server_filters.ListServerFiltersTestJSON.test_list_servers_filter_by_image | 0.100 | success |
| tempest.api.compute.servers.test_list_server_filters.ListServerFiltersTestJSON.test_list_servers_filter_by_limit | 0.136 | success |
| tempest.api.compute.servers.test_list_server_filters.ListServerFiltersTestJSON.test_list_servers_filter_by_server_name | 0.165 | success |
| tempest.api.compute.servers.test_list_server_filters.ListServerFiltersTestJSON.test_list_servers_filter_by_server_status | 0.108 | success |
| tempest.api.compute.servers.test_list_server_filters.ListServerFiltersTestJSON.test_list_servers_filtered_by_ip | 0.496 | success |
| tempest.api.compute.servers.test_list_server_filters.ListServerFiltersTestJSON.test_list_servers_filtered_by_ip_regex | 0.007 | skip |
| tempest.api.compute.servers.test_list_server_filters.ListServerFiltersTestJSON.test_list_servers_filtered_by_name_wildcard | 0.254 | success |
| tempest.api.compute.servers.test_list_servers_negative.ListServersNegativeTestJSON.test_list_servers_by_limits | 0.082 | success |
| tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON.test_lock_unlock_server | 37.387 | success |
| tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON.test_rebuild_server | 80.857 | success |
| tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON.test_resize_server_confirm | 41.418 | success |
| tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON.test_resize_server_revert | 50.959 | success |
| tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON.test_stop_start_server | 44.271 | success |
| tempest.api.compute.servers.test_server_metadata.ServerMetadataTestJSON.test_delete_server_metadata_item | 1.025 | success |
| tempest.api.compute.servers.test_server_metadata.ServerMetadataTestJSON.test_get_server_metadata_item | 0.910 | success |
| tempest.api.compute.servers.test_server_metadata.ServerMetadataTestJSON.test_list_server_metadata | 0.624 | success |
| tempest.api.compute.servers.test_server_metadata.ServerMetadataTestJSON.test_set_server_metadata | 0.832 | success |
| tempest.api.compute.servers.test_server_metadata.ServerMetadataTestJSON.test_set_server_metadata_item | 0.977 | success |
| tempest.api.compute.servers.test_server_metadata.ServerMetadataTestJSON.test_update_server_metadata | 0.678 | success |
| tempest.api.compute.servers.test_servers.ServersTestJSON.test_create_server_with_admin_password | 2.509 | success |
| tempest.api.compute.servers.test_servers.ServersTestJSON.test_create_specify_keypair | 7.713 | success |
| tempest.api.compute.servers.test_servers.ServersTestJSON.test_create_with_existing_server_name | 15.460 | success |
| tempest.api.compute.servers.test_servers.ServersTestJSON.test_update_access_server_address | 9.006 | success |
| tempest.api.compute.servers.test_servers.ServersTestJSON.test_update_server_name | 7.960 | success |
| tempest.api.compute.test_quotas.QuotasTestJSON.test_get_default_quotas | 0.427 | success |
| tempest.api.compute.test_quotas.QuotasTestJSON.test_get_quotas | 0.127 | success |
| tempest.api.compute.volumes.test_attach_volume.AttachVolumeTestJSON.test_attach_detach_volume | 128.877 | success |
| tempest.api.compute.volumes.test_attach_volume.AttachVolumeTestJSON.test_list_get_volume_attachments | 42.423 | success |
| tempest.api.compute.volumes.test_volumes_list.VolumesTestJSON.test_volume_list | 0.142 | success |
| tempest.api.compute.volumes.test_volumes_list.VolumesTestJSON.test_volume_list_with_details | 0.142 | success |
| tempest.api.identity.v3.test_tokens.TokensV3Test.test_create_token | 0.147 | success |
| tempest.api.image.v2.test_images.ListImagesTest.test_list_no_params | 0.100 | success |
| tempest.api.object_storage.test_object_expiry.ObjectExpiryTest.test_get_object_after_expiry_time | 12.339 | success |
| tempest.api.object_storage.test_object_services.ObjectTest.test_copy_object_2d_way | 1.457 | success |
| tempest.api.object_storage.test_object_services.ObjectTest.test_copy_object_across_containers | 0.482 | success |
| tempest.api.object_storage.test_object_services.ObjectTest.test_copy_object_in_same_container | 0.214 | success |
| tempest.api.object_storage.test_object_services.ObjectTest.test_copy_object_to_itself | 0.186 | success |
| tempest.api.object_storage.test_object_services.ObjectTest.test_delete_object | 0.113 | success |
| tempest.api.object_storage.test_object_services.ObjectTest.test_get_object_if_different | 0.096 | success |
| tempest.api.object_storage.test_object_services.ObjectTest.test_object_upload_in_segments | 1.027 | success |
| tempest.api.object_storage.test_object_temp_url.ObjectTempUrlTest.test_get_object_using_temp_url | 0.057 | success |
| tempest.api.object_storage.test_object_temp_url.ObjectTempUrlTest.test_put_object_using_temp_url | 0.106 | success |
| tempest.api.object_storage.test_object_version.ContainerTest.test_versioned_container | 1.281 | success |
+-----------------------------------------------------------------------------------------------------------------------------------+---------+---------+
To output results to a HTML file, run
rally verify detail --html > /tmp/test.html
Then you can open it in a web browser.
