OpenStack Rules: How OpenVswitch works inside OpenStack

OpenStack Fuel Rules: How OpenVSwitch works inside OpenStack

Understanding OpenFlow rules

OpenVswitch (OVS) is a virtual switch that connects virtual machines together using virtual links and ports. Traditionally this would be done by a physical switch over physical links and network cards and switch ports. In OpenStack, OVS also plays an important role which provides virtualised network services and both the Neutron node, and the compute node are running OpenVSwitches.

But what is important about OVS is its role in manipulating and directing the coming in and out. In this article we intend to describe the flow rules installed on OVS inside OpenStack Mitaka.

For example:
ssh root@192.168.127.101

[root@mitaka ~]# ssh compute Welcome to Ubuntu 14.04.5 LTS (GNU/Linux 4.4.0-135-generic x86_64) * Documentation: https://help.ubuntu.com/ Last login: Wed Sep 26 06:40:57 2018 from 10.20.0.2 root@node-4:~#

Print the information of the br-tun of OpenStack as it provides communication inside and outside of the OpenStack:

root@node-4:~# ovs-ofctl dump-flows br-tun NXST_FLOW reply (xid=0x4): 1- cookie=0xbb7b3cdd52626a01, duration=13003.029s, table=0, n_packets=183, n_bytes=28498, idle_age=4, priority=1,in_port=1 actions=resubmit(,2) 2- cookie=0xbb7b3cdd52626a01, duration=9917.985s, table=0, n_packets=198, n_bytes=36045, idle_age=4, priority=1,in_port=2 actions=resubmit(,4) 3- cookie=0xbb7b3cdd52626a01, duration=13003.030s, table=0, n_packets=0, n_bytes=0, idle_age=13003, priority=0 actions=drop 4- cookie=0xbb7b3cdd52626a01, duration=13003.029s, table=2, n_packets=1, n_bytes=42, idle_age=9913, priority=1,arp,dl_dst=ff:ff:ff:ff:ff:ff actions=resubmit(,21) 5- cookie=0xbb7b3cdd52626a01, duration=13003.029s, table=2, n_packets=168, n_bytes=26780, idle_age=4, priority=0,dl_dst=00:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,20) 6- cookie=0xbb7b3cdd52626a01, duration=13003.029s, table=2, n_packets=14, n_bytes=1676, idle_age=9904, priority=0,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,22) 7- cookie=0xbb7b3cdd52626a01, duration=13003.029s, table=3, n_packets=0, n_bytes=0, idle_age=13003, priority=0 actions=drop 8- cookie=0xbb7b3cdd52626a01, duration=9921.166s, table=4, n_packets=198, n_bytes=36045, idle_age=4, priority=1,tun_id=0x2 actions=mod_vlan_vid:1,resubmit(,10) 9- cookie=0xbb7b3cdd52626a01, duration=13003.029s, table=4, n_packets=0, n_bytes=0, idle_age=13003, priority=0 actions=drop 10- cookie=0xbb7b3cdd52626a01, duration=13003.029s, table=6, n_packets=0, n_bytes=0, idle_age=13003, priority=0 actions=drop 11- cookie=0xbb7b3cdd52626a01, duration=13003.029s, table=10, n_packets=198, n_bytes=36045, idle_age=4, priority=1 actions=learn(table=20,hard_timeout=300,priority=1,cookie=0xbb7b3cdd52626a01,NXM_OF_VLAN_TCI[0..11],NXM_OF_ETH_DST[]=NXM_OF_ETH_SRC[],load:0->NXM_OF_VLAN_TCI[],load:NXM_NX_TUN_ID[]->NXM_NX_TUN_ID[],output:NXM_OF_IN_PORT[]),output:1 12- cookie=0xbb7b3cdd52626a01, duration=9917.984s, table=20, n_packets=102, n_bytes=14108, idle_age=9435, priority=2,dl_vlan=1,dl_dst=fa:16:3e:0b:cf:10 actions=strip_vlan,set_tunnel:0x2,output:2 13- cookie=0xbb7b3cdd52626a01, duration=9917.984s, table=20, n_packets=66, n_bytes=12672, idle_age=4, priority=2,dl_vlan=1,dl_dst=fa:16:3e:4a:10:2b actions=strip_vlan,set_tunnel:0x2,output:2 14- cookie=0xbb7b3cdd52626a01, duration=9913.613s, table=20, n_packets=0, n_bytes=0, hard_timeout=300, idle_age=9913, hard_age=4, priority=1,vlan_tci=0x0001/0x0fff,dl_dst=fa:16:3e:4a:10:2b actions=load:0->NXM_OF_VLAN_TCI[],load:0x2->NXM_NX_TUN_ID[],output:2 15- cookie=0xbb7b3cdd52626a01, duration=13003.029s, table=20, n_packets=0, n_bytes=0, idle_age=13003, priority=0 actions=resubmit(,22) 16- cookie=0xbb7b3cdd52626a01, duration=9917.985s, table=21, n_packets=1, n_bytes=42, idle_age=9913, priority=1,arp,dl_vlan=1,arp_tpa=192.168.111.1 actions=move:NXM_OF_ETH_SRC[]->NXM_OF_ETH_DST[],mod_dl_src:fa:16:3e:0b:cf:10,load:0x2->NXM_OF_ARP_OP[],move:NXM_NX_ARP_SHA[]->NXM_NX_ARP_THA[],move:NXM_OF_ARP_SPA[]->NXM_OF_ARP_TPA[],load:0xfa163e0bcf10->NXM_NX_ARP_SHA[],load:0xc0a86f01->NXM_OF_ARP_SPA[],IN_PORT 17- cookie=0xbb7b3cdd52626a01, duration=9917.984s, table=21, n_packets=0, n_bytes=0, idle_age=9917, priority=1,arp,dl_vlan=1,arp_tpa=192.168.111.2 actions=move:NXM_OF_ETH_SRC[]->NXM_OF_ETH_DST[],mod_dl_src:fa:16:3e:4a:10:2b,load:0x2->NXM_OF_ARP_OP[],move:NXM_NX_ARP_SHA[]->NXM_NX_ARP_THA[],move:NXM_OF_ARP_SPA[]->NXM_OF_ARP_TPA[],load:0xfa163e4a102b->NXM_NX_ARP_SHA[],load:0xc0a86f02->NXM_OF_ARP_SPA[],IN_PORT 18- cookie=0xbb7b3cdd52626a01, duration=13003.028s, table=21, n_packets=0, n_bytes=0, idle_age=13003, priority=0 actions=resubmit(,22) 19- cookie=0xbb7b3cdd52626a01, duration=9917.956s, table=22, n_packets=10, n_bytes=1336, idle_age=9904, dl_vlan=1 actions=strip_vlan,set_tunnel:0x2,output:2 20- cookie=0xbb7b3cdd52626a01, duration=13003.002s, table=22, n_packets=4, n_bytes=340, idle_age=9920, priority=0 actions=drop

Explanation of the Rules:

Table 0:
1- cookie=0xbb7b3cdd52626a01, duration=13003.029s, table=0, n_packets=183, n_bytes=28498, idle_age=4, priority=1,in_port=1 actions=resubmit(,2) 2- cookie=0xbb7b3cdd52626a01, duration=9917.985s, table=0, n_packets=198, n_bytes=36045, idle_age=4, priority=1,in_port=2 actions=resubmit(,4) 3- cookie=0xbb7b3cdd52626a01, duration=13003.030s, table=0, n_packets=0, n_bytes=0, idle_age=13003, priority=0 actions=drop

Rule 1	Has priority=1 and checks if the packets coming on port in_port=”patch-int” then the action is: go to table 2
Rule 2	Checks if the packets coming on port in_port=vxlan-c0a80202 then the action is: go to table 4
Rule 3	Has priority=0 (lowest priority) and drop the packets that don’t match rule 1 and rule 2

Table 2:
4- cookie=0xbb7b3cdd52626a01, duration=13003.029s, table=2, n_packets=1, n_bytes=42, idle_age=9913, priority=1,arp,dl_dst=ff:ff:ff:ff:ff:ff actions=resubmit(,21) 5- cookie=0xbb7b3cdd52626a01, duration=13003.029s, table=2, n_packets=168, n_bytes=26780, idle_age=4, priority=0,dl_dst=00:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,20) 6- cookie=0xbb7b3cdd52626a01, duration=13003.029s, table=2, n_packets=14, n_bytes=1676, idle_age=9904, priority=0,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,22)

Rule 4

Has priority=1 and checks if the packets are ARP packet with destination MAC address set to broadcast

then the action is: go to table 21

Rule 5

Has priority=0 and checks if the packets has dl_dst=00:00:00:00:00:00/01:00:00:00:00:00

(match all unicast Ethernet packets) then the action is: go to table 20

Rule 6

Has priority=0 and checks if the packets has dl_dst=01:00:00:00:00:00/01:00:00:00:00:00

(match all multicast(including broadcast Ethernet packets) then the action is: go to table 22

Table 3:

7- cookie=0xbb7b3cdd52626a01, duration=13003.029s, table=3, n_packets=0, n_bytes=0, idle_age=13003, priority=0 actions=drop

Rule 7

drop the packets

Table 4:

8- cookie=0xbb7b3cdd52626a01, duration=9921.166s, table=4, n_packets=198, n_bytes=36045, idle_age=4, priority=1,tun_id=0x2 actions=mod_vlan_vid:1,resubmit(,10) 9- cookie=0xbb7b3cdd52626a01, duration=13003.029s, table=4, n_packets=0, n_bytes=0, idle_age=13003, priority=0 actions=drop

Rule 8	Has priority=1 and checks if the packets tun_id=0x20 hen the action is to add the vlan_vid:1 and go to table 10
Rule 9	Has priority=0 (lower priority) and drop the packets that don’t match rule 8

Table 6:

10- cookie=0xbb7b3cdd52626a01, duration=13003.029s, table=6, n_packets=0, n_bytes=0, idle_age=13003, priority=0 actions=drop

Rule 10

drop the packets

Table 10:

11- cookie=0xbb7b3cdd52626a01, duration=13003.029s, table=10, n_packets=198, n_bytes=36045, idle_age=4, priority=1 actions=learn(table=20,hard_timeout=300,priority=1,cookie=0xbb7b3cdd52626a01,NXM_OF_VLAN_TCI[0..11],NXM_OF_ETH_DST[]=NXM_OF_ETH_SRC[],load:0->NXM_OF_VLAN_TCI[],load:NXM_NX_TUN_ID[]->NXM_NX_TUN_ID[],output:NXM_OF_IN_PORT[]),output:1

Rule 11

Has priority=1 and the action has two parts:

Part one:

Is to install a rule in table 20. This table (20) will be a MAC learning table.

The “learn” action modifies a flow table based on the content of the flow currently being processed by table 4.

Here’s how you can interpret each part of the “learn” action above:

table=20 Modify flow table 20. This will be the MAC learning table.

hard_timeout=300

Causes the flow to expire after the 300 seconds, regardless of activity.

priority=1

The priority at which a wildcarded entry will match in comparison to others

cookie=0x407518fa3ccd67d2 NXM_OF_VLAN_TCI[0..11] Make the flow that we add to flow table 20 match the same VLAN ID that the packet we’re currently processing contains. This effectively scopes the MAC learning entry to a single VLAN, which is the ordinary behavior for a VLAN-aware switch. NXM_OF_ETH_DST[]=NXM_OF_ETH_SRC[] Make the flow that we add to flow table 20 match, as Ethernet destination, the Ethernet source address of the packet we’re currently processing.

load:0->NXM_OF_VLAN_TCI[],

Strip off the VLAN ID by loading 0 as a VLAN ID

load:NXM_NX_TUN_ID[]->NXM_NX_TUN_ID[],

Load the tunnel ID of the proceesing packet as a tunnel id of the packet

output:OXM_OF_IN_PORT[]),

Send the packet out via input port

Part Two:

output:”patch-int”

sends the packet out via port patch-int

Table 20:
12- cookie=0xbb7b3cdd52626a01, duration=9917.984s, table=20, n_packets=102, n_bytes=14108, idle_age=9435, priority=2,dl_vlan=1,dl_dst=fa:16:3e:0b:cf:10 actions=strip_vlan,set_tunnel:0x2,output:2 13- cookie=0xbb7b3cdd52626a01, duration=9917.984s, table=20, n_packets=66, n_bytes=12672, idle_age=4, priority=2,dl_vlan=1,dl_dst=fa:16:3e:4a:10:2b actions=strip_vlan,set_tunnel:0x2,output:2 14- cookie=0xbb7b3cdd52626a01, duration=9913.613s, table=20, n_packets=0, n_bytes=0, hard_timeout=300, idle_age=9913, hard_age=4, priority=1,vlan_tci=0x0001/0x0fff,dl_dst=fa:16:3e:4a:10:2b actions=load:0->NXM_OF_VLAN_TCI[],load:0x2->NXM_NX_TUN_ID[],output:2 15- cookie=0xbb7b3cdd52626a01, duration=13003.029s, table=20, n_packets=0, n_bytes=0, idle_age=13003, priority=0 actions=resubmit(,22)

Rule 12,13

Have priority=2 and check if the packets has VLAN id = 1 and a certain dl_dst addresses

then the action is: strip the VLAN id and load the tunnel id of 0x2 and send the packets out via output:vxlan-c0a80202

Rule 14

These rule are installed via the learn action of table 10:

Has priority=1 and checks if the packets has vlan_tci=0x0001/0x0fff (VLAN id = 1) and ,dl_dst=fa:16:3e:4a:10:2b

then the action is: strip the VLAN id and load the tunnel id of 0x2 and send the packets out via output:vxlan-c0a80202

Rule 15

Has priority=0 (lower priority) and the action is: go to table 22

Table 21:
16- cookie=0xbb7b3cdd52626a01, duration=9917.985s, table=21, n_packets=1, n_bytes=42, idle_age=9913, priority=1,arp,dl_vlan=1,arp_tpa=192.168.111.1 actions=move:NXM_OF_ETH_SRC[]->NXM_OF_ETH_DST[],mod_dl_src:fa:16:3e:0b:cf:10,load:0x2->NXM_OF_ARP_OP[],move:NXM_NX_ARP_SHA[]->NXM_NX_ARP_THA[],move:NXM_OF_ARP_SPA[]->NXM_OF_ARP_TPA[],load:0xfa163e0bcf10->NXM_NX_ARP_SHA[],load:0xc0a86f01->NXM_OF_ARP_SPA[],IN_PORT 17- cookie=0xbb7b3cdd52626a01, duration=9917.984s, table=21, n_packets=0, n_bytes=0, idle_age=9917, priority=1,arp,dl_vlan=1,arp_tpa=192.168.111.2 actions=move:NXM_OF_ETH_SRC[]->NXM_OF_ETH_DST[],mod_dl_src:fa:16:3e:4a:10:2b,load:0x2->NXM_OF_ARP_OP[],move:NXM_NX_ARP_SHA[]->NXM_NX_ARP_THA[],move:NXM_OF_ARP_SPA[]->NXM_OF_ARP_TPA[],load:0xfa163e4a102b->NXM_NX_ARP_SHA[],load:0xc0a86f02->NXM_OF_ARP_SPA[],IN_PORT 18- cookie=0xbb7b3cdd52626a01, duration=13003.028s, table=21, n_packets=0, n_bytes=0, idle_age=13003, priority=0 actions=resubmit(,22)

Rule 16, 17

Has priority=1 and checks if the packets are ARP packet and have certain VLAN ID (e.g. dl_vlan=1) and

a certain destination IP address (e.g. arp_tpa=192.168.111.1)

then the action of the flow is:

move:NXM_OF_ETH_SRC[]->NXM_OF_ETH_DST[] → move the Ethernet destination of the processing packet as an Ethernet source address of the flow
mod_dl_src:fa:16:3e:0b:cf:10 → change the Ethernet source address to a certain value
load:0x2→NXM_OF_ARP_OP[] → Load the tunnel ID 0x2
move:NXM_NX_ARP_SHA[]→NXM_NX_ARP_THA[] → move the ARP source MAC address of the processing packet as an ARP target MAC address of the flow
move:NXM_OF_ARP_SPA[]→NXM_OF_ARP_TPA[] → move the ARP source IP address of the processing packet as an ARP target IP address of the flow
load:0xfa163e0bcf10→NXM_NX_ARP_SHA[] → load 0xfa163e0bcf10 as an ARP source MAC address
load:0xc0a86f01→NXM_OF_ARP_SPA[] → load 0xc0a86f01 as an ARP IP address
IN_PORT → send the packet out via input port

Note: the above flow indicate that the switch which is close to the host replies to arp MAC address

Rule 18

Has priority=0 (lower priority) and the action is: go to table 22

Table 22:
19- cookie=0xbb7b3cdd52626a01, duration=9917.956s, table=22, n_packets=10, n_bytes=1336, idle_age=9904, dl_vlan=1 actions=strip_vlan,set_tunnel:0x2,output:2 20- cookie=0xbb7b3cdd52626a01, duration=13003.002s, table=22, n_packets=4, n_bytes=340, idle_age=9920, priority=0 actions=drop

Rule 19	Checks if the packet has VLAN ID=1 then the action is: strip the VLAN id and load the tunnel id of 0x2 and send the packets out via output:vxlan-c0a80202
Rule 20	Has priority=0 (lower priority) and drop the packets that don’t match rule 19

Having a good understanding of these rules will help us troubleshooting network traffic. If there are any connectivity issues in the network (internal/external) which result in the packet loss, we can easily follow the trail of packets within the engaged flow rules to find the leakage in the network.

For example, if we run ping between two OpenStack endpoints, first we need to understand which flow rules are being hit by the ping packets and then observe if there are any incremental changes in the “n_packets” count of the rule. The “n_packet” feature inform us if the packets are begin forwarded to another endpoint or being dropped in the network.