Technical Documentation

OpenStack Rules: How OpenVswitch works inside OpenStack

OpenStack Fuel Rules: How OpenVSwitch works inside OpenStack

Understanding OpenFlow rules

OpenVswitch (OVS) is a virtual switch that connects virtual machines together using virtual links and ports. Traditionally this would be done by a physical switch over physical links and network cards and switch ports. In OpenStack, OVS also plays an important role which provides virtualised network services and both the Neutron node, and the compute node are running OpenVSwitches.

But what is important about OVS is its role in manipulating and directing the coming in and out. In this article we intend to describe the flow rules installed on OVS inside OpenStack Mitaka.

Login to Mitaka node using the following:

ssh root@Mitaka’s IP address

For example:

ssh root@192.168.127.101

Login to the compute node:


[root@mitaka ~]# ssh compute
Welcome to Ubuntu 14.04.5 LTS (GNU/Linux 4.4.0-135-generic x86_64)
* Documentation: https://help.ubuntu.com/
Last login: Wed Sep 26 06:40:57 2018 from 10.20.0.2
root@node-4:~#

Print the information of the br-tun of OpenStack as it provides communication inside and outside of the OpenStack:


root@node-4:~# ovs-ofctl dump-flows br-tun
NXST_FLOW reply (xid=0x4):
1- cookie=0xbb7b3cdd52626a01, duration=13003.029s, table=0, n_packets=183, n_bytes=28498, idle_age=4, priority=1,in_port=1 actions=resubmit(,2)
2- cookie=0xbb7b3cdd52626a01, duration=9917.985s, table=0, n_packets=198, n_bytes=36045, idle_age=4, priority=1,in_port=2 actions=resubmit(,4)
3- cookie=0xbb7b3cdd52626a01, duration=13003.030s, table=0, n_packets=0, n_bytes=0, idle_age=13003, priority=0 actions=drop
4- cookie=0xbb7b3cdd52626a01, duration=13003.029s, table=2, n_packets=1, n_bytes=42, idle_age=9913, priority=1,arp,dl_dst=ff:ff:ff:ff:ff:ff actions=resubmit(,21)
5- cookie=0xbb7b3cdd52626a01, duration=13003.029s, table=2, n_packets=168, n_bytes=26780, idle_age=4, priority=0,dl_dst=00:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,20)
6- cookie=0xbb7b3cdd52626a01, duration=13003.029s, table=2, n_packets=14, n_bytes=1676, idle_age=9904, priority=0,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,22)
7- cookie=0xbb7b3cdd52626a01, duration=13003.029s, table=3, n_packets=0, n_bytes=0, idle_age=13003, priority=0 actions=drop
8- cookie=0xbb7b3cdd52626a01, duration=9921.166s, table=4, n_packets=198, n_bytes=36045, idle_age=4, priority=1,tun_id=0x2 actions=mod_vlan_vid:1,resubmit(,10)
9- cookie=0xbb7b3cdd52626a01, duration=13003.029s, table=4, n_packets=0, n_bytes=0, idle_age=13003, priority=0 actions=drop
10- cookie=0xbb7b3cdd52626a01, duration=13003.029s, table=6, n_packets=0, n_bytes=0, idle_age=13003, priority=0 actions=drop
11- cookie=0xbb7b3cdd52626a01, duration=13003.029s, table=10, n_packets=198, n_bytes=36045, idle_age=4, priority=1 actions=learn(table=20,hard_timeout=300,priority=1,cookie=0xbb7b3cdd52626a01,NXM_OF_VLAN_TCI[0..11],NXM_OF_ETH_DST[]=NXM_OF_ETH_SRC[],load:0->NXM_OF_VLAN_TCI[],load:NXM_NX_TUN_ID[]->NXM_NX_TUN_ID[],output:NXM_OF_IN_PORT[]),output:1
12- cookie=0xbb7b3cdd52626a01, duration=9917.984s, table=20, n_packets=102, n_bytes=14108, idle_age=9435, priority=2,dl_vlan=1,dl_dst=fa:16:3e:0b:cf:10 actions=strip_vlan,set_tunnel:0x2,output:2
13- cookie=0xbb7b3cdd52626a01, duration=9917.984s, table=20, n_packets=66, n_bytes=12672, idle_age=4, priority=2,dl_vlan=1,dl_dst=fa:16:3e:4a:10:2b actions=strip_vlan,set_tunnel:0x2,output:2
14- cookie=0xbb7b3cdd52626a01, duration=9913.613s, table=20, n_packets=0, n_bytes=0, hard_timeout=300, idle_age=9913, hard_age=4, priority=1,vlan_tci=0x0001/0x0fff,dl_dst=fa:16:3e:4a:10:2b actions=load:0->NXM_OF_VLAN_TCI[],load:0x2->NXM_NX_TUN_ID[],output:2
15- cookie=0xbb7b3cdd52626a01, duration=13003.029s, table=20, n_packets=0, n_bytes=0, idle_age=13003, priority=0 actions=resubmit(,22)
16- cookie=0xbb7b3cdd52626a01, duration=9917.985s, table=21, n_packets=1, n_bytes=42, idle_age=9913, priority=1,arp,dl_vlan=1,arp_tpa=192.168.111.1 actions=move:NXM_OF_ETH_SRC[]->NXM_OF_ETH_DST[],mod_dl_src:fa:16:3e:0b:cf:10,load:0x2->NXM_OF_ARP_OP[],move:NXM_NX_ARP_SHA[]->NXM_NX_ARP_THA[],move:NXM_OF_ARP_SPA[]->NXM_OF_ARP_TPA[],load:0xfa163e0bcf10->NXM_NX_ARP_SHA[],load:0xc0a86f01->NXM_OF_ARP_SPA[],IN_PORT
17- cookie=0xbb7b3cdd52626a01, duration=9917.984s, table=21, n_packets=0, n_bytes=0, idle_age=9917, priority=1,arp,dl_vlan=1,arp_tpa=192.168.111.2 actions=move:NXM_OF_ETH_SRC[]->NXM_OF_ETH_DST[],mod_dl_src:fa:16:3e:4a:10:2b,load:0x2->NXM_OF_ARP_OP[],move:NXM_NX_ARP_SHA[]->NXM_NX_ARP_THA[],move:NXM_OF_ARP_SPA[]->NXM_OF_ARP_TPA[],load:0xfa163e4a102b->NXM_NX_ARP_SHA[],load:0xc0a86f02->NXM_OF_ARP_SPA[],IN_PORT
18- cookie=0xbb7b3cdd52626a01, duration=13003.028s, table=21, n_packets=0, n_bytes=0, idle_age=13003, priority=0 actions=resubmit(,22)
19- cookie=0xbb7b3cdd52626a01, duration=9917.956s, table=22, n_packets=10, n_bytes=1336, idle_age=9904, dl_vlan=1 actions=strip_vlan,set_tunnel:0x2,output:2
20- cookie=0xbb7b3cdd52626a01, duration=13003.002s, table=22, n_packets=4, n_bytes=340, idle_age=9920, priority=0 actions=drop

Explanation of the Rules:

Table 0:

1- cookie=0xbb7b3cdd52626a01, duration=13003.029s, table=0, n_packets=183, n_bytes=28498, idle_age=4, priority=1,in_port=1 actions=resubmit(,2)
2- cookie=0xbb7b3cdd52626a01, duration=9917.985s, table=0, n_packets=198, n_bytes=36045, idle_age=4, priority=1,in_port=2 actions=resubmit(,4)
3- cookie=0xbb7b3cdd52626a01, duration=13003.030s, table=0, n_packets=0, n_bytes=0, idle_age=13003, priority=0 actions=drop

Rule 1 Has priority=1 and checks if the packets coming on port in_port=”patch-int” then the action is: go to table 2 
Rule 2  Checks if the packets coming on port in_port=vxlan-c0a80202 then the action is: go to table 4
Rule 3 Has priority=0 (lowest priority) and drop the packets that don’t match rule 1 and rule 2

Table 2:

4- cookie=0xbb7b3cdd52626a01, duration=13003.029s, table=2, n_packets=1, n_bytes=42, idle_age=9913, priority=1,arp,dl_dst=ff:ff:ff:ff:ff:ff actions=resubmit(,21)
5- cookie=0xbb7b3cdd52626a01, duration=13003.029s, table=2, n_packets=168, n_bytes=26780, idle_age=4, priority=0,dl_dst=00:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,20)
6- cookie=0xbb7b3cdd52626a01, duration=13003.029s, table=2, n_packets=14, n_bytes=1676, idle_age=9904, priority=0,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,22)

Rule 4 

Has priority=1 and checks if the packets are ARP packet with destination MAC address set to broadcast

then the action is: go to table 21

Rule 5

Has priority=0 and checks if the packets has dl_dst=00:00:00:00:00:00/01:00:00:00:00:00 

(match all unicast Ethernet packets) then the action is: go to table 20 

Rule 6 

Has priority=0 and checks if the packets has dl_dst=01:00:00:00:00:00/01:00:00:00:00:00

(match all multicast(including broadcast Ethernet packets) then the action is: go to table 22 

Table 3:


7- cookie=0xbb7b3cdd52626a01, duration=13003.029s, table=3, n_packets=0, n_bytes=0, idle_age=13003, priority=0 actions=drop

Rule 7 drop the packets

Table 4:


8- cookie=0xbb7b3cdd52626a01, duration=9921.166s, table=4, n_packets=198, n_bytes=36045, idle_age=4, priority=1,tun_id=0x2 actions=mod_vlan_vid:1,resubmit(,10)
9- cookie=0xbb7b3cdd52626a01, duration=13003.029s, table=4, n_packets=0, n_bytes=0, idle_age=13003, priority=0 actions=drop

Rule 8 Has priority=1 and checks if the packets tun_id=0x20 hen the action is to add the vlan_vid:1 and go to table 10 
Rule 9 Has priority=0 (lower priority) and drop the packets that don’t match rule 8

Table 6:


10- cookie=0xbb7b3cdd52626a01, duration=13003.029s, table=6, n_packets=0, n_bytes=0, idle_age=13003, priority=0 actions=drop

Rule 10 drop the packets

Table 10:


11- cookie=0xbb7b3cdd52626a01, duration=13003.029s, table=10, n_packets=198, n_bytes=36045, idle_age=4, priority=1 actions=learn(table=20,hard_timeout=300,priority=1,cookie=0xbb7b3cdd52626a01,NXM_OF_VLAN_TCI[0..11],NXM_OF_ETH_DST[]=NXM_OF_ETH_SRC[],load:0->NXM_OF_VLAN_TCI[],load:NXM_NX_TUN_ID[]->NXM_NX_TUN_ID[],output:NXM_OF_IN_PORT[]),output:1

Rule 11

Has priority=1 and the action has two parts: 

Part one:

Is to install a rule in table 20. This table (20) will be a MAC learning table. 

The “learn” action modifies a flow table based on the content of the flow currently being processed by table 4.

Here’s how you can interpret each part of the “learn” action above:

table=20     Modify flow table 20.  This will be the MAC learning table.

    

      hard_timeout=300

       Causes the flow to expire after the 300 seconds, regardless of activity.

    

      priority=1

       The priority at which a wildcarded entry will match in comparison to others

    

      cookie=0x407518fa3ccd67d2 NXM_OF_VLAN_TCI[0..11]     Make the flow that we add to flow table 20 match the same VLAN    ID that the packet we’re currently processing contains.  This    effectively scopes the MAC learning entry to a single VLAN,    which is the ordinary behavior for a VLAN-aware switch. NXM_OF_ETH_DST[]=NXM_OF_ETH_SRC[]     Make the flow that we add to flow table 20 match, as Ethernet    destination, the Ethernet source address of the packet we’re    currently processing.

    

      load:0->NXM_OF_VLAN_TCI[],

    

          Strip off the VLAN ID by loading 0 as a VLAN ID

    

      load:NXM_NX_TUN_ID[]->NXM_NX_TUN_ID[],

    

          Load the tunnel ID of the proceesing packet as a tunnel id of the packet

    

      output:OXM_OF_IN_PORT[]),

    

          Send the packet out via input port

    

      Part Two:

      output:”patch-int”

      sends the packet out via port patch-int

Table 20:

12- cookie=0xbb7b3cdd52626a01, duration=9917.984s, table=20, n_packets=102, n_bytes=14108, idle_age=9435, priority=2,dl_vlan=1,dl_dst=fa:16:3e:0b:cf:10 actions=strip_vlan,set_tunnel:0x2,output:2
13- cookie=0xbb7b3cdd52626a01, duration=9917.984s, table=20, n_packets=66, n_bytes=12672, idle_age=4, priority=2,dl_vlan=1,dl_dst=fa:16:3e:4a:10:2b actions=strip_vlan,set_tunnel:0x2,output:2
14- cookie=0xbb7b3cdd52626a01, duration=9913.613s, table=20, n_packets=0, n_bytes=0, hard_timeout=300, idle_age=9913, hard_age=4, priority=1,vlan_tci=0x0001/0x0fff,dl_dst=fa:16:3e:4a:10:2b actions=load:0->NXM_OF_VLAN_TCI[],load:0x2->NXM_NX_TUN_ID[],output:2
15- cookie=0xbb7b3cdd52626a01, duration=13003.029s, table=20, n_packets=0, n_bytes=0, idle_age=13003, priority=0 actions=resubmit(,22)

Rule 12,13

Have priority=2 and check if the packets has VLAN id = 1 and  a certain dl_dst addresses

then the action is: strip the VLAN id and load the tunnel id of 0x2 and send the packets out via output:vxlan-c0a80202

Rule 14 

These rule are installed via the learn action of table 10:

Has priority=1 and checks if the packets has vlan_tci=0x0001/0x0fff (VLAN id = 1) and ,dl_dst=fa:16:3e:4a:10:2b

then the action is: strip the VLAN id and load the tunnel id of 0x2 and send the packets out via output:vxlan-c0a80202

Rule 15 Has priority=0 (lower priority) and the action is: go to table 22

Table 21:

16- cookie=0xbb7b3cdd52626a01, duration=9917.985s, table=21, n_packets=1, n_bytes=42, idle_age=9913, priority=1,arp,dl_vlan=1,arp_tpa=192.168.111.1 actions=move:NXM_OF_ETH_SRC[]->NXM_OF_ETH_DST[],mod_dl_src:fa:16:3e:0b:cf:10,load:0x2->NXM_OF_ARP_OP[],move:NXM_NX_ARP_SHA[]->NXM_NX_ARP_THA[],move:NXM_OF_ARP_SPA[]->NXM_OF_ARP_TPA[],load:0xfa163e0bcf10->NXM_NX_ARP_SHA[],load:0xc0a86f01->NXM_OF_ARP_SPA[],IN_PORT
17- cookie=0xbb7b3cdd52626a01, duration=9917.984s, table=21, n_packets=0, n_bytes=0, idle_age=9917, priority=1,arp,dl_vlan=1,arp_tpa=192.168.111.2 actions=move:NXM_OF_ETH_SRC[]->NXM_OF_ETH_DST[],mod_dl_src:fa:16:3e:4a:10:2b,load:0x2->NXM_OF_ARP_OP[],move:NXM_NX_ARP_SHA[]->NXM_NX_ARP_THA[],move:NXM_OF_ARP_SPA[]->NXM_OF_ARP_TPA[],load:0xfa163e4a102b->NXM_NX_ARP_SHA[],load:0xc0a86f02->NXM_OF_ARP_SPA[],IN_PORT
18- cookie=0xbb7b3cdd52626a01, duration=13003.028s, table=21, n_packets=0, n_bytes=0, idle_age=13003, priority=0 actions=resubmit(,22)

Rule 16, 17

Has priority=1 and checks if the packets are ARP packet and  have certain VLAN ID (e.g. dl_vlan=1) and 

a certain destination IP address (e.g. arp_tpa=192.168.111.1)

then the action of the flow is:

  • move:NXM_OF_ETH_SRC[]->NXM_OF_ETH_DST[] → move the Ethernet destination of the processing packet as an Ethernet source address of the flow
  • mod_dl_src:fa:16:3e:0b:cf:10 → change the Ethernet source address to a certain value
  • load:0x2→NXM_OF_ARP_OP[] → Load the tunnel ID 0x2 
  • move:NXM_NX_ARP_SHA[]→NXM_NX_ARP_THA[] → move the ARP source MAC address of the processing packet as an ARP target MAC address of the flow
  • move:NXM_OF_ARP_SPA[]→NXM_OF_ARP_TPA[] → move the ARP source IP address of the processing packet as an ARP target IP address of the flow
  • load:0xfa163e0bcf10→NXM_NX_ARP_SHA[] → load 0xfa163e0bcf10 as an ARP source MAC address 
  • load:0xc0a86f01→NXM_OF_ARP_SPA[] →  load 0xc0a86f01 as an ARP IP address 
  • IN_PORT → send the packet out via input port 

Note: the above flow indicate that the switch which is close to the host replies to arp MAC address  

Rule 18 Has priority=0 (lower priority) and the action is: go to table 22

Table 22:

19- cookie=0xbb7b3cdd52626a01, duration=9917.956s, table=22, n_packets=10, n_bytes=1336, idle_age=9904, dl_vlan=1 actions=strip_vlan,set_tunnel:0x2,output:2
20- cookie=0xbb7b3cdd52626a01, duration=13003.002s, table=22, n_packets=4, n_bytes=340, idle_age=9920, priority=0 actions=drop

Rule 19 Checks if the packet has VLAN ID=1 then the action is: strip the VLAN id and load the tunnel id of 0x2 and send the packets out via output:vxlan-c0a80202
Rule 20 Has priority=0 (lower priority) and drop the packets that don’t match rule 19

Having a good understanding of these rules will help us troubleshooting network traffic. If there are any connectivity issues in the network (internal/external) which result in the packet loss, we can easily follow the trail of packets within the engaged flow rules to find the leakage in the network.

For example, if we run ping between two OpenStack endpoints, first we need to understand which flow rules are being hit by the ping packets and then observe if there are any incremental changes in the “n_packets” count of the rule. The “n_packet” feature inform us if the packets are begin forwarded to another endpoint or being dropped in the network.

How can we make OpenStack work for you?
Find out what else we can do with OpenStack.

Find Out Here

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.