Introduction to VXLAN in SD-Access

What is VXLAN?

Virtual Extensible Local Area Network or VXLAN is the underlying tunnelling protocol used in SD-Access. In SDA it is the Data Plane mechanism. Data plane is the actual forwarding of the payload at Layer 3. VXLAN is in charge of this. In order to understand VXLAN we must understand what a traditional network is like without VXLAN and what it is like with VXLAN.

Networks without VXLAN

In a traditional LAN we typically have the same setup of access, layer and core switches which is similar in SDA. However, the difference is we typically run the below protocols in a NON-SDA network:

• Spanning-tree
• VLANs
• Trunk links between Access and Distribution switches, or if collapsed core, between Access and Core.

Our end hosts are connected to the end switches and typically either segmented into the same or different VLAN. They can be segmented on the same or a different switch.

If we look at the diagram again. Carol initiates a ping to David in the same VLAN. The packet information is below

We are currently unaware of what the destination MAC address is for David. In order to find the MAC we use ARP (Address Resolution Protocol). In a traditional network in order to forward the Packet we need help of ARP. ARP here acts as the "Control Plane" it will map the missing value that we need.

How ARP works

Carol's NIC will send an ARP broadcast message to resolve David's MAC address. The structure of an ARP Request packet simplified is below:

We know what the Target IP Address is. In the real world, the user will likely browse to a website using the URL, in the backend this is an FQDN tied to an IP which we have to resolve the next hop - Usually the default gateway. For example, when you enter google.com it will resolve to a public IP and the ARP process will initiate on the switch to find where the packet needs to be forwarded.

ARP is basically just asking what is the MAC address to reach this target IP address.

Anyhow, when Carol's ARP request broadcast reaches SW1, it will create an entry in its CAM (Content Addressable Memory) AKA MAC table. It will include the VLAN, MAC and interface configuration. A switch creates an entry in its MAC table based on the source MAC Address.

When SW1 gets this all F's MAC, it will flood the broadcast message. You must be aware of spanning tree if redundant links. Broadcast messages are flooded out of all ports except the one it was received on, and blocked ports by spanning-tree. This only applies in a L2 Access design, if this is a Layer 3 routed design then Spanning-tree is not a concern.

If Layer 2 it will forward it over the available trunk link and attach a Dot1Q Tag for this broadcast message so that receiving switches know its for VLAN 10. This broadcast request will reach SW2 where David is, and SW2 will put in its MAC table the source MAC address of the packet...which is Carol's CCCC.CCCC.CCCC MAC address and the port/vlan information.

The ARP broadcast finally gets to David who will de-encapsulate this packet and see that the destination is for him and then will send a unicast ARP reply back to Carol. As mentioned above, since SW2 now knows the MAC of Carol, there is no need to do another ARP broadcast. The reply will be unicast. However, when SW2 receives the ARP reply it will populate its MAC table with David's MAC/VLAN/IP information.

SW1 will receive this unicast ARP packet and now know which interface David is reachable on, as well as David's MAC/VLAN/IP information and then will forward the unicast ARP reply to Carol.

Carol can now complete the initial ping packet and send it to David and they will both be able to communicate.

Networks with VXLAN

The major change with VXLAN in an SDA Network is that between access and distribution/core design, we are running full Layer 3 links in our network. From Access to Distribution and Distribution to Core we would run Layer 3 routed links. Loopbacks also are created on each of the switches in the SDA Fabric. These are always a /32 loopback address.

What are the consequences of it becoming Layer 3? There is no more trunk links. There is no more 802.1Q tagging and no more ARP requests.

We have the choice of running any IGP within SDA - ISIS & OSPF are most common but we can even run iBGP or RIP. The only thing that matters is we use this routing protocol to achieve loopback to loopback communication between all nodes in the SDA fabric. They will all be advertised into the IS-IS, BGP, OSPF or RIP domain. So if SW1 for example has a loopback of 10.10.10.1 and SW2 has a loopback of 10.10.10.2. They should both be able to reach each other via the IGP.

The issue encountered in Layer 3

If Carol is on 192.168.1.1 and David is on 192.168.1.2, both on different switches, and Carol sends an ICMP ping to David on 192.168.1.2. We will encounter a familiar issue. Carol will need to resolve what the MAC address of David's device is.

I will not explain the process again, however Carol will attempt to send another ARP Request broadcast to SW1. SW1 will update its MAC table as normal. However it cannot broadcast the ARP request because ARP is Layer 2. It cannot be broadcasted over a Layer 3 link. For example, you can remember that routers are usually the boundary/blocker between broadcast domains. Its the same with Layer 3 routed links.

How VXLAN solves the problem

VXLAN solves this problem by encapsulating the ARP packet into a VXLAN packet. It will look like the below

In the VXLAN header it includes a VNI. A VNI is "Virtual Network Identifier" and it includes the VLAN information.

To facilitate VXLAN transmitting over the L3 links, the packet is further encapsulated into a UDP and IP header.

We now have an OSA and an ODA. An OSA is the "Outer Source Address" and the ODA it the "Outer Destination Address". The OSA will be the loopback present on that switch receiving this VXLAN packet. In this example it will be loopback 10.10.10.1. This IP is also used to identify the switch.

SW1 does not know what the IP address of the ODA is. The way it finds this information is through LISP. LISP stands for "Location Identity Separation Protocol" is a Control Plane Protocol. The purpose of LISP is to tell SW1 what the IP of the Outer Destination Address is. We typically will have a dedicated node called the Control Plane node which will run LISP and SW1 will send a request to the control plane.

Technically that LISP does not actually provide David's IP/MAC. But instead provides the IP of the switch that David is connected to. In this example, David is connected to SW2. So the IP address LISP will send back to SW1 for the ODA is 10.10.10.2. If you recall, 10.10.10.2 is the loopback address of SW2.

Therefore, to reach David we need to send the packet to 10.10.10.2. The VXLAN encapsulated packet will eventually reach SW2. SW2 will de-encapsulate the packet and see the ARP request and put Carol's MAC, IP, and VLAN in its MAC table. It will also see the ARP request was for David's IP address.

Key to note, in the SW2 MAC table it will also have knowledge that Carol is reachable through loopback 10.10.10.1. The benefit of this? SW2 does not have to send a request to the control plane node to know how to send the request back to Carol.

When David Replies with an ARP reply message, SW2 will log this in its MAC table as normal, it will encapsulate the packet with a VXLAN packet and SW1 will de-encapsulate the packet and forward it through to Carol.

Now Carol has all the information required and can send the ICMP pings.