Reading Time: 20 minutes

General

I think most of the people in the Citrix community have been reading a lot about the NetScaler SD-WAN the last months. I want to show you, how to configure the appliances to give you a straight start with this technology. First of all: Why should a company consider investing money in Software Defined WAN?

• Improve application performance in the datacenter and branch offices
• Reduce your operating costs dramatically. You don’t need to spend money for the expensive MPLS connection to South America/Asia when you can bundle n-WAN connections from different Internet Service Providers.
• Maintain network connectivity (without interrupt) to the Datacenter/Branch even multiple network connections go down.
• Reduce network hardware (stateful firewall function)
• Centralized and easy management
• Detailed reports and network analysis

If you check the SDWAN datasheet, you will notice there are three kind of appliances.

• Standard (SE): Bundle different Internet Service Provider links to a Virtual WAN
• WANOP (WO): Optimize the WAN Connection. Many people will know this as Branch Repeater or CloudBridge.
• Enterprise (EE): You get the ability to use Standard and WANOP on a single appliance.

My opinion to WANOP: I made bad experience with Citrix CloudBridge some time ago. I think this should only be used when there really is no other ISP at the branch available and you need to squeeze the last drop out of the connection. Regarding HDX optimization –> With adaptive transport (EDT) the traffic will be UDP based and there will be nothing to optimize. Keep this in mind.

NetScaler SD-WAN is available as a physical appliance or virtualized on the hypervisor of your choice.

• Hyper-V (2012 R2)
• XenServer 6.5 SP1
• VMware ESX 5.5 & ESX 6.0
• KVM (Ubuntu 16.04)

The Enterprise edition is only available as a physical appliance.

Like in the NetScaler ADC family the purchased license will specify how much bandwidth you can use and how many virtual paths you will be able to create. What is a virtual path? Let’s say your datacenter/branch is getting WAN access from two different ISPs. If you now create a Virtual WAN to the branch office, this will result in a total of 4 virtual paths.

1. ISP1_Datacenter –> ISP1_BranchOffice
2. ISP1_BranchOffice –> ISP1_Datacenter
3. ISP2_Datacenter –> ISP2_BranchOffice
4. ISP2_BranchOffice –> ISP2_Datacenter

When running a VPX the license upgrade will also give you the ability stock up the available virtual paths. A physical appliance can use the maximum of the virtual paths starting from the smallest license. Depending on your company size you should size the appliance properly.

The quality of each path will be measured by the following information:

• BOWT (Latency – Best one-way time)
• Jitter (mS)
• Packet-loss (%)
• kbps (Bandwidth)
• Congestion (will detect if there is a problem on the ISP backbone. Example: unexcepted packet flow)

SD-WAN will provide out-of-the-box configured QoS settings for the most common protocols used in an enterprise network:

Prioritization 1 – Real Time – Minimum Bandwidth 30% – VOIP (Skype for Business, ICA Audio)

Prioritization 2 – Interactive – Minimum Bandwidth 40 % –  XenDesktop, Exchange

Prioritization 3 – Bulk – Minimum Bandwidth 30 % –  Moving Data (FTP, Backup Replication)

If you are not comfortable with the default settings, you can create a custom “Classes” for prioritizing specific protocols.

Deployment Methods

You have a variety of possibilities to connect the SD-WAN to your corporate network.  Before starting to show you the different deployment modes it’s important to know that you can configure the ethernet interfaces as “Fail-To-Wire” or “Fail-To-Block”.

Fail-To-Wire: Only available on physical appliances. If the appliance goes offline, the specified ports will bridge the traffic from the SD-WAN to the LAN side. This will be used for a MPLS connection to prevent a complete network outage. The communication to the branch office is still guaranteed.

Fail-To-Block: When you have connected an unsecure Internet connection (without Firewall) the port will block the traffic. This is the default setting on a VPX.

For High Availability you have the possibility to create an active-passive setup.

Datacenter Gateway Deployment

The SD-WAN appliance will be put in the physical path between LAN and WAN (Two-Arm Deployment). SD-WAN will be the default gateway for the entire datacenter LAN network. If the appliance will go down, you will not have any internet or MPLS connection because SD-WAN is acting as Layer 3 device only (no Fail-To-Wire functionality).

Branch Inline Deployment

The SD-WAN appliance will be put in the physical path between LAN and WAN (Two-Arm Deployment). The branch LAN will be connected directly to the appliance.

Virtual Inline Mode (PBR)

The SD-WAN appliance will be connected with a single interface to the datacenter router (One-Arm Deployment). Policy Based Routing (PBR) needs to be configured on the core switch/router.

Edge Mode

In Edge Mode the SD-WAN is connected directly to the WAN links. You can seamlessly replace the legacy routers.

Building a NetScaler SD-WAN Lab

Now let’s come to the interesting part. How can I create a NetScaler SD-WAN Lab? You should bring some network knowledge with you. If you are new to this stuff I will try to explain as detailed as possible. My lab environment is hosted on an “Intel NUC- i7-6770HQ” with XenServer Release 7.2

Network Diagram

Overview Lab Infrastructure

VM	vCPU	RAM	HDD	Description
RTR_INET_DC	1	256 MB	5 GB	Internet Router (DC)
RTR_INET_BR	1	256 MB	5 GB	Internet Router (Branch)
WANEM-INET	1	256 MB	5 GB	WAN Emulator Internet
RTR_MPLS_DC	1	256 MB	5 GB	MPLS Router (DC)
RTR_MPLS_BR	1	256 MB	5 GB	MPLS Router (Branch)
WANEM-MPLS	1	256 MB	5 GB	WAN Emulator MPLS
SDWAN_DC	4	4 GB	40 GB	SDWAN (DC)
SDWAN_BR	4	4 GB	40 GB	SDWAN (Branch)
DC_Client	2	2 GB	25 GB	Windows Client (DC)
BR_Client	2	2 GB	25 GB	Windows Client (Branch)

Preparations

• Download VyOS (Virtual Router)
• Download WANEM v2.3
• Download SD-WAN VPX for your Hypervisor
• Download Windows 10 or Server 2016 ISO for the datacenter/branch client.
• Get a Trial License for SD-WAN

Create the virtual networks

Network	Description	VLAN ID (optional)
LAN-DC	192.168.2.0/24	2
LAN-BR	192.168.3.0/24	3
WAN-INET-DC	192.168.50.0/24	50
WAN-INET-BR	192.168.50.0/24	55
INET-DC	192.168.51.0/24	51
INET-BR	192.168.61.0/24	61
WAN-MPLS-DC	192.168.60.0/24	60
WAN-MPLS-BR	192.168.60.0/24	66
MPLS-DC	192.168.52.0/24	52
MPLS-BR	192.168.62.0/24	62

Before we can start to configure the VyOS routers we need to create the virtual networks. If you build the lab on a single machine you don’t need to create external networks with a VLAN ID. A single server private network is enough. When using a single XenServer for the lab, it’s very important to disable TCP checksum offload on the virtual networks otherwise no request except ICMP will reach your clients/servers in a different network. Thanks to Benjamin Ruoff from Citrix who discovered this issue. Here is the bash script from Benjamin to disable the checksum offload on all virtual network interfaces. This was consuming most of my time during the lab build.

1. Click on “Networking” in XenCenter and select “Add Network…”
2. Choose “Single-Server Private Network”, when running on a single lab machine. Otherwise you need to take “External Network” and specify the VLAN ID.
3. Provide the name and a description for the new network.
4. Uncheck the box “Automatically add this network to new virtual machines”.
5. Create the missing networks. After this we can start to install the VyOS routers.

Install VyOS Routers

VM	Network 1	Network 2	Description
RTR_INET_DC	WAN-INET-DC 192.168.50.1	INET-DC 192.168.51.1	Internet Router (DC)
RTR_INET_BR	WAN-INET-BR 192.168.50.3	INET-BR 192.168.61.1	Internet Router (Branch)
RTR_MPLS_DC	WAN-MPLS-DC 192.168.52.1	MPLS-DC 192.168.60.1	MPLS Router (DC)
RTR_MPLS_BR	WAN-MPLS-BR 192.168.60.3	MPLS-BR 192.168.62.1	MPLS Router (Branch)

1. Right click on the XenServer and select “New VM…”. Now we need to create the first router called “RTR_INET_DC”.
2. Put in the name for the first router “RTR_INET_DC”.
3. Choose the downloaded VyOS image as install media.
4. Assign 1vCPU and 256MB
5. Configure the used virtual disk. 5GB is enough.
6. Attach the needed network interfaces (refer to the table).
7. Click on “Create Now” and start the new VM automatically.
8. Before we can start with the configuration we need to install the VyOS image to the harddisk and install the XenServer Guest Tools. The default login is vyos;vyos.
9. Use the command “install Image”, confirm the user prompts and set a new password for the vyos account. After the successful installation restart the VM with the command “reboot”.
10. Mount the “guest-tools.iso” to the virtual dvd drive and login with the vyos user.
11. Run the following commands to install the XenServer Tools.
    sudo su
    mount /dev/cdrom /mnt
    /mnt/Linux/install.sh
    reboot
12. After the successful reboot login again and execute the command “show interfaces” you should now be able to see the virtual network adapters inside VyOS.
13. Repeat these steps to create the missing routers
    • RTR_INET_BR
    • RTR_MPLS_DC
    • RTR_MPLS_BR

Configure VyOS Routers

Execute the commands on the routers.

###  RTR_INET_DC ###

### RTR_INET_BR ###

### RTR_MPLS_DC ###

## RTR_MPLS_BR ##

 

Install & Configure the WAN Emulators

VM	Network 1	Network 2	IP Address
WANEM-INET	WAN-INET-DC	WAN-INET-BR	192.168.50.2 (bride)
WANEM-MPLS	WAN-MPLS-DC	WAN-MPLS-BR	192.168.60.2 (bridge)

1. Create the VM “WANEM-INET”, attach the needed networks and mount the WANEMv2.3.iso.
2. Press “Enter” to boot from the CD.
3. Do you want to configure all interfaces via DHCP: y
4. Enter new UNIX password: ********
5. WANemControl@Perc> exit2shell
6. root!tty1:/# knx2hd
7. Initialization of Knoppix-Installation: OK
8. Initialization of Knoppix-Installation: OK
9. Knoppix Partition-Menu: “ Partition”
10. Partitioning harddisk for Knoppix: “had” and NEXT
11. Partitioning harddisk for Knoppix: “ Template 1”
12. Partitioning Harddisk: Yes
13. Partitioning Harddisk: Yes
14. Partitioning harddisk for Knoppix: OK
15. Knoppix Partition-Menu: “Quit”
16. Save configuration?: Yes
17. root!tty1:/# knx2hd
18. Initialization of Knoppix-Installation: OK
19. Knoppix Main-Menu: “1. Configure Installation”
20. Choose your system type: “debian”
21. Creating Knoppix Configuration (Step1/7): “/dev/hda2”
22. Choose filesystem-type: ext3
23. Input your whole name (name surname): “dummy”
24. Input your user name: “ddummy”
25. Input user password: ***** (will not be used)
26. Input your administration password: *****
27. Input your preferred hostname: “WANEM-INET”
28. Choose where the bootloader (grub) shall be installed: mbr
29. Knoppix Main-Menu: “2. Start Installation”
30. Starting Knoppix Installation: Next
31. Creating floppy disk: No
32. root!tty1:/# reboot and remove the mounted image
33. The WAN Emulator will boot from harddisk and you can login with the admin password you choose in Step 26. Now we need to configure the ethernet interfaces to bridge mode. Execute “vi /etc/network/interfaces”
34. Remove/Uncomment the DHCP interfaces (eth0, eth1) settings and add this to the configuration file.
    auto br0
    iface br0 inet static
         address 192.168.50.2
         netmask 255.255.255.0
         gateway 192.168.50.1
         bridge_ports all
         bridge_fd 0
         bridge_stp off
35. Save the changes with “:wq!”
36. Reboot the VM.
37. Now create the WAN MPLS Emulator. The only difference is the IP configuration of the bridged network interface.
    auto br0
    iface br0 inet static
         address 192.168.60.2
         netmask 255.255.255.0
         gateway 192.168.60.1
         bridge_ports all
         bridge_fd 0
         bridge_stp off

Network Test

At this point you should have installed and configured the VyOS routers and the WAN emulators. Time for a short test if the components can communicate with each other. Run a “ping” command from the VyOS console. If there is a problem with the network, please check your VyOS configuration.

Source: RTR_INET_DC
Destination: 192.168.50.2 (WANEM INET)

Source: RTR_INET_DC
Destination: 192.168.50.3 (RTR_INET_BR – ISP WAN Side)

Source: RTR_INET_DC
Destination: 192.168.61.1 (RTR_INET_BR – INET Branch Side)

Source: RTR_MPLS_DC
Destination: 192.168.60.2 (WANEM MPLS)

Source: RTR_MPLS_DC
Destination: 192.168.60.3 (RTR_MPLS_BR –  MPLS WAN Side)

Source: RTR_MPLS_DC
Destination: 192.168.62.1 (RTR_MPLS_BR – MPLS Branch Side)

Create the Datacenter and Branch Client

Before we can start with the import and configuration of the SD-WAN appliance we need to create the Datacenter and Branch client that, we can access the management ip address of the VPX.

VM	vCPU	RAM	HDD	IP Address	Network	Description
DC_Client	2	2 GB	25 GB	192.168.2.100 255.255.255.0 192.168.2.254	LAN-DC	Windows Client (DC)
BR_Client	2	2 GB	25 GB	192.168.3.100 255.255.255.0 192.168.3.254	LAN-BR	Windows Client (Branch)

1. Right click on the XenServer and select “New VM…”
2. Select “Legacy Windows” as Template and click Next.
3. Enter the Name of the datacenter client “DC_Client”.
4. Select your Windows 10/Server 2016 install media.
5. Place the VM on the server which is simulating the datacenter.
6. Assign 2vCPU and 2048 MB
7. Configure the virtual disk.
8. Attach the needed network interface “LAN-DC”.
9. Click on “Create Now” and start the new VM automatically.
10. Complete the Windows installation and install the XenServer Guest Tools.
11. Configure the TCP/IPv4 Properties.
12. Repeat the steps and create the branch client “BR_Client”.

Import the NetScaler SD-WAN VPX (Datacenter)

VM	eth0	eth1	eth2	eth3
SDWAN_DC	LAN-DC	INET-DC	MPLS-DC	LAN-DC

1. In XenCenter click on File -> Import
2. Browse to the “ns-sdw-os-xxxx.xva”file and click Next.
3. Set a location for the VM.
4. Select the storage where the VM should be imported.
5. Assign the Datacenter LAN (LAN-DC) to interface 0. This is always the management port. Connect the missing interfaces.
6. Uncheck “Start VM(s) after import” and click “Finish”.
7. Right click on the imported VPX and select “Properties”
8. Change the Name to “SDWAN_DC”.

Configure Management IP Address

Note: If there is a DHCP server available on the management interface, the SD-WAN appliance will request an IP address.

1. Start the VM “SDWAN_DC”
2. After the run once commands are complete, login to the console as “admin” and “password”.
3. Enter “management_ip” and “set interface 192.168.2.253 255.255.255.0 192.168.2.1”
4. Enter “apply” and “y” to save the new configured management ip address.

Licensing

1. Download your License.
2. Select “Citrix Store Cloud Bridge VPX Virtualization” and Continue.
3. Enter the MAC address from the VM “SDWAN_DC” interface 0 (MGMT) and download the license file.
4. Open the console of “DC_Client” and browse to the URL: https://192.168.2.253
5. Login with default user “admin” and “password”.
6. Click on “Configuration”
7. Click on “Appliance Settings” and Licensing and upload your trial license file. Its also possible to import the .lic file to your Citrix Licensing Server and activate the VPX via this method.

General Settings (NTP, TimeOut)

1. Click on Appliance Settings, Administrator Interface and choose “Miscellaneous”. Change the Time Out of the Management Web Console to 500 minutes. This will save you some login procedures while building the lab. Not recommended for production environment.
2. Under “System Maintenance” and “Date/Time Settings” specify your NTP server and change to the correct time zone.

Create the Master Control Node (MCN)

The Master Control Node is the heart of the Virtual WAN. The MCN will create the config for the Datacenter and for the branch offices. Any changes you will make will happen on this machine.

1. Go to “Appliance Settings”, “Administrator Interface” and switch to “Miscellaneous”. Hit the Button to “Switch Console” to make this appliance to the primary MCN.
2. Your Session will be terminated and you need to login again with admin;password.

Create the Site “Datacenter”

1. Go to “Configuration” and expand the “Virtual WAN” menu.
2. Click on “Configuration Editor” and create a “New” configuration.
3. Select the arrow icon to expand the window size.
4. Change the mode to “Advanced”.
5. Click on the “+ Add” button under “Sites”.
6. Specify the Site Name to “DC” and change the Model to “CBVPX”. Click “Add”.
7. Click on the “+” button.
8. Mark the Ethernet Interface “1”, set the Bypass Mode to “Fail-To-Block and change the Security to “Untrusted”. Click on the “+” button on the left side. The Interface is untrusted because on Interface 1 will be connected the Public Internet connection without a firewall.
9. Click again on the “+” button.
10. Specify the name “DC-INET” for the virtual interface.
11. Repeat the procedure to create the virtual interface for the MPLS connection. (Ethernet 2, Trusted Security).
12. Repeat the procedure to create the virtual interface for the LAN connection. (Ethernet 3, Trusted Security).
13. Now its time to specify the virtual ip addresses.
14. Enter the ip addresses/subnet mask and the matching virtual interface for the VPX.
15. The next step is to create the WAN links.
16. Click on the pencil to edit the WAN link configuration.
17. Specify the available bandwidth for the Public Internet connection (16000kbps/4000kbps). There is a text field with the description “Public IP Address”. We don’t need this setting in the lab environment. In a real-world environment, the MCN needs to be configured with at least one public ip address. Otherwise the branch appliance will not be able to connect to the MCN.
18. Create the missing access interface “DC-INET-AI-1” (192.168.51.2, 192.168.51.1)
19. Repeat the steps for the MPLS WAN Link. a. Access Type: Private Intranet, b.Bandwidth: 6000kbps/6000kbps, c. IP Address: 192.168.52.2, Gateway: 192.168.52.1
20. Save the configuration.

Create the Site – Branch

1. Create the Site “Branch” with the Configuration Editor.
2. Specify the Name “Branch”, Model: “CBVPX” and the mode to “client”.
3. Now we need to create the interface groups for the branch site. When finished it should look like this.
4. Specify the virtual ip addresses for the branch.
5. Create the WAN Links for the Branch.
   a.BR-INET – 16000kbps/4000kbps – Public Internet – IP: 192.168.61.2, Gateway 192.168.61.1
   b.BR-MPLS – 6000kbps/6000kbps – Private Internet – IP: 192.168.62.2, Gateway 192.168.62.1
6. Save the configuration.

MPLS Path Association

You always need to specify that a MPLS Link will be used for a connection. For Public Internet connections this will be done automatically.

1. Go to “Connections”, select “WAN Links” and edit the “DC-MPLS” link.
2. Mark the path with “Use” and apply the change.
3. Do the same with the MPLS link in the branch site.
4. Head to “Connections” –> “Branch” –> “Virtual Paths” –> “DC-Branch” –> “Paths” and create a new path.
5. From Site: Branch, From WAN LINK: BR-MPLS, To Site: DC, To WAN Link: DC-MPLS Make sure you check the “Reverse Also” option. (This will create the path in thedatacenter)                                                           
6. Now your configuration editor shouldn’t throw any errors and you have completed the configuration of the two sites. Congratulations. Was quiet easy hum? Its no rocket science 😊 Please save the config one more time!

Import the NetScaler SD-WAN VPX (Branch)

VM	eth0	eth1	eth2	eth3
SDWAN_BR	LAN-BR	INET-BR	MPLS-BR	LAN-BR

1. 1. 1. 1. Import the .ova Template and create the VPX for the branch site. The network configuration should look like this.
         2. Rename the VM to “SDWAN_BR” and set the management ip address.
            management_ip
            set interface 192.168.3.253 255.255.255.0 192.168.3.1
            apply
            y
         3. Import a valid license file.
         4. Change NTP, time zone and timeout interval.

Provision the Configuration (Staging)

1. Login to the SDWAN_DC again – https://192.168.2.253
2. Go to the “Configuration Editor” of the Virtual WAN and click on “Export”. If the configuration is not loaded just open it.
3. Export the Configuration the “Change Management Inbox”.
4. Switch to “Change Management”
5. If you start the staging process with “Stage Appliances” you will get an error message because the software package is missing. The software packing is containing the firmware file and needs to be imported to the MCN. This is needed because imagine your physical MCN was shipped with Firmware 9.3 but the branch appliance is still running on Firmware 9.2. With this procedure its secured that the firmware will be same on every appliance you build the Virtual WAN.
6. Download the Software Package for the matching SD-WAN Release.
7. Upload the package and try again to start the staging.
8. Accept the EULA
9. Go to “Next”
10. Click on “Activate Staged”.
11. Click again on “Activate Staged” and wait until the appliance is ready again. Can take up to 180 seconds.
12. Enable the “Citrix Virtual WAN Service” under Virtual WAN à “Enable/Disable/Purge Flows”

Setup the Branch VPX

There is also a way provision the configuration file without the following manual steps. This is called Zero-Touch Deployment. The configuration file will be uploaded to the Citrix Cloud. The configuration file need to be linked to the serial number of the appliance. The co-worker in the branch office just need connect an ethernet link on the management interface with DHCP and active internet connection. The config will be downloaded, and the appliance should be ready to use.

1. Go to Configuration, Virtual WAN, Change Management on the MCN. Download the active configuration file for the branch appliance.
2. Login to the Branch VPX – https://192.168.3.253
3. Go to Configuration, Local Change Management and upload the configuration file from step 1 –> Next
4. Click on “Activate Staged”
5. Select “OK” and wait until the webpage reloaded.
6. Enable the Citrix Virtual WAN Service.
7. Click on “Monitoring” and you should see that all the WAN Links are up and running. That’s it 🙂

Working with WANEM

With WANEM we have the ability to simulate stuff like lower bandwith and packetloss. Its only a short introduction I think you can handle the rest without problems.

1.  Connect the branch or datacenter Client to the network “WAN-INET-BR” or WAN-INET-DC”
2. Set a static ip address for the ethernet interface (192.168.50.10/24)
3. Browse to http://192.168.50.2/WANem  (case sensitive!)
4. Select “Advanced Mode” and click on Start.
5. Create a packet-loss of 25% and apply the settings.
6. Check the SD-WAN Dashboard (Monitoring).

SD-WAN as DHCP Server

You can use the SD-WAN appliance in the branch office as a DHCP server. Very usefull when there is no other server infrastructure available.

1. 1. Open the Configuration Editor and import your current configuration.
   2. Click on the “Branch Site” and create a new DHCP “Server Subnet”
   3. Specify the DHCP settings.
   4. Save and Export the configuration to the Change Management.
   5. Start the Staging procedure.
   6. Activate the new configuration.
   7. The configuration rollout was succesfull.
   8. Switch to your Branch Client and change the Ethernet Interface to DHCP to verify if its working.                            

Direct Internet Breakout at Branch

If the Branch Office should use the WAN Links directly to surf in the WWW  its possible to configure a direct Internet Breakout.

1. Open the Configuration Editor and import your current configuration.
2. Click on the “Branch Site” –> Connections –> Virtual Paths –> Add
3. Click on “WAN Links” and press the edit button.
4. Now you can specify which WAN Links should be used for the direct internet access. You have the ability to choose three different modes.
   1. Primary
   2. Second
   3. Balance
5. After you have applied the setting the routing table will be updated.
6. Rollout the new configuration.