Veeam Cloud Connect Replication does not only manage virtual machine replication, but also offers a complete networking solution to easily publish failed over virtual machines in case of a disaster. This has always proved to be a tremendous feature of the solution. There are however some specific use cases where the Network Extension Appliance (NEA) may be better replaced with a different solution. One of the use cases is when a tenant needs to have more than 9 virtual networks to publish his virtual machines.
The limit of VMware virtual machines
Where does this limit of 9 networks comes from? Actually, it’s not a Veeam limit, rather a VMware one. Veeam NEA is deployed at the service provider in the form of a virtual machine, running on top of the same hypervisor where the replicated VMs are also saved. Thanks to its networking capabilities, the NEA can connect together all the different networks assigned to a given tenant, and allow the VMs to either reach Internet, or to be reached from Internet. This happens because the NEA leverages the routing and firewalling capabilities of the Linux kernel that is used: like in any firewall, one network in used for the external interface, and the others as different internal networks.
Now, VMware virtual machines have a limit of 10 network interfaces. So, if we do the maths, 1 is the external, and we are left with a maximum of 9 internal networks. For small customers this has never been a problem, many of them use way less than 9 networks in their infrastructure, often I see just one virtual machine network where every VM is connected to. But as the solution is growing and bigger customers are consuming DR services using Veeam Cloud Connect Replication (VCC-R), this limit has surfaced already a couple of times, for me at least. And while the NEA cannot be increase any further, there is a different solution to this limit.
vSphere VLAN tagging
VLANs are used by VMware vSphere to isolate multiple networks from each other, while running over the same ethernet cable. At the same time, VCC-R uses vSphere port groups to create new dedicated networks to its tenants, by tagging each newly created portgroup with a unique VLAN id. In vSphere terms, this is called Virtual Switch Tagging (VST). This is not however the only available method: as you can read in VMware KB1003806, there are also External Switch Tagging (EST) and Virtual Guest Tagging (VGT).
In VCC-R, VST is used. Usually, this means that each portgroup is tagged with one VLAN id, and because of the limit of 10 (minus 1 in our scenario) network interfaces, VST also brings a limit of 10 different VLANs that can be mapped to a virtual machine. This is clearly stated in VCC-R when we create a new hardware plan:
But this is not the only available method.
If Distributed Port Groups are used, we can also use VLAN trunking: thanks to this option, we still need one portgroup for each VLAN we have to create, but we can also assign an entire range of IDs to the same interface. On the other side of the “cable”, we need a device that is capable of splitting the traffic based on the VLANid of a given packet.
Let’s see how this can be accomplished by using an example.
VCC-R setup
First, we need to create two hardware plans. Because of the 9 networks limits in VCC-R, for our customer that needs 12 networks, we will create a hardware plan with 9 interfaces, and another one with 3, like this:
Then, when we create the tenant and assign him replica resources, we assign to him both the hardware plans:
We also have to unselect the “Use built-in network management capabilities during failover”; this means that the Veeam Network Extension Appliance will not be deployed. This brings two consequences: partial failover will not ba available, and the provider will have to manually configure and give to the tenant a different firewall/router solution. This is what we will do next.
vSphere network configuration
As soon as the two hardware plans are assigned to the tenant, all the needed networking objects are automatically configured by Veeam Cloud Connect inside vSphere:
If we look into vSphere networking, we can clearly see all the new portgroups that have been created, each with their VLANid:
These are the 12 new portgroups where all the tenant virtual machines will be connected. What we need now is also another portgroup where all the VLANs are tagged at the same time. This one has to be created manually:
Since the new VM portgroups are created with consecutive IDs from 113 to 124, we create a new portgroup with a recognizable name, and we select the VLAN option of “VLAN trunk”, then, we specify the range of the trunk to be the same of the created portgroups. If the range is made of individual IDs, they can be specified one by one by separating them with a comma.
pfSense configuration
In this example we are going to deploy and configure pfSense; providers can choose to use a different solution depending on their needs. pfSense can be deployed in a VMware environment by creating a new empty virtual machine and then installing the appliance software. The machine needs to be FreeBSD 64 bit; it also needs to have two network cards:
As you can see immediately, we don’t need to have (we wouldn’t be able to have them anyway) 13 network connections to manage all the tenant’s portgroups. While the external networks is DPG-vcc_public, that is the external interface of Cloud Connect, the internal network is only connected to the VLAN trunk we created before. This network will receive all the traffic from all the VLANs we added to the trunk; then, pfSense will be the one responsible for identifying and splitting the different traffic based on its VLAN id.
Once the firewall is installed, we need to configure it. First, we need a new public IP to be assigned to the wan interface ,this is not only for managing the tenant’s traffic, but also to allow the tenant himself to self-configure the firewall; the internal interfaces are used for the DR virtual machines, so there’s no way for the tenant to connect from these networks to pfSense, he will have to use the wan interface. So, it’s important that the admin user has a really strong password, and that the appliance is kept updated using it’s integrated update features. When the IP is loaded in the wan interface, we need to access the web interface to configure the firewall. To do so, there’s a quick and dirty trick: go into the console, choose option 8) Shell, and in the command line run:
pfctl -d
This command will disable completely the FreeBSD integrate PF firewall. Don’t worry, this is only done for a few minutes will we configure the rest of the appliance. Now that the firewall is disabled, we can access the WebConfigurator with a browser, buy pointing it to https://wan_IP_address. Here, we follow the initial configuration wizard, where we can immediately change the admin password. We can also install any available update and even configure them to be automated.
One quick note here, since it drove me mad for a couple of minutes: ANY change in the firewall configuration immediately re-enables the firewall itself, so you my end up with a connetion timeout to the webinterface. It you still need to complete the different configurations, run again the command to disable PF until you are done.
So, my suggestion is that the first operation would be to go into Firewall -> Rules -> WAN, and create a new rule to allow the remote connection from outside to port 443 or another custom port that we may want to use for the Web interface. Once this rule is created, first you need to disable again PF, then if you have decided to use a custom port, you go into System -> Advanced -> Admin Access and you configure pfSense to listen on the new custom port (and no, this is not the one I’m using, it’s just an example):
As soon as you apply this configuration, PF will be enabled again, but if you have also created the new firewall rule, from here on you should be able to just access the web interface from WAN anyway.
Now, it’s finally time to setup the networks. We start by adding all the created VLANs to the appropriate list, and map them to the LAN interface:
Then, we go into Interface Assignments, and we’ll see, together with WAN and LAN, a new “Available network ports”, and in the dropdown list, all the VLANs we just created:
Here, we can add each VLAN as a new virtual interface:
We now have a firewall with 1 physical interface on the inside, but at the same time 12 different “virtual” interfaces, separated by the different VLAN ids.
We can finally pass the management interface of the firewall to our tenant, where he can create multiple firewall rules to allow traffic between the different subnets and the Internet.
We’ll see in the next post how the tenant can properly replicate all his virtual machines to the different networks and setup pfSense to allow the different communications.