Sometimes and for various reasons, it is necessary to configure the VLAN directly at the level of our Nutanix cluster, in particular to ensure network segmentation.

Use case

Having had a little time for myself during the Christmas holidays, I set about resuming the configuration of my local network in order to isolate my Nutanix lab from my internal network.

To do this, I had to reconfigure my Ubiquiti equipment in order to:

• create VLAN 84 at the Dream Machine Pro level

• propagate VLAN 84 on the 24-port switch then on the 5-port switch on which the cluster is connected

Changing the VLAN on AHV

Before starting the modifications, I start by checking the network configuration of my host:

[root@NTNX-5e8f7308-A ~]# ovs-vsctl list port br0
_uuid : b76f885d-59b2-4153-99d3-27605a729ab8
bond_active_slave : []
bond_downdelay : 0
bond_fake_iface : false
bond_mode : []
bond_updelay : 0
cvlans : []
external_ids : {}
fake_bridge : false
interfaces : [17e8b0de-2ef5-4f6f-b253-94a766ec9603]
lacp : []
mac : []
name : br0
other_config : {}
protected : false
qos : []
rstp_statistics : {}
rstp_status : {}
statistics : {}
status : {}
tag : 0
trunks : []
vlan_mode : []

The output of the command shows us that there is no tag on my host. We will fix this with the following command:

[root@NTNX-5e8f7308-A ~]# ovs-vsctl set port br0 tag=84

The command “ovs-vsctl set port br0 tag=” allows me to tag my host interface with the VLAN ID that I have dedicated to my Nutanix network. We then check that the configuration is applied:

[root@NTNX-5e8f7308-A ~]# ovs-vsctl show
Bridge br0
    Port vnet4
        tag: 0
        Interface vnet4
    Port br0-up
        Interface eth4
        Interface eth0
        Interface eth5
        Interface eth2
        Interface eth1
        Interface eth3
    Port br0.u
        Interface br0.u
            type: patch
            options: {peer=br.dmx.d.br0}
    Port br0
        tag: 84
        Interface br0
            type: internal
    Port br0-dhcp
        Interface br0-dhcp
            type: vxlan
            options: {key="1", remote_ip="192.168.84.200"}
    Port br0-arp
        Interface br0-arp
            type: vxlan
            options: {key="1", remote_ip="192.168.5.2"}
    Port vnet2
        Interface vnet2
ovs_version: "2.14.8"

We can now see that the VLAN is configured on my host, we must now do the configuration on the CVM side…

Configuring the VLAN on the CVM

We start by checking the network configuration of our CVM:

[root@NTNX-5e8f7308-A ~]# ovs-vsctl show
    Bridge br0
        Port br0-up
            Interface eth4
            Interface eth0
            Interface eth5
            Interface eth2
            Interface eth1
            Interface eth3
        Port br0-arp
            Interface br0-arp
                type: vxlan
                options: {key="1", remote_ip="192.168.5.2"}
        Port br0.u
            Interface br0.u
                type: patch
                options: {peer=br.dmx.d.br0}
        Port vnet5
            Interface vnet5
        Port br0
            tag: 84
            Interface br0
                type: internal
        Port br0-dhcp
            Interface br0-dhcp
                type: vxlan
                options: {key="1", remote_ip="192.168.84.200"}
        Port vnet2
            Interface vnet2
    ovs_version: "2.14.8"

Here we can see that my network interface does not have any vlan information. So I proceed to configure the VLAN ID by connecting to my CVM and then typing the command

change_cvm_vlan VLANID

nutanix@NTNX-5e8f7308-A-CVM:192.168.84.200:~$ change_cvm_vlan 84
This operation will perform a network restart. Please enter [y/yes] to proceed or any other key to cancel: y
Changing vlan tag to 84
Replacing external NIC in CVM, old XML:
<interface type="bridge">
      <mac address="52:54:00:8e:69:bc" />
      <source bridge="br0" />
      <virtualport type="openvswitch">
        <parameters interfaceid="356e3bf3-5700-4131-b1b2-4fa65195a6e2" />
      </virtualport>
      <target dev="vnet0" />
      <model type="virtio" />
      <driver name="vhost" queues="4" />
      <alias name="ua-1decc31c-2764-416a-b509-d54ecd1a684f" />
      <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x0" />
    </interface>

        new XML:
<interface type="bridge">
      <mac address="52:54:00:8e:69:bc" />
      <model type="virtio" />
      <driver name="vhost" queues="4" />
      <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x0" />
    <source bridge="br0" /><virtualport type="openvswitch" /><vlan><tag id="84" /></vlan></interface>

CVM external NIC successfully updated.
Performing a network restart

We now check the CVM network configuration to verify that the tag has been configured correctly:

[root@NTNX-5e8f7308-A ~]# ovs-vsctl show
    Bridge br0
        Port br0-up
            Interface eth4
            Interface eth0
            Interface eth5
            Interface eth2
            Interface eth1
            Interface eth3
        Port br0-arp
            Interface br0-arp
                type: vxlan
                options: {key="1", remote_ip="192.168.5.2"}
        Port br0.u
            Interface br0.u
                type: patch
                options: {peer=br.dmx.d.br0}
        Port vnet5
            tag: 84
            Interface vnet5
        Port br0
            tag: 84
            Interface br0
                type: internal
        Port br0-dhcp
            Interface br0-dhcp
                type: vxlan
                options: {key="1", remote_ip="192.168.84.200"}
        Port vnet2
            Interface vnet2
    ovs_version: "2.14.8"

My CVM is now on VLAN 84. All I have to do now is repeat these operations on all my nodes and then check that everything works properly.

WARNING: the change_cvm_vlan command has a known bug in 6.8 with AHV 20230302.100173 that causes the VLAN ID not to be preserved when repeating the host: https://portal.nutanix.com/page/documents/kbs/details?targetId=kA0VO0000002uJ30AI