Team Leader - Nutanix Technology Champion - Nutanix NTC Storyteller

Julien DUMUR
Infrastructure in a Nutshell

This weekend I set out to move the IPMI of my lab Nutanix node onto my management segment. I changed the card’s address and the switch port’s VLAN profile within the same minute. The node had just been wiped, it had no operating system left, so there was no way to fix anything from the inside, and the card vanished from the network.

What gave me back the controls came down to an IPv6 link-local address and two commands, a quarter of an hour all told. Here is the method, on Windows 11 and on Rocky Linux.

The evening my IPMI vanished from the network

Two changes in the same minute left me no way of telling which one had broken what. I plugged a laptop straight into the dedicated IPMI port of this Supermicro SYS-5019D-4C-FN8TP to get back in.

On a running node, ipmitool locally settles that kind of blunder in a single command through the KCS interface, without going anywhere near the network, but a freshly wiped server closes that door. If you are in the opposite situation, with the card reachable and the password lost, I covered that case here: lost IPMI password on a Nutanix or Supermicro node.

Why link-local answers when IPv4 stays silent

Every active IPv6 interface carries an address starting with fe80::, which it builds on its own from its MAC and without DHCP. A machine whose IPv4 is wrong or missing therefore keeps a reachable identity on its segment. The ff02::1 address is the all-nodes multicast group: a ping to that group makes everything living at the end of the cable answer, and an IPMI powered in standby answers even with the server switched off. Multicast does not cross a router, so you do have to sit on the same segment as the card.

One detail settles the case for this method, which I only saw later while reading back my card’s configuration on firmware 01.74.13: the line “ARP Responses Enabled, Gratuitous ARP Disabled”. The IPMI never announces itself on the network, so listening passively on the segment gets you nowhere.

Finding the card on the cable

The principle is the same on both systems: identify the interface, send the multicast, read the neighbour cache.

On Windows 11

  • Spotting the adapter and its index:
Get-NetAdapter

The ifIndex column gives you the number to reuse afterwards, index 3 in my case.

On a direct link there is no DHCP server at all, and an interface left on automatic falls back to APIPA on 169.254.x.x, in which case nothing will ever ping, even against an IPMI in perfect health. Set a static address:

New-NetIPAddress -InterfaceAlias "Ethernet" -IPAddress 192.168.10.50 -PrefixLength 24
  • Sending the multicast, with the interface index as the zone suffix:
ping -6 ff02::1%3
  • Reading the discovered neighbours, with their MAC address:
Get-NetNeighbor -InterfaceIndex 3

On Rocky Linux

  • Identifying the interface:
ip link show
  • Sending the multicast:
ping6 -c4 ff02::1%ens0
  • Command output:
PING ff02::1%ens0(ff02::1%ens0) 56 data bytes
64 bytes from fe80:0:0:0:225:90ff:feb8:f313%ens0: icmp_seq=1 ttl=64 time=0.659 ms
64 bytes from fe80:0:0:0:225:90ff:feb8:f313%ens0: icmp_seq=2 ttl=64 time=0.288 ms
  • Matching the addresses to their MAC:
ip -6 neigh show dev ens0

One subtlety costs time when you move from one system to the other: the zone suffix takes an interface name on Linux and a numeric index on Windows, and an address copied from one console to the other fails without any explicit message.

Reading the vendor straight from the address

When a link-local address carries ff:fe in the middle, it derives from the MAC through the EUI-64 method, and the operation reverses in your head. Drop the two ff:fe bytes from the middle of fe80::225:90ff:feb8:f313, you are left with 02:25:90:b8:f3:13, then flip the second bit of the first byte: 02 becomes 00, which is a Super Micro Computer OUI.

Opening the web interface over link-local

The address you have recovered goes straight into a browser, in square brackets and followed by its zone suffix:

https://[fe80::225:90ff:feb8:f313%3]

The suffix is mandatory, a link-local address being unique only on a given segment. If your browser refuses that format, curl will at least confirm the service answers, with the % escaped as %25 and the -g option, without which the brackets are treated as globbing:

curl -k -g "https://[fe80::225:90ff:feb8:f313%25ens0]/"

What the IPMI told me

Once the interface was open, the answer came down to one line on its network page: the card was carrying 192.168.10.11, when I thought I had configured 192.168.10.1. The port’s VLAN profile had nothing to do with it, and the address I had typed belongs to the machine hosting my DNS. Reconfigured on the right address, and it was back.

My management cards now live in a reserved range at the bottom of the segment, with a matching DHCP exclusion, the lab node on .1 and the production node on .2. And I have stuck to a simple rule since: one change at a time, checked before the next one.

Read More

Sometimes and for various reasons, it is necessary to configure the VLAN directly at the level of our Nutanix cluster, in particular to ensure network segmentation.

Use case

Having had a little time for myself during the Christmas holidays, I set about resuming the configuration of my local network in order to isolate my Nutanix lab from my internal network.

To do this, I had to reconfigure my Ubiquiti equipment in order to:

  • create VLAN 84 at the Dream Machine Pro level
  • propagate VLAN 84 on the 24-port switch then on the 5-port switch on which the cluster is connected

Changing the VLAN on AHV

Before starting the modifications, I start by checking the network configuration of my host:

[root@NTNX-5e8f7308-A ~]# ovs-vsctl list port br0
_uuid : b76f885d-59b2-4153-99d3-27605a729ab8
bond_active_slave : []
bond_downdelay : 0
bond_fake_iface : false
bond_mode : []
bond_updelay : 0
cvlans : []
external_ids : {}
fake_bridge : false
interfaces : [17e8b0de-2ef5-4f6f-b253-94a766ec9603]
lacp : []
mac : []
name : br0
other_config : {}
protected : false
qos : []
rstp_statistics : {}
rstp_status : {}
statistics : {}
status : {}
tag : 0
trunks : []
vlan_mode : []

The output of the command shows us that there is no tag on my host. We will fix this with the following command:

[root@NTNX-5e8f7308-A ~]# ovs-vsctl set port br0 tag=84

The command “ovs-vsctl set port br0 tag=” allows me to tag my host interface with the VLAN ID that I have dedicated to my Nutanix network. We then check that the configuration is applied:

[root@NTNX-5e8f7308-A ~]# ovs-vsctl show
Bridge br0
    Port vnet4
        tag: 0
        Interface vnet4
    Port br0-up
        Interface eth4
        Interface eth0
        Interface eth5
        Interface eth2
        Interface eth1
        Interface eth3
    Port br0.u
        Interface br0.u
            type: patch
            options: {peer=br.dmx.d.br0}
    Port br0
        tag: 84
        Interface br0
            type: internal
    Port br0-dhcp
        Interface br0-dhcp
            type: vxlan
            options: {key="1", remote_ip="192.168.84.200"}
    Port br0-arp
        Interface br0-arp
            type: vxlan
            options: {key="1", remote_ip="192.168.5.2"}
    Port vnet2
        Interface vnet2
ovs_version: "2.14.8"

We can now see that the VLAN is configured on my host, we must now do the configuration on the CVM side…

Configuring the VLAN on the CVM

We start by checking the network configuration of our CVM:

[root@NTNX-5e8f7308-A ~]# ovs-vsctl show
    Bridge br0
        Port br0-up
            Interface eth4
            Interface eth0
            Interface eth5
            Interface eth2
            Interface eth1
            Interface eth3
        Port br0-arp
            Interface br0-arp
                type: vxlan
                options: {key="1", remote_ip="192.168.5.2"}
        Port br0.u
            Interface br0.u
                type: patch
                options: {peer=br.dmx.d.br0}
        Port vnet5
            Interface vnet5
        Port br0
            tag: 84
            Interface br0
                type: internal
        Port br0-dhcp
            Interface br0-dhcp
                type: vxlan
                options: {key="1", remote_ip="192.168.84.200"}
        Port vnet2
            Interface vnet2
    ovs_version: "2.14.8"

Here we can see that my network interface does not have any vlan information. So I proceed to configure the VLAN ID by connecting to my CVM and then typing the command

change_cvm_vlan VLANID
nutanix@NTNX-5e8f7308-A-CVM:192.168.84.200:~$ change_cvm_vlan 84
This operation will perform a network restart. Please enter [y/yes] to proceed or any other key to cancel: y
Changing vlan tag to 84
Replacing external NIC in CVM, old XML:
<interface type="bridge">
      <mac address="52:54:00:8e:69:bc" />
      <source bridge="br0" />
      <virtualport type="openvswitch">
        <parameters interfaceid="356e3bf3-5700-4131-b1b2-4fa65195a6e2" />
      </virtualport>
      <target dev="vnet0" />
      <model type="virtio" />
      <driver name="vhost" queues="4" />
      <alias name="ua-1decc31c-2764-416a-b509-d54ecd1a684f" />
      <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x0" />
    </interface>

        new XML:
<interface type="bridge">
      <mac address="52:54:00:8e:69:bc" />
      <model type="virtio" />
      <driver name="vhost" queues="4" />
      <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x0" />
    <source bridge="br0" /><virtualport type="openvswitch" /><vlan><tag id="84" /></vlan></interface>

CVM external NIC successfully updated.
Performing a network restart

We now check the CVM network configuration to verify that the tag has been configured correctly:

[root@NTNX-5e8f7308-A ~]# ovs-vsctl show
Bridge br0
Port br0-up
Interface eth4
Interface eth0
Interface eth5
Interface eth2
Interface eth1
Interface eth3
Port br0-arp
Interface br0-arp
type: vxlan
options: {key="1", remote_ip="192.168.5.2"}
Port br0.u
Interface br0.u
type: patch
options: {peer=br.dmx.d.br0}
Port vnet5
tag: 84
Interface vnet5
Port br0
tag: 84
Interface br0
type: internal
Port br0-dhcp
Interface br0-dhcp
type: vxlan
options: {key="1", remote_ip="192.168.84.200"}
Port vnet2
Interface vnet2
ovs_version: "2.14.8"

My CVM is now on VLAN 84. All I have to do now is repeat these operations on all my nodes and then check that everything works properly.

WARNING: the change_cvm_vlan command has a known bug in 6.8 with AHV 20230302.100173 that causes the VLAN ID not to be preserved when repeating the host: https://portal.nutanix.com/page/documents/kbs/details?targetId=kA0VO0000002uJ30AI

Read More

To be able to deploy a virtual machine on your Nutanix cluster and have it reachable on your network, you will need to start by configuring the network(s) on your cluster.

Creating a network using Prism Element

Under Prism Element, in the “Settings > Network Configuration” menu is the list of all existing networks on the cluster, click on “Create Subnet”:

Then enter your network information, namely the name and vlan ID:

If you do not have a DHCP server, you can let Nutanix manage the addressing of the network created using the “Enable IP address management” option:

You will then need to complete all the options that would normally have been delivered by a traditional DHCP server:

Click “Save” once the settings are correct. Repeat for each VLAN you need on your infrastructure.

Creating a network using Prism Central

In Prism Central, network management is carried out in “Network & Security > Subnets”:

To add a new network, click “Create Subnet”:

It is then a form similar to that of Prism Element that must be completed by activating, or not, the “IP Address Management” option if you wish to leave the management of your addressing to Nutanix.

Official Nutanix documentation

Link to official documentation: https://portal.nutanix.com/page/documents/solutions/details?targetId=BP-2071-AHV-Networking:bp-ahv-network-management.html

Read More

Recently, my Ubiquiti USW-Pro-24-PoE switch had some connection issues with my Unifi console.

Read More