Reinstalling a Nutanix Community Edition node on hardware that already hosted an AHV. The Phoenix installer boots, detects the hardware, then halts on a Python traceback:
FATAL An exception was raised: Traceback (most recent call last):
File "/root/phoenix/./phoenix", line 155, in <module>
main()
...
File "/root/phoenix/sysUtil.py", line 2086, in mount_ahv_lvm_volumes
shell_cmd(['mount', '-t ext4', dev, target + mntpnt])
Exception: Failed command: [mount -t ext4 /dev/mapper/ahv-varlog /mnt/stage/var/log]
with error: [mount: /mnt/stage/var/log: special device /dev/mapper/ahv-varlog does not exist.]
The message seems perfectly accurate, and that is what makes it misleading: the device does not exist. The real question is why Phoenix is looking for it.
Understanding Why the Nutanix Installer Crashes
The call stack tells the whole story, provided you read it from bottom to top:
get_params → determine_actions → get_hyp_state
get_hyp_state → customize_kvm.get_state → __mount_kvm_partition → mount_ahv_lvm_volumes
Phoenix does not crash during the installation. It crashes beforehand, during the detection phase, when it tries to determine which actions to offer. It found an existing hypervisor on the disk, so it switched to its customize path (the one that updates an AHV in place rather than installing a fresh one), and this path requires mounting the volumes of the existing installation.
The preceding log lines confirm this reasoning:
INFO Calling dmsetup remove /dev/mapper/ahvstorage-varlogaudit
INFO Calling dmsetup remove /dev/mapper/ahvstorage-varlog
INFO LVM volume group ahv detected
INFO Found block device entry /dev/mapper/ahv-root, calling vgchange -an ahv
Two distinct volume groups appear: ahv and ahvstorage. An lvs command from the rescue shell confirms the layout:
| LV | VG | Size |
| home | ahv | 200 MB |
| root | ahv | 6.84 GB |
| tmp | ahv | 1.95 GB |
| var | ahv | 2.93 GB |
| varlog | ahvstorage | 39.06 GB |
| varlogaudit | ahvstorage | 1000 MB |
There is the discrepancy. On the disk, varlog lives in the ahvstorage VG. The Phoenix ISO, however, is looking for /dev/mapper/ahv-varlog, meaning varlog inside the ahv VG. It is applying an LVM schema that does not match what is present on the machine.
The most likely explanation is a generational gap: the AHV layout evolved toward a separate storage VG, and the media used for the reinstallation expects the older layout. As a result, Phoenix recognizes the existing installation well enough to want to reuse it, but not well enough to know how to mount it.
The Solution: Wiping the System Disk Clean
Since the blockage stems entirely from the detection of a pre-existing installation, the solution is to ensure there is nothing left to detect.
Phoenix leaves a root shell accessible after the failure. We start by mapping the layout before destroying it:
lsblk
pvs
vgs
lvs
In my case, there were two PVs on a single disk: /dev/sda3 for ahv, and /dev/sda5 for ahvstorage.
Next, we deactivate and remove the VGs:
vgchange -an ahvstorage
vgremove -f ahvstorage
ahvstorage disappears without resistance. ahv, however, is stubborn:
Logical volume ahv/root contains a filesystem in use.
Can't deactivate volume group "ahv" with 2 open logical volume(s)
The Second Problem: Volumes Left Mounted
This message is the real turning point of the procedure, and the lvs output provides the key:
root ahv -wi-ao----
var ahv -wi-ao----
The o attribute means open. These two logical volumes are mounted. They are the “2 open logical volume(s)” from the error message, and this is what subsequently causes the whole chain to fail:
pvremove -ff -y /dev/sda3
Can't open /dev/sda3 exclusively. Mounted filesystem?
Cannot use /dev/sda3: device has a signature
wipefs -a /dev/sda
wipefs: error: /dev/sda: probing initialization failed: Device or resource busy
The cause is simple: Phoenix had mounted root and var under /mnt/stage before crashing on varlog, and it unmounted nothing upon exiting. The failure left the disk in an intermediate state.
Therefore, unmounting must be done before anything else:
findmnt -R /mnt/stage
umount -R /mnt/stage
If a process is still holding the partition:
fuser -vm /mnt/stage
fuser -km /mnt/stage
Verify that the LVs are properly closed; the attribute should be -wi-a-----, without the o:
lvs
And only then, proceed with the complete wipe:
vgchange -an ahv
vgremove -f ahv
pvremove -ff -y /dev/sda3
wipefs -a /dev/sda
sgdisk --zap-all /dev/sda
dd if=/dev/zero of=/dev/sda bs=1M count=100 conv=fsync
The sgdisk --zap-all is not decorative. A GPT table has a backup copy at the end of the disk, which the dd on the first 100 megabytes leaves perfectly intact. This is why volume groups often “come back” after a wipe you thought was complete.
If the CVM data disks also hold remnants, apply the same treatment to them: run wipefs -a then sgdisk --zap-all on each.
Reboot, restart the installer, and Phoenix will no longer detect anything: it will offer a fresh installation.
My Takeaways from this Reinstallation
An accurate error message can point to the wrong problem:
“ahv-varlog does not exist” is literally true and completely beside the point. The problem is not the missing volume; it is that the installer is applying a schema that doesn’t match the disk. Without reading the call stack, you set out looking for a missing volume instead of realizing you are in the wrong branch of the program.
An installer failure leaves state behind:
Phoenix crashed after mounting two volumes, and these mounts survived its termination. All the subsequent sequence failures (pvremove, wipefs) stemmed directly from this. Before fighting against a “Device or resource busy” error, you must find out what is holding the device.
An explicitly destructible cluster changes the nature of the problem:
This procedure is brutal: it wipes everything. It is only acceptable because this node belongs to an environment designated as disposable, on which no irreplaceable data depends. Deciding in advance which cluster is disposable does not change the failure itself, but it completely changes the amount of time spent solving it.
