Leaving vSphere is a disk convert, then ordinary replication

Leaving vSphere is a disk convert, then ordinary replication. Power the guest off, copy the consolidated VMDK off the datastore, and convert it onto that guest’s dataset. Write a domain XML for the firmware and a disk bus the guest already has a driver for. Boot a clone on a bridge with no uplink. zfs send of the snapshot that clone booted from is the same send you already use for every other guest.

You do not need a backup platform’s permission to leave. A cross-hypervisor restore moves the bytes, and then you still have to boot the clone and send the snapshot.

How do you migrate a VMware VM to Proxmox or KVM?

Stop the guest first. A VMDK copied from a running VM still has writes in flight, and the convert keeps them. If you cannot power it off, the copy is only as consistent as the freeze you took. Make application-consistent virtual machine backups.

Consolidate snapshots on vSphere before you copy. web-01-000001.vmdk is a delta, not the disk. Back up VMware virtual machines is how that copy leaves the datastore. Start here once the files are on the new host.

Copy the VMDK, the .vmx, and the .nvram when the firmware line says efi. A raw device mapping is not in the VMDK. Neither is an independent disk the backup skipped. Convert the boot disk alone and the guest can reach a login while the data disk is still on the old datastore.

Put the converted file on a dataset that holds only this guest. One dataset per virtual machine is the job, the quota, and the key. On Proxmox, importing the VMDK onto a storage is this same convert. The VM config is the catalog there. On libvirt, the catalog is the domain XML. Back up Proxmox virtual machines.

How do you convert a VMDK to qcow2 or raw?

Open the file with the format forced:

qemu-img info -f vmdk web-01.vmdk

Without -f, qemu-img probes. A raw file renamed to .vmdk probes as raw and looks openable. -f vmdk rejects it.

For one qcow2 file on the dataset:

qemu-img convert -f vmdk -O qcow2 web-01.vmdk /tank/vm/web-01/disk0.qcow2

For a raw file on that dataset:

qemu-img convert -f vmdk -O raw web-01.vmdk /tank/vm/web-01/disk0.raw

Do not run those paths against a live guest. The script below builds a 1 MiB disk in a temp directory.

Skip -c, and skip --salvage. A compressed qcow2 is the wrong boot disk here. --salvage replaces unreadable ranges with zeros and can still exit 0. Skip -n unless the target zvol already exists and is large enough. The script does not create a zvol.

qemu-img compare is the byte check while the VMDK is still there. After you delete it, a matching virtual size is all qemu-img info can tell you, and that is not a boot. Keep one raw file, one qcow2 file, or a zvol on the guest’s dataset. Do not add a qcow2 overlay on top. A snapshot is not a backup. The recovery point, after the boot, is a ZFS snapshot under the disk.

#!/usr/bin/env bash
set -euo pipefail

root=$(mktemp -d)
case "$root" in
  /*/tmp.*) ;;
  *) printf 'unexpected temp dir %s\n' "$root" >&2
     exit 1 ;;
esac
trap 'rm -rf "$root"' EXIT

virtual_size() {
  local fmt="$1"
  local path="$2"
  qemu-img info -f "$fmt" "$path" | awk -F'[()]' '/^virtual size:/ { print $2; exit }' | awk '{ print $1 }'
}

qemu-img create -f raw "$root/src.raw" 1M >/dev/null
printf 'migrate-marker' | dd of="$root/src.raw" bs=1 conv=notrunc status=none

qemu-img convert -f raw -O vmdk "$root/src.raw" "$root/src.vmdk"
qemu-img convert -f vmdk -O qcow2 "$root/src.vmdk" "$root/out.qcow2"
qemu-img convert -f vmdk -O raw "$root/src.vmdk" "$root/out.raw"

qemu-img compare -f raw "$root/src.raw" "$root/out.raw"

raw_size=$(virtual_size raw "$root/src.raw")
vmdk_size=$(virtual_size vmdk "$root/src.vmdk")
qcow_size=$(virtual_size qcow2 "$root/out.qcow2")
out_size=$(virtual_size raw "$root/out.raw")

[[ "$raw_size" == "$vmdk_size" ]]
[[ "$raw_size" == "$qcow_size" ]]
[[ "$raw_size" == "$out_size" ]]
[[ "$raw_size" == "1048576" ]]

cp "$root/src.raw" "$root/misnamed.vmdk"
if qemu-img info -f vmdk "$root/misnamed.vmdk" >/dev/null 2>&1; then
  printf 'forced vmdk format accepted a raw file\n' >&2
  exit 1
fi

printf 'Convert check passed.\n'
printf 'Proves: on a 1 MiB fixture, qemu-img convert from vmdk to qcow2 and to raw kept the virtual size, and the raw output matched the source bytes. -f vmdk rejected a raw file named .vmdk.\n'
printf 'Does not prove: a vSphere VMDK opened, a guest booted, or a domain XML matched the firmware.\n'

Which VMDK do you refuse?

A name ending in -000001.vmdk, or any six-digit snapshot suffix, is a delta. Do not convert it as the migration disk. qemu-img will open a delta if the parent sits beside it, and the output is that snapshot, not a consolidate you skipped. If qemu-img info -f vmdk prints backing file:, stop and consolidate, unless you meant that snapshot and the parent file is on this host.

A monolithic flat disk is a short descriptor plus a -flat.vmdk extent. The descriptor alone does not open. Copy the extent with the descriptor. The -flat file is not a delta, and it is not the file you pass to convert. Pass the descriptor.

A file that opens at the wrong virtual size is the wrong disk. Write the size down from vSphere before you wipe the datastore. The script uses the fixture’s own size as that number.

If info cannot open the file, do not point convert at a different VMDK. A missing extent fails open. A vSphere-encrypted disk fails open, because qemu-img does not have the vCenter key. Decrypt or export an unencrypted copy on vSphere, then convert that file.

#!/usr/bin/env bash
set -euo pipefail

root=$(mktemp -d)
case "$root" in
  /*/tmp.*) ;;
  *) printf 'unexpected temp dir %s\n' "$root" >&2
     exit 1 ;;
esac
trap 'rm -rf "$root"' EXIT

virtual_size() {
  local path="$1"
  qemu-img info -f vmdk "$path" | awk -F'[()]' '/^virtual size:/ { print $2; exit }' | awk '{ print $1 }'
}

has_backing() {
  local path="$1"
  qemu-img info -f vmdk "$path" | awk 'BEGIN { found=0 } /^backing file:/ { found=1 } END { exit (found ? 0 : 1) }'
}

is_delta_name() {
  local base
  base=$(basename "$1")
  [[ "$base" =~ -[0-9]{6}\.vmdk$ ]]
}

accept_disk() {
  local path="$1"
  local expect="$2"
  local size
  if is_delta_name "$path"; then
    return 1
  fi
  if ! qemu-img info -f vmdk "$path" >/dev/null 2>&1; then
    return 1
  fi
  if has_backing "$path"; then
    return 1
  fi
  size=$(virtual_size "$path") || return 1
  [[ "$size" == "$expect" ]]
}

must_fail() {
  local label="$1"
  shift
  if "$@"; then
    printf 'checker accepted %s\n' "$label" >&2
    exit 1
  fi
}

qemu-img create -f raw "$root/good.raw" 1M >/dev/null
qemu-img convert -f raw -O vmdk "$root/good.raw" "$root/web-01.vmdk"
expect=$(virtual_size "$root/web-01.vmdk")

cp "$root/web-01.vmdk" "$root/web-01-000001.vmdk"
qemu-img create -f vmdk -b "$root/web-01.vmdk" -F vmdk "$root/web-01-child.vmdk" 1M >/dev/null
qemu-img create -f raw "$root/small.raw" 64K >/dev/null
qemu-img convert -f raw -O vmdk "$root/small.raw" "$root/small.vmdk"

cat > "$root/missing-extent.vmdk" << 'DESC'
# Disk DescriptorFile
version=1
encoding="UTF-8"
CID=fffffffe
parentCID=ffffffff
createType="monolithicFlat"
# Extent description
RW 2048 FLAT "missing-flat.vmdk" 0
# The Disk Data Base
ddb.virtualHWVersion = "14"
ddb.adapterType = "lsilogic"
DESC

accept_disk "$root/web-01.vmdk" "$expect"
must_fail 'delta name' accept_disk "$root/web-01-000001.vmdk" "$expect"
must_fail 'backing file' accept_disk "$root/web-01-child.vmdk" "$expect"
must_fail 'missing extent' accept_disk "$root/missing-extent.vmdk" "$expect"
must_fail 'wrong size' accept_disk "$root/small.vmdk" "$expect"

printf 'VMDK accept check passed.\n'
printf 'Proves: the fixture accepts one consolidated vmdk at the expected virtual size, and rejects a -000001 name, a vmdk with a backing file, a descriptor whose extent is missing, and a smaller disk.\n'
printf 'Does not prove: the file came from vSphere, the guest boots, or every disk of a multi-disk VM was copied.\n'

What has to match in the domain XML?

The convert does not read the .vmx. You do. The catalog is the domain XML saved with the snapshot.

Which firmware do you set?

If the .vmx has firmware = "efi", define OVMF and keep the .nvram you copied. If that line is missing, treat the guest as BIOS, and confirm it on the powered-off VM before you wipe the datastore. A BIOS guest on OVMF stops in the firmware. An EFI guest on SeaBIOS does the same. The boot entry may live in the .nvram file and not in the VMDK. Convert cannot rebuild that file.

Which disk bus can the guest already see?

QEMU has no vmxnet3 NIC and no pvscsi controller. Use virtio or e1000 for the NIC. For the disk, use a bus the guest already has a driver for. A Windows image with only the pvscsi driver will not boot from virtio. It reports an inaccessible boot device. Boot that guest on SATA, or on an LSI SCSI controller, until a virtio driver is installed and that second boot works. A Linux initramfs built only for the VMware SCSI driver fails the same way: the firmware sees the disk, and the root filesystem does not.

Attach the disks in the .vmx boot order. The largest file is often the data disk.

Leave the NIC on a bridge with no uplink for the first start. A kept MAC on the production bridge answers for the production address if the original VM is still registered. A cleanroom is a clone on a bridge with no uplink.

How do you prove the converted disk boots?

Snapshot the dataset before the first start. A wrong bus in the XML should cost you another clone, not another convert. Boot the clone, not the only copy.

Look for a record written before the VMware shutdown. A file you noted, a row, a log line with a time from before power-off. The firmware splash is not that record. A login prompt is not the application. Test a virtual machine restore.

If the clone fails, change the XML and boot a new clone of the same snapshot. Convert again only if the virtual size was wrong. Destroy the test domain when the look is done. If this boot is the cutover, leave the VMware guest powered off. Two guests with one hostname, on a network that reaches production, will both answer.

When does ordinary replication start?

After that boot. Send the snapshot you cloned. That is the first recovery point on this stack. A snapshot you have not booted a clone of is still a convert you have not started.

From there the job matches every other guest. The recovery point objective is the time since the last successful send. The copy has to leave this pool. Keep an off-site virtual machine backup. When you expire local dailies, keep a bookmark so the next send still has a base. Bookmarks let you delete dailies without breaking the next send.

The replica does not need a vSphere host after that send. The next incremental does not convert the disk again. The replica still expires on its own clock. The replica keeps snapshots the source is no longer allowed to expire.

What fails when qemu-img convert exits 0?

A delta converted with its parent present is that snapshot, not the guest at power-off. Check the name before you send.

A boot on the production bridge can update DNS while the original is still registered. open-vm-tools left installed is not a failed convert. Removing it does not add the virtio driver.

A send that starts before the clone boots copies a disk you have not started.