The replica keeps snapshots the source is no longer allowed to expire
An immutable backup is a replica that still has snapshots the source already deleted. Refuse zfs receive -F. A hold the production host can release is not that lock.
The replica keeps snapshots the source is no longer allowed to expire. That is an immutable backup. The source may delete yesterday because its count says so, or because someone with the backup password deleted it. The replica still has that snapshot. A receive that makes the replica’s snapshot list match the source’s list copied the delete. It did not finish a backup.
What is an immutable backup?
An immutable backup is a copy the account that runs production cannot delete, or shorten, before the copy’s own clock says so. On this stack the copy is the replica’s snapshots. The source’s retention is a different clock. When the clocks match, a delete on the source becomes a delete on the replica.
A snapshot on the production pool dies with those disks and that root password. A snapshot is not a backup. Three copies means three pools. The building is an off-site backup. The keep count is backup retention for virtual machines.
What is the difference between an immutable backup and a normal backup?
A normal backup job may delete its own old points. You shorten a daily count, the job prunes, and that side loses the day. That is retention, and you want it on the source so the pool does not fill with dailies you will not boot.
An immutable backup is a second clock that job cannot turn. If the same password can prune both, you have one backup in two folders.
Should the replica keep snapshots the source expired?
Yes, when the failure you are planning for is a source that deletes its own history. Ransomware does that. So does a retention change the replica is set to follow. Skip the extra points for a guest you can rebuild. Keep them for the guest that still needs October after the source pruned it.
The source can keep 14 dailies while the replica keeps those dailies and the monthlies. If the replica’s count copies the source’s count, there is nothing left to refuse.
Why must the replica refuse zfs receive -F?
zfs receive -F forces a rollback of the destination to its most recent snapshot before the receive. On an incremental replication stream, the stream zfs send -R -i or zfs send -R -I produces, that flag also destroys destination snapshots and file systems the sender no longer has. OpenZFS documents both effects. The source destroyed @2026-10-01. The replica still has it. A receive with -F destroys it so the stream can land, and the command exits 0.
A replication package from zfs send -R is the same class of stream. On receive, that package is documented to destroy destination snapshots that do not exist on the sender, using zfs destroy -d. Do not use that package as the replica’s retention.
Send one dataset: zfs send -i from the previous snapshot, or from the bookmark that replaced it, into zfs receive with no -F. A plain incremental requires the replica’s most recent snapshot to match the start of the stream. It does not delete older snapshots the source has already destroyed. The replica pulls, and the production host cannot log in there.
If receive refuses because the replica still has a snapshot the source no longer sends, leave the refusal. The recovery-point age grows until the next send the replica can take without deleting its own points. The recovery point objective is the time since the last successful send. Do not add -F to make the timer look healthy.
-F also rolls back a clone you took on the replica after the last receive. Clone the test onto a dataset the receive does not touch. Instant recovery is a clone you can promote.
What does a hold stop?
zfs hold tag snapshot adds a reference. zfs destroy of that snapshot returns EBUSY while the hold exists. zfs release tag snapshot removes it. The account that can run zfs release can lift the tag.
A hold on the source stops the source from destroying the base snapshot of a send that is still running. It does not stop the replica from expiring. Bookmarks let you delete dailies without breaking the next send. A bookmark is not a snapshot the replica can boot.
A hold on the replica stops destroy, including the destroy zfs receive -F would run, until someone releases the tag. If production can log in and release it, the hold changes nothing. Root on the replica is the boundary. The replica’s own retention job releases the tag when its count says the point can go.
How do VMware, Hyper-V, and Proxmox keep the copy?
On VMware the lock is the repository, not the VM snapshot on the datastore. If vCenter, or the backup job, can delete those files early, the repository follows the source. Back up VMware virtual machines. On Hyper-V a checkpoint is the same miss. The backup target’s delete right is the lock. Back up Hyper-V virtual machines. On Proxmox, the token that runs vzdump must not prune the archive. A flag that same token can clear is not a lock. Back up Proxmox virtual machines. A ZFS replica of the Proxmox disk follows the receive rule above.
What fails?
The receive wrapper adds -F after a refused stream, because the last run failed. The next run exits 0, and October is gone. Alert on a receive command that contains -F, not only on a non-zero exit.
The replica’s prune job reads the source’s snapshot list and deletes anything missing. That is -F written as a script. It should read the replica’s count only.
A hold is set, and the same SSH key that pulls the stream can zfs release it. The replica disk fills because its count is longer and nobody sized the pool for that. A full pool is not a reason to pass -F. Shorten the replica’s count on purpose, after you restore the oldest point that shorten would delete. Choose backup retention for virtual machines.
The source snapshot list cannot show a point the source was not allowed to keep. Read the replica, and boot the snapshot the source no longer has, not the newest one. Keeping the name does not repair a torn write. The freeze had to finish when the point was taken. Make application-consistent virtual machine backups.
How do you prove the replica still has the point the source expired?
On the source, zfs list -t snapshot -o name,guid tank/vm-db. On the replica, the same command for the received dataset. The proof is a name and guid on the replica that the source list does not have, and that the replica’s count still wants. A matching newest guid only proves the last send arrived.
Clone that older snapshot onto storage that is not the production pool. New name, NIC on a bridge with no uplink. A record written before that snapshot has to be present. A record written after it has to be absent. If the later record is there, you booted production, or a newer point. Test a virtual machine restore. The bridge with no uplink is a cleanroom clone on a bridge with no uplink. Write down the dataset, the snapshot name, the guid, and whether that record was there.
Check the replica kept what the source expired
These scripts build their own fixtures under a temporary directory. They do not run zfs, log in to a replica, or read a virtual disk.
The first rejects zfs receive -F and a clustered -vF, accepts a receive with no -F, and requires the replica list to still contain a snapshot the source list no longer has.
#!/usr/bin/env bash
set -euo pipefail
root="$(mktemp -d)"
trap 'rm -rf "$root"' EXIT
printf '%s\n' \
'tank/vm-db@2026-11-06' \
'tank/vm-db@2026-11-07' \
'tank/vm-db@2026-11-08' \
> "${root}/source-snapshots"
printf '%s\n' \
'tank/vm-db@2026-10-01' \
'tank/vm-db@2026-11-06' \
'tank/vm-db@2026-11-07' \
'tank/vm-db@2026-11-08' \
> "${root}/replica-snapshots"
printf '%s\n' 'tank/vm-db@2026-10-01' > "${root}/replica-must-keep"
printf '%s\n' 'tank/vm-db@2026-09-01' > "${root}/missing-on-replica"
printf '%s\n' 'zfs receive tank/replica/vm-db' > "${root}/receive-ok"
printf '%s\n' 'zfs receive -F tank/replica/vm-db' > "${root}/receive-force"
printf '%s\n' 'zfs receive -vF tank/replica/vm-db' > "${root}/receive-cluster"
receive_forces_rollback() {
local token
local -a words
read -r -a words < "$1"
for token in "${words[@]}"; do
case "$token" in
-F) return 0 ;;
--*) ;;
-*)
if [[ "$token" == *F* ]]; then
return 0
fi
;;
esac
done
return 1
}
replica_kept_expired() {
local must="$1" source_list="$2" replica_list="$3" name
[[ -s "$must" ]] || return 1
while IFS= read -r name; do
[[ -n "$name" ]] || continue
grep -Fxq -- "$name" "$replica_list" || return 1
if grep -Fxq -- "$name" "$source_list"; then
echo "source still has ${name}" >&2
return 1
fi
done < "$must"
}
if receive_forces_rollback "${root}/receive-ok"; then
echo "rejected a receive that has no -F" >&2
exit 1
fi
if ! receive_forces_rollback "${root}/receive-force"; then
echo "accepted zfs receive -F" >&2
exit 1
fi
if ! receive_forces_rollback "${root}/receive-cluster"; then
echo "accepted clustered -vF" >&2
exit 1
fi
if replica_kept_expired "${root}/missing-on-replica" "${root}/source-snapshots" "${root}/replica-snapshots"; then
echo "accepted a snapshot the replica list does not have" >&2
exit 1
fi
replica_kept_expired "${root}/replica-must-keep" "${root}/source-snapshots" "${root}/replica-snapshots"
echo "Replica retention check passed."
echo "Proves: on this fixture, a receive without -F is accepted, receive -F and -vF are rejected, and the replica list still contains a snapshot the source list no longer has."
echo "Does not prove: a pool refused the stream, a hold is set, or the guest boots."The second reads a fixture catalog. A snapshot the source no longer has is kept only when it has a hold the production account cannot release. A snapshot that is still on the source, a snapshot with no hold, and a hold the production account can release are not that point. The catalog is a fixture, not zfs holds output.
#!/usr/bin/env python3
"""Check a fixture catalog: the replica still holds a snapshot the source expired.
This does not run zfs, open a pool, or read a virtual disk.
"""
import tempfile
from pathlib import Path
def load_rows(path):
rows = []
for raw in path.read_text(encoding="utf-8").splitlines():
line = raw.strip()
if not line or line.startswith("#"):
continue
parts = line.split()
if len(parts) != 4:
raise SystemExit(f"bad row: {line}")
name, on_source, hold, releasable_by = parts
if on_source not in ("yes", "no"):
raise SystemExit(f"on_source must be yes or no: {line}")
rows.append(
{
"name": name,
"on_source": on_source,
"hold": hold,
"releasable_by": releasable_by,
}
)
return rows
def immutable_points(rows, production_account):
kept = []
for row in rows:
if row["on_source"] != "no":
continue
if row["hold"] in ("-", "none"):
continue
if row["releasable_by"] == production_account:
continue
kept.append(row["name"])
return kept
def main():
production = "backup-admin"
catalogs = {
"good": """\
# name on_source hold releasable_by
tank/vm-db@2026-10-01 no replica-monthly replica-root
tank/vm-db@2026-11-08 yes - -
""",
"still-on-source": """\
# name on_source hold releasable_by
tank/vm-db@2026-11-08 yes source-daily backup-admin
""",
"production-can-release": """\
# name on_source hold releasable_by
tank/vm-db@2026-10-01 no source-daily backup-admin
""",
"no-hold": """\
# name on_source hold releasable_by
tank/vm-db@2026-10-01 no none -
""",
}
with tempfile.TemporaryDirectory() as tmp:
root = Path(tmp)
results = {}
for name, text in catalogs.items():
path = root / f"{name}.txt"
path.write_text(text, encoding="utf-8")
results[name] = immutable_points(load_rows(path), production)
if results["good"] != ["tank/vm-db@2026-10-01"]:
raise SystemExit(f"good catalog kept the wrong points: {results['good']}")
for name in ("still-on-source", "production-can-release", "no-hold"):
if results[name]:
raise SystemExit(f"accepted {name}: {results[name]}")
print("Replica hold check passed.")
print(
"Proves: on this fixture, a snapshot the source no longer has is kept only when its hold is not releasable by the production account."
)
print(
"Does not prove: zfs hold is set on a pool, receive -F was refused, or the guest boots."
)
if __name__ == "__main__":
main()