Bookmarks let you delete dailies without breaking the next send

Bookmarks let you delete a daily snapshot and still start the next incremental from that point. zfs send -i needs an earlier point in the new snapshot’s history. A bookmark is that point, stored as a name. It does not keep the blocks. After the replica has the snapshot, bookmark it, destroy the snapshot on the source, and the next send is zfs send -i tank/vm-db#2026-11-04 tank/vm-db@2026-11-05.

The daily you deleted is not a recovery point. The recovery point is the snapshot the replica still has. A snapshot is not a backup. A bookmark is not one either.

What is a ZFS bookmark?

A bookmark is a name on the dataset, tank/vm-db#2026-11-04, that records when a snapshot was created. Create it from a snapshot, or from another bookmark, with zfs bookmark tank/vm-db@2026-11-04 tank/vm-db#2026-11-04.

@ is the snapshot. # is the bookmark. The short name can match. zfs list -t bookmark -r tank lists them. zfs destroy tank/vm-db#2026-11-04 removes the name and does not free guest data. The bookmark was not holding it.

The pool feature bookmarks has to be enabled. zpool get feature@bookmarks tank is enabled or active. active means a bookmark exists. If it is disabled, zfs bookmark will not create the name. Do not delete dailies on that pool.

A bookmark is not a snapshot. zfs rollback takes a snapshot. You cannot clone a bookmark, and you cannot mount one. zfs rollback -r to an older snapshot destroys bookmarks newer than that snapshot.

Destroying the snapshot does not destroy the bookmark. zfs destroy of a snapshot fails if a clone exists, or if a hold is set. A bookmark is neither, so the daily can be destroyed and the name can stay.

When is the daily safe to destroy?

Not when you create the bookmark. When the replica’s most recent snapshot is the one you are about to delete.

zfs receive of an incremental stream requires the destination dataset to exist, and its most recent snapshot to match the incremental source. The bookmark keeps that identity on the source after the snapshot is gone. The replica still needs the snapshot. A bookmark on the replica does not pass that check. The replica keeps snapshots the source is no longer allowed to expire.

You can create the bookmark as soon as the snapshot exists. You can destroy the snapshot only after the receive has it.

1. Snapshot the dataset: zfs snapshot tank/vm-db@2026-11-05. If the guest needed a freeze, that already finished. A bookmark does not make a torn write consistent. Make application-consistent virtual machine backups.
2. Send that snapshot. The first copy is zfs send tank/vm-db@2026-11-05. Later copies start at the previous bookmark. Who opens the SSH session is separate. The replica pulls, and the production host cannot log in there.
3. On the replica, the last name from zfs list -t snapshot -o name -s creation tank/vm-db has to be tank/vm-db@2026-11-05. A send that exited 0 on the source is not that check. The connection can drop after the source finishes writing and before the receive commits.
4. Bookmark it: zfs bookmark tank/vm-db@2026-11-05 tank/vm-db#2026-11-05.
5. If you do not need that day as a local recovery point, zfs destroy tank/vm-db@2026-11-05.
6. Destroy the previous bookmark, tank/vm-db#2026-11-04, only after this receive has committed. That bookmark was the -i source. The next send uses #2026-11-05.

A hold pins the blocks until you release it. zfs hold send tank/vm-db@2026-11-05 makes zfs destroy return EBUSY. Release it after the receive with zfs release send tank/vm-db@2026-11-05, then bookmark, then destroy. A bookmark does not pin the blocks. Use a hold only while a timer might expire the snapshot during the send.

Weekly and monthly points you still want to clone or roll back to stay as snapshots. Choose backup retention for virtual machines is which points you keep.

What command is the next send?

zfs send -i tank/vm-db#2026-11-04 tank/vm-db@2026-11-05 is the form that accepts a bookmark. The bookmark has to be earlier in that snapshot’s history. The short name #2026-11-04 is legal on the same dataset. Use the full name so the bookmark is on the dataset you mean.

The stream is one delta. It does not name the destroyed dailies. zfs send -I does name intermediate snapshots, and that form takes a snapshot, not a bookmark.

zfs send -R packages children and properties, and its -i source is a snapshot. A bookmark send is one dataset. The disk has to be its own dataset, or this stream is not the whole guest. One dataset per VM is the job, the quota, and the key. Copy the domain XML with the snapshot. It is not in the disk dataset. The catalog is the domain XML saved with the snapshot. zfs bookmark takes one snapshot or one bookmark, so a parent bookmark does not cover a child.

The delta is the blocks written between those two points. The incremental is the pool’s own birth times. If earlier sends of an encrypted dataset were raw, add -w. A raw incremental after a non-raw receive fails, and a non-raw send decrypts on the way out. A raw send replicates ciphertext and leaves the key at home.

What fails?

Destroy the snapshot before the replica has it, and the bookmark cannot put it back. The bookmark is only a later -i source. The recovery point you still have is the last snapshot that arrived. RPO is the time since the last successful send.

Destroy the bookmark the next send still names, with the snapshot already gone, and zfs send -i has no source. You cannot recreate that bookmark from nothing. The next copy is a full send, or an incremental from an older snapshot that still exists on both sides. On a large disk, that full send is the gap you opened by deleting the name.

A snapshot created on the replica between receives becomes the most recent name, and the receive refuses the stream. Do not snapshot that dataset locally. zfs receive -F on a replication stream destroys destination snapshots the sender no longer has. The sender no longer has the dailies. Do not point that flag at the replica.

zfs rollback -r on the source destroys newer bookmarks. After a rollback, snapshot again and start from a point both sides still share.

How do you prove the send?

Read the replica. The new snapshot is there, and it is the last name from zfs list -t snapshot -o name -s creation on that dataset. Then the source may destroy the daily and keep the bookmark. zfs list -t bookmark on the source only shows the name. It does not show that the receive committed.

These scripts build their own fixtures. They do not run zfs, create a pool, or send a stream. The first accepts an incremental source only when that point still exists as a snapshot or a bookmark, the new snapshot still exists, the source point is older, and the replica’s latest guid matches the source point.

#!/usr/bin/env bash
set -euo pipefail

root="$(mktemp -d)"
trap 'rm -rf "$root"' EXIT

write_point() {
  local file="$1"
  local guid="$2"
  local txg="$3"
  mkdir -p "$(dirname "$file")"
  printf 'guid=%s\ntxg=%s\n' "$guid" "$txg" > "$file"
}

field() {
  local file="$1"
  local key="$2"
  local line
  while IFS= read -r line || [[ -n "$line" ]]; do
    if [[ "$line" == "${key}="* ]]; then
      printf '%s\n' "${line#"${key}="}"
      return 0
    fi
  done < "$file"
  printf 'missing %s in %s\n' "$key" "$file" >&2
  return 1
}

seed_ready() {
  local base="$1"
  mkdir -p "${base}/source/snapshots" "${base}/source/bookmarks" "${base}/replica"
  write_point "${base}/source/bookmarks/2026-11-04" 104 4
  write_point "${base}/source/snapshots/2026-11-05" 105 5
  printf 'name=2026-11-04\nguid=104\n' > "${base}/replica/latest"
}

can_incremental() {
  local base="$1"
  local from_kind="$2"
  local from_name="$3"
  local to_name="$4"
  local from_file to_file from_guid from_txg to_txg replica_guid
  to_file="${base}/source/snapshots/${to_name}"
  [[ -f "$to_file" ]] || return 1
  if [[ "$from_kind" == "bookmark" ]]; then
    from_file="${base}/source/bookmarks/${from_name}"
  elif [[ "$from_kind" == "snapshot" ]]; then
    from_file="${base}/source/snapshots/${from_name}"
  else
    return 1
  fi
  [[ -f "$from_file" ]] || return 1
  from_guid="$(field "$from_file" guid)"
  from_txg="$(field "$from_file" txg)"
  to_txg="$(field "$to_file" txg)"
  [[ "$from_txg" -lt "$to_txg" ]] || return 1
  replica_guid="$(field "${base}/replica/latest" guid)"
  [[ "$from_guid" == "$replica_guid" ]] || return 1
}

must_fail() {
  local label="$1"
  shift
  if "$@"; then
    printf 'checker accepted %s\n' "$label" >&2
    exit 1
  fi
}

ready="${root}/ready"
seed_ready "$ready"
can_incremental "$ready" bookmark 2026-11-04 2026-11-05

gone="${root}/snapshot-gone"
seed_ready "$gone"
rm -f "${gone}/source/bookmarks/2026-11-04"
must_fail "a destroyed snapshot with no bookmark" \
  can_incremental "$gone" snapshot 2026-11-04 2026-11-05

missing="${root}/bookmark-missing"
seed_ready "$missing"
rm -f "${missing}/source/bookmarks/2026-11-04"
must_fail "a missing bookmark" \
  can_incremental "$missing" bookmark 2026-11-04 2026-11-05

stale="${root}/replica-stale"
seed_ready "$stale"
printf 'name=2026-11-03\nguid=103\n' > "${stale}/replica/latest"
must_fail "a replica whose latest guid is not the bookmark" \
  can_incremental "$stale" bookmark 2026-11-04 2026-11-05

printf 'Send-source check passed.\n'
printf 'Proves: a bookmark of the replica'\''s latest guid is accepted, and a missing snapshot, a missing bookmark, or a mismatched replica guid is rejected.\n'
printf 'Does not prove: zfs send ran, the bookmark holds blocks, or the replica received a stream.\n'

The second script checks the destroy order. A daily snapshot may be destroyed only when its bookmark exists and the replica’s latest snapshot is that same name and guid. The bookmark the next send still needs may not be destroyed. A bookmark is not a rollback target.

#!/usr/bin/env bash
set -euo pipefail

root="$(mktemp -d)"
trap 'rm -rf "$root"' EXIT

write_point() {
  local file="$1"
  local guid="$2"
  local txg="$3"
  mkdir -p "$(dirname "$file")"
  printf 'guid=%s\ntxg=%s\n' "$guid" "$txg" > "$file"
}

field() {
  local file="$1"
  local key="$2"
  local line
  while IFS= read -r line || [[ -n "$line" ]]; do
    if [[ "$line" == "${key}="* ]]; then
      printf '%s\n' "${line#"${key}="}"
      return 0
    fi
  done < "$file"
  printf 'missing %s in %s\n' "$key" "$file" >&2
  return 1
}

seed_sent() {
  local base="$1"
  mkdir -p "${base}/source/snapshots" "${base}/source/bookmarks" "${base}/replica"
  write_point "${base}/source/snapshots/2026-11-05" 105 5
  write_point "${base}/source/bookmarks/2026-11-05" 105 5
  write_point "${base}/source/bookmarks/2026-11-04" 104 4
  printf 'name=2026-11-05\nguid=105\n' > "${base}/replica/latest"
}

safe_to_destroy_daily() {
  local base="$1"
  local name="$2"
  local snap="${base}/source/snapshots/${name}"
  local mark="${base}/source/bookmarks/${name}"
  local snap_guid mark_guid replica_name replica_guid
  [[ -f "$snap" && -f "$mark" ]] || return 1
  snap_guid="$(field "$snap" guid)"
  mark_guid="$(field "$mark" guid)"
  [[ "$snap_guid" == "$mark_guid" ]] || return 1
  replica_name="$(field "${base}/replica/latest" name)"
  replica_guid="$(field "${base}/replica/latest" guid)"
  [[ "$replica_name" == "$name" && "$replica_guid" == "$snap_guid" ]] || return 1
}

safe_to_destroy_bookmark() {
  local base="$1"
  local old_name="$2"
  local new_name="$3"
  local old_file="${base}/source/bookmarks/${old_name}"
  local new_file="${base}/source/bookmarks/${new_name}"
  local new_guid replica_guid old_txg new_txg
  [[ -f "$old_file" && -f "$new_file" ]] || return 1
  new_guid="$(field "$new_file" guid)"
  replica_guid="$(field "${base}/replica/latest" guid)"
  [[ "$new_guid" == "$replica_guid" ]] || return 1
  old_txg="$(field "$old_file" txg)"
  new_txg="$(field "$new_file" txg)"
  [[ "$old_txg" -lt "$new_txg" ]] || return 1
}

is_rollback_target() {
  local base="$1"
  local kind="$2"
  local name="$3"
  [[ "$kind" == "snapshot" ]] || return 1
  [[ -f "${base}/source/snapshots/${name}" ]]
}

must_fail() {
  local label="$1"
  shift
  if "$@"; then
    printf 'checker accepted %s\n' "$label" >&2
    exit 1
  fi
}

sent="${root}/sent"
seed_sent "$sent"
safe_to_destroy_daily "$sent" 2026-11-05
safe_to_destroy_bookmark "$sent" 2026-11-04 2026-11-05
is_rollback_target "$sent" snapshot 2026-11-05

unsent="${root}/unsent"
seed_sent "$unsent"
printf 'name=2026-11-04\nguid=104\n' > "${unsent}/replica/latest"
must_fail "destroying a daily the replica does not have as latest" \
  safe_to_destroy_daily "$unsent" 2026-11-05
must_fail "destroying the bookmark the next send still needs" \
  safe_to_destroy_bookmark "$unsent" 2026-11-04 2026-11-05

must_fail "rolling back to a bookmark" \
  is_rollback_target "$sent" bookmark 2026-11-05

printf 'Destroy-order check passed.\n'
printf 'Proves: a received daily with a matching bookmark may be destroyed, the previous bookmark may be destroyed only after that receive, and a bookmark is not a rollback target.\n'
printf 'Does not prove: a snapshot was destroyed on a pool, a receive committed, or a guest boots from the replica.\n'

How do you prove a restore?

You do not restore from the bookmark. The blocks are in a snapshot that still exists, on the replica if the source daily is already gone. Receive that snapshot onto storage that is not this pool, put the NIC on a network with no path to production, and boot that guest. Test a virtual machine restore. zfs list -t bookmark proves the name. It does not prove the guest starts.

A clone of a snapshot still on this pool is not that test. Instant recovery is a clone you can promote. Write down the replica snapshot name, the source bookmark, and the time the receive finished. If the replica is in this building, it is not the off-site copy. Keep an off-site virtual machine backup.