Replacing A ZFS Root Disk With Proxmox 5.x

What I will be detailing here is how to replace a root ZFS disk on proxmox, we have all had it where a disk fails, and some systems are simpler than others.

When replacing a root ZFS disk in proxmox it's not as simple as just replacing the disk you also need to partition it and install grub.

First off you will need to make sure that you have a replacement disk the same size or larger, most systems will support hot swap but if yours dosnt then you will need to shutdown the server so do ensure you allow time for this specially if you are running a production server with live VPS/Containers

Ffirst offcheck the status of your pool

zpool status

  pool: rpool
 state: DEGRADED
status: One or more devices has been taken offline by the administrator.
        Sufficient replicas exist for the pool to continue functioning in a
        degraded state.
action: Online the device using 'zpool online' or replace the device with
        'zpool replace'.
  scan: scrub repaired 0 in 3h41m with 0 errors on Sat Aug 10 12:34:45 2017
config:

        NAME        STATE     READ WRITE CKSUM
        rpool       DEGRADED     0     0     0
          raidz1-0  DEGRADED     0     0     0
            sda2    ONLINE       0     0     0
            sdb2    OFFLINE      0     0     0
            sdc2    ONLINE       0     0     0

errors: No known data errors

As you will see on rpool this is degraded and one disk is offline sdb2, if your disk is not offline then you should take it offline before replacing it run zpool offline rpool /dev/sdb2 to take this disk offline.

Once you have the disk ofline you can then replace the disk, if your machine does not support hot swap then shutdown and replace.

After the disk has been replaced check new disk is visible, you can use fdisk -l to list your disks in this instance the new disk will be sdb

1. As proxmox uses GPT we need to copy the partition table layout with the right tool, sgdisk. Copy your partition layout from a good disk to your new disk sdb, here we copy the partition table from sda to sdb

   sgdisk --replicate=/dev/sdb /dev/sda

2. Next we need to randomize the disk guids, this may not always be required but I feal that this is good practice to do it anyway.

   sgdisk --randomize-guids /dev/sdb
   The operation has completed successfully.

3. Install grup to your new disk sdb

   grub-install /dev/sdb 

4. Now you need to resilver/rebuild your ZFS pool, once you start the resilver process it's advised that you don't reboot your machine utill its completed.

   zpool replace rpool /dev/sdb2

You can check the status of the rebuild with zpool status under the scan section it will details the rebuild progress.

 zpool status
  pool: rpool
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Tue Sep  12 19:55:07 2017
    34.6G scanned out of 576G at 17.75M/s, 8h52m to go
    34.6G resilvered, 6.01% done
config:

        NAME                        STATE     READ WRITE CKSUM
        rpool                       DEGRADED     0     0     0
          mirror-0                  DEGRADED     0     0     0
            sda2                    ONLINE       0     0     0
            replacing-1             UNAVAIL      0     0     0
              16461405243654394612  UNAVAIL      0     0     0  was /dev/sdb2/old
              sdb2                  ONLINE       0     0     0  (resilvering)

errors: No known data errors

Hope this help you.