Ivan Tomica

Replacing a broken drive on FreeBSD ZFS

These days, one of the drives on one of my machines started developing bad habits of reallocating blocks which is an early sign that drive might go bad. To prevent such misfortune I decided to replace it with a new one.

Once the old drive was yanked out and new one was put into the machine (there are other ways, but I was forced to do it this way) it was only left to partition the new drive and add it to appropriate places.

First step was to check how the remaining drive was partitioned:

root@ncc-1701:~ # gpart show ada0
=>        40  5860533088  ada0  GPT  (2.7T)
          40        1024     1  freebsd-boot  (512K)
        1064         984        - free -  (492K)
        2048     8388608     2  freebsd-swap  (4.0G)
     8390656  5852141568     3  freebsd-zfs  (2.7T)
  5860532224         904        - free -  (452K)

Situation here is pretty much self-explanatory. The partition /dev/ada1p3 is one of the providers in ZFS mirror (mind that this is situation after everything was done and is here just for illustrative purposes):

root@ncc-1701:~ # zpool list -v                                                                          
zroot       2.72T   132G  2.59T         -     6%     4%  1.00x  ONLINE  -
  mirror    2.72T   132G  2.59T         -     6%     4%
    ada0p3      -      -      -         -      -      -
    ada1p3      -      -      -         -      -      -

OK, now that we know where goes where, let’s partition the new ada1 drive the same way as the ada0 is partitioned:

Create new partition table:

root@ncc-1701:~ # gpart create -s gpt ada1
ada1 created

Create boot partition:

root@ncc-1701:~ # gpart add -b 40 -s 1024 -t freebsd-boot ada1
ada1p1 added

Switch -b here, specifies LBA at which partition starts and -s specifies the size.

Install bootcode to the drive since it is part of a mirror from which server boots:

root@ncc-1701:~ # gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada1
partcode written to ada1p1
bootcode written to ada1

Create swap partition:

root@ncc-1701:~ # gpart add -b 2048 -s 8388608 -t freebsd-swap -l swap ada1
ada1p2 added

Create ZFS storage provider partition:

root@ncc-1701:~ # gpart add -b 8390656 -s 5852141568 -t freebsd-zfs -l disk2 ada1
ada1p3 added

Partition is used as a provider, but this can be any GEOM device. This is just the way FreeBSD does things on default install.

With this done, I got the following situation:

root@ncc-1701:~ # gpart show ada1
=>        40  5860533088  ada1  GPT  (2.7T)
          40        1024     1  freebsd-boot  (512K)
        1064         984        - free -  (492K)
        2048     8388608     2  freebsd-swap  (4.0G)
     8390656  5852141568     3  freebsd-zfs  (2.7T)
  5860532224         904        - free -  (452K)

Exactly like on ada0 :-)

Now all is left to do is to attach the drive back to the ZFS pool again:

root@ncc-1701:~ # zpool attach zroot ada0p3 ada1p3 

To explain this further; ada0p3 was already part of a ZFS mirror VDEV but was “missing a pair”, here, we simply attached ada1p3 to it, making it a healthy mirror again (once resilver process is complete).

Monitoring resilver process can be done this way:

root@ncc-1701:~ # zpool status -v
  pool: zroot
 state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Tue Dec 19 18:50:48 2017
        71.6M scanned out of 132G at 2.11M/s, 17h46m to go
        71.3M resilvered, 0.05% done

        NAME        STATE     READ WRITE CKSUM
        zroot       ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            ada0p3  ONLINE       0     0     0
            ada1p3  ONLINE       0     0     0  (resilvering)

errors: No known data errors

That may take some time, depending on how much data you have :-)

You may notice that there is some random device in that mirror, called 15380924691328095690 or something similar. That is basically identifier of the missing device that was yanked out. In previous step I used zpool attach command to add the device since I’ve already performed:

zpool detach zroot 15380924691328095690

If I didn’t do that, to replace the drive, I would need to use:

zpool replace zroot 15380924691328095690 ada1p3

At this point, after filtering through all the mess I put here, you might wonder; “what about that swap partition”?

  • Nothing, I simply ran swapon -a and it was used again as the swap since my /etc/fstab already had it specified by the name (ada1p2).
Tagged in:,
Sysadmin on the everlasting journey of learning. Always in search for an opportunity to prove myself and to learn something new. My addiction is learning and my main goal is to excel in every aspect of Linux/Unix system administration.


Leave a Reply

Your email address will not be published. Required fields are marked *