FreeBSD-10 on a ZFS mirror

With FreeBSD-10, one has the option to install FreeBSD to a ZFS based system. One of the options given at that time is a mirror. Replacing a drive in such a mirror is bit less trivial than I'd expected, so I'm putting up this page in the hope it saves someone some time. During installation, we'll choose GPT, mirroring drives ada0 and ada1. It's a little different than most ZFS replacement because we have to worry about a small boot partition. Mirrors can have slightly different syntax than RAIDZ. Don't use zpool replace. See this thread on FreeBSD forums.

Replacing a failed drive

First run
zpool status zroot

You should see that the state is DEGRADED and get information about which drive is unavailable. It will look something like like
123606844632523523    UNAVAIL   was /dev/gpt/zfs1

First detach that drive. (This works even if a drive has died and been completely removed from the machine, to the point where it is no longer visible in your /dev directory)
zpool detach zroot /dev/gpt/zfs1

Run gpart show to see the layout of the working drive. (This is a drive with no swap partition, not always a good idea, but it's what I had when I first wrote this.)
40    1024     1  freebsd-boot (512K)
1064  209715200 2 freebsd-zfs (100.0G)

Add the new drive. In this case, let's say we're replacing /dev/ada1. It will need to get a GPT scheme.
gpart create -s GPT ada1

Next we add a boot partition, specifying the size of 512k and make the rest zfs. The -l is for label, and will show up under /dev/gpt.
gpart add -b 40 -l gptboot1 -s 512k -t freebsd-boot ada1
gpart add -t freebsd-zfs -l zfs1 ada1

Now that we have the new drive successfully partitioned, we are going to add the zfs partition to our zfs mirror. The syntax is zpool attach poolname existing_dev new_dev. Use the /dev/gpt name for the devices.
zpool attach zroot /dev/gpt/zfs0 /dev/gpt/zfs1

We are just attaching the freebsd-zfs partition. Had there been a swap partition, we wouldn't attach that either, we're just concerned about the one zfs partition. You should see that it is attached and also a message to run gpart bootcode on the newly attached drive. The message gives the syntax (though not the drive) and it's run on the entire ada1 drive, not the partition that you just attached.
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 /dev/ada1
Run zpool status to see if the drive has been resilvered. Once it has, the system can once again boot from either drive.

Adding a Fresh Install

This is probably a rare situation, but as it took me awhile to figure it out, with a friend's help, I thought I'd add it here.

I had a 2 disk mirror running 10.3. I then put in a single disk, and installed 11.0-RC2 to the new disk. In both installs, the 10.3 and the 11.0, I used the default name of zroot.

When I booted into the 11.0 install, it gave the old host name, and had all the packages I'd installed on the 10.3 install. This caused problems, for example, when I wanted to load an NVidia module, and couldn't, because it was seeing a kernel mismatch. However, uname -r showed 11.0, and I had definitely booted the correct disk.

The problem turned out to be that both systems had the same default zroot name for the root partition. The 11.0 installation was pulling in all sorts of files from the 10.3 installation. My original plan had been to copy over various things from the old installation, then change the 11.0 single disk into a mirror, using one of the two disks from the old 10.2 install.

Once I figured out the problem, I just did a quick reinstall, this time calling the new zroot zroot11. I then created a directory for the old zroot, mounted it, and was easily able to copy over what I wanted. (Don't do zpool import zroot, or you'll wind up with the old files mounted all over the place.) So, I did something like
mkdir oldzroot
import -R oldzroot zroot

I was then able to go into oldzroot/home/myuser and copy over various things I wanted. As I used the -R option, to specify a different mount point, the mount was only for that current session, which is what I wanted.