Encrypted root on ZFS FreeBSD with an SSH unlock rescue system

2026-11-19 - How I install FreeBSD nowadays
Tag: FreeBSD

Introduction

This article explains how I setup FreeBSD systems nowadays. My use case is mostly to have a FreeBSD rescue handy at cheap VPS providers that do not natively support FreeBSD, and protect my data against providers that might not properly erase disks when I stop using a VPS. Remote unlock over SSH is for convenience.

This is achieved by creating a small unencrypted ZFS pool that holds /boot and a minimal FreeBSD system that will be called the outerbase in this article. On boot, I SSH into it and unlock the GELI partition of the main system and then perform a soft reboot into it.

Bootstrap the FreeBSD installer on a local virtual machine

Create the RAW hard drive image for your virtual machine. Using the minimal necessary size will speed up later image transfer if it needs uploading to some VPS or cloud provider. At the time of this writing, a minimal system seems to fit in about 4GB, but feel free to use a bigger image if you do not intend to transfer it later.

qemu-img create -f raw ~/freebsd.raw 4G

Download the installer image from an official mirror then start up the installer in the virtual machine.

I start a UEFI virtual machine with something like:

qemu-system-x86_64 \
    -drive if=none,id=disk,file=$HOME/freebsd.raw,format=raw,cache=writeback \
    -bios /usr/share/edk2/OvmfX64/OVMF_CODE.fd \
    -cpu host -smp 2 -m 4096 \
    -nic user,model=virtio-net-pci,hostfwd=tcp::10022-:22,hostfwd=tcp::10023-:2222 \
    -device virtio-blk-pci,drive=disk \
    -device virtio-serial-pci \
    -device virtserialport,chardev=spicechannel0,name=com.redhat.spice.0 \
    -vga qxl -spice port=5902,addr=127.0.0.1,disable-ticketing=on \
    -chardev spicevmc,id=spicechannel0,name=vdagent \
    -boot d -machine type=q35,accel=kvm \
    -cdrom $HOME/Downloads/FreeBSD-15.1-RELEASE-amd64-bootonly.iso

If you are short on memory, tune down the -m 4096 flag that configures the amount allocated to the virtual machine.

This virtual machine starts up with a SPICE display device, which I like better than VNC, and can be accessed with a SPICE client like spicy. If you would rather use VNC instead, replace the lines mentioning SPICE with the following to start a VNC server on port 5900:

    -display vnc=127.0.0.1:0 \
    -vga none -device virtio-vga,edid=on,xres=2560,yres=1440

Select Shell on the first installer menu screen, then set up the network however is appropriate in your situation. At this point I like to set up OpenSSH so that I can copy and paste at my leisure in a true terminal instead of a SPICE or VNC one.

Prepare the disk

Partition the disk with:

DISK=/dev/vtbd0
gpart create -s gpt $DISK

If you are preparing the disk for a UEFI VPS provider, continue the partitioning process with:

gpart add -a 1m -s 260m -t efi         -l efi    $DISK   # p1 EFI
gpart add -a 1m -s 1g   -t freebsd-zfs -l unlock $DISK   # p2 unencrypted unlocker
gpart add -a 1m         -t freebsd-zfs -l zroot  $DISK   # p3 encrypted root
newfs_msdos -F 32 -c 1 ${DISK}p1
mount_msdosfs ${DISK}p1 /mnt
mkdir -p /mnt/EFI/BOOT
cp /boot/loader.efi /mnt/EFI/BOOT/BOOTX64.EFI
umount /mnt

If you are preparing for a BIOS VPS provider instead, continue with:

gpart add -a 4k -s 512k -t freebsd-boot -l boot   $DISK   # p1 gptzfsboot
gpart add -a 1m -s 1g   -t freebsd-zfs  -l unlock $DISK   # p2 unencrypted unlocker
gpart add -a 1m         -t freebsd-zfs  -l zroot  $DISK   # p3 encrypted root
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 $DISK

Then create the encrypted space with:

geli init -e AES-XTS -l 256 -s 4096 /dev/gpt/zroot
geli attach /dev/gpt/zroot

Now we prepare ZFS for the main system:

zpool create \
      -O acltype=posixacl \
      -O atime=off \
      -O compression=zstd-fast \
      -O mountpoint=none \
      -O relatime=on \
      -O xattr=sa \
      -o altroot=/mnt \
      -o ashift=12 \
      -o autoexpand=on \
      -o autotrim=on \
      -m none zroot /dev/gpt/zroot.eli
zfs create zroot/system
zfs create zroot/system/ROOT
zfs create -o mountpoint=/ -o canmount=noauto zroot/system/ROOT/default
zfs mount zroot/system/ROOT/default
zfs create -o mountpoint=/tmp                         -o setuid=off zroot/system/tmp
zfs create -o mountpoint=/usr                                       zroot/system/usr
zfs create -o mountpoint=/usr/local                   -o setuid=off zroot/system/usr/local
zfs create -o mountpoint=/usr/ports                   -o setuid=off zroot/system/usr/ports
zfs create -o mountpoint=/usr/src                     -o setuid=off zroot/system/usr/src
zfs create -o mountpoint=/var             -o exec=off -o setuid=off zroot/system/var
zfs create -o mountpoint=/var/db          -o exec=off -o setuid=off zroot/system/var/db
zfs create -o mountpoint=/var/spool       -o exec=off -o setuid=off zroot/system/var/spool
zfs create -o mountpoint=/var/tmp         -o exec=off -o setuid=off zroot/system/var/tmp
zfs create zroot/data
zfs create -o mountpoint=/home            -o exec=off -o setuid=off zroot/data/home
zfs create -o mountpoint=/home/julien                 -o setuid=off zroot/data/home/julien
zfs create -o mountpoint=/root                        -o setuid=off zroot/data/home/root
zfs create -o canmount=off                                          zroot/data/var
zfs create -o canmount=off                                          zroot/data/var/db
zfs create -o mountpoint=/var/db/postgres -o exec=off -o setuid=off zroot/data/var/db/postgres
zfs create -o mountpoint=/var/log         -o exec=off -o setuid=off zroot/data/var/log
chmod 1777 /mnt/tmp /mnt/var/tmp
zpool set bootfs=zroot/system/ROOT/default zroot

And then we prepare ZFS for the outerbase rescue system:

zpool create \
      -O acltype=posixacl \
      -O atime=off \
      -O compression=zstd-fast \
      -O mountpoint=none \
      -O relatime=on \
      -O xattr=sa \
      -o altroot=/mnt/unlock \
      -o ashift=12 \
      -o autoexpand=on \
      -o autotrim=on \
      -m none broot /dev/gpt/unlock
zfs create -o mountpoint=/ -o canmount=noauto broot/unlock
zpool set bootfs=broot/unlock broot
mkdir -p /mnt/unlock
zfs mount broot/unlock

We want the main system’s /boot to point to the outerbase one, and the symlink needs to be written in such a way that it resolves correctly both during this installation, and from the main system:

mkdir -p /mnt/unlock/boot
ln -s unlock/boot /mnt/boot

With this done, the main system can install its kernel into the outerbase system.

Install FreeBSD

Let’s bootstrap the package manager configurations of both the main and the outerbase systems to use latest instead of quarterly, and enable PkgBase like so:

for R in /mnt /mnt/unlock; do
    mkdir -p $R/etc/pkg $R/usr/local/etc/pkg/repos/
    sed -e s/quarterly/latest/g \
        -e 's/enabled: no/enabled: yes/g' \
        /etc/pkg/FreeBSD.conf > $R/usr/local/etc/pkg/repos/FreeBSD.conf
    mkdir -p $R/usr/share/keys
    cp -R /usr/share/keys/* $R/usr/share/keys/
done

We can then install the main system with:

export ASSUME_ALWAYS_YES=yes
export TMPDIR=/mnt/tmp
pkg -r /mnt -R /mnt/usr/local/etc/pkg/repos update
pkg -r /mnt -R /mnt/usr/local/etc/pkg/repos install \
    FreeBSD-kernel-generic pkg \
    FreeBSD-set-base

We install the outerbase system with:

pkg -r /mnt/unlock -R /mnt/unlock/usr/local/etc/pkg/repos update
pkg -r /mnt/unlock -R /mnt/unlock/usr/local/etc/pkg/repos install \
    FreeBSD-set-minimal \
    FreeBSD-ssh pkg

Let’s generate a ZFS hostid and set it in both systems so that the pools are not considered foreign:

uuidgen > /mnt/etc/hostid
cp /mnt/etc/hostid /mnt/unlock/etc/hostid

Now we generate the SSH authorized_keys and host keys, and set a root password for both systems with:

for R in /mnt /mnt/unlock; do
    mkdir -p -m 700 $R/root/.ssh
    fetch -o $R/root/.ssh/authorized_keys https://git.adyxax.org/adyxax.keys
    chmod 600 $R/root/.ssh/authorized_keys
    mount -t devfs none $R/dev
    chroot $R ssh-keygen -A
    chroot $R passwd
    umount $R/dev
done

We configure the FreeBSD loader with:

cat > /mnt/unlock/boot/loader.conf <<'EOF'
geom_eli_load="YES"
zfs_load="YES"
autoboot_delay="5"
EOF

If like me you want to keep the serial console accessible, add something like this to your loader.conf:

boot_multicons="YES"
boot_serial="YES"
comconsole_speed="115200"
console="comconsole,vidconsole,efi"

The outerbase system does not have much to do, I bootstrap it with:

cat > /mnt/unlock/etc/rc.conf <<'EOF'
cron_enable="NO"
dumpdev="NO"
hostname="unlock"
network_interfaces="vtnet0"
sendmail_enable="NONE"
sshd_enable="YES"
syslogd_flags="-ss"
EOF
cat > /mnt/unlock/etc/start_if.vtnet0 <<'EOF'
dhclient vtnet0
EOF
: > /mnt/unlock/etc/fstab
echo "nameserver 1.1.1.1" > /mnt/unlock/etc/resolv.conf

echo "Port 2222" >> /mnt/unlock/etc/ssh/sshd_config
echo "PermitRootLogin without-password" >> /mnt/unlock/etc/ssh/sshd_config

I use the following small script to unlock the main system from the outerbase one:

mkdir -p /mnt/unlock/usr/local/bin/
cat > /mnt/unlock/usr/local/bin/unlock.sh <<'EOF'
#!/bin/sh
# Attach the encrypted root and reroot into it. Run over ssh or on console.
set -eu
DEV=/dev/gpt/zroot
POOL=zroot

geli status "${DEV##/dev/}.eli" >/dev/null 2>&1 || geli attach "$DEV"

# pick the BE that bectl/zpool bootfs says is active
zpool import -f -N "$POOL"
BE=$(zpool get -H -o value bootfs "$POOL")
zpool export "$POOL"

echo "rerooting into zfs:${BE}"
kenv vfs.root.mountfrom="zfs:${BE}"
exec reboot -r
EOF
chmod 700 /mnt/unlock/usr/local/bin/unlock.sh

The main system configuration has a tiny bit more to it:

cat > /mnt/etc/rc.conf <<'EOF'
clear_tmp_enable="YES"
dumpdev="NO"
hostname="myth"
local_unbound_enable="YES"
moused_nondefault_enable="NO"
network_interfaces="vtnet0"
ntpd_enable="YES"
ntpd_sync_on_start="YES"
sendmail_enable="NONE"
sshd_enable="YES"
syslogd_flags="-ss"
zfs_enable="YES"
EOF

cat > /mnt/etc/start_if.vtnet0 <<'EOF'
dhclient vtnet0
EOF

cat > /mnt/etc/fstab <<'EOF'
# Device          Mountpoint  FStype  Options  Dump  Pass
broot/unlock      /unlock     zfs     rw       0     0
EOF

echo "PermitRootLogin without-password" >> /mnt/etc/ssh/sshd_config

Do any remaining configuration you need, and finish with:

umount /mnt/unlock
zpool export broot
zpool export zroot
halt -p

To test your new installation, update the QEMU command to -boot c instead of d (which points to the installer ISO) then start the virtual machine again. If your kept my hostfwd NIC configuration, you should be able to SSH and unlock your system with:

ssh -p 10023 root@localhost unlock.sh
ssh -p 10022 root@localhost

After copying this minimal image to a VPS provider, you will want to resize the disks with (update the disk to match what your provider exposes):

gpart recover da0
gpart resize -i 3 -a 1m da0
zpool online -e zroot

Conclusion

I have been using an outerbase rescue system on my FreeBSD hosted servers for a few years already, and I really like how tidy this has become thanks to PkgBase!