Over my December 2020 break, I spent a bit of time transitioning my hosting from an older SoYouStart server to a newer one - less disk but it was SSD, but double the RAM and faster CPUs. Since I host a Killing Floor 2 server for myself and some friends to use, I figured I would transition that from my old VirtualBox setup (screw you, Oracle) to bhyve. I'd first heard of bhyve nearly immediately after I set up the VBox in 2018 (I told a friend, he immediately said "why didn't you use bhyve?") and finally took the time to get into it. I also transitioned from a UFS2 filesystem to ZFS on root. ZFS isn't necessarily ideal for a server setup like mine - the SYS install template for FreeBSD ZFS really wants you to use their RAID controller against a single RAID1 block device. You can break that RAID1 apart but then the template does some silly things. I could fix that with a custom install but then they want like $30 for a single day's worth of console access, and it's just not worth that much to me.

After a lot of false starts with some of the things meant to make things easier - cbsd, vm-bhyve chief amongst them - I went back to plain old bhyve, right from the FreeBSD Handbook. Once I got things mostly working on my SYS server, I replicated them on a little i7 PC that occupies some floor space behind me and found a bunch of mistakes. What follows is what I think will work if I follow it a third time. :) It results in a fairly minimal Debian 10.7 running on a network internal to the host+guests, which can NAT out to the Internet (and have ports redirected into it).


  • ZFS on root, or at least, an available ZFS filesystem. Using ZFS has lots of advantages and tradeoffs, but I was interested in more easily being able to manage disk usage, on the fly compression, and using ZFS snapshots to handle OS upgrades. The main difference here if you're on UFS will be how you handle creating hard disk images to back your VM - it will use a file instead of a ZFS volume.
  • A single public Internet IPv4 address available to the host. You don't have to NAT but SYS charges a good amount of money for extra v4 addresses, and I just don't need them. For my VM I use, with the bridge device getting the .1 and starting with .11 for the VM guest. Your host can have ipv6 - mine does - but I didn't bother with that for the guest.
  • pf as the system firewall. I started in the BSDs with OpenBSD back in 1999 and got used to ipf. FreeBSD's pf isn't exactly OpenBSD's, any more, so lots of pf documentation on the net may or may not work for you. But for this, the needs are very basic. You can probably substitute in ipfw if that's what you like.
  • Device names are embedded in a few places. My SYS system has an igb driver, my home system uses em. Don't do what I do when I was re-testing my documentation and leave in some igb0 references, if you need em0 or some other device.
  • I don't bother with sudo in this documentation. I typically sudo -i before doing any of this stuff. You may be more paranoid than I am.

This document does not cover automatically start/stopping the guest at boot/reboot. I'll cover that later (once I've figured it out).

Howto (more or less)

First, you will need to install a few packages: bhyve-firmware grub2-bhyve

The firmware package should handle some UEFI firmwares as well, which I don't use.

kldload vmm
kldload nmdm
ifconfig tap0 create
ifconfig bridge0 create
ifconfig bridge0 up
ifconfig bridge0 addm em0 addm tap0

This loads the vmm device (needed for VMs), nmdm device (needed for null modems, used later for the console), a tap0 interface (for I'm not sure why), and a bridge0 device (essentially, your gateway for your NAT network).

sysctl net.inet.ip.forwarding=1

This sets things up so the tap interface automatically comes up when it's opened, and allows IP forwarding to happen.

You will also need to set things up in /boot/loader.conf:


and /etc/sysctl.conf:

I wrote this little blob as well:

cat << EOM >> /etc/rc.conf
# bhyve VMs etc
cloned_interfaces="bridge0 tap0"
ifconfig_bridge0="addm tap0 addm em0"
kld_list="nmdm vmm"
ifconfig_bridge0="inet netmask broadcast"

If all goes well, you should see something like this in ifconfig -a:

[root@monkey ~]# ifconfig -a
    ether 90:b1:1c:74:0b:62
    inet netmask 0xffffff00 broadcast
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
    inet netmask 0xff000000
    groups: lo
pflog0: flags=141<UP,RUNNING,PROMISC> metric 0 mtu 33160
    groups: pflog
tap0: flags=8902<BROADCAST,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 58:9c:fc:10:be:5c
    groups: tap
    media: Ethernet autoselect
    status: no carrier
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 02:99:94:b7:b1:00
    inet netmask 0xffffff00 broadcast
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto stp-rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 4 priority 128 path cost 2000000
            ifmaxaddr 0 port 1 priority 128 path cost 20000
    groups: bridge
    nd6 options=9<PERFORMNUD,IFDISABLED>
[root@monkey ~]#

I used the following barebones pf.conf. Order matters with pf, so if you have existing rules, then the nat rule needs to come before any rdr or block rules.

set skip on lo0
scrub in all
scrub out all
nat on $ExtIF from $vmnet to any -> ($ExtIF)
# rdr rules go after this line
block in log on $ExtIF from any to any
block out log on $ExtIF from any to any
# This needs to be last
pass out quick on $ExtIF inet from $ExtIF to any modulate state

Next, make a dataset for the VMs:

zfs create zroot/bhyve

and a 60GB volume inside of it for your Debian guest:

zfs create -V60G -o volmode=dev zroot/bhyve/debian107

zfs should look something like this when you're done:

zroot/ROOT                               2.20G   379G    96K  none
zroot/ROOT/default                       2.20G   379G  2.20G  /
zroot/bhyve                              61.9G   379G    96K  /zroot/bhyve
zroot/bhyve/debian107                    61.9G   440G    56K  -

Next, get yourself a Debian ISO:

cd /zroot/bhyve

The fact it's a net install will make it really obvious if your networking is broken. It will be frustrating if you missed a step above, but at least you won't waste your time post-install wondering what's busted. It's your networking. :)

Next, make a map file for GRUB:

cat << EOF >>
(hd0) /dev/zvol/zroot/bhyve/debian107
(cd0) /zroot/bhyve/debian-10.7.0-amd64-netinst.iso

This tells grub about the two devices you'll need, the hard disk volume and your ISO file.

Next, start GRUB:

grub-bhyve -m /zroot/bhyve/ -r cd0 -M 8192M debian107

This will put you into an installer menu, from which you want to pick Install. It will throw you back to your FreeBSD shell afterwards, this is normal; all you've done is start the bootloader process.

Next, tell bhyve about it:

bhyve -A -c 2 -m 8192 -H -s 0:0,hostbridge -s1:0,lpc -s 2:0,virtio-net,tap0 -s3:0,virtio-blk,/dev/zvol/zroot/bhyve/debian107 -s4:0,ahci-cd,/zroot/bhyve/debian-10.7.0-amd64-netinst.iso -l com1,stdio debian107

Breakdown of this command:

-A                  ACPI tables - found out the hard way this is _critical_ on a 64 bit system
-c 2                2 CPUs
-m 8192             8GB RAM
-H                  see bhyve manual
-s 0:0,hostbridge   
-s 2:0,virtio-net,tap0
-s 3:0,virtio-blketc Set up a virtio device on that path
-s 4:0,ahci-cdpath  Set up a CD device on that path
-l com1,stdio       Redirect serial console to stdio
debian107           Name the thing

If you're on a busier system, you might consider pinning processes to CPUs (-p). This will take a while to boot, but you should eventually get to an installer menu. DHCP and such will fail, you can hurry them along with cancel.

Use for IP, netmask, etc. Unless you hate yourself, use regular partitioning, no LVM.

If you don't know how to do the rest, I don't know how to help you.

Once the installer finishes, it will halt the VM, at which point you can destroy the VM:

bhyvectl destroy --vm=debian107

Note that this command will set up the serial console on stdio. This can be awkward if you're not in a screen and your network connection drops, but it's fine to run this in screen. Alternatively, you can use the null modem device.

Now that the install is done, you need to re-start grub with a different root device:

grub-bhyve -m /zroot/bhyve/ -r hd0,msdos1 -M 8192 debian107

The next command elides the CD device, you don't need it any more, and also changes the console to a null modem device instead of stdio.

bhyve -A -c 2 -m 8192 -H -s 0:0,hostbridge -s1:0,lpc -s 2:0,virtio-net,tap0 -s3:0,virtio-blk,/dev/zvol/zroot/bhyve/debian107 -l com1,/dev/nmdm0A debian107

If you use the null modem, cu will connect to it:

cu -l /dev/nmdm0B

at which point \~^D (tilde, then control-D) will detach.

As I mentioned, this setup will not handle autostart/shutdown. Make sure you halt -p the guest before you reboot the host.

pf hints: if you want to redirect a port from the outside to your VM, add an rdr rule like this:

rdr pass on $ExtIF proto tcp from any to any port 1234 -> port 1234

or any ports you like. If you like filling logs on the guest, you can even redirect, say, 20222 on the host to 22 on the guest, and then log in directly from "outside".

I found the following links useful in building out my knowledge of bhyve:

Also a copy of Michael W. Lucas's Absolute FreeBSD (third edition) is always a handy reference. I taught myself the very rudiments of ZFS from the relevant chapter there, although Lucas does not pretend that it's comprehensive. The same author's Jails book also mentions that you can run Linux in a jail instead of needing bhyve, and one of the things in the FreeBSD 12.2 release notes mentioned better support for this, but that's another post. I think a jail might be a bit "thinner" than a full-on bhyve guest but I have plenty of resources, so I don't care so much.