Preamble
Over my December 2020 break, I spent a bit of time transitioning my hosting from an older SoYouStart server to a newer one - less disk but it was SSD, but double the RAM and faster CPUs. Since I host a Killing Floor 2 server for myself and some friends to use, I figured I would transition that from my old VirtualBox setup (screw you, Oracle) to bhyve. I'd first heard of bhyve nearly immediately after I set up the VBox in 2018 (I told a friend, he immediately said "why didn't you use bhyve?") and finally took the time to get into it. I also transitioned from a UFS2 filesystem to ZFS on root. ZFS isn't necessarily ideal for a server setup like mine - the SYS install template for FreeBSD ZFS really wants you to use their RAID controller against a single RAID1 block device. You can break that RAID1 apart but then the template does some silly things. I could fix that with a custom install but then they want like $30 for a single day's worth of console access, and it's just not worth that much to me.
After a lot of false starts with some of the things meant to make things easier - cbsd, vm-bhyve chief amongst them - I went back to plain old bhyve, right from the FreeBSD Handbook. Once I got things mostly working on my SYS server, I replicated them on a little i7 PC that occupies some floor space behind me and found a bunch of mistakes. What follows is what I think will work if I follow it a third time. :) It results in a fairly minimal Debian 10.7 running on a network internal to the host+guests, which can NAT out to the Internet (and have ports redirected into it).
Assumptions
- ZFS on root, or at least, an available ZFS filesystem. Using ZFS has lots of advantages and tradeoffs, but I was interested in more easily being able to manage disk usage, on the fly compression, and using ZFS snapshots to handle OS upgrades. The main difference here if you're on UFS will be how you handle creating hard disk images to back your VM - it will use a file instead of a ZFS volume.
- A single public Internet IPv4 address available to the host. You don't have to NAT but SYS charges a good amount of money for extra v4 addresses, and I just don't need them. For my VM I use 192.168.100.0/24, with the bridge device getting the .1 and starting with .11 for the VM guest. Your host can have ipv6 - mine does - but I didn't bother with that for the guest.
- pf as the system firewall. I started in the BSDs with OpenBSD back in 1999 and got used to ipf. FreeBSD's pf isn't exactly OpenBSD's, any more, so lots of pf documentation on the net may or may not work for you. But for this, the needs are very basic. You can probably substitute in ipfw if that's what you like.
- Device names are embedded in a few places. My SYS system has an igb driver, my home system uses em. Don't do what I do when I was re-testing my documentation and leave in some igb0 references, if you need em0 or some other device.
- I don't bother with sudo in this documentation. I typically sudo -i before doing any of this stuff. You may be more paranoid than I am.
This document does not cover automatically start/stopping the guest at boot/reboot. I'll cover that later (once I've figured it out).
Howto (more or less)
First, you will need to install a few packages: bhyve-firmware grub2-bhyve
The firmware package should handle some UEFI firmwares as well, which I don't use.
kldload vmm
kldload nmdm
ifconfig tap0 create
ifconfig bridge0 create
ifconfig bridge0 192.168.100.1/24 up
ifconfig bridge0 addm em0 addm tap0
This loads the vmm device (needed for VMs), nmdm device (needed for null modems, used later for the console), a tap0 interface (for I'm not sure why), and a bridge0 device (essentially, your gateway for your NAT network).
sysctl net.link.tap.up_on_open=1
sysctl net.inet.ip.forwarding=1
This sets things up so the tap interface automatically comes up when it's opened, and allows IP forwarding to happen.
You will also need to set things up in /boot/loader.conf:
if_bridge_load="YES"
if_tap_load="YES"
nmdm_load="YES"
vmm_load="YES"
and /etc/sysctl.conf:
net.link.tap.up_on_open=1
net.inet.ip.forwarding=1
I wrote this little blob as well:
bash
cat << EOM >> /etc/rc.conf
# bhyve VMs etc
cloned_interfaces="bridge0 tap0"
ifconfig_bridge0="addm tap0 addm em0"
vmm_load="YES"
kld_list="nmdm vmm"
ifconfig_bridge0="inet 192.168.100.1 netmask 255.255.255.0 broadcast 192.168.100.255"
EOM
If all goes well, you should see something like this in ifconfig -a:
[root@monkey ~]# ifconfig -a
em0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=812099<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER>
ether 90:b1:1c:74:0b:62
inet 192.168.0.99 netmask 0xffffff00 broadcast 192.168.0.255
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
inet 127.0.0.1 netmask 0xff000000
groups: lo
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=141<UP,RUNNING,PROMISC> metric 0 mtu 33160
groups: pflog
tap0: flags=8902<BROADCAST,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=80000<LINKSTATE>
ether 58:9c:fc:10:be:5c
groups: tap
media: Ethernet autoselect
status: no carrier
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether 02:99:94:b7:b1:00
inet 192.168.100.1 netmask 0xffffff00 broadcast 192.168.100.255
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto stp-rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 4 priority 128 path cost 2000000
member: em0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 1 priority 128 path cost 20000
groups: bridge
nd6 options=9<PERFORMNUD,IFDISABLED>
[root@monkey ~]#
I used the following barebones pf.conf. Order matters with pf, so if you have existing rules, then the nat rule needs to come before any rdr or block rules.
set skip on lo0
ExtIF="em0"
vmnet="192.168.100.0/24"
scrub in all
scrub out all
nat on $ExtIF from $vmnet to any -> ($ExtIF)
# rdr rules go after this line
block in log on $ExtIF from any to any
block out log on $ExtIF from any to any
# This needs to be last
pass out quick on $ExtIF inet from $ExtIF to any modulate state
Next, make a dataset for the VMs:
zfs create zroot/bhyve
and a 60GB volume inside of it for your Debian guest:
zfs create -V60G -o volmode=dev zroot/bhyve/debian107
zfs should look something like this when you're done:
zroot/ROOT 2.20G 379G 96K none
zroot/ROOT/default 2.20G 379G 2.20G /
zroot/bhyve 61.9G 379G 96K /zroot/bhyve
zroot/bhyve/debian107 61.9G 440G 56K -
Next, get yourself a Debian ISO:
cd /zroot/bhyve
fetch https://mirror.csclub.uwaterloo.ca/debian-cd/10.7.0/amd64/iso-cd/debian-10.7.0-amd64-netinst.iso
The fact it's a net install will make it really obvious if your networking is broken. It will be frustrating if you missed a step above, but at least you won't waste your time post-install wondering what's busted. It's your networking. :)
Next, make a map file for GRUB:
cat << EOF >> debian.map
(hd0) /dev/zvol/zroot/bhyve/debian107
(cd0) /zroot/bhyve/debian-10.7.0-amd64-netinst.iso
EOF
This tells grub about the two devices you'll need, the hard disk volume and your ISO file.
Next, start GRUB:
grub-bhyve -m /zroot/bhyve/debian.map -r cd0 -M 8192M debian107
This will put you into an installer menu, from which you want to pick Install. It will throw you back to your FreeBSD shell afterwards, this is normal; all you've done is start the bootloader process.
Next, tell bhyve about it:
bhyve -A -c 2 -m 8192 -H -s 0:0,hostbridge -s1:0,lpc -s 2:0,virtio-net,tap0 -s3:0,virtio-blk,/dev/zvol/zroot/bhyve/debian107 -s4:0,ahci-cd,/zroot/bhyve/debian-10.7.0-amd64-netinst.iso -l com1,stdio debian107
Breakdown of this command:
-A ACPI tables - found out the hard way this is _critical_ on a 64 bit system
-c 2 2 CPUs
-m 8192 8GB RAM
-H see bhyve manual
-s 0:0,hostbridge
-s1:0,lpc
-s 2:0,virtio-net,tap0
-s 3:0,virtio-blketc Set up a virtio device on that path
-s 4:0,ahci-cdpath Set up a CD device on that path
-l com1,stdio Redirect serial console to stdio
debian107 Name the thing
If you're on a busier system, you might consider pinning processes to CPUs (-p). This will take a while to boot, but you should eventually get to an installer menu. DHCP and such will fail, you can hurry them along with cancel.
Use 192.168.100.11 for IP, 255.255.255.0 netmask, etc. Unless you hate yourself, use regular partitioning, no LVM.
If you don't know how to do the rest, I don't know how to help you.
Once the installer finishes, it will halt the VM, at which point you can destroy the VM:
bhyvectl destroy --vm=debian107
Note that this command will set up the serial console on stdio. This can be awkward if you're not in a screen and your network connection drops, but it's fine to run this in screen. Alternatively, you can use the null modem device.
Now that the install is done, you need to re-start grub with a different root device:
grub-bhyve -m /zroot/bhyve/debian.map -r hd0,msdos1 -M 8192 debian107
The next command elides the CD device, you don't need it any more, and also changes the console to a null modem device instead of stdio.
bhyve -A -c 2 -m 8192 -H -s 0:0,hostbridge -s1:0,lpc -s 2:0,virtio-net,tap0 -s3:0,virtio-blk,/dev/zvol/zroot/bhyve/debian107 -l com1,/dev/nmdm0A debian107
If you use the null modem, cu will connect to it:
cu -l /dev/nmdm0B
at which point \~^D (tilde, then control-D) will detach.
As I mentioned, this setup will not handle autostart/shutdown. Make sure you halt -p the guest before you reboot the host.
pf hints: if you want to redirect a port from the outside to your VM, add an rdr rule like this:
rdr pass on $ExtIF proto tcp from any to any port 1234 -> 192.168.100.11 port 1234
or any ports you like. If you like filling logs on the guest, you can even redirect, say, 20222 on the host to 22 on the guest, and then log in directly from "outside".
I found the following links useful in building out my knowledge of bhyve:
Also a copy of Michael W. Lucas's Absolute FreeBSD (third edition) is always a handy reference. I taught myself the very rudiments of ZFS from the relevant chapter there, although Lucas does not pretend that it's comprehensive. The same author's Jails book also mentions that you can run Linux in a jail instead of needing bhyve, and one of the things in the FreeBSD 12.2 release notes mentioned better support for this, but that's another post. I think a jail might be a bit "thinner" than a full-on bhyve guest but I have plenty of resources, so I don't care so much.