Using Windows on FreeBSD's vm-bhyve

I've been finding Windows 10 in FreeBSD's version of VirtualBox unbearably slow, so decided to give bhyve a try. Much of this is taken from vm-bhyve's Quick start guide.

The first steps are fairly straightforward. Install vm-bhyve. You can use pkg for this.
pkg install vm-bhyve bhyve-firmware 

The firmware package is recommended when having a guest, such as Windows, that can use UEFI. It's a meta package that also installs a couple of other uefi bhyve packages.

Create your vm directory. We'll just call it vm for purposes of this tutorial. If using zfs then use zfs create pool/vm if using UFS, then mkdir /vm can be used. Now that we have our main directory, we can add these lines to /etc/rc.conf. (In this case, let's assume we used UFS, and the directory is under /vm).

Now run vm init. This will create the required subdirectories under /vm. Next we want the templates. We can just get the Windows one, but we may as well get all the sample ones that were installed when we install vm-bhyve.
cp /usr/local/share/examples/vm-bhyve/* /vm/.templates

Note the dot in front of .templates, it's a hidden file.

Now we have to create a switch which the vm will use for networking. We'll keep this as simple. The switch will be called public and attached to our main network card. Say the main card connecting you to the Internet is em0.
vm switch create public
vm switch add public em0

We need a Windows10 (or whatever--in this case, we'll assume Windows-10) iso. You can put it where you like, for convenience, I would put it in the same directory that you have for the vms. We'll assume it's called Windows-10.iso.

We can now begin to create the guest. The -t refers to type (I think--it may refer to template), so in this case, we use -t windows. The -s refers to size. The default size is 20G for Windows, but we'll make it 40G. You can call it whatever you want, in this case, we'll call it winguest.
vm create -t windows -s 40G winguest

If you wish to reconfigure from the defaults, now run vm configure winguest and a file will open showing the various defaults. For example, memory is set at 2G. You may wish to raise this, or the number of processors which is set to 2.

Once created, we can begin the installation. We can use a vncviewer, e.g., remmina, tightvnc, or anything else you prefer.

vm install winguest /Windows-10.iso

You will see a message that it has booted. Now to see what's going on, you will need a vncviewer. It uses port 5900, so if you were using tightvnc's vncviewer the command would be vncviewer localhost:5900. This should take you to the Windows install. It will reboot once or twice.

When complete, it reboots again. (You probably have to reconnect the vncviwer each time you reboot). This time it will go through setup, asking location, choice of keyboard, and so on. When complete, you should have a working Windows 10 installation.

In the future, when you want to run it, you can start it with the command vm start winguest. You will need to run a vncviewer on port 5900 to view its screen. I have found it to run far more smoothly than Windows in FreeBSD's VirtualBox. There are various improvements you can make, such as changing the default e1000 virtual NIC with virtIO, but as I only need Windows for a few quick things, I haven't looked into that.

Speaking of VirtualBox, you can have it as a host on the same machine as vm-bhyve. However, you can't run them simultaneously. Before starting a VirtualBox machine, you have to first unload the vmm module as root or with root privilege.
kldunload vmm

Once VirtualBox has stopped, you should be able to once again run any vm-bhyve machine without having to load modules. If the VirtualBox machine has used bridged networking, I have sometimes, but not consistently found, that I had to run service vboxnet stop before being able to reach the bhyve vm.

It can also be run with Linux VMs. RedHat and clones are installed in the same way, but for type use centos7.
vm create -t centos7 -s 20G redhat9

Make the size whatever you wish but use centos7 for type, as long as it's RHEL7 or later. It works fine with RHEL9.

However, most other versions of Linux need you to run vm config <machine_name> to change their defaults. There is a wiki which has a guest example section. I found that for a Debian install, if I wanted a gui, I had the change the Debian default of loader="grub" to loader="uefi" and add a line graphics="yes". Otherwise, the built in vnc server wouldn't work, although I could do an install by using vm console debian. In contrast, the console doesn't work with a RH install that boots into a GUI.

Apparently the VNC only working with uefi is something that most people know. I'm sure it's in the docs somewhere, but I've missed it. A couple of other things that I found from experience were that if creating a RedHat or clone server, I might need 2048 MB of memory--this wasn't consistent, e.g., it was true with genuine RH but not with AlmaLinux. For RHEL9 and clones, I used the centos7 template which defaults to 512 MB of memory.

I also found (I think this this is in the documentation) that with uefi booting installs, it only takes up to 3 additional disks. With grub or other boot methods, you don't have this limitation. However, as of May, 2023, I have been able to add more than three disks to a Rocky9 bhyve install. Note that once I added a disk, the connected interface name would change. For example with one disk, it was enp0s3, if I added one more disk, it would change to enp0s4, etc. So, if adding a disk, at least to a uefi boot, I recommend connecting afterwards with a vncviewer and checking if you need to alter the network configuration.

As it gets more mature, I'm sure there will be more documentation and more methods, such as vm-bhyve, to handle bhyve virtual machines.

In September of 2023, an issue arose with bhyve machines that use uefi, which would include Windows and RedHat bhyve VM's. The machines would fail to start. With a Windows VM it one error, not a BSOD, but an error showing a problem, with RH, it seemed to be starting but never reached the login screen. The solution as given in this freebsdforums thread by user vermaden in post #7. With vm-bhyve, just run vm config <vm_name> and at the end of the file add

Interested readers can investigate the links in the thread for further information, but the solution is to add that -A to the config options of your vm.