I've been finding Windows 10 in FreeBSD's version of VirtualBox unbearably slow, so decided to give bhyve a try. Much of this is taken from vm-bhyve's Quick start guide.
The first steps are fairly straightforward. Install vm-bhyve. You can use pkg for this.
pkg install vm-bhyve bhyve-firmware |
The firmware package is recommended when having a guest, such as Windows, that can use UEFI. It's a meta package that also installs a couple of other uefi bhyve packages.
Create your vm directory. We'll just call it vm for purposes of this tutorial. If using zfs then use zfs create pool/vm; if using UFS, then mkdir /vm can be used. Now that we have our main directory, we can add these lines to /etc/rc.conf. (In this case, let's assume we used UFS, and the directory is under /vm).
vm_enable="YES" vm_dir="/vm" |
If on ZFS, substitute an existing pool for "pool". For example, on a default FreeBSD system on ZFS, my only existing pool is zroot, so I'd use zfs create zfsroot/vm. The directory path line in /etc/rc.conf would be
vm_dir="zfs:zroot/vm" |
(This seems to have changed--in 14.2-RELEASE it's /zroot/vm)
By the way, one can copy a vm installed on UFS to a vm-bhyve install one zfs. Once you've completed a vm-bhyve guest install, say of a vm called debian, set up permissions, do vm create debian on the host with zfs, then just do rsync -avz vm/debian/ newhost:zroot/vm/debian/ or whatever path it is. The Vm will work on either filesystem. The exception is something like a distro mentioned below that won't install on an AMD sytem--if you copy it from an Intel system, as mentioned below it won't boot properly, regardless of the filesystem used.
Now run vm init. This will create the required subdirectories under /vm. Next we want the templates. We can just get the Windows one, but we may as well get all the sample ones that were installed when we install vm-bhyve.
cp /usr/local/share/examples/vm-bhyve/* /vm/.templates |
Note the dot in front of .templates, it's a hidden file.
Now we have to create a switch which the vm will use for networking. We'll keep this as simple. The switch will be called public and attached to our main network card. Say the main card connecting you to the Internet is em0.
vm switch create public vm switch add public em0 |
We need a Windows10 (or whatever--in this case, we'll assume Windows-10) iso. You can put it where you like, for convenience, I would put it in the same directory that you have for the vms. We'll assume it's called Windows-10.iso.
We can now begin to create the guest. The -t refers to template so in this case, we use -t windows. The -s refers to size. The default size is 20G for Windows, but we'll make it 40G. You can call it whatever you want, in this case, we'll call it winguest.
vm create -t windows -s 40G winguest |
If you wish to reconfigure from the defaults, now run vm configure winguest and a file will open showing the various defaults. For example, memory is set at 2G. You may wish to raise this, or the number of processors which is set to 2.
You can also do this on the command line, with -m for memory and -c for cpu. So if I wanted something with 4G of memory and 4 cpus, my creation line could read
vm create -t windows -s 40G -m 4G -c 4 |
Various templates have various default sizes. For example, the centos7 template, if you don't specify size, will create a disk of 20G.
I like to add a couple of lines to any conf, one for resolution and one for sound. So I might, when doing my vm config winguest, put in this under the graphics="yes" line.
graphics_res="1920x1080" |
(Depending on what you're installing, that may be the maximum available resolution. I know that it is with a FreeBSD guest, using scfb. With vesa I think it's 1024x768.).
I also like to have sound. I got this from vermaden's excellent page on bhyve. It works on my two main workstations, I'm not sure if it has to be altered for various machines.
bhyve_options="-s 8,hda,play=/dev/dsp,rec=/dev/dsp" |
I put that at the end of the configuration file.
Once created, we can begin the installation. We can use a vncviewer, e.g., remmina, tightvnc, or anything else you prefer.
vm install winguest /Windows-10.iso
You will see a message that it has booted. Now to see what's going on, you will need a vncviewer. It uses port 5900, so if you were using tightvnc's vncviewer the command would be vncviewer localhost:5900, or vncviewer localhost:0. Either should work. This should take you to the Windows install. It will reboot once or twice.
When complete, it reboots again. (You probably have to reconnect the vncviwer each time you reboot). This time it will go through setup, asking location, choice of keyboard, and so on. When complete, you should have a working Windows 10 installation.
In the future, when you want to run it, you can start it with the command vm start winguest. You will need to run a vncviewer on port 5900 to view its screen. I have found it to run far more smoothly than Windows in FreeBSD's VirtualBox. There are various improvements you can make, such as changing the default e1000 virtual NIC with virtIO, but as I only need Windows for a few quick things, I haven't looked into that.
Speaking of VirtualBox, you can have it as a host on the same machine as vm-bhyve. However, you can't run them simultaneously. Before starting a VirtualBox machine, you have to first unload the vmm module as root or with root privilege.
kldunload vmm |
Once VirtualBox has stopped, you should be able to once again run any vm-bhyve machine without having to load modules. If the VirtualBox machine has used bridged networking, I have sometimes, but not consistently found, that I had to run service vboxnet stop before being able to reach the bhyve vm.
It can also be run with Linux VMs. RedHat and clones are installed in the same way, but for type use centos7.
vm create -t centos7 -s 20G redhat9 |
Make the size whatever you wish but use centos7 for type, as long as it's RHEL7 or later. It works fine with RHEL9, Rocky, or other RH types. (I'll get to Fedora in a bit).
However, most other versions of Linux need you to run vm config <machine_name> to change their defaults. There is a wiki which has a guest example section. I found that for a Debian install, if I wanted a gui, I had the change the Debian default of loader="grub" to loader="uefi" and add a line graphics="yes". Otherwise, the built in vnc server wouldn't work, although I could do an install by using vm console debian. In contrast, the console doesn't work with a RH install that boots into a GUI.
Also see below about bhyve on an AMD machine. Many versions of Linux may not install on AMD.
Apparently the VNC only working with uefi is something that most people know. I'm sure it's in the docs somewhere, but I've missed it. Basically, I've found that if you want to run any Linux with a GUI, the best thing to do is use the centos7 template, increasing the memory if the boot from the install image doesn't work. For example, in my experience, if creating a RedHat or clone server, I might need 2048 MB of memory--this wasn't consistent, e.g., it was true with genuine RH but not with AlmaLinux. For RHEL9 and clones, I used the centos7 template which defaults to 512 MB of memory.
I also found (I think this this is in the documentation) that with uefi booting installs, it only takes up to 3 additional disks. With grub or other boot methods, you don't have this limitation. However, as of May, 2023, I have been able to add more than three disks to a Rocky9 bhyve install. Note that once I added a disk, the connected interface name would change. For example with one disk, it was enp0s3, if I added one more disk, it would change to enp0s4, etc. So, if adding a disk, at least to a uefi boot, I recommend connecting afterwards with a vncviewer and checking if you need to alter the network configuration.
As it gets more mature, I'm sure there will be more documentation and more methods like as vm-bhyve, to handle bhyve virtual machines.
In September of 2023, an issue arose with bhyve machines that use uefi, which would include Windows and RedHat bhyve VM's. The machines would fail to start. With a Windows VM it one error, not a BSOD, but an error showing a problem, with RH, it seemed to be starting but never reached the login screen. The solution as given in this freebsdforums thread by user vermaden in post #7. With vm-bhyve, just run vm config <vm_name> and at the end of the file add
bhyve_options="-A" |
Interested readers can investigate the links in the thread for further information, but the solution was to add that -A to the config options of your vm. This is necessary with Windows, RH, and RH clones, but I haven't needed it with Fedora or Arch.
I have also found this unnecessary in recent VMs of RH and clones, as of November, 2024, but leave it here in case it arises again. In January, 2024, I went to install Fedora, and found that the mouse wouldn't work with the GUI install. I also tried installing Arch, and found that once I installed X and ran startx, I had no keyboard or mouse input. I asked on freebsdforums and user _al pointed me to a thread with a solution. That thread had done a bit more, but in my case, all I had to do was comment out the line that read xhci_mouse="yes" in the default centos7 template that I was using and mouse and keyboard worked in these Linux installs with X.
Hopefully, this will get fixed soon. See the FreeBSD bug report. There's already a working patch to fix it that should be in packages soon.
This also, as of November, 2024, is usually no longer necessary. The config can have the xhci_mouse="yes" line, without it causing a problem. The only exception I've found is if running a FreeBSD bhyve guest, which seems to still require that xhci is commented out, as well as having to add, assuming that it's being booted with uefi, a /usr/local/etc/X11/xorg.conf.d entry. I just call mine card.conf, but you can call it anything ending with conf.
Section "Device"
Identifier "card0"
Driver "scfb"
|
An alternative that also worked for me is supplying the BusID. (Both of these tips were given me on FreeBSD forums, by users T-Daemon and Emiron). First get the bus id with
pciconf -l |grep vga |
In my case it comes back as
pci0@pci0:0:6:0: |
so, my xorg.conf.d entry can read
Section "Device"
Identifier "card0"
BusID "pci0:0:6:0"
|
Both of these worked for me.
With some distributions when one uses uefi boot, I've had other issues. The problem is that if you want a GUI, you have to choose UEFI. What will happen is, once everything's installed, the reboot may go smoothly, but if you then power down the vm, the next time you start it you'll see something like
BdsDxe: failed to load Boot0001 "UEFI BHYVE SATA DISK BHYVE-48FF-992B-D5E0" from PciRoot(0x0)/Pci(0x4,0x0)/Sata(0x0,0xFFFF,0x0): Not Found >>Start PXE over IPv4. |
Most of what I have here, is taken from an article from davidschlatchter.com
It can't find the EFI bootloader so it's trying to boot over the network. Now, if you wait, and you may be waiting for a couple of minutes, eventually, it goes to an EFI shell. At that shell's prompt, you type
FS0: |
If your fonts don't make that clear it's F,S, zero, and a colon. Then your screen should show FS0:\> and at that point, various commands such as ls will work. You're looking for where the grubx64.efi file is. In the davidsclachter.com page, he's using Debian and it's in EFI/debian. Once you've found the grubx64.efi, you can cd into its directory, type grubx64.efi and it will boot. That page gives suggestions for fixing it on a Debian system (also works for Devuan), just run
grub-install --bootloader-id=boot |
which will install it into EFI/boot instead of EFI/debian. The article also mentions that you will have to rename the grubx64.efi file to bootx64.efi. (The article says you'll probably have to rename it, but there's no probably about it). Rename it.
With Fedora, and other RH based distributions, it just works, you don't have to worry about EFI installs not booting.
Back to ArchLinux and others not booting, one can modify the solution from davidschlachter.com. If, when installing grub on Arch, if one follows the wiki, they suggest the command
grub-install --target=x86_64-efi --efi-directory=esp --bootloader-id=GRUB |
with esp being the directory you're installing into, usually /boot. The wiki does point out that the bootloader-id of GRUB can be changed. So, even if you've already installed grub, you can reinstall grub with
grub-install --target=x86_64-efi --efi-directory=/boot --bootloader-id=boot |
This puts the efi file in /boot/EFI/boot. Then, as with Debian and Devuan, rename grubx64.efi to bootx64.efi and it will boot without problems. I only installed Arch to play around with it a bit, so I don't know how one deals with kernel updates, I imagine you might have to just reinstall grub.
Alpine Linux also works well, but not with their suggested templates. Rather than use grub as the bootloader, choose uefi. If you do this, you don't have to worry about the various grub entries in the default template. In this example, I'm using 4G of memory but you can use whatever you want. It's a very light system.
loader="uefi" cpu=1 memory=4096M network0_type="virtio-net" network0_switch="public" disk0_type="virtio-blk" disk0_name="disk0.img" |
The various Alpine configs on the vm-bhyve wiki use grub, and some other people have mentioned using them, changing the word "vanilla" in the config to "lts". However, when I tried that, it didn't work. The config I list above is thanks to user ArgentoSoma on the FreeBSD forums. With this config, I was able to install with the console--note that though it says uefi it doesn't have the typical line for a GUI, the graphics="yes" line. Once installed, I could either access it with sudo vm console alpine, or just by ssh'ing into it.
I have found that with some linux distributions, on my SER5 Beelink, that the GUI install will crash shortly after starting. In Anaconda using distros, (RH, Fedora, Alma, Rocky) it gives a message that the pane is dead (Anaconda uses tmux). In MX Linux, it just froze while starting the install. I haven't had this happen on my other main workstation which uses Intel. The Beelink is an AMD machine. In such cases, the only solution I've found so far for RH and clones is to go down to Alma 9.4 and upgrade, but the problem seems to occur in all later versions that use Anaconda, including Fedora.
It doesn't even work with installing with reduced graphics or text mode.
As of February 2025, I was able to use the boot.iso with Alma, however, it didn't work with Rocky-9.4.
I've had no trouble with LinuxMint. This seems to only be on AMD machines, they all install without a problem on my Intel workstation.
The relevant bug seems to possibly be in glibc. See bugs.freebsd.org to follow it.
There is also a bug on AlmaLinux, a RedHat clone, dealing with it. As mentioned elsewhere on this page, it's working with Alma Kitten and CentOS Stream, though not with RHEL10 Beta.
On the other hand Arch and Mint VMs worked without problems for me, so I don't know. Currently, on the SER5, I also run VirtualBox (VBox-7 is available in ports) so for RH systems, that's what I'm using.
Out of the distributions I've tested, I had problems on RH and clones, as well as Fedora, MX Linux, and Zorin, based on Ubuntu, I think, that can install on ZFS. As mentioned, Mint and Arch installed without issue, and so far, those are the only ones I've tried. Zorin and MX just freeze after beginning the install, the Anaconda ones give that pane dead message shortly after booting, before gettin to the choose language screen.
I also tried creating a VM on an Intel and then transferring it to the AMD machine, but it booted with a bunch of messages about failed services and hung on trying to start the Accounting Service. (This was with Rocky 9.4,the only one that I tried.
These are the onews that I tried to install that didn't work with a uefi template, either freezing, or having an obvious crash.
With OpenBSD if one goes to vm-bhve's default templates you'll see that it's been changed from booting with grub to booting with uefi. (As of January, 2024, the default OpenBSD template that comes with the vm-bhyve package uses grub). So, go get the new template and use that. My template reads
loader="uefi" cpu=1 memory=2048M network0_type="virtio-net" network0_switch="public" disk0_type="nvme" disk0_name="disk0.img" graphics="yes" uuid="45ac0f0a-bad7-11ee-abcc-b8ca3abc453f" network0_mac="58:9c:fc:0f:1f:f7" |
I do keep the graphics="yes" stuff in though, as the vncviewer method of install works well for me. Some have had success with just using vm console, and if I figure it out, I'll post it.
When it asks if you want to use the whole disk, GPT, or OpenBSD area, choose the default of gtp. Otherwise when it's booted up after install, it won't be able to find the efi file and won't boot.
Also, when finished installing shutdown the vm and then start it again with vm start, rather than rebooting, as a reboot takes you back to the installation. Unfortunately, the only way I've found to do this is vm poweroff openbsd;vm start openbsd. However, before doing this choose S for shell and do
halt -p |
While that won't turn the vm off, at least it puts everything in a state where it's unlikely to be damaged by vm poweroff.
When I do an install, I usually choose to include xbase74.tgz as otherwise I find I'm missing something when I go to install some scanning tools. If you don't want some, or any of them, the way to remove selected sets is to type -<setname>, for example, after it lists the sets, to get rid of xbase, type in
-xbash74.tgz |
(That information is on the console screen that lists the sets). You can do that for all the sets beginning with an X. Then, you can either use ssh or use vm console. (You can use vm console even if you include all xsets. You don't have to use vncviewer or a GUI).
During the install, there's a point where it asks if you want com0 to be the default console, with a default of no. Choose yes, and you can accept the default of 9600. This way, if you need to use vm console, it will work properly.
I'll mention this last part in case someone runs into it. Recently, I did a fresh install, leaving my VMs in place on a different hard drive. I then installed vm-bhyve and saw that all my machines were there, but none of them had a network. So I just destroyed and recreated the public switch with vm switch destroy public and vm switch create public; vm switch add public em0, and all was well again. I don't think it's a typical situation,but perhaps the reader may find themself with the same problem.