setenv D /jails/templatemkdir -p $D cd /usr/src make buildworld
make installworld DESTDIR=$D make distribution DESTDIR=$D
Thanks to Mark S. at work, who showed me how easy this can be.
IMPORTANT! This was written using FreeBSD-9.2. Some things, specifically mounting of devfs, have issues in FreeBSD-10. See the FreeBSD-10 section for more information.
Using nullfs (somewhat similar to bind-mounts in Linux), one can easily run multiple jails by reusing a base install. The handbook has a page on this, but I prefer my own (well, actually, the aforementioned Mark's) method, which I find far simpler.
We are going to create a template jail, then reuse it to run a web server and a zabbix server. Our host has an address of 192.168.1.50/24 and our two servers will use the addresses of 192.168.1.51 and 192.168.1.52. The creation of the template will be almost identical to creating a single jail, as described on my jail page.
Most of this will have to be done as root or with root privilege. We are going to keep our jails under a directory called, oddly enough, /jails. We will first create our template, then the other two jails, which we'll call www_jail and zabbix_jail. You might be able to skip make buildworld, see below.
setenv D /jails/template |
The tcsh default shell for root on FreeBSD uses setenv to
create a variable. If using a Bourne style shell, e.g., bash or sh, use
D=/jails/template to create the $D variable.
If you've recently run a make buildworld on the system, you may be able to
skip the buildworld part. You can try, after doing a cd to /usr/src,
running make installworld DESTDIR=$D. If it succeeds, great. If it fails,
you'll have to go back and start with buildworld. If it does fail, as
there will be some files already in it, I usually just delete the directory
and recreate it. Depending upon how far the installworld got before
failing, I might have to run chflags, otherwise, I'll be unable to delete
some files. So, if your installworld failed
cd /jails chflags -R noschg template rm -rf template mkdir template |
Then go back to /usr/src/ and start from the make buildworld step above.
Once done, copy over the host's /etc/resolv.conf to the jail.
cp /etc/resolv.conf /jails/template/etc/ |
If you want to add packages, you can now chroot into the jail and do so. For example, if you want the essential tmux
chroot /jails/template pkg_add -r tmux |
If desired, one can set up the template jail to use the now recommended
pkgng.If using FreeBSD-10, pkgng is used by default, and the syntax would be (and will be, if you update your 9.2 system to use pkgng)
pkg install tmux |
While in the jail, edit /etc/rc.conf. I want to enable ssh, put in a reminder to change the hostname on the actual jails, and make sure it's not running sendmail or dumpdev. My template jail's rc.conf will read
hostname="CHANGEME" sshd_enable="YES" sendmail_enable="NO" dumpdev="NO" |
While in the template, set the timezone with
tzsetup |
At this point, we're done with the template, so type exit to get out of it.
Our jails will need to use devfs. While I discuss this more fully on my jails page, in brief, we'll copy over the set of rules pertaining to jails from /etc/defaults/devfs.rules to /etc/devfs.rules. Open the file /etc/devfs.rules (you may have to create it with your favorite text editor) and add
[jail=4] add include $devfsrules_hide_all add include $devfsrules_unhide_basic add include $devfsrules_unhide_login |
If using zfs you may want the add path zfs unhide line from the defaults version, if desired.
Our two jails will each need an ip alias. As mentioned, we are going to use 192.168.1.51 and 192.168.1.52, so we'll create the aliases then add an entry to the host's rc.conf so that they'll be there on reboot. Assuming our ethernet card is em0
ifconfig em0 alias 192.168.1.51/32 ifconfig em0 alias 192.168.1.52/32 |
Add the following to the host's /etc/rc.conf
ifconfig_em0_alias0="inet 192.168.1.51/32" ifconfig_em0_alias1="inet 192.168.1.52/32" |
In the /jails directory, we'll now create two directories, one for the www_jail and a mount point for it.
cd /jails mkdir www_jail _www_jail |
Next we add an entry to /etc/fstab. We are going to mount the template read only, using nullfs, and then mount www_jail on top of it. Add the lines
/jails/template /jails/_www_jail nullfs ro 0 0 /jails/www_jail /jails/_www_jail unionfs rw,noatime 0 0 |
Now add entries for jails and specifically, www_jail to /etc/rc.conf
jail_enable="YES" jail_list="www_jail" jail_www_jail_rootdir="/jails/_www_jail" jail_www_jail_hostname="www_jail" jail_www_jail_ip="192.168.1.51" jail_www_jail_devfs_enable="YES" jail_www_jail_devfs_ruleset="jail" jail_www_jail_exec="/bin/sh /etc/rc" |
Note that the root directory is _www_jail, not www_jail. We can now mount the new jail.
mount -a mount |
The output of the mount command should now include
/jails/template on /jails/_www_jail (nullfs, local, read-only) <above>:/jails/www_jail on /jails/_www_jail (unionfs, local, noatime) |
The <above> indicates that our www_jail has successfully been mounted above the nullfs mount.
With the new jail mounted, we can edit its rc.conf and change the hostname from CHANGEME to www_jail. You may get a notice that a newer copy of the file exists and to hit ! to override, but this can be safely ignored. It seems to happen if the template jail's timezone is set to UTC, and seems to only occur the first time you edit the file. We can then start the jail with
service jail start www_jail |
Running the jls command should show us that the jail is running.
We can enter www_jail via ssh or running chroot /jails/_www_jail and install apache, or other web server, and any other desired programs without affecting the template.
Adding the zabbix_jail is now absurdly simple. We'll create the directories, add the fstab and rc.conf entries, and we'll be done. We've already created its IP alias.
cd /jails mkdir zabbix_jail _zabbix_jail |
Add to /etc/fstab
/jails/template /jails/_zabbix_jail nullfs ro 0 0 /jails/zabbix_jail /jails/_zabbix_jail unionfs rw,noatime 0 0 |
Edit rc.conf. We'll add the zabbix_jail to the jail_list line so that it now reads
jail_list="www_jail zabbix_jail" |
We'll also add lines for the zabbix_jail.
jail_zabbix_jail_rootdir="/jails/_zabbix_jail" jail_zabbix_jail_hostname="zabbix_jail" jail_zabbix_jail_ip="192.168.1.52" jail_zabbix_jail_devfs_enable="YES" jail_zabbix_jail_devfs_ruleset="jail" jail_zabbix_jail_zabbix_jail_parameters="allow.sysvipc=1" jail_zabbix_jail_exec="/bin/sh /etc/rc" |
As explained on both my jail page and my zabbix page, the allow.sysvipc=1 line is necessary for zabbix to work within a jail. (It's also necessary for a jail running postgresql).
We can now mount the zabbix_jail with mount -a, and once again, run mount to make sure that it shows both the template and zabbix_jail are mounted on _zabbix_jail. It should show something like
/jails/template on /jails/_zabbix_jail (nullfs, local, read-only) <above>:/jails/zabbix_jail on /jails/_zabbix_jail (unionfs, local, noatime) |
Now that it's mounted, edit /jails/_zabbix_jail/etc/rc.conf to change the hostname from CHANGEME and the start the jail with service jail start zabbix_jail. To then install zabbix, either ssh or chroot into the jail and install desired programs.
If the reader later decides to add another jail, the process is quick and easy. Create two new directories in /etc/jails/, add the two lines to /etc/fstab (remember that each jail requires mounting template on nullfs), add the necessary lines to /etc/rc.conf, mount the new jail with mount -a, change its hostname in its own /etc/rc.conf and start the jail. If there's a program that you want to all the jails do chroot to template and make the changes. For example
chroot /jails/template pkg install w3m |
Now all the jails will have w3m available.
As the template is not using devfs, the ports tree may not install successfully, giving an error that the snapshot is corrupt. Personally, I've just installed the ports tree in a specific jail if I've needed it. Another option (untested by me) might be to temporarily or permanently mount devfs on /jails/template's /dev.
Another method is to temporarily mount the host's /usr/ports on a specific jail's /usr/ports. On the host
mount -t nullfs -o ro /usr/ports/ /jails/_www_jail/usr/ports |
Then chroot into the jail and add to the jail's /etc/make.conf (or create an /etc/make.conf)
WRKDIRPREFIX=/tmp/ports/work DISTDIR=/tmp/ports/distfiles PACKAGES=/tmp/ports/packages |
If those entries aren't put in /etc/make.conf, any attempt at installing a port will fail with a Read-only filesystem error.
ifconfig em0 alias 192.168.1.53/32 |
Add to the host's /etc/rc.conf
ifconfig_em0_alias2="inet 192.168.1.53/32" |
Change the line in /etc/rc.conf that reads
jail_zabbix_jail_ip="192.168.1.52" |
to read
| jail_zabbix_jail_ip="192.168.1.52,192.168.1.53" |
In other words, just add it to the zabbix jail's /etc/rc.conf ip line, using a comma to separate the two addresses.
Now restart the zabbix jail and both IPs will work for it.
service jail restart zabbix_jail |
service jail stop www_jail umount /jails/_www_jail umount /jails/_www_jail |
As we've mounted _www_jail twice, I've found that I have
to umount it twice as well. I'm not even sure it's necessary to umount it
first, but it only takes a moment.
Change the two lines in /etc/fstab to so that they refer to apache_jail and _apache_jail rather than www_jail and _www_jail. and change all www_jail entries in /etc/rc.conf to read apache_jail. At this point, you can run mount -a, and edit /jails/_apache_jail's /etc/rc.conf to change the host name from www_jail to apache_jail. Start the jail again and jls should show you that it's now listed as apache_jail rather than www_jail.
jail_zabbix_jail_rootdir="/jails/_zabbix_jail" jail_zabbix_jail_hostname="zabbix_jail" jail_zabbix_jail_ip="192.168.1.52" jail_zabbix_jail_devfs_enable="YES" jail_zabbix_jail_devfs_ruleset="jail" jail_zabbix_jail_zabbix_jail_parameters="allow.sysvipc=1" jail_zabbix_jail_exec="/bin/sh /etc/rc" |
On FreeBSD-10, the only entry in /etc/rc.conf is the one to enable jails.
jail_enable="YES" |
Rather than the other entries, one creates an /etc/jail.conf file that, in this case would read
zabbix_jail {
path = /jails/_zabbix_jail;
mount.devfs;
devfs_ruleset = 4;
|
|
When using /etc/jail.conf, one refers to a devfs.rules
ruleset by number rather than name.
The exec.preset line is necessary to mount /dev in the jail.
Otherwise, the jail will start but
the jail's /dev only shows null and log.
Another workaround that I've used is
to create an /etc/rc.local file. The file just has the line
mount -t devfs /dev /jails/_zabbix_jail/dev |
This works, as does manually mounting devfs after starting, but seems a kludgy way to do it. Using the exec.prestart line was a solution from a respondent to the PR I submitted, which can be seen here.
On the FreeBSD forums,user amiramix has a somewhat different solution, adding a line to /etc/fstab. Details are in this forum thread.
This thread's post number 8 shows a method that worked for user bforest on daemonforums, using ZFS, that I haven't yet tried for myself.
If creating a standard jail, that is, without the nullfs mount, jails behave as expected, that is, mounting the expected devices on the jail's /dev partition.