Poudriere in a jail

2016-08-03T13:12:45Z

Tykling:

Tykling: /* Syslog jail */

= Background =
This is my personal checklist for when I am setting up a new ezjail host. I like my jail hosts configured in a very specific way. There is a good chance that what is right for me is not right for you. As always, YMMV.

Also note that I talk a lot about the German hosting provider Hetzner, if you are using another provider or you are doing this at home, just ignore the Hetzner specific stuff. Much of the content here can be used with little or no changes outside Hetzner.

= Installation =
== OS install with mfsbsd ==
After receiving the server from Hetzner I boot it using the rescue system which puts me at an mfsbsd prompt via SSH. This is perfect for installing a zfs-only server.

=== Changes to zfsinstall ===
I edit the ''zfsinstall'' script <code>/root/bin/zfsinstall</code> and add "usr" to ''FS_LIST'' near the top of the script. I do this because I like to have /usr as a seperate ZFS dataset.

=== Check disks ===
I create a small zpool using just 30gigs, enough to confortably install the base OS and so on. The rest of the diskspace will be used for GELI which will have the other zfs pool on top. This encrypted zpool will house the actual jails and data. This setup allows me to have all the important data encrypted, while allowing the physical server to boot without human intervention like full disk encryption would require.

Note that the disks in this server are not new, they have been used for around two years (18023 hours/24 = 702 days):
<pre>
[root@rescue ~]# grep "ada[0-9]:" /var/run/dmesg.boot | grep "MB "
ada0: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada1: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
[root@rescue ~]# smartctl -a /dev/ada0 | grep Power_On_Hours
9 Power_On_Hours 0x0032 096 096 000 Old_age Always - 18023
[root@rescue ~]# smartctl -a /dev/ada1 | grep Power_On_Hours
9 Power_On_Hours 0x0032 096 096 000 Old_age Always - 18023
[root@rescue ~]#
</pre>

=== Destroy existing partitions ===
Any existing partitions need to be deleted first. This can be done with the ''destroygeom'' command like shown below:
<pre>
[root@rescue ~]# destroygeom -d ada0 -d ada1
Destroying geom ada0:
Deleting partition 3 ... done
Destroying geom ada1:
Deleting partition 1 ... done
Deleting partition 2 ... done
Deleting partition 3 ... done
</pre>

=== Install FreeBSD ===
Installing FreeBSD with mfsbsd is easy. I run the below command, adjusting the release I want to install of course:

<pre>
[root@rescue ~]# zfsinstall -d ada0 -d ada1 -r mirror -z 30G -t /nfs/mfsbsd/10.0-release-amd64.tbz
Creating GUID partitions on ada0 ... done
Configuring ZFS bootcode on ada0 ... done
=> 34 3907029101 ada0 GPT (1.8T)
34 2014 - free - (1M)
2048 128 1 freebsd-boot (64k)
2176 62914560 2 freebsd-zfs (30G)
62916736 3844112399 - free - (1.8T)

Creating GUID partitions on ada1 ... done
Configuring ZFS bootcode on ada1 ... done
=> 34 3907029101 ada1 GPT (1.8T)
34 2014 - free - (1M)
2048 128 1 freebsd-boot (64k)
2176 62914560 2 freebsd-zfs (30G)
62916736 3844112399 - free - (1.8T)

Creating ZFS pool tank on ada0p2 ada1p2 ... done
Creating tank root partition: ... done
Creating tank partitions: var tmp usr ... done
Setting bootfs for tank to tank/root ... done
NAME USED AVAIL REFER MOUNTPOINT
tank 270K 29.3G 31K none
tank/root 127K 29.3G 34K /mnt
tank/root/tmp 31K 29.3G 31K /mnt/tmp
tank/root/usr 31K 29.3G 31K /mnt/usr
tank/root/var 31K 29.3G 31K /mnt/var
Extracting FreeBSD distribution ... done
Writing /boot/loader.conf... done
Writing /etc/fstab...Writing /etc/rc.conf... done
Copying /boot/zfs/zpool.cache ... done

Installation complete.
The system will boot from ZFS with clean install on next reboot

You may type "chroot /mnt" and make any adjustments you need.
For example, change the root password or edit/create /etc/rc.conf for
for system services.

WARNING - Don't export ZFS pool "tank"!
[root@rescue ~]#
</pre>

=== Post install configuration (before reboot) ===
Before rebooting into the installed FreeBSD I need to make certain I can reach the server through SSH after the reboot. This means:
# Adding network settings to <code>/etc/rc.conf</code>
# Adding sshd_enable="YES" to <code>/etc/rc.conf</code>
# Change ''PermitRootLogin'' to ''Yes'' in <code>/etc/ssh/sshd_config</code> ''Note: In the current This is now the default in the zfsinstall image that Hetzner provides''
# Add nameservers to <code>/etc/resolv.conf</code>
# Finally I set the root password.

'''All of these steps are essential if I am going to have any chance of logging in after reboot.''' Most of these changes can be done from the mfsbsd shell but the password change requires ''chroot'' into the newly installed environment.

I use the ''chroot'' command but start another shell as bash is not installed in /mnt:
<pre>
[root@rescue ~]# chroot /mnt/ csh
rescue# ee /etc/rc.conf
rescue# ee /etc/ssh/sshd_config
rescue# passwd
New Password:
Retype New Password:
rescue#
</pre>

So, the network settings are sorted, root password is set, and root is permitted to ssh in. Time to reboot (this is the exciting part).

'''Remember to use <code>shutdown -r now</code> and not <code>reboot</code> when you reboot. <code>shutdown -r now</code> performs the proper shutdown process including rc.d scripts and disk buffer flushing. <code>reboot</code> is the "bigger hammer" to use when something is preventing <code>shutdown -r now</code> from working.

== Basic config after first boot ==
If the server boots without any problems, I do some basic configuration before I continue with the disk partitioning.

=== Timezone ===
I run the command <code>tzsetup</code> to set the proper timezone, and set the time using <code>ntpdate</code> if neccesary.

''Note: The current hetzner freebsd image has the timezone set to CEST, I like my servers configured as UTC''

=== Basic ports ===
I also add some basic ports with <code>pkg</code> so I can get screen etc. up and running as soon as possible:
<pre>
# pkg install bash screen sudo portmaster
</pre>

I then add the following to <code>/usr/local/etc/portmaster.rc</code>:
<pre>
ALWAYS_SCRUB_DISTFILES=dopt
PM_DEL_BUILD_ONLY=pm_dbo
SAVE_SHARED=wopt
PM_LOG=/var/log/portmaster.log
PM_IGNORE_FAILED_BACKUP_PACKAGE=pm_ignore_failed_backup_package
</pre>
An explanation of these options can be found on the [[Portmaster]] page.

After a <code>rehash</code> and adding my non-root user with <code>adduser</code>, I am ready to continue with the disk configuration. I also remember to disable root login in <code>/etc/ssh/sshd_config</code>.

== Further disk configuration ==
After the reboot into the installed FreeBSD environment, I need to do some further disk configuration.
=== Create swap partitions ===
Swap-on-zfs is not a good idea for various reasons. To keep my swap encrypted but still off zfs I use geli onetime encryption. To avoid problems if a disk dies I also use gmirror. First I add the partitions with gpart:

<pre>
$ sudo gpart add -t freebsd-swap -s 10G /dev/ada0
ada0p3 added
$ sudo gpart add -t freebsd-swap -s 10G /dev/ada1
ada1p3 added
$
</pre>

Then I make sure gmirror is loaded, and loaded on boot:
<pre>
$ sudo sysrc -f /boot/loader.conf geom_mirror_load="YES"
$ sudo kldload geom_mirror
</pre>

Then I create the gmirror:
<pre>
sudo gmirror label swapmirror /dev/ada0p3 /dev/ada1p3
</pre>

Finally I add the following line to <code>/etc/fstab</code> to get encrypted swap on top of the gmirror:
<pre>
/dev/mirror/swapmirror.eli none swap sw,keylen=256,sectorsize=4096 0 0
</pre>

I can enable the new swap partition right away:
<pre>
$ sudo swapon /dev/mirror/swapmirror.eli
$ swapinfo
Device 1K-blocks Used Avail Capacity
/dev/mirror/swapmirror.eli 8388604 0 8388604 0%
$
</pre>

=== Create GELI partitions ===
First I create the partitions to hold the geli devices:
<pre>
$ sudo gpart add -t freebsd-ufs ada0
ada0p4 added
$ sudo gpart add -t freebsd-ufs ada1
ada1p4 added
</pre>

I add them as <code>freebsd-ufs</code> type partitions, as there is no dedicated <code>freebsd-geli</code> type.

=== Create GELI key ===
To create a GELI key I copy some data from <code>/dev/random</code>:
<pre>
$ sudo dd if=/dev/random of=/root/geli.key bs=256k count=1
1+0 records in
1+0 records out
262144 bytes transferred in 0.003347 secs (78318372 bytes/sec)
$
</pre>

=== Create GELI volumes ===
I create the GELI volumes with 4k blocksize and 256bit AES encryption:

<pre>
$ sudo geli init -s 4096 -K /root/geli.key -l 256 /dev/ada0p4
Enter new passphrase:
Reenter new passphrase:

Metadata backup can be found in /var/backups/ada0p4.eli and
can be restored with the following command:

# geli restore /var/backups/ada0p4.eli /dev/ada0p4

$ sudo geli init -s 4096 -K /root/geli.key -l 256 /dev/ada1p4
Enter new passphrase:
Reenter new passphrase:

Metadata backup can be found in /var/backups/ada1p4.eli and
can be restored with the following command:

# geli restore /var/backups/ada1p4.eli /dev/ada1p4

$
</pre>

=== Enable AESNI ===
Most Intel CPUs have hardware acceleration of AES which helps a lot with GELI performance. I load the <code>aesni</code> module during boot from <code>/boot/loader.conf</code>:
<pre>
$ sudo sysrc -f /boot/loader.conf aesni_load="YES"
</pre>

=== Attach GELI volumes ===
Now I just need to attach the GELI volumes before I am ready to create the second zpool:
<pre>
$ sudo geli attach -k /root/geli.key /dev/ada0p4
Enter passphrase:
$ sudo geli attach -k /root/geli.key /dev/ada1p4
Enter passphrase:
$
</pre>

=== Create second zpool ===
<pre>
$ sudo zpool create gelipool mirror /dev/ada0p4.eli /dev/ada1p4.eli
$ zpool status
pool: gelipool
state: ONLINE
scan: none requested
config:

NAME STATE READ WRITE CKSUM
gelipool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
ada0p4.eli ONLINE 0 0 0
ada1p4.eli ONLINE 0 0 0

errors: No known data errors

pool: tank
state: ONLINE
scan: none requested
config:

NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
ada0p2 ONLINE 0 0 0
ada1p2 ONLINE 0 0 0

errors: No known data errors
$
</pre>

=== Create ZFS filesystems on the new zpool ===
The last remaining thing is to create a filesystem in the new zfs pool:
<pre>
$ zfs list
NAME USED AVAIL REFER MOUNTPOINT
gelipool 624K 3.54T 144K /gelipool
tank 704M 28.6G 31K none
tank/root 704M 28.6G 413M /
tank/root/tmp 38K 28.6G 38K /tmp
tank/root/usr 291M 28.6G 291M /usr
tank/root/var 505K 28.6G 505K /var
$ sudo zfs set mountpoint=none gelipool
$ sudo zfs set compression=on gelipool
$ sudo zfs create -o mountpoint=/usr/jails gelipool/jails
$ zfs list
NAME USED AVAIL REFER MOUNTPOINT
gelipool 732K 3.54T 144K none
gelipool/jails 144K 3.54T 144K /usr/jails
tank 704M 28.6G 31K none
tank/root 704M 28.6G 413M /
tank/root/tmp 38K 28.6G 38K /tmp
tank/root/usr 291M 28.6G 291M /usr
tank/root/var 505K 28.6G 505K /var
$
</pre>

=== Disable atime ===
One last thing I like to do is to disable <code>atime</code> or ''access time'' on the filesystem. Access times are recorded every time a file is read, and while this can have it's use cases, I never use it. Disabling it means a lot fewer write operations, as a read operation doesn't automatically include a write operation when <code>atime</code> is disabled. Disabling it is easy:

<pre>
$ sudo zfs set atime=off tank
$ sudo zfs set atime=off gelipool
$
</pre>

The next things are post-install configuration stuff like OS upgrade, ports, firewall and so on. The basic install is finished \o/

=== Reserved space ===
Running out of space in ZFS is bad. Stuff will run slowly and may stop working entirely until some space is freed. The problem is that ZFS is a journalled filesystem which means that all writes, even a deletion, requires writing data to the disk. I've more than once wound up in a situation where I couldn't delete file to free up diskspace because the disk was full.

Sometimes this can be resolved by overwriting a large file, like:
<pre>
$ echo > /path/to/a/large/file
</pre>

This will overwrite the file thereby freeing up some space, but sometimes even this is not possible. This is where reserved space comes in. I create a new filesystem in each pool, set them readonly and without a mountpoint, and with 1G reserved each:
<pre>
$ sudo zfs create -o mountpoint=none -o reservation=1G -o readonly=on gelipool/reserved
$ sudo zfs create -o mountpoint=none -o reservation=1G -o readonly=on tank/reserved
</pre>

If I run out of space for some reason, I can just delete the dataset, or unset the <code>reserved</code> property, and I immediately have 1G diskspace available. Yay!

= Ports =
== Installing the ports tree ==
I need to bootstrap the ports system, I use <code>portsnap</code> as it is way faster than using c(v)sup. Initially I run <code>portsnap fetch extract</code> and when I need to update the tree later I use <code>portsnap fetch update</code>.

== smartd ==
I install smartd to monitor the disks for problems:
<pre>
$ sudo pkg install smartmontools
</pre>

I create the file <code>/usr/local/etc/smartd.conf</code> and add this line to it:
<pre>
DEVICESCAN -a -m thomas@gibfest.dk
</pre>

This makes smartd monitor all disks and send me an email if it finds an error.

Remember to enable smartd in <code>/etc/rc.conf</code> and start it:
<pre>
sudo sysrc smartd_enable="YES"
sudo service smartd start
</pre>

== openntpd ==
I install <code>net/openntpd</code> to keep the clock in sync. I find this a lot easier to configure than the base ntpd.

<pre>
sudo pkg install openntpd
</pre>

I enable <code>openntpd</code> in <code>/etc/rc.conf</code>:

<pre>
sudo sysrc openntpd_enable="YES"
</pre>

and add a one line config file:
<pre>
$ grep -v "^#" /usr/local/etc/ntpd.conf | grep -v "^$"
servers de.pool.ntp.org
$
</pre>

sync the clock and start openntpd:

<pre>
sudo ntpdate de.pool.ntp.org
sudo service openntpd start
</pre>

== ntpdate ==
I also enable <code>ntpdate</code> to help set the clock after a reboot. I add the following two lines to <code>/etc/rc.conf</code>:
<pre>
sudo sysrc ntpdate_enable="YES"
sudo sysrc ntpdate_hosts="de.pool.ntp.org"
</pre>

= Upgrade OS (buildworld) =
I usually run -STABLE on my hosts, which means I need to build and install a new world and kernel. I also like having [http://www.freebsd.org/cgi/man.cgi?query=rctl rctl] available on my jail hosts, so I can limit jail ressources in all kinds of neat ways. I also like having DTRACE available. Additionally I also need the built world to populate ezjails basejail.

'''Note''': I will need to update the host and the jails many times during the lifespan of this server, which is likely > 2-3 years. As new security problems are found or features are added that I want, I will update host and jails. There is a section about staying up to date later in this page. This section (the one you are reading now) only covers the OS update I run right after installing the server.

== Fetching sources ==
I install <code>subversion</code>:
<pre>
sudo pkg install subversion
</pre>

and then get the sources:
<pre>
sudo svn checkout https://svn0.eu.freebsd.org/base/stable/10/ /usr/src/
Error validating server certificate for 'https://svn0.eu.freebsd.org:443':
- The certificate is not issued by a trusted authority. Use the
fingerprint to validate the certificate manually!
Certificate information:
- Hostname: svnmir.bme.FreeBSD.org
- Valid: from Jun 29 12:24:17 2013 GMT until Jun 29 12:24:17 2015 GMT
- Issuer: clusteradm, FreeBSD.org, CA, US(clusteradm@FreeBSD.org)
- Fingerprint: 39:B0:53:35:CE:60:C7:BB:00:54:96:96:71:10:94:BB:CE:1C:07:A7
(R)eject, accept (t)emporarily or accept (p)ermanently? p
A /usr/src/sys
A /usr/src/sys/arm
A /usr/src/sys/arm/ti
A /usr/src/sys/arm/ti/cpsw
A /usr/src/sys/arm/ti/am335x
A /usr/src/sys/arm/ti/usb
A /usr/src/sys/arm/ti/twl
A /usr/src/sys/arm/ti/ti_mmchs.c
A /usr/src/sys/arm/ti/ti_cpuid.h
A /usr/src/sys/arm/ti/ti_mmchs.h
A /usr/src/sys/arm/ti/ti_i2c.c
A /usr/src/sys/arm/ti/ti_i2c.h
A /usr/src/sys/arm/ti/am335x/am335x_pwm.c
A /usr/src/sys/arm/ti/ti_sdma.c
<snip>
A /usr/src/sys/fs/nandfs/nandfs_cleaner.c
A /usr/src/sys/fs/nandfs/nandfs_bmap.c
A /usr/src/sys/fs/nandfs/bmap.c
A /usr/src/sys/fs/nandfs/nandfs_subr.c
A /usr/src/sys/fs/nandfs/bmap.h
A /usr/src/sys/fs/nandfs/nandfs_vfsops.c
A /usr/src/sys/fs/nandfs/nandfs_sufile.c
U /usr/src
Checked out revision 282899.
$
</pre>

This takes a while the first time, but subsequent runs are much faster.

'''Note:''' The reason I use the full Subversion port instead of just svnup is that using full SVN client means that you get the SVN revision in <code>uname -a</code> when checking out code with the full SVN client.

== Create kernel config ==
After the sources finish downloading, I create a new kernel config file <code>/etc/TYKJAIL</code> with the following content:
<pre>
include GENERIC
ident TYKJAIL

#rctl
options RACCT
options RCTL
</pre>

I then create a symlink to the kernel config file in /etc/:
<pre>
# ln -s /etc/TYKJAIL /usr/src/sys/amd64/conf/
# ls -l /usr/src/sys/amd64/conf/TYKJAIL
lrwxr-xr-x 1 root wheel 9 Jul 22 16:14 /usr/src/sys/amd64/conf/TYKJAIL -> /etc/TYKJAIL
</pre>

Finally I enable the kernel config in <code>/etc/make.conf</code>:
<pre>
$ cat /etc/make.conf
KERNCONF=TYKJAIL
</pre>

== Building world and kernel ==
Finally I start the build. I use -j to start one thread per core in the system. <code>sysctl hw.ncpu</code> shows the number of available cores:
<pre>
# sysctl hw.ncpu
hw.ncpu: 12
</pre>

To build the new system:
<pre>
# cd /usr/src/
# time (sudo make -j$(sysctl -n hw.ncpu) buildworld && sudo make -j$(sysctl -n hw.ncpu) kernel) && date
</pre>

After the build finishes, reboot and run mergemaster, installworld, and mergemaster again:
<pre>
# cd /usr/src/
# sudo mergemaster -pFUi && sudo make installworld && sudo mergemaster -FUi
</pre>

'''DO NOT OVERWRITE /etc/group AND /etc/master.passwd AND OTHER CRITICAL FILES!'''

Reboot after the final mergemaster completes, and boot into the newly built world.

= Preparing ezjail =
ezjail needs to be installed and a bit of configuration is also needed, in addition to bootstrapping <code>/usr/jails/basejail</code> and <code>/usr/jails/newjail</code>.

== Installing ezjail ==
Just install it with pkg:
<pre>
sudo pkg install ezjail
</pre>

== Configuring ezjail ==
Then I go edit the ezjail config file <code>/usr/local/etc/ezjail.conf</code> and add/change these three lines near the bottom:
<pre>
ezjail_use_zfs="YES"
ezjail_jailzfs="gelipool/jails"
ezjail_use_zfs_for_jails="YES"
</pre>

This makes ezjail use seperate zfs datasets under <code>gelipool/jails</code> for the <code>basejail</code> and <code>newjail</code>, as well as for each jail created. <code>ezjail_use_zfs_for_jails</code> is supported since ezjail 3.2.2.

== Bootstrapping ezjail ==
Finally I populate <code>basejail</code> and <code>newjail</code> from the world I build earlier:
<pre>
$ sudo ezjail-admin update -i
</pre>

The last line of the output is a message saying:
<pre>
Note: a non-standard /etc/make.conf was copied to the template jail in order to get the ports collection running inside jails.
</pre>

This is because ezjail defaults to symlinking the ports collection in the same way it symlinks the basejail. I prefer having seperate/individual ports collections in each of my jails though, so I remove the symlink and make.conf from newjail:
<pre>
$ sudo rm /usr/jails/newjail/etc/make.conf /usr/jails/newjail/usr/ports /usr/jails/newjail/usr/src
$ sudo mkdir /usr/jails/newjail/usr/src
</pre>

== ZFS goodness ==
Note that ezjail has created two new ZFS datasets to hold <code>basejail</code> and <code>newjail</code>:
<pre>
$ zfs list -r gelipool/jails
NAME USED AVAIL REFER MOUNTPOINT
gelipool/jails 239M 3.54T 476K /usr/jails
gelipool/jails/basejail 236M 3.54T 236M /usr/jails/basejail
gelipool/jails/newjail 3.10M 3.54T 3.10M /usr/jails/newjail
</pre>

== ezjail flavours ==
ezjail has a pretty awesome feature that makes it possible to create templates or ''flavours'' which apply common settings when creating a new jail. I always have a ''basic'' flavour which adds a user for me, installs an SSH key, adds a few packages like bash, screen, sudo and portmaster - and configures those packages. Basically, everything I find myself doing over and over again every time I create a new jail.

It is also possible, of course, to create more advanced flavours, I've had one that installs a complete nginx+php-fpm server with all the neccesary packages and configs.

ezjail flavours are technically pretty simple. By default, they are located in the same place as ''basejail'' and ''newjail'', and ezjail comes with an example flavour to get you started. Basically a flavour is a file/directory hierachy which is copied to the jail, and a shell script called ''ezjail.flavour'' which is run once, the first time the jail is started, and then deleted.

For reference, I've included my basic flavour here. First is a listing of the files included in the flavour, and then the ''ezjail.flavour'' script which performs tasks beyond copying config files.

<pre>
$ find /usr/jails/flavours/tykbasic
/usr/jails/flavours/tykbasic
/usr/jails/flavours/tykbasic/ezjail.flavour
/usr/jails/flavours/tykbasic/usr
/usr/jails/flavours/tykbasic/usr/local
/usr/jails/flavours/tykbasic/usr/local/etc
/usr/jails/flavours/tykbasic/usr/local/etc/portmaster.rc
/usr/jails/flavours/tykbasic/usr/local/etc/sudoers
/usr/jails/flavours/tykbasic/usr/home
/usr/jails/flavours/tykbasic/usr/home/tykling
/usr/jails/flavours/tykbasic/usr/home/tykling/.ssh
/usr/jails/flavours/tykbasic/usr/home/tykling/.ssh/authorized_keys
/usr/jails/flavours/tykbasic/usr/home/tykling/.screenrc
/usr/jails/flavours/tykbasic/etc
/usr/jails/flavours/tykbasic/etc/fstab
/usr/jails/flavours/tykbasic/etc/rc.conf
/usr/jails/flavours/tykbasic/etc/periodic.conf
/usr/jails/flavours/tykbasic/etc/resolv.conf
</pre>

As you can see, the flavour contains files like ''/etc/resolv.conf'' and other stuff to make the jail work. The name of the flavour here is ''tykbasic'' which means that if I want a file to end up in ''/usr/home/tykling'' after the flavour has been applied, I need to put that file in the folder ''/usr/jails/flavours/tykbasic/usr/home/tykling/'' - '''remember to also chown the files in the flavour appropriately'''.

Finally, my ''ezjail.flavour'' script looks like so:

<pre>
#!/bin/sh
#
# BEFORE: DAEMON
#
# ezjail flavour example

# Timezone
###########
#
ln -s /usr/share/zoneinfo/Europe/Copenhagen /etc/localtime

# Groups
#########
#
pw groupadd -q -n tykling

# Users
########
#
# To generate a password hash for use here, do:
# openssl passwd -1 "the password"
echo -n '$1$L/fC0UrO$bi65/BOIAtMkvluDEDCy31' | pw useradd -n tykling -u 1001 -s /bin/sh -m -d /usr/home/tykling -g tykling -c 'tykling' -H 0

# Packages
###########
#
env ASSUME_ALWAYS_YES=YES pkg bootstrap
pkg install -y bash
pkg install -y sudo
pkg install -y portmaster
pkg install -y screen

#change shell to bash
chsh -s bash tykling

#update /etc/aliases
echo "root: thomas@gibfest.dk" >> /etc/aliases
newaliases

#remove adjkerntz from crontab
cat /etc/crontab | grep -E -v "(Adjust the time|adjkerntz)" > /etc/crontab.new
mv /etc/crontab.new /etc/crontab

#remove ports symlink
rm /usr/ports

# create symlink to /usr/home in / (adduser defaults to /usr/username as homedir)
ln -s /usr/home /home
</pre>

Creating a flavour is easy: just create a folder under <code>/usr/jails/flavours/</code> that has the name of the flavour, and start adding files and folders there. The ezjail.flavour script should be placed in the root (see the example further up the page).

Finally I add the following to <code>/usr/local/etc/ezjail.conf</code> to make ezjail always use my new flavour:
<pre>
ezjail_default_flavour="tykbasic"
</pre>

= Configuration =
This section outlines what I do to further prepare the machine to be a nice ezjail host.

== Firewall ==
One of the first things I fix is to enable the pf firewall from OpenBSD. I add the following to <code>/etc/rc.conf</code> to enable pf at boot time:
<pre>
sudo sysrc pf_enable="YES"
sudo sysrc pflog_enable="YES"
</pre>

I also create a very basic <code>/etc/pf.conf</code>:
<pre>
[root@ ~]# cat /etc/pf.conf
### macros
if="em0"
table <portknock> persist

#external addresses
tykv4="a.b.c.d"
tykv6="2002:ab:cd::/48"
table <allowssh> { $tykv4,$tykv6 }

#local addresses
glasv4="w.x.y.z"

### scrub
scrub in on $if all fragment reassemble

################
### filtering
### block everything
block log all

################
### skip loopback interface(s)
set skip on lo0

################
### icmp6
pass in quick on $if inet6 proto icmp6 all icmp6-type {echoreq,echorep,neighbradv,neighbrsol,routeradv,routersol}

################
### pass outgoing
pass out quick on $if all

################
### portknock rule (more than 5 connections in 10 seconds to the port specified will add the "offending" IP to the <portknock> table)
pass in quick on $if inet proto tcp from any to $glasv4 port 32323 synproxy state (max-src-conn-rate 5/10, overload <portknock>)

### pass incoming ssh and icmp
pass in quick on $if proto tcp from { <allowssh>, <portknock> } to ($if) port 22
pass in quick on $if inet proto icmp all icmp-type { 8, 11 }

################
### pass ipv6 fragments (hack to workaround pf not handling ipv6 fragments)
pass in on $if inet6
block in log on $if inet6 proto udp
block in log on $if inet6 proto tcp
block in log on $if inet6 proto icmp6
block in log on $if inet6 proto esp
block in log on $if inet6 proto ipv6
</pre>

To load pf without rebooting I run the following:
<pre>
[root@ ~]# kldload pf
[root@ ~]# kldload pflog
[root@ ~]# pfctl -ef /etc/pf.conf && sleep 60 && pfctl -d
No ALTQ support in kernel
ALTQ related functions disabled
</pre>

I get no prompt after this because pf has cut my SSH connection. But I can SSH back in if I did everything right, and if not, I can just wait 60 seconds after which pf will be disabled again. I SSH in and reattach to the screen I am running this in, and press control-c, so the "sleep 60" is interrupted and pf is not disabled. Neat little trick for when you want to avoid locking yourself out :)

== Replacing sendmail with Postfix ==
I always replace Sendmail with Postfix on every server I manage. See [[Replacing_Sendmail_With_Postfix]] for more info.

== Listening daemons ==
When you add an IP alias for a jail, any daemons listening on * will also listen on the jails IP, which is not what I want. For example, I want the jails sshd to be able to listen on the jails IP on port 22, instead of the hosts sshd. Check for listening daemons like so:
<pre>
$ sockstat -l46
USER COMMAND PID FD PROTO LOCAL ADDRESS FOREIGN ADDRESS
root master 1554 12 tcp4 *:25 *:*
root master 1554 13 tcp6 *:25 *:*
root sshd 948 3 tcp6 *:22 *:*
root sshd 948 4 tcp4 *:22 *:*
root syslogd 789 6 udp6 *:514 *:*
root syslogd 789 7 udp4 *:514 *:*
[tykling@glas ~]$
</pre>

This tells me that I need to change Postfix, sshd and syslogd to stop listening on all IP addresses.
=== Postfix ===
The defaults in Postfix are really nice on FreeBSD, and most of the time a completely empty config file is fine for a system mailer (sendmail replacement). However, to make Postfix stop listening on port 25 on all IP addresses, I do need one line in <code>/usr/local/etc/postfix/main.cf</code>:
<pre>
$ cat /usr/local/etc/postfix/main.cf
inet_interfaces=localhost
$
</pre>

=== sshd ===
To make <code>sshd</code> stop listening on all IP addresses I uncomment and edit the <code>ListenAddress</code> line in <code>/etc/ssh/sshd_config</code>:
<pre>
$ grep ListenAddress /etc/ssh/sshd_config
ListenAddress x.y.z.226
#ListenAddress ::
</pre>
(IP address obfuscated..)

=== syslogd ===
I don't need my syslogd to listen on the network at all, so I add the following line to <code>/etc/rc.conf</code>:
<pre>
$ grep syslog /etc/rc.conf
syslogd_flags="-ss"
</pre>

=== Restarting services ===
Finally I restart Postfix, sshd and syslogd:
<pre>
$ sudo /etc/rc.d/syslogd restart
Stopping syslogd.
Waiting for PIDS: 789.
Starting syslogd.
$ sudo /etc/rc.d/sshd restart
Stopping sshd.
Waiting for PIDS: 948.
Starting sshd.
$ sudo /usr/local/etc/rc.d/postfix restart
postfix/postfix-script: stopping the Postfix mail system
postfix/postfix-script: starting the Postfix mail system
</pre>

A check with <code>sockstat</code> reveals that no more services are listening on all IP addresses:
<pre>
$ sockstat -l46
USER COMMAND PID FD PROTO LOCAL ADDRESS FOREIGN ADDRESS
root master 1823 12 tcp4 127.0.0.1:25 *:*
root master 1823 13 tcp6 ::1:25 *:*
root sshd 1617 3 tcp4 x.y.z.226:22 *:*
$
</pre>

== Network configuration ==
Network configuration is a big part of any jail setup. If I have enough IP addresses (ipv4 and ipv6) I can just add IP aliases as needed. If I only have one or a few v4 IPs I will need to use rfc1918 addresses for the jails. In that case, I create a new loopback interface, <code>lo1</code> and add the IP aliases there. I then use the pf firewall to redirect incoming traffic to the right jail, depending on the port in use.

=== IPv4 ===
If rfc1918 jails are needed, I add the following to <code>/etc/rc.conf</code> to create the lo1 interface on boot:
<pre>
### lo1 interface for ipv4 rfc1918 jails
cloned_interfaces="lo1"
</pre>

When the lo1 interface is created, or if it isn't needed, I am ready to start adding IP aliases for jails as needed.

=== IPv6 ===
On the page [[Hetzner_ipv6]] I've explained how to make IPv6 work on a Hetzner server where the supplied IPv6 default gateway is outside the IPv6 subnet assigned.

When basic IPv6 connectivity works, I am ready to start adding IP aliases for jails as needed.

=== Allow ping from inside jails ===
I add the following to <code>/etc/sysctl.conf</code> so the jails are allowed to do icmp ping. This enables raw socket access, which can be a security issue if you have untrusted root users in your jails. Use with caution.

<pre>
#allow ping in jails
security.jail.allow_raw_sockets=1
</pre>

== Tips & tricks ==
=== Get jail info out of ''top'' ===
To make top show the jail id of the jail in which the process is running in a column, I need to specify the -j flag to top. Since this is a multi-cpu server I am working on, I also like giving the -P flag to top, to get a seperate line of cpu stats per core. Finally, I like -a to get the full commandline/info of the running processes. I add the following to my .bashrc in my homedir on the jail host:
<pre>
alias top="nice top -j -P -a"
</pre>

...this way I don't have to remember passing <code>-j -P -a</code> to top every time. Also, I've been told to run top with ''nice'' to limit the cpu used by top itself. I took the advice so the complete alias looks like above.

= Base jails =
My jails need various services, so:
* I have a DNS jail with a caching DNS server which is used by all the jails.
* I also have a syslog server on each jailhost to collect syslog from all the jails and send them to my central syslog server.
* I have a postgres jail on each jail host, so I only need to maintain one postgres server.
* I have a reverse webproxy jail since many of my jails serve some sort of web application, and I like to terminate SSL for those in one place in an attempt to keep the individual jails as simple as possible.

This section describes how I configure the postgres and web jails that provide services to the other jails.

== DNS jail ==
I don't need a public v4 IP for this jail, so I configure it with an RFC1918 v4 IP on a loopback interface, and of course a real IPv6 address.

I am used to using bind but unbound is just as good. Use whatever you are comfortable with. Make sure you permit DNS traffic from the other jails to the DNS jail.

This jail needs to be started first since the rest of the jails need it for DNS. ezjail runs rcorder on the config files in <code>/usr/local/etc/ezjail</code> which means I can use the normal <code>PROVIDE:</code> and <code>REQUIRE:</code> to control the jail dependencies. I change the ezjail config for my DNS jail to have the following <code>PROVIDE:</code> line:
<pre>
# PROVIDE: dns
</pre>

The rest of the jails all get the following <code>REQUIRE:</code> line:
<pre>
# REQUIRE: dns
</pre>

== Syslog jail ==
I don't need a public v4 IP for this jail, so I configure it with an RFC1918 v4 IP on a loopback interface, and of course a real IPv6 address.

This jail gets the following <code>PROVIDE:</code> line in it's ezjail config file:
<pre>
# PROVIDE: syslog
</pre>

I use <code>syslog-ng</code> for this. I install the package using <code>pkg install syslog-ng</code> and then add a few things to the default config:

* I use the following options, YMMV:
<code>options { chain_hostnames(off); flush_lines(0); threaded(yes); use_fqdn(yes); keep_hostname(no); use_dns(yes); stats-freq(60); stats-level(0); };</code>

* I remove <code>udp()</code> from the default source and define a new source:
<code>source jailsrc { udp(ip("10.0.0.1")); udp6(ip("w:x:y:z:10::1")); };</code>

* I also remove the line: <code>destination console { file("/dev/console"); };</code> since the jail does not have access to <code>/dev/console</code>. I also remove any corresponding <code>log</code> statements that use <code>destination(console);</code>.

* I add a new destination:
<pre>
destination loghost{
syslog(
"syslog.tyktech.dk"
transport("tls")
port(1999)
tls(peer-verify(required-untrusted))
localip("10.0.0.1")
log-fifo-size(10000)
);
};
</pre>

* Finally tell syslog-ng to send all logdata from <code>jailsrc</code> to <code>loghost</code>:
<code>log { source(jailsrc); destination(loghost); flags(flow_control); };</code>

The rest of the jails get the following <code>REQUIRE:</code> line:
<pre>
# REQUIRE: dns syslog
</pre>

... and of course the jails default <code>/etc/syslog.conf</code> all get this line so they log to the syslog jail:
<code>*.* @10.0.0.1</code>

== Postgres jail ==
I don't need a public v4 IP for this jail, so I configure it with an RFC1918 v4 IP on a loopback interface, and of course a real IPv6 address. I add a AAAA record in DNS for the v6 IP so I have something to point the clients at.

I install the latest Postgres server port, at the time of writing that is <code>databases/postgresql93-server</code>. But before I can run <code>/usr/local/etc/rc.d/postgresql initdb</code> I need to permit the use of SysV shared memory in the jail. This is done in the ezjail config file for the jail, in the _parameters line. I need to add <code>allow.sysvipc=1</code> so I change the line from:
<pre>
export jail_postgres_kush_tyknet_dk_parameters=""
</pre>
to:
<pre>
export jail_postgres_kush_tyknet_dk_parameters="allow.sysvipc=1"
</pre>

While I'm there I also change the jails <code># PROVIDE:</code> line to:
<pre>
# PROVIDE: postgres
</pre>

And ofcourse the standard <code># REQUIRE:</code> line:
<pre>
# REQUIRE: dns syslog
</pre>

After restarting the jail I can run <code>initdb</code> and start Postgres. When a jail needs a database I need to:
* Add a DB user (with the <code>createuser -P someusername</code> command)
* Add a database with the new user as owner (<code>createdb -O someusername somedbname</code>)
* Add permissions in <code>/usr/local/pgsql/data/pg_hba.conf</code>
* Open a hole in the firewall so the jail can reach the database on TCP port <code>5432</code>
* Add <code>postgres</code> to the <code># REQUIRE:</code> line in the ezjail config file

== Web Jail ==
I need a public V4 IP for the web jail and I also give it a V6 IP. Since I use a different V6 IP per website, I will need additional v6 addresses when I start adding websites. I add the v6 addresses to the web jail in batches of 10 as I need them. After creating the jail and bootstrapping the ports collection I install <code>security/openssl</code> and <code>www/nginx</code> and configure it. More on that later.

= ZFS snapshots and backup =
So, since all this is ZFS based, there is a few tricks I do to make it easier to restore data in case of accidental file deletion or other dataloss.

== Periodic snapshots using sysutils/zfs-periodic ==
<code>sysutils/zfs-periodic</code> is a little script that uses the FreeBSD <code>periodic(8)</code> system to make snapshots of filesystems with regular intervals. It supports making hourly snapshots with a small change to <code>periodic(8)</code>, but I've settled for daily, weekly and monthly snapshots on my servers.

After installing <code>sysutils/zfs-periodic</code> I add the following to <code>/etc/periodic.conf</code>:
<pre>
#daily zfs snapshots
daily_zfs_snapshot_enable="YES"
daily_zfs_snapshot_pools="tank gelipool"
daily_zfs_snapshot_keep=7

#weekly zfs snapshots
weekly_zfs_snapshot_enable="YES"
weekly_zfs_snapshot_pools="tank gelipool"
weekly_zfs_snapshot_keep=5

#monthly zfs snapshots
monthly_zfs_snapshot_enable="YES"
monthly_zfs_snapshot_pools="tank gelipool"
monthly_zfs_snapshot_keep=6

#monthly zfs scrub
monthly_zfs_scrub_pools="tank gelipool"
monthly_zfs_scrub_enable="YES"
</pre>

Note that the last bit also enables a monthly scrub of the filesystem. Remember to change the pool name and remember to set the number of snapshots to retain to something appropriate. These things are always a tradeoff between diskspace and safety. Think it over and find some values that make you sleep well at night :)

After this has been running for a few days, you should have a bunch of daily snapshots:
<pre>
$ zfs list -t snapshot | grep gelipool@
gelipool@daily-2012-09-02 0 - 31K -
gelipool@daily-2012-09-03 0 - 31K -
gelipool@daily-2012-09-04 0 - 31K -
gelipool@daily-2012-09-05 0 - 31K -
</pre>

== Back-to-back ZFS mirroring ==
I am lucky enough to have more than one of these jail hosts, which is the whole reason I started writing down how I configure them. One of the advantages to having more than one is that I can configure <code>zfs send/receive</code> jobs and make server A send it's data to server B, and vice versa.

=== Introduction ===
The concept is pretty basic, but as it often happens, security considerations turn what was a simple and elegant idea into something... else. To make the back-to-back backup scheme work without sacrificing too much security, I first make a jail on each jailhost called <code>backup.jailhostname</code>. This jail will have control over a designated zfs dataset which will house the backups sent from the other server.

=== Create ZFS dataset ===
First I create the zfs dataset:
<pre>
$ sudo zfs create cryptopool/backups
$ sudo zfs set jailed=on cryptopool/backups
</pre>

=== 'jail' the new dataset ===
I create the jail like I normally do, but after creating it, I edit the ezjail config file and tell it which extra zfs dataset to use:
<pre>
$ grep dataset /usr/local/etc/ezjail/backup_glas_tyknet_dk
export jail_backup_glas_tyknet_dk_zfs_datasets="cryptopool/backups"
</pre>

This makes <code>ezjail</code> run the <code>zfs jail</code> command with the proper jail id when the jail is started.

=== jail sysctl settings ===
I also add the following to the jails ezjail config:

<pre>
# grep parameters /usr/local/etc/ezjail/backup_glas_tyknet_dk
export jail_backup_glas_tyknet_dk_parameters="allow.mount.zfs=1 enforce_statfs=1"
</pre>

=== Configuring the backup jail ===
The jail is ready to run now, and inside the jail a <code>zfs list</code> looks like this:
<pre>
$ zfs list
NAME USED AVAIL REFER MOUNTPOINT
cryptopool 3.98G 2.52T 31K none
cryptopool/backups 62K 2.52T 31K none
$
</pre>

I don't want to open up root ssh access to this jail, but the remote servers need to call <code>zfs receive</code> which requires root permissions. <code>zfs allow</code> to the rescue! <code>zfs allow</code> makes it possible to say "user X is permitted to do action Y on dataset Z" which is what I need here. In the backup jail I add a user called <code>tykbackup</code> which will be used as the user receiving the zfs snapshots from the remote servers.

I then run the following commands to allow the user to work with the dataset:
<pre>
$ sudo zfs allow tykbackup atime,compression,create,mount,mountpoint,readonly,receive cryptopool/backups
$ sudo zfs allow cryptopool/backups
---- Permissions on cryptopool/backups -------------------------------
Local+Descendent permissions:
user tykbackup atime,compression,create,mount,mountpoint,readonly,receive
$
</pre>

Testing if it worked:
<pre>
$ sudo su tykbackup
$ zfs create cryptopool/backups/test
$ zfs list cryptopool/backups/test
NAME USED AVAIL REFER MOUNTPOINT
cryptopool/backups/test 31K 2.52T 31K none
$ zfs destroy cryptopool/backups/test
cannot destroy 'cryptopool/backups/test': permission denied
$
</pre>

Since the user <code>tykbackup</code> only has the permissions <code>create,mount,mountpoint,receive</code> on the <code>cryptopool/backups</code> dataset, I get Permission Denied (as I expected) when trying to destroy <code>cryptopool/backups/test</code>. Works like a charm.

To allow automatic SSH operations I add the public ssh key for the root user of the server being backed up to <code>/usr/home/tykbackup/.ssh/authorized_keys</code>:
<pre>
$ cat /usr/home/tykbackup/.ssh/authorized_keys
from="ryst.tyknet.dk",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty,command="/usr/home/tykbackup/zfscmd.sh $SSH_ORIGINAL_COMMAND" ssh-rsa AAAAB3......KR2Z root@ryst.tyknet.dk
</pre>

The script called <code>zfscmd.sh</code> is placed on the backup server to allow the ssh client to issue different command line arguments depending on what needs to be done. The script is very simple:
<pre>
#!/bin/sh
shift
/sbin/zfs $@
exit $?
</pre>

A few notes: Aside from restricting the command this SSH key can run, I've restricted it to only be able to log in from the IP of the server being backed up. These are very basic restrictions that should always be in place no matter what kind of backup you are using.

=== Add the periodic script ===
I then add the script <code>/usr/local/etc/periodic/daily/999.zfs-mirror</code> to each server being backed up with the following content:
<pre>
#!/bin/sh

### check pidfile
if [ -f /var/run/$(basename $0).pid ]; then
echo "pidfile /var/run/$(basename $0).pid exists, bailing out"
exit 1
fi
echo $$ > /var/run/$(basename $0).pid

### If there is a global system configuration file, suck it in.
if [ -r /etc/defaults/periodic.conf ]; then
. /etc/defaults/periodic.conf
source_periodic_confs
fi

case "$daily_zfs_mirror_enable" in
[Yy][Ee][Ss])
;;
*)
exit
;;
esac

pools=$daily_zfs_mirror_pools
if [ -z "$pools" ]; then
pools='tank'
fi

targethost=$daily_zfs_mirror_targethost
if [ -z "$targethost" ]; then
echo '$daily_zfs_mirror_targethost must be set in /etc/periodic.conf'
exit 1
fi

targetuser=$daily_zfs_mirror_targetuser
if [ -z "$targetuser" ]; then
echo '$daily_zfs_mirror_targetuser must be set in /etc/periodic.conf'
exit 1
fi

targetfs=$daily_zfs_mirror_targetfs
if [ -z "$targetfs" ]; then
echo '$daily_zfs_mirror_targetfs must be set in /etc/periodic.conf'
exit 1
fi

if [ -n "$daily_zfs_mirror_skip" ]; then
egrep="($(echo $daily_zfs_mirror_skip | sed "s/ /|/g"))"
fi

### get todays date for later use
tday=$(date +%Y-%m-%d)

echo -n "Doing daily ZFS mirroring - "
date

### loop through the configured pools
for pool in $pools; do
echo " Processing pool $pool ..."
### enumerate datasets with daily snapshots from today
#echo $egrep
#echo "@daily-$tday"
if [ -n "$egrep" ]; then
datasets=$(zfs list -t snapshot -o name | grep "^$pool/" | egrep -v "$egrep" | grep "@daily-$tday")
else
datasets=$(zfs list -t snapshot -o name | grep "^$pool/" | grep "@daily-$tday")
fi

echo "found datasets: $datasets"

for snapshot in $datasets; do
dataset=$(echo -n $snapshot | cut -d "@" -f 1 | cut -d "/" -f 2-)
echo "working on dataset $dataset"
### find the latest daily snapshot of this dataset on the remote node, if any
echo ssh ${targetuser}@${targethost} /usr/home/tykbackup/zfscmd.sh list -t snapshot \| grep "^${targetfs}/${dataset}@daily-" \| cut -d " " -f 1 \| tail -1
lastgoodsnap=$(ssh ${targetuser}@${targethost} /usr/home/tykbackup/zfscmd.sh list -t snapshot | grep "^${targetfs}/${dataset}@daily-" | cut -d " " -f 1 | tail -1)
if [ -z $lastgoodsnap ]; then
echo "No remote daily snapshot found for local daily snapshot $snapshot - cannot send incremental - sending full backup"
zfs send -v $snapshot | ssh ${targetuser}@${targethost} /usr/home/tykbackup/zfscmd.sh receive -v -F -u -d $targetfs
if [ $? -ne 0 ]; then
echo " Unable to send full snapshot of $dataset to $targetfs on host $targethost"
else
echo " Successfully sent a full snapshot of $dataset to $targetfs on host $targethost - future sends will be incremental"
fi
else
#check if this snapshot has already been sent for some reason, skip if so..."
temp=$(echo $snapshot | cut -d "/" -f 2-)
lastgoodsnap="$(echo $lastgoodsnap | sed "s,${targetfs}/,,")"
if [ "$temp" = "$lastgoodsnap" ]; then
echo " The snapshot $snapshot has already been sent to $targethost, skipping..."
else
### add pool name to lastgoodsnap
lastgoodsnap="${pool}/${lastgoodsnap}"
### zfs send the difference between latest remote snapshot and todays local snapshot
echo " Sending the diff between local snapshot $(hostname)@$lastgoodsnap and $(hostname)@$snapshot to ${targethost}@${targetfs} ..."
zfs send -I $lastgoodsnap $snapshot | ssh ${targetuser}@${targethost} /usr/home/tykbackup/zfscmd.sh receive -F -u -d $targetfs
if [ $? -ne 0 ]; then
echo " There was a problem sending the diff between $lastgoodsnap and $snapshot to $targetfs on $targethost"
else
echo " Successfully sent the diff between $lastgoodsnap and $snapshot to $targethost"
fi
fi
fi
done
done

### remove pidfile
rm /var/run/$(basename $0).pid
</pre>

Remember to also <code>chmod +x /usr/local/etc/periodic/daily/999.zfs-mirror</code> and enable it in <code>/etc/periodic.conf</code>:

<pre>
#daily zfs mirror
daily_zfs_mirror_enable="YES"
daily_zfs_mirror_targethost="backup.glas.tyknet.dk"
daily_zfs_mirror_targetuser="tykbackup"
daily_zfs_mirror_targetfs="cryptopool/backups/kush.tyknet.dk"
daily_zfs_mirror_pools="tank gelipool"
daily_zfs_mirror_skip="gelipool/reserved gelipool/backups" # space seperated list of datasets to skip
</pre>

=== Create the target filesystems in the backup jail ===
Finally I create the destination filesystems in the backupjail, one per server being backed up. The filesystem that needs to be created is the one specified in the setting <code>daily_zfs_mirror_targetfs</code> in <code>/etc/periodic.conf</code> on the server being backed up.

=== Run the periodic script ===
I usually to the initial run of the periodic script by hand, so I can catch and fix any errors right away. The script will loop over all datasets in the configured pools and zfs send them including their snapshots to the backup server. Next time the script runs it will send an incremental diff instead of the full dataset.

=== Caveats ===
This script does not handle deleting datasets (including their snapshots) on the backup server when the dataset is deleted from the server being backed up. You will need to do that manually. This could be considered a feature, or a missing feature, depending on your preferences. :)

= Staying up-to-date =
I update my ezjail hosts and jails to track -STABLE regularly. This section describes the procedure I use. '''It is essential that the jail host and the jails use the same world version, or bad stuff will happen.'''

== Updating the jail host ==
First I update world and kernel of the jail host like I normally would. This is described earlier in this guide, see [[Ezjail_host#Building_world_and_kernel]].

== Updating ezjails basejail ==
To update ezjails basejail located in <code>/usr/jails/basejail</code>, I run the same commands as when bootstrapping ezjail, see the section [[Ezjail_host#Bootstrapping_ezjail]].

== Running mergemaster in the jails ==
Finally, to run mergemaster in all jails I use the following script. It will run mergemaster in each jail, the script comments should explain the rest. When it is finished the jails can be started:
<pre>
#! /bin/sh

### check if .mergemasterrc exists,
### move it out of the way if so
MM_RC=0
if [ -e /root/.mergemasterrc ]; then
MM_RC=1
mv /root/.mergemasterrc /root/.mergemasterrc.old
fi

### loop through jails
for jailroot in $(ezjail-admin list | cut -c 57- | tail +3 | grep "/"); do
echo "processing ${jailroot}:"
### check if jailroot exists
if [ -n "${jailroot}" -a -d "${jailroot}" ]; then
### create .mergemasterrc
cat <<EOF > /root/.mergemasterrc
AUTO_INSTALL=yes
AUTO_UPGRADE=yes
FREEBSD_ID=yes
PRESERVE_FILES=yes
PRESERVE_FILES_DIR=/var/tmp/mergemaster/preserved-files-$(basename ${jailroot})-$(date +%y%m%d-%H%M%S)
IGNORE_FILES="/boot/device.hints /etc/motd"
EOF
### remove backup of /etc from previous run (if it exists)
if [ -d "${jailroot}/etc.bak" ]; then
rm -rfI "${jailroot}/etc.bak"
fi

### create backup of /etc as /etc.bak
cp -pRP "${jailroot}/etc" "${jailroot}/etc.bak"

### check if mtree from last mergemaster run exists
if [ ! -e ${jailroot}/var/db/mergemaster.mtree ]; then
### delete /etc/rc.d/*
rm -rfI ${jailroot}/etc/rc.d/*
fi
### run mergemaster for this jail
mergemaster -D "${jailroot}"
else
echo "${jailroot} doesn't exist"
fi
sleep 2
done

### if an existing .mergemasterrc was moved out of the way in the beginning, move it back now
if [ ${MM_RC} -eq 1 ]; then
mv /root/.mergemasterrc.old /root/.mergemasterrc
else
rm /root/.mergemasterrc
fi

### done, a bit of output
echo "Done. If everything went well the /etc.bak backup folders can be deleted now."
exit 0
</pre>

To restart all jails I run the command <code>ezjail-admin restart</code>.

= Replacing a defective disk =
I had a broken harddisk on one of my servers this evening. This section describes how I replaced the disk to make everything work again.

== Booting into the rescue system ==
After Hetzner staff physically replaced the disk my server was unable to boot because the disk that died was the first one on the controller. The cheap Hetzner hardware is unable to boot from the secondary disk, bios restrictions probably. If the other disk had broken the server would have booted fine and this whole process would be done with the server running. Anyway, I booted into the rescue system and partitioned the disk, added a bootloader and added it to the root zpool. After this I was able to boot the server normally, so the rest of the work was done without the rescue system.

== Partitioning the new disk ==
The following shows the commands I ran to partition the disk:
<pre>
[root@rescue ~]# gpart create -s GPT /dev/ad4
ad4 created
[root@rescue ~]# /sbin/gpart add -b 2048 -t freebsd-boot -s 128 /dev/ad4
ad4p1 added
[root@rescue ~]# gpart add -t freebsd-zfs -s 30G /dev/ad4
ad4p2 added
[root@rescue ~]# gpart add -t freebsd-ufs /dev/ad4
ad4p3 added
[root@rescue ~]# gpart show
=> 34 1465149101 ad6 GPT (698G)
34 2014 - free - (1M)
2048 128 1 freebsd-boot (64k)
2176 62914560 2 freebsd-zfs (30G)
62916736 1402232399 3 freebsd-ufs (668G)

=> 34 1465149101 ad4 GPT (698G)
34 2014 - free - (1M)
2048 128 1 freebsd-boot (64k)
2176 62914560 2 freebsd-zfs (30G)
62916736 1402232399 3 freebsd-ufs (668G)

[root@rescue ~]#
</pre>

== Importing the pool and replacing the disk ==

Next step is importing the zpool (remember altroot=/mnt !) and replacing the defective disk:

<pre>
[root@rescue ~]# zpool import
pool: tank
id: 3572845459378280852
state: DEGRADED
status: One or more devices are missing from the system.
action: The pool can be imported despite missing or damaged devices. The
fault tolerance of the pool may be compromised if imported.
see: http://www.sun.com/msg/ZFS-8000-2Q
config:

tank DEGRADED
mirror-0 DEGRADED
11006001397618753837 UNAVAIL cannot open
ad6p2 ONLINE
[root@rescue ~]# zpool import -o altroot=/mnt/ tank
[root@rescue ~]# zpool status
pool: tank
state: DEGRADED
status: One or more devices could not be opened. Sufficient replicas exist for
the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
see: http://www.sun.com/msg/ZFS-8000-2Q
scan: scrub repaired 0 in 0h2m with 0 errors on Thu Nov 1 05:00:49 2012
config:

NAME STATE READ WRITE CKSUM
tank DEGRADED 0 0 0
mirror-0 DEGRADED 0 0 0
11006001397618753837 UNAVAIL 0 0 0 was /dev/ada0p2
ad6p2 ONLINE 0 0 0

errors: No known data errors
[root@rescue ~]# zpool replace tank 11006001397618753837 ad4p2

Make sure to wait until resilver is done before rebooting.

If you boot from pool 'tank', you may need to update
boot code on newly attached disk 'ad4p2'.

Assuming you use GPT partitioning and 'da0' is your new boot disk
you may use the following command:

gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da0

[root@rescue ~]#
[root@rescue ~]# zpool status
pool: tank
state: DEGRADED
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Tue Nov 27 00:24:41 2012
823M scanned out of 3.11G at 45.7M/s, 0h0m to go
823M resilvered, 25.88% done
config:

NAME STATE READ WRITE CKSUM
tank DEGRADED 0 0 0
mirror-0 DEGRADED 0 0 0
replacing-0 UNAVAIL 0 0 0
11006001397618753837 UNAVAIL 0 0 0 was /dev/ada0p2
ad4p2 ONLINE 0 0 0 (resilvering)
ad6p2 ONLINE 0 0 0

errors: No known data errors
[root@rescue ~]# zpool status
pool: tank
state: ONLINE
scan: resilvered 3.10G in 0h2m with 0 errors on Tue Nov 27 01:26:45 2012
config:

NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
ada0p2 ONLINE 0 0 0
ada1p2 ONLINE 0 0 0

errors: No known data errors
[root@rescue ~]#
</pre>

== Reboot into non-rescue system ==
At this point I rebooted the machine into the normal FreeBSD system.

== Re-create geli partition ==
To recreate the geli partition on p3 of the new disk, I just follow the same steps as when I originally created it, [http://wiki.tyk.nu/index.php?title=Ezjail_host#Create_GELI_volumes more info here].

To attach the new geli volume I run <code>geli attach</code> as [http://wiki.tyk.nu/index.php?title=Ezjail_host#Attach_GELI_volumes described here].

== Add the geli device to the encrypted zpool ==
First I check that both geli devices are available, and I check the device name that needs replacing in <code>zpool status</code> output:

<pre>
[tykling@haze ~]$ geli status
Name Status Components
ada1p3.eli ACTIVE ada1p3
ada0p3.eli ACTIVE ada0p3
</pre>

<pre>
[tykling@haze ~]$ zpool status gelipool
pool: gelipool
state: DEGRADED
status: One or more devices has been removed by the administrator.
Sufficient replicas exist for the pool to continue functioning in a
degraded state.
action: Online the device using 'zpool online' or replace the device with
'zpool replace'.
scan: scrub repaired 68K in 0h28m with 0 errors on Thu Nov 1 05:58:08 2012
config:

NAME STATE READ WRITE CKSUM
gelipool DEGRADED 0 0 0
mirror-0 DEGRADED 0 0 0
18431995264718840299 REMOVED 0 0 0 was /dev/ada0p3.eli
ada1p3.eli ONLINE 0 0 0

errors: No known data errors
[tykling@haze ~]$
</pre>

To replace the device and begin resilvering:

<pre>
[tykling@haze ~]$ sudo zpool replace gelipool 18431995264718840299 ada0p3.eli
Password:
[tykling@haze ~]$ zpool status
pool: gelipool
state: DEGRADED
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Tue Nov 27 00:53:40 2012
759M scanned out of 26.9G at 14.6M/s, 0h30m to go
759M resilvered, 2.75% done
config:

NAME STATE READ WRITE CKSUM
gelipool DEGRADED 0 0 0
mirror-0 DEGRADED 0 0 0
replacing-0 REMOVED 0 0 0
18431995264718840299 REMOVED 0 0 0 was /dev/ada0p3.eli/old
ada0p3.eli ONLINE 0 0 0 (resilvering)
ada1p3.eli ONLINE 0 0 0

errors: No known data errors
[tykling@haze ~]$
</pre>

When the resilver is finished, the system is good as new.

Ezjail host

2015-10-26T10:34:19Z

Tykling: /* Syslog jail */

= Background =
This is my personal checklist for when I am setting up a new ezjail host. I like my jail hosts configured in a very specific way. There is a good chance that what is right for me is not right for you. As always, YMMV.

Also note that I talk a lot about the German hosting provider Hetzner, if you are using another provider or you are doing this at home, just ignore the Hetzner specific stuff. Much of the content here can be used with little or no changes outside Hetzner.

= Installation =
== OS install with mfsbsd ==
After receiving the server from Hetzner I boot it using the rescue system which puts me at an mfsbsd prompt via SSH. This is perfect for installing a zfs-only server.

=== Changes to zfsinstall ===
I edit the ''zfsinstall'' script <code>/root/bin/zfsinstall</code> and add "usr" to ''FS_LIST'' near the top of the script. I do this because I like to have /usr as a seperate ZFS dataset.

=== Check disks ===
I create a small zpool using just 30gigs, enough to confortably install the base OS and so on. The rest of the diskspace will be used for GELI which will have the other zfs pool on top. This encrypted zpool will house the actual jails and data. This setup allows me to have all the important data encrypted, while allowing the physical server to boot without human intervention like full disk encryption would require.

Note that the disks in this server are not new, they have been used for around two years (18023 hours/24 = 702 days):
<pre>
[root@rescue ~]# grep "ada[0-9]:" /var/run/dmesg.boot | grep "MB "
ada0: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada1: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
[root@rescue ~]# smartctl -a /dev/ada0 | grep Power_On_Hours
9 Power_On_Hours 0x0032 096 096 000 Old_age Always - 18023
[root@rescue ~]# smartctl -a /dev/ada1 | grep Power_On_Hours
9 Power_On_Hours 0x0032 096 096 000 Old_age Always - 18023
[root@rescue ~]#
</pre>

=== Destroy existing partitions ===
Any existing partitions need to be deleted first. This can be done with the ''destroygeom'' command like shown below:
<pre>
[root@rescue ~]# destroygeom -d ada0 -d ada1
Destroying geom ada0:
Deleting partition 3 ... done
Destroying geom ada1:
Deleting partition 1 ... done
Deleting partition 2 ... done
Deleting partition 3 ... done
</pre>

=== Install FreeBSD ===
Installing FreeBSD with mfsbsd is easy. I run the below command, adjusting the release I want to install of course:

<pre>
[root@rescue ~]# zfsinstall -d ada0 -d ada1 -r mirror -z 30G -t /nfs/mfsbsd/10.0-release-amd64.tbz
Creating GUID partitions on ada0 ... done
Configuring ZFS bootcode on ada0 ... done
=> 34 3907029101 ada0 GPT (1.8T)
34 2014 - free - (1M)
2048 128 1 freebsd-boot (64k)
2176 62914560 2 freebsd-zfs (30G)
62916736 3844112399 - free - (1.8T)

Creating GUID partitions on ada1 ... done
Configuring ZFS bootcode on ada1 ... done
=> 34 3907029101 ada1 GPT (1.8T)
34 2014 - free - (1M)
2048 128 1 freebsd-boot (64k)
2176 62914560 2 freebsd-zfs (30G)
62916736 3844112399 - free - (1.8T)

Creating ZFS pool tank on ada0p2 ada1p2 ... done
Creating tank root partition: ... done
Creating tank partitions: var tmp usr ... done
Setting bootfs for tank to tank/root ... done
NAME USED AVAIL REFER MOUNTPOINT
tank 270K 29.3G 31K none
tank/root 127K 29.3G 34K /mnt
tank/root/tmp 31K 29.3G 31K /mnt/tmp
tank/root/usr 31K 29.3G 31K /mnt/usr
tank/root/var 31K 29.3G 31K /mnt/var
Extracting FreeBSD distribution ... done
Writing /boot/loader.conf... done
Writing /etc/fstab...Writing /etc/rc.conf... done
Copying /boot/zfs/zpool.cache ... done

Installation complete.
The system will boot from ZFS with clean install on next reboot

You may type "chroot /mnt" and make any adjustments you need.
For example, change the root password or edit/create /etc/rc.conf for
for system services.

WARNING - Don't export ZFS pool "tank"!
[root@rescue ~]#
</pre>

=== Post install configuration (before reboot) ===
Before rebooting into the installed FreeBSD I need to make certain I can reach the server through SSH after the reboot. This means:
# Adding network settings to <code>/etc/rc.conf</code>
# Adding sshd_enable="YES" to <code>/etc/rc.conf</code>
# Change ''PermitRootLogin'' to ''Yes'' in <code>/etc/ssh/sshd_config</code> ''Note: In the current This is now the default in the zfsinstall image that Hetzner provides''
# Add nameservers to <code>/etc/resolv.conf</code>
# Finally I set the root password.

'''All of these steps are essential if I am going to have any chance of logging in after reboot.''' Most of these changes can be done from the mfsbsd shell but the password change requires ''chroot'' into the newly installed environment.

I use the ''chroot'' command but start another shell as bash is not installed in /mnt:
<pre>
[root@rescue ~]# chroot /mnt/ csh
rescue# ee /etc/rc.conf
rescue# ee /etc/ssh/sshd_config
rescue# passwd
New Password:
Retype New Password:
rescue#
</pre>

So, the network settings are sorted, root password is set, and root is permitted to ssh in. Time to reboot (this is the exciting part).

'''Remember to use <code>shutdown -r now</code> and not <code>reboot</code> when you reboot. <code>shutdown -r now</code> performs the proper shutdown process including rc.d scripts and disk buffer flushing. <code>reboot</code> is the "bigger hammer" to use when something is preventing <code>shutdown -r now</code> from working.

== Basic config after first boot ==
If the server boots without any problems, I do some basic configuration before I continue with the disk partitioning.

=== Timezone ===
I run the command <code>tzsetup</code> to set the proper timezone, and set the time using <code>ntpdate</code> if neccesary.

''Note: The current hetzner freebsd image has the timezone set to CEST, I like my servers configured as UTC''

=== Basic ports ===
I also add some basic ports with <code>pkg</code> so I can get screen etc. up and running as soon as possible:
<pre>
# pkg install bash screen sudo portmaster
</pre>

I then add the following to <code>/usr/local/etc/portmaster.rc</code>:
<pre>
ALWAYS_SCRUB_DISTFILES=dopt
PM_DEL_BUILD_ONLY=pm_dbo
SAVE_SHARED=wopt
PM_LOG=/var/log/portmaster.log
PM_IGNORE_FAILED_BACKUP_PACKAGE=pm_ignore_failed_backup_package
</pre>
An explanation of these options can be found on the [[Portmaster]] page.

After a <code>rehash</code> and adding my non-root user with <code>adduser</code>, I am ready to continue with the disk configuration. I also remember to disable root login in <code>/etc/ssh/sshd_config</code>.

== Further disk configuration ==
After the reboot into the installed FreeBSD environment, I need to do some further disk configuration.
=== Create swap partitions ===
Swap-on-zfs is not a good idea for various reasons. To keep my swap encrypted but still off zfs I use geli onetime encryption. To avoid problems if a disk dies I also use gmirror. First I add the partitions with gpart:

<pre>
$ sudo gpart add -t freebsd-swap -s 10G /dev/ada0
ada0p3 added
$ sudo gpart add -t freebsd-swap -s 10G /dev/ada1
ada1p3 added
$
</pre>

Then I make sure gmirror is loaded, and loaded on boot:
<pre>
$ sudo sysrc -f /boot/loader.conf geom_mirror_load="YES"
$ sudo kldload geom_mirror
</pre>

Then I create the gmirror:
<pre>
sudo gmirror label swapmirror /dev/ada0p3 /dev/ada1p3
</pre>

Finally I add the following line to <code>/etc/fstab</code> to get encrypted swap on top of the gmirror:
<pre>
/dev/mirror/swapmirror.eli none swap sw,keylen=256,sectorsize=4096 0 0
</pre>

I can enable the new swap partition right away:
<pre>
$ sudo swapon /dev/mirror/swapmirror.eli
$ swapinfo
Device 1K-blocks Used Avail Capacity
/dev/mirror/swapmirror.eli 8388604 0 8388604 0%
$
</pre>

=== Create GELI partitions ===
First I create the partitions to hold the geli devices:
<pre>
$ sudo gpart add -t freebsd-ufs ada0
ada0p4 added
$ sudo gpart add -t freebsd-ufs ada1
ada1p4 added
</pre>

I add them as <code>freebsd-ufs</code> type partitions, as there is no dedicated <code>freebsd-geli</code> type.

=== Create GELI key ===
To create a GELI key I copy some data from <code>/dev/random</code>:
<pre>
$ sudo dd if=/dev/random of=/root/geli.key bs=256k count=1
1+0 records in
1+0 records out
262144 bytes transferred in 0.003347 secs (78318372 bytes/sec)
$
</pre>

=== Create GELI volumes ===
I create the GELI volumes with 4k blocksize and 256bit AES encryption:

<pre>
$ sudo geli init -s 4096 -K /root/geli.key -l 256 /dev/ada0p4
Enter new passphrase:
Reenter new passphrase:

Metadata backup can be found in /var/backups/ada0p4.eli and
can be restored with the following command:

# geli restore /var/backups/ada0p4.eli /dev/ada0p4

$ sudo geli init -s 4096 -K /root/geli.key -l 256 /dev/ada1p4
Enter new passphrase:
Reenter new passphrase:

Metadata backup can be found in /var/backups/ada1p4.eli and
can be restored with the following command:

# geli restore /var/backups/ada1p4.eli /dev/ada1p4

$
</pre>

=== Enable AESNI ===
Most Intel CPUs have hardware acceleration of AES which helps a lot with GELI performance. I load the <code>aesni</code> module during boot from <code>/boot/loader.conf</code>:
<pre>
$ sudo sysrc -f /boot/loader.conf aesni_load="YES"
</pre>

=== Attach GELI volumes ===
Now I just need to attach the GELI volumes before I am ready to create the second zpool:
<pre>
$ sudo geli attach -k /root/geli.key /dev/ada0p4
Enter passphrase:
$ sudo geli attach -k /root/geli.key /dev/ada1p4
Enter passphrase:
$
</pre>

=== Create second zpool ===
<pre>
$ sudo zpool create gelipool mirror /dev/ada0p4.eli /dev/ada1p4.eli
$ zpool status
pool: gelipool
state: ONLINE
scan: none requested
config:

NAME STATE READ WRITE CKSUM
gelipool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
ada0p4.eli ONLINE 0 0 0
ada1p4.eli ONLINE 0 0 0

errors: No known data errors

pool: tank
state: ONLINE
scan: none requested
config:

NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
ada0p2 ONLINE 0 0 0
ada1p2 ONLINE 0 0 0

errors: No known data errors
$
</pre>

=== Create ZFS filesystems on the new zpool ===
The last remaining thing is to create a filesystem in the new zfs pool:
<pre>
$ zfs list
NAME USED AVAIL REFER MOUNTPOINT
gelipool 624K 3.54T 144K /gelipool
tank 704M 28.6G 31K none
tank/root 704M 28.6G 413M /
tank/root/tmp 38K 28.6G 38K /tmp
tank/root/usr 291M 28.6G 291M /usr
tank/root/var 505K 28.6G 505K /var
$ sudo zfs set mountpoint=none gelipool
$ sudo zfs set compression=on gelipool
$ sudo zfs create -o mountpoint=/usr/jails gelipool/jails
$ zfs list
NAME USED AVAIL REFER MOUNTPOINT
gelipool 732K 3.54T 144K none
gelipool/jails 144K 3.54T 144K /usr/jails
tank 704M 28.6G 31K none
tank/root 704M 28.6G 413M /
tank/root/tmp 38K 28.6G 38K /tmp
tank/root/usr 291M 28.6G 291M /usr
tank/root/var 505K 28.6G 505K /var
$
</pre>

=== Disable atime ===
One last thing I like to do is to disable <code>atime</code> or ''access time'' on the filesystem. Access times are recorded every time a file is read, and while this can have it's use cases, I never use it. Disabling it means a lot fewer write operations, as a read operation doesn't automatically include a write operation when <code>atime</code> is disabled. Disabling it is easy:

<pre>
$ sudo zfs set atime=off tank
$ sudo zfs set atime=off gelipool
$
</pre>

The next things are post-install configuration stuff like OS upgrade, ports, firewall and so on. The basic install is finished \o/

=== Reserved space ===
Running out of space in ZFS is bad. Stuff will run slowly and may stop working entirely until some space is freed. The problem is that ZFS is a journalled filesystem which means that all writes, even a deletion, requires writing data to the disk. I've more than once wound up in a situation where I couldn't delete file to free up diskspace because the disk was full.

Sometimes this can be resolved by overwriting a large file, like:
<pre>
$ echo > /path/to/a/large/file
</pre>

This will overwrite the file thereby freeing up some space, but sometimes even this is not possible. This is where reserved space comes in. I create a new filesystem in each pool, set them readonly and without a mountpoint, and with 1G reserved each:
<pre>
$ sudo zfs create -o mountpoint=none -o reservation=1G -o readonly=on gelipool/reserved
$ sudo zfs create -o mountpoint=none -o reservation=1G -o readonly=on tank/reserved
</pre>

If I run out of space for some reason, I can just delete the dataset, or unset the <code>reserved</code> property, and I immediately have 1G diskspace available. Yay!

= Ports =
== Installing the ports tree ==
I need to bootstrap the ports system, I use <code>portsnap</code> as it is way faster than using c(v)sup. Initially I run <code>portsnap fetch extract</code> and when I need to update the tree later I use <code>portsnap fetch update</code>.

== smartd ==
I install smartd to monitor the disks for problems:
<pre>
$ sudo pkg install smartmontools
</pre>

I create the file <code>/usr/local/etc/smartd.conf</code> and add this line to it:
<pre>
DEVICESCAN -a -m thomas@gibfest.dk
</pre>

This makes smartd monitor all disks and send me an email if it finds an error.

Remember to enable smartd in <code>/etc/rc.conf</code> and start it:
<pre>
sudo sysrc smartd_enable="YES"
sudo service smartd start
</pre>

== openntpd ==
I install <code>net/openntpd</code> to keep the clock in sync. I find this a lot easier to configure than the base ntpd.

<pre>
sudo pkg install openntpd
</pre>

I enable <code>openntpd</code> in <code>/etc/rc.conf</code>:

<pre>
sudo sysrc openntpd_enable="YES"
</pre>

and add a one line config file:
<pre>
$ grep -v "^#" /usr/local/etc/ntpd.conf | grep -v "^$"
servers de.pool.ntp.org
$
</pre>

sync the clock and start openntpd:

<pre>
sudo ntpdate de.pool.ntp.org
sudo service openntpd start
</pre>

== ntpdate ==
I also enable <code>ntpdate</code> to help set the clock after a reboot. I add the following two lines to <code>/etc/rc.conf</code>:
<pre>
sudo sysrc ntpdate_enable="YES"
sudo sysrc ntpdate_hosts="de.pool.ntp.org"
</pre>

= Upgrade OS (buildworld) =
I usually run -STABLE on my hosts, which means I need to build and install a new world and kernel. I also like having [http://www.freebsd.org/cgi/man.cgi?query=rctl rctl] available on my jail hosts, so I can limit jail ressources in all kinds of neat ways. I also like having DTRACE available. Additionally I also need the built world to populate ezjails basejail.

'''Note''': I will need to update the host and the jails many times during the lifespan of this server, which is likely > 2-3 years. As new security problems are found or features are added that I want, I will update host and jails. There is a section about staying up to date later in this page. This section (the one you are reading now) only covers the OS update I run right after installing the server.

== Fetching sources ==
I install <code>subversion</code>:
<pre>
sudo pkg install subversion
</pre>

and then get the sources:
<pre>
sudo svn checkout https://svn0.eu.freebsd.org/base/stable/10/ /usr/src/
Error validating server certificate for 'https://svn0.eu.freebsd.org:443':
- The certificate is not issued by a trusted authority. Use the
fingerprint to validate the certificate manually!
Certificate information:
- Hostname: svnmir.bme.FreeBSD.org
- Valid: from Jun 29 12:24:17 2013 GMT until Jun 29 12:24:17 2015 GMT
- Issuer: clusteradm, FreeBSD.org, CA, US(clusteradm@FreeBSD.org)
- Fingerprint: 39:B0:53:35:CE:60:C7:BB:00:54:96:96:71:10:94:BB:CE:1C:07:A7
(R)eject, accept (t)emporarily or accept (p)ermanently? p
A /usr/src/sys
A /usr/src/sys/arm
A /usr/src/sys/arm/ti
A /usr/src/sys/arm/ti/cpsw
A /usr/src/sys/arm/ti/am335x
A /usr/src/sys/arm/ti/usb
A /usr/src/sys/arm/ti/twl
A /usr/src/sys/arm/ti/ti_mmchs.c
A /usr/src/sys/arm/ti/ti_cpuid.h
A /usr/src/sys/arm/ti/ti_mmchs.h
A /usr/src/sys/arm/ti/ti_i2c.c
A /usr/src/sys/arm/ti/ti_i2c.h
A /usr/src/sys/arm/ti/am335x/am335x_pwm.c
A /usr/src/sys/arm/ti/ti_sdma.c
<snip>
A /usr/src/sys/fs/nandfs/nandfs_cleaner.c
A /usr/src/sys/fs/nandfs/nandfs_bmap.c
A /usr/src/sys/fs/nandfs/bmap.c
A /usr/src/sys/fs/nandfs/nandfs_subr.c
A /usr/src/sys/fs/nandfs/bmap.h
A /usr/src/sys/fs/nandfs/nandfs_vfsops.c
A /usr/src/sys/fs/nandfs/nandfs_sufile.c
U /usr/src
Checked out revision 282899.
$
</pre>

This takes a while the first time, but subsequent runs are much faster.

'''Note:''' The reason I use the full Subversion port instead of just svnup is that using full SVN client means that you get the SVN revision in <code>uname -a</code> when checking out code with the full SVN client.

== Create kernel config ==
After the sources finish downloading, I create a new kernel config file <code>/etc/TYKJAIL</code> with the following content:
<pre>
include GENERIC
ident TYKJAIL

#rctl
options RACCT
options RCTL
</pre>

I then create a symlink to the kernel config file in /etc/:
<pre>
# ln -s /etc/TYKJAIL /usr/src/sys/amd64/conf/
# ls -l /usr/src/sys/amd64/conf/TYKJAIL
lrwxr-xr-x 1 root wheel 9 Jul 22 16:14 /usr/src/sys/amd64/conf/TYKJAIL -> /etc/TYKJAIL
</pre>

Finally I enable the kernel config in <code>/etc/make.conf</code>:
<pre>
$ cat /etc/make.conf
KERNCONF=TYKJAIL
</pre>

== Building world and kernel ==
Finally I start the build. I use -j to start one thread per core in the system. <code>sysctl hw.ncpu</code> shows the number of available cores:
<pre>
# sysctl hw.ncpu
hw.ncpu: 12
</pre>

To build the new system:
<pre>
# cd /usr/src/
# time (sudo make -j$(sysctl -n hw.ncpu) buildworld && sudo make -j$(sysctl -n hw.ncpu) kernel) && date
</pre>

After the build finishes, reboot and run mergemaster, installworld, and mergemaster again:
<pre>
# cd /usr/src/
# sudo mergemaster -pFUi && sudo make installworld && sudo mergemaster -FUi
</pre>

'''DO NOT OVERWRITE /etc/group AND /etc/master.passwd AND OTHER CRITICAL FILES!'''

Reboot after the final mergemaster completes, and boot into the newly built world.

= Preparing ezjail =
ezjail needs to be installed and a bit of configuration is also needed, in addition to bootstrapping <code>/usr/jails/basejail</code> and <code>/usr/jails/newjail</code>.

== Installing ezjail ==
Just install it with pkg:
<pre>
sudo pkg install ezjail
</pre>

== Configuring ezjail ==
Then I go edit the ezjail config file <code>/usr/local/etc/ezjail.conf</code> and add/change these three lines near the bottom:
<pre>
ezjail_use_zfs="YES"
ezjail_jailzfs="gelipool/jails"
ezjail_use_zfs_for_jails="YES"
</pre>

This makes ezjail use seperate zfs datasets under <code>gelipool/jails</code> for the <code>basejail</code> and <code>newjail</code>, as well as for each jail created. <code>ezjail_use_zfs_for_jails</code> is supported since ezjail 3.2.2.

== Bootstrapping ezjail ==
Finally I populate <code>basejail</code> and <code>newjail</code> from the world I build earlier:
<pre>
$ sudo ezjail-admin update -i
</pre>

The last line of the output is a message saying:
<pre>
Note: a non-standard /etc/make.conf was copied to the template jail in order to get the ports collection running inside jails.
</pre>

This is because ezjail defaults to symlinking the ports collection in the same way it symlinks the basejail. I prefer having seperate/individual ports collections in each of my jails though, so I remove the symlink and make.conf from newjail:
<pre>
$ sudo rm /usr/jails/newjail/etc/make.conf /usr/jails/newjail/usr/ports /usr/jails/newjail/usr/src
$ sudo mkdir /usr/jails/newjail/usr/src
</pre>

== ZFS goodness ==
Note that ezjail has created two new ZFS datasets to hold <code>basejail</code> and <code>newjail</code>:
<pre>
$ zfs list -r gelipool/jails
NAME USED AVAIL REFER MOUNTPOINT
gelipool/jails 239M 3.54T 476K /usr/jails
gelipool/jails/basejail 236M 3.54T 236M /usr/jails/basejail
gelipool/jails/newjail 3.10M 3.54T 3.10M /usr/jails/newjail
</pre>

== ezjail flavours ==
ezjail has a pretty awesome feature that makes it possible to create templates or ''flavours'' which apply common settings when creating a new jail. I always have a ''basic'' flavour which adds a user for me, installs an SSH key, adds a few packages like bash, screen, sudo and portmaster - and configures those packages. Basically, everything I find myself doing over and over again every time I create a new jail.

It is also possible, of course, to create more advanced flavours, I've had one that installs a complete nginx+php-fpm server with all the neccesary packages and configs.

ezjail flavours are technically pretty simple. By default, they are located in the same place as ''basejail'' and ''newjail'', and ezjail comes with an example flavour to get you started. Basically a flavour is a file/directory hierachy which is copied to the jail, and a shell script called ''ezjail.flavour'' which is run once, the first time the jail is started, and then deleted.

For reference, I've included my basic flavour here. First is a listing of the files included in the flavour, and then the ''ezjail.flavour'' script which performs tasks beyond copying config files.

<pre>
$ find /usr/jails/flavours/tykbasic
/usr/jails/flavours/tykbasic
/usr/jails/flavours/tykbasic/ezjail.flavour
/usr/jails/flavours/tykbasic/usr
/usr/jails/flavours/tykbasic/usr/local
/usr/jails/flavours/tykbasic/usr/local/etc
/usr/jails/flavours/tykbasic/usr/local/etc/portmaster.rc
/usr/jails/flavours/tykbasic/usr/local/etc/sudoers
/usr/jails/flavours/tykbasic/usr/home
/usr/jails/flavours/tykbasic/usr/home/tykling
/usr/jails/flavours/tykbasic/usr/home/tykling/.ssh
/usr/jails/flavours/tykbasic/usr/home/tykling/.ssh/authorized_keys
/usr/jails/flavours/tykbasic/usr/home/tykling/.screenrc
/usr/jails/flavours/tykbasic/etc
/usr/jails/flavours/tykbasic/etc/fstab
/usr/jails/flavours/tykbasic/etc/rc.conf
/usr/jails/flavours/tykbasic/etc/periodic.conf
/usr/jails/flavours/tykbasic/etc/resolv.conf
</pre>

As you can see, the flavour contains files like ''/etc/resolv.conf'' and other stuff to make the jail work. The name of the flavour here is ''tykbasic'' which means that if I want a file to end up in ''/usr/home/tykling'' after the flavour has been applied, I need to put that file in the folder ''/usr/jails/flavours/tykbasic/usr/home/tykling/'' - '''remember to also chown the files in the flavour appropriately'''.

Finally, my ''ezjail.flavour'' script looks like so:

<pre>
#!/bin/sh
#
# BEFORE: DAEMON
#
# ezjail flavour example

# Timezone
###########
#
ln -s /usr/share/zoneinfo/Europe/Copenhagen /etc/localtime

# Groups
#########
#
pw groupadd -q -n tykling

# Users
########
#
# To generate a password hash for use here, do:
# openssl passwd -1 "the password"
echo -n '$1$L/fC0UrO$bi65/BOIAtMkvluDEDCy31' | pw useradd -n tykling -u 1001 -s /bin/sh -m -d /usr/home/tykling -g tykling -c 'tykling' -H 0

# Packages
###########
#
env ASSUME_ALWAYS_YES=YES pkg bootstrap
pkg install -y bash
pkg install -y sudo
pkg install -y portmaster
pkg install -y screen

#change shell to bash
chsh -s bash tykling

#update /etc/aliases
echo "root: thomas@gibfest.dk" >> /etc/aliases
newaliases

#remove adjkerntz from crontab
cat /etc/crontab | grep -E -v "(Adjust the time|adjkerntz)" > /etc/crontab.new
mv /etc/crontab.new /etc/crontab

#remove ports symlink
rm /usr/ports

# create symlink to /usr/home in / (adduser defaults to /usr/username as homedir)
ln -s /usr/home /home
</pre>

Creating a flavour is easy: just create a folder under <code>/usr/jails/flavours/</code> that has the name of the flavour, and start adding files and folders there. The ezjail.flavour script should be placed in the root (see the example further up the page).

Finally I add the following to <code>/usr/local/etc/ezjail.conf</code> to make ezjail always use my new flavour:
<pre>
ezjail_default_flavour="tykbasic"
</pre>

= Configuration =
This section outlines what I do to further prepare the machine to be a nice ezjail host.

== Firewall ==
One of the first things I fix is to enable the pf firewall from OpenBSD. I add the following to <code>/etc/rc.conf</code> to enable pf at boot time:
<pre>
sudo sysrc pf_enable="YES"
sudo sysrc pflog_enable="YES"
</pre>

I also create a very basic <code>/etc/pf.conf</code>:
<pre>
[root@ ~]# cat /etc/pf.conf
### macros
if="em0"
table <portknock> persist

#external addresses
tykv4="a.b.c.d"
tykv6="2002:ab:cd::/48"
table <allowssh> { $tykv4,$tykv6 }

#local addresses
glasv4="w.x.y.z"

### scrub
scrub in on $if all fragment reassemble

################
### filtering
### block everything
block log all

################
### skip loopback interface(s)
set skip on lo0

################
### icmp6
pass in quick on $if inet6 proto icmp6 all icmp6-type {echoreq,echorep,neighbradv,neighbrsol,routeradv,routersol}

################
### pass outgoing
pass out quick on $if all

################
### portknock rule (more than 5 connections in 10 seconds to the port specified will add the "offending" IP to the <portknock> table)
pass in quick on $if inet proto tcp from any to $glasv4 port 32323 synproxy state (max-src-conn-rate 5/10, overload <portknock>)

### pass incoming ssh and icmp
pass in quick on $if proto tcp from { <allowssh>, <portknock> } to ($if) port 22
pass in quick on $if inet proto icmp all icmp-type { 8, 11 }

################
### pass ipv6 fragments (hack to workaround pf not handling ipv6 fragments)
pass in on $if inet6
block in log on $if inet6 proto udp
block in log on $if inet6 proto tcp
block in log on $if inet6 proto icmp6
block in log on $if inet6 proto esp
block in log on $if inet6 proto ipv6
</pre>

To load pf without rebooting I run the following:
<pre>
[root@ ~]# kldload pf
[root@ ~]# kldload pflog
[root@ ~]# pfctl -ef /etc/pf.conf && sleep 60 && pfctl -d
No ALTQ support in kernel
ALTQ related functions disabled
</pre>

I get no prompt after this because pf has cut my SSH connection. But I can SSH back in if I did everything right, and if not, I can just wait 60 seconds after which pf will be disabled again. I SSH in and reattach to the screen I am running this in, and press control-c, so the "sleep 60" is interrupted and pf is not disabled. Neat little trick for when you want to avoid locking yourself out :)

== Replacing sendmail with Postfix ==
I always replace Sendmail with Postfix on every server I manage. See [[Replacing_Sendmail_With_Postfix]] for more info.

== Listening daemons ==
When you add an IP alias for a jail, any daemons listening on * will also listen on the jails IP, which is not what I want. For example, I want the jails sshd to be able to listen on the jails IP on port 22, instead of the hosts sshd. Check for listening daemons like so:
<pre>
$ sockstat -l46
USER COMMAND PID FD PROTO LOCAL ADDRESS FOREIGN ADDRESS
root master 1554 12 tcp4 *:25 *:*
root master 1554 13 tcp6 *:25 *:*
root sshd 948 3 tcp6 *:22 *:*
root sshd 948 4 tcp4 *:22 *:*
root syslogd 789 6 udp6 *:514 *:*
root syslogd 789 7 udp4 *:514 *:*
[tykling@glas ~]$
</pre>

This tells me that I need to change Postfix, sshd and syslogd to stop listening on all IP addresses.
=== Postfix ===
The defaults in Postfix are really nice on FreeBSD, and most of the time a completely empty config file is fine for a system mailer (sendmail replacement). However, to make Postfix stop listening on port 25 on all IP addresses, I do need one line in <code>/usr/local/etc/postfix/main.cf</code>:
<pre>
$ cat /usr/local/etc/postfix/main.cf
inet_interfaces=localhost
$
</pre>

=== sshd ===
To make <code>sshd</code> stop listening on all IP addresses I uncomment and edit the <code>ListenAddress</code> line in <code>/etc/ssh/sshd_config</code>:
<pre>
$ grep ListenAddress /etc/ssh/sshd_config
ListenAddress x.y.z.226
#ListenAddress ::
</pre>
(IP address obfuscated..)

=== syslogd ===
I don't need my syslogd to listen on the network at all, so I add the following line to <code>/etc/rc.conf</code>:
<pre>
$ grep syslog /etc/rc.conf
syslogd_flags="-ss"
</pre>

=== Restarting services ===
Finally I restart Postfix, sshd and syslogd:
<pre>
$ sudo /etc/rc.d/syslogd restart
Stopping syslogd.
Waiting for PIDS: 789.
Starting syslogd.
$ sudo /etc/rc.d/sshd restart
Stopping sshd.
Waiting for PIDS: 948.
Starting sshd.
$ sudo /usr/local/etc/rc.d/postfix restart
postfix/postfix-script: stopping the Postfix mail system
postfix/postfix-script: starting the Postfix mail system
</pre>

A check with <code>sockstat</code> reveals that no more services are listening on all IP addresses:
<pre>
$ sockstat -l46
USER COMMAND PID FD PROTO LOCAL ADDRESS FOREIGN ADDRESS
root master 1823 12 tcp4 127.0.0.1:25 *:*
root master 1823 13 tcp6 ::1:25 *:*
root sshd 1617 3 tcp4 x.y.z.226:22 *:*
$
</pre>

== Network configuration ==
Network configuration is a big part of any jail setup. If I have enough IP addresses (ipv4 and ipv6) I can just add IP aliases as needed. If I only have one or a few v4 IPs I will need to use rfc1918 addresses for the jails. In that case, I create a new loopback interface, <code>lo1</code> and add the IP aliases there. I then use the pf firewall to redirect incoming traffic to the right jail, depending on the port in use.

=== IPv4 ===
If rfc1918 jails are needed, I add the following to <code>/etc/rc.conf</code> to create the lo1 interface on boot:
<pre>
### lo1 interface for ipv4 rfc1918 jails
cloned_interfaces="lo1"
</pre>

When the lo1 interface is created, or if it isn't needed, I am ready to start adding IP aliases for jails as needed.

=== IPv6 ===
On the page [[Hetzner_ipv6]] I've explained how to make IPv6 work on a Hetzner server where the supplied IPv6 default gateway is outside the IPv6 subnet assigned.

When basic IPv6 connectivity works, I am ready to start adding IP aliases for jails as needed.

=== Allow ping from inside jails ===
I add the following to <code>/etc/sysctl.conf</code> so the jails are allowed to do icmp ping. This enables raw socket access, which can be a security issue if you have untrusted root users in your jails. Use with caution.

<pre>
#allow ping in jails
security.jail.allow_raw_sockets=1
</pre>

== Tips & tricks ==
=== Get jail info out of ''top'' ===
To make top show the jail id of the jail in which the process is running in a column, I need to specify the -j flag to top. Since this is a multi-cpu server I am working on, I also like giving the -P flag to top, to get a seperate line of cpu stats per core. Finally, I like -a to get the full commandline/info of the running processes. I add the following to my .bashrc in my homedir on the jail host:
<pre>
alias top="nice top -j -P -a"
</pre>

...this way I don't have to remember passing <code>-j -P -a</code> to top every time. Also, I've been told to run top with ''nice'' to limit the cpu used by top itself. I took the advice so the complete alias looks like above.

= Base jails =
My jails need various services, so:
* I have a DNS jail with a caching DNS server which is used by all the jails.
* I also have a syslog server on each jailhost to collect syslog from all the jails and send them to my central syslog server.
* I have a postgres jail on each jail host, so I only need to maintain one postgres server.
* I have a reverse webproxy jail since many of my jails serve some sort of web application, and I like to terminate SSL for those in one place in an attempt to keep the individual jails as simple as possible.

This section describes how I configure the postgres and web jails that provide services to the other jails.

== DNS jail ==
I don't need a public v4 IP for this jail, so I configure it with an RFC1918 v4 IP on a loopback interface, and of course a real IPv6 address.

I am used to using bind but unbound is just as good. Use whatever you are comfortable with. Make sure you permit DNS traffic from the other jails to the DNS jail.

This jail needs to be started first since the rest of the jails need it for DNS. ezjail runs rcorder on the config files in <code>/usr/local/etc/ezjail</code> which means I can use the normal <code>PROVIDE:</code> and <code>REQUIRE:</code> to control the jail dependencies. I change the ezjail config for my DNS jail to have the following <code>PROVIDE:</code> line:
<pre>
# PROVIDE: dns
</pre>

The rest of the jails all get the following <code>REQUIRE:</code> line:
<pre>
# REQUIRE: dns
</pre>

== Syslog jail ==
I don't need a public v4 IP for this jail, so I configure it with an RFC1918 v4 IP on a loopback interface, and of course a real IPv6 address.

This jail gets the following <code>PROVIDE:</code> line in it's ezjail config file:
<pre>
# PROVIDE: syslog
</pre>

I use <code>syslog-ng</code> for this. I install the package using <code>pkg install syslog-ng</code> and then add a few things to the default config:

* I use the following options, YMMV:
<code>options { chain_hostnames(off); flush_lines(0); threaded(yes); use_fqdn(yes); keep_hostname(no); use_dns(yes); stats-freq(60); stats-level(0); };</code>

* I remove <code>udp()</code> from the default source and define a new source:
<code>source jailsrc { udp(ip("10.0.0.1")); udp6(ip("w:x:y:z:10::1")); };</code>

* I also remove the line: <code>destination console { file("/dev/console"); };</code> since the jail does not have access to <code>/dev/console</code>. I also remove any corresponding <code>log</code> statements that use <code>destination(console);</code>.

* I add a new destination:
<pre>
destination loghost{
syslog(
"syslog.tyktech.dk"
transport("tls")
port(1999)
tls(peer-verify(required-untrusted))
localip("10.0.0.1")
log-fifo-size(10000)
);
};
</pre>

* Finally tell syslog-ng to send all logdata from <code>jailsrc</code> to <code>loghost</code>:
<code>log { source(jailsrc); destination(loghost); flags(flow_control); };</code>

The rest of the jails get the following <code>REQUIRE:</code> line:
<pre>
# REQUIRE: dns syslog
</pre>

== Postgres jail ==
I don't need a public v4 IP for this jail, so I configure it with an RFC1918 v4 IP on a loopback interface, and of course a real IPv6 address. I add a AAAA record in DNS for the v6 IP so I have something to point the clients at.

I install the latest Postgres server port, at the time of writing that is <code>databases/postgresql93-server</code>. But before I can run <code>/usr/local/etc/rc.d/postgresql initdb</code> I need to permit the use of SysV shared memory in the jail. This is done in the ezjail config file for the jail, in the _parameters line. I need to add <code>allow.sysvipc=1</code> so I change the line from:
<pre>
export jail_postgres_kush_tyknet_dk_parameters=""
</pre>
to:
<pre>
export jail_postgres_kush_tyknet_dk_parameters="allow.sysvipc=1"
</pre>

While I'm there I also change the jails <code># PROVIDE:</code> line to:
<pre>
# PROVIDE: postgres
</pre>

And ofcourse the standard <code># REQUIRE:</code> line:
<pre>
# REQUIRE: dns syslog
</pre>

After restarting the jail I can run <code>initdb</code> and start Postgres. When a jail needs a database I need to:
* Add a DB user (with the <code>createuser -P someusername</code> command)
* Add a database with the new user as owner (<code>createdb -O someusername somedbname</code>)
* Add permissions in <code>/usr/local/pgsql/data/pg_hba.conf</code>
* Open a hole in the firewall so the jail can reach the database on TCP port <code>5432</code>
* Add <code>postgres</code> to the <code># REQUIRE:</code> line in the ezjail config file

== Web Jail ==
I need a public V4 IP for the web jail and I also give it a V6 IP. Since I use a different V6 IP per website, I will need additional v6 addresses when I start adding websites. I add the v6 addresses to the web jail in batches of 10 as I need them. After creating the jail and bootstrapping the ports collection I install <code>security/openssl</code> and <code>www/nginx</code> and configure it. More on that later.

= ZFS snapshots and backup =
So, since all this is ZFS based, there is a few tricks I do to make it easier to restore data in case of accidental file deletion or other dataloss.

== Periodic snapshots using sysutils/zfs-periodic ==
<code>sysutils/zfs-periodic</code> is a little script that uses the FreeBSD <code>periodic(8)</code> system to make snapshots of filesystems with regular intervals. It supports making hourly snapshots with a small change to <code>periodic(8)</code>, but I've settled for daily, weekly and monthly snapshots on my servers.

After installing <code>sysutils/zfs-periodic</code> I add the following to <code>/etc/periodic.conf</code>:
<pre>
#daily zfs snapshots
daily_zfs_snapshot_enable="YES"
daily_zfs_snapshot_pools="tank gelipool"
daily_zfs_snapshot_keep=7

#weekly zfs snapshots
weekly_zfs_snapshot_enable="YES"
weekly_zfs_snapshot_pools="tank gelipool"
weekly_zfs_snapshot_keep=5

#monthly zfs snapshots
monthly_zfs_snapshot_enable="YES"
monthly_zfs_snapshot_pools="tank gelipool"
monthly_zfs_snapshot_keep=6

#monthly zfs scrub
monthly_zfs_scrub_pools="tank gelipool"
monthly_zfs_scrub_enable="YES"
</pre>

Note that the last bit also enables a monthly scrub of the filesystem. Remember to change the pool name and remember to set the number of snapshots to retain to something appropriate. These things are always a tradeoff between diskspace and safety. Think it over and find some values that make you sleep well at night :)

After this has been running for a few days, you should have a bunch of daily snapshots:
<pre>
$ zfs list -t snapshot | grep gelipool@
gelipool@daily-2012-09-02 0 - 31K -
gelipool@daily-2012-09-03 0 - 31K -
gelipool@daily-2012-09-04 0 - 31K -
gelipool@daily-2012-09-05 0 - 31K -
</pre>

== Back-to-back ZFS mirroring ==
I am lucky enough to have more than one of these jail hosts, which is the whole reason I started writing down how I configure them. One of the advantages to having more than one is that I can configure <code>zfs send/receive</code> jobs and make server A send it's data to server B, and vice versa.

=== Introduction ===
The concept is pretty basic, but as it often happens, security considerations turn what was a simple and elegant idea into something... else. To make the back-to-back backup scheme work without sacrificing too much security, I first make a jail on each jailhost called <code>backup.jailhostname</code>. This jail will have control over a designated zfs dataset which will house the backups sent from the other server.

=== Create ZFS dataset ===
First I create the zfs dataset:
<pre>
$ sudo zfs create cryptopool/backups
$ sudo zfs set jailed=on cryptopool/backups
</pre>

=== 'jail' the new dataset ===
I create the jail like I normally do, but after creating it, I edit the ezjail config file and tell it which extra zfs dataset to use:
<pre>
$ grep dataset /usr/local/etc/ezjail/backup_glas_tyknet_dk
export jail_backup_glas_tyknet_dk_zfs_datasets="cryptopool/backups"
</pre>

This makes <code>ezjail</code> run the <code>zfs jail</code> command with the proper jail id when the jail is started.

=== jail sysctl settings ===
I also add the following to the jails ezjail config:

<pre>
# grep parameters /usr/local/etc/ezjail/backup_glas_tyknet_dk
export jail_backup_glas_tyknet_dk_parameters="allow.mount.zfs=1 enforce_statfs=1"
</pre>

=== Configuring the backup jail ===
The jail is ready to run now, and inside the jail a <code>zfs list</code> looks like this:
<pre>
$ zfs list
NAME USED AVAIL REFER MOUNTPOINT
cryptopool 3.98G 2.52T 31K none
cryptopool/backups 62K 2.52T 31K none
$
</pre>

I don't want to open up root ssh access to this jail, but the remote servers need to call <code>zfs receive</code> which requires root permissions. <code>zfs allow</code> to the rescue! <code>zfs allow</code> makes it possible to say "user X is permitted to do action Y on dataset Z" which is what I need here. In the backup jail I add a user called <code>tykbackup</code> which will be used as the user receiving the zfs snapshots from the remote servers.

I then run the following commands to allow the user to work with the dataset:
<pre>
$ sudo zfs allow tykbackup atime,compression,create,mount,mountpoint,readonly,receive cryptopool/backups
$ sudo zfs allow cryptopool/backups
---- Permissions on cryptopool/backups -------------------------------
Local+Descendent permissions:
user tykbackup atime,compression,create,mount,mountpoint,readonly,receive
$
</pre>

Testing if it worked:
<pre>
$ sudo su tykbackup
$ zfs create cryptopool/backups/test
$ zfs list cryptopool/backups/test
NAME USED AVAIL REFER MOUNTPOINT
cryptopool/backups/test 31K 2.52T 31K none
$ zfs destroy cryptopool/backups/test
cannot destroy 'cryptopool/backups/test': permission denied
$
</pre>

Since the user <code>tykbackup</code> only has the permissions <code>create,mount,mountpoint,receive</code> on the <code>cryptopool/backups</code> dataset, I get Permission Denied (as I expected) when trying to destroy <code>cryptopool/backups/test</code>. Works like a charm.

To allow automatic SSH operations I add the public ssh key for the root user of the server being backed up to <code>/usr/home/tykbackup/.ssh/authorized_keys</code>:
<pre>
$ cat /usr/home/tykbackup/.ssh/authorized_keys
from="ryst.tyknet.dk",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty,command="/usr/home/tykbackup/zfscmd.sh $SSH_ORIGINAL_COMMAND" ssh-rsa AAAAB3......KR2Z root@ryst.tyknet.dk
</pre>

The script called <code>zfscmd.sh</code> is placed on the backup server to allow the ssh client to issue different command line arguments depending on what needs to be done. The script is very simple:
<pre>
#!/bin/sh
shift
/sbin/zfs $@
exit $?
</pre>

A few notes: Aside from restricting the command this SSH key can run, I've restricted it to only be able to log in from the IP of the server being backed up. These are very basic restrictions that should always be in place no matter what kind of backup you are using.

=== Add the periodic script ===
I then add the script <code>/usr/local/etc/periodic/daily/999.zfs-mirror</code> to each server being backed up with the following content:
<pre>
#!/bin/sh

### check pidfile
if [ -f /var/run/$(basename $0).pid ]; then
echo "pidfile /var/run/$(basename $0).pid exists, bailing out"
exit 1
fi
echo $$ > /var/run/$(basename $0).pid

### If there is a global system configuration file, suck it in.
if [ -r /etc/defaults/periodic.conf ]; then
. /etc/defaults/periodic.conf
source_periodic_confs
fi

case "$daily_zfs_mirror_enable" in
[Yy][Ee][Ss])
;;
*)
exit
;;
esac

pools=$daily_zfs_mirror_pools
if [ -z "$pools" ]; then
pools='tank'
fi

targethost=$daily_zfs_mirror_targethost
if [ -z "$targethost" ]; then
echo '$daily_zfs_mirror_targethost must be set in /etc/periodic.conf'
exit 1
fi

targetuser=$daily_zfs_mirror_targetuser
if [ -z "$targetuser" ]; then
echo '$daily_zfs_mirror_targetuser must be set in /etc/periodic.conf'
exit 1
fi

targetfs=$daily_zfs_mirror_targetfs
if [ -z "$targetfs" ]; then
echo '$daily_zfs_mirror_targetfs must be set in /etc/periodic.conf'
exit 1
fi

if [ -n "$daily_zfs_mirror_skip" ]; then
egrep="($(echo $daily_zfs_mirror_skip | sed "s/ /|/g"))"
fi

### get todays date for later use
tday=$(date +%Y-%m-%d)

echo -n "Doing daily ZFS mirroring - "
date

### loop through the configured pools
for pool in $pools; do
echo " Processing pool $pool ..."
### enumerate datasets with daily snapshots from today
#echo $egrep
#echo "@daily-$tday"
if [ -n "$egrep" ]; then
datasets=$(zfs list -t snapshot -o name | grep "^$pool/" | egrep -v "$egrep" | grep "@daily-$tday")
else
datasets=$(zfs list -t snapshot -o name | grep "^$pool/" | grep "@daily-$tday")
fi

echo "found datasets: $datasets"

for snapshot in $datasets; do
dataset=$(echo -n $snapshot | cut -d "@" -f 1 | cut -d "/" -f 2-)
echo "working on dataset $dataset"
### find the latest daily snapshot of this dataset on the remote node, if any
echo ssh ${targetuser}@${targethost} /usr/home/tykbackup/zfscmd.sh list -t snapshot \| grep "^${targetfs}/${dataset}@daily-" \| cut -d " " -f 1 \| tail -1
lastgoodsnap=$(ssh ${targetuser}@${targethost} /usr/home/tykbackup/zfscmd.sh list -t snapshot | grep "^${targetfs}/${dataset}@daily-" | cut -d " " -f 1 | tail -1)
if [ -z $lastgoodsnap ]; then
echo "No remote daily snapshot found for local daily snapshot $snapshot - cannot send incremental - sending full backup"
zfs send -v $snapshot | ssh ${targetuser}@${targethost} /usr/home/tykbackup/zfscmd.sh receive -v -F -u -d $targetfs
if [ $? -ne 0 ]; then
echo " Unable to send full snapshot of $dataset to $targetfs on host $targethost"
else
echo " Successfully sent a full snapshot of $dataset to $targetfs on host $targethost - future sends will be incremental"
fi
else
#check if this snapshot has already been sent for some reason, skip if so..."
temp=$(echo $snapshot | cut -d "/" -f 2-)
lastgoodsnap="$(echo $lastgoodsnap | sed "s,${targetfs}/,,")"
if [ "$temp" = "$lastgoodsnap" ]; then
echo " The snapshot $snapshot has already been sent to $targethost, skipping..."
else
### add pool name to lastgoodsnap
lastgoodsnap="${pool}/${lastgoodsnap}"
### zfs send the difference between latest remote snapshot and todays local snapshot
echo " Sending the diff between local snapshot $(hostname)@$lastgoodsnap and $(hostname)@$snapshot to ${targethost}@${targetfs} ..."
zfs send -I $lastgoodsnap $snapshot | ssh ${targetuser}@${targethost} /usr/home/tykbackup/zfscmd.sh receive -F -u -d $targetfs
if [ $? -ne 0 ]; then
echo " There was a problem sending the diff between $lastgoodsnap and $snapshot to $targetfs on $targethost"
else
echo " Successfully sent the diff between $lastgoodsnap and $snapshot to $targethost"
fi
fi
fi
done
done

### remove pidfile
rm /var/run/$(basename $0).pid
</pre>

Remember to also <code>chmod +x /usr/local/etc/periodic/daily/999.zfs-mirror</code> and enable it in <code>/etc/periodic.conf</code>:

<pre>
#daily zfs mirror
daily_zfs_mirror_enable="YES"
daily_zfs_mirror_targethost="backup.glas.tyknet.dk"
daily_zfs_mirror_targetuser="tykbackup"
daily_zfs_mirror_targetfs="cryptopool/backups/kush.tyknet.dk"
daily_zfs_mirror_pools="tank gelipool"
daily_zfs_mirror_skip="gelipool/reserved gelipool/backups" # space seperated list of datasets to skip
</pre>

=== Create the target filesystems in the backup jail ===
Finally I create the destination filesystems in the backupjail, one per server being backed up. The filesystem that needs to be created is the one specified in the setting <code>daily_zfs_mirror_targetfs</code> in <code>/etc/periodic.conf</code> on the server being backed up.

=== Run the periodic script ===
I usually to the initial run of the periodic script by hand, so I can catch and fix any errors right away. The script will loop over all datasets in the configured pools and zfs send them including their snapshots to the backup server. Next time the script runs it will send an incremental diff instead of the full dataset.

=== Caveats ===
This script does not handle deleting datasets (including their snapshots) on the backup server when the dataset is deleted from the server being backed up. You will need to do that manually. This could be considered a feature, or a missing feature, depending on your preferences. :)

= Staying up-to-date =
I update my ezjail hosts and jails to track -STABLE regularly. This section describes the procedure I use. '''It is essential that the jail host and the jails use the same world version, or bad stuff will happen.'''

== Updating the jail host ==
First I update world and kernel of the jail host like I normally would. This is described earlier in this guide, see [[Ezjail_host#Building_world_and_kernel]].

== Updating ezjails basejail ==
To update ezjails basejail located in <code>/usr/jails/basejail</code>, I run the same commands as when bootstrapping ezjail, see the section [[Ezjail_host#Bootstrapping_ezjail]].

== Running mergemaster in the jails ==
Finally, to run mergemaster in all jails I use the following script. It will run mergemaster in each jail, the script comments should explain the rest. When it is finished the jails can be started:
<pre>
#! /bin/sh

### check if .mergemasterrc exists,
### move it out of the way if so
MM_RC=0
if [ -e /root/.mergemasterrc ]; then
MM_RC=1
mv /root/.mergemasterrc /root/.mergemasterrc.old
fi

### loop through jails
for jailroot in $(ezjail-admin list | cut -c 57- | tail +3 | grep "/"); do
echo "processing ${jailroot}:"
### check if jailroot exists
if [ -n "${jailroot}" -a -d "${jailroot}" ]; then
### create .mergemasterrc
cat <<EOF > /root/.mergemasterrc
AUTO_INSTALL=yes
AUTO_UPGRADE=yes
FREEBSD_ID=yes
PRESERVE_FILES=yes
PRESERVE_FILES_DIR=/var/tmp/mergemaster/preserved-files-$(basename ${jailroot})-$(date +%y%m%d-%H%M%S)
IGNORE_FILES="/boot/device.hints /etc/motd"
EOF
### remove backup of /etc from previous run (if it exists)
if [ -d "${jailroot}/etc.bak" ]; then
rm -rfI "${jailroot}/etc.bak"
fi

### create backup of /etc as /etc.bak
cp -pRP "${jailroot}/etc" "${jailroot}/etc.bak"

### check if mtree from last mergemaster run exists
if [ ! -e ${jailroot}/var/db/mergemaster.mtree ]; then
### delete /etc/rc.d/*
rm -rfI ${jailroot}/etc/rc.d/*
fi
### run mergemaster for this jail
mergemaster -D "${jailroot}"
else
echo "${jailroot} doesn't exist"
fi
sleep 2
done

### if an existing .mergemasterrc was moved out of the way in the beginning, move it back now
if [ ${MM_RC} -eq 1 ]; then
mv /root/.mergemasterrc.old /root/.mergemasterrc
else
rm /root/.mergemasterrc
fi

### done, a bit of output
echo "Done. If everything went well the /etc.bak backup folders can be deleted now."
exit 0
</pre>

To restart all jails I run the command <code>ezjail-admin restart</code>.

= Replacing a defective disk =
I had a broken harddisk on one of my servers this evening. This section describes how I replaced the disk to make everything work again.

== Booting into the rescue system ==
After Hetzner staff physically replaced the disk my server was unable to boot because the disk that died was the first one on the controller. The cheap Hetzner hardware is unable to boot from the secondary disk, bios restrictions probably. If the other disk had broken the server would have booted fine and this whole process would be done with the server running. Anyway, I booted into the rescue system and partitioned the disk, added a bootloader and added it to the root zpool. After this I was able to boot the server normally, so the rest of the work was done without the rescue system.

== Partitioning the new disk ==
The following shows the commands I ran to partition the disk:
<pre>
[root@rescue ~]# gpart create -s GPT /dev/ad4
ad4 created
[root@rescue ~]# /sbin/gpart add -b 2048 -t freebsd-boot -s 128 /dev/ad4
ad4p1 added
[root@rescue ~]# gpart add -t freebsd-zfs -s 30G /dev/ad4
ad4p2 added
[root@rescue ~]# gpart add -t freebsd-ufs /dev/ad4
ad4p3 added
[root@rescue ~]# gpart show
=> 34 1465149101 ad6 GPT (698G)
34 2014 - free - (1M)
2048 128 1 freebsd-boot (64k)
2176 62914560 2 freebsd-zfs (30G)
62916736 1402232399 3 freebsd-ufs (668G)

=> 34 1465149101 ad4 GPT (698G)
34 2014 - free - (1M)
2048 128 1 freebsd-boot (64k)
2176 62914560 2 freebsd-zfs (30G)
62916736 1402232399 3 freebsd-ufs (668G)

[root@rescue ~]#
</pre>

== Importing the pool and replacing the disk ==

Next step is importing the zpool (remember altroot=/mnt !) and replacing the defective disk:

<pre>
[root@rescue ~]# zpool import
pool: tank
id: 3572845459378280852
state: DEGRADED
status: One or more devices are missing from the system.
action: The pool can be imported despite missing or damaged devices. The
fault tolerance of the pool may be compromised if imported.
see: http://www.sun.com/msg/ZFS-8000-2Q
config:

tank DEGRADED
mirror-0 DEGRADED
11006001397618753837 UNAVAIL cannot open
ad6p2 ONLINE
[root@rescue ~]# zpool import -o altroot=/mnt/ tank
[root@rescue ~]# zpool status
pool: tank
state: DEGRADED
status: One or more devices could not be opened. Sufficient replicas exist for
the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
see: http://www.sun.com/msg/ZFS-8000-2Q
scan: scrub repaired 0 in 0h2m with 0 errors on Thu Nov 1 05:00:49 2012
config:

NAME STATE READ WRITE CKSUM
tank DEGRADED 0 0 0
mirror-0 DEGRADED 0 0 0
11006001397618753837 UNAVAIL 0 0 0 was /dev/ada0p2
ad6p2 ONLINE 0 0 0

errors: No known data errors
[root@rescue ~]# zpool replace tank 11006001397618753837 ad4p2

Make sure to wait until resilver is done before rebooting.

If you boot from pool 'tank', you may need to update
boot code on newly attached disk 'ad4p2'.

Assuming you use GPT partitioning and 'da0' is your new boot disk
you may use the following command:

gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da0

[root@rescue ~]#
[root@rescue ~]# zpool status
pool: tank
state: DEGRADED
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Tue Nov 27 00:24:41 2012
823M scanned out of 3.11G at 45.7M/s, 0h0m to go
823M resilvered, 25.88% done
config:

NAME STATE READ WRITE CKSUM
tank DEGRADED 0 0 0
mirror-0 DEGRADED 0 0 0
replacing-0 UNAVAIL 0 0 0
11006001397618753837 UNAVAIL 0 0 0 was /dev/ada0p2
ad4p2 ONLINE 0 0 0 (resilvering)
ad6p2 ONLINE 0 0 0

errors: No known data errors
[root@rescue ~]# zpool status
pool: tank
state: ONLINE
scan: resilvered 3.10G in 0h2m with 0 errors on Tue Nov 27 01:26:45 2012
config:

NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
ada0p2 ONLINE 0 0 0
ada1p2 ONLINE 0 0 0

errors: No known data errors
[root@rescue ~]#
</pre>

== Reboot into non-rescue system ==
At this point I rebooted the machine into the normal FreeBSD system.

== Re-create geli partition ==
To recreate the geli partition on p3 of the new disk, I just follow the same steps as when I originally created it, [http://wiki.tyk.nu/index.php?title=Ezjail_host#Create_GELI_volumes more info here].

To attach the new geli volume I run <code>geli attach</code> as [http://wiki.tyk.nu/index.php?title=Ezjail_host#Attach_GELI_volumes described here].

== Add the geli device to the encrypted zpool ==
First I check that both geli devices are available, and I check the device name that needs replacing in <code>zpool status</code> output:

<pre>
[tykling@haze ~]$ geli status
Name Status Components
ada1p3.eli ACTIVE ada1p3
ada0p3.eli ACTIVE ada0p3
</pre>

<pre>
[tykling@haze ~]$ zpool status gelipool
pool: gelipool
state: DEGRADED
status: One or more devices has been removed by the administrator.
Sufficient replicas exist for the pool to continue functioning in a
degraded state.
action: Online the device using 'zpool online' or replace the device with
'zpool replace'.
scan: scrub repaired 68K in 0h28m with 0 errors on Thu Nov 1 05:58:08 2012
config:

NAME STATE READ WRITE CKSUM
gelipool DEGRADED 0 0 0
mirror-0 DEGRADED 0 0 0
18431995264718840299 REMOVED 0 0 0 was /dev/ada0p3.eli
ada1p3.eli ONLINE 0 0 0

errors: No known data errors
[tykling@haze ~]$
</pre>

To replace the device and begin resilvering:

<pre>
[tykling@haze ~]$ sudo zpool replace gelipool 18431995264718840299 ada0p3.eli
Password:
[tykling@haze ~]$ zpool status
pool: gelipool
state: DEGRADED
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Tue Nov 27 00:53:40 2012
759M scanned out of 26.9G at 14.6M/s, 0h30m to go
759M resilvered, 2.75% done
config:

NAME STATE READ WRITE CKSUM
gelipool DEGRADED 0 0 0
mirror-0 DEGRADED 0 0 0
replacing-0 REMOVED 0 0 0
18431995264718840299 REMOVED 0 0 0 was /dev/ada0p3.eli/old
ada0p3.eli ONLINE 0 0 0 (resilvering)
ada1p3.eli ONLINE 0 0 0

errors: No known data errors
[tykling@haze ~]$
</pre>

When the resilver is finished, the system is good as new.