Managing jails with ezjail

This is a list to help me remember the details of managing FreeBSD jails with ezjail. This example was all done on FreeBSD 8.0-BETA2 with ezjail-3.0.

Getting started

To get started, install ezjail from sysutils/ezjail and run the command:

[tykling@peace ~]$ sudo ezjail-admin install

This will make ezjail connect to a FreeBSD ftp site and download the files neccesary for building the jail. Consult the ezjail documentation for more information.

Allow ping from the jails

I add the following line to /etc/sysctl.conf to allow the jails raw socket access, which is neccesary for icmp since icmp is a layer two protocol. This opens up the possibility of a compromized jail binding to other IP addresses than the one configured to the jail, but I'll take my chances:

security.jail.allow_raw_sockets=1

Get jail info out of top

To make top show the jail id of the jail in which the process is running in a column, I need to specify the -j flag to top. Since this is a multi-cpu server I am working on, I also like giving the -P flag to top, to get a seperate line of cpu stats per core. I add the following to my .bashrc in my homedir on the jail host:

alias top="nice top -j -P"

...this way I don't have to remember passing -j -P to top every time. Also, I've been told to run top with nice to limit the cpu used by top itself. I took the advice so the complete alias looks like above.

Services listening on all IP addresses

I need to make a few services stop listening on all addresses before I start creating jails (I don't want the host system sshd listening on port 22 on the jails IP, for example). Candidates in the default install are sshd and syslogd.

sshd

To make the host system sshd listen only on it's own IP address, I uncommented and edited the following line in /etc/ssh/sshd_config:

[tykling@peace ~]$ grep 10.16 /etc/ssh/sshd_config
ListenAddress 10.16.0.75

syslogd

To make the host system syslogd stop listening on all interfaces I add the following to rc.conf:

#make syslogd listen on loopback only
syslogd_flags="-s -b 127.0.0.1"

net-snmp

I also have the port /usr/ports/net-mgmt/net-snmp installed on the host system for monitoring. This is what I have in rc.conf for net-snmp:

#enable snmpd
snmpd_enable="YES"
snmpd_conffile="/usr/local/etc/snmpd.conf"
snmpd_flags="10.16.0.75"

And this line needs to go into /usr/local/etc/snmpd.conf:

smuxsocket 10.16.0.75

..along with the rest of the configuration in that file.

I restart all three daemons, and now everything is listening on the host systems main or loopback address, nothing is bound to *:port in the column "LOCAL ADDRESS", which is what I wanted:

[tykling@peace ~]$ sockstat -l4
USER     COMMAND    PID   FD PROTO  LOCAL ADDRESS         FOREIGN ADDRESS
root     syslogd    3032  6  udp4   127.0.0.1:514         *:*
root     snmpd      2858  9  tcp4   10.16.0.75:199        *:*
root     snmpd      2858  10 udp4   10.16.0.75:161        *:*
root     sshd       1802  3  tcp4   10.16.0.75:22         *:*
root     sendmail   956   4  tcp4   127.0.0.1:25          *:*

Setting up a new jail

This section is a step by step guide to creating a new jail.

Configure an IP address for the jail

I need an IP address on this jail. I always use loopback addresses and redirect the traffic in pf as needed. I also use a seperate lo interface, and leave the normal lo0 interface alone. This can be accomplished in rc.conf like so:

cloned_interfaces="lo1"
ifconfig_lo1="inet 127.0.0.2/8"
ifconfig_lo1_alias0="127.0.1.25/32"
ifconfig_lo1_alias1="127.0.2.25/32"

To make the settings in rc.conf take effect without rebooting, run the following commands:

[tykling@peace ~]$ sudo ifconfig lo1 create
[tykling@peace ~]$ sudo /etc/rc.d/netif restart lo1
ifconfig: ioctl (SIOCDIFADDR): Can't assign requested address
ifconfig: ioctl (SIOCDIFADDR): Can't assign requested address
Stopping Network: lo1.
lo1: flags=8048<LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=3<RXCSUM,TXCSUM>
Starting Network: lo1.
lo1: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=3<RXCSUM,TXCSUM>
        inet 127.0.0.2 netmask 0xff000000
        inet 127.0.1.25 netmask 0xffffffff
        inet 127.0.2.25 netmask 0xffffffff
[tykling@peace ~]$

You can safely ignore the first two errors. As you can see, the lo1 interface has been created with the desired IP addresses.

Create the jail

I am now ready to create the jail with the following command:

sudo ezjail-admin create postfix1.peace.skabet.cn.dom 127.0.1.25

Start the jail

The jail is now ready to run, I start ezjail using the rc.d script:

[tykling@peace ~]$ sudo /usr/local/etc/rc.d/ezjail.sh start
 ezjailConfiguring jails:.
Starting jails: postfix2.peace.skabet.cn.dom postfix1.peace.skabet.cn.dom.
[tykling@peace ~]$ jls
   JID  IP Address      Hostname                      Path
     1  127.0.2.25      postfix2.peace.skabet.cn.dom  /usr/jails/postfix2.peace.skabet.cn.dom
     2  127.0.1.25      postfix1.peace.skabet.cn.dom  /usr/jails/postfix1.peace.skabet.cn.dom
[tykling@peace ~]$

As you can see, I created another jail by repeating the steps above before I started ezjail. The output of the jls command shows that the two new jails are running as expected.

"Getting into" a new jail

To get a shell in the new jail, run the jexec command with the jail id and the sh command like so:

[tykling@peace ~]$ jls
   JID  IP Address      Hostname                      Path
     1  127.0.2.25      postfix2.peace.skabet.cn.dom  /usr/jails/postfix2.peace.skabet.cn.dom
     2  127.0.1.25      postfix1.peace.skabet.cn.dom  /usr/jails/postfix1.peace.skabet.cn.dom
[tykling@peace ~]$ sudo jexec 1 sh
# hostname
postfix2.peace.skabet.cn.dom
#

Initial configuration of a new jail

There is a few things that I need to remember whenever configuring a new jail in this fashion.

Create rc.conf

There is no /etc/rc.conf by default, so I create it and the first thing I set is the hostname, even though is it set by ezjail, I like having it here for the sake of completeness.

Services listening on *

Same deal as with the host system - sshd and syslogd by default listens to all available IP addreses, this is not strictly neccesary to change since the jail cannot see other addresses than the one assigned to it:

# ifconfig lo0
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=3<RXCSUM,TXCSUM>
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4
        inet6 ::1 prefixlen 128
        inet 127.0.2.25 netmask 0xffffffff

I still like to bind the services explicitly though, so I edit /etc/ssh/sshd_config to make it listen on 127.0.2.25 specifically. I do the same with syslogd, by adding syslogd_flags="-s -b 127.0.2.25" to /etc/rc.conf.

The jails rc.conf now looks like this:

# cat /etc/rc.conf
hostname="postfix2.peace.skabet.cn.dom"
syslogd_flags="-s -b 127.0.2.25"
sshd_enable="YES"

Getting the jail online

Since this jail has no real world IP address, I need a bit of pf magic to get it online. This is only for basic connectivity, I will configure pf to allow access to the postfix instances that will eventually run in these jails later.

I enable pf on the host machine with the following rc.conf entries:

#enable pf
pf_enable="YES"                 # Enable PF (load module if required)
pflog_enable="YES"              # start pflogd(8)

I use the following very simple pf.conf ruleset to make traffic from all jails with an IP address on the lo1 interface get translated to the public address of the jail hosts physical network interface, em1:

#translate outgoing traffic from jails
nat on em1 from 127.0.1.25 to any -> (em1)
nat on em1 from 127.0.2.25 to any -> (em1)

Note that I have a pf firewall at the network perimeter taking care of scrubbing and filtering and such, I do not recommend using a pf.conf like the one above on a machine that does not have any other protection :).

I start up pf by first loading the kernel modules (this will be done automatically at the next reboot), and then I start pf using pfctl:

[tykling@peace ~]$ sudo kldload pf
[tykling@peace ~]$ sudo kldload pflog
[tykling@peace ~]$ sudo pfctl -ef /etc/pf.conf
No ALTQ support in kernel
ALTQ related functions disabled
pf enabled
[tykling@peace ~]$

I confirm that it works immediately with ping from inside the jail:

# hostname
postfix1.peace.skabet.cn.dom
# ifconfig lo1
lo1: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=3<RXCSUM,TXCSUM>
        inet 127.0.1.25 netmask 0xffffffff
# ping 10.16.0.1
PING 10.16.0.1 (10.16.0.1): 56 data bytes
64 bytes from 10.16.0.1: icmp_seq=0 ttl=64 time=0.393 ms
64 bytes from 10.16.0.1: icmp_seq=1 ttl=64 time=0.329 ms
^C
--- 10.16.0.1 ping statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.329/0.361/0.393/0.032 ms
#

The last thing missing is to configure DNS in the jails. I just copy the resolv.conf from the host machine to each of the jails:

[tykling@peace ~]$ sudo cp /etc/resolv.conf /usr/jails/postfix1.peace.skabet.cn.dom/etc/resolv.conf
[tykling@peace ~]$ sudo cp /etc/resolv.conf /usr/jails/postfix2.peace.skabet.cn.dom/etc/resolv.conf
[tykling@peace ~]$

I can test name resolution right away from inside the jails:

# ping wiki.tyk.nu
PING wiki.tyk.nu (89.233.43.70): 56 data bytes
64 bytes from 89.233.43.70: icmp_seq=0 ttl=64 time=0.426 ms
64 bytes from 89.233.43.70: icmp_seq=1 ttl=64 time=0.301 ms
^C

Install basic ports

I always install a few ports in a new jail:

• ports-mgmt/portmaster - my favourite port management tool.
• security/openssh-portable - I like running deamons from ports, ports are easier to upgrade than things in the base system. After installing this I change sshd_enable="YES" to openssh_enable="YES" in /etc/rc.conf, and then stop the old sshd and start the new one.
• shells/bash - can't live without it.
• security/sudo - to avoid working as root.

Before I can install any ports I need the ports tree. ezjail defaults to mounting a read-only copy of the ports tree from the basejail in /usr/jails/basejail, but I like each jail to have it's own ports tree. So the first thing I do is to remove the symlink created by ezjaill, before I run portsnap:

# rm /usr/ports

The first time I run portsnap I need to use the following command:

# portsnap fetch extract
Looking up portsnap.FreeBSD.org mirrors... 3 mirrors found.
Fetching public key from portsnap2.FreeBSD.org... done.
Fetching snapshot tag from portsnap2.FreeBSD.org... done.
Fetching snapshot metadata... done.
Fetching snapshot generated at Sun Aug  9 01:36:49 UTC 2009:
11571513eccc326b8f19efcc8a3ac26b4e01faddb3500c 18% of   58 MB  239 kBps 03m27s
..... snip ........

Any future runs of portsnap will be with the command portsnap fetch update instead of portsnap fetch extract.

Then I install portmaster, and use that to install the rest of the ports.

Adding a user

Since a jail is a completely seperate machine, I need to add a user to log in with using SSH. This is done in the regular way using adduser. I add the user to sudoers.

Forwarding root system mail

To make sure I get the system mails from the jail I change the following line in /etc/aliases:

root:   me@example.com

..and then run the command 'newaliases' to rebuild the alias database:

# newaliases
/etc/mail/aliases: 28 aliases, longest 17 bytes, 296 bytes total

Further configuration

After this is completed the jails are ready to be configured to whatever they are meant to do - in this case, I will install postfix and configure them to act as incoming mail servers for a few domains. I will configure seperate jails for other parts of the mailserver, a jail for dovecot (actually two), a jail for spamassassin (actually two) and so on.