EuroBSDCon 2018 Trip Report

The developer summit began on Thursday morning with an unconference feel. After introductions and greetings, we got down to brass tacks by hacking in smaller groups. I took the opportunity to work with dch@ on a review, catch up on some core@ email, and work on a few port updates.

Eventually seanc@ stepped up to lead a room-wide discussion. Some of the topics were:

For me, the most notable point from the 'Core.10 expectations' discussion was that Core.10 is planning a developer survey. To use Sean's phrase, "We are flying blindly". With a well-crafted survey, we can begin to make decisions based on what people really want. If you have questions that you think should be added to the survey, please discuss them on freebsd-arch@. Once the survey of developers is complete, the plan is to send out a version to the wider community. I have high expectations for the surveys. They have the potential to shape the direction of the project for years to come. On a related note, there were rumblings about either revitalizing or coming up with something similar to the BSDstats project, so we can get good system data to direct development. It is a waste of time and energy, for example, to maintain drivers that no one is using and conversely, it would be a shame to remove parts of the OS that are still useful to some number of users.

During the 'Workflow and version control' discussion, I sensed that we (mostly) agreed and reaffirmed that we want: I am not sure that our attempt to approach this topic with an open mind was successful. There were undertones of "switching to Git is inevitable". When Sean asked for a show of hands to determine who already uses Git, almost everyone in the room lifted a hand. If I recall correctly, the second question was something along the lines of "Do you choose to use Git?" and the result was the about the same. It was noted that this likely does not represent the developer community as a whole, but it was compelling anecdotal support for a switch. However, there are major challenges involved with switching:

Later on Twitter, des@ reminded us of a post by phk@ in one of our 'Git versus Subversion' threads of pain. In it, phk@ argues that, in some respects, comparing Subversion and Git is dubious. Subversion supports the traditional features of a version control system, whereas Git is different. Git is more of a collaboration tool that can mostly substitute as a version control system. It is a worthwhile (re-)read for anyone interested in this topic.

Considering the contentious nature of the topic, the discussions were civil and technical and even though it has been discussed a few times in the past, some new, important technical questions were raised. Hopefully the developer survey will help us clarify one important point: What proportion of developers feel that the benefits of switching will outweigh the costs?

The most discussion time was dedicated to 'Goals for FreeBSD 13'. Since there are notes, I will only give an overview of what I found most interesting We want:

dch@ gave a short talk/demo about an interesting idea to revive something similar to RedPorts, which offered package building and testing as a service. Hopefully this will get more people interested in helping out with ports. Given the many combinations of supported versions and architectures, it can be difficult to tests all those combinations on a desktop or laptop.

The conference proper started on Saturday with a keynote talk by Costin Raiciu. In a nutshell, he is working on something he calls Unikraft, which reduces the time and effort required to strip down a Linux kernel, so that it only includes the components necessary for specific target applications. I learned that these stripped down kernels are referred to as unikernels. He also described other unikernel work designed to improve the efficiency of lightweight virtual machines (VM), so that they approach the speed and size of containers. An example use-case involved creating ephemeral VM instances whenever requests came in from mobile devices. The motivation for doing this is to ensure that each request is run with a high level of isolation. When there are thousands of such instances running simultaneously, efficiency is a priority.

A few talks that stood out for me were Advances in OpenBSD packages: https is a lie, by Marc Espie, Using Boot Environments at Scale by Allan Jude, The End of DNS as we know it, by Carsten Strotmann, and FreeBSD Graphics by Niclas Zeising.

In Marc's talk he described work to 1. securely and efficiently fetch packages with OpenBSD's pkg_add and 2. separate privileges when building packages on OpenBSD. Marc is an enthusiastic and opinionated speaker, which I like (within limits) to keep my attention. There was also some good back-and-forth with the audience to tease out a few technical questions, which kept things lively. Could TLS 1.3 solve some of the challenges with session resumption?

The piece of mind ZFS boot environments offer when making changes to the OS have been well publicized. Allan described his efforts to enhance this work by wrangling boot environments and poudriere image into something that could be used to install and upgrade many of his company's servers without having to deal with mergemaster or etcupdate. A few other advantages to the approach are: 1. no need for console access, 2. failures are handled gracefully, 3. less manual intervention, and 4. releases can be created without compiling. A nice part of the talk was when Allan described the pitfalls he faced along the way. For example, using a separate filesystem for /etc, so that it can persist through an upgrade sounds like a good idea, but when /etc/fstab or /etc/rc are not available at the right time, bad things happen. Contact Allan to help with ZFS support for poudriere image or to report bugs and request features.

The End of DNS as we know it was a fantastic talk that sparked lots of interesting comments from a knowledgeable audience. Carsten did a nice job finding a good technical balance. The talk was not light on technical information, but did not venture too far into the weeds. It was interesting to hear a comprehensive review of where we are headed with such a fundamental Internet protocol.

If you run FreeBSD on your desktop or laptop, then Niclas's FreeBSD Graphics talk would have helped you appreciate the work the small graphics team does so that we can run our favourite window manager. There are different parts the team takes responsibility for: drivers, libraries, the X server itself, and a variety of other applications that sum up to about 300 ports. After giving an overview of these different parts, Niclas described some of the challenges and future work. If you would like to get in touch with the Graphics team, contact them at,, #freebsd-xorg on efnet, or on the FreeBSD Desktop GitHub page.

Monday was secretary-tourist-day as René Laden and I set out to play tourists. Unfortunately, Mondays are less than ideal for tourist activities, since many of the museums and other attractions are closed, so we decided to have a free walk around the city without much of a plan. Some highlights were the Palace of the Parliament (which apparently is the second largest administrative building and the heaviest building in the world), our stroll along the Dambovita River, and the city parks.

We discussed the developer summit, our roles as secretaries, and the challenges and opportunities facing the project. In one of our personal conversations, which René permitted me to share, he described some of his Autism-related challenges, in particular his difficulties with communication. On the one hand, this was surprising, because René's reports and other writings are high quality. On the other hand, the clarity of his speech can vary, but luckily René does not mind repeating himself if something is unclear the first time around. He also explained how his speech depends on how comfortable he feels in a social situation. I saw evidence of this, especially in the relaxed atmosphere of a quiet café, because our discussions were natural, stimulating and a high point of my trip.

Indeed, the talks and the developer summit were interesting, but a meaningful aspect of a conference like this for me is to step away from the comfort of the Irc/mail client and engage in face-to-face discussions with others. We spend significant amounts of time working on a common goal, so it is especially rewarding for me to make a connection with other members of the team.

Thank you to the FreeBSD Foundation for supporting my trip to EuroBSDCon 2018.

Posted 2019-02-16 20:52 | Comments

Upgrading Mastodon on FreeBSD

This post is now out of date. The FreeBSD port/package of Mastodon, net-im/mastodon, is no longer available. To install Mastodon on FreeBSD, refer to the Mastodon Production Guide. To upgrade Mastodon, refer to the Mastodon release notes.

Stop the three Mastodon servers.
# service mastodon_web stop
# service mastodon_workers stop
# service mastodon_stream stop
Upgrade your packages.
# pkg upgrade

Check the Mastodon release notes to see if any special scripts need to be run. For example, if a data migration is required, run the commands below, but do not prepend commands with `bundle exec` and never run `bundle install`, `yarn install`, or `assets:precompile`.

# su - mastodon
% echo -n > Gemfile.lock
% RAILS_ENV=production rails db:migrate

Do a diff between nginx-include.conf.sample and nginx-include.conf to check for changes. Update nginx-include.conf if necessary.

Return to root privileges.

If you made changes to nginx-include.conf, reload the nginx configuration.
# service nginx reload
Finally, restart the Mastodon daemons.
# service mastodon_web start
# service mastodon_workers start
# service mastodon_stream start

Posted 2017-04-27 12:55 | Comments

Running Mastodon on FreeBSD

This post is now out of date. The FreeBSD port/package of Mastodon, net-im/mastodon, is no longer available. To install Mastodon on FreeBSD, refer to the Mastodon Production Guide.

Install the packages

# pkg install nginx postgresql95-server postgresql95-contrib mastodon

Enable and start required services

# sysrc redis_enable="YES"
# service redis start
# sysrc postgresql_enable="YES"
# service postgresql initdb
# service postgresql start
The Mastodon port ships with two sample nginx configuration files, a complete nginx.conf and nginx-include.conf, which mostly just includes the server block. If the web server is going to be dedicated to Mastodon, you can create a new nginx profile.
# sysrc nginx_profiles=mastodon
# sysrc nginx_mastodon_configfile="/usr/local/www/mastodon/nginx.conf"
If you prefer to continue using your current nginx.conf, you can add the line below to it. Make sure you put it inside the http block.
include /usr/local/www/mastodon/nginx-include.conf;
In either case, you need to customize nginx-include.conf. At minimum, you will have to change all instances of Once you are satisfied with its configuration, start nginx.
# sysrc nginx_enable="YES"
# service nginx start

Create mastodon database user

# psql -d template1 -U pgsql -c "CREATE USER mastodon CREATEDB;"

Switch to the mastodon user

# su - mastodon

Customize .env.production

Customize .env.production to suit your needs, but you must at least set values for LOCAL_DOMAIN and SMTP_FROM_ADDRESS. You also have to generate secrets for PAPERCLIP_SECRET, SECRET_KEY_BASE, and OTP_SECRET. Generate a different secret for each of these fields by running the command below three time. Save .env.production before moving on to the next step.
% RAILS_ENV=production rake secret
If you are installing version 1.5.0 or later, to enable Web Push notifications, you need to generate a few extra secrets and put them in .env.production.
% RAILS_ENV=production rake mastodon:webpush:generate_vapid_key

Set up the database for the first time

% RAILS_ENV=production rails db:setup

Start the Mastodon daemons

Run the following commands with root privileges.
# sysrc mastodon_web_enable="YES"
# sysrc mastodon_workers_enable="YES"
# sysrc mastodon_stream_enable="YES"
# service mastodon_web start
# service mastodon_workers start
# service mastodon_stream start

Optional Configuration in /etc/rc.conf

As of version 2.0.0, values for some options can be read from /etc/rc.conf instead of .env.production. See the comments in Mastodon's three rc scripts, mastodon_web, mastodon_workers, and mastodon_stream for details.

Give users administration rights

After the user alice has created an account through the web interface, you can give her administration rights by using the command below.
# sudo su - mastodon
% RAILS_ENV=production rails mastodon:make_admin USERNAME=alice

Posted 2017-04-23 21:18 | Comments

ZFS Snapshot Management and Replication with zap

In this post, I describe how I manage ZFS snapshots and remote replication from a source host, phe, to a target host, bravo. I assume you have version 0.7.3 or newer of zap installed via the FreeBSD package and the sudo command is available.

Target host (bravo)

Partition the disks to be used for the backup pool.

# for i in 1 2; do
    gpart create -s gpt ada$i
    gpart add -a 1M -l gptboot$i -t freebsd-boot -s 512k ada$i
    gpart bootcode -b boot/pmbr -p /boot/gptzfsboot -i 1 ada$i
    gpart add -a 1m -l zfs$i -t freebsd-zfs ada$i

Create the backup pool and a top-level filesystem with the necessary permissions. Make the backup datasets readonly, otherwise incremental replication may fail.

# zpool create -m none zback mirror gpt/zfs1 gpt/zfs2
# zfs create -o readonly=on zback/phe
# zfs allow -u zap canmount,compression,create,mountpoint,mount,receive,refreservation,userprop,volmode zback/phe

Source host (phe)

Set permissions on the datasets to be replicated.

# zfs allow -u zap bookmark,diff,hold,send,snapshot zroot/ROOT
# zfs allow -u zap bookmark,diff,hold,send,snapshot zroot/usr/home
# zfs allow -u zap bookmark,diff,hold,send,snapshot zroot/var

Set the zap:snap user property to on for datasets to snapshot. The property will be inherited, so explicitly turn it off for datasets that we do not want to snapshot.

# zfs set zap:snap=on zroot/ROOT zroot/usr/home zroot/var
# zfs set zap:snap=off zroot/var/crash zroot/var/tmp zroot/var/mail

Create the first snapshots.

# sudo -u zap zap snap -v 1w

Set the zap:rep property for datasets that will be replicated to bravo.

# zfs set zap:rep='zap@bravo:zback/phe' zroot/ROOT zroot/usr/home zroot/var
# zfs set zap:rep=off zroot/var/crash zroot/var/tmp zroot/var/mail

Before replicating datasets, key-based ssh authentication to bravo must be set up. An important decision at this point is whether or not to protect the private key that will be generated with a passphrase. If anyone else has access to the source host, the decision is clear: add a passphrase. The downside to adding a passphrase is that you will have to enter it before you can replicate. The ssh-agent makes this more manageable.

# sudo -u zap
zap@phe % ssh-keygen

Manually copy the contents of the newly created public key on the source host (~/.ssh/ by default) to ~/.ssh/authorized_keys on the target host and confirm that you can ssh from the source host to the target host using key-based authentication.

Since this is the first time these datasets are being replicated to bravo, a full stream will be sent.

zap@phe % zap rep -v

Target host (bravo)

Optionally mount the backup datasets under /zback/phe.

# mkdir -p /zback/phe
# zfs set mountpoint=/zback/phe zback/phe/ROOT/default
# zfs mount zback/phe/ROOT/default
# zfs set mountpoint=/zback/phe/var zback/phe/var
# zfs mount zback/phe/var
# zfs set mountpoint=/zback/phe/usr/home zback/phe/usr/home
# zfs mount zback/phe/usr/home

Since I also use zap on bravo, I turn off the zap:snap and zap:rep properties for the backups.

# zfs set zap:snap=off zback/phe/ROOT zback/phe/usr/home zback/phe/var

Source host (phe)

Create new snapshots and send the incremental changes to bravo.

zap@phe % zap snap -v 1d
zap@phe % zap rep -v

Automate rolling spanshots and replication with cron. Taking snapshots is normally cheap, so do it often. Destroying snapshots can thrash disks, so I only do it every 24 hours. Sensible replication frequencies can vary with different factors. Adjust according to your needs.

Edit the crontab for the zap user by adding the contents shown below.

zap@phe % crontab -e
#minute	hour	mday	month	wday	command

# take snapshots
*/5	*	*	*	*	zap snap 1d
14	*/4	*	*	*	zap snap 1w
14	00	*	*	1	zap snap 1m

# replicate datasets
54	*/1	*	*	*	zap rep -v

Edit the crontab for the root user by adding the contents shown below.

# crontab -e
#minute	hour	mday	month	wday	command

# destroy snapshots
44	04	*	*	*	zap destroy

Posted 2017-01-04 12:22 | Comments

ZFS on Root and Full Disk Encryption: FreeBSD 10.3 to 11.0 or One pool to rule them all

Thanks to Allan Jude for steering me through this on IRC and Warren Block for his feedback.

The new boot loader in 11.0 is able to boot encrypted ZFS pools directly. Yes, that means you can have full disk encryption (FDE) with ZFS on root and boot environments (BEs)! However, after you upgrade from 10.3, some tinkering is necessary to get this working. The instructions that follow are for a ZFS mirror installation. The two disks (ada0 and ada1) each have the same partition layout: p1: freebsd-boot, p2: freebsd-zfs (boot pool), p3: swap, p4: freebsd-zfs (main pool). Specify your disk(s) and partition indices according to your setup.

Reencrypt the master key with only a passphrase. You can use the same passphrase as before.

# geli setkey -k /boot/encryption.key ada0p4
# geli setkey -k /boot/encryption.key ada1p4

Set the geliboot flag.

# geli configure -g ada0p4
# geli configure -g ada1p4

Remove the /boot symbolic link pointing to /bootpool/boot and copy /boot from /bootpool/ to /.

# rm /boot
# cp -r /bootpool/boot /

Install the GPT boot code into the boot partition.

# gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0
# gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada1

Set the partition type of the old boot pool partition to freebsd-vinum, so it does not get detected by the boot code as a ZFS partition.

# gpart modify -t freebsd-vinum -i 2 ada0
# gpart modify -t freebsd-vinum -i 2 ada1

Remove geli_ada0p4_*, geom_eli_passphrase_prompt, and (optional) zpool_cache_* from /boot/loader.conf.

Set canmount=noauto for all BEs, including the default.

# zfs set canmount=noauto zroot/ROOT/default
# zfs set canmount=noauto zroot/ROOT/some_other_be

Reboot to confirm everything is working. If you are satisfied, you can destroy the old boot pool.

# zpool destroy bootpool

Delete the old boot pool partitions.

# gpart delete -i2 ada0
# gpart delete -i2 ada1

Delete the old swap partitions.

# swapoff -a
# gpart delete -i3 ada0
# gpart delete -i3 ada1

Use the reclaimed space for larger swap partitions.

# gpart add -t freebsd-swap -l swap0 ada0
# gpart add -t freebsd-swap -l swap1 ada1

Update /etc/fstab to use the new swap partition indices.

# Device          Mountpoint  FStype  Options  Dump  Pass#
/dev/ada0p2.eli   none        swap    sw       0     0
/dev/ada1p2.eli   none        swap    sw       0     0

Turn swap back on.

# swapon -a

Posted 2016-09-17 15:48 | Comments

Recent posts

Monthly Archives

Yearly Archives