New server: away from Kolab

Yesterday I finally flipped the switch on the MX record and moved to a new server (still aptly named after another character from アキハバラ電脳組). The machine was found through Hetzner’s server auctions and has quite a good configuration for the price paid:

• Intel Xeon CPU E31245 @ 3.30GHz
• 16G ECC RAM (ECC was the deciding factor)
• LSI Hardware RAID 1 with two 3x2T disks

Now, since the old server has been chugging along since 2011, the 5 eurocent question is why did you move at all? It turns out there were a pretty good number of reasons (to me, at least).

OS and software

tsugumi (the old server) ran CentOS 7, which was a requirement to properly run Kolab Server (more on this later). As the base is very stable, some of the bits and pieces there were already pretty outdated. That usually is not a problem, except for PHP, whose version is EOL upstream. There are alternatives to get a newer PHP version on CentOS, but they were impractical (again, more on this later). There were also a few unfortunate choices I made during the years that I did not bother fixing to avoid any kind of breakage.

In addition I wasn’t too happy with CentOS at all, mainly because of familiarity. I mean, I mostly use openSUSE Tumbleweed on desktops, and the HPC system I use at work is based on Ubuntu. I wouldn’t choose Ubuntu or any Debian derivative (a matter of personal taste), so the obvious choice was openSUSE Leap, and in particular 15.1 that had just been released.

Reinstalling was not an option, as the server accepted mail, so I did not want to risk downtime. Like the other times I did this, I put a new server in parallel, migrated all the mail, web and other services to it, then decommissioned the old one.

For most of the bits I run the process was simple enough as rsyncing the files to the new ones, or turning off services for the time needed to make database dumps, restore them, and flipping the DNS if required. With the occasional hiccup, I would say that I got most of them working within a couple of days (including DNS propagation).

Mail, however, was a different issue.

Saying goodbye to Kolab, forever

I used to run Kolab Server 3.4, but as time passed I found many barely tolerable issues with it. Some of these were related to the software itself, some in the way Kolab Systems (the company which does most of the Kolab Server work, and the owner of the Kolab Now! mail service) handled the software itself and the relationships with the community and downstreams. Let’s get to each of these issues in order.

Quirks with the software

Kolab is an incredibly complex piece of software which includes C++ parts (including forks of some libraries developed originally by KDE), a series of daemons, Postfix for SMTP, Cyrus as POP/IMAP server, and Amavis/ClamAV/Spamassassin for spam handling. It offers a user database based on LDAP, and a web interface to administer users and mailboxes. The first problem is exactly this complexity: Kolab is incredibly fragile and some configuration changes may break everything with obscure errors.

Also, some of the defaults make sense for Kolab Now, but have little sense for many other users, for example making the primary username the surname (real or fake) of the user. Changing these is not trivial and very poorly documented (read more on this later). Multi domain support, by default, is basically non existent, and to be productive you might want to rely on scripts made by third party contributors to make everything manageable (in this case, again the documentation is unhelpful). Lastly, it doesn’t work with SELinux: you have to run everything in permissive mode.

And here also comes the software problem. Kolab has PHP bindings for its C++ libraries, which are tied to the PHP version in the distro. So you can’t really use PHP from another repository without risking breaking anything (I know on CentOS you can use SCLs, but this means an additional setup cost). You can’t run Kolab in LXC, nor in Docker. I had to resort to very ugly hacks to run other software that required at least PHP 7.

Documentation is a sore point of Kolab in general: painfully incomplete, sometimes hard to read, or woefully out of date. A perfect opportunity for someone from outside. to fix it… if how the whole thing worked was actually understandable. Some advanced features of Kolab are completely undocumented: the Chwala file sharing component, for example, can tie to the Seafile web file sharing application, but it was not documented anywhere and the configuration options were hidden in mailing list threads. It took a lot of digging to get all of this in shape and contribute as a HOWTO.

Upgrade paths are complex, unsupported, and require manual migrations of the databases. It is thanks to the efforts of a couple of community members that they were available at all. In addition, some newer versions introduced major regressions, like with the new component Guam, which was supposed to act as a proxy in front of Cyrus IMAP and hide the special IMAP folders Kolab uses for calendars, contacts, and even files, which was obviously not ready when Kolab 16 was released (it couldn’t even understand SSL). On top of that, there hasn’t been a new version since Kolab 16: upstream development moved to “Winterfell”, a continuously developed, integrated, and deployed version. Probably very useful for what Kolab Systems does with Kolab Now!: I’m not sure it would benefit others, though.

Relationship with upstreams and downstreams

First and foremost, all bits of Kolab are Free Software and use open standards, and their developers know this very well. A huge point in their favor. However, Kolab Systems has a very poor handling of projects that are upstream and downstream of Kolab, unless they are part of the upstream themselves (like with the Roundcube Mail web mail client, or Cyrus IMAP itself). In fact, I think their extent of collaboration with upstreams that aren’t directly related to their interest is non-existent: KDE developed Kirigami, whose only dependency is Qt, yet the Kube mail client just got rid of the “unnecessary dependency”, and also went on to fork a couple of PIM libraries (PIM and KDAV): while probably they needed this for legitimate reasons, it only caused extra fractures in a community (KDE PIM) which is already very short on developers. Developed projects that had no interest to them were basically abandoned without notice, like the Qt5 version of libkolab, which was ultimately forked by KDE PIM due to its lack of maintenance. And let’s not forget a rather heated discussion on KDE Frameworks release policies on kde-frameworks-devel.

In addition, they don’t seem to care at all about downstreams. Packaging Kolab for a distribution is an exercise in futility, due to the number of downstream patches, occasional git snapshots, and other very Kolab-specific bits that do not fit well in an integrator’s workflow. I don’t mind if Kolab Systems don’t do that themselves (they need to keep on making money to stay alive), but enabling or facilitating integration downstream would help a lot.

Lastly, contributions. For a project that is this complex, it’s not unusual to expect few contributions. Yet, basically every interaction with the community ended up in silence, no communication, and outright failure. On this topic, let’s note the failed Indiegogo campaign for Roundcube Next (never ever did anything with zero communication), the almost dead mailing lists, and the fact that the official forums are a spamfest (and for some time, they weren’t even reachable).

The verdict

I had to move away from Kolab because I felt that the limitations of the software were a problem and the issues around the community at large were too much (good luck trying to find help). Thus, I had to move to another solution. I chose mailcow-dockerized mostly because its feature set overlapped most with what I needed (including shared IMAP folders through email addresses, which was a killer feature of Kolab), and also had a nice web interface to configure part of the system (including some extra bits like DKIM, 2FA, or temporary mail addresses). I don’t really like the reliance on Docker and it’s not perfect, but it does the job good enough for what I needed to do.

Migration of the mail

First, I created users corresponding to whatever was on the Kolab server, including a few extra aliases. Then, I changed the SPF records to also include the new mail server (but keep the old one as well). Migrating the mail was a matter of using imapsync with a cron job running every 15 minutes (and many special cases to map folders correctly between the two instances). Some users were very opinionated on spam handling, so I had to make sure some corner cases (like no graylisting) were covered, and the spam thresholds were set for them so that messages would never get dropped.

The effort paid off as when I migrated the accounts (I don’t have that many users, so a few Thunderbird instance on desktops and K-9 Mail on Android) most people did not even notice the change.

Wrapping up

Then it was time to move my own (large) account to the new infrastructure. I couldn’t point the old KMail account to it as the special Kolab folders would be missing, so I armed myself with patience and created a new account from scratch. Calendar and contacts were exported to ICS and VCF, and then imported back in the new platform. KMail has native support of CardDAV/CalDAV used by mailcow (SOGo Groupware) so adding the relevant address book and contacts was easy.

Overall I’m pretty satisfied. The software stack is more up to date and on a platform (openSUSE Leap 15.1) that I am far more familiar with, and while complex, I can understand the components used by mailcow (Dovecot, rspamd, postfix) better than what I used to be able to with Kolab. As a plus, the newer software stack allowed me to do some changes in the web part, finally scrapping GitLab (no fault of its own: too big for what I needed), replaced by Gitea and Drone for CI.

As usual it was a pretty good learning experience and there are a couple more additions I plan on doing in the future.

After migrating my users (some did not even notice the change), I took the plunge myself and configured KMail for CalDAV/CardDAV for contacts and calendars, plus a new IMAP account. Within three or four days of testing (plus many other tests I did with a VM a couple of weeks before, and a few bug reports filed and fixed upstream), I removed the MX record pointing to the old server, and kept only the new one. Everything seems to be running smoothly so far, and the old server will be officially decommissioned tomorrow.