The goal is to reanimate gpg-mailgate (of which Lacre is a fork) by porting it to Python 3.x, improve its documentation and in the future, maybe adapt to other services. We also plan to integrate it with Disroot’s webmail.

Before changing anything, I’ve spent some time reading the code.

• There are no tests.
• Main Python script called by Postfix is quite large. By moving most of the code to a dedicated module we could improve performance by having that code compiled.
• There’s a GnuPG module (a wrapper for /usr/bin/gpg). We should consider reusing an existing module (e.g. https://github.com/isislovecruft/python-gnupg).
• gpg-mailgate-web module, which is expected to accept public keys provided by users, deletes keys before checking if it can actually add keys. This could be used by a malicious actor to delete user’s keys and have their email stored in cleartext.
• According to gpg-mailgate documentation, the user nobody should be used for the filter. This is probably a workaround of some sort and we’ll need to see how to fix the root cause.
• Main script isn’t configurable enough, e.g. it’s not always true that system configuration is stored in /etc directory (on FreeBSD it could be /usr/local/etc).

I’ve asked two of the authors of the original gpg-mailgate project about their commits mentioning user nobody. One said they just didn’t want to make the system user’s name to reveal the purpose, the other didn’t remember the reasoning behind it.

My conclusion is that perhaps we can use a dedicated system user for GPG Lacre in the future.

Since I need to test my work somewhere, I’m setting up a FreeBSD jail to run a test instance Postfix server. I’ve created the jail following instructions from the handbook: 15.3. Creating and Controlling Jails.

Then, to install software inside the jail, I’ve used pkg -j $name from the host system, so I don’t have to connect the jail to the network.

To use GnuPG in a FreeBSD jail, one has to add the allow-loopback-pinentry option to ~/.gnupg/gpg-agent.conf and use --pinentry-mode loopback whenver calling gpg, as reported in this FreeBSD bug report: security/gnupg: pinentry-tty dumps core because of missing privelege (esp. see comment #11 and #12).

Before modifying GPG-Mailgate code, I wanted to cover it with at least some tests, because there were none and I would feel much more confidend modifying the code if I had a chance to verify it still works as expected.

I’ve started by writing an E2E test script that would perform the following steps:

• Set up a test mail relay listening on a given TCP port.
• Write a test GPG-Mailgate configuration file.
• Run GPG-Mailgate with a test message and the configuration prepared in the previous step.
• Collect output from test mail relay, verify it matches expected output.

However, GPG–encrypted messages include a packet with a timestamp, so we cannot just compare output against a message encrypted at another time. I’d like to avoid including private keys in the repository, so at the moment I’m looking for other options.

Having covered the project with at least these tests, which can be triggered using a simple command (see testing documentation for details), I’ve migrated the code to Python 3.x automatically. Implementing tests first proved to be the right approach, because auto–migrated code didn’t work out of the box. The change between Python 2.x and 3.x that caused most of the issues was the introduction of bytes type used by IPC. Thanks to the tests, it was very easy to fix the issues.

Finally I’ve managed to open a pull-request to share my work with those who still maintain or use the original repository. It’s been merged soon after my submission.

On our test environment, we’d found an issue with the cron-job. To fix it, I’ve written some tests for the cron-job too, but since this script reads keys from database, I needed a test database. To do that, I’ve replaced MySQLdb package with SQLAlchemy and started using SQLite for tests. This also makes it possible to use Lacre with other databases.

Of course I've started by making sure I can even test the advanced filter, so I've set up end-to-end tests similar to those already used by the simple content filter. Eventually I've rewritten tests for both content filters to use unittest Python library. Since all these tests are defined as sets of parameters for the same test code, I've just used subTest generator to run them all.

Benefits:

• The same test code can be executed many repeatedly (instead of being interrupted by the first fail).
• Passing keyword arguments to identify a sub-test makes failure messages more useful.

Downsides:

• These tests don't produce any output, so it might look like they were frozen.

While implementing advanced filter, I've also learned about event loops used by coroutines.

• One should make sure that event-loops are used consistently.
• Starting a task is very easy.
• There are semaphores and other synchronisation primitives for asyncio.

In the near future I’d like to:

• polish the code handling configuration;
• update documentation.

I'm having issues with integrating SQLAlchemy and aiosmtpd.

Stack Overflow entries:

• RuntimeError: This event loop is already running in python
• Python RuntimeError: This event loop is already running