Trace:

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Next revision		Previous revision
lacre:notes [2023/06/29 19:49] – created pfmlacre:notes [2023/10/12 22:35] (current) – sqlalchemy/aiosmtpd conflict pfm
Line 56: Line 56:
  
 On our test environment, we’d found an [[https://git.disroot.org/Disroot/gpg-lacre/pulls/64|issue with the cron-job]]. To fix it, I’ve written some tests for the cron-job too, but since this script reads keys from database, I needed a test database. To do that, I’ve replaced ''%%MySQLdb%%'' package with [[https://www.sqlalchemy.org/|SQLAlchemy]] and started using SQLite for tests. This also makes it possible to use Lacre with other databases. On our test environment, we’d found an [[https://git.disroot.org/Disroot/gpg-lacre/pulls/64|issue with the cron-job]]. To fix it, I’ve written some tests for the cron-job too, but since this script reads keys from database, I needed a test database. To do that, I’ve replaced ''%%MySQLdb%%'' package with [[https://www.sqlalchemy.org/|SQLAlchemy]] and started using SQLite for tests. This also makes it possible to use Lacre with other databases.
 +
 +====== Implementing an Advanced Content Filter ======
 +
 +Of course I've started by making sure I can even test the advanced filter, so I've set up end-to-end tests similar to those already used by the simple content filter. Eventually I've rewritten tests for both content filters to use [[https://docs.python.org/3/library/unittest.html|unittest]] Python library. Since all these tests are defined as sets of parameters for the same test code, I've just used [[https://docs.python.org/3/library/unittest.html#unittest.TestCase.subTest|subTest]] generator to run them all.
 +
 +Benefits:
 +  * The same test code can be executed many repeatedly (instead of being interrupted by the first fail).
 +  * Passing keyword arguments to identify a sub-test makes failure messages more useful.
 +
 +Downsides:
 +
 +  * These tests don't produce any output, so it might look like they were frozen.
 +
 +While implementing advanced filter, I've also learned about [[https://docs.python.org/3/library/asyncio-eventloop.html#asyncio-event-loop|event loops]] used by coroutines.
 +
 +  * One should make sure that event-loops are used consistently.
 +  * Starting a task is very easy.
 +  * There are semaphores and other [[https://docs.python.org/3/library/asyncio-sync.html|synchronisation primitives]] for ''asyncio''.
  
 ====== Next steps ====== ====== Next steps ======
Line 61: Line 79:
 In the near future I’d like to: In the near future I’d like to:
  
-  * improve the logging code (it’s already in progress); 
   * polish the code handling configuration;   * polish the code handling configuration;
-  * consider using [[https://www.postfix.org/FILTER_README.html#advanced_filter|advanced Postfix content filters]], which would make the overall solution more efficient (simple filters start a new process for each email while advanced filters are based on a separate daemon which is started once and lives for a long time).+  * update documentation. 
 + 
 +====== Asynchronous I/O ====== 
 + 
 +I'm having issues with integrating SQLAlchemy and aiosmtpd. 
 + 
 +Stack Overflow entries: 
 + 
 +  * [[https://stackoverflow.com/q/46827007|RuntimeError: This event loop is already running in python]] 
 +  * [[https://stackoverflow.com/q/63535313|Python RuntimeError: This event loop is already running]]