11.10.2017
How to set up Fetchmail, Maildrop and Dovecot as a spam filtering IMAP “proxy”
Why?
OVH is my web hoster and domain name registrar, it also provides me with an email service for my domain names, e.g.: xxxx@mrhide.eu.
It comes with an SSL encrypted POP3 and IMAP endpoint, anti-spam, and a webmail GUI (Roundcube).
Their anti-spam feature adds a header and prepend the subject with [SPAM], there is one option to automatically delete
junk email, or forward it to another address, but there is no option to put it in a junk or spam folder.
I could have made it delete all junk emails, but there are many false positive!
They have a paid for Exchange (Microsoft) service that comes with many more features.
Because they don't want their free offer to compete with the commercial one, they only provide a crippled free service.
What happens if you ask them for server-side filtering? They either tell you to upgrade your email plan, or to set up Google Apps to benefit from Gmail's server-side filtering feature.
I can use client-side filtering, it works well in thunderbird, but the android email client misses that feature ...
I came up with a not so simple, not so elegant solution, that requires a dedicated server (that I already have).
Note: this article is a simple workaround, there is a more professional approach https://www.rosehosting.com/blog/how-to-set-up-server-side-email-filtering-with-dovecot-sieve-and-roundcube-on-a-centos-6-vps/.
Requirements
- Fetch emails from the IMAP service at OVH.
- Do not remove emails from the original IMAP service (If my server crashes, I don't want to lose emails).
- Put spam in the junk folder.
- Create a new IMAP service to serve the filtered mail box.
Fetching and filtering emails
Install Fetchmail, Maildrop and
Dovecot, these are widely used programs and should be available through your Linux
distribution's package manager.
Fetchmail will act as a MTA (Mail Transfer Agent), it will fetch mails from my hoster, and send them
to Maildrop, our MDA (Mail Delivery Agent) (or LDA (Local Delivery Agent)).
Maildrop can filter emails based on a language that applies
on the source file of an email (its headers and content) and put it in a mailbox.
Then Dovecot will serve emails from that mailbox.
All the following shell commands must be run as root.
First we need to create a user that will contain dotfiles to configure software and our mailbox:
groupadd -g 5000 emailproxy useradd -m -d /srv/emailproxy -s /bin/false -u 5000 -g emailproxy emailproxy
Second we create a mailbox using the maildir++ format:
maildirmake /srv/emailproxy/.maildir
We also create a Junk folder in our mailbox:
maildirmake -f Junk /srv/emailproxy/.maildir
Created file structure should look like this:
tree -a /srv/emailproxy/.maildir /srv/emailproxy/.maildir ├── cur ├── .Junk │ ├── cur │ ├── maildirfolder │ ├── new │ └── tmp ├── new └── tmp 7 directories, 1 file
Create a .mailfilter filter for Maildrop (this file will be parsed after /etc/maildroprc):
cat > /srv/emailproxy/.mailfilter << EOF
if (/^X-Ovh-Message-Type: SPAM$/)
to $DEFAULT/.Junk/
EOF
This file tells Maildrop to put emails in the Junk folder if it matches a pattern. Here, the header added by OVH's anti-spam service. Of course, you can define more rules to sort your emails here, eg: put email from mailing lists in a special folder.
Create a .fetchmailrc file to configure Fetchmail:
cat > /srv/emailproxy/.fetchmailrc << EOF poll ssl0.ovh.net protocol imap username "××××××××" with ssl with password "××××××××" mda "/usr/bin/maildrop" keep EOF
The syntax is verbose and explicit, it's just like an order you could have told to somebody.
Fetchmail when started will read this file, poll the given IMAP service, fetch all new emails, pass them to Maildrop and
mark them as read, it will not delete emails (keep).
Finally, fix ownership and permissions:
chown -R emailproxy:emailproxy /srv/emailproxy/ chmod -R 2770 /srv/emailproxy
Invoking fetchmail as user emailproxy should populate the maildir mailbox.
Serving emails
First, I don't want anyone but me to be able to read my emails, so I decided to use SSL here. Hopefully dovecot has a script to ease the generation of an RSA key and an SSL certificate, it should be in its documentation: /usr/share/doc/dovecot/mkcert.sh using the configuration file /usr/share/doc/dovecot/dovecot-openssl.cnf (that script is short and very easy to understand, and can be customized using environment variables).
Invoking mkcert.sh will create a key in /etc/ssl/private/dovecot.pem and generate an SLL certificate in /etc/ssl/certs/dovecot.pem.
First I'll edit /etc/dovecot/dovecot.conf to override one parameter:
# Only serve using IMAP protocol protocols = imap
Then I'll tell it where to look for inboxes, overriding only one parameter in file /etc/dovecot/conf.d/10-mail.conf:
mail_location = maildir:~/.maildir
Then I'll enable SSL, in file /etc/dovecot/conf.d/10-ssl.conf:
ssl = required ssl_cert = </etc/ssl/certs/server.pem ssl_key = </etc/ssl/private/server.key
Finally, I'll configure dovecot to use a passwd user/password database in file /etc/dovecot/conf.d/auth-passwdfile.conf.ext:
passdb {
driver = passwd-file
args = scheme=CRYPT username_format=%u /etc/dovecot/users
}
userdb {
driver = passwd-file
args = username_format=%u /etc/dovecot/users
}
And the corresponding /etc/dovecot/users file:
emailproxy:××××××××:5000:5000::/srv/emailproxy:
The syntax here is very simple, it's the same as /etc/passwd, because the purpose of this driver is tu use that very file. For more information, see dovecot's wiki page.
Systemd setup
Because systemd is sacred, systemd is great! ... nah I really like systemd, it is a nice tool, and it's the default init daemon of my favourite Linux distribution.
Autostart fetchmail daemon for user emailproxy:
cat > /etc/systemd/system/fetchmail-proxy.service << EOF [Unit] Description=A remote-mail retrieval utility After=network.target [Service] User=emailproxy ExecStart=/usr/bin/fetchmail -d 60 RestartSec=1 [Install] WantedBy=multi-user.target EOF systemctl enable fetchmail-proxy systemctl start fetchmail-proxy
Autostart dovecot: my distro has a correct systemd unit for dovecot, so I will just enable and start it:
systemctl enable dovecot systemctl start dovecot
Client setup
I will use OVH's SMTP server for outgoing emails and my newly setup IMAP server for incoming emails, I found no reason to put a proxy in front of a SMTP that I don't control.
Use the credentials defined in the /etc/dovecot/users file, the IP or domain name of the machine hosting our proxy,
port is 993 (SSL).
Usually you'll have to tell your client to trust the self signed certificate generated by the mkcert.sh script.
And .... we're done, I don't know if it's the best setup to achieve that simple goal, but it works well for me!
10.09.2017
LD39 Postmortem
For 5 years, with my friend Cyto, We have participated in the Ludum Dare Jam
event of august. We didn't deliver every time but we always had fun.
This year we delivered our most polished game since we started doing this!
Learn more about this game on its ldjam entry page:
I have to say that this game received the highest score we ever had, and this is rewarding, because while making this game, we had the feeling that we were well prepared for once. We tried our best to avoid the errors and pitfalls we used to fell into every time.
The questions are:
- What went wrong?
- What went right?
- What was different this time?
- How could we improve?
What went wrong
This is the hardest question, because to me it felt like everything went right, but of course some things must have gone wrong, the hardest was to keep our motivation, especially because after two straight days of gamedev, every small distraction is hard to ignore.
What went right? What was different this time?
This time we weren’t more prepared or more motivated than the previous times, we didn’t have a better or more complete codebase to kickstart coding, we didn’t warm up the week before. We were just as unprepared as everytime we joined in in the past.
This jam was a success because of a few things:
- We live in the UTC+2 timezone … we decided to go to sleep early to wake up at 9am. why used to wait until 3am for the theme, now I understand why this is a stupid idea, you start the jam by shifting your body clock which makes you tired, then waking up in the morning to discover the theme and start working with a fresh and rested brain is a real advantage that we, European participants, have over Americans.
- We’ve found a good idea almost immediately (brainstorming while having a nice breakfast is effective!)
- We decided to use GitHub’s new Projects feature to keep track of tasks using these columns: minimum to do to deliver, extra features, in progress, done.
- We started by breaking up the work to be done in smaller tasks and filled the first two columns according to the priority of these tasks.
- We focused our efforts on the first column.
- We managed to finish most of its tasks during the first two days! (epic success).
It seems that sleeping well and being organised is the way to go to deliver.
How could we improve?
The main reason why we felt more comfortable this time is probably because it’s our fifth participation, we are getting used to game jams and did not fall in the usual traps. But nobody’s perfect and there is still room to improve:
- Be prepared:
Every time we joined in we weren’t prepared, I do enjoy making games from scratch but I also want to save time by not rewriting the same code every game jam, our codebase has been rewritten a few time, we switched from C to C++ to benefit from object oriented programming, templates and the STL, all of this has improved our efficiency, we no longer waste time writing “engine” parts for the game. - Practice:
I’ve bought some equipment to create assets, a USB microphone (Samson C01U) and a pop filter, a USB midi keyboard (Akai LPK25) and a graphic tablet (Wacom Intuos). But having fancy gear is useless if you don’t know how to use it properly, I spent some time playing with it to learn how to create using various software (Aseprite, Krita, LMMS, Audacious, …) - Jam:
Participate in game jams more often, it’s fun!
28.07.2017
New release: libTMX ver. 1.0
libTMX version 1.0 has just been released!
Reminder: libTMX is an open source C library to load maps made with Tiled, its name comes from the extension of map file: .tmx.
This is a milestone! 1.0 is always a special version for every project,
even for small projects.
I have been maintaining that C library for more than four years now, and its
version numbers have always followed those of the supported Tiled release.
Even though I do not really decide on version numbers, I wanted to make that release special and implement many new features!
Here's the changelog:
New Features:
- Use a hashtable to store properties
- Add more IO options: load maps from a file descriptor, a buffer or using a callback
- Add a tileset manager to hold references to external tilesets
First, a long time requested feature: access properties via their names. Since the removal of support for JSON formatted map, the XML parser has been the only supported parser, hence it became mandatory (you cannot build libTMX without the XML parser). Therefore libxml2 has became a compulsory dependency of libTMX. Now I have to share a little secret ... libxml2 has an implementation for the hashtable datastructure! I just added a few fuctions that delegate to libxml2's implementation.
/* Returns the tmx_property from given hashtable and key, returns NULL if not found */ TMXEXPORT tmx_property* tmx_get_property(tmx_properties *hash, const char *key); /* ForEach callback type to be used with function tmx_property_foreach(...) */ typedef void (*tmx_property_functor)(tmx_property *property, void *userdata); /* Calls `callback` for each entry in the property hashtable, order of entries is random */ TMXEXPORT void tmx_property_foreach(tmx_properties *hash, tmx_property_functor callback, void *userdata);
Second, another requested feature, TMX should be able to load a map from a buffer, as libxml2's reader accepts many kind of IO as input, once again I just added new functions that delegates to libxml2.
/* Loads a map from file at `path` and returns the head of the data structure returns NULL if an error occurred and set tmx_errno */ TMXEXPORT tmx_map* tmx_load(const char *path); /* Loads a map from file at `path` and returns the head of the data structure returns NULL if an error occurred and set tmx_errno */ TMXEXPORT tmx_map* tmx_load_buffer(const char *buffer, int len); /* Loads a map from a file descriptor and returns the head of the data structure The file descriptor will not be closed returns NULL if an error occurred and set tmx_errno */ TMXEXPORT tmx_map* tmx_load_fd(int fd); /* allback used by tmx_load to delegate reading to client code userdata(in): user data passed to tmx_load() buffer(in): to store read bytes len: how many bytes to read (length of buffer) */ typedef int (*tmx_read_functor)(void *userdata, char *buffer, int len); /* Loads a map using the given read callback and returns the head of the data structure returns NULL if an error occurred and set tmx_errno */ TMXEXPORT tmx_map* tmx_load_callback(tmx_read_functor callback, void *userdata);
The main addition here is the tileset manager, its main purpose is to avoid loading tilesets multiple times, another benefit is to preload tilesets to make them available when using the new IO load methods.
/* Tileset Manager type (private hashtable) */ typedef void tmx_tileset_manager; /* Creates a Tileset Manager that holds a hashtable of loaded tilesets Only external tilesets (in .TSX files) are indexed in a tileset manager This is particularly useful to only load once tilesets needed by many maps The key is the `source` attribute of a tileset element */ TMXEXPORT tmx_tileset_manager* tmx_make_tileset_manager(); /* Frees the tilesetManager and all its loaded Tilesets All maps holding a pointer to external tileset loaded by the given manager now hold a pointer to freed memory */ TMXEXPORT void tmx_free_tileset_manager(tmx_tileset_manager *ts_mgr); /* Loads a tileset from file at `path` and stores it into given tileset manager `path` will be used as the key Returns 1 on success */ TMXEXPORT int tmx_load_tileset(tmx_tileset_manager *ts_mgr, const char *path); /* Loads a tileset from a buffer and stores it into given tileset manager Returns 1 on success */ TMXEXPORT int tmx_load_tileset_buffer(tmx_tileset_manager *ts_mgr, const char *buffer, int len, const char *key); /* Loads a tileset from a file descriptor and stores it into given tileset manager The file descriptor will not be closed Returns 1 on success */ TMXEXPORT int tmx_load_tileset_fd(tmx_tileset_manager *ts_mgr, int fd, const char *key); /* Loads a tileset using the given read callback and stores it into given tileset manager Returns 1 on success */ TMXEXPORT int tmx_load_tileset_callback(tmx_tileset_manager *ts_mgr, tmx_read_functor callback, void *userdata, const char *key); /* Load map using a Tileset Manager */ /* Same as tmx_load (tmx.h) but with a Tileset Manager. */ TMXEXPORT tmx_map* tmx_tsmgr_load(tmx_tileset_manager *ts_mgr, const char *path); /* Same as tmx_load_buffer (tmx.h) but with a Tileset Manager. */ TMXEXPORT tmx_map* tmx_tsmgr_load_buffer(tmx_tileset_manager *ts_mgr, const char *buffer, int len); /* Same as tmx_load_fd (tmx.h) but with a Tileset Manager. */ TMXEXPORT tmx_map* tmx_tsmgr_load_fd(tmx_tileset_manager *ts_mgr, int fd); /* Same as tmx_load_callback (tmx.h) but with a Tileset Manager. */ TMXEXPORT tmx_map* tmx_tsmgr_load_callback(tmx_tileset_manager *ts_mgr, tmx_read_functor callback, void *userdata);
Also, because libxml2 has embedded http and ftp support, you can load maps and
tilesets from the internet! just pass the URL to load_tmx(), simple isn't it?
Tiled 1.0 TMX format support:
- Add text object support
- Add group layer support
- Add tile type support
See the TMX format changelog for Tiled 1.0 for more details on these new elements and attributes.
24.06.2017
About myself
My name is Jonathan Bayle, I'm also known as baylej and Mr-Hide, I'm a 27
year old French guy and I'm a wannabe game developer.
I have a masters degree in computer science with a speciality in 3D rendering.
I don't work in the video game industry at the moment but I would really like to.
Right now I make scalable webapps for the ESA.
I make video game as a hobby, click the portfolio link in the navbar to see all my small and mostly shitty games.
My favorite languages are C/C++ and I like to use low level libraries instead of full featured engines. I like C so much that I made an emscripten port of a javascript game library.
My favorite library is Allegro5, it's like an enhanced
SDL that has many more features, but also allows you to use it just to create a
window and handle input while you use OpenGL directly for rendering.
Is Allegro5 superior to SDL2? Yes!
Most of my games were made during Game Jams, I like the competitive aspect of games jams, it's a good incentive and rules plus the limited time stimulates each one of us. Every year I participate in the LumdumDare August jam with my good friend Cyto, we don't deliver each time but we do our best and learn a few new things.
Of course I'm an avid gamer, my favourite genre is the plateformer genre, games like Cave Story, Rayman legends, Abe's Oddysee, ... I'm also into several competitive multiplayer games that I play with my mates.
I have a few other hobbies, I own an old car, a Citroën 2cv6 that I restore and
maintain, I seldom drive it, only when the weather is great and/or someone want
to go for a ride.
I'm also a paraglider pilot.