Day: January 31, 2007

I must be a little bit crazy and anti-paranoid about security for advertising this, but I also think that testing infrastructure is very important.

That’s why I have begun setting up different test IMAP servers for tinymail development. The domains might still have to be distributed to your providers DNS servers, I still have to make the mailbox for the user, etc etc.

I already created a little web interface that allows you start and stop the IMAP servers. Of course if I see too many planet users trying that out at the same time, I will disable it for a day or two. Just assume that it works :-).

The unit test environment of tinymail will probably get some scripts that will turn off and on certain IMAP servers for in case they will be testing capabilities that are specific for a specific IMAP server. They will indeed test online and reset the mailbox to its original state (which reminds me that I still have to add that to the web script).

Other projects are allowed to use the IMAP servers too (but do read the abuse item on the wiki page). Though there’s no promise that the services will always be available at all times and in its current shape.

I configured three virtual machines for this. Two of them are already running IMAP server software: one is running Dovecot and one Cyrus (the webscript allows for multiple installs to be stopped and started). Both the very latest and most unstable version that I could find. Yes I’m anti-paranoid about security and I do understand the risk that I’m taking here (and by blogging about it, I just increased that risk).

I welcome help from the server developers. If they want root on the machine and perform upgrades to the install of their softwares, then we can definitely discuss that. I’m also going to allow commercial IMAP softwares. But once many such servers have to run on the hardware simultaneously, I will not sponsor the hardware myself any longer. I guess that the commercial ones will have to get me hardware then.

So you can check more about it out here. I will be fine tuning it this evening and I might already change some unit tests to start using them too.

With the focus on Lemonade IMAP features and (therefore) wanting to optimize bandwidth utilization, I added support for the BINARY IMAP capability to tinymail’s IMAP code. The binary retrieval reduces retrieval size of certain messages (encoded is typically larger than the original message). Together with the existing support for CONDSTORE, STARTTLS with OpenSSL or NSS (which should compress too) and the optimizations for when you are retrieving the summary, this should mean that tinymail is indeed getting better and better at making sure that it’s not wasting your expensive GPRS connection.

I’m planning to write or steal somebodies ENVELOPE response parser so that I can use that in stead of the optimized summary retrieval code. This ain’t going to make a huge difference (just a few bytes per summary item). Those few bytes do add when retrieving really large folders, so it’s still worth the development time (… indeed, and I agree with you. Don’t worry).

Other plans include again rewriting the summary code (that’s the mmap stuff). This time I want to store segments of the summary data in mmap()ed files. For example 1000 items per such file. Then read-only mmap all of the frozen ones. Then when the next 1000th summary item is added, store the last 1000th in a file and mmap it too. Adding it to the list of segments.

The reason for this is that with the Lemonade IDLE code, adding summary items is going to happen more sporadically. Right now it’s very predictable when lots of summary items are going to be added to the summary: when you are retrieving or synchronizing your summary of course. With Lemonade’s IDLE there is less control over it. It’s the IMAP server that will “Push” this to the client.

I want to get it that stable and good that while this is happening, I want the view to start updating itself. This is what the TnyFolderObserver and TnyFolderMonitor are going to be responsible for: the IDLE event will trigger the invocation of the update method on all registered observers of the folder instance. The monitor will be such an observer that will deliver it to the list models of your views (those are TnyList implementations like TnyGtkHeaderListModel which is one that also implements GtkTreeModel).

Another reason is that that way I will also put the changeable data, which is only the flags, in a separate file. This will reduce level wearing on file systems that run on flash devices whenever changes to the summary happen. That’s because all the flags together will easily fit in a few blocks, and it’s not interesting to keep integers in the mmap. The pointer to that integer would consume as much memory as the integer itself. Leaving it out of the mmap()s will reduce the filesize of the segments, and the complexity to parse through it.

A last reason is that I can make looking up a summary item using the message’s uid a lot faster this way: I would store the summary items sorted on uid in those segments. So a little bit of simple math to find out in which segment it will belong (which would serve as a first index) and then utilizing a search that is optimized for sorted data (preferable one that doesn’t make big steps while comparing, as that would thrash possible caches). This is also going to help me a lot with mimicking a VFolder feature in tinymail, that works offline too (not just SEARCH commands on IMAP servers), that is — which is important — fast enough (and will also push found items to the views using the TnyFolderObserver stuff, while searching the read-only mmap()s in the background).

Equally important about such a search feature is that tinymail can NOT, no absolutely NOT, consume as much memory as Evolution does by keeping all summary of all folders in memory at all times.

There are a few other todo items too. Which are more bound to tinymail itself and less to tinymail’s camel-lite code. But those are all documented here already.

I hope the plans sound good :-). But I’m absolutely interested in your technical opinions. The future is going to be challenging, but definitely not undoable. The undoable part (starting the project and not giving up on it), is already done. Right?