With the focus on Lemonade IMAP features and (therefore) wanting to optimize bandwidth utilization, I added support for the BINARY IMAP capability to tinymail’s IMAP code. The binary retrieval reduces retrieval size of certain messages (encoded is typically larger than the original message). Together with the existing support for CONDSTORE, STARTTLS with OpenSSL or NSS (which should compress too) and the optimizations for when you are retrieving the summary, this should mean that tinymail is indeed getting better and better at making sure that it’s not wasting your expensive GPRS connection.
I’m planning to write or steal somebodies ENVELOPE response parser so that I can use that in stead of the optimized summary retrieval code. This ain’t going to make a huge difference (just a few bytes per summary item). Those few bytes do add when retrieving really large folders, so it’s still worth the development time (… indeed, and I agree with you. Don’t worry).
Other plans include again rewriting the summary code (that’s the mmap stuff). This time I want to store segments of the summary data in mmap()ed files. For example 1000 items per such file. Then read-only mmap all of the frozen ones. Then when the next 1000th summary item is added, store the last 1000th in a file and mmap it too. Adding it to the list of segments.
The reason for this is that with the Lemonade IDLE code, adding summary items is going to happen more sporadically. Right now it’s very predictable when lots of summary items are going to be added to the summary: when you are retrieving or synchronizing your summary of course. With Lemonade’s IDLE there is less control over it. It’s the IMAP server that will “Push” this to the client.
I want to get it that stable and good that while this is happening, I want the view to start updating itself. This is what the TnyFolderObserver and TnyFolderMonitor are going to be responsible for: the IDLE event will trigger the invocation of the update method on all registered observers of the folder instance. The monitor will be such an observer that will deliver it to the list models of your views (those are TnyList implementations like TnyGtkHeaderListModel which is one that also implements GtkTreeModel).
Another reason is that that way I will also put the changeable data, which is only the flags, in a separate file. This will reduce level wearing on file systems that run on flash devices whenever changes to the summary happen. That’s because all the flags together will easily fit in a few blocks, and it’s not interesting to keep integers in the mmap. The pointer to that integer would consume as much memory as the integer itself. Leaving it out of the mmap()s will reduce the filesize of the segments, and the complexity to parse through it.
A last reason is that I can make looking up a summary item using the message’s uid a lot faster this way: I would store the summary items sorted on uid in those segments. So a little bit of simple math to find out in which segment it will belong (which would serve as a first index) and then utilizing a search that is optimized for sorted data (preferable one that doesn’t make big steps while comparing, as that would thrash possible caches). This is also going to help me a lot with mimicking a VFolder feature in tinymail, that works offline too (not just SEARCH commands on IMAP servers), that is — which is important — fast enough (and will also push found items to the views using the TnyFolderObserver stuff, while searching the read-only mmap()s in the background).
Equally important about such a search feature is that tinymail can NOT, no absolutely NOT, consume as much memory as Evolution does by keeping all summary of all folders in memory at all times.
There are a few other todo items too. Which are more bound to tinymail itself and less to tinymail’s camel-lite code. But those are all documented here already.
I hope the plans sound good :-). But I’m absolutely interested in your technical opinions. The future is going to be challenging, but definitely not undoable. The undoable part (starting the project and not giving up on it), is already done. Right?