Author: admin

It’s not yet accepted but it looks like Dirk-Jan proposed doing a little chat about Modest at Guadec this year.

I figured it’s going to be a tinymail related presentation and maybe, who knows, will others do talks and presentations on the subject in future too? Although I might improve the location by giving it an actual page like the demos page, you can already find my own slides there. I left out the FOSDEM ’06 one as that one was about software design with GObject, not about tinymail. The FOSDEM ’07 one was about tinymail, so it’s available.

Oh .. I might be interested to do some marketing around tinymail. Nothing big though. I’m not sure yet: once Modest and other E-mail clients start getting used, it might automatically sell? Maybe marketing is overkill? The project might not be mature enough? Perhaps it’s just a bad idea? Perhaps it’s a great idea? I also don’t think there are a lot competent marketing people who understand free software (except for two crazy Australians, though that’s not marketing but rather ‘strategic consultancy’, right?).

Boy, it should be clear by now that I truly lack experience in leading a project. The idea of “leading a project” also scares the hell out of me. Given that I’m actually quite incompetent at some important social skills. Well, Daniel Coleman writes in his books about emotional intelligence that being self aware helps a lot. So I’m not fully incompetent! :-). At least one contributor actually wrote on his blog that he’s happy. I’m proud of that achievement! I never thought I could do that through a project (you know, making people happy).

Now these Subversion guys tell us, project leaders, that we need to be careful of both poisonous people and even of not becoming poisonous ourselves. I mostly agree with the end of their presentation (not with everything): that you need to be very careful with quickly identifying people as poisonous.

Over time a lot projects start consuming rather more than less memory. Mostly it’s not due to the added features themselves. Often it’s rather due to the amount of code growing and maintainability getting harder. A result is often that the good ideas and great ideals about memory consumption fade away a little bit.

I disliked that strategy from day one for the tinymail project.

Also note that most of the times just throwing more into ram, does not necessarily mean that your application will perform better. Often the exact opposite is the case: because it’s often due to your design being broken. Most hacks are just what they are: hacks. Most hacks are in reality slowing down things too.

I made the decision that I will design in a flexible and adaptive way starting day one. This means that new features have always been easy to add to the framework, cause of its design. It’s a misconception that good design automatically means a lot memory consumption. In fact, the contrary is often true. It’s also a misconception that you can’t use C for this. When designing, the programming language is mostly irrelevant. Although D is indeed a kick ass programming language :-) (boy, it feels good to say subjective crap in my blogs. Moeha!).

I remember the days when I flooded my own blog about how tinymail was only consuming something like say ~18MB of heap allocations for displaying a folder of 50,000 items. Well, I have improved that figure a little bit. Today tinymail consumes 8,613K when displaying a folder of 50,000 items, with a ~10MB mmap (which hasn’t significantly changed since).

I made a new memory report which is available on the trac. I added quite a lot documentation about mmap, about how tinymail uses the memory and about the slab allocator. I did this to make sure that people who’ll be really reading the report will have as few misconceptions as possible.

You can of course reproduce the report and all steps for reproducing it are documented too.

Imagine you just created a nice GObject with a so called “priv thing” in it. Yes, “thing” is a good name for now. In your code you access that “priv thing” using the G_TYPE_INSTANCE_GET_PRIVATE GType “stuff”. Most of the time you are a happy hacker with a “priv thing”. Then suddenly you have a bug somewhere in client code of that GObject type of yours (by client code I mean: code that is using an instance of your type, like calling API on it).

You’ll most likely want to peek at that “priv thing”? Doesn’t sound very unusual, right?

Well it’s actually quite simple:

We have a GtkTreeModel, which is a GTypeInterface implemented as TnyGtkHeaderListModel. The instance of that TnyGtkHeaderListModel is called “self”. The instance (which in this debugging session plays the role of the client code for a TnyFolder instance) will do some stuff with another instance stored at self->folder. With some casting voodoo magic we can easily get the “folder” instance out of that. Nothing difficult here.

Let’s do this step by step …

(gdb) print self
$17 = (GtkTreeModel *) 0x82f8090
(gdb) print *self
$18 = <incomplete type>
(gdb) print ((TnyGtkHeaderListModel*)self)->folder
$19 = (TnyFolder *) 0x8253ba8
(gdb) print *((TnyGtkHeaderListModel*)self)->folder
$20 = <incomplete type>
(gdb)

We are after the “priv thing” of that folder instance: we’re not there yet. But let’s dig a little bit deeper!

(gdb) print * (TnyCamelFolder*) ((TnyGtkHeaderListModel*)self)->folder
$21 = {parent = {g_type_instance = {g_class = 0x80e0b98}, ref_count = 2, qdata = 0x0}}
(gdb)

We’re still not there, and we still haven’t even seen a sign of that “priv thing”. That’s because it’s somewhere deep inside the “crazyness” of the glib GType stuff. Don’t worry, we have yet another gdb trick as tool in our pocket. We can actually launch some of our own code. Including code from the glib library.

(gdb) print g_type_instance_get_private (((TnyGtkHeaderListModel*)self)->folder, tny_camel_folder_get_type())
$22 = 136657848
(gdb)

Okay, let’s not care about the number and just use that $22 label.

(gdb) print *$22
$23 = 1
(gdb)

Well…of course! Our beloved debugger doesn’t know how that pointer looks like! Let’s tell it. Because “we” .. know, right?

(gdb) print *(TnyCamelFolderPriv*)$22
$24 = {loaded = 1, headers_list_type = 0, folder_changed_id = 132, headers_managed = 1898, folder_lock = 0x826af80, folder = 0x81f7868,
  folder_name = 0x826afb0 "INBOX/30000", account = 0x80e58f8, store = 0x80e5a08, cached_length = 1898, unread_length = 49, unread_sync = 0,
  local_size = 258652, subscribed = 1, has_summary_cap = 1, iter = 0x81f6800, iter_parented = 1, cached_name = 0x826afc0 "30000",
  cached_folder_type = TNY_FOLDER_TYPE_NORMAL, remove_strat = 0x820a8b0, receive_strat = 0x820a6a0, observers = 0x8259fb0,
  sobservers = 0x8259fc8, self = 0x8253ba8, want_changes = 1, dont_fkill = 0, parent = 0x81f3ee0}
(gdb) 

Maybe we should make some gdb-shortcuts and do some integration with gdb and glib? I do like things like refdbg and I like all those other tools too. Yet I still have to fall back to using gdb most of the times.

Let’s play some more, digging deeper:

(gdb) print ((TnyCamelFolderPriv*)$22)->folder
$25 = (CamelFolder *) 0x81f7868
(gdb) print *((TnyCamelFolderPriv*)$22)->folder
$26 = {parent_object = {klass = 0x82a9130, hooks = 0x82f9190, ref_count = 1, flags = 0}, priv = 0x8216e00, name = 0x82f75f0 "30000",
  full_name = 0x825d4f0 "INBOX/30000", description = 0x0, parent_store = 0x80e5a08, summary = 0x80b8368, folder_flags = 3,
  permanent_flags = 8194079}
(gdb)

(gdb) print *((TnyCamelFolderPriv*)$22)->folder->parent_store
$31 = {parent_object = {parent_object = {klass = 0x80de0e0, hooks = 0x80bca80, ref_count = 7, flags = 0}, priv = 0x80e55e0,
    session = 0x80b9c00, provider = 0xb6233da0, status = CAMEL_SERVICE_CONNECTED, connect_op = 0x0, url = 0x80e6230, data = 0x80e5ba0},
  priv = 0x80e5778, folders = 0x80e57a8, flags = 11, mode = 3}
(gdb)

Modest’s source code hasn’t been available for more than a week, and it is already supporting Push E-mail. I created a little video demo showing this feature.

We of course have Modest running on our Maemo devices, like the Nokia 770 and the N800. The Push E-mail feature works on the device too. You can expect video demos of Modest running on the device soon. For packages you will have to wait: we want to make sure that it’s rock stable and really works well.

Modest and the tinymail demo-ui showing Push-Email

Youtube version

Some notes: If you are going to experiment with your own Cyrus install, then make sure that you have the idled daemon of Cyrus running. You can also use these test IMAP servers for experimenting. They all support IMAP IDLE (therefore they support the Push E-mail of both tinymail and Modest). The Cyrus one also has condstore turned on on its INBOX folder, which is another Lemonade enhancement that you might want to test with.

I would like to stress that tinymail nor Modest have had intensive testing. Please don’t yet use it on real accounts unless you really don’t care about losing your E-mails and data. I try very hard to make sure that you do have testing infrastructure. You don’t have to test tinymail or Modest with your own accounts.

Oeps, some people are blogging that Modest is available. I guess that means that there’s no more escaping it. Its code is indeed available at http://modest.garage.maemo.org.

I would like to warn though, that both Tinymail and Modest still need a lot of work. It will not yet replace your Evolution. Whether or not it will eventually do that, depends on you wanting to invest time in this.

This is indeed an invitation for the community to get involved in creating both a framework for creating E-mail applications on mobile devices, and an E-mail client on top of it that might someday be used on both your desktop and your mobile device.

People are working on the IMAP code and Lemonade, on the POP3 code, on the code for handling the local cache and summaries, on language bindings. At FOSDEM I announced my plan to build-up a network of competence around the project. Nokia is interested, OpenMoko is interested, the One Laptop Per Child project seems to be interested. Igalia did some work, Kernel Concepts did some work. A large list of contributors did some work.

So, let us make this another success story like the Telepathy framework is for VoIP. And what can you do? A lot!

Maybe integrate Telepathy/Galago with an optional tinymail component like a summary and/or message view? Integrate a calendaring application? A contacts one? Enable the integration in Modest? Or maybe you prefer to do a language binding and write your own mail user agent? Or use the existing Python language binding for that? Maybe write a ui for timsieved (a daemon for deploying Sieve scripts) and that way make it possible for normal people to have server side filtering of messages into mailboxes, configurable with their cellphone? Maybe write a SyncML observable for the Push E-mail features of tinymail? Help me implementing more of the Lemonade IMAP enhancements? Help me with improving the summary code? Maybe integrate it with an existing desktop searcher and indexer, making searching E-mail on your phone possible? Melting E-mail, MMS and SMS together? Porting to WinCE? Why not?

There really is a huge to do list. It’s simply not doable for one person to ever finish all of the possibilities of what E-mail can and should offer to mobile users.

Oh, by the way. Implementing something that plays a sound when a new message arrives (a Push E-mail event) with the current API (just to show what tinymail can already do for a mobile device like a cellular phone):

public class OnMsgArrivesBeeper
	implements Tny.FolderObserver
{
   public void update (Tny.FolderChange change)
   {
      Tny.FolderChangeChanged c = change.get_changed ();
      if (c.ADDED_HEADERS)
         Phone.PlayRingTone (RingTones.NewMsg);
   }
}

So tinymail has these default components. And people have been confused by them as it causes them to believe that tinymail strictly depends on Gtk+ (or whatever it also supports through those default components). This is not true. In fact, I put a lot effort in making sure that I don’t depend on anything that might make it impossible to use tinymail on a different architecture.

I, for example, foresee that someday those who are building web applications want to use tinymail. What does this mean? This means: forget the availability of signals, most uses of singletons, the mainloop and some other things that I have yet to discover. Also forget the availability of things like gdk_threads_enter and gdk_threads_leave in the core library: I would have to link with the gtk+ libraries for this .. nope, not possible.

So, whatever you have been telling your friends about tinymail’s dependency on gtk+: you were probably wrong. Although you might have seen “Gtk+”-code in its distribution, it doesn’t necessarily depend on Gtk+.

Nevertheless are some of these components becoming powerful tools when you want to create quickly a Mail User Agent. This weekend I finished letting a GtkTreeModel type implement the TnyList, the TnyFolderObserver and TnyFolderStoreObserver types. Don’t worry, don’t worry, I’ll explain.

Well, I explain it here. That’s the documentation of that type. Because I figured that, well, a powerful component is worth nothing for the vast majority of you software developers, unless I document it. Right?

So I documented it. I, indeed, want tinymail to excel in its documentation.

Note that the vast majority of components in tinymail are optional. Being the application developer, you choose the ones you will use. The vast majority can be disabled at compile time, too. Tinymail is about the interfaces between the many components. Their implementations are reimplementable, inheritable, choosable, replaceable, …

Update: same story for this component (a list model for a folder summary): here. This one also documents how to get Push E-mail (Imap’s IDLE support) to auto-update your summary view, with tinymail.

I added basic folder management to the demo user interface of tinymail so that people can start testing this. The framework itself had this since long, but now I’m going to really start testing it. That’s because a Modest release is coming closer too. It’s of course going to be one of the basic functionalities of that MUA. The trivial stuff has to work too.

Basic folder management means creating, deleting and renaming folders. What you will be testing, if you are going to, is the entire thing behind all this: error reporting, creation on the IMAP service, creation of an instance for this, memory and reference counting when the amount of folders change, the folder observer stuff: does the view get updated?, the local cache refreshing: is the folder also removed locally?, etc. So when reporting bugs, don’t just focus on crashers. Not seeing it crash doesn’t mean that it worked. Go after the entire thing. It has to be a.b.s.o.l.u.t.e.l.y correct (can you imagine the despair of a user who has lost his INBOX because of a recursiveness mistake? I can, been there done that with another MUA).

For that reason we have the testing infrastructure. Please, please, people, I beg you, don’t yet use this functionality on your real accounts. I have set up testing servers: so I really did all that I can to give you the testing tools. Don’t use it on your own & real accounts. Final warning.

I also documented how to start using the testing servers in tinymail (I think it’s in the README). I even recommend only configuring those accounts. If you don’t configure any other account, no harm can be done. The testing infrastructure recovers automatically from disaster each 3th hour on IMAP2 and each service restart on IMAP1. Don’t worry about trashing it.

Sergio and Dirk-Jan are often discussing in parallel with me on software development of tinymail over instant messaging, so I decided to start an IRC channel on the topic. People with questions or who are actively developing on or with tinymail are invited to #tinymail on GimpNET.

Today I finished the observer stuff. You can register an observer to both a folderstore and a folder and get notified about folder renames, header adds and removes, count changes, folder removes and folder deletions (and other things). The default ui components will all auto act on such changes (by auto updating themselves as good as possible). You can of course register your own view (or model) as an observer of a folder too.

The observer/observable part of tinymail is the core-principle of the Push E-mail feature too. Actually of all the changes that need action use it. Implementing the role that the observable plays is what is flexible about this: It doesn’t matter whether it’s IMAP IDLE doing this, or an event coming from SyncML. And maybe tomorrow we will get these notifications from the telephone network just like how SMS messages are also being notified to our phone software? That tomorrow will be possible with tinymail’s infrastructure. Or maybe will some network start supporting some way of communicating this information over GPRS without needing an active TCP/IP connection? Maybe we need a small kernel module to communicate this to the application layer? Using D-BUS maybe? Multicast? inotify? Four such techniques at the same time? Pickable by the user as a plugin? A closed-source one from PhoneCodersDotCom? Also that will tinymail’s design for this cope with.

So .. this: gimpnet.#tinymail.add_observer (you), where ‘you’ is an instance of a type that implements an update method. And a ‘foreach observer o: o.update()’ in the channel type whenever something happens (Don’t call me, I call you). And not this: while (true) you.pollforstuff (gimpnet.#tinymail). It’s sooooo simple, but you would be surprised to see how many projects and cowboy-programmers get it totally wrong over and over again.

After all these improvements to the IMAP code of tinymail I have put in place all the infrastructure to support incrementally loading the summary view (the thing that shows the headers of your E-mails) while you are downloading it in the background.

Polymer and Telomer do this too by the way. Which is why it looks so fast. On top of that it uses pipelining. This makes it possible to queue multiple commands on the server and then letting it start replying the results for them.

So basically, while using a single connection, Dave managed to allow the user to start downloading and viewing your summary while that user can already start viewing messages too. Using lots of clever techniques, tricks and most importantly a lot knowledge about how IMAP servers work. I would, by the way, hereby like to thank Dave for his help on IMAP while I have been improving things for the last weeks.

The idea is indeed to get tinymail there too. As I said, step by step. I already saw my own hack working in the demo ui of tinymail. It’s however done a little bit too gross to commit this hack somewhere.

But indeed, you click on a folder and the treeview starts getting filled up with headers while downloading is taking place. That, in the end, is what you spoiled users want. Right?

Well, on tinymail-based E-mail clients you will usually see this “fancy effect” only once. After that first time your folder’s summary will have a local state that doesn’t need a lot such changes to get synchronized.

Unless you added really a lot stuff using another IMAP client. And of course while the folder is active and you have a connection you will see the Push E-mail feature in action: new messages will be automatically added to the view too. This should give you the same fancy feeling! Whoohoo.

So that is what you can expect in a near future. This type of features might, however, be for a 2.0 release of the API. I don’t know yet. I’m a.t.m. too focused on coding to worry about releases.

A demo of the hack in action? No sorry, it’s late here in Belgium. Maybe tomorrow. It’s reusing a lot of the code of the Push-Email / IMAP IDLE feature. I posted a demo that shows that a few days ago.

My first todo item that I wanted to finish last night is now finished: I finished the support for both unsolicited EXPUNGE and unsolicited FETCH events. Together with the unsolicited EXISTS this forms the support for Push E-Mail or the so called IMAP IDLE.

There is still some work to also fully and correctly act on the changes in the higher tinymail layers. Most is acted upon already though. For example added and removed messages are pushed to the models of the user interface if you register the model with the folder observer TnyFolderMonitor.

Next on the list is finally going back to non-camel-but-tinymail-code coding. Although I think I must by now know more or less everything about Camel that there is to know, both its beauties and uglies, tinymail code is still far more easy for me. Probably because I designed it myself and because it’s designed exactly how I want it.

Dirk-Jan and Sergio have been finding various nasty pieces in both the API and implementation that I didn’t foresee when testing using the demo user interface. This is good and was to be expected. No human, not even a freak like me, can get things right from the first time. It’s not in the design of our species. The result of their pioneering will be a better library. One that just works. As usual, I’m only satisfied when it’ll be excellent. In the end, what’s the point in making it easy for yourself?

My opinion? Life is about the path to your results, not about the result themselves. Getting there is the joyful part … But maybe I just need a shrink? Or more time with Tinne, in sauna, in bed sleeping, on my skateboard and less time behind my computer screen?

Yes, maybe. Fucking coding addiction.

Although I still need to get acting on expunges 100% right, I just completed Push-Email and I already have a bunch of new TODO items for that little framework which I now started over a year ago.

Dirk-Jan asked me whether it would be possible for the folder model to get auto updated upon changes. Changes like when the “unread” and the “read” changed. But also changes like folder renames, creations and folder deletions. The idea will most likely be to let the folder model implement the TnyFolderObserver type, to register it as an observer of every folder that it stores in itself and to add some extra information to the TnyFolderChange delta or helper object. As usual, people can help me with this.

Another item that I want to get done is to read more of the unsolicited events from the IMAP server. At this moment I’m indeed reading unsolicited EXISTS and EXPUNGE responses. But there is also the unsolicited FETCH response that aids with getting the flags updated when they changed remotely. In combination with the already implemented CONDSTORE support, will this reduce a lot of the needed bandwidth. Not only will new messages be pushed and expunged ones deleted, but also will changes to the flags of your messages be auto updated (without a network-traffic expensive full synchronization needed).

Anyway, I have no idea how to get informed on remote folder renames, remote folder creates and remote folder deletions. I wonder whether the Lemonade group can do something about this? It seems that nothing in the IMAP specification makes this possible? Periodically doing a LIST in combination with STATUS doesn’t really sound practical to me.

At FOSDEM my plan is to give an overview of both future plans and the current state of the project. Last year at FOSDEM and at GUADEC I did a technical talk on design patterns and GObject. I think this one will be less technical in its nature. A little bit like my talk at T-DOSE. I mean, I only have 45 minutes or so. There’s no way you can do a technical talk in 45 minutes unless you focus on one micro feature. At T-DOSE I was lucky that my audience was still interested (which is very strange, I agree) and that it was lunch time. I wasn’t very prepared at T-DOSE though. I mean, I had been coding like crazy those weeks. I fear FOSDEM wont be much different.

Somebody on an IRC channel told that it’s better to burn out, than to fade away. I think it was our bugmaster Andre. He’s right. And thanks for those few non-coding moments on IRC Andre.

I have been doing some testing with the recent support for IMAP IDLE (RFC 2177) or Push E-mail using IMAP.

It might surprise people but IDLE is currently supported by most IMAP servers. I tested with Cyrus, Exchange, Isode M-Box, Courier and Dovecot. The only one that didn’t worked with the current solution in tinymail was Dovecot. I did see the right EXISTS coming through, it seems that the UID FETCH that follows gets wrong information. I really haven’t in depth looked at this though. So I’m not yet blaming Dovecot for this. It might very well be a discrepancy in how I’m using IDLE.

ps. I’m tracking the testing of IMAP IDLE & Push E-mail here.

Although it will need an enormous amount of testing, I just committed support for IMAP IDLE to the tinymail framework.

The by-the-server pushed summary information will be passed to the TnyFolderObservers of a folder. If you configured a TnyFolderMonitor to update your TnyList of TnyHeaders, which is usually going to be a TnyGtkHeaderListModel, then your view will also be automatically updated.

I created a screencast of the demo user interface where you can see this taking place.

Youtube version of the screencast.

You can use imap2.tinymail.org for testing. The server is also running Cyrus’s idled tool to speed up its IDLE notifications. Note that for now, only the Inbox folder has condstore-features enabled (sometimes, depends on my mood). You can remove and play with messages as much as you like. Under normal circumstances tinymail should update itself fully automatically not long after somebody does something (that’s what IDLE is all about: Push E-mail).

I would also like to note that fejj has restarted working on libspruce. Which is amazingly great news. I will most likely join sooner or later (as a warm-up I already posted some untested condstore code for libspruce, which I ported from camel-lite). And you can bet that once libspruce is usable, that tinymail’s camel-lite will be replaced with it. There’s still a lot of work for libspruce though.

Please note that this IMAP IDLE support is extremely experimental and will most likely deadlock and crash a few times. But then again, that’s just bugs. I mean, they’ll get fixed.

Important to know is that the observer / observable pattern being used by tinymail, can be implemented by any observer and by any observable in a flexible way. If tomorrow somebody implements a observable using SyncML (like me, as I’m nonetheless planning to also support it), then the tinymail API and design will cope with that. By the way, this is the design-idea page of all this.

It looks like GPE-Phone is using tinymail for their little E-mail application. I truly didn’t know that until this evening.

Ok, well. While I recommend waiting for Modest to read the code of an E-mail client that uses tinymail, or simply taking a look at the demo user interface, I guess this is good news.

A little clarification about evolution-data-server and tinymail.

About

Tinymail will not conflict, in any way, with evolution-data-server. The Camel in evolution-data-server is not, in any important technical way, “integrated” with the other components of evolution-data-server. If somebody once did integration with the Camel API part of evolution-data-server, exactly the same will still work with the embedded camel-lite of tinymail. That’s because the API has not changed (it’s true that some API has been added).

Tinymail also (by far) isn’t copying everything of evolution-data-server. It only contains the Camel part of it. And actually only those pieces of Camel that tinymail supports and uses (some providers have been disabled). It’s a size reduced one (and the size reductions are real). This means that tinymail users who don’t need other evolution-data-server components (and these users exist), are only using that exact specific part that is really used by tinymail.

One might think: then why isn’t Camel a separate library? From a technical point of view that is a very reasonable, sane and good question that I asked myself, too.

The camel-lite of tinymail has immensely changed. These changes are part of what tinymail is. Tinymail also depends on these changes. Not only did I change memory consumption, I also rewrote big parts the POP3 provider to support summaries and to make it possible to use it offline. I rewrote pieces of the IMAP code and I have recently added support for the condstore and binary capability. People are working on getting that upstream, indeed.

Getting it upstream

The speed at which these changes can go upstream is, however, not compatible with the speed at which the tinymail project is being developed. Although I’m trying to get as much upstream as possible, I also understand that some of the changes are very specific for mobile devices. It’s exactly those changes on which tinymail, naturally, depends the most.

API

The good part is that since only API was added, the only change that people who integrated with evolution-data-server using the Camel API have to make, is this one in their configure.ac:

- PKG_CHECK_MODULES(CML, camel camel-provider)
+ PKG_CHECK_MODULES(CML, camel-lite camel-lite-provider)

For mobile developers it’s also very easy to disable the camel part of evolution-data-server’s build environment. The building of Camel actually used to be disabled in the eds-dbus fork of evolution-data-server because not a single mobile application was using it nor was any of the other evolution-data-server components using it (and still isn’t as far as I know). The only thing you have to do is remove the word “camel” from the toplevel Makefile.am’s SUBDIRS variable. Although then you would have to make a small change to the tinymail build environment to compile tinymail, you could also copy the camel directory of tinymail’s camel-lite to that location.

A few weeks ago I wrote this official explanation on the subject.

I must be a little bit crazy and anti-paranoid about security for advertising this, but I also think that testing infrastructure is very important.

That’s why I have begun setting up different test IMAP servers for tinymail development. The domains might still have to be distributed to your providers DNS servers, I still have to make the mailbox for the user, etc etc.

I already created a little web interface that allows you start and stop the IMAP servers. Of course if I see too many planet users trying that out at the same time, I will disable it for a day or two. Just assume that it works :-).

The unit test environment of tinymail will probably get some scripts that will turn off and on certain IMAP servers for in case they will be testing capabilities that are specific for a specific IMAP server. They will indeed test online and reset the mailbox to its original state (which reminds me that I still have to add that to the web script).

Other projects are allowed to use the IMAP servers too (but do read the abuse item on the wiki page). Though there’s no promise that the services will always be available at all times and in its current shape.

I configured three virtual machines for this. Two of them are already running IMAP server software: one is running Dovecot and one Cyrus (the webscript allows for multiple installs to be stopped and started). Both the very latest and most unstable version that I could find. Yes I’m anti-paranoid about security and I do understand the risk that I’m taking here (and by blogging about it, I just increased that risk).

I welcome help from the server developers. If they want root on the machine and perform upgrades to the install of their softwares, then we can definitely discuss that. I’m also going to allow commercial IMAP softwares. But once many such servers have to run on the hardware simultaneously, I will not sponsor the hardware myself any longer. I guess that the commercial ones will have to get me hardware then.

So you can check more about it out here. I will be fine tuning it this evening and I might already change some unit tests to start using them too.

With the focus on Lemonade IMAP features and (therefore) wanting to optimize bandwidth utilization, I added support for the BINARY IMAP capability to tinymail’s IMAP code. The binary retrieval reduces retrieval size of certain messages (encoded is typically larger than the original message). Together with the existing support for CONDSTORE, STARTTLS with OpenSSL or NSS (which should compress too) and the optimizations for when you are retrieving the summary, this should mean that tinymail is indeed getting better and better at making sure that it’s not wasting your expensive GPRS connection.

I’m planning to write or steal somebodies ENVELOPE response parser so that I can use that in stead of the optimized summary retrieval code. This ain’t going to make a huge difference (just a few bytes per summary item). Those few bytes do add when retrieving really large folders, so it’s still worth the development time (… indeed, and I agree with you. Don’t worry).

Other plans include again rewriting the summary code (that’s the mmap stuff). This time I want to store segments of the summary data in mmap()ed files. For example 1000 items per such file. Then read-only mmap all of the frozen ones. Then when the next 1000th summary item is added, store the last 1000th in a file and mmap it too. Adding it to the list of segments.

The reason for this is that with the Lemonade IDLE code, adding summary items is going to happen more sporadically. Right now it’s very predictable when lots of summary items are going to be added to the summary: when you are retrieving or synchronizing your summary of course. With Lemonade’s IDLE there is less control over it. It’s the IMAP server that will “Push” this to the client.

I want to get it that stable and good that while this is happening, I want the view to start updating itself. This is what the TnyFolderObserver and TnyFolderMonitor are going to be responsible for: the IDLE event will trigger the invocation of the update method on all registered observers of the folder instance. The monitor will be such an observer that will deliver it to the list models of your views (those are TnyList implementations like TnyGtkHeaderListModel which is one that also implements GtkTreeModel).

Another reason is that that way I will also put the changeable data, which is only the flags, in a separate file. This will reduce level wearing on file systems that run on flash devices whenever changes to the summary happen. That’s because all the flags together will easily fit in a few blocks, and it’s not interesting to keep integers in the mmap. The pointer to that integer would consume as much memory as the integer itself. Leaving it out of the mmap()s will reduce the filesize of the segments, and the complexity to parse through it.

A last reason is that I can make looking up a summary item using the message’s uid a lot faster this way: I would store the summary items sorted on uid in those segments. So a little bit of simple math to find out in which segment it will belong (which would serve as a first index) and then utilizing a search that is optimized for sorted data (preferable one that doesn’t make big steps while comparing, as that would thrash possible caches). This is also going to help me a lot with mimicking a VFolder feature in tinymail, that works offline too (not just SEARCH commands on IMAP servers), that is — which is important — fast enough (and will also push found items to the views using the TnyFolderObserver stuff, while searching the read-only mmap()s in the background).

Equally important about such a search feature is that tinymail can NOT, no absolutely NOT, consume as much memory as Evolution does by keeping all summary of all folders in memory at all times.

There are a few other todo items too. Which are more bound to tinymail itself and less to tinymail’s camel-lite code. But those are all documented here already.

I hope the plans sound good :-). But I’m absolutely interested in your technical opinions. The future is going to be challenging, but definitely not undoable. The undoable part (starting the project and not giving up on it), is already done. Right?

For the Maemo ui, I’ll let Dirk-Jan surprise you.

Recently I was talking with Dave about using ENVELOPE in stead of what Evolution’s Camel does for fetching the summary information. The camel-lite of tinymail, however, already reduced the query a lot by cutting away everything that was not being used.

Months ago I cut away headers like REFERENCES, IN-REPLY-TO, MIME-VERSION, CONTENT-TYPE and in some situations also X-MAILING-LIST, X-LOOP, LIST-ID, LIST-POST, MAILING-LIST, ORIGINATOR, X-LIST, SENDER, RETURN-PATH, X-BEENTHERE. Though that last list was conditional they do get received by most people’s Evolution. Only if you have put “basic_headers” in the “url string” of your IMAP account, this last list of headers isn’t used in Evolution.

In contrast with that I figured early on that an E-mail client for a mobile-device does not need all the information, per message, about the mailing list. The summary view was not showing the reply-to field and tinymail doesn’t yet support showing the headers in a thread view. Which is what the references header is used for.

I’m mentioning this because I want to get rid of the reputation that Evolution has when it comes to bandwidth utilization. Tinymail is using an extensively modified Camel for a reason, indeed.

This was before I learned about Lemonade. You must know that back when I started with tinymail, I was not very experienced in all this E-mail & IMAP stuff. I’m still not, nor is it easy to know everything about it.

I got an account on one of the Isode servers for testing. I used their server for testing this, because that server (I assume) is a very efficient one when it comes to interpreting the IMAP specification versus bandwidth utilization (they are one of the companies that are pushing Lemonade, hence the assumption).

The most efficient way to ask for the summary is probably to use something like this (Don’t worry, people like Dave are going to correct me if it’s wrong):

A01 UID FETCH 1:2 (UID FLAGS RFC822.SIZE ENVELOPE)

ENVELOPE is often cached information, so it’s fast. In bandwidth, which is what matters most for tinymail because IMAP “servers” usually have enough power when comparing it with a mobile device, this consumed something like 800 bytes depending on the headers themselves. I’ll use the actual numbers of the account that I got from Isode.

This, however, is what tinymail is using today. You don’t want to see the query that Evolution is using :-). In a few words, it’s something like three times larger in what it asks for. You can probably compare it with a “TOP uid 0” on a POP3 server (most headers). It’s not about bashing Evolution. I’m just comparing facts here.

A01 UID FETCH 1:2 (FLAGS RFC822.SIZE INTERNALDATE BODY.PEEK[HEADER.FIELDS (DATE FROM TO CC SUBJECT MESSAGE-ID)])

This consumed 825 bytes. Don’t forget that back when I was optimizing this, I wasn’t into writing a parser for ENVELOPE nor did I actually know about that IMAP feature. Remember: I was not very experienced with IMAP at all. My focus was on getting tinymail to do something interesting in terms of memory consumption while wrapping that in the nicest API that I can imagine. You know, I’m just one person with only two hands.

Luckily only a few of the things that are needed do consume a lot bandwidth. Important are receiving the summary, updating the flags and knowing about expunges and other such changes. In other words: synchronizing the local offline status with the online status.

And that is precisely what Lemonade is improving in the specification. Condstore, for example, returns me a sequence number that uniquely identifies the current status of the mailbox. I can create a very simple query that says: well, give me all changes that have happened since a state. Important here is that I only receive the changes. I don’t have to get all of it and start comparing things.

The difference of 25 bytes sure is significant enough to want to improve this. This will eventually happen. I also have to check whether using ENVELOPE on all IMAP servers works correctly of course. With worst-case servers around (experts know which server I mean), this isn’t self evident. This indeed, is what makes developing an IMAP client difficult. My opinion, and I like to express that as people already know, is that it’s better not to create an IMAP server than to create one that sucks. The problem is that users will depend on your defects. And that E-mail client developers need to cope with it.

So, why did I write this lengthy blog item?

Because I wanted to tell people not to get the focus of the tinymail framework wrong. Memory consumption is only one aspect of the project. Bandwidth utilization is actually even more important. Other aspects are design, testing, documentation, language bindings, flexibility and being adaptive to change.

By the way, didn’t I whine a lot about being adaptive to change a few months ago? I might have irritated some people, right? Guess what Lemonade brings us?

Change

This proofs my point. Once I implement support for the IDLE Lemonade feature, I already have a TnyFolderObserver type ready for this.

E-mail nor any other technology is in a stable state. E-mail has never been in that state though it’s one of the oldest Internet standards. Change is among us.

Be adaptive.

Support for the Lemonade IMAP condstore capability (RFC 4551) has been added to tinymail. This is the first support for a Lemonade feature. I added a point of view page on the subject to the tinymail documentation. It outlines the strategy that tinymail developers (as I’m no longer the only one) will take to one day let tinymail become a MUA framework that will outperform existing commercial solutions like the Blackberry E-mail software. That’s, indeed, what I’m aiming at.

From a financial point of view I’m taking things more seriously now. I now understand that there might be commercial interest in the framework. I also understand the simple fact that I could die in a car accident tomorrow, is a major risk for companies interested in adopting tinymail. For this risk reason, I’m building up a network of TnyPartners who can help customers with tinymail consultancy. I’m being at FOSDEM next month, so catch me there if this sounds interesting to you.

Rather than see this as competition, I’m planning to actively help both customers and those partners finding each other. If I would make this a competition-battle between me and them, the risk reason would turn it against the project. I’m aware of that.

My biggest concern is keeping the project a healthy free software one. Ever heard of Peter Principle? I have and I’m aware of my own limits. I’m not (yet) much of a project-leader (but hey, I have started reading books on emotional intelligence. Soon maybe). I think most people by now understand that I’m the engineer type of guy, who passionately and unstoppable enjoys building software.

This is one of the reasons why I haven’t yet done any releases: it would put an extra burden on me and it would awake the crazy “I hate and bash everything”-people. I’m not interested in that. Since tinymail is for software developers, I figured that those guys can checkout from the subversion repository. Some day, however, I will release something. Nevertheless, if somebody wants to be that project leader or release maintainer, call for it.