Day: January 23, 2007

I’ll guide people to how they can help Evolution (and therefore also tinymail) get support for at least certain interesting Lemonade IMAP protocol features. You can all start looking at me to do it for you, but as my todo list just keeps growing and growing, you’ll be looking for a longer time than that it would take you implementing it yourself.

I’ll explain the CONDSTORE capability. CONDSTORE is interesting because the thing that does change about summary of messages are the flags. For example the \Recent and the \Seen flag of your E-mails, change. The CONDSTORE capability makes it possible to quickly synchronize these flags with what you have been storing locally (without having to retrieve all of them one by one again).

In Camel’s camel-imap-folder.c there’s a not-very-difficult implementation for this, called imap_rescan. And in Camel’s camel-imap-store.c you’ll find a static global table called “capabilities”. Simply adding the “CONDSTORE” capability there and putting an if (store->capabilities & IMAP_CAPABILITY_CONDSTORE) { /* your new code */ } else { /* original camel code */ } at that location, would probably be sufficient to actually implement it.

imap_rescan (CamelFolder *folder, int exists, CamelException *ex)

The existing implementation uses a trick to avoid calling the hashtable lookup that I mentioned yesterday. But for a few lookups (only the changed ones since last sync) it’s probably better to simply utilize camel_folder_summary_uid to get the CamelMessageInfo instance for a specific uid. Now use camel_message_info_set_flags on that instance and feed it the new flags which you just received from the server. You even already have the helper functions that turn strings like “\Seen” and “\Recent” into the correct bitfield flags.

If you need to add “(CONDSTORE)” to the “SELECT”-IMAP command: The “SELECT” command is implemented in camel_imap_command (a quick search in the code for “SELECT %F” will do wonders). Just make sure that you only add it in case you have IMAP_CAPABILITY_CONDSTORE in the store->capabilities flags.

So why am I whining about this? Because well, honestly .. not much people are working on this type of things. Yet this is something that both our desktop and mobile devices desperately need.

I decided to link to Camel’s SVN this time because I’m pretty certain that the Evolution Mailer maintainer would accept a good patch that adds support for the CONDSTORE capability. If you are more into experimenting with things that are rather unlikely to go in Evolution’s Camel soon, you can of course join me on experimenting with it in camel-lite. This isn’t a pure isolated fork. I’m trying to extract useful deltas out of those experiments and pushing them upstream. Which takes time on a per-patch basis, of course.

I’m willing to dedicate my entire FOSDEM-time to co-developing Lemonade IMAP support with interested people. I mean, we have developer rooms over there, right? Why not? We can even continue at night if you are not afraid of spending the night coding. Somebody?

In the end, it’s really about “just doing it”.

Nokia N800

Thanks to Nokia and more specifically Dirk-Jan C. Binnema for getting me a discount for the N800. I will of course make sure tinymail runs on it. Not that I expect a lot changes for that.

FWD: Misinformation

Jono makes this interesting argument about misinformation. I recently started reading this book about emotional intelligence written by Daniel Goleman. This is is, In my opinion, a good followup subject on Jono’s point.

I read an older book of Daniel Goleman on emotional intelligence before, but this one is more practical (I think I have the Dutch version of Working with Emotional Intelligence).

Psychology is interesting, indeed.

Telomer & Polymer

I’ll immediately respond on Telomer before I get flooded with questions on why doing tinymail if it’s all that simple:

Telomer is interesting of course, but it does not cache E-mail locally. This means that you must always be online to read E-mail. Getting the summary information and getting messages (actually, getting MIME parts) is not the hardest part of an E-mail client. The hardest part is parsing the MIME parts and dealing with the offline cache in an efficient way. Which is what tinymail is really about. Actually, from a technical pov it’s what camel(-lite) is about.

Tinymail of course also provides code for connectivity with IMAP servers (also for POP3 and NNTP) otherwise it can not get this information in that cache.

Another already working solution when your mobile device is always online, is also called webmail like Yahoo mail, GMail, Hotmail, Squirrel, Roundcube, IMP, etc. The problem is, however, that mobile devices are not always online.

You could argue that making a simple quick ‘n dirty cache is quickly implementable. The implementation of this E-mail client or library will, however, get more complex once synchronizing changes to that offline cache must somehow be reflected to the IMAP server once there is connectivity. And once you want to do this efficiently. Etc. But I’m interested, as it’s on the TODO list for Telomer & Polymer (I just promise that it wont be easy, hehe).

The Lemonade stuff, however, is very interesting. For example the P-IMAP (Push-IMAP concept) for when new messages arrive. You can be certain that I’m already looking into ways to use these IMAP protocol features on IMAP servers that support it (emphasis on that).

It’s very nice and extremely interesting to see Polymer and Telomer use and push the Lemonade IMAP protocol features. In time tinymail will obviously support all of it.

Anybody who wants to work on reimplementing the IMAP provider of camel(-lite) or on improving the existing one by for example adding support for the Lemonade protocol features is absolutely invited.

It’s time for some monthly bashing-others about memory wasting. Moehaha! But hey, Camel was not done that bad. This is a friendly one. Some people in the past had this strange idea that I was questioning the competence of those who’ve built Camel. That’s absolutely not true. There’s a reason why I use it myself, don’t forget that (and some day, people will bash the fuck out of tinymail, right? Good! keeps us going).

Some developers enjoy the fact that glib makes their lives more easy a lot, it seems. And I agree with them! You should leverage the fact that glib is a well tested library. Who wants to implement his own hashtable or doubly-linked list each project he starts? I honestly don’t.

However. In the CamelFolderSummary, the number one piece of code that consumes most memory of E-mail clients that are being build on top of Camel (that includes Evolution and tinymail with its camel-lite), we surprisingly see a GHashTable being used in parallel with a GPtrArray! Both holds the exact same instances?!

Somebody like me … questions that. Because both in memory and in performance this (in this case) is a loose-loose situation: adding it to the hashtable takes time, searching a GPtrArray is (in this case) not going to take a lot longer (read below for more information on the “in this case” situation).

Consider the memory caused by each item you add to the GHashTable:

struct _GHashNode {
  gpointer   key;
  gpointer   value;
  GHashNode *next;
  guint      key_hash;
};

That’s on a typical x86 architecture 4 + 4 + 4 + 4 bytes, right? 16 bytes per item (Or am I calculating this wrong? There’s no waste caused by mem alignment here I think, and it uses g_slice_new so there’s also not a lot heap admin). Multiply that with 10,000 headers, that’s 152KB of memory. Nothing you say? Okay, I agree (well, except that for a mobile application this is a significant memory improvement).

However, consider that each and every summary item that you can see in your Evolution consumes this. With the help of some mailing list subscriptions, I use Evolution to manage up to 1,000,000 messages that way: 15,625KB or 15MB of GHashNode instances (gah, I must be miscalculating because that’s much more than what I expected).

Because this:

static CamelMessageInfo*
find_message_info_with_uid (CamelFolderSummary *s, const char *uid)
{
	CamelMessageInfo *retval = NULL;
	guint i = 0, len = strlen (uid);
	for (i=0; i < s->messages->len; i++) {
		CamelMessageInfo *info = s->messages->pdata[i];
		if (info && !strncmp (info->uid, uid, len)) {
			retval = info;
			break;
		}
	}
	return retval;
}

Is more difficult than this:

g_hash_table_lookup(s->messages_uid, uid)

I don’t whine without a patch, right? right!

Update: and on the tinymail front, I removed the need for often calling this function (update: which is an important pre-condition for applying patches like this). This made loading large folders a lot faster when using the framework for this. You can svn diff -r 1451:1452 to check what I changed for that.

Update on CPU consumption (because some people have questions about that): The hashtable lookup would indeed be faster, but the hashtable lookup can (and usually is) avoided in Camel. Not sure whether it’s also avoided (and avoidable) in Evolution code (that’s a lot code to check), but all occurrences where the hashtable lookup is needed are avoidable and are therefore avoided by tinymail. In other words: looking up by uid is not often needed. But you are right that a hashtable lookup is faster than that find_message_info_with_uid implementation.