Introduction
Unlike what some strange people think wont E-mail disappear anytime soon. I also believe that more and more of the use cases of the desktop computer will migrate to specialized devices. Examples are music, navigation, simple messaging, movies, contact and meeting/agenda management. With devices like the Blackberry we are seeing a market for E-mail too.
I want to create the next generation of E-mail clients for mobile devices. Perhaps even do more with the technology than what people nowadays imagine E-mail can do.
CONVERT
IMAP will soon be one of the few content services that will make it possible to do conversions at the server. A mobile device often can’t ship with support for many document formats. Often are certain document formats inadequate and too large for the popular wireless data networks being used in mobile. Just to make a document easily editable can a format needlessly grow in size. Often is this capability not required on the mobile. Another problem is too much data quality. A device that has poor audio capabilities does not need a CD quality WAV file. A device that has a screen of only a few hundred pixels, does not need to download images that got attached to an E-mail as-is when they where taken from the digital camera. If just for the purpose of displaying on the device, they can be made a lot smaller in size.
Selective summary downloading
Not only CONVERT but also techniques that have been possible for ages, can be applied with IMAP. Mulberry, Pine and especially Polymer have done interesting experiments with existing IMAP capabilities like pipelining, envelope downloading in a specific way and downloading of individual MIME parts of a message as they are needed. Two of those three techniques are not surprisingly missing in my own project, Tinymail.
Tinymail will download the entire folder’s summary because it wants to serve as infrastructure to write E-mail clients that can work offline in a convenient way. This doesn’t mean that the envelope downloading technique being used by Pine, Mulberry and Polymer is not suitable for offline E-mail clients. I have, however, never found a good E-mail client that combines this technique with a fully offline available summary of your folder. I know Dave Cridland is interested in experimenting with Polymer to make it become an offline E-mail client too.
What is this technique about? In stead of doing a simple query that requests all summary items top to bottom, you start with requesting the items that are near the ones the user wants to view as soon as possible (the visible ones). After that you continue in the background with receiving less important items. This works surprisingly good for E-mail clients that require being online. You more or less integrate it with your summary viewing component’s scrollbar and you make the magic do its work. To avoid roundtrips you try to group items together in your queries (to end up having to send less queries). The better implementation will on top of that pipeline queries and responses.
For an offline E-mail client you need to store that, of course. Once stored you’ll also need to pick a strategy for deciding what the next most important group of items will be. It’s for example likely that you already have the group locally cached. Storing them is less trivial than it sounds. Especially if you want to use the items efficiently in your memory. For example by mmap()-ing the file that stores the cache, or by using one of those embedded database engines (which makes this a lot more simple, of course).
It’s likely that the user will be scrolling up and down. It’s equally likely that the user wont make huge leaps in his scrolling behaviour. Therefore you can make the assumption that you want to start growing the group of locally available items around the spot where the user is currently viewing things. Somehow, store it in that retrieval order.
Sorting, threading
Meanwhile you want to ask for a SORT and a THREAD which will assist your E-mail client with sorting and threading in a much faster way than it could do this by itself (remember that we are on a mobile with limited memory bandwidth and CPU capacity to reduce battery consumption). You also want to request all the possible sorts that the user can request, and store the results offline.
Selective MIME part downloading
It’s quite common that the user is not interested in the entire message. A good example is a message with five image attachments. The user will often pick just one of the images. With IMAP’s BODYSTRUCTURE and FETCH program you can selectively download the exact MIME part that the user is requesting, in stead of the entire message. For that reason you want to fetch BODYSTRUCTURE together with ENVELOPE when dealing with summary retrievals. That way your summary will also serve as the skeletons for your messages: you can prepare your user interface components before even having to download the actual content of the messages. Also important is that you can do this without having to ‘store’ the content of the messages: storage capacity is rather limited on a mobile compared to the typical E-mail account of a lot of people. Not having to store and not having to download the attachments, is kinda nice. Don’t you think?
The problem, current state of affairs
There’s not really a problem. A lot of the ideas that I had in mind last year are now implemented in Tinymail. Examples are support for CONDSTORE, QRESYNC, IDLE, Selective MIME downloading (if you enable it) and BINARY (if you enabled selective MIME downloading).
One problem is funding. The ideas that I have in mind imply developing on these ideas full time for at least another two years. Another problem is the experience level in E-mail client development: it’s no secret that all competent E-mail client developers seem to have the tendency of running away from E-mail client development. Often they join forces with IMAP server developers (hi there Dave), or they no longer like E-mail client development (hi Jeffrey!). Leaving the scene with inexperienced developers, like me. It seems those people get more experienced and end up doing a new project, they burn out or they get hired by server developers. It looks like they don’t often focus their experience on a exciting new kind of E-mail client, based on their in-depth experience with IMAP.
I finally kinda understand IMAP, I think (but I’m not sure). I of course think that if I’d start an E-mail client infrastructure now, it would be excellent stuff. But I also have to survive, so I found myself a contract that is mostly unrelated to E-mail client development. Starting April will Tinymail be my non-professional pet project. Which means that the speed of development will probably drastically go down.
Being mostly a technical guy, I don’t know how to market these ideas in such a way that somebody wants to pay me for a few years to make it happen. Unlike what most people want to believe, would it indeed take a few years to do all this right.