Month: November 2007

An IMAP server is smarter than a POP server. This, I hope, comes as no surprise to most people. An IMAP server has a MIME parser and can give you the BODYSTRUCTURE of a message. This is useful because that means that you can individually request the parts of a message, rather than always fetching the entire message. In future you can also CONVERT those to for example thumbnail versions. (I already started implementing support for CONVERT in Tinymail)

For a mobile user this makes a difference. Suddenly he wont have to pay for penis enlargement advertising images while working roaming over a GPRS connection at five euros per megabyte. This is an actual price in Belgium when you are five kilometers across the border in the Netherlands (I live 10 kilometers from the border, I can walk the 15 kilometers to have to deal with that price).

With today’s currency fluctuation I’ll describe what five Euros are: five Euros are two cans of CocaCola to download putty.exe and one can of CocaCola to receive two spam messages about how small your penis is with Evolution (I guess some people’s penis will actually spontaneously shrink if they’d know how much they payed for receiving the spam). I guess I don’t have to explain you what this means for Google Maps? I think that buying a Garmin or TomTom device is going to be less expensive than one month of using Google Maps with a phone. Usually it’s because you are in a foreign country that you need a service like Google Maps, right?

This realism of today should give desktop users an idea why bandwidth consumption is important. Another reason is of course latency. I’m sure UMTS and EDGE fans will explain me that soon, oh dear soon, all such problems will be gone. Bad luck as both UMTP and EDGE still have a very high latency (although slightly better than GPRS).

That what makes the user think that a service is slow, is what technical people call latency. Pure bandwidth performance matters for non-interactive protocols. E-mail is a highly interactive protocol by nature. A E-mail client user clicks and views a lot of E-mails and folders and wants the application to respond reasonably fast nonetheless. He doesn’t want to either wait for nor pay a lot of money while synchronizing E-mail data.

Synchronizing data is not how most people use their phones. Most people press the power button and want it to just fucking work. Mobile can only be a success if things like this just work. Users don’t want to be bothered with setting up an E-mail data synchronization software on their laptop, drag that laptop with them, and from time to times synchronize data to their phone.

The reality that Western people usually can’t grasp is that most mobile phone users in the world don’t even have one computer.

With LEMONADE, IMAP does pretty well at avoiding round trips. Fewer round trips and using IMAP’s pipeling capabilities means less latency. Less latency means faster responding E-mail clients. Given that the HTTP protocol is not really designed to avoid round trips, a web client in a web browser wont ever perform well with mobile networks. On top of that have a lot of web client services the annoying habit of adding bandwidth-expensive advertisements on the page itself. On top of that don’t all people like the privacy aspect. Especially businesses don’t like this (although they can of course buy a web service appliance for E-mail too). On top of that do some people want to fetch messages locally, and read them while being offline.

With BODYSTRUCTURE we can selectively fetch the parts of an E-mail as they are needed, rather than always.

Use the Tinymail API to build your E-mail client, and it’ll do both automatically for you.

After that much text, a video demo is a requirement. Right? Right! You’ll see that each time I expand the expander, the image gets downloaded. You can see this because Nautilus’s directory monitor will add the file to its directory viewer.

I guess sometimes I do need a lot of words :-)

ps. Build instructions for everything, here. No I don’t know whether that will work with your 770. Yes it’ll work with your N800 and your N810.

I just added the bodystructure parser that I mentioned a few hours ago to a Subversion repository. I also fixed a bunch of parsing mistakes and figured out a few of the unknown fields.

Edit

I just added error checking. The parser now supports recursive bodystructures for multipart/*, application/*, text/*, message/rfc822, audio/* and image/*. It will parse as much fields as possible (I think I’m only missing two or three fields right now, they seem the be always NIL anyway). Fields include the parameters, content disposition, octets, lines, encoding, description, language, …

By requesting the BODYSTRUCTURE you can get a preview of the structure of a message without having to download it. Most E-mail clients parse this structure to the point of what they need from it. I did some searching and as I expected did few of the opensource E-mail clients do a complete parse. Some just scanned for some words, others did a reasonably well job but skipped information that they are simply not interested in. And that’s fine, if you don’t need it.

However

Soon will CONVERT be unleashed by the IMAP server developers. CONVERT is very interesting for high latency networks, mobile devices with limited display capabilities and other situations where you want to convert something in the E-mail to something else. A perfect example is scaling down the dimensions of an image that is embedded in a HTML message.

At the Lemonade interop that we did this week I met Roger Grönberg. The company where Roger works has an interesting but quite different solution for this. To me it’s interesting because it shows how wanted such a feature is on mobiles like phones: at the SMTP server first they somehow tag each image, then they rewrite it in the E-mail by scaling it to the perfect size for your mobile (they store your mobile’s capabilities at their SMTP server), and then the SMTP server delivers it to their IMAP server. Your phone’s standard E-mail client will now get a nicely formatted E-mail. Because they tagged it they’ll revert this change when you forward the image to somebody else. So they find the original image back, and correct the forwarded E-mail with that original image.

This approach of course implies various difficulties and changing the content of the E-mail at the location of the IMAP server. Although the server can still recover from this change, of course. With Lemonade we are, however, interested in making this possible without needing such tricks.

I started my blog item with BODYSTRUCTURE, didn’t I? Let me get back to that. You can probably imagine that if you want the IMAP server to instruct converting a MIME part to something suitable for your situation, that you’ll need to have a lot of information about the original MIME part before you start converting. Right? In case of a bitmap you probably want to know about the dimensions of the original one, so that you can ask CONVERT to convert while maintaining the aspect ratio of the image.

Well. For that we need a full BODYSTRUCTURE parser. Especially the content’s parameters are needed. I know I should probably have used sexp for parsing the S-expression like structures and I know that I probably got a bunch of things wrong. I even know that this current version leaks memory. For now it’s the parsing that matters most. Next on my list is hardening it against Courier-like IMAP servers. With some versions of Courier, you never know what you’ll get.

I prepared the sample so that you can just run “./test_app.sh”. That script will fetch 10,000 BODYSTRUCTUREs from my test IMAP server and then it will start parsing them.

I hope other E-mail client authors will test this stuff and point me to bugs, send me patches. Perhaps use it in their E-mail clients as soon as it’s actually usable. If somebody wants to convert this away from glib, to no-strings-attached pure C with POSIX, or C with the Linux kernel’s library then that’s fine and cute, send me the patch.

Oh yes. I released Tinymail pre release 0.0.5 and TMut 1.1.0 yesterday evening. The reason is that a specific series of bugs got fixed shortly after Tinymail’s pre release 0.0.4. I adapted TMut to leverage those fixes. The fixes also include moving folders and making the TnyGtkFolderStoreTreeModel automatically update. I also fixed a few embarrassing reference count fixes on the accounts and folders. The instances that where leaked where not consuming a lot of memory, but it’s not nice to leak entire objects of course. Especially not if you claim to focus on mobiles, like I do. Refdbg helped me a lot with solving this. Thanks Josh!

TMut now also supports displaying which messages are marked as seen, unseen and deleted. Note that TMut does this at the event of a unsolicited response from the IMAP server. This means that with today’s IMAP servers you’ll get instant updates to your view.

I have started implementing consuming NOTIFY in Tinymail. NOTIFY isn’t very different from IDLE and it actually complements IDLE when you are in that mode. With NOTIFY you specify which events you are interested in. You also get certain events as unsolicited responses about folders that are not currently selected. A few samples of events that you can request are new mailbox creations, mailbox deletions, mailbox renames, subscription changes and status changes. Status changes happen when the total amount of messages change. Regretfully doesn’t NOTIFY at this moment specify telling you when the total amount of unseen messages changed (although an IMAP server can probably nonetheless implement this). But then again, next week we’re doing an IMAP Lemonade interop in Munich. NOTIFY is already on the agenda.

I’m also planning to rewrite the summary storage in such a way that I can selectively retrieve summary information. For example Polymer, Mulberry and Pine retrieve summary information in ‘waves’ (that’s a name for it that I just invented). My claim has always been that this is typical something for an E-mail client that works while you are online, and that Tinymail has a strong emphasis on working offline too. Yet it’s possible to focus on retrieving exactly what the user is looking at the first time he opens a mailbox. A lot of the technical users would be more happy if Tinymail based clients would do this too. Most normal users, who have average mailbox sizes (like 20,000 items or less), are not going to notice the difference that much: once Tinymail has this cached locally, it of course only needs to fetch the new messages’s summary data. It’s not that it’s usual that you have to ‘wave’ download those. It’s rather usual that the amount of new summary data is small enough to fit in one TCP frame anyway.

It’s a technical challenge to get this working, so I will make Tinymail do this too. The most difficult part is making my mmap() based summary store cope with this. The retrieving part itself is rather simple and implementable in five minutes. Mulberry is an example of how such an E-mail client can work both offline and yet do this kind of stuff.

I just released TMut 1.0.0. I recommend using TMut with what you’ll find in Tinymail‘s Subversion repository. It has been tested with Tinymail’s pre release 0.0.4 too.

In pre release 0.0.4 there’s one known bug about moving folders that has been fixed in trunk.

This version of TMut doesn’t come with an account wizard or account management screen. You’ll have to configure TMut using GConf if you want to use it on your GNOME desktop. TMut’s target devices are for example phones and other mobile devices.

gconftool-2 -s /apps/tinymail/cache_dir -t string .tmut
gconftool-2 -s /apps/tinymail/accounts/count -t int 1
gconftool-2 -s /apps/tinymail/accounts/0/name -t string "My GMail account"
gconftool-2 -s /apps/tinymail/accounts/0/proto -t string imap
gconftool-2 -s /apps/tinymail/accounts/0/type -t string store
gconftool-2 -s /apps/tinymail/accounts/0/user -t string user.name
gconftool-2 -s /apps/tinymail/accounts/0/hostname -t string imap.gmail.com
gconftool-2 -s /apps/tinymail/accounts/0/options -t list --list-type=string '[use_ssl=wrapped]'

Or just use this GConf dump. And change “user.name” to your user name on GMail (or change the hostname to your IMAP server etc etc). This dump comes with GMail’s SMTP server configured too (then you can also send E-mails).

gconftool-2 --load tmut.gmail.gconfdump

I decided to do another Tinymail pre release. I haven’t yet run out of numbers, so this is pre-release v0.0.4.

Now that TMut is more or less (basic) feature complete, I’m thinking about doing a TMut 1.0.0 too. I don’t know when Modest will be released but I do think it’s almost the time of the Tny E-mail clients. If not Modest then I’ll make a few other E-mail client flavours … like TMut.

There are (probably) still small problems here and there in Tinymail. Mostly the kind of problems that can only be found by putting the code in the hands of users. There have been two or three API changes so the Python bindings are not synchronized (Python developers who started using Tinymail might want to stick with pre-release 0.0.3 until this is fixed).

Oh. I was taking a look at QAbstractItemModel recently. I think it’s doable to make a TnyList that also implements QAbstractItemModel in C++. Having convenient models for views like GtkComboBox and GtkTreeView is more or less what most of libtinymailui-gtk is about. Doing the same thing for QTreeView and QComboBox sounds like a doable idea to me.

I also identified all uses of the GMainLoop in Tinymail. I can easily abstract that (although I know the people at Qt are interested in having GMainLoop integration too).

If you want me to make Tinymail as easy to use with Qt as with Gtk+, fund me and I will :-). Technically it’s perfectly possible and not really hard.

Yet another video demo of TMut. This time I’m consuming most of the operations. Like creating, moving, renaming and deleting folders, sending an E-mail to a GMail account. Then seeing that E-mail arrive on the GMail IMAP server, utilizing the support for IDLE, making a reply and a forward, using a local Maildir folder, moving messages. Stuff.

The video demo also shows the recent new TMutFolderView that reuses Modest’s GtkCellRenderers in the GtkTreeView. These cell renderer reduces space and shows the E-mail header line in the folder summary in two rows.

All this TMut code is still hot, fresh and new. Be careful not to burn your fingers.

While I was not searching for a new project, I have been throwing TMut together. After six quick sessions of working on it, I made TMut:

• Support multiple accounts simultaneously
• Open and display text/plain and text/html E-mail
• Open and display IMAP folders
• Open and display the INBOX of a POP account
• Delete IMAP and Maildir folders
• Create IMAP and Maildir folders
• Move IMAP and Maildir folders
• Copy IMAP, POP and Maildir folders
• Rename IMAP and Maildir folders
• Delete messages from IMAP folders
• Delete messages from POP folders
• Move messages from one folder to another
• Put the connection in IDLE on IMAP and receive notifications
• Create new messages
• Create a reply of a message
• Create a forward of a message, as attachment
• Have accurate progress information

Not unimportant but because Tinymail comes with this support I also made TMut:

• Do CONDSTORE on IMAP
• Do QRESYNC on IMAP
• Do BINARY on IMAP
• Do IDLE on IMAP
• Do various SSL and TLS encrypting
• Do various authentication methods
• …

I’m planning to fine tune all this a little bit, then do a GPEPhone port of it. This will include overriding some of the virtual methods that are right now creating a more or less standard user interface for a GNOME desktop.

I also have to make some sort of optional account editor. Optional because some device vendors have the account editing done by a web page and then uploaded to the device via an SMS (this is not uncommon on for example Nokia phones). I just remember I still have to make a little bit UI for adding and removing attachments from a new message too. When forwarding the forwared message is right now added as an attachment already.

Once all that is finished I of course have to prepare a video demo of running code. I guess the VMWare image will help me a lot with that. After that demo, I will most likely do a release of TMut. Perhaps at the same time release a first Tinymail version too? I’m not sure of that yet. Perhaps I will do a few more pre-releases in stead. It has to be perfect.

I think TMut is ready for contributors now. If you want to join and hack together your favourite E-mail client feature, just get on the Tinymail mailing list and post me your great patches. I also have to document the code of TMut a bit more. And make TMut’s wiki page a bit better too.

The joy of starting yet another cute little project starts …

As in the world of mobile and embedded aggressive changes in technology and devices’ capabilities occur at a high frequency, I decided to make Tinymail‘s API very flexible or adaptive.

Yet I wanted to provide a certain functionality, which means a message view component that also shows the typical message headers.

What if the developer wants to wrap the message view in a scrolled window, have a message viewer as a new window or put the thing that displays the header in an expander? On a mobile device with a limited screen height it is a waste of space to show four or five lines of information. If the user wants this info, he can decide to expand it.

Imagine the same software vendor want to adjust the E-mail client for another device? However, nowadays software development must be fast. The change is something that should not destabilize everything that was already working. It must be easy to develop a new version of the message view component that perhaps behaves in a new way. Who knows what the screen will be capable of?

This diagram shows the flexibility of Tinymail‘s types. I’m using the decorator pattern here. Decorator is also used by the streams API of Java, .NET and GVFS. I explained Tinne about decorator by simply referring to a painting and its borders. When decorating a painting with borders, you can still look at the painting. Nothing about its functionality really changes, but you add things to it, you decorate it.

In stead of subclassing TnyGtkMsgView, I decided to decorate it with a new implementation of the TnyMsgView interface. This one wraps the TnyGtkMsgView instance (it makes a composition with it). I did the same thing for TnyGtkHeaderView that implements the TnyHeaderView interface.

Because TnyGtkMsgView‘s method for creating its TnyHeaderView is abstract, I inherited it as TMutTnyGtkMsgView (which will be the type that we’ll decorate). The TnyGtkHeaderView is decorated by TMutHeaderView. This TMutHeaderView is then created by the TMutTnyGtkMsgView.

TMutHeaderView decorates TnyGtkHeaderView by surrounding it with a GtkExpander. TMutMsgView decorates TnyGtkMsgView (TMutTnyGtkMsgView, actually) by surrounding it with a GtkScrolledWindow.