Category: english

By requesting the BODYSTRUCTURE you can get a preview of the structure of a message without having to download it. Most E-mail clients parse this structure to the point of what they need from it. I did some searching and as I expected did few of the opensource E-mail clients do a complete parse. Some just scanned for some words, others did a reasonably well job but skipped information that they are simply not interested in. And that’s fine, if you don’t need it.

However

Soon will CONVERT be unleashed by the IMAP server developers. CONVERT is very interesting for high latency networks, mobile devices with limited display capabilities and other situations where you want to convert something in the E-mail to something else. A perfect example is scaling down the dimensions of an image that is embedded in a HTML message.

At the Lemonade interop that we did this week I met Roger Grönberg. The company where Roger works has an interesting but quite different solution for this. To me it’s interesting because it shows how wanted such a feature is on mobiles like phones: at the SMTP server first they somehow tag each image, then they rewrite it in the E-mail by scaling it to the perfect size for your mobile (they store your mobile’s capabilities at their SMTP server), and then the SMTP server delivers it to their IMAP server. Your phone’s standard E-mail client will now get a nicely formatted E-mail. Because they tagged it they’ll revert this change when you forward the image to somebody else. So they find the original image back, and correct the forwarded E-mail with that original image.

This approach of course implies various difficulties and changing the content of the E-mail at the location of the IMAP server. Although the server can still recover from this change, of course. With Lemonade we are, however, interested in making this possible without needing such tricks.

I started my blog item with BODYSTRUCTURE, didn’t I? Let me get back to that. You can probably imagine that if you want the IMAP server to instruct converting a MIME part to something suitable for your situation, that you’ll need to have a lot of information about the original MIME part before you start converting. Right? In case of a bitmap you probably want to know about the dimensions of the original one, so that you can ask CONVERT to convert while maintaining the aspect ratio of the image.

Well. For that we need a full BODYSTRUCTURE parser. Especially the content’s parameters are needed. I know I should probably have used sexp for parsing the S-expression like structures and I know that I probably got a bunch of things wrong. I even know that this current version leaks memory. For now it’s the parsing that matters most. Next on my list is hardening it against Courier-like IMAP servers. With some versions of Courier, you never know what you’ll get.

I prepared the sample so that you can just run “./test_app.sh”. That script will fetch 10,000 BODYSTRUCTUREs from my test IMAP server and then it will start parsing them.

I hope other E-mail client authors will test this stuff and point me to bugs, send me patches. Perhaps use it in their E-mail clients as soon as it’s actually usable. If somebody wants to convert this away from glib, to no-strings-attached pure C with POSIX, or C with the Linux kernel’s library then that’s fine and cute, send me the patch.

Oh yes. I released Tinymail pre release 0.0.5 and TMut 1.1.0 yesterday evening. The reason is that a specific series of bugs got fixed shortly after Tinymail’s pre release 0.0.4. I adapted TMut to leverage those fixes. The fixes also include moving folders and making the TnyGtkFolderStoreTreeModel automatically update. I also fixed a few embarrassing reference count fixes on the accounts and folders. The instances that where leaked where not consuming a lot of memory, but it’s not nice to leak entire objects of course. Especially not if you claim to focus on mobiles, like I do. Refdbg helped me a lot with solving this. Thanks Josh!

TMut now also supports displaying which messages are marked as seen, unseen and deleted. Note that TMut does this at the event of a unsolicited response from the IMAP server. This means that with today’s IMAP servers you’ll get instant updates to your view.

I have started implementing consuming NOTIFY in Tinymail. NOTIFY isn’t very different from IDLE and it actually complements IDLE when you are in that mode. With NOTIFY you specify which events you are interested in. You also get certain events as unsolicited responses about folders that are not currently selected. A few samples of events that you can request are new mailbox creations, mailbox deletions, mailbox renames, subscription changes and status changes. Status changes happen when the total amount of messages change. Regretfully doesn’t NOTIFY at this moment specify telling you when the total amount of unseen messages changed (although an IMAP server can probably nonetheless implement this). But then again, next week we’re doing an IMAP Lemonade interop in Munich. NOTIFY is already on the agenda.

I’m also planning to rewrite the summary storage in such a way that I can selectively retrieve summary information. For example Polymer, Mulberry and Pine retrieve summary information in ‘waves’ (that’s a name for it that I just invented). My claim has always been that this is typical something for an E-mail client that works while you are online, and that Tinymail has a strong emphasis on working offline too. Yet it’s possible to focus on retrieving exactly what the user is looking at the first time he opens a mailbox. A lot of the technical users would be more happy if Tinymail based clients would do this too. Most normal users, who have average mailbox sizes (like 20,000 items or less), are not going to notice the difference that much: once Tinymail has this cached locally, it of course only needs to fetch the new messages’s summary data. It’s not that it’s usual that you have to ‘wave’ download those. It’s rather usual that the amount of new summary data is small enough to fit in one TCP frame anyway.

It’s a technical challenge to get this working, so I will make Tinymail do this too. The most difficult part is making my mmap() based summary store cope with this. The retrieving part itself is rather simple and implementable in five minutes. Mulberry is an example of how such an E-mail client can work both offline and yet do this kind of stuff.

I just released TMut 1.0.0. I recommend using TMut with what you’ll find in Tinymail‘s Subversion repository. It has been tested with Tinymail’s pre release 0.0.4 too.

In pre release 0.0.4 there’s one known bug about moving folders that has been fixed in trunk.

This version of TMut doesn’t come with an account wizard or account management screen. You’ll have to configure TMut using GConf if you want to use it on your GNOME desktop. TMut’s target devices are for example phones and other mobile devices.

gconftool-2 -s /apps/tinymail/cache_dir -t string .tmut
gconftool-2 -s /apps/tinymail/accounts/count -t int 1
gconftool-2 -s /apps/tinymail/accounts/0/name -t string "My GMail account"
gconftool-2 -s /apps/tinymail/accounts/0/proto -t string imap
gconftool-2 -s /apps/tinymail/accounts/0/type -t string store
gconftool-2 -s /apps/tinymail/accounts/0/user -t string user.name
gconftool-2 -s /apps/tinymail/accounts/0/hostname -t string imap.gmail.com
gconftool-2 -s /apps/tinymail/accounts/0/options -t list --list-type=string '[use_ssl=wrapped]'

Or just use this GConf dump. And change “user.name” to your user name on GMail (or change the hostname to your IMAP server etc etc). This dump comes with GMail’s SMTP server configured too (then you can also send E-mails).

gconftool-2 --load tmut.gmail.gconfdump

I decided to do another Tinymail pre release. I haven’t yet run out of numbers, so this is pre-release v0.0.4.

Now that TMut is more or less (basic) feature complete, I’m thinking about doing a TMut 1.0.0 too. I don’t know when Modest will be released but I do think it’s almost the time of the Tny E-mail clients. If not Modest then I’ll make a few other E-mail client flavours … like TMut.

There are (probably) still small problems here and there in Tinymail. Mostly the kind of problems that can only be found by putting the code in the hands of users. There have been two or three API changes so the Python bindings are not synchronized (Python developers who started using Tinymail might want to stick with pre-release 0.0.3 until this is fixed).

Oh. I was taking a look at QAbstractItemModel recently. I think it’s doable to make a TnyList that also implements QAbstractItemModel in C++. Having convenient models for views like GtkComboBox and GtkTreeView is more or less what most of libtinymailui-gtk is about. Doing the same thing for QTreeView and QComboBox sounds like a doable idea to me.

I also identified all uses of the GMainLoop in Tinymail. I can easily abstract that (although I know the people at Qt are interested in having GMainLoop integration too).

If you want me to make Tinymail as easy to use with Qt as with Gtk+, fund me and I will :-). Technically it’s perfectly possible and not really hard.

Yet another video demo of TMut. This time I’m consuming most of the operations. Like creating, moving, renaming and deleting folders, sending an E-mail to a GMail account. Then seeing that E-mail arrive on the GMail IMAP server, utilizing the support for IDLE, making a reply and a forward, using a local Maildir folder, moving messages. Stuff.

The video demo also shows the recent new TMutFolderView that reuses Modest’s GtkCellRenderers in the GtkTreeView. These cell renderer reduces space and shows the E-mail header line in the folder summary in two rows.

All this TMut code is still hot, fresh and new. Be careful not to burn your fingers.

While I was not searching for a new project, I have been throwing TMut together. After six quick sessions of working on it, I made TMut:

• Support multiple accounts simultaneously
• Open and display text/plain and text/html E-mail
• Open and display IMAP folders
• Open and display the INBOX of a POP account
• Delete IMAP and Maildir folders
• Create IMAP and Maildir folders
• Move IMAP and Maildir folders
• Copy IMAP, POP and Maildir folders
• Rename IMAP and Maildir folders
• Delete messages from IMAP folders
• Delete messages from POP folders
• Move messages from one folder to another
• Put the connection in IDLE on IMAP and receive notifications
• Create new messages
• Create a reply of a message
• Create a forward of a message, as attachment
• Have accurate progress information

Not unimportant but because Tinymail comes with this support I also made TMut:

• Do CONDSTORE on IMAP
• Do QRESYNC on IMAP
• Do BINARY on IMAP
• Do IDLE on IMAP
• Do various SSL and TLS encrypting
• Do various authentication methods
• …

I’m planning to fine tune all this a little bit, then do a GPEPhone port of it. This will include overriding some of the virtual methods that are right now creating a more or less standard user interface for a GNOME desktop.

I also have to make some sort of optional account editor. Optional because some device vendors have the account editing done by a web page and then uploaded to the device via an SMS (this is not uncommon on for example Nokia phones). I just remember I still have to make a little bit UI for adding and removing attachments from a new message too. When forwarding the forwared message is right now added as an attachment already.

Once all that is finished I of course have to prepare a video demo of running code. I guess the VMWare image will help me a lot with that. After that demo, I will most likely do a release of TMut. Perhaps at the same time release a first Tinymail version too? I’m not sure of that yet. Perhaps I will do a few more pre-releases in stead. It has to be perfect.

I think TMut is ready for contributors now. If you want to join and hack together your favourite E-mail client feature, just get on the Tinymail mailing list and post me your great patches. I also have to document the code of TMut a bit more. And make TMut’s wiki page a bit better too.

The joy of starting yet another cute little project starts …

As in the world of mobile and embedded aggressive changes in technology and devices’ capabilities occur at a high frequency, I decided to make Tinymail‘s API very flexible or adaptive.

Yet I wanted to provide a certain functionality, which means a message view component that also shows the typical message headers.

What if the developer wants to wrap the message view in a scrolled window, have a message viewer as a new window or put the thing that displays the header in an expander? On a mobile device with a limited screen height it is a waste of space to show four or five lines of information. If the user wants this info, he can decide to expand it.

Imagine the same software vendor want to adjust the E-mail client for another device? However, nowadays software development must be fast. The change is something that should not destabilize everything that was already working. It must be easy to develop a new version of the message view component that perhaps behaves in a new way. Who knows what the screen will be capable of?

This diagram shows the flexibility of Tinymail‘s types. I’m using the decorator pattern here. Decorator is also used by the streams API of Java, .NET and GVFS. I explained Tinne about decorator by simply referring to a painting and its borders. When decorating a painting with borders, you can still look at the painting. Nothing about its functionality really changes, but you add things to it, you decorate it.

In stead of subclassing TnyGtkMsgView, I decided to decorate it with a new implementation of the TnyMsgView interface. This one wraps the TnyGtkMsgView instance (it makes a composition with it). I did the same thing for TnyGtkHeaderView that implements the TnyHeaderView interface.

Because TnyGtkMsgView‘s method for creating its TnyHeaderView is abstract, I inherited it as TMutTnyGtkMsgView (which will be the type that we’ll decorate). The TnyGtkHeaderView is decorated by TMutHeaderView. This TMutHeaderView is then created by the TMutTnyGtkMsgView.

TMutHeaderView decorates TnyGtkHeaderView by surrounding it with a GtkExpander. TMutMsgView decorates TnyGtkMsgView (TMutTnyGtkMsgView, actually) by surrounding it with a GtkScrolledWindow.

I know Tinymail has been a far-from-my-bed show for a lot of people. Especially since Modest hasn’t yet been released, it’s taking ages for me to do a first Tinymail release, etc etc.

My opinion is that all these are just the normal troubles that most medium-size software projects go through. I know some people expected certain wonders coming from all this. Wonders we will create. My opinion is although what can be better must be better, Tinymail is getting nice. It’s not perfect. But nothing will ever be perfect in my opinion. Ask Tinne, I’m full of criticism about nearly everything, including myself. Nothing is ever good enough. Which is why Tinymail can be better and therefore it must be better.

Balancing perfectionism with pragmatism is probably one thing about my own personality that can be better. It’s definitely something that I’m improving. The good thing is that my philosophy is that it’s not the destination, it’s the path itself that makes life worthwhile. Let’s not be perfect.

The API to build an E-mail client is okay. For the first time I tried putting a simple one together from scratch. Before that, I was focused on what Modest needed. The advice on complexity in “Framework Design Guidelines” by the .NET gurus Krzysztof Cwalina and Brad Adams is to make sure that the simple use cases are easy, and make the difficult ones possible. Don’t try to make both easy. This too is certain pragmatism. I’m learning that real quality lies in this pragmatism. Being too pragmatic most definitely doesn’t lead to quality. Being always a purist, leads to total failure. Quality lies in the balance. I think this applies to quite a lot of domains in life.

TMut is definitely an example of a simple use case. Yet it supports Push E-mail using IDLE, multiple folders, IMAP, POP, NNTP. It does QRESYNC and CONDSTORE. It downloads messages in BINARY. It does local caching of once retrieved data. It consumes not only few bandwidth, it also uses few roundtrips to synchronize a folder.

Much more important in my opinion is that TMut’s code is clean. I could give you guys a line count and everybody would be amazed by how few lines are needed. It’s not (not at all) the point of software development. The COBOL days that software developers got payed based on the amount of typed lines are over. Much more important is that code makes sense.

In my opinion is TMut, while written completely in C, clean and easy to understand. Nonetheless it supports a lot, using little effort.

Finally, after putting TMut together, I’m feeling satisfied about this aspect of Tinymail. It is a good API, it does work, it might not yet be ideal. Even from the point of view where you want to write code that makes sense, it’s doable to build a simple but capable E-mail client in less than a few hours.

That, was the goal that I had in mind a year ago.

What you’ll find in in TMut’s sources is what you need to do to implement a basic E-mail client for a phone using Tinymail. It’s not even a pragmatic quick/dirty thing. In stead (in my opinion) it’s clean and clear code. As typesafe as possible. Proper design. You can probably clean up a bunch of things if you’d do this with Vala or by replacing the GObject boilerplate code with macro cruft.

Chances are high that a project that was being planned will be canceled or at least delayed for weeks. This of course means that I will have to find myself something else to do with my time. Since finding another project usually takes a few weeks, I started with putting together that TMut thingy.

A few months ago I announced that I would start an E-mail client specifically targeted at very small screens. For example mobile phones. Not tablets with a high resolution and relatively wide screens. I was thinking more about those poor phones. I threw together some pieces here and there to get something started. And that was it, I was too busy with Tinymail itself, with Modest, with going to conferences, codecamps, yadi yada.

Today I committed a bunch of stuff for TMut so that it at least shows a first screen with your configured E-mail accounts and the selected account’s folder tree. I still have to make a folder view, then a message view. Perhaps all of it will be finished by this morning. Although I’m planning to go to a Dr. Lektroluv party in Amsterdam tomorrow night and I have to drive home. Maybe it’s a better idea to get some sleep this time?

Update: I’m still up, so that means that a bunch of things got implemented. Like reading E-mails (including HTML ones), IDLE and Push E-mail, getting a summary of a folder, selecting a folder, a thing that plays ringtones. Short version: all that a read-only E-mail client should do.

After that, a message editor, an account editor and then putting some operations in place. Like moving messages around, deleting them, renaming folders, deleting folders, creating folders. Nothing really difficult since TMut will use Tinymail of course. Tinymail’s demo-ui has most of this functionality already and I made it relatively simple to have it.

Something else that I have in mind is putting something together that converts SMS and MMS messages to correct RFC822 MIME messages in a Maildir folder, and make a system that activates Tinymail’s notification infrastructure. That way the application developer will see an SMS or MMS arrive exactly how a Push E-mail arrives during IDLE on IMAP. With a correctly set up folder view, he wont have to do anything. Except maybe implement playing a ring tone or something.

Something else I hope to fine tune, in Tinymail itself, is correctly using the socket in such a way that a typical GPRS chip turns itself off until an event in IDLE arrives, to avoid battery consumption. We obviously don’t want to drain the battery just for having Push E-mail on a phone.

TMut is just a little bit of UI to make it easy to start using all this juicy IMAP stuff. Not a demo-ui, but something that should be relatively usable or at least reusable by phone projects who’ll be using Tinymail as their E-mail client solution. This means that I’m trying to get its types and design as good as possible. You can indeed throw a three-screen read-only E-mail client together in a few minutes using Tinymail. But that’s not the idea here. If you want a sample of a quick/dirty one, check out Tinymail’s demo-ui which I mostly use for debugging Tinymail itself.

If you want me to make one for your device then well, I’m looking for a project! Perhaps we can use TMut as a starting point for this?

With GMail now supporting free IMAP, I predict E-mail is going to become increasingly popular on phones. Although rather limited in its capabilities, GMail’s IMAP server has IDLE (so it supports doing Push E-mail on phones). I hope GMail’s IMAP team will soon add support for CONDSTORE and QRESYNC.

I made the video demo that I promised last night. In this video demo I use ettercap and wireshark to show the difference in network traffic between a typical IMAP client and a Tinymail based one when we have an IMAP server that deals with QRESYNC, IDLE and CONDSTORE (like modern IMAP servers do).

Ettercap is a useful tool to follow the traffic while using the applications. Wireshark is interesting to see how frames are formed while the traffic is being transmitted. When working with GPRS, the latency can be very high. This means that with an interactive protocol like IMAP, the time to retrieve and send a frame (the roundtrip time) has a big influence on the network performance of the application.

In the video demo I explain the difference between the old way of working with an IMAP server, and the modern way with QRESYNC and CONDSTORE. My goal is to convince IMAP server developers to consider implementing both CONDSTORE and QRESYNC so that IMAP clients can make better use of the GPRS network, and of course to show that my precious little Tinymail is supporting this already, if available on the IMAP server.

The video

I just released Tinymail‘s pre-release 0.0.3. This release contains the latest QRESYNC improvements, a lot of other bandwidth improvements, some fixes for the support for IDLE, various fixes for Maildir, merging from Camel upstream, a lot of compilation warnings when turning on -pedantic, improvements in the Gecko HTML message viewer and yet even more improvements for the Python bindings.

As usual brings this release Tinymail a bit closer to its v1.0. The stability has improved a lot and we’re again adding new features.

I’ll try to do another video demo this week. This time about QRESYNC, CONDSTORE and IDLE. Probably with ettercap and wireshark showing you how these IMAP features are saving you roundtrips and bandwidth and why they are more ideal when you’re working with GPRS.

While trying to get QRESYNC working I found a few problems in Tinymail‘s IDLE handling, specifically in its handling of (unsolicited) FETCH responses.

I used to believe some IMAP servers simply ‘forget’ to send certain FETCH responses, but while testing with MBox I noticed that I was receiving all of them correctly. It turned out that I was simply not parsing the FETCH response correctly as the sequence ended up being invalid. Once that was fixed, I found that, would it have been correct, the index was not subtracted with one. IMAP sends sequence numbers starting at one, whereas an array in C obviously starts at zero. If it had worked, it would have marked locally the wrong E-mail as seen or unseen. Ugh.

As a result, the flag changes were not instantly visible. Only notifications about expunges and new messages were really working. This has been fixed of course. This means that you can now start a Tinymail based E-mail client, continue working in another E-mail client, and with the right IMAP server the Tinymail one gets instantly updated about the state changes of messages.

With NOTIFY, the same granularity of change notifications can be achieved for folders. For example, their subscription state, renames, deletions, creations and status changes like the unread and the total counts of each folder. With CONTEXT we can also provide in the future a vfolder that is completely stored at the server. As usual, if you are interested in having this: either fund me or help me implementing it.

An IMAP draft that I really like is CONVERT. Imagine having in a message a document attached in a funny format like Microsoft Word. Do we expect that each and every phone deploys OpenOffice.org just to parse and convert all that data to something that can be viewed? Yeah yeah, the AbiWord guys now want to tell me they have something for phones too, but I can redo the sample with any other existing document format.

Why not putting that AbiWord code in the plugin of an IMAP server in stead? Why consume more bandwidth if perhaps the server can convert it to a smaller format that is immediately suitable for the phone’s display? What about converting large image attachments to the exact size of the display?

Oh, support for QRSYNC is also committed. I will most likely build pre-release 0.0.3 this weekend. I know it has taken me longer than the promised three weeks. The problem was that the amount of bugfixes has decreased drastically. The Modest developers keep telling me that everything is stable and they asked me not to change too much anymore. But now the amount of fixes and even new features is again worthwhile to do a new release.

I wanted to be the first in something, so I just implemented support for the QRESYNC capability in Tinymail. Of course has Dave implemented this a few weeks ago in Polymer too (aaarrg!!). As usual to have an experimental E-mail client to test the capability with their Isode IMAP server.

Anyway, this makes Tinymail implement IDLE, BINARY, CONDSTORE and QRESYNC. These are the ones related to Lemonade.

Next on my list is of course NOTIFY, and pick that over IDLE where possible. NOTIFY is interesting because it gives more information like when a mailbox (a folder) gets created and when it gets deleted. It also tells me about new messages in folders that are not currently selected. In contrast with that is IDLE rather limited.

I still need to test this support for QRESYNC a bit, so it’s not yet committed in trunk. The patch is available on the mailinglist if you want to help testing (and if you can find an IMAP server that already offers the QRESYNC capability). I obviously hope more IMAP server developers will implement QRESYNC. I think it’s a better approach than CONDSTORE. I’ll explain.

QRESYNC is interesting because it avoids having to do roundtrips to synchronize your current folder. Roundtrips are far more expensive (in terms of speed) than most people think while they are using GPRS. The nice thing about not doing roundtrips is that often the combined and atomic reply just fits in one tcp/ip packet. So with QRESYNC you get very fast and cheap folder synchronizations over GPRS.

That’s exactly what we want on our phones, right?

Regretfully few IMAP servers in the field have this already (if any, QRESYNC is still in draft state). But especially if you provide your users with an E-mail service combined with the mobile device that you sold them, you can give them this little bit of extra that many other far more expensive solutions don’t have. It’s yet another reason why to go for Tinymail and IMAP4 Lemonade. You can trust me on it that I’ll create more reasons like this in future.