Author: admin

This guy from Igalia has the new Air laptop thing already, go check it out!

Video by this guy.

The same rfc2047 decoder fixes that Jeffrey did for upstream Camel are of course ported to Tinymail‘s parsing code. So E-mail clients like Modest will also parse those broken E-mails correctly (once you update your Modest packages on your Nokia devices, of course).

Of course are Nokia’s testers testing the application with such broken E-mails. We are indeed seeing that more E-mails can be displayed correctly with Jeffrey’s new rfc2047 decoder. Usually spam succeeds more often now. Legitimate E-mails are less frequently broken. I guess spammers want to fool weak rfc2047 decoder implementations in spam detection softwares.

For the last few weeks I have been synchronizing the embedded Camel of Tinymail with Camel upstream. Other than bringing upstream’s bugfixes to Tinymail, this will of course make it more easy to port features back to Camel upstream. I must stress again that a lot of the new features are specific for mobile use cases and that a lot of them are not done in such a way that they can easily be ported. Others are simply not very interesting for a desktop E-mail client, and some are.

First of all, a summary is the overview of your E-mail folder or mailbox. It shows you the cc, to, from, subject of each E-mail. In IMAP terminology people also call this the ENVELOPE of each E-mail. Showing all ENVELOPEs of a folder is showing the summary to the user. Some people want more than just the cc, to, from and subject to be visible. Most E-mail clients also indicate the read and importance status of the E-mails in this view. Some E-mail clients also show the size of the E-mail. Whatever yours shows, that is what I’ll here call … the summary.

About a year ago I was telling people about how few memory Tinymail consumed: it consumes fewer memory than most other E-mail clients because it maps the summary data. That’s still true and, in my opinion, a lot better than just copying the strings in memory. However …

The implementation didn’t care a enough about the VmRss. The analysis of memory usage was focused on what valgrind’s massif tool showed. That of course shows just the heap and the stack. Both are relevant, but not the only kind of numbers that you have to consider. The referred-to page explains this too (don’t worry, I was never trying to hide this fact).

What matters for a mobile device is the VmRss. The VmRss is a number that indicates how much of your data is in real memory modules. For a mobile device this is important because these devices often can’t have a swap partition, or a slow one on a level wearing flash device, have relatively few RAM installed, or consume more battery and/or are more expensive per unit to produce. I know RAM is not important for your server, and I don’t care about your server.

Oh .. that desktop or laptop that you just bought? In terms of amount of memory, it’s a server too in my eyes. I don’t care about your desktop machine either.

Another something that ain’t really good about the current implementation is that writing the file takes a long time, that it grows relatively large, and that it can contain redundant information. This redundant information contributes to the VmSize growing.

Note: VmSize is the total amount of memory being used, including things that got swapped out, libraries, heap, stack, everything. As a number it’s good for making wannabe geeks scared about the memory consumption of your application. Other than that, it’s not interesting as a number unless you have a good idea what exactly it means (like, know how shared libraries are handled nowadays and things like that).

What is worse is that because of the redundancy of data the locality of the data that might be required becomes fragmented (you’ll get more page faults). This contributes to a growing VmRss, which is bad news for the mobile device that is sparse on memory availability. I’d love to explain why this contributes to VmRss, … it comes down to “a kernel can only page in using buffers/blocks of 4k”, “no matter how small your string is”, “so keep the ‘needed’ things close together and try to fit them in as few pages as possible”.

Average Joe Six Pack the kernel developer will simply translate that to: try to avoid page faults.

I started writing an experimental new summary storage engine which will in future be used by Tinymail. This one will store duplicate strings uniquely, will sort the strings on reference count, will store lists of addresses (the to and cc fields) in a sorted way (in the hopes of creating more duplicate strings this way) and finally to speedup the writing it will store blocks of summary data rather than all summary of a folder in one big file. The reason for the blocks is that a summary is relatively read only and usually only grows in size. (ps. Desrt is the cool guy who came up with the idea of sorting the lists of addresses)

Next to all these improvements, it’ll have a few new features like out of order adding and expunging of items using both the UID and the sequence number of the item. The reason for this flexibility in the API is that modern IMAP servers more or less require you to look them up using both the UID and the sequence number while handling EXPUNGE, FETCH and VANISHED responses.

These are the results of a summary with 50,000 completely unique summary items (each string in each field of the summary item is unique). It’s a worse case scenario. Everybody probably understands that a large amount of strings in the summary view of his E-mail’s INBOX are duplicates. Right? The strings where Around 20 – 30 bytes each. This is a low average, most E-mails have larger strings. (But most blog items have smaller texts, I know, patience, I’m almost finished)

Mmap file's size: 13 Mb

VmPeak:    22076 kB
VmSize:    22076 kB <-
VmLck:         0 kB
VmHWM:      7604 kB
VmRSS:      7604 kB <-
VmData:     7084 kB
VmStk:        88 kB
VmExe:        16 kB
VmLib:      2152 kB
VmPTE:        16 kB

We can see the large VmSize (large, as we expected, since the mapped file is 13 Mb in size). Interesting the VmRss is just around 8 Mb. These eight megs of mostly heap and stack is being used by pointers that point to the data in the mapping, hashtable nodes and admin info like the reference-count integer of the items. I did the measurement before touching any of the items's data. The kernel has therefore effectively not paged in any of the mapped file's data (on demand paging). This memory, however, is what I call "must have": you wont ever get rid of those eight megs with a folder that has 50,000 items loaded. If after those the VmRss grows caused by pages from the mapped file, when memory availability gets sparse your other applications can get it back from the kernel (depending on how active you are using the E-mail client's summary data at the moment, of course).

You can get the experimental summary store here (it's attached to the mail).

I just released Tinymail’s pre-release 0.0.6. I’m hoping (and thinking) that this will be one of the last pre-releases before I’ll do a final first version of Tinymail. Although I have probably been telling this about every previous pre-release.

This release has neat new features and a lot of important bugfixes. Among the new stuff you’ll find Vala bindings, Namespace support, symbol annotations, a lot of API documentation improvements, improved decoding & language support and it has endured a lot of testing for Modest’s beta releases (which might explain the many bugfixes).

You can find the release’s announcement here.

I just committed support for NAMESPACE in Tinymail. This means that now you can show your boss’s shared folders in Tinymail based E-mail clients.

The “Shared” and “Other” namespaces are often used to put large lists in like NNTP newsgroups, so I only support the list LSUB returns for those two namespaces. LSUB gives you only the mailboxes that you are subscribed to. For the “Personal” namespace, I get both the LIST and the LSUB or in case you have LIST-EXTENDED I ask for LIST (SUBSCRIBED) and filter-out the \NonExistent ones, and flag the \Subscribed ones.

The better way to use “Shared” and “Other” is probably by not recursively asking for the entire list. In stead, when the list is unfolded you ask for the flat list of children under the level of the list. This of course means that you can’t know whether or not the UI must show an unfold arrow. For that you have the \HasChildren flag in IMAP. I’m not yet using this, though. More work for me, yippee!

Once this is done, you could make a UI to help to user subscribe to folders in accounts that have very large amounts of folders.

Evolution’s default IMAP provider does not support shared folders. The problem is of course in the code that handles namespaces. I fixed this in Tinymail‘s camel-lite. I know this is a much requested feature for Evolution too.

I would like to point out to the heroes who’ll, in stead of begging for me to port the patch to Evolution, this time port the patch themselves, that the patch is a.t.m. in a very ugly state (I made a few shortcuts in the code). That camel-lite’s LIST code for IMAP has been adapted in multiple places to consume less round trips (it might not apply easily on upstream Camel, you’ll have to do quite a lot of merging). That this stuff is experimental and that a raw port of the patch would most likely not get accepted upstream yet.

If you want to work on this perhaps join one of the IRC channels that I use and let’s discuss how we’ll clean it up and bring it upstream.

Edit: I’m making more intrusive changes for Tinymail’s version right now. Check Tinymail’s mailing list to follow up on this (for example this one).

Igalia and Dirk-Jan C. Binnema from Nokia just announced the availability of Modest‘s first public beta. You can find the one-click install link at their announcements.

The demo E-mail client can be found here, the Vala bindings for Tinymail can be found here.

Many thanks to Jürg!

Here are the instructions on how to get the binding regenerated.

Just do this to get the Vala demo E-mail client running:

• Get Vala’s latest release here. You don’t need anything special. Just configure, make and sudo make install it with –prefix=/opt/vala for example.
• export VALAC=/opt/vala/bin/valac
• export PATH=$PATH:/opt/vala/bin
• Build Tinymail: ./configure –prefix=/opt/tinymail –enable-vala-bindings && make && sudo make install
• Setup your Tinymail (demoui) accounts (described here)
• cd bindings/vala ; make && sudo make install
• cd tests/vala-demo
• make ; ./tinymail-vala-test

[17:03 today] <juergbi> pvanhoof: i’m reading email with a vala tinymail client now :)

I just documented TMut. I think this documentation will help E-mail client developers who are planning to use Tinymail a lot. Some of them will probably take a look at how I did TMut and now they’ll have some documentation about how and why I did certain things.

Also the client developers who’ll start their E-mail client based on TMut will enjoy reading the documentation. If somebody wants to create a client based on TMut, and if for that we need to make TMut a static library in stead, just ask of course (or just send me the patch that adapts the Makefile.am for this).

You can enjoy my normal English as Tinne is at this moment sleeping and I don’t want to wake her up to correct my poor English. The language purists will have to wait for her language correcting until at least tomorrow, or correct it themselves (it’s a wiki, go ahead). Perhaps we should go over all of Tinymail’s and TMut’s documentation once more. It has been a while since we last did a massive spelling correction campaign.

Anyway, here are the most interesting added subjects:

• Creating messages
• Select a folder window
• TMut’s menu, its first screen
• Showing the summary of a folder
• The message view component
• TMut’s frame & window setup

Have fun

Note (edit): Some people prefer me to call IMAP’s Push-Email “notifications”. That’s fine of course. Just replace “Push” with “notification”.

Everybody who’s into IMAP already knows: IMAP does Push E-mail. Nowadays nearly all IMAP servers are implementing the necessary features for it.

Marketing people wanting to sell their proprietary mail server solution have been claiming that IMAP is not true push e-mail because it doesn’t push the entire E-mail to the client. This is incorrect and just marketing. It’s true that IMAP gives you as a client developer the opportunity to decide to waste bandwidth or not at the cost of one single round trip. Round trips are less important for Push E-mail, as a push event is a background event.

What happens is very simple. In stead of pumping the entire message through, it sends the new total count of your folder. This makes sense, because if you are not interested in push events, you only wasted a few bytes of bandwidth and if subsequent push events happen you get automatically upgraded. Just compare your local number with the new number and make a decision.

C: a IDLE
S: idling+
S: ...
S: * 3 EXISTS
C: DONE
C: a OK
C: b FETCH 3 (ENVELOPE)
S: * 3 FETCH (ENVELOPE ("Tue, 19 May...))
S: b OK
C: c IDLE
S: idling+

We see that we have to exit IDLE to fetch the message’s ENVELOPE. This is why NOTIFY is being introduced. With NOTIFY you simply set at the start of the session what you want the IMAP server to send you during a push event.

If you want it to pump the entire E-mail through, you can instruct it. True push E-mail according to the marketing guys! Nobody really cares about this. though. People with a technical understanding know that you just want to fetch the fields that are interesting and absolutely not the entire mail: that way a spammer could bankrupt you if he knows that you are using expensive GPRS while roaming.

NOTIFY makes the client implementation more simple. You can e.g. request the X-Priority header and/or your company’s custom header. “X-Came-From The-Boss: Yes”. Based on that info you can choose to make the phone perform certain actions.

C: a notify set (selected MessageNew (uid
            body.peek[header.fields (from to subject)]) (all)
            MessageExpunge) (subtree Lists MessageNew (uid) (all))
S: ...
S: * 127 FETCH (UID 127001 BODY[HEADER.FIELDS (From To Subject)] {75}
S: Subject: Re: good morning
S: From: alice@example.org
S: To: bob@example.org
S:
S: )
S: ...

GPRS and IDLE or NOTIFY are fine by itself too. When we wait long enough in the select(), most GPRS chips will automatically go into low power mode. This way you can wait up to 30 minutes before an IMAP server thinks that you got disconnected. Such very high timeout times are specified for e.g. mobile purposes and high latency situations (IMAP from Mars!). Misconfigured NAT gateways that make timeout decisions more early are a problem. The biggest problem is that there’s no way to find out what the timeout for a session at the NAT is.

IMAP as a protocol makes it possible to process alien Push events too. Most IMAP servers, including Cyrus, provide a small daemon that handles the pushing. Cyrus, for example, has “idled” for this purpose. Very often you can quickly make a plugin for the IMAP server such as converting the IDLE event to an SMS and sending it to the user’s phone.

Just remember that with a good GPRS chip with correct power management, this ain’t really needed

If you are a phone builder and you are integrating a GPRS, UMTS or EDGE chip, I recommend getting the power management of it right. In the future your users will stay connected with the IMAP server: the timeout for the NOOP can be high. I hope the people who maintain the NAT gateways of GPRS, UMTS and EDGE at the many mobile networks will consider such use cases in stead of closing sessions at extremely small timeouts.

Tinymail does Push E-mail with IMAP :-)

An IMAP server is smarter than a POP server. This, I hope, comes as no surprise to most people. An IMAP server has a MIME parser and can give you the BODYSTRUCTURE of a message. This is useful because that means that you can individually request the parts of a message, rather than always fetching the entire message. In future you can also CONVERT those to for example thumbnail versions. (I already started implementing support for CONVERT in Tinymail)

For a mobile user this makes a difference. Suddenly he wont have to pay for penis enlargement advertising images while working roaming over a GPRS connection at five euros per megabyte. This is an actual price in Belgium when you are five kilometers across the border in the Netherlands (I live 10 kilometers from the border, I can walk the 15 kilometers to have to deal with that price).

With today’s currency fluctuation I’ll describe what five Euros are: five Euros are two cans of CocaCola to download putty.exe and one can of CocaCola to receive two spam messages about how small your penis is with Evolution (I guess some people’s penis will actually spontaneously shrink if they’d know how much they payed for receiving the spam). I guess I don’t have to explain you what this means for Google Maps? I think that buying a Garmin or TomTom device is going to be less expensive than one month of using Google Maps with a phone. Usually it’s because you are in a foreign country that you need a service like Google Maps, right?

This realism of today should give desktop users an idea why bandwidth consumption is important. Another reason is of course latency. I’m sure UMTS and EDGE fans will explain me that soon, oh dear soon, all such problems will be gone. Bad luck as both UMTP and EDGE still have a very high latency (although slightly better than GPRS).

That what makes the user think that a service is slow, is what technical people call latency. Pure bandwidth performance matters for non-interactive protocols. E-mail is a highly interactive protocol by nature. A E-mail client user clicks and views a lot of E-mails and folders and wants the application to respond reasonably fast nonetheless. He doesn’t want to either wait for nor pay a lot of money while synchronizing E-mail data.

Synchronizing data is not how most people use their phones. Most people press the power button and want it to just fucking work. Mobile can only be a success if things like this just work. Users don’t want to be bothered with setting up an E-mail data synchronization software on their laptop, drag that laptop with them, and from time to times synchronize data to their phone.

The reality that Western people usually can’t grasp is that most mobile phone users in the world don’t even have one computer.

With LEMONADE, IMAP does pretty well at avoiding round trips. Fewer round trips and using IMAP’s pipeling capabilities means less latency. Less latency means faster responding E-mail clients. Given that the HTTP protocol is not really designed to avoid round trips, a web client in a web browser wont ever perform well with mobile networks. On top of that have a lot of web client services the annoying habit of adding bandwidth-expensive advertisements on the page itself. On top of that don’t all people like the privacy aspect. Especially businesses don’t like this (although they can of course buy a web service appliance for E-mail too). On top of that do some people want to fetch messages locally, and read them while being offline.

With BODYSTRUCTURE we can selectively fetch the parts of an E-mail as they are needed, rather than always.

Use the Tinymail API to build your E-mail client, and it’ll do both automatically for you.

After that much text, a video demo is a requirement. Right? Right! You’ll see that each time I expand the expander, the image gets downloaded. You can see this because Nautilus’s directory monitor will add the file to its directory viewer.

I guess sometimes I do need a lot of words :-)

ps. Build instructions for everything, here. No I don’t know whether that will work with your 770. Yes it’ll work with your N800 and your N810.

I just added the bodystructure parser that I mentioned a few hours ago to a Subversion repository. I also fixed a bunch of parsing mistakes and figured out a few of the unknown fields.

Edit

I just added error checking. The parser now supports recursive bodystructures for multipart/*, application/*, text/*, message/rfc822, audio/* and image/*. It will parse as much fields as possible (I think I’m only missing two or three fields right now, they seem the be always NIL anyway). Fields include the parameters, content disposition, octets, lines, encoding, description, language, …

By requesting the BODYSTRUCTURE you can get a preview of the structure of a message without having to download it. Most E-mail clients parse this structure to the point of what they need from it. I did some searching and as I expected did few of the opensource E-mail clients do a complete parse. Some just scanned for some words, others did a reasonably well job but skipped information that they are simply not interested in. And that’s fine, if you don’t need it.

However

Soon will CONVERT be unleashed by the IMAP server developers. CONVERT is very interesting for high latency networks, mobile devices with limited display capabilities and other situations where you want to convert something in the E-mail to something else. A perfect example is scaling down the dimensions of an image that is embedded in a HTML message.

At the Lemonade interop that we did this week I met Roger Grönberg. The company where Roger works has an interesting but quite different solution for this. To me it’s interesting because it shows how wanted such a feature is on mobiles like phones: at the SMTP server first they somehow tag each image, then they rewrite it in the E-mail by scaling it to the perfect size for your mobile (they store your mobile’s capabilities at their SMTP server), and then the SMTP server delivers it to their IMAP server. Your phone’s standard E-mail client will now get a nicely formatted E-mail. Because they tagged it they’ll revert this change when you forward the image to somebody else. So they find the original image back, and correct the forwarded E-mail with that original image.

This approach of course implies various difficulties and changing the content of the E-mail at the location of the IMAP server. Although the server can still recover from this change, of course. With Lemonade we are, however, interested in making this possible without needing such tricks.

I started my blog item with BODYSTRUCTURE, didn’t I? Let me get back to that. You can probably imagine that if you want the IMAP server to instruct converting a MIME part to something suitable for your situation, that you’ll need to have a lot of information about the original MIME part before you start converting. Right? In case of a bitmap you probably want to know about the dimensions of the original one, so that you can ask CONVERT to convert while maintaining the aspect ratio of the image.

Well. For that we need a full BODYSTRUCTURE parser. Especially the content’s parameters are needed. I know I should probably have used sexp for parsing the S-expression like structures and I know that I probably got a bunch of things wrong. I even know that this current version leaks memory. For now it’s the parsing that matters most. Next on my list is hardening it against Courier-like IMAP servers. With some versions of Courier, you never know what you’ll get.

I prepared the sample so that you can just run “./test_app.sh”. That script will fetch 10,000 BODYSTRUCTUREs from my test IMAP server and then it will start parsing them.

I hope other E-mail client authors will test this stuff and point me to bugs, send me patches. Perhaps use it in their E-mail clients as soon as it’s actually usable. If somebody wants to convert this away from glib, to no-strings-attached pure C with POSIX, or C with the Linux kernel’s library then that’s fine and cute, send me the patch.

Oh yes. I released Tinymail pre release 0.0.5 and TMut 1.1.0 yesterday evening. The reason is that a specific series of bugs got fixed shortly after Tinymail’s pre release 0.0.4. I adapted TMut to leverage those fixes. The fixes also include moving folders and making the TnyGtkFolderStoreTreeModel automatically update. I also fixed a few embarrassing reference count fixes on the accounts and folders. The instances that where leaked where not consuming a lot of memory, but it’s not nice to leak entire objects of course. Especially not if you claim to focus on mobiles, like I do. Refdbg helped me a lot with solving this. Thanks Josh!

TMut now also supports displaying which messages are marked as seen, unseen and deleted. Note that TMut does this at the event of a unsolicited response from the IMAP server. This means that with today’s IMAP servers you’ll get instant updates to your view.

I have started implementing consuming NOTIFY in Tinymail. NOTIFY isn’t very different from IDLE and it actually complements IDLE when you are in that mode. With NOTIFY you specify which events you are interested in. You also get certain events as unsolicited responses about folders that are not currently selected. A few samples of events that you can request are new mailbox creations, mailbox deletions, mailbox renames, subscription changes and status changes. Status changes happen when the total amount of messages change. Regretfully doesn’t NOTIFY at this moment specify telling you when the total amount of unseen messages changed (although an IMAP server can probably nonetheless implement this). But then again, next week we’re doing an IMAP Lemonade interop in Munich. NOTIFY is already on the agenda.

I’m also planning to rewrite the summary storage in such a way that I can selectively retrieve summary information. For example Polymer, Mulberry and Pine retrieve summary information in ‘waves’ (that’s a name for it that I just invented). My claim has always been that this is typical something for an E-mail client that works while you are online, and that Tinymail has a strong emphasis on working offline too. Yet it’s possible to focus on retrieving exactly what the user is looking at the first time he opens a mailbox. A lot of the technical users would be more happy if Tinymail based clients would do this too. Most normal users, who have average mailbox sizes (like 20,000 items or less), are not going to notice the difference that much: once Tinymail has this cached locally, it of course only needs to fetch the new messages’s summary data. It’s not that it’s usual that you have to ‘wave’ download those. It’s rather usual that the amount of new summary data is small enough to fit in one TCP frame anyway.

It’s a technical challenge to get this working, so I will make Tinymail do this too. The most difficult part is making my mmap() based summary store cope with this. The retrieving part itself is rather simple and implementable in five minutes. Mulberry is an example of how such an E-mail client can work both offline and yet do this kind of stuff.

I just released TMut 1.0.0. I recommend using TMut with what you’ll find in Tinymail‘s Subversion repository. It has been tested with Tinymail’s pre release 0.0.4 too.

In pre release 0.0.4 there’s one known bug about moving folders that has been fixed in trunk.

This version of TMut doesn’t come with an account wizard or account management screen. You’ll have to configure TMut using GConf if you want to use it on your GNOME desktop. TMut’s target devices are for example phones and other mobile devices.

gconftool-2 -s /apps/tinymail/cache_dir -t string .tmut
gconftool-2 -s /apps/tinymail/accounts/count -t int 1
gconftool-2 -s /apps/tinymail/accounts/0/name -t string "My GMail account"
gconftool-2 -s /apps/tinymail/accounts/0/proto -t string imap
gconftool-2 -s /apps/tinymail/accounts/0/type -t string store
gconftool-2 -s /apps/tinymail/accounts/0/user -t string user.name
gconftool-2 -s /apps/tinymail/accounts/0/hostname -t string imap.gmail.com
gconftool-2 -s /apps/tinymail/accounts/0/options -t list --list-type=string '[use_ssl=wrapped]'

Or just use this GConf dump. And change “user.name” to your user name on GMail (or change the hostname to your IMAP server etc etc). This dump comes with GMail’s SMTP server configured too (then you can also send E-mails).

gconftool-2 --load tmut.gmail.gconfdump

I decided to do another Tinymail pre release. I haven’t yet run out of numbers, so this is pre-release v0.0.4.

Now that TMut is more or less (basic) feature complete, I’m thinking about doing a TMut 1.0.0 too. I don’t know when Modest will be released but I do think it’s almost the time of the Tny E-mail clients. If not Modest then I’ll make a few other E-mail client flavours … like TMut.

There are (probably) still small problems here and there in Tinymail. Mostly the kind of problems that can only be found by putting the code in the hands of users. There have been two or three API changes so the Python bindings are not synchronized (Python developers who started using Tinymail might want to stick with pre-release 0.0.3 until this is fixed).

Oh. I was taking a look at QAbstractItemModel recently. I think it’s doable to make a TnyList that also implements QAbstractItemModel in C++. Having convenient models for views like GtkComboBox and GtkTreeView is more or less what most of libtinymailui-gtk is about. Doing the same thing for QTreeView and QComboBox sounds like a doable idea to me.

I also identified all uses of the GMainLoop in Tinymail. I can easily abstract that (although I know the people at Qt are interested in having GMainLoop integration too).

If you want me to make Tinymail as easy to use with Qt as with Gtk+, fund me and I will :-). Technically it’s perfectly possible and not really hard.