I, Philip, do hereby pledge to practice absinth-tinence by remaining absinth-tinent from Absinthe …
Since absinthe incidents in many instances induce incipient synesthetic inspiration and sinister synthetic insistence on sin, I sincerely insist I will be absent from instances of Absinthe ingestion, this instant.
I know Tinymail has been a far-from-my-bed show for a lot of people. Especially since Modest hasn’t yet been released, it’s taking ages for me to do a first Tinymail release, etc etc.
My opinion is that all these are just the normal troubles that most medium-size software projects go through. I know some people expected certain wonders coming from all this. Wonders we will create. My opinion is although what can be better must be better, Tinymail is getting nice. It’s not perfect. But nothing will ever be perfect in my opinion. Ask Tinne, I’m full of criticism about nearly everything, including myself. Nothing is ever good enough. Which is why Tinymail can be better and therefore it must be better.
Balancing perfectionism with pragmatism is probably one thing about my own personality that can be better. It’s definitely something that I’m improving. The good thing is that my philosophy is that it’s not the destination, it’s the path itself that makes life worthwhile. Let’s not be perfect.
The API to build an E-mail client is okay. For the first time I tried putting a simple one together from scratch. Before that, I was focused on what Modest needed. The advice on complexity in “Framework Design Guidelines” by the .NET gurus Krzysztof Cwalina and Brad Adams is to make sure that the simple use cases are easy, and make the difficult ones possible. Don’t try to make both easy. This too is certain pragmatism. I’m learning that real quality lies in this pragmatism. Being too pragmatic most definitely doesn’t lead to quality. Being always a purist, leads to total failure. Quality lies in the balance. I think this applies to quite a lot of domains in life.
TMut is definitely an example of a simple use case. Yet it supports Push E-mail using IDLE, multiple folders, IMAP, POP, NNTP. It does QRESYNC and CONDSTORE. It downloads messages in BINARY. It does local caching of once retrieved data. It consumes not only few bandwidth, it also uses few roundtrips to synchronize a folder.
Much more important in my opinion is that TMut’s code is clean. I could give you guys a line count and everybody would be amazed by how few lines are needed. It’s not (not at all) the point of software development. The COBOL days that software developers got payed based on the amount of typed lines are over. Much more important is that code makes sense.
In my opinion is TMut, while written completely in C, clean and easy to understand. Nonetheless it supports a lot, using little effort.
Finally, after putting TMut together, I’m feeling satisfied about this aspect of Tinymail. It is a good API, it does work, it might not yet be ideal. Even from the point of view where you want to write code that makes sense, it’s doable to build a simple but capable E-mail client in less than a few hours.
That, was the goal that I had in mind a year ago.
What you’ll find in in TMut’s sources is what you need to do to implement a basic E-mail client for a phone using Tinymail. It’s not even a pragmatic quick/dirty thing. In stead (in my opinion) it’s clean and clear code. As typesafe as possible. Proper design. You can probably clean up a bunch of things if you’d do this with Vala or by replacing the GObject boilerplate code with macro cruft.
Chances are high that a project that was being planned will be canceled or at least delayed for weeks. This of course means that I will have to find myself something else to do with my time. Since finding another project usually takes a few weeks, I started with putting together that TMut thingy.
A few months ago I announced that I would start an E-mail client specifically targeted at very small screens. For example mobile phones. Not tablets with a high resolution and relatively wide screens. I was thinking more about those poor phones. I threw together some pieces here and there to get something started. And that was it, I was too busy with Tinymail itself, with Modest, with going to conferences, codecamps, yadi yada.
Today I committed a bunch of stuff for TMut so that it at least shows a first screen with your configured E-mail accounts and the selected account’s folder tree. I still have to make a folder view, then a message view. Perhaps all of it will be finished by this morning. Although I’m planning to go to a Dr. Lektroluv party in Amsterdam tomorrow night and I have to drive home. Maybe it’s a better idea to get some sleep this time?
Update: I’m still up, so that means that a bunch of things got implemented. Like reading E-mails (including HTML ones), IDLE and Push E-mail, getting a summary of a folder, selecting a folder, a thing that plays ringtones. Short version: all that a read-only E-mail client should do.
After that, a message editor, an account editor and then putting some operations in place. Like moving messages around, deleting them, renaming folders, deleting folders, creating folders. Nothing really difficult since TMut will use Tinymail of course. Tinymail’s demo-ui has most of this functionality already and I made it relatively simple to have it.
Something else that I have in mind is putting something together that converts SMS and MMS messages to correct RFC822 MIME messages in a Maildir folder, and make a system that activates Tinymail’s notification infrastructure. That way the application developer will see an SMS or MMS arrive exactly how a Push E-mail arrives during IDLE on IMAP. With a correctly set up folder view, he wont have to do anything. Except maybe implement playing a ring tone or something.
Something else I hope to fine tune, in Tinymail itself, is correctly using the socket in such a way that a typical GPRS chip turns itself off until an event in IDLE arrives, to avoid battery consumption. We obviously don’t want to drain the battery just for having Push E-mail on a phone.
TMut is just a little bit of UI to make it easy to start using all this juicy IMAP stuff. Not a demo-ui, but something that should be relatively usable or at least reusable by phone projects who’ll be using Tinymail as their E-mail client solution. This means that I’m trying to get its types and design as good as possible. You can indeed throw a three-screen read-only E-mail client together in a few minutes using Tinymail. But that’s not the idea here. If you want a sample of a quick/dirty one, check out Tinymail’s demo-ui which I mostly use for debugging Tinymail itself.
If you want me to make one for your device then well, I’m looking for a project! Perhaps we can use TMut as a starting point for this?
With GMail now supporting free IMAP, I predict E-mail is going to become increasingly popular on phones. Although rather limited in its capabilities, GMail’s IMAP server has IDLE (so it supports doing Push E-mail on phones). I hope GMail’s IMAP team will soon add support for CONDSTORE and QRESYNC.
I made the video demo that I promised last night. In this video demo I use ettercap and wireshark to show the difference in network traffic between a typical IMAP client and a Tinymail based one when we have an IMAP server that deals with QRESYNC, IDLE and CONDSTORE (like modern IMAP servers do).
Ettercap is a useful tool to follow the traffic while using the applications. Wireshark is interesting to see how frames are formed while the traffic is being transmitted. When working with GPRS, the latency can be very high. This means that with an interactive protocol like IMAP, the time to retrieve and send a frame (the roundtrip time) has a big influence on the network performance of the application.
In the video demo I explain the difference between the old way of working with an IMAP server, and the modern way with QRESYNC and CONDSTORE. My goal is to convince IMAP server developers to consider implementing both CONDSTORE and QRESYNC so that IMAP clients can make better use of the GPRS network, and of course to show that my precious little Tinymail is supporting this already, if available on the IMAP server.
I just released Tinymail‘s pre-release 0.0.3. This release contains the latest QRESYNC improvements, a lot of other bandwidth improvements, some fixes for the support for IDLE, various fixes for Maildir, merging from Camel upstream, a lot of compilation warnings when turning on -pedantic, improvements in the Gecko HTML message viewer and yet even more improvements for the Python bindings.
As usual brings this release Tinymail a bit closer to its v1.0. The stability has improved a lot and we’re again adding new features.
I’ll try to do another video demo this week. This time about QRESYNC, CONDSTORE and IDLE. Probably with ettercap and wireshark showing you how these IMAP features are saving you roundtrips and bandwidth and why they are more ideal when you’re working with GPRS.
While trying to get QRESYNC working I found a few problems in Tinymail‘s IDLE handling, specifically in its handling of (unsolicited) FETCH responses.
I used to believe some IMAP servers simply ‘forget’ to send certain FETCH responses, but while testing with MBox I noticed that I was receiving all of them correctly. It turned out that I was simply not parsing the FETCH response correctly as the sequence ended up being invalid. Once that was fixed, I found that, would it have been correct, the index was not subtracted with one. IMAP sends sequence numbers starting at one, whereas an array in C obviously starts at zero. If it had worked, it would have marked locally the wrong E-mail as seen or unseen. Ugh.
As a result, the flag changes were not instantly visible. Only notifications about expunges and new messages were really working. This has been fixed of course. This means that you can now start a Tinymail based E-mail client, continue working in another E-mail client, and with the right IMAP server the Tinymail one gets instantly updated about the state changes of messages.
With NOTIFY, the same granularity of change notifications can be achieved for folders. For example, their subscription state, renames, deletions, creations and status changes like the unread and the total counts of each folder. With CONTEXT we can also provide in the future a vfolder that is completely stored at the server. As usual, if you are interested in having this: either fund me or help me implementing it.
An IMAP draft that I really like is CONVERT. Imagine having in a message a document attached in a funny format like Microsoft Word. Do we expect that each and every phone deploys OpenOffice.org just to parse and convert all that data to something that can be viewed? Yeah yeah, the AbiWord guys now want to tell me they have something for phones too, but I can redo the sample with any other existing document format.
Why not putting that AbiWord code in the plugin of an IMAP server in stead? Why consume more bandwidth if perhaps the server can convert it to a smaller format that is immediately suitable for the phone’s display? What about converting large image attachments to the exact size of the display?
Oh, support for QRSYNC is also committed. I will most likely build pre-release 0.0.3 this weekend. I know it has taken me longer than the promised three weeks. The problem was that the amount of bugfixes has decreased drastically. The Modest developers keep telling me that everything is stable and they asked me not to change too much anymore. But now the amount of fixes and even new features is again worthwhile to do a new release.
I wanted to be the first in something, so I just implemented support for the QRESYNC capability in Tinymail. Of course has Dave implemented this a few weeks ago in Polymer too (aaarrg!!). As usual to have an experimental E-mail client to test the capability with their Isode IMAP server.
Anyway, this makes Tinymail implement IDLE, BINARY, CONDSTORE and QRESYNC. These are the ones related to Lemonade.
Next on my list is of course NOTIFY, and pick that over IDLE where possible. NOTIFY is interesting because it gives more information like when a mailbox (a folder) gets created and when it gets deleted. It also tells me about new messages in folders that are not currently selected. In contrast with that is IDLE rather limited.
I still need to test this support for QRESYNC a bit, so it’s not yet committed in trunk. The patch is available on the mailinglist if you want to help testing (and if you can find an IMAP server that already offers the QRESYNC capability). I obviously hope more IMAP server developers will implement QRESYNC. I think it’s a better approach than CONDSTORE. I’ll explain.
QRESYNC is interesting because it avoids having to do roundtrips to synchronize your current folder. Roundtrips are far more expensive (in terms of speed) than most people think while they are using GPRS. The nice thing about not doing roundtrips is that often the combined and atomic reply just fits in one tcp/ip packet. So with QRESYNC you get very fast and cheap folder synchronizations over GPRS.
That’s exactly what we want on our phones, right?
Regretfully few IMAP servers in the field have this already (if any, QRESYNC is still in draft state). But especially if you provide your users with an E-mail service combined with the mobile device that you sold them, you can give them this little bit of extra that many other far more expensive solutions don’t have. It’s yet another reason why to go for Tinymail and IMAP4 Lemonade. You can trust me on it that I’ll create more reasons like this in future.
Ubuntu Gutsy has Tinymail‘s packages, I think this is pre-release 0.0.2, I will most likely release pre-release 0.0.3 today and there are a few major things that got fixed in trunk. Nevertheless it’s good to see distributions picking it up!
Desrt asks, Philip does. So I changed my blog’s system authentication system to use reCaptcha.
I already E-mailed the author of the Maildir standard about this, but I didn’t get a reply. Other people who asked the same question told me they got the same reaction: none. Of course I don’t know for sure whether they actually tried.
Nowadays people use FAT32 a lot for storing their data on flash disk and USB sticks. They buy these at a computer shop or supermarket, plug it in their mobile device and they expect it to just work. They don’t want us to require them to reformat the device’s partition. Quite often do these people already have actual (thus, important) data on those storage devices.
I know that some people live in a imaginary Utopical world where FAT32 is a piece of shit filesystem where we not only puke on it, we also ignore it and religiously hate its authors. The problem is that I live in the realistic and pragmatic world and that I ignore people who try to force me to live in their imaginary Utopical worlds. I really have no time for the religious anti Microsoft rebellions who want me to become one of their crusaders.
In stead, I try to use the time nature has given to me to do things that can actually be done. You see, trying to get Microsoft to support the ‘:’ character on FAT32 is not among those goals of mine. Although, sure, I too don’t know why they don’t support it. I too dislike the fact that they don’t support it. I too wish live was different. In fact, I think I remember that I wanted to be born as an ant with gigantic colourful butterfly wings living in a delicious and fresh cow .. shed. In stead, they created me as a normal human.
It’s not fair!
Nevertheless, this is what happened.
So I noticed that one person decided to ignore the Maildir standard and replace the vicious ‘:’ character with ‘!’. I propose to make the Maildir standard more meaningful by changing this ‘:’ character to ‘!’. This way we can allow our users to put the E-mail data on standard USB sticks, flash cards and other media that comes with the evil, satanic, bad, ugly and patented FAT32 partition type.
I know people will tell me to use a different format for FAT32 partitions. Well, Maildir is not really bad for slow storages and it gives you easy atomic operations: rename() is, as far as I know, atomic on all (and if not most) platforms. The kernel is good at doing its own indexing for filesystems, so it’s not slow either. The idea behind Maildir is not a bad one. Most of the other local formats are kinda broken in different ways too. I could create my own standard, of course, but then I’ll get that other group of anti NIH religious rebellions who’ll start armed protest marches in front of my house.
My little Tinymail is becoming like a small fire that I just need to keep burning. Various people are contributing both large parts and small improvements to it. We’re releasing pre-releases and those are working increasingly well. I received less requests for improvements based on the pre-releases as expected. It seems most people are still using ‘trunk’ to learn about Tinymail. At GUADEC a lot of people told me that if there would be just a release, they would start using it. Well, now there are releases (although, pre-releases), yet I’m seeing no significant differences here. The ‘release often, release quickly’ mantra is not always true, it seems. I rather think that if a project is useful, it’ll be used. If not, it wont. Full stop.
To increase users of Tinymail I think that in stead of rushing to make a clean release I rather have to focus on supporting it on SymbianOS and WinCE, showing that you can build a client with it (which is what Modest is doing, and more likewise clients will follow soon), support higher programming languages and models and having good documentation. In other words: adding value. In the end is “a release” just a wrapper for that value in my opinion. Although I believe that for packagers and distributions “a release” by itself add a lot of value.
I already know what I have to do to create more site visitors: making fancy looking video demos, being part of GMAE and its press releases, getting my blog aggregated on the planets, being mentioned by Lortie (+4000 unique visitors that day!), providing easily browsable sourcecode (you’d be surprised to know how many people seems to be just reading Tinymail’s code via the trac source code browser daily) and again having good documentation (increases the hits coming from search engines). Just a lot of visitors is not really what I’m seeking for as the project maintainer. Rather people who’ll either contribute, learn from the code (the browsable-code visitors) or people who’ll want to use it for their own projects.
To get more contributors I try to get as much companies involved as possible. The major contributors to Tinymail (other than me, of course) so far have been Codethink, Nokia, Openismus and Igalia. Let’s try to get some more such organizations interested. I would love to see more individuals like contractors, students and hobbyists getting involved. I just have no idea how to get them to be interested. I think the problem is that creating an E-mail client is usually a project for a small or medium sized team of people. Although with Tinymail’s API available in higher programming languages will this be doable as a one person activity. Or, that’s the idea.
Jono thinks the Perl script of this site did what its description tells you it did: making a top list of Linux related bloggers. What it really does is detecting what programming language those people would like to use.
A blogger and now Slashdot too (oh dear, now we’re doomed) are reporting that Microsoft is releasing the source code of their .NET libraries under a not-really opensource license.
Although only readable for people who accept a license agreement that forbids editing, copying, and rebuilding, I think it’s a step in the right direction. I hope the company will learn that making things truly opensource does not necessarily mean that there will be endless amounts of forks and support problems. I hope to see more like this. I’m among the opensource developers who’d love to see Microsoft join the love.
I’m certain that there are a lot of technically brilliant people working for Microsoft who’d like to see that happen too. At the end of the day, their (technical) goals are our (technical) goals. We just sometimes differ in opinion on how we want to get there. Typical for humans.
I know there are a lot of people who want to believe that the majority of opensource developers do what they do just because it’s fun to hate Microsoft. Regretfully for the people who want to believe that, that’s definitely not the case for the majority of us. The principles of opensource are older than Microsoft, so it’s also not the reason why it exists.
