Author: admin

Mark Doffman wrote a roadmap for the Python bindings of Tinymail. He just committed the possibility to implement Tinymail’s types in Python. Although I haven’t tested this yet, it looks like he’s making the bindings kick ass.

I guess this means that good Python bindings will be a feature of the first version of Tinymail. Thanks a lot Mark!

IMAP has the STATUS command which can give you information like how many of the messages in a folder are not flagged with \Seen. That’s cute, but not as useful as it might sound to most people. This is the problem:

What I want to avoid on a mobile device is having to do a synchronization of a folder. A synchronization the first time means retrieving the summary information like the Uid, the flags and some headers. What use is all this information if this is a folder that will never be viewed by the user of a mobile device? Why waste bandwidth on retrieving this, especially when bandwidth over GPRS is expensive for many people?

How will you know (before said synchronization) whether a message is unread but not deleted and how many of that unseen count, that you got from STATUS, are like this? It’s perfectly possible for a message to be marked both deleted (but not yet expunged) and unread.

“Simple”, I imagining hearing somebody in the audience say: “just mark all messages that you delete as \Seen!”. The problem here is that a mobile E-mail client is most of the times not the primary E-mail client of a user and I can’t control the behaviour of the many E-mail clients being used in the field.

The nice people at Isode specified ESEARCH for things like this. To all developers of IMAP servers: please hurry up implementing this one. With ESEARCH I can get an accurate unread count suitable for mobile devices without having to do a folder synchronization for each and every folder.

As some people at GUADEC asked me to continue the technical stories about Tinymail, here’s a preview on how it goes:

When you open a local E-mail folder that is stored on a MMC card, you obviously need to open some resources. Those include, for example, the summary’s mmapable file. You might also need to opendir() the directory and perhaps even chdir() to it? Perhaps rename items from a “new” or “tmp” to a “cur” subdirectory? In the process you might have left a directory or two open?

Suddenly … you get bug reports from system testers telling you that if the E-mail client is running, they can’t unmount the MMC card.

Today, Tinymail’s implementation for a few local E-mail folders makes sure that no directory related resources are left open, chdir() always goes to the root folder and the mapped files are closed as soon as the last component stopped using the folder.

Mark Doffman wants to maintain and fixed the Python bindings of Tinymail. He committed a huge patch restructuring some things and making things work again. I guess this means that very soon and perhaps already you can write your own cute little E-mail client in Python?

I know that for example one person plans to make a mutt-like using Tinymail. Maybe that one can be done in Python? I think the Python bindings will be most interesting for the OLPC project, though.

Vala bindings will of course be made very soon too. Though, my current plan for Tinymail 2.0 is to rewrite all interfaces and refactor a few of them to abstract classes … in Vala. I hope that by that time Vala will generate a bunch of bindings to other languages. If I look at the progress that the authors of this programming language are making recently, I’m confident it will.

I just noticed that it has been since May that I did a video demo. I’ll try to make something soon. What about Tinymail on the Openmoko phone?

For most of the organizations that needed it, it comes a few years to late (when i started Tinymail, they wanted it yesterday). I promised to start the process of making early releases the first time one full year ago. I completely underestimated it, indeed.

Finally, a first pre-release of Tinymail. I hear people think “What? Just a pre-release? Not even a final release?!”. The pre-releases are to aid people like the packagers with preparing their environments and to give a solid basis for application developers who want to start focusing on an API that will not change a lot anymore.

It also means that everything involved with releasing a final will now start. You can find the announcement with all the information on the mailing list.

ps. I have no idea when or if a pre-release of Modest will be made. I have no idea when or if packages will be made available.

I have been reading “The Transparent Society” by David Brin and a .NET book “Framework Design Guidelines” by Brad Abrams and Krzysztof Cwalina. On page 10 of the former book I found something interesting:

“In all of history, we have found just one cure for error — a partial antidote against making and repeating grand, foolish mistakes, a remedy against self-deception, That antidote is criticism. Scientists have known this for a long time. It is the keystone of their success. A scientific theory gains respect only by surviving repeated attempts to demolish it. (page 10)”

I have been closely following the criticism on Tinymail. I indeed noticed that there has been certain difficulties. The book continues with:

“Only after platoons of clever critics have striven to come up with refuting evidence, forcing changes, do a few hypotheses eventually graduate from mere theories to accepted models of the world. (page 11)”

This is why I’m a critic myself. This is what makes me want to listen to criticism for my own things. Because criticism is a partial antidote against making and repeating grand, foolish mistakes. Against self-deception.

My plan is simple: I’ll take the criticism, I’ll learn from it. I will educate myself using the .NET book and I will improve both my own skills and the framework.

I like the .NET book because of an interesting quote:

“Eight out of eight failed to read text from a file in the 30 minutes we allocated for the task. We believe this was due in part to problems with the documentation search engine and insufficient sample coverage; however, it is clear that the API itself had several usability problems. (page 17 and 18)”.

They explain that they designed the .NET API scenario driven. This means that they wrote the code of a scenario and only then constructed the classes and interfaces.

I think this is an important lesson in framework design: do usability studies on your API and you’ll learn that with your own frameworks eight out of eight will fail while performing the simplest tasks. The authors of the .NET book write that their goal was to provide a pit of success for their customers, not a pit of failure. I want the same for Tinymail: to have a pit of success for the application developers.

ps. The former book is not about a technical subject like designing a framework of course (I know, you don’t have to tell me). Rather about privacy vs. technology. The connection between the two subjects is only about these quotes: I’m reading both books at the same time.

Murray’s blog spurred some controversy about Evolution and the adapted Camel that Tinymail internally uses.

Having done most of the adjustments I’d like to point out that some of the changes are indeed focused on mobile use. Some of the changes alter the IMAP communication in such a way that less bandwidth is consumed. Examples are omitting the references header while downloading so-called summary information per message and the support for CONDSTORE.

CONDSTORE is a recent enhancement to the IMAP protocol that makes it possible to ask the IMAP servers for just the differences since a certain known state. Rather than asking for all the flags of an item, comparing that with your local version and synchronizing the two, you ask for only what has changed and you synchronize the recent local changes remotely.

I removed the support for the references header in Tinymail because this was poorly implemented in my opinion: a doubly linked list was used for each reference. This as you know consumes eight extra bytes per reference for the pointers (for the purists: on a 32bit computer). I think just storing the string and using strtok would be better here. The current E-mail clients being developed using Tinymail, however, don’t need the threading granularity of a desktop E-mail client.

Most modern E-mail clients are also less and less utilising this feature the way most experienced E-mail users know it: usually E-mails are grouped together based on relevance (flagging by a human and determining this with an algorithm) or on indeed the discussion thread. Full threading is often overkill: most people just want the messages to be grouped together somehow (in a flat list per group, not really the actual tree). Mind that most mobile devices have rather limited display capabilities and that a tree wastes a lot of whitespace this way. What is the use of message threading if you can only display at most five rows of E-mail summary info?

This “references information” must be downloaded and GPRS sometimes costs the end-user money per downloaded megabyte. Those reasons combined made me decide to simply not request this information. Later I might make the projection of the actual query definable by the application developer who builds on top of Tinymail.

My conclusion? Certain things can most likely be backported to Evolution. Other things might just not be interesting. Cooperating is definitely more interesting than not doing so. Right now is an internal highly adapted copy of Camel the pragmatic right solution for Tinymail.

I promised my precious Tinne that I was going to take two weeks of vacation. I’ll start with the Belgian rock festival Pukkelpop and after that, two weeks of holiday together with Tinne.

I promised a release candidate of Tinymail last week so I guess I will still be doing that in the evenings. Other than that I’m planning not to work on anything for the first time in .. already a few years I think (all the holidays that I took at my former employer have been going to Tinymail development). I kinda forgot how holidays feel. I’m even a little bit scared of getting bored.

Well, no. I can’t get bored! I have a bunch of things to read, A girlfriend who’ll finally get some attention and a release to make. It’s not much to do, but it’s something.

I’m at the point of (naively) believing that Tinymail‘s thread difficulties are all solved. The developers working on Modest seem to be telling the same thing (today’s quotes from the developers: “Modest is quite robust today”, “yes!!! it works”). Whoo!

The POP code had some difficulties too, mostly related to deleted and deleting messages. These have all been fixed too. We have bugs like a translation bug about the SSL-question text: it’s not yet translated to your locale. That type of bugs. Still too much such bugs, in my opinion. An E-mail client has quite a lot of functions to support, if you just want something very basic (you’d be amazed). We’re getting there and I have this (again naive) feeling Modest is going to please most people.

Next week will be an interesting one as I’m planning to do a release candidate of Tinymail. That’ll be an actual tarball and a branch tag in a “releases” sub directory of Tinymail’s repository. We’ll make sure Modest’s repository will be synchronized and tagged with this RC. I’ll make sure all the build information will be available. Chances are high that we’ll put some packages for N800 online too. That’s not a promise, though. Especially not for Modest (I don’t make such decisions for Modest, but I can probably prepare packages for Tinymail on N800).

I’m expecting quite a lot of bug reports. Especially if we’ll provide packages, as that will increase the amount of people who’ll try it out of course. I will make test accounts available for people who don’t yet trust Tinymail with their precious E-mail data. I will also prepare a QEmu image with an IMAP server and some test data. Although this QEmu image will only be bundled with a release officially when I’ll do the final onepointzero release.

It has been a year and a half of hard work, especially last few months, but I think it’s getting ready now for beta phase, yes.

If people want to help with all the “make distcheck” work, let me know. I’m not very optimistic that a lot people will, doing a release is really boring work. That’s because, and I repeat, all our typical development tools (autotools, gtk-doc, etc) suck and are absolutely not synchronized with how developers really work.

Jürg Billeter finally has his blog setup. He finished the first building blocks of a DBUS binding for Vala and made a sample. Go check it out!

Vala

A lot of people are visiting #vala on GimpNET to tell Jürg Billeter how much they like Vala. One such person already finished a Vala binding for GtkMozEmbed, another is now working on bindings for Clutter.

This is very cool indeed. Soon we’ll have solutions in place for the vast majority of our libraries. I hope our community realizes that the emerge of a new programming language can only succeed if the creators get some help with implementing required instrumentation.

There’s for example a challenge in supporting .vala files in a clean way in autotools and Makefile.am. Although the majority of people dislike autotools & co., it’s used by nearly all of GNOME’s libraries. To make things work smoothly is a good integration needed. The difficulty is that a Vala compiler, unlike a C compiler for typical C projects, needs a list of all Vala files to start generating the .c files. For Python they needed something similar using a helping script called py-compile. Java can cope with a file per class, as there’s a file-name convention.

Vala is missing documentation, a website and there’s a lot of low hanging fruit that can very easily be picked up by new people.

As for Tinymail, me and Jürg are planning to replace the TnyList and TnyIterator interfaces with the ones in libgee sooner or later (probably later, as I’d like to release a one point zero of Tinymail before that). I heard the Anjuta folks are interested in replacing their “from IDL generated interfaces” with Vala-ones too. I might just bite the bullet and rewrite all Tinymail’s interfaces in Vala. As soon as Jürg has the binding generation for Vala in place, that would mean I would get the generation of language bindings for free. Right?

This illustrates why I think Vala is such a good idea. Especially for GObject based libraries that have interfaces for their API.

Tinymail

Today I finished the work on Tinymail to make it completely “Gdk lock” aware. This means that Tinymail’s signals, observer update invocations and callbacks are all “Gdk lock” aware. In practise this means that you, as application developer, don’t need to acquire the Gdk lock yourself. For your own timeout and idle callbacks and for your own threads calling Gtk+ or Gdk subsystems you of course still need to do this.

This work will be committed tomorrow. I don’t know when Modest’s public repository will have the required changes for this (some people are on vacation, etc etc).

Documentation still has the top priority at the Tinymail project, so some new documentation items got added. Mostly about how Tinymail works internally related to the recent improvements. I have asynchronous operations, asynchronous connecting and dealing with the Gdk lock for you.

These recent improvements combined together should make Tinymail more robust against threading problems. There’s locking order, queuing and guaranteed handling of the Gdk lock in signals, observer update invokes and callbacks. The queue waits for the return of the callbacks made by the application developer, avoiding that the application developer can launch a method in that callback that would lock with the original call that invoked the callback. Lot’s of improvements like that.

All these things have delayed the release candidate that I promised to some people at GUADEC this year. I also want to let the improvements stabilize a little bit. On top of that there’s quite a lot of work cleaning certain things up. I would also like to create a QEmu image with an IMAP server and the Tinymail demoui that I’ll release in parallel. These are certainly things people can help me with, if they want me to speedup a release for Tinymail.

The make distcheck should be again working, some of the unit tests need reviewing, the !@#*@* gtk-doc scripts need to be fixed .. again, the !@#*@* Python bindings are broken .. again, the enum and flags GTypes are synchronized once more (although some don’t have documentation in gtk-doc yet), etc. I guess all these are the typical 60% of work a project maintainer has to do. A lot of work is indeed because all our development tools are completely broken and not integrated with how people really work.

Gdk Lock aware?

Right, some people have reviewed that wiki page that I mentioned yesterday. Hoping more people will do reviews.

IRC

If you can’t join a GNOME or Gtk+ channel on GimpNET because it’s invite only: /msg dorito invite #channel. The ops had to set some channels to invite-only because of spam bots.

I’ll try to make it more of a habit to write about Tinymail‘s future enhancements before I commit them. This gives people who are following the project the opportunity to comment on and maybe improve the actual work.

In the good old current days Tinymail’s callbacks for asynchronous operations are guaranteed to happen in the GMainLoop of your application. Tinymail uses a g_idle_add_full and ensures that the operation itself happens in a queue that runs a single thread, and a GCond that ensures that the queue will wait for the callback to happen before entering the next item on the queue.

That’s not enough because a g_idle_add_full does not enter the GDK lock. In near future (the patch is ready), Tinymail will do this too. This will make all of Tinymail’s callback invokes behave like signal handlers of Gtk+ components.

Tinymail’s observer update functions will do the same thing. Although I know you can use glib’s signals for this, I used the original observer pattern for some situations. Usually for situations where it must be clear for the application developer what he’s doing. Just like signal emissions must these update invokes or notifies happen with the GDK lock locked. Next on my list is going after all signal emissions that happen outside of the GDK lock.

The thing with this GDK lock is that it’s completely not well documented, that nearly nobody in the GNOME community fully understands how it works (except maybe people like Tim Janik), how you need to use it and more importantly when you need to do it. It’s at this moment definitely not (yet) clear at all. I started this wiki page in the hopes that our experienced developers will continue improving it. This way we might someday have a real and full description of how to actually correctly use gdk_threads_enter and gdk_threads_leave.

Two or three months ago Krzysztof Cwalina made one of his Microsoft Research presentations available. It’s a presentation of around four hours about how to design framework APIs.

He talks about quite a lot of subjects that I think the community of people working on future GNOME related frameworks and infrastructure should carefully listen to.

I’m one of those software developers who have very much respect for those people that Microsoft employs who are passionate and knowledgeable. I truly believe a lot of the technical people at Microsoft do want to make things better. This makes it interesting to listen to their thoughts and ideas, to be open minded about their research.

If you have Windows Media Player you can watch the presentation here. You can download this presentation for offline view here. It contains a .asf file in one of the folders. You can probably view it using one of those media players that we have. If not, in stead of ignoring the content I advise you to use your mother’s Windows computer to nevertheless watch the presentation. You’ll suffer from having to use Windows and you’ll suffer from having to watch four hours of presentation material, but I promise you that it wont kill you and that the presentation itself is quite interesting.

I know that this blog item misses real argumentation on why you should view it and waste four hours on this. Well, just do. There you have it.

Due to naively designing the internal organs of Tinymail’s async methods, I introduced a lot of threading and locking difficulties to the application developer who’s building an application on top of Tinymail.

We noticed this and therefore we did something about it (we is Rob Taylor and me). The solution was to introduce an internal queue that guarantees locking order for operations and during the connecting of accounts. The callback, which the application developer can implement, will make such a queue item waits for its completion. They happen in the mainloop, so that is done using a small little “GCond” construction. Dialogs like the password dialog, the SSL question and the code that gets launched in case the password was wrong will now all happen in the mainloop of the application. This makes it much more easy for the application developer to implement them using user interface components like the GNOME password manager’s dialogs.

This work was being done in this branch of Tinymail and will, with the migration of an E-mail client that is being build on top of Tinymail, be merged back to Tinymail’s trunk starting tomorrow. Some of the work has already been merged in trunk. Mostly work that was not related to the account connecting and initialization (which includes the password -and the SSL question things).

Hereby apologies for making these horrible internal design decisions. I guess I focused at the start of the project too much on the external design. I guess I hoped too much that internal organs would automatically become good too. This kludge made the external API behave difficult for application developers. It’s fixed now, and we wont make the same mistakes again. I believe this work is one of the key pieces in the puzzle that can turn Tinymail from a hobby project into an actual product.

The good thing about the kludge is that it forced me to make a lot of the locking both very fine grained and yet very correct. With the locking order in place now, this should mean that the overall quality should have improved a lot. We’ll see. Let the testers begin!

Another key piece in the puzzle is to someday have a libtinymail-dbus. We’re already planning and designing this, indeed.