Month: February 2006

The slides and a document with the text that I planned to say per slide are now online.

I got a little bit sick this weekend. Nevertheless managed to do my presentation at FOSDEM. I decided to stay in bed on Sunday, which is why I wasn’t at FOSDEM yesterday. Sorry if you wanted to talk to me and hoped to catch me on Sunday.

I was a little bit afraid a lot people where going to tell me how much they dislike the “buzzword” “design pattern”. I guess I shouldn’t have been afraid as i.m.h.o. a lot people where interested in my talk and a lot (interesting) questions where asked. Regretfully I was a little bit sick so I might have been answering those questions like an uninterested asshole. Sorry if that was the case. And I hope I didn’t make somebody else in the GNOME room sick.

If I did (I had stomach flu), visit GUADEC at Spain this year, I’ll buy you some free beer at the meeting. Don’t worry about me, I feel a lot better today.

I’ll put the slides online soon. You can find the samples here, here and here.

wow.

Yesterday night I worked on the gtk-doc API documentation of libtinymail, libtinymailui, libtinymailui-gtk, libtinymail-gnomevfs and libtinymail-camel. You can find the API documentation here and a (little bit outdated) class diagram here.

Mono and NHibernate

A friend of mine, Bjorn Monnens, did a little project using Mono, Glade-Sharp and NHibernate. Very interesting

Tinymail

The new default gtk+ summary-window implementation of tinymail now has paned widgets and embeds a msg-view:

Some early dude using tinymail on his Zaurus:

Tinymail can now not only show attachments, it can also store them on a VFS destination.

But because GnomeVFS might be something that’s difficult to support on a mobile platform, I decided to go further than usual with the concept of shared libraries:

• First I decoupled libtinymailui from it’s Gtk+ dependency and created a libtinymailui-gtk that implements the interfaces defined in libtinymailui. I did this in case somebody decides to write another user interface implementation. For example an ncurses or a qt one.
• Then I created libtinymail-gnomevfs which at this moment only implements a VfsStream. I reused most of its code from EmVFSStream which you can find in Evolution. It’s stuff like this that illustrates my opinion about that we should split Evolution in multiple reusable libraries. Not all Evolution code is bad or bloated, its good pieces just aren’t very reusable at this moment. Note that this library can be implemented differently (for example by not using gnomevfs but something more suitable for your mobile platform). Look at the source code, it would be extremely easy to do that.
• Then I adjusted tinymails MsgWindow Gtk+ implementation in such a way that it gets the selected MsgMimePart from the IconView and writes it to the VfsStream that gets created using the uri that the filechooser dialog returned.

Somebody on IRC asked me to add the possibility to create plugins in Python. I might sooner or later introduce a plugin system. I already decided to use the strategy design pattern for the plugin system. Once such infrastructure is in place, I might create Python plugin infrastructure using the decorator design pattern. I’ll create a plugin that decorates a normal plugin type.

But I’ll first try to finish tinymails basic functionality a bit. Still on the todo-list is of course the possibility to create e-mails and to send them using an SMTP service. And implementing a GeckoStream like TextBufferStream so that I can write HTML-mails to a Gecko component. Perhaps also a stream that can write to an Opera HTML widget.

Today tinymail not only uses less than 4 MB RAM for viewing large e-mail folders, it’s also becoming blazing fast and it’s getting the minimum features a small e-mail client needs. The big missing feature for being alpha-release-ready is creating e-mails (and sending using a transport camel provider).

After some attempts to remove the allocation peek, I squeezed it with a megabyte by more quickly unreferencing the camelfolder instances. Peeks aren’t interesting because they require the hardware builder to nevertheless add memory to the device. It’s good that memory doesn’t stay allocated, but simply not needing the memory yet being fast enough is of course better. I’m never satisfied! Now tinymail is actually faster yet its folder-load allocation peek is lower. Win-win.

Some guys on IRC want me to build an ncurses version of tinymail, using libtinymail. They are, of course, invited to start. I’d certainly join. The libtinymail and libtinymail-camel libraries don’t depend on any GUI thing.

API boutification

• Refactored some things from the application implementation to the libtinymailui.
• The header-list model no longer has an add_header nor inject_headers method. Now it simply has a set_folder method.

Performance tweaking

• Header-list model: Caching the total length of a folder replaces a g_list_length on the header glist.
• Folder type: Caching the priv pointer of the folder type in the loop over each message header, makes the slower gobject get_private stuff happen less often.
• Folder and message header types: Not duplicating the uid of the messages, but keeping the gptrarray until the folder must finalize itself, letting the message header instances have their uid point to the string instances in the gptrarray.
• Message header type: Removed the priv pointer of the header proxy type. Made the entire type opaque (forward typedef). This makes the slower gobject get_private stuff not happen for this type anymore.

New features

• Completed the mime-part type. It now as properties like filename, mime-type and others.
• Added a attachment viewer in the message window, added an attachment list model type that will set the mime-type icon correctly. Since that uses a tiny gnome api I had to let libtinymailui depend on libgnomeui-2 (but this can very easily be changed of course)

Hey fejj, that poor-man’s version (the proxy technique) is one of the very few possibilities for reducing memory usage when using camel outside of Evolution. As you know is the (current) focus of tinymail to run on mobile devices. Implementing a “poor man’s” solution is IMHO better then depending on a forked and modified camel.

On IRC I did confess (to you, personally, on #gnome-hackers) that vfolders would, indeed, be more difficult to implement in a mail client like tinymail because of the technique. But not impossible (the technique might be useless because, indeed, it needs a reference to the real camelfolder instance for each folder that is used in such a vfolder — but only for such folders).

But believe me that probably a lot people don’t care about those vfolders.

For the mobile-device case: nobody will ask for vfolders on a mobile device if such a feature means paying eight times the price of a typical mobile device because viewing their mail suddenly needs hundreds of megabytes of ram memory.

Perhaps should it be made possible to turn off the vfolder feature in Evolution until Novell decides to finish the disksummary branch? People who aren’t using it, are wasting serious amounts of memory while using Evolution. By the way., I very recently contacted Nat about this disksummary branch. My opinion is that it should be finished. And I’ve already offered my help (in my freetime, I can’t spend my daytime job-hours on it).

But if Novell doesn’t start working on it: I’m not going to do it on my own. So I have to implement what you call “poor-man’s” solutions. It’s the only solution that works, at this moment. If it’s a “poor-man’s” solution, I guess Novell needs to start working on the disksummary branch. Right? (I know some guys of the Evolution team are planning to do that. I’m very interested in that, by the way).

Nevertheless, my prove of concept was that a mail client “can” show large folders without using a lot ram memory using camel. Perhaps I shouldn’t have compared it with Evolution as Evolution does, indeed, have some interesting features that do require memory. But my opinion is that 60 megabytes of ram, is just to much. Way to much. VFolders aren’t a good excuse IMHO.

I’m not the typical asshole that only whines about Evolution. I care about the product and I’m offering my help. But it looks like people are afraid of finishing that disksummary branch.

I told some people that I “will” prove that an e-mail client can load huge folders using very few memory. So I will prove it now.

Today tinymail can show the folders and subfolders of multiple accounts, it can show the message headers in a summary view and it can show the content of an E-mail picked from that view.

I did a session where I launch both evolution and tinymail from a cold start using the massif tool of valgrind. Both Evolution and Tinymail showed all the folders. Both evolution and tinymail gave a view of the message headers in the folder. Both evolution and tinymail viewed the last e-mail that appeared in the list. Using both evolution and tinymail I scrolled to the last message header and back to the first.

I do understand comparing tinymail with evolution might sound unfair. I do think a lot of the by evolution allocated memory is not just the non-trivial user interface stuff. In fact, by turning off my proxy design pattern tricks, I get surprisingly very similar memory usage results with tinymail.

Evolution used 60 megabytes of ram. Tinymail used 3 – 6 megabytes.

Both evolution and tinymail used camel. But that doesn’t mean that camel doesn’t need improvements. It certainly does.

At stage one the folders are being loaded. At stage two the (visible) message headers are loaded. And at stage three I was scrolling the message header summary view’s scrollbar down and up a bit. The tree stages where repeated on both tinymail and evolution.

Note that I’m uncertain about the three marked points on the evolution graph.

I am, however, certain about these three marked points (I did multiple valgrind measurements and disabled some steps to watch the memory status at known points in time).

I kindly invite people to reproduce these measurements. You can get a checkout of tinymail here. Use “valgrind –tool=massif tinymail or evolution”. Read this piece of source code to know how to create an account.

(I think) The biggest reason why evolution uses a lot memory is because of the CamelFolder instances that get delivered by camel_store_get_folder. I created a treemodel that uses proxy instances for this CamelFolder type. I made sure the real CamelFolder instance gets freed when it’s not used anymore.

For 50 folders that reduced the total amount of memory with ten megabytes. You can see that allocation peek in the valgrind massif report of tinymail. You can’t see it as clear in the evolution report because, well, it simply never gets freed. If you don’t load the folders in tinymail recursively (for example only the INBOX folder), the peek goes away. This proves that it’s indeed the allocation of all the CamelFolder instances (in fact I ironed that out to the very call to camel_store_get_folder, as I wanted to make sure it wasn’t my own code causing these allocations). I think a lot of evolutions memory goes into those CamelFolder instances.

So I was wrong when I said that it’s the camel_folder_get_uids function. As a solution, evolution could create a proxy class for CamelFolder and attempt to camel_object_unref the real folder instance as soon as possible. For a sample, take a look at how tinymail does it.

I created this tinymail screencast demo using Byzanz.

I tried showing how tinymail actively tries to deallocate message headers as soon as possible. Regretfully isn’t the disksummary branch of !z’s camel finished.

That branch would also make it possible to keep the list of message id’s on disk. At this moment, that list has to be loaded in memory. If you have a mailfolder with 10.000 e-mails, that’s 10.000 strings in your memory. Which isn’t a good idea on a mobile device.

All I can say is: !z, we need you! I hope Ajay Parthasarathi Susarla and Shreyas Srinivasan are going to work on this branch a little bit. Get it stable etcetera. I’m very likely going to join the fun of camel hacking sooner or later. This branch would, of course, greatly reduce the memory usage of Evolution for people using large mail folders.

Read more about the design decisions of tinymail here. A per definition always outdated class diagram is available here and the code here.

Tinymail is going very well. I’m making progress really fast.

I wrote some explanation about tinymail. For example about why I picked this development model and why I’m so keen on using interfaces a lot. Because it’s rather long, I put it on a page instead of in this blog entry.

Have fun.

This has quite amazing amounts of stop energy.

Please make it stop!

If people want to develop crazy things and explore new ideas, please let them. If you don’t agree, please shut up or prove the guy wrong using code.

I don’t care whether or not it’s crack. The very fact that somebody is exploring alternatives and putting his time in it, makes it already interesting for at least one person. If he wants others to enjoy his work, it’s his right to do so. It doesn’t mean that somebody has to agree. And Joe, it certainly doesn’t mean that somebody has to insult him.

If fact, you and Ross are in my opinion scaring away potential developers who’d like to join our free software community.

I just added a quick class diagram that describes the layout of libtinymail, libtinymail-camel and libtinymailui.

Not yet related, but might become related, is this tryout that I’ve been working on a few hours ago. I created for example a GIterIface which is an interface for an iterator. I implemented, as an example, a completely useless glistiter (and a demo of course). Actually it’s not useless as the iterator pattern abstracts iterating over list-types. So the iterator makes it possible to iterate over multiple types of iterateable list-types. I’m planning to, for example, also create a ghashtableiter. Using the exact same interface you could then iterate both a ghashtable and a glist while it wouldn’t matter for the consuming component what the list-type originally was. This might sound totally useless if you haven’t used the iterator pattern before. Most Java and .NET programmers can probably tell you why this is important. It often makes iterating slower yadiyada, but for a lot beginner-programmers it’s nice as they will be forced to correctly use the list-type (the iterator implementation for a specific list-type is typically implemented by an experienced developer).

There’s a problem with the current comparable type in gtk+: the gtktreeiter. GtkTreeIter is a type by itself, it’s not implemented as an implementation of an interface that can have a different implementation. This makes it more difficult to create gtktreemodel implementations with a specific iterator for a specific list-type. Or it makes it more difficult to reuse that iterator implemenation for other parts of your project.

This could be, in my humble opinion, one of the reasons why it might be complex and/or frightening to implement a custom tree model. Something that a lot developers should do instead of using the standard gtktree -and list store.

Other things I’m trying out: a GBindableIface, a GModelIface and a GListModelIface. All these things need a lot more brain-time. It’s something I’m cooking. I know I’m being stupid by already showing this. I know I will very likely get flamed about this. That’s okay, I’m getting used to that anyway (xdg-list). Danw is right.

Anyway. The idea of this concept is to bind data to components. For example make gtktextbuffer implement gbindableiface and then g_bindable_iface_bind (buffer, datasource). Make it possible to define a gbindablebehaviour and assign it to the gbindableiface. Something like g_bindable_iface_set_bind_behaviour (buffer, gda_datasource_to_string_bindbehaviour_new()) and datasource = gda_datasource_new(). Anyway. For this to work on a gtktreemodel, I need an iterator which I can instruct how to function when calling its next method. With the current gtktreeiter I can’t do that.

You can tryout this demo which implements a datasource that reads from a file on the filesystem. Just a silly example.

Also make sure you checkout this thingy. It makes it possible to synchronize gobject properties.

As you can see, I’m sickening glib and gobject with sick evil interfaces. And I love doing that! I guess I should be burned and severely punished. You can do that, visit FOSDEM.

Tinymail now shows both the folders and the headers for each folder you select. Gaaah, I know you guys don’t believe me if I don’t post a stupid cute screenshot. So here you go:

I immediately picked my largest folder: my spam account INBOX folder. This is already using the custom treemodel, so it loaded extremely fast. But that is normal, as I’m only requesting the visible camel-message-info instances. I haven’t yet fully tweaked it. This is just the very first prove of concept. I will prove that you can, using camel and some very simple design patterns, write an e-mail client that has only the visible items in memory. And that a device like the Nokia 770 can, that way, easily show an extremely large e-mail folder. Once I’m really sure only the visible ones are requested, I’m planning to experiment with compressing the items in the base-directory which camel uses to on-disk-cache headers. Once you have full control over which items are requested, such tricks become useful. As it will need to decompress recently-new-visible header information, scrolling the treeview will of course become a little bit slower (I can also do it in a thread, using something like asyncworker).

If I can reduce the amount of disk-space, a 256 mb flash disk might hold all my spam E-mails headers of the last three years. Or perhaps there’s a better database-engine solution for this? Perhaps I should check out the status of that disksummary branch. I sure have plenty ideas to make camel a good candidate for mobile devices.