Category: english

Chris did some nice integration with tinymail, Dates and Contact. I, of course, accepted his patch.

After 43 bookmarks in khexedit, three afternoons of hacking and calculating memory positions that get defined by recursive loops that write a file recursively (and other crazy shit). After adding a ‘\0’ after each pstring and all content-type tokens in that summary file and making sure the lenghts of the strings are still correct so that I know the offset to the next info.

I’m finally done

Lets start small with the least impressive images. In these two images I open and closed a a folder of 658 headers a few times in tinymail. The first graph is without using my patched camel. It’s using the current Camel of my Ubuntu Dapper. I gimped two red lines on it. They illustrate the amount of memory this folder needs.

This is a graph that shows loading and unloading the same folder a few times but now using my patched Camel. By looking at the green lines you can clearly see that I still have a memory leak. The leak grows each time you open/close a folder, which is of course perfectly normal in case of a real leak. This doesn’t mean that the mmap patch *needs* this leak. I’m of course going to hunt it down.

The leak, however, isn’t very interesting. What is interesting is that the same two red lines now illustrate that I need less memory.

So now the impressive ones. First a graph that shows loading a big folder of 14558 headers using the default Ubuntu Dapper Camel library. You can see that valgrind decided to print the 10M barrier. I made red lines at 8MB and at 10MB. You’ll see the same red lines in the next graph ;-).

I guess words cannot describe this as good as the graph can. As you can see I’m saving a few megabytes. I’m guessing after some more tweaking (for example using posix_madvise), showing 14558 headers will be easy for a mobile device like the Nokia 770 and the OLPC device. I’m aiming at making it possible to display 50,000 headers on the Nokia 770 now. Seeing these valgrind graphs, I learned there’s still a lot room for improvement in my own code.

About speed. Well. I didn’t yet measure it. But I really think this is a lot faster than the fread() implementation. It probably depends a little bit on the posix_madvise hint and your kernels mmap() implementation. Sorting (on my desktop) is definitely not slow for the 658 folder when using the mmap() implementation. I was a bit afraid of that because reading the mmap() data happens at random during the sorting. Sorting the 14558 headers folder felt a little bit slower. But I really need to measure it, and tweak with posix_madvise first.

This is the patch. I don’t think it’s going to work with Evolution already, unless you remove your $HOME/.evolution directory and stuff like that. It should, however, work with tinymail. Anyway, don’t think that this stuff is ready yet. It’s not. The patch is for development purposes. It’s an experiment.

Suggestion if you want to test: install e-d-s (which contains camel) with the patch in prefix /opt/camel and do LD_LIBRARY_PATH=/opt/camel/lib before running tinymail (for example using the massif tool of valgrind).

Harish told me yesterday (on IRC) he would like this to be one of the big patches for the next Evolution release. I don’t think that would be a good idea Harish. It really needs a lot more testing and experimenting first. I’m actually surprised that I got it working.

It is absolutely in experimental state. It needs enormous amounts of refactoring work. It’s definitely a hack at this moment. It doesn’t really work yet. It’s ugly (like, I have to save things in a load method). Yadi yada.

But some of it is actually working and you can find the current patch here. It often crashes and burns and does other cool stuff like throwing keyboard keys in your eyes. In tinymail, if you do this and if you do that, you can actually already see the summary of headers in the view. I still have to do the “content_info” part, and disable a few more things in the camel providers.

Once it works, I’m interested in what tweaking the mmap addr pointer with posix_madvise is going to do when for example the sorting and loading of the summary happens. My target is to support a summary view of ~80,000 messages on a mobile device with less than 50MB ram. Cause of how the summary information is at this moment loaded, tinymail can only support something like ~15,000 messages.

Anyway, it’s Friday evening. Tinne is waiting for me! So I’m forcing myself to stop hacking on it and spend the evening doing some other stuff. Like going out.

Some people are requesting for a tinymail based E-mail client that would integrate well with the standard GNOME desktop.

There’s already a libtinymail-gnome-desktop which implements some of this desktop integration (like NetworkManager usage, Gecko HTML component and GnomeVFS support for when you want to save an attachment). Nevertheless I’d like to empathize that I don’t have a focus on developing an E-mail framework for the desktop.

However. That doesn’t mean that I’m not interested in contributions or E-mail clients on top of tinymail that would implement this. I have, however, been telling people that such an E-mail client is NOT a one person job. A team of people, in an agile development setting like Extreme Programming, would have to spend at least a few months time on it.

I very often have to repeat that I am not going to implement their new desktop E-mail client. People, please get realistic: it’s not doable if I’m the only person doing it. If you guys really want a new E-mail client, then start forming a team that will implement it. It’s as simple as it is: good code doesn’t grow biologically. Code doesn’t just happen. Code must be written by software developers. You cannot magically emerge an E-mail client from nothing, or by just talking about it. You have to implement it.

If you want to do it well, I really suggest that team would use the extreme programming development methods.

There’s going to be more on tinymail.org very soon. Like a trac and probably also a wiki. For now I moved the Subversion repository from our company one to one which I’m hosting myself.

So, if you are interested in the tinymail source code, you can do something like this:

svn checkout https://svn.tinymail.org/svn/tinymail/trunk

The original location has a RewriteRule. But I don’t think that works with the standard (current) Subversion client. But references to the old locations should automatically direct you to the right location.

• I just received the OLPC motherboard. I will definitely bring tinymail to the device. (Picture taken with a bad phone-camera)
• Tinymail is experiencing some aggressive API changes that will make generating language bindings much more easy.
• I registered tinymail.org. Contact me if you are interested in helping me building a nice website for tinymail

During the GUADEC week I had to explain a few times why you pronounce tinymail as tinniemail rather than the English taainymail. The reason is of course obvious: because that is also how you pronounce my girlfriends name, Tinne.

Maybe are some GUADIANS now curious about who this Tinne person is? After all, she’s the reason why all you guys have to pronounce tinymail as tinniemail.

So on these photos you can see miss Tinne saying “Telepathy rocks, because they picked both a name and logo that doesn’t sound nor look geeky”. Which basically translates to “I like this T-shirt, now it’s mine!”. Thanks Rob Taylor! You made my Tinne happy with it. And I gained a few more credits for being a good boyfriend. Well, actually the deal was that I could go to GUADEC, if I would bring a cute penguin teddy bear. But since those aren’t sold at GUADEC, I had to come back with these T-shirts.

Thank you Telepathy guys for being smart enough to make the logo and name girl-friendly!

It certainly looked like a lot people, companies and organizations are interested in tinymail. Among them are Nokia and OLPC. Some talks between Nokia and Collabora for making the Python language bindings of tinymail kick ass are happening. I talked with Christopher Blizzard and Jim Gettys about tinymail on the OLPC laptop. They where very interested. Especially if there’s going to be Python bindings (I guess that way they can do prototyping and experiments more easily). Nokia is of course also extremely interested. Meeting them and the GPE guys working on Modest was a great experience.

This blog entry would be multiple pages in length if I would discuss all the meetings, discussions and talks I did. So I’m going to skip that. I’d like to thank GNOME and whoever was responsible for what happened at GUADEC. It was great. We GNOME people did great. We did so much social networking. This social networking is going to be the reason of so many GNOME developments. Incredible.

Thanks to Luis for making his great statement: GNOME is people

Oh, one very nice technology company that I’d like to link to: Polymer Vision. Check it out, their stuff is nice.

For all my non-pgo readers, Edd Dumbill blogs about some interesting methods that will help you with making your software development model more agile.

ps. Why (imho) females should participate in agile software development.

The demos on the N770 device reminded me of a blog entry I wrote six months ago. We are halfway, time for me to look back and do an evaluation:

These are the/my “free software” subjects that I have in mind for this year. I guess that in 2007 I will look back at this blog-entry of mine and laugh with the fact that none of my plans got achieved.

• Redesign and recreate osso-email…

That’s the part I’m satisfied with. Nokia and Kernelconcepts are working on Modest, which will be build on top of my tinymail framework. I’ve created a demo-ui that is (imho) better than the current osso-mail application on the Nokia 770. You can find demos here.

Modest will be available as a desktop, a GPE and a Maemo application. I have also started looking at the OLPC device and I’ve started porting tinymail (the framework) to it.

I haven’t been spending a lot time on building a libmainloop nor libcrack (hey Robert and Rob!) nor have I implemented deconf-spec yet (and people hate the idea without knowing what it is anyway) nor improving codegen nor dvfs nor gnome-schedule (but Gaute has restarted fixing bugs, I noticed) nor with replacing GtkHTML with Gecko in Evolution. I don’t know which topics will still interest me in a few weeks. I think my visit to GUADEC will tell. Of course there’s also some work to do for tinymail. But at some point I would like other people to start building applications with it. Give them the chance to learn it and contribute to it. I have been going quite fast with tinymail, indeed.

This month during my free time I will focus a little bit on improving my communication skills, attending GUADEC and course in a little bit more relaxed way keep improving tinymail. Note, you pronounce tinymail as tinni-mail. Not taainiemail.

I created two new tinymail demos. The first new one has some “go offline”, “go online” and some other tests. If you look carefully, you can see that I trigger a bug that causes the GNOME Hackers mailing list not to get displayed. Very strange. I guess I’ll need to figure out what was happening with that specific folder. Btw. I noticed that loading the summary file of a folder with > 15,000 headers makes the device unresponsiveness. I think (and hope) the mmap() idea will make a big difference for opening huge folders on the device.

The second new demo shows performance. First opening folders while being online, which takes longer because then the folders will be synchronized with the IMAP server. Then in offline mode, which shows the true speed of tinymail. Most folders in these demo’s have ~1,000 messages each. Without the mmap summary idea implemented, expect to load max. ~4,000 headers per active folder with tinymail on a Nokia 770. With the mmap implementation, the maximum amount will be at, I think (estimation), 50,000 or 100,000 headers per active folder.During the demo, I (for privacy reasons) tried making sure I only show spam and/or messages that are also public comments on for example my blog. If you see me searching for a message, I’m searching for such messages ;-).

I was thinking, well, somebody has to do it once. No? So I did. I compiled eds-dbus with support for camel and tinymail for the maemo platform. Both for the ARM cpu. I transferred the two binaries to my N770 device. Configured an account, sudo gainroot and untarred both to /opt, launched it. I took a camera and I made a little movie.

Oh, at the end of the movie I was indeed thinking: “crap, where’s tinymail?” But of course, the demo-ui isn’t fully integrated with the Hildon Maemo framework. Therefore it wont minimize correctly. In other words, if you minimize the demo-ui, it’ll disappear. But just wait for Modest. Okay? That one is going to be the real application for using E-mail on your Nokia 770. This is just a demo-ui.

Oh, sorry for the crap first elektro song in the background. The song that comes around 01:55, is better. No, I’m not planning to build packages for this one. Else people will also ask me to build an account wizard. It’s a bloody demo-ui. Wait for Modest! Others are of course allowed to do packages. Just don’t make *your* users ask *me* for account wizards ;-).

ps. Yes, this means that with some hacky hacky magicy, you can port evolution-exchange‘s camel provider to the device, and connect with Exchange servers. I’m sure o-hand would love that. Maybe a challenge for Novell? Or integrate the N770 with Hula? Tinymail could easily do it. Because it’s build on top of camel.
ps. I repeat (I often do) that Evolution should be split up in many (more) such reusable libraries. This is indeed a direct questions to people like Nat, who as far as I know at Novell gets to decide about it: please allow your Evolution and Camel/mailer teams to work on this. We would improve the libraries. Look at what Ross did with eds-dbus.

I added platform libraries for OLPC, GPE and Maemo to the build environment of the tinymail framework. The GNOME desktop one is now fully implemented with support for things like GnomeVFS, Gnome Keyring, GnomePasswordDialog and NetworkManager. I will personally implement the OLPC platform one. The GPE and Maemo platform ones will be implemented by me and the people who are doing Modest.

The Maemo one has been tested and works, but improved support for some Maemo specifics are soon going to be added.

This list illustrates which platforms will be tested and supported when I’ll release version 2.0 (if ever). A first version of the tinymail framework will probably be released a few minutes before the first public appearance of Modest and will definitely support GPE, Maemo and the GNOME desktop. To give you an ETA, this wont happen before or during GUADEC.

Modest will be much cooler. But tinymail comes with a demo-ui. That demo-ui looks like this if you compile with –with-platform=maemo and –enable-gnome=no.

The “give me your f. password!” – dialog box when you are not using GnomePasswordDialog nor GNOME Keyring:

This is the GNOME Hackers mailing list. I had some fun shooting holes in the E-mail addresses of Joe Shaw and Ross Golder:

Hi Robert Staudinger,

Since you have expressed your criticism, could that mean you care?

I’m aware that I lack good communication skills, but I’m planning to improve this.

Feel free to give some advice on this matter. That is a challenge, isn’t it?

Hey Luis. My opinion is that criticism, for a wise receiver, is probably the most enlightening form of communication. It of course also depends a little bit on the person who does the criticism (it shouldn’t be an idiot who really doesn’t know what he’s talking about). I don’t think it depends on the format of the criticism. Abrasive criticism, can also be enlightening.

Therefore I thank people like oGALAXYo and Bowie Poag for sacrificing their popularity in exchange of our wisdom.

I would very much like more community members to learn how to accept all formats of criticism. My opinion is that exaggerated emotional intelligence (like being extremely empathic) only obfuscates the real message. What really matters is the real message. The fact that the person who expresses criticism does it in a not-so-nice way, is not what the message itself is about. I say: focus on the message itself first and then care about him being empathic or not.

I see many people with eye-flaps on their heads, not looking at criticism at all. Because, they think, criticism is purely something negative. Therefore, they often seem to think, is the message itself also wrong. Which is, in my opinion, unwise. Criticism means the person who expressed it, at least took the time to tell you why your stuff sucks. That by itself means that the person cares about it. Use the message. Let it enlighten you.

That’s my opinion. That’s why I say thank you for your criticism.

ps. I do agree that a person who’s going to express criticism should try to do it in a polite way. Maybe even using a little bit emotional intelligence. I don’t think the emotional intelligence should be all over the message. It should be possible to say: what you did there, sucks. Period. Because if something sucks, it sucks. There’s no so called good empathic way to tell it.

I’m searching for a few holy knights to help me replace the fopen()/fread() implementation in camel-folder-summary.c at GUADEC. Such a replacement would (I think) dramatically improve the memory usage of Evolution (and tinymail. On small devices the improvement would be even more noticeable).

The current implementation copies the string after fread()ing it, it also searches a hash-table to avoid duplicates and NULL terminates the copy. Because the used string format in the summary file isn’t NULL terminated C char pointers, but in stead pascal-string like (with length information in front of the string), while the infrastructure that uses the information doesn’t use the strings this way, I think the best solution will be to replace the file-format with a more mmap()-friendly one. For example with both the length information in front of the string and a NULL termination byte at the end of the string.

The reason why I think it would dramatically improve the situation is that with mmap(), the kernel gets to decide about whether or not putting the memory in its buffers/cache. Note that access to the information should, however, be fast. For example sorting headers depends on this. The access is (while sorting) random (qsort or mergesort or something like that). I don’t want to change thousands of Evolution lines for just this optimization.

Using valgrind I measured that a quite large part of the total amount of memory being allocated during one Evolution (or tinymail) session, goes to this summary information. Being mmap()ed, I think this data (mostly being buffer/cache) wouldn’t harm as much. We would have to drop the hashtable trick that avoids duplication, or we would have to implement it in such a way that duplication is also avoided in the file itself (which isn’t the case at this moment).

I think it would be a nice temporary solution until the disk-summary branch of camel (or libspruce) is finished.

While Dirk-Jan‘s modest-on-maemo screenshot is of course a lot better integrated with Maemo

I have ported the default tinymail to Maemo. This is how the tinymail demo-ui looks after doing a standard compilation:

And if you run it using run-standalone.sh you’ll get this:

I had to make two patches. One for eds-dbus which replaces the symlinks with simple copies and changed a few small buggies and dependencies. It looks like the Subversion client in a scatchbox doesn’t support restoring the symlinks. Maybe I’m doing something wrong? I also had to make a patch for tinymail. But soon will the Nokia guys give me their libtinymail-maemo which will make this patch unnecessary (support for maemo will be a standard tinymail component).

svn co http://svn.o-hand.com/repos/eds-dbus/trunk eds-dbus
cd eds-dbus; patch -p0 < eds-dbus-maemo.patch
./autogen.sh --enable-camel=yes --prefix=/opt/eds --with-dbus=yes --with-soup=no
make && make install ; cd ..; mkdir tinymail; cd tinymail
svn co https://svn.cronos.be/svn/tinymail/trunk ; cd trunk
patch -p0 < tinymail-maemo.diff
PKG_CONFIG_PATH=/opt/eds/lib/pkgconfig ./autogen.sh --prefix=/opt/tinymail/ \
    --enable-gnome=no --with-html-component=none
make && make install

I also started with getting myself a OLPC development environment working in a QEmu. So you can expect a port to that, soon.

There’s also a new modest-on-desktop screenshot by Dirk-Jan:

The second-most important feature, next to viewing spam, is of course deleting spam. I just added support for both deleting messages and expunging folders.

if (gtk_tree_selection_get_selected (sel, &model, &iter))
{
        TnyMsgHeaderIface *header;
        TnyMsgFolderIface *folder = ... ;
        gtk_tree_model_get (model, &iter,
          TNY_MSG_HEADER_LIST_MODEL_INSTANCE_COLUMN,
          &header, -1);
        /* This also automatically updates the GtkTreeView,
            so you don't have to reassign a new model. */
        tny_list_iface_remove (TNY_LIST_IFACE (model), header);
        /* The actual removal from the folder itself */
        tny_msg_folder_iface_remove_message (folder, header);
        /* Optionally you can immediately expunge the folder
        tny_msg_folder_iface_expunge (folder); */
}

Lot’s of *bla bla*. Time for the *bling bling* in my blog. Here’s a screenshot of Modest (it might be renamed to Shy) on the Maemo development platform:

Graph made using G_SLICE=always-malloc G_DEBUG=gc-friendly and a –disable-visibility glib

You can see on the new graph a tinymail demo-ui session where I opened my Inbox of ~1000 IMAP messages and then switched to my nearly empty Drafts box and repeated that four times.

By the way, the GObject debugging info on this page is very interesting. Thanks.