Month: July 2006

Hey, look at this cool thing I did, I sure hope the friendly Evolution maintainers commit it upstream so that we can all benefit in a happy orgy of peace and love. Note that without the mmap() patch, (my) Evolution consumes a lot more of course.

The patch, The mmap information from proc

cp -a $HOME/.evolution $HOME/evo.backup
cvs co evolution-data-server; cd evolution-data-server
cd camel; patch -p0 < patch ; cd ..
./autogen.sh --prefix=/opt/camel
make && sudo make install
export LD_LIBRARY_PATH=/opt/camel/lib

and then

/opt/tinymail/bin/tinymail

or

evolution --force-shutdown ; evolution

It will rewrite all “summary” files in your $HOME/.evolution/mail/type/$account directory. Don’t worry, if you go back to the old implementation, it will reverse that by doing the exact same thing. After that, go in offline mode. Shut down Evolution. Restart Evolution, for example using valgrind and massif. If things go terribly wrong, you still have the old files in “evo.backup” (right?). The patch only supports MBOX and IMAP. If you use another provider type, you will have to wait for support for that.

In online mode, Evolution will still (imho) consume to much memory. But I already have a very good idea where and how to fix that (imap_rescan for example).

Note that this is highly experimental. It will most likely crash. You will most likely have to launch Evolution in your debugger a few times, and fix things before you can get this working. Normal users don’t want this yet. I’m serious!

I created a new patch that fixes a few things (about the previous patch). The mailinglists are slow (the hackers and patches mailinglists of evolution are in CC) so I can’t yet point to the changelog E-mails. But I guess you can soon follow it up there.

Of course, what you guys care about is fancy graphs. The first shows tinymail opening and closing the ~650 headers folder a few times (this time without the leak). And the second is tinymail opening and closing the ~14550 headers folder.

While finding my leak I also learned that the original implementation of camel-folder-summary.c did an unnecessary strdup() and did some funny unneeded memory segmentation. I created this patch that fixes the unnecessary strdup. The memory segmentation fix is already committed upstream (and is for-free fixed by the mmap patch of course).

The strdup caused ~300kb unnecessary memory for the 14558 headers folder, and ~40kb for the 658 headers folder. In my opinion that’s a lot for memory that isn’t needed at all.

Everybody: please start using valgrinds massive tool aggressively on every change you make to your code. The difference between token = strdup(tokens[i]) and token = tokens[i] can be a lot memory! You can make little children, that will use such a OLPC device, happy with it.

If you want to make a real difference in Evolutions memory usage, check the imap_rescan method in camel-imap-folder.c. There’s huge room for improvement in memory usage in that method (or something that is related to “scanning messages”). Valgrind shows me up to 50MB of memory usage just for scanning for the new messages of ‘one’ folder (the ~14550 headers one).

Don’t think Evolution doesn’t need our community help. Let us fight this memory monster together shall we? Hows that for a positive message, jdub? ;-). There’s definitely very interesting components (like Camel) that aren’t very monolithic. You *can* make a difference, just checkout the code and start hacking on it.

Chris did some nice integration with tinymail, Dates and Contact. I, of course, accepted his patch.

After 43 bookmarks in khexedit, three afternoons of hacking and calculating memory positions that get defined by recursive loops that write a file recursively (and other crazy shit). After adding a ‘\0’ after each pstring and all content-type tokens in that summary file and making sure the lenghts of the strings are still correct so that I know the offset to the next info.

I’m finally done

Lets start small with the least impressive images. In these two images I open and closed a a folder of 658 headers a few times in tinymail. The first graph is without using my patched camel. It’s using the current Camel of my Ubuntu Dapper. I gimped two red lines on it. They illustrate the amount of memory this folder needs.

This is a graph that shows loading and unloading the same folder a few times but now using my patched Camel. By looking at the green lines you can clearly see that I still have a memory leak. The leak grows each time you open/close a folder, which is of course perfectly normal in case of a real leak. This doesn’t mean that the mmap patch *needs* this leak. I’m of course going to hunt it down.

The leak, however, isn’t very interesting. What is interesting is that the same two red lines now illustrate that I need less memory.

So now the impressive ones. First a graph that shows loading a big folder of 14558 headers using the default Ubuntu Dapper Camel library. You can see that valgrind decided to print the 10M barrier. I made red lines at 8MB and at 10MB. You’ll see the same red lines in the next graph ;-).

I guess words cannot describe this as good as the graph can. As you can see I’m saving a few megabytes. I’m guessing after some more tweaking (for example using posix_madvise), showing 14558 headers will be easy for a mobile device like the Nokia 770 and the OLPC device. I’m aiming at making it possible to display 50,000 headers on the Nokia 770 now. Seeing these valgrind graphs, I learned there’s still a lot room for improvement in my own code.

About speed. Well. I didn’t yet measure it. But I really think this is a lot faster than the fread() implementation. It probably depends a little bit on the posix_madvise hint and your kernels mmap() implementation. Sorting (on my desktop) is definitely not slow for the 658 folder when using the mmap() implementation. I was a bit afraid of that because reading the mmap() data happens at random during the sorting. Sorting the 14558 headers folder felt a little bit slower. But I really need to measure it, and tweak with posix_madvise first.

This is the patch. I don’t think it’s going to work with Evolution already, unless you remove your $HOME/.evolution directory and stuff like that. It should, however, work with tinymail. Anyway, don’t think that this stuff is ready yet. It’s not. The patch is for development purposes. It’s an experiment.

Suggestion if you want to test: install e-d-s (which contains camel) with the patch in prefix /opt/camel and do LD_LIBRARY_PATH=/opt/camel/lib before running tinymail (for example using the massif tool of valgrind).

Harish told me yesterday (on IRC) he would like this to be one of the big patches for the next Evolution release. I don’t think that would be a good idea Harish. It really needs a lot more testing and experimenting first. I’m actually surprised that I got it working.

It is absolutely in experimental state. It needs enormous amounts of refactoring work. It’s definitely a hack at this moment. It doesn’t really work yet. It’s ugly (like, I have to save things in a load method). Yadi yada.

But some of it is actually working and you can find the current patch here. It often crashes and burns and does other cool stuff like throwing keyboard keys in your eyes. In tinymail, if you do this and if you do that, you can actually already see the summary of headers in the view. I still have to do the “content_info” part, and disable a few more things in the camel providers.

Once it works, I’m interested in what tweaking the mmap addr pointer with posix_madvise is going to do when for example the sorting and loading of the summary happens. My target is to support a summary view of ~80,000 messages on a mobile device with less than 50MB ram. Cause of how the summary information is at this moment loaded, tinymail can only support something like ~15,000 messages.

Anyway, it’s Friday evening. Tinne is waiting for me! So I’m forcing myself to stop hacking on it and spend the evening doing some other stuff. Like going out.

Some people are requesting for a tinymail based E-mail client that would integrate well with the standard GNOME desktop.

There’s already a libtinymail-gnome-desktop which implements some of this desktop integration (like NetworkManager usage, Gecko HTML component and GnomeVFS support for when you want to save an attachment). Nevertheless I’d like to empathize that I don’t have a focus on developing an E-mail framework for the desktop.

However. That doesn’t mean that I’m not interested in contributions or E-mail clients on top of tinymail that would implement this. I have, however, been telling people that such an E-mail client is NOT a one person job. A team of people, in an agile development setting like Extreme Programming, would have to spend at least a few months time on it.

I very often have to repeat that I am not going to implement their new desktop E-mail client. People, please get realistic: it’s not doable if I’m the only person doing it. If you guys really want a new E-mail client, then start forming a team that will implement it. It’s as simple as it is: good code doesn’t grow biologically. Code doesn’t just happen. Code must be written by software developers. You cannot magically emerge an E-mail client from nothing, or by just talking about it. You have to implement it.

If you want to do it well, I really suggest that team would use the extreme programming development methods.

There’s going to be more on tinymail.org very soon. Like a trac and probably also a wiki. For now I moved the Subversion repository from our company one to one which I’m hosting myself.

So, if you are interested in the tinymail source code, you can do something like this:

svn checkout https://svn.tinymail.org/svn/tinymail/trunk

The original location has a RewriteRule. But I don’t think that works with the standard (current) Subversion client. But references to the old locations should automatically direct you to the right location.

• I just received the OLPC motherboard. I will definitely bring tinymail to the device. (Picture taken with a bad phone-camera)
• Tinymail is experiencing some aggressive API changes that will make generating language bindings much more easy.
• I registered tinymail.org. Contact me if you are interested in helping me building a nice website for tinymail

During the GUADEC week I had to explain a few times why you pronounce tinymail as tinniemail rather than the English taainymail. The reason is of course obvious: because that is also how you pronounce my girlfriends name, Tinne.

Maybe are some GUADIANS now curious about who this Tinne person is? After all, she’s the reason why all you guys have to pronounce tinymail as tinniemail.

So on these photos you can see miss Tinne saying “Telepathy rocks, because they picked both a name and logo that doesn’t sound nor look geeky”. Which basically translates to “I like this T-shirt, now it’s mine!”. Thanks Rob Taylor! You made my Tinne happy with it. And I gained a few more credits for being a good boyfriend. Well, actually the deal was that I could go to GUADEC, if I would bring a cute penguin teddy bear. But since those aren’t sold at GUADEC, I had to come back with these T-shirts.

Thank you Telepathy guys for being smart enough to make the logo and name girl-friendly!

It certainly looked like a lot people, companies and organizations are interested in tinymail. Among them are Nokia and OLPC. Some talks between Nokia and Collabora for making the Python language bindings of tinymail kick ass are happening. I talked with Christopher Blizzard and Jim Gettys about tinymail on the OLPC laptop. They where very interested. Especially if there’s going to be Python bindings (I guess that way they can do prototyping and experiments more easily). Nokia is of course also extremely interested. Meeting them and the GPE guys working on Modest was a great experience.

This blog entry would be multiple pages in length if I would discuss all the meetings, discussions and talks I did. So I’m going to skip that. I’d like to thank GNOME and whoever was responsible for what happened at GUADEC. It was great. We GNOME people did great. We did so much social networking. This social networking is going to be the reason of so many GNOME developments. Incredible.

Thanks to Luis for making his great statement: GNOME is people

Oh, one very nice technology company that I’d like to link to: Polymer Vision. Check it out, their stuff is nice.