Author: admin

Yesterday I told you guys that I didn’t yet implement a account store that doesn’t use GConf, right?

Here’s the one for OLPC that uses gkeyfile: libtinymail-olpc/tny-account-store.c

I also improved camel-lite-builder so that it no longer links with GnomeVFS (nor ORBit-2 nor Bonobo). I don’t know whether the final OLPC device will ship with GConf and the D-Bus version of GnomeVFS. It no longer matters for tinymail.

By the way, great blog post by Mukund where he explains how we, human species, are in fact nothing but tests that test whether some newer genetic instructions successfully work or not.

Guys, remember I promised that I was not going to post new photos on my blog until I have one with tinymail running on the OLPC mobo?

Well, I will need my blog-space for posting pictures soon. So here’s a picture of tinymail running on the OLPC mobo!

I haven’t yet implemented a account store that doesn’t use GConf for reading the account configuration. Nor have I already added a patch for Camel in camel-lite-builder that removes Camels GnomeVFS dependency. Since a month or so, Camel depends on GnomeVFS, which at this moment pulls things like Bonobo and ORBit-2.

So .. I don’t think it’s a good idea not to patch it out of Camel before starting to use Camel on devices like the OLPC laptop. The OLPC image indeed doesn’t seem to come with libraries like GConf, Bonobo and ORBit-2. Other than that, it’s a very easy device to target. It’s no surprise that tinymail runs on it.

There’s of course a gnome-vfs-dbus. Very interesting of course. Might also be a solution for this. Well, the current hack for getting it to run was simply removing the two files from Camels Makefile.am.

I just uploaded the new tinymail.org website.

The small size of the web-design is indeed a hint that the website can also be used on mobile devices, who are a target of the tinymail framework.

Some of the website content might change as some people will probably give me advises. The purpose of this website is to be a view for also the commercial world. The idea is indeed to encourage commercial activities around the LGPL tinymail framework.

I’m also planning to make a website with a much more technical point of view. Being a website for a development framework, it will include developer documentation like the API reference, class diagrams, demos and guidelines.

I hope people understand that I only have two hands and that making such documentation takes time. I am, however, committed to do it. There’s already an API reference which will of course very soon be updated. Using the source code from the Subversion repository you can of course rebuild the API reference yourself.

Things like unit tests and documentation have a high priority and some of it is already available. I will probably not release unless both are perfect.

I will of course welcome people who want to help me creating or improving developer documentation, unit tests, design, implementation, a first release and the website.

I hope you will hear about Modest very soon. And yes, I was told they are designing the Modest user interface in such a way that it will be possible to use it on both a desktop and on a mobile device.

It is my opinion that using our blogs and the Internet being our speech instruments, we should somehow start organizing a huge anti-war protest.

Israel is clearly targeting civilians and civilian infrastructure. I condemn these actions and will no longer feel any compassion whatsoever for Israeli soldiers that decide to fight this war for their government. I can not agree with Hezbollah for launching missiles on Israeli cities. I can understand, however, that Lebanese people (like Hezbollah) no longer agree with what Israel has been doing the past years and during the last 12 days.

This way Israel is creating new hatred and aggression against themselves. Question yourself: what would you do if another country bombs your village killing your woman, baby, kid and family members? Imagine finding your kids head and other body-parts in a bloody bombed street or building. Would you not hate the people that did this? Western people live behind their CNN screen. Enjoying “television war”. It actually makes me sick that so many of us, Westerns, have lost reality with what war really is (please, for once: look at it. Even if you are frightned of pictures showing dead children. Look at the reality. See it. Understand it).

Our grandparents fought World War I and II. They learned us, the generations after World War II, never again to go to war. They died for our peace. We, young people, might never understand what they have seen. But please respect them. War is never a solution. War against citizens, what in my opinion Israel is doing, is evil.

It is insanity. I refuse to call this human. We, Europeans, do not want this. If our European politicians would want this, I would tell them here and now that I no longer call myself a citizen of their European union. Luckily so far most of them have been condemning Israel and their recent actions. I hope this will continue. I hope everybody who makes himself guilty of war crimes (both Hezbollah and Israeli) will get severely punished for these crimes against the human race (almost 400 unarmed citizen after only 12 days of war, 350 of them Lebanese people).

I hope the Nuremberg trials will repeat themselves (but please read the interesting 5th comment of Max, I also agree with that).

People who are going to reply with the “terrorist” word in their comment, please read this first. Maybe you should stop being so naive and learn about indoctrination using fear. You might understand what this American president is really doing with his “terrorists are everywhere”-speeches.

To world leaders and the UN: if American leaders show no interest in peace, please no longer consider their opinion to be of any value. Proceed with your peace plans without American support.

I’m guessing most people know these lounge cafes where they have relax music and where the waiter writes down your order on a PocketPC like device. About the PocketPC, in some such cafes they do that already, in others they don’t. If the one in your city doesn’t, maybe dancings or Café Del Mar style cafes do?

This device could be a Linux device that uses GNOME components. Why does it have to be Windows CE? Oh, it definitely at this moment most of the times is using Windows CE. Very often with an application done in Visual Basic Embedded or maybe Compact Framework .NET.

What is the business interested in? Selling cocktails, spirits and other fancy drinks that are very expensive. The customer expects such drink to arrive fast and oh, let us not forget: in total style. The bar guy constantly does tricks like throwing bottles around, The waiter, all dressed up chic (or maybe even naked at some places), brings you a nice big cocktail with a lot fruits, toys and other crap. There’s one word for it, or maybe two: style and eye candy.

The PocketPC probably doesn’t help a lot with making it faster. In dancings where they often also use them, it of course does. But it makes it more fancy. Fancy sells. Don’t underestimate it.

Apart from being fancy, what does this device needs to do?

• Take the order in a very fast way;
• Write down the table number;
• Send to order over wifi to the bar guys computer;
• The computer of the bar guy needs to show information about the cocktail. Ingredients and stuff like that;
• For customers that ask, the PocketPC needs to show a nice big picture of the cocktail so that the waiter can (in a fancy way) show it to the customer;
• Show the price and print the ticket in a few ways. For example a ticket per person, per drink or a ticket per table. Yes, you have portable printers and they look VERY fancy. It’s totally accepted by the owner of the business. He’s the guy who is going to pay for your consultancy, right? Listen to him!
• Get updated every month, about new items on the menu of the place (or do you want to make a locally served website maybe? Might work, might not work)

Does the device need addressbook functionality? Does it need contactbook functionality? Does it need E-mail functionality? Messaging functionality? Voice over IP? All that might be true! It depends on the business activity. On how the people who work at that place interact with each other. Maybe does the bar guy wants to send a message to the waiter like: “we don’t have the ingredients of this cocktail, can you ask the customer whether he wants it without that ingredient at a discount prince?”. Can your solution cope with that event?

The point that I’m trying to make is, choice and change. Today the owner of the business starts using your solution. They will definitely find defects in the system like: “what if we don’t have the ingredients for the cocktail and the user wants to cancel the order or pick another drink with a different price?”, “we can’t undo it using the device today, the waiter has to go to the main computer and that takes needlessly time”. The situation changes. Can your solution cope with that? Can the situation be avoided? And how?

Have you thought about how our GNOME desktop components needs to (cope with) change for them to be suitable for such application domains? What about tomorrow? How can we make the Linux device less expensive than the Windows CE device? Are only experienced GNOME developers going to write a gazillion of such applications? No, it will be nine to five developers that don’t care about C. How can we attract them to our platform?

One thing the waterfall model learned us is that we CAN NOT catch all use-cases in a first run. Be agile. Be adaptive to change. Be flexible. Develop in style, and you will be.

Here are the patches to get the upstream version of evolution-data-server and evolution-exchange to start using the mmap technique for loading the header and content info summary data of Evolution and tinymail.

I expect it to reduce memory usage of Evolution with approximately fourty to sixty megabytes of ram, depending on the amount of folders you have.

• evolution_data_server__mmap_summary.diff
• evolution_exchange__mmap_summary.diff

I’m very interested in your test results:

cvs -z3 co evolution-data-server
cd evolution-data-server
patch -p0 < evolution_data_server__mmap_summary.diff
./autogen.sh --prefix=/opt/eds-mmap && make && sudo make install
evolution --force-shutdown
export LD_LIBRARY_PATH=/opt/eds-mmap/lib
evolution (and wait a few minutes to let the summary files get build)
evolution --force-shutdown
valgrind --tool=massif evolution

Please do this procedure with and without the LD_LIBRARY_PATH and measure the difference. If you have any technical questions about it, feel free to ask me.

This version of the mmap patch for Camel is finally working on the Nokia 770 device (on an ARM cpu which needs aligned memory access).

For people who might not know what that means, it means that tinymail can open very large folders (I expect > 20,000 headers) easily on the device. I’m testing it now.

While doing these mmap patches for Camel, I had to leave the OLPC mobo untouched. I decided to undo that and install it so that it will be powered on (the idea here is that this way it will push me to give more attention to it). I’ll just put a promise here so that I can’t escape it: my next picture will be tinymail running on the OLPC mobo.

Just look at it! How cute! In a box :-)

Hey Nokia (or OLPC or whoever really wants it)! When will you guys pay Collabora (or somebody else) to do Python bindings for tinymail :-) ? The API is more or less ready for this (the API will definitely change a little bit. But not aggressive anymore. I Hope). If somebody else is interested, don’t let me saying somebody else is planning to do it stop you. Go ahead!

Very interesting read. Don’t just read the soldiers letter. Also read the reply beneath it.

Because

One, I share the opinion with Zaheer Abbas Merali that what Israel is doing is wrong, that I hope the international community will indeed condemn Israel’s actions and that the people in Lebanon and Palestine are getting a oppressive treatment by Israel.

Two, it’s my opinion that a personal blog, being syndicated on planet.gnome.org or not, is a free-speech instrument that definitely can be used for making political statements and expressing opinions. That what we, software developers working on GNOME software, write in our personal blog should not be censored because we get to come on a popular blog aggregator like planet.gnome.org.

We are very much allowed to write anything we want in our blogs. Knowing them personally, I’m very sure every single person in charge of planet.gnome.org agrees with that. I’m sure all of them also know that if they wouldn’t allow us to write anything we want, very few GNOME developers would want to be syndicated on planet.gnome.org. Nor had any of them, as far as I know, ever had the intention to forbid any such thing.

To make sure I piss off everybody who made these comments to Zaheer, I repeated his opinion and made sure it gets on the planet.gnome.org aggregator again. I strongly disagree with the opinion of anybody who says that we, GNOME developers, shouldn’t be allowed to write anything at anytime on our own personal blogs. I will also fight for my right to say whatever I want. Whenever I want. Wherever I want. As long as I don’t intend to discredit somebody or some group by making false statements (like unproven racist statements or telling lies about somebody to destroy his or her reputation).

If you cannot grasp that, you definitely have a huge culture difference with me. Maybe you shouldn’t use the Internet if you don’t understand this? The Internet is, for me, the free-speech instrument of this century. If necessary, I will strongly defend the free opinion and free speech of Zaheer.

“The greatest threat to freedom is the absence of criticism. — Wole Soyinka”
“I may disagree with what you have to say, but I shall defend, to the death, your right to say it. — Francois Marie Arouet Voltaire”

Camel with the mmap patch and the latest tinymail on your Nokia 770.

Note to Ross: there’s also a version of the patch (in the tarball) that doesn’t yet include the mmap patch and that will update the Camel of your eds-dbus. What do I do to get this committed in your Subversion?

If you want to build it yourself:

~$ cd /scratchbox/users/$username/
$ svn co http://svn.o-hand.com/repos/eds-dbus/trunk eds-dbus-mmap
$ svn co https://svn.tinymail.org/svn/tinymail
$ cd eds-dbus-mmap; patch -p0 < eds-dbus-mmap-camel-update.diff
$ cp e-data-server-utils.c e-data-server-utils.h libedataserver/
$ /scratchbox/login
[sbox-SDK_ARMEL: ~] > cd eds-dbus-mmap
[sbox-SDK_ARMEL: ~] > ./autogen.sh --prefix=/opt/eds --with-soup=no \
	--enable-camel=yes --with-dbus=yes
[sbox-SDK_ARMEL: ~] > make && make install
[sbox-SDK_ARMEL: ~] > cd .. ; cd tinymail/trunk
[sbox-SDK_ARMEL: ~] > export PKG_CONFIG_PATH=/opt/eds/lib
[sbox-SDK_ARMEL: ~] > ./autogen.sh --prefix=/opt/tinymail --enable-gnome=no \
	--with-platform=maemo --with-html-component=none
[sbox-SDK_ARMEL: ~] > make && make install
[sbox-SDK_ARMEL: ~] > cd /opt
[sbox-SDK_ARMEL: ~] > tar zcvf tinymail-maemo.tar.gz eds/ tinymail/

On your Nokia 770 device:

- xterm
- sudo gainroot
- cd /opt
- tar zxvf /media/mmc1/tinymail-maemo.tar.gz
- exit
- Run your account-create script
- /opt/tinymail/bin/tinymail

Update: sadly there seems to be something unstable about using mmap() on the Nokia 770 itself (at this moment, the kernel often kills the process). In stead of the eds-dbus-mmap-camel-update.diff you can also use the eds-dbus-fread-camel-update.diff. Tinymail will then not use mmap(). In scratchbox, mmap(), however, does work correctly. I will check with Nokia what this mmap() issue is about. I’ve put an eds that doesn’t use mmap() online here. On your device untar it in /opt (overwriting the /opt/eds of tinymail-maemo.tar.gz).

This new eds is also (cause of other issues that I’ve found while I was developing the mmap() patch) improved a lot in terms of memory usage. Using this new eds, showing 14,500 headers is now possible on a Nokia 770 with tinymails demo-ui. With the mmap() patch working on the device, it will be possible to display much larger folders.

I might have put binaries online. But be advised to always build from source until I do a release. Whatever is in Subversion is always going to be much more stable than whatever alpha/beta or unofficial binary I make and put online.

Update: You can get a version of both in one tarball here.

Chris is doing more crazy things with Dates, tinymail and Contact

I have this funny feeling this version of the patch for the mmap summary stuff is actually really working. I’ve been running Evolution with it for two hours now. I’ve let it rebuild all of my summary files and opened all my imap folders a few times.

I wrote a page that explains the Camel stuff that I have been blogging and mailing (like a crazy horse) about.

ps. Performance people: if you are technically interested, I can definitely use your help. There’s still a lot to do. We can reduce the memory usage even more (mainly for online usage of Evolution). It will also help tinymail with doing its initial load of a large folder. More info about this in the article and by simply asking me.

I’ve been fixing bugs and implementing the provider specifics for summary files in all default Camel providers. On my Evolution, this version of the patch actually worked without any strange crashes.

You simply apply the patch to evolution-data-server/camel, you build e-d-s in /opt/camel and you set LD_LIBRARY_PATH to /opt/camel/lib before launching evolution –force-shutdown followed by evolution. You can of course also use tinymail after setting the library path. Tinymail is the real reason why I did this patch.

As I warned in my previous blog, it will rewrite all summary files (which takes some time). If you restart evolution without setting the library path, Evolution will again rewrite all summary files using the old format. So nothing really dangerous happens. But still make a backup of your evolution data directory.

If you are online, Evolution still consumes a lot memory. That’s because of memory problems in the Camel provider implementations that scan for new messages. I hope I (finally) inspired people to take a look at these problems. If these are also fixed, Evolution will on average (I think) consume less then 50MB. Try Evolution (with the patch) in offline mode and measure it with valgrind. You’ll see.

In general, visit my files and check for a new version of the patch before applying anything. The filename is appended with fixesXX.diff

Hey, look at this cool thing I did, I sure hope the friendly Evolution maintainers commit it upstream so that we can all benefit in a happy orgy of peace and love. Note that without the mmap() patch, (my) Evolution consumes a lot more of course.

The patch, The mmap information from proc

cp -a $HOME/.evolution $HOME/evo.backup
cvs co evolution-data-server; cd evolution-data-server
cd camel; patch -p0 < patch ; cd ..
./autogen.sh --prefix=/opt/camel
make && sudo make install
export LD_LIBRARY_PATH=/opt/camel/lib

and then

/opt/tinymail/bin/tinymail

or

evolution --force-shutdown ; evolution

It will rewrite all “summary” files in your $HOME/.evolution/mail/type/$account directory. Don’t worry, if you go back to the old implementation, it will reverse that by doing the exact same thing. After that, go in offline mode. Shut down Evolution. Restart Evolution, for example using valgrind and massif. If things go terribly wrong, you still have the old files in “evo.backup” (right?). The patch only supports MBOX and IMAP. If you use another provider type, you will have to wait for support for that.

In online mode, Evolution will still (imho) consume to much memory. But I already have a very good idea where and how to fix that (imap_rescan for example).

Note that this is highly experimental. It will most likely crash. You will most likely have to launch Evolution in your debugger a few times, and fix things before you can get this working. Normal users don’t want this yet. I’m serious!

I created a new patch that fixes a few things (about the previous patch). The mailinglists are slow (the hackers and patches mailinglists of evolution are in CC) so I can’t yet point to the changelog E-mails. But I guess you can soon follow it up there.

Of course, what you guys care about is fancy graphs. The first shows tinymail opening and closing the ~650 headers folder a few times (this time without the leak). And the second is tinymail opening and closing the ~14550 headers folder.

While finding my leak I also learned that the original implementation of camel-folder-summary.c did an unnecessary strdup() and did some funny unneeded memory segmentation. I created this patch that fixes the unnecessary strdup. The memory segmentation fix is already committed upstream (and is for-free fixed by the mmap patch of course).

The strdup caused ~300kb unnecessary memory for the 14558 headers folder, and ~40kb for the 658 headers folder. In my opinion that’s a lot for memory that isn’t needed at all.

Everybody: please start using valgrinds massive tool aggressively on every change you make to your code. The difference between token = strdup(tokens[i]) and token = tokens[i] can be a lot memory! You can make little children, that will use such a OLPC device, happy with it.

If you want to make a real difference in Evolutions memory usage, check the imap_rescan method in camel-imap-folder.c. There’s huge room for improvement in memory usage in that method (or something that is related to “scanning messages”). Valgrind shows me up to 50MB of memory usage just for scanning for the new messages of ‘one’ folder (the ~14550 headers one).

Don’t think Evolution doesn’t need our community help. Let us fight this memory monster together shall we? Hows that for a positive message, jdub? ;-). There’s definitely very interesting components (like Camel) that aren’t very monolithic. You *can* make a difference, just checkout the code and start hacking on it.

Chris did some nice integration with tinymail, Dates and Contact. I, of course, accepted his patch.

After 43 bookmarks in khexedit, three afternoons of hacking and calculating memory positions that get defined by recursive loops that write a file recursively (and other crazy shit). After adding a ‘\0’ after each pstring and all content-type tokens in that summary file and making sure the lenghts of the strings are still correct so that I know the offset to the next info.

I’m finally done

Lets start small with the least impressive images. In these two images I open and closed a a folder of 658 headers a few times in tinymail. The first graph is without using my patched camel. It’s using the current Camel of my Ubuntu Dapper. I gimped two red lines on it. They illustrate the amount of memory this folder needs.

This is a graph that shows loading and unloading the same folder a few times but now using my patched Camel. By looking at the green lines you can clearly see that I still have a memory leak. The leak grows each time you open/close a folder, which is of course perfectly normal in case of a real leak. This doesn’t mean that the mmap patch *needs* this leak. I’m of course going to hunt it down.

The leak, however, isn’t very interesting. What is interesting is that the same two red lines now illustrate that I need less memory.

So now the impressive ones. First a graph that shows loading a big folder of 14558 headers using the default Ubuntu Dapper Camel library. You can see that valgrind decided to print the 10M barrier. I made red lines at 8MB and at 10MB. You’ll see the same red lines in the next graph ;-).

I guess words cannot describe this as good as the graph can. As you can see I’m saving a few megabytes. I’m guessing after some more tweaking (for example using posix_madvise), showing 14558 headers will be easy for a mobile device like the Nokia 770 and the OLPC device. I’m aiming at making it possible to display 50,000 headers on the Nokia 770 now. Seeing these valgrind graphs, I learned there’s still a lot room for improvement in my own code.

About speed. Well. I didn’t yet measure it. But I really think this is a lot faster than the fread() implementation. It probably depends a little bit on the posix_madvise hint and your kernels mmap() implementation. Sorting (on my desktop) is definitely not slow for the 658 folder when using the mmap() implementation. I was a bit afraid of that because reading the mmap() data happens at random during the sorting. Sorting the 14558 headers folder felt a little bit slower. But I really need to measure it, and tweak with posix_madvise first.

This is the patch. I don’t think it’s going to work with Evolution already, unless you remove your $HOME/.evolution directory and stuff like that. It should, however, work with tinymail. Anyway, don’t think that this stuff is ready yet. It’s not. The patch is for development purposes. It’s an experiment.

Suggestion if you want to test: install e-d-s (which contains camel) with the patch in prefix /opt/camel and do LD_LIBRARY_PATH=/opt/camel/lib before running tinymail (for example using the massif tool of valgrind).

Harish told me yesterday (on IRC) he would like this to be one of the big patches for the next Evolution release. I don’t think that would be a good idea Harish. It really needs a lot more testing and experimenting first. I’m actually surprised that I got it working.

It is absolutely in experimental state. It needs enormous amounts of refactoring work. It’s definitely a hack at this moment. It doesn’t really work yet. It’s ugly (like, I have to save things in a load method). Yadi yada.

But some of it is actually working and you can find the current patch here. It often crashes and burns and does other cool stuff like throwing keyboard keys in your eyes. In tinymail, if you do this and if you do that, you can actually already see the summary of headers in the view. I still have to do the “content_info” part, and disable a few more things in the camel providers.

Once it works, I’m interested in what tweaking the mmap addr pointer with posix_madvise is going to do when for example the sorting and loading of the summary happens. My target is to support a summary view of ~80,000 messages on a mobile device with less than 50MB ram. Cause of how the summary information is at this moment loaded, tinymail can only support something like ~15,000 messages.

Anyway, it’s Friday evening. Tinne is waiting for me! So I’m forcing myself to stop hacking on it and spend the evening doing some other stuff. Like going out.

Some people are requesting for a tinymail based E-mail client that would integrate well with the standard GNOME desktop.

There’s already a libtinymail-gnome-desktop which implements some of this desktop integration (like NetworkManager usage, Gecko HTML component and GnomeVFS support for when you want to save an attachment). Nevertheless I’d like to empathize that I don’t have a focus on developing an E-mail framework for the desktop.

However. That doesn’t mean that I’m not interested in contributions or E-mail clients on top of tinymail that would implement this. I have, however, been telling people that such an E-mail client is NOT a one person job. A team of people, in an agile development setting like Extreme Programming, would have to spend at least a few months time on it.

I very often have to repeat that I am not going to implement their new desktop E-mail client. People, please get realistic: it’s not doable if I’m the only person doing it. If you guys really want a new E-mail client, then start forming a team that will implement it. It’s as simple as it is: good code doesn’t grow biologically. Code doesn’t just happen. Code must be written by software developers. You cannot magically emerge an E-mail client from nothing, or by just talking about it. You have to implement it.

If you want to do it well, I really suggest that team would use the extreme programming development methods.