Month: July 2006

This week will be a documentation week for tinymail. A normal XP iteration means meet, design, implement, document and test. I very often do the document part (mostly the API). I, however, never brought all that documentation together. There’s of course still a lot other things to document. For example making some sample code embedded in the documentation.

The API reference is updated and moved to its final location (this doesn’t mean that this is the final 1.0 API. It might still change until I actually do a 1.0 release). I have setup a trac website for tinymail and one for camel-lite-builder. These trac websites will function as the developers website for tinymail.

I discussed some ideas with Ivan Krstic about E-mail and the One Laptop Per Child project. The current idea is to make a local webserver that will serve webpages like, I guess, Roundcube does. But then much smaller in memory footprint. Which is why he’s interested in tinymail and its Python bindings. I will of course make sure that the tinymail framework can function perfectly in a web environment.

Christopher Blizzard then asked me to fill in this page on the OLPCWiki. It contains some information about tinymail on the OLPC laptop.

I hope people will help me this week with setting up all the wiki pages of the tinymail trac. Everybody is of course invited to improve things like the website (just wget -r it) or documentation or the trac infrastructure and its wiki pages. If you have an idea: please go ahead and edit it. I mean .. if I don’t like it, I can remove it right?

I also hope there’s going to be more people helping me with language bindings. Like improving the Python ones and implementing the .NET ones. The more of this stuff that I will have before making a first release, the better this first release will be. If somebody climbs the stage, I will definitely join him on this. If nobody does, it might be delayed until after my holiday: Tinne decided that I need a break next week.

A few days ago, on some mailing list, I wrote:

I actually but truly believe that running most GNOME programs under a garbage collector would actually save (a lot) more memory then the virtual machine would add.

In stead of just saying such crazy things, I should of course also prove it. It’s of course not easy to prove this. What I can do is show you two programs that do exactly the same. One program uses Python language bindings. The other doesn’t use language bindings but directly invokes the API in C using GObject.

The two applications loaded the same IMAP folder of ~1000 headers. And unloaded that folder.

The usage of Python bindings causes some Python proxy objects to be created. I think (but I’m not sure) that most of the extra memory that you will see is caused by these.

• The Python application
• The C application

In Python

In C

Consider that most desktop applications consume more than 20MB and that the test application (tinymail) is designed and developed with low memory consumption in mind, I still believe the overhead introduced by a higher programming language isn’t significant for most desktop applications. Not even for mobile devices with not-so-much memory.

On Mono, the results would probably differ a little bit. But probably not a lot. I indeed hope people will join me on creating good .NET bindings for tinymail.

I proudly present you, a fully working E-mail client that supports IMAP, Exchange, POP (and their SSL equivalents) written in Python.

It’s build on top of tinymail. It supports going online and viewing received information offline. It supports refreshing the folder asynchronous. It has a progressbar that shows the status of the worker thread that is getting the new message information. It can show E-mails in HTML. Proved using an enormous amount of valgrind testing and if you use a Camel build by camel-lite-builder, it consumes less then ten megabytes of ram when displaying a folder that is larger than 15,000 messages.

Its code (the Python one) is 64 lines long using only four event handlers: on folder select, on header select, on status update and on folder refreshed. This is proof that the tinymail API kicks ass. Updated gtk-doc reference API documentation will be online very soon.

The design of the tinymail framework allows for ultimate flexibility. You can reinvent and reimplement every tinymail type without even having to rebuild for example the Python bindings. Without losing ABI nor API compatibility (but there’s no release, so this doesn’t mean a lot yet. After a first release, it will mean a lot). You can do this for both underlying infrastructure (such as connectivity with the E-mail service) and user interface infrastructure (such as a component for viewing a message). For example: the design of tinymail allows you to (re)implement the user interface implementations using Qt or Win32.

A screenshot:

The bindings are working. Nevertheless I hope other people will be inspired to go ahead and make the Python bindings of tinymail even better. Because now there’s Python bindings, there’s a chance that your work would be used by the One Laptop Per Child project. For me personally, that alone is enough fuel. Once I know more, you know I will blog about it :-).

Words? who needs words when he has a picture?

This is indeed the 42 lines Python E-mail client on top of tinymail.

I hope this way I inspired Python developers to join the fun of making the Python bindings for tinymail the best humanly possible. My appreciation to all the people in #pygtk who helped me the last three days. Especially Gustavo J. A. M. Carneiro, James Livingston and Johan Dahlin. Thanks to the #pygtk people for not kicking me because I flooded their IRC channel with questions and pasted code.

I know I don’t make it my public secret that I’m not a Python evangelist. As a programming environment and language, I don’t like Python nor PyGtk+. It’s not my programming language. I’m more the .NET, D and Java developer when it comes to higher programming languages.

But that doesn’t mean I don’t love Python developers. I do! I decided to prove it by implementing the Python bindings for tinymail.

There’s still a few minor and solvable issues like that the binding generator doesn’t yet generate the stuff needed for a typedef to a function pointer. You know, like the GFunc for Gtk+. Sooner or later, I’ll figure out how to do that. Or maybe you, the experienced Python code warrior, can help me? The current resulting .so files also aren’t tested yet. There’s definitely still some work to do.

Other than that, are bindings for libtinymail, libtinymailui, libtinymailui-gtk and the platform library you selected being generated automatically. Just add –enable-python-bindings to the configure line.

You can find the overrides and makefile here.

So far, two people told me they are interested in doing the .NET bindings. They are of course encouraged to do so.

The Power Of Nightmares 1/3

The Power Of Nightmares 2/3

The Power Of Nightmares 3/3

Yesterday I told you guys that I didn’t yet implement a account store that doesn’t use GConf, right?

Here’s the one for OLPC that uses gkeyfile: libtinymail-olpc/tny-account-store.c

I also improved camel-lite-builder so that it no longer links with GnomeVFS (nor ORBit-2 nor Bonobo). I don’t know whether the final OLPC device will ship with GConf and the D-Bus version of GnomeVFS. It no longer matters for tinymail.

By the way, great blog post by Mukund where he explains how we, human species, are in fact nothing but tests that test whether some newer genetic instructions successfully work or not.

Guys, remember I promised that I was not going to post new photos on my blog until I have one with tinymail running on the OLPC mobo?

Well, I will need my blog-space for posting pictures soon. So here’s a picture of tinymail running on the OLPC mobo!

I haven’t yet implemented a account store that doesn’t use GConf for reading the account configuration. Nor have I already added a patch for Camel in camel-lite-builder that removes Camels GnomeVFS dependency. Since a month or so, Camel depends on GnomeVFS, which at this moment pulls things like Bonobo and ORBit-2.

So .. I don’t think it’s a good idea not to patch it out of Camel before starting to use Camel on devices like the OLPC laptop. The OLPC image indeed doesn’t seem to come with libraries like GConf, Bonobo and ORBit-2. Other than that, it’s a very easy device to target. It’s no surprise that tinymail runs on it.

There’s of course a gnome-vfs-dbus. Very interesting of course. Might also be a solution for this. Well, the current hack for getting it to run was simply removing the two files from Camels Makefile.am.

I just uploaded the new tinymail.org website.

The small size of the web-design is indeed a hint that the website can also be used on mobile devices, who are a target of the tinymail framework.

Some of the website content might change as some people will probably give me advises. The purpose of this website is to be a view for also the commercial world. The idea is indeed to encourage commercial activities around the LGPL tinymail framework.

I’m also planning to make a website with a much more technical point of view. Being a website for a development framework, it will include developer documentation like the API reference, class diagrams, demos and guidelines.

I hope people understand that I only have two hands and that making such documentation takes time. I am, however, committed to do it. There’s already an API reference which will of course very soon be updated. Using the source code from the Subversion repository you can of course rebuild the API reference yourself.

Things like unit tests and documentation have a high priority and some of it is already available. I will probably not release unless both are perfect.

I will of course welcome people who want to help me creating or improving developer documentation, unit tests, design, implementation, a first release and the website.

I hope you will hear about Modest very soon. And yes, I was told they are designing the Modest user interface in such a way that it will be possible to use it on both a desktop and on a mobile device.

It is my opinion that using our blogs and the Internet being our speech instruments, we should somehow start organizing a huge anti-war protest.

Israel is clearly targeting civilians and civilian infrastructure. I condemn these actions and will no longer feel any compassion whatsoever for Israeli soldiers that decide to fight this war for their government. I can not agree with Hezbollah for launching missiles on Israeli cities. I can understand, however, that Lebanese people (like Hezbollah) no longer agree with what Israel has been doing the past years and during the last 12 days.

This way Israel is creating new hatred and aggression against themselves. Question yourself: what would you do if another country bombs your village killing your woman, baby, kid and family members? Imagine finding your kids head and other body-parts in a bloody bombed street or building. Would you not hate the people that did this? Western people live behind their CNN screen. Enjoying “television war”. It actually makes me sick that so many of us, Westerns, have lost reality with what war really is (please, for once: look at it. Even if you are frightned of pictures showing dead children. Look at the reality. See it. Understand it).

Our grandparents fought World War I and II. They learned us, the generations after World War II, never again to go to war. They died for our peace. We, young people, might never understand what they have seen. But please respect them. War is never a solution. War against citizens, what in my opinion Israel is doing, is evil.

It is insanity. I refuse to call this human. We, Europeans, do not want this. If our European politicians would want this, I would tell them here and now that I no longer call myself a citizen of their European union. Luckily so far most of them have been condemning Israel and their recent actions. I hope this will continue. I hope everybody who makes himself guilty of war crimes (both Hezbollah and Israeli) will get severely punished for these crimes against the human race (almost 400 unarmed citizen after only 12 days of war, 350 of them Lebanese people).

I hope the Nuremberg trials will repeat themselves (but please read the interesting 5th comment of Max, I also agree with that).

People who are going to reply with the “terrorist” word in their comment, please read this first. Maybe you should stop being so naive and learn about indoctrination using fear. You might understand what this American president is really doing with his “terrorists are everywhere”-speeches.

To world leaders and the UN: if American leaders show no interest in peace, please no longer consider their opinion to be of any value. Proceed with your peace plans without American support.

I’m guessing most people know these lounge cafes where they have relax music and where the waiter writes down your order on a PocketPC like device. About the PocketPC, in some such cafes they do that already, in others they don’t. If the one in your city doesn’t, maybe dancings or Café Del Mar style cafes do?

This device could be a Linux device that uses GNOME components. Why does it have to be Windows CE? Oh, it definitely at this moment most of the times is using Windows CE. Very often with an application done in Visual Basic Embedded or maybe Compact Framework .NET.

What is the business interested in? Selling cocktails, spirits and other fancy drinks that are very expensive. The customer expects such drink to arrive fast and oh, let us not forget: in total style. The bar guy constantly does tricks like throwing bottles around, The waiter, all dressed up chic (or maybe even naked at some places), brings you a nice big cocktail with a lot fruits, toys and other crap. There’s one word for it, or maybe two: style and eye candy.

The PocketPC probably doesn’t help a lot with making it faster. In dancings where they often also use them, it of course does. But it makes it more fancy. Fancy sells. Don’t underestimate it.

Apart from being fancy, what does this device needs to do?

• Take the order in a very fast way;
• Write down the table number;
• Send to order over wifi to the bar guys computer;
• The computer of the bar guy needs to show information about the cocktail. Ingredients and stuff like that;
• For customers that ask, the PocketPC needs to show a nice big picture of the cocktail so that the waiter can (in a fancy way) show it to the customer;
• Show the price and print the ticket in a few ways. For example a ticket per person, per drink or a ticket per table. Yes, you have portable printers and they look VERY fancy. It’s totally accepted by the owner of the business. He’s the guy who is going to pay for your consultancy, right? Listen to him!
• Get updated every month, about new items on the menu of the place (or do you want to make a locally served website maybe? Might work, might not work)

Does the device need addressbook functionality? Does it need contactbook functionality? Does it need E-mail functionality? Messaging functionality? Voice over IP? All that might be true! It depends on the business activity. On how the people who work at that place interact with each other. Maybe does the bar guy wants to send a message to the waiter like: “we don’t have the ingredients of this cocktail, can you ask the customer whether he wants it without that ingredient at a discount prince?”. Can your solution cope with that event?

The point that I’m trying to make is, choice and change. Today the owner of the business starts using your solution. They will definitely find defects in the system like: “what if we don’t have the ingredients for the cocktail and the user wants to cancel the order or pick another drink with a different price?”, “we can’t undo it using the device today, the waiter has to go to the main computer and that takes needlessly time”. The situation changes. Can your solution cope with that? Can the situation be avoided? And how?

Have you thought about how our GNOME desktop components needs to (cope with) change for them to be suitable for such application domains? What about tomorrow? How can we make the Linux device less expensive than the Windows CE device? Are only experienced GNOME developers going to write a gazillion of such applications? No, it will be nine to five developers that don’t care about C. How can we attract them to our platform?

One thing the waterfall model learned us is that we CAN NOT catch all use-cases in a first run. Be agile. Be adaptive to change. Be flexible. Develop in style, and you will be.

Here are the patches to get the upstream version of evolution-data-server and evolution-exchange to start using the mmap technique for loading the header and content info summary data of Evolution and tinymail.

I expect it to reduce memory usage of Evolution with approximately fourty to sixty megabytes of ram, depending on the amount of folders you have.

• evolution_data_server__mmap_summary.diff
• evolution_exchange__mmap_summary.diff

I’m very interested in your test results:

cvs -z3 co evolution-data-server
cd evolution-data-server
patch -p0 < evolution_data_server__mmap_summary.diff
./autogen.sh --prefix=/opt/eds-mmap && make && sudo make install
evolution --force-shutdown
export LD_LIBRARY_PATH=/opt/eds-mmap/lib
evolution (and wait a few minutes to let the summary files get build)
evolution --force-shutdown
valgrind --tool=massif evolution

Please do this procedure with and without the LD_LIBRARY_PATH and measure the difference. If you have any technical questions about it, feel free to ask me.

This version of the mmap patch for Camel is finally working on the Nokia 770 device (on an ARM cpu which needs aligned memory access).

For people who might not know what that means, it means that tinymail can open very large folders (I expect > 20,000 headers) easily on the device. I’m testing it now.

While doing these mmap patches for Camel, I had to leave the OLPC mobo untouched. I decided to undo that and install it so that it will be powered on (the idea here is that this way it will push me to give more attention to it). I’ll just put a promise here so that I can’t escape it: my next picture will be tinymail running on the OLPC mobo.

Just look at it! How cute! In a box :-)

Hey Nokia (or OLPC or whoever really wants it)! When will you guys pay Collabora (or somebody else) to do Python bindings for tinymail :-) ? The API is more or less ready for this (the API will definitely change a little bit. But not aggressive anymore. I Hope). If somebody else is interested, don’t let me saying somebody else is planning to do it stop you. Go ahead!

Very interesting read. Don’t just read the soldiers letter. Also read the reply beneath it.

Because

One, I share the opinion with Zaheer Abbas Merali that what Israel is doing is wrong, that I hope the international community will indeed condemn Israel’s actions and that the people in Lebanon and Palestine are getting a oppressive treatment by Israel.

Two, it’s my opinion that a personal blog, being syndicated on planet.gnome.org or not, is a free-speech instrument that definitely can be used for making political statements and expressing opinions. That what we, software developers working on GNOME software, write in our personal blog should not be censored because we get to come on a popular blog aggregator like planet.gnome.org.

We are very much allowed to write anything we want in our blogs. Knowing them personally, I’m very sure every single person in charge of planet.gnome.org agrees with that. I’m sure all of them also know that if they wouldn’t allow us to write anything we want, very few GNOME developers would want to be syndicated on planet.gnome.org. Nor had any of them, as far as I know, ever had the intention to forbid any such thing.

To make sure I piss off everybody who made these comments to Zaheer, I repeated his opinion and made sure it gets on the planet.gnome.org aggregator again. I strongly disagree with the opinion of anybody who says that we, GNOME developers, shouldn’t be allowed to write anything at anytime on our own personal blogs. I will also fight for my right to say whatever I want. Whenever I want. Wherever I want. As long as I don’t intend to discredit somebody or some group by making false statements (like unproven racist statements or telling lies about somebody to destroy his or her reputation).

If you cannot grasp that, you definitely have a huge culture difference with me. Maybe you shouldn’t use the Internet if you don’t understand this? The Internet is, for me, the free-speech instrument of this century. If necessary, I will strongly defend the free opinion and free speech of Zaheer.

“The greatest threat to freedom is the absence of criticism. — Wole Soyinka”
“I may disagree with what you have to say, but I shall defend, to the death, your right to say it. — Francois Marie Arouet Voltaire”

Camel with the mmap patch and the latest tinymail on your Nokia 770.

Note to Ross: there’s also a version of the patch (in the tarball) that doesn’t yet include the mmap patch and that will update the Camel of your eds-dbus. What do I do to get this committed in your Subversion?

If you want to build it yourself:

~$ cd /scratchbox/users/$username/
$ svn co http://svn.o-hand.com/repos/eds-dbus/trunk eds-dbus-mmap
$ svn co https://svn.tinymail.org/svn/tinymail
$ cd eds-dbus-mmap; patch -p0 < eds-dbus-mmap-camel-update.diff
$ cp e-data-server-utils.c e-data-server-utils.h libedataserver/
$ /scratchbox/login
[sbox-SDK_ARMEL: ~] > cd eds-dbus-mmap
[sbox-SDK_ARMEL: ~] > ./autogen.sh --prefix=/opt/eds --with-soup=no \
	--enable-camel=yes --with-dbus=yes
[sbox-SDK_ARMEL: ~] > make && make install
[sbox-SDK_ARMEL: ~] > cd .. ; cd tinymail/trunk
[sbox-SDK_ARMEL: ~] > export PKG_CONFIG_PATH=/opt/eds/lib
[sbox-SDK_ARMEL: ~] > ./autogen.sh --prefix=/opt/tinymail --enable-gnome=no \
	--with-platform=maemo --with-html-component=none
[sbox-SDK_ARMEL: ~] > make && make install
[sbox-SDK_ARMEL: ~] > cd /opt
[sbox-SDK_ARMEL: ~] > tar zcvf tinymail-maemo.tar.gz eds/ tinymail/

On your Nokia 770 device:

- xterm
- sudo gainroot
- cd /opt
- tar zxvf /media/mmc1/tinymail-maemo.tar.gz
- exit
- Run your account-create script
- /opt/tinymail/bin/tinymail

Update: sadly there seems to be something unstable about using mmap() on the Nokia 770 itself (at this moment, the kernel often kills the process). In stead of the eds-dbus-mmap-camel-update.diff you can also use the eds-dbus-fread-camel-update.diff. Tinymail will then not use mmap(). In scratchbox, mmap(), however, does work correctly. I will check with Nokia what this mmap() issue is about. I’ve put an eds that doesn’t use mmap() online here. On your device untar it in /opt (overwriting the /opt/eds of tinymail-maemo.tar.gz).

This new eds is also (cause of other issues that I’ve found while I was developing the mmap() patch) improved a lot in terms of memory usage. Using this new eds, showing 14,500 headers is now possible on a Nokia 770 with tinymails demo-ui. With the mmap() patch working on the device, it will be possible to display much larger folders.

I might have put binaries online. But be advised to always build from source until I do a release. Whatever is in Subversion is always going to be much more stable than whatever alpha/beta or unofficial binary I make and put online.

Update: You can get a version of both in one tarball here.

Chris is doing more crazy things with Dates, tinymail and Contact

I have this funny feeling this version of the patch for the mmap summary stuff is actually really working. I’ve been running Evolution with it for two hours now. I’ve let it rebuild all of my summary files and opened all my imap folders a few times.

I wrote a page that explains the Camel stuff that I have been blogging and mailing (like a crazy horse) about.

ps. Performance people: if you are technically interested, I can definitely use your help. There’s still a lot to do. We can reduce the memory usage even more (mainly for online usage of Evolution). It will also help tinymail with doing its initial load of a large folder. More info about this in the article and by simply asking me.

I’ve been fixing bugs and implementing the provider specifics for summary files in all default Camel providers. On my Evolution, this version of the patch actually worked without any strange crashes.

You simply apply the patch to evolution-data-server/camel, you build e-d-s in /opt/camel and you set LD_LIBRARY_PATH to /opt/camel/lib before launching evolution –force-shutdown followed by evolution. You can of course also use tinymail after setting the library path. Tinymail is the real reason why I did this patch.

As I warned in my previous blog, it will rewrite all summary files (which takes some time). If you restart evolution without setting the library path, Evolution will again rewrite all summary files using the old format. So nothing really dangerous happens. But still make a backup of your evolution data directory.

If you are online, Evolution still consumes a lot memory. That’s because of memory problems in the Camel provider implementations that scan for new messages. I hope I (finally) inspired people to take a look at these problems. If these are also fixed, Evolution will on average (I think) consume less then 50MB. Try Evolution (with the patch) in offline mode and measure it with valgrind. You’ll see.

In general, visit my files and check for a new version of the patch before applying anything. The filename is appended with fixesXX.diff