Author: admin

I would like to mirror a comment on my previous blog entry so that people in the US would at least see how their “democratic” decisions are having serious consequences on non-US peoples lives.

I wouldn’t agree if the commenter meant that US citizens are indeed fully responsible. I would, however, agree if he meant that US citizen are at least partially responsible for what is the result of US foreign policy.

It’s easy to throw the guilt at your government. But you are the people who should protest; You are the people who shouldn’t have voted for this government; You are the people that, while very few non-US people on this planet believed that it was possible, reelected this absurd current US government; You are the parents, brothers and sisters of soldiers that agree to fight absurd wars on behalf of their equally absurd governments; And YOU mister US-solder, are the person who pulls the trigger. In this case it where the Israel proxy soldiers pulling that trigger. I don’t see a significant difference. Both types of soldiers are murderers. Both get or got their highest orders from US politicians (don’t tell us the US had nothing to do with it, we are not idiots).

The comment:

As a Lebanese and an Arab having to deal daily with the results of the US foreign policy, I totally appreciate and salute your stance.

However, I feel that the US citizens (GNOMErs or otherwise) are responsible for the US foreign policy killing hundreds of thousands of people. They are responsible in the sense that they brought the current or previous governments to power and they have the power to change or protest their government’s actions since they live in a so called democracy.

Venezuela’s leader Hugo Chavez, said Mr. Bush promoted “a false democracy of the elite” and a “democracy of bombs”. I agree. Hugo Chavez said “He (Mr. Bush) came here talking as if he were the owner of the world”, I agree. “The UN in its current form doesn’t work”, he said. I agree.

I will not visit the Boston Summit because I’m against its country’s foreign policy. Try again when you guys have real and good leaders in Washington. I know the people organizing the summit aren’t responsible for American foreign policy. Their non-guilt isn’t the point. The point is philosophical: I can’t visit a country that doesn’t respect human rights, is directly responsible for thousands of innocent dead and of which its politicians think that they own the planet. They don’t. For me, visiting the country wouldn’t feel right.

My apologizes to them who wanted to talk with me. At GUADEC I hesitated because the group of people representing GNOME are great. It nearly overruled the problems I have with visiting the US. Recent happenings in the world, however, convinced me I shouldn’t visit the country. I simply can’t.

Xen might be very promising, yet this type of features I simply haven’t seen or successfully used with it:

The thing is that someday I want to try porting Tors Win32 Gtk+ work to WinCE. It’s definitely not a promise. However, I did start installing the environments that I would need for this. Among them are, for example, Visual eMbedded C++ 4.0, Visual Studio Embedded 5.0 and because you often also need it a Visual Studio .NET.

Surprisingly Microsoft gives away some of these products. The eMbedded stuff, for example, is free (not free as in free speech, only free as in priceless). For the others, like the operating system and VS.NET, I have access to volume licenses from my employer. Lucky me.

I first did a clean Windows XP Pro install in a VMWare Workstation 5.5.2 virtual machine instance, I upgraded the thing with SP2 and let the “Windows update” tool run over it a few times. This is indeed a few hours work just to get this done. We Ubuntu (Dapper) users are spoiled indeed (enjoying the wonders of apt-get install gnome-devel). First thing I did after I was finally done, was to create a snapshot of the VM instance.

Then I created a linked clone and called the clone “Windows XP pro with VS.NET”. I rebooted the clone, renamed the hostname, and installed VS.NET in the new instance. I created another linked clone and called that one “Windows XP pro with 2003 mobile”. I installed a bunch of eMbedded C++ 4.0 related tools in it. Like an SDK for PocketPC.

The result? The result is that I now have a setup where it’s easy to clone a fresh copy of a virtual machine that has everything I’ll need for testing and building my WinCE stuff. I can experiment, rollback, snapshot and clone stuff. I can archive them. I can very rapidly suspend and unsuspend the virtual machines (not the operating system, the entire virtual machine and its full state). And what is really great, is that it’s not consuming a lot disk space! Only ~10GB for all these installations together. That’s mostly because I used the “linked clone” feature of VMWare Workstation 5.5.2.

Using a COW device with LVM, something like this is probably also doable with Xen. But let’s agree that we don’t have these fancy ChipX86 wizards that do all the dirty stuff for me, for Xen, right? I also wouldn’t have the nice desktop integration. A terminal server client just doesn’t always cut it. For example, I want to resize the VMWare window and the guest OS should automatically adapt its resolution. Click on a button and have my USB stick bridged to the guest OS. Same for bridging the USB cabled connection with my mobile device so that Ms. ActiveSync which runs on the guest OS finds it. All that just works in VMWare. It’s amazing.

Anyway. That’s why I have to thank the people who are working at VMWare for this great piece of software. The software has already been worth my money, and will be in future. Nevertheless I’m of course also following up on Xen and other virtualization technologies. So far, however, VMWare hasn’t been beaten at the level of desktop integration and taking care of the type of features a software developer needs from virtualization.

Lots of respect for you guys at VMWare. An amazing piece of software you guys have made. Really impressive.

New tinymail development documentation (that might also be useful for software developers who are interested in contributing to GNOME related technologies, as most of the documentation explains the GObject type system which is also being used by a lot GNOME softwares):

• How to implement a unit test for tinymail
• How to implement a type (an interface) in tinymail
• How to implement a type by inheriting an existing one in tinymail

These are wiki pages which means that you (yes, you) are allowed to improve them. You can find more of this stuff here.

For all my non-pgo blog readers out there, check Federico’s blog about testing out. Very interesting papers he linked to.

Because I truly believe that documentation and unit testing is far more important than implementation, I have added micro samples and beautified the API reference manual of tinymail a lot. This makes tinymail not only a E-mail framework that helps you a lot with making sure you don’t consume a lot memory, it’s also thoroughly documented, has a nicely designed API, has a unit test for almost every significant type and has Python bindings.

And I’m definitely not done yet with the project. I will keep adding value to it as I’m not satisfied about it yet. It’s still missing something magic somewhere. I don’t know yet where, but I will find the magic and insert it in the project. I’m already thinking about supporting higher programming languages in all possible directions. Including making it possible and easy to implement types, that tinymail will consume, in your higher programming language. I’m already looking at Cilc and of course GAPI for this. The define-virtual of the Python language binding generator also does a nice job.

I’m also thinking about ways to implement query folders which will have features like a virtual folders in Evolution. The goal, however, is not to use as much memory consumption yet keep it possible to query all folders. Not just the active one. I’m also thinking a lot about reducing memory consumption even more. I’m considering implementing multiple libtinymail implementation libraries (at this moment there’s only a libtinymail-camel implementation).

And I’m thinking about supporting Qtopia, Windows CE and porting the dependencies and tinymail itself to targets like SymbianOS, QNX and eCos. Yes, I’m indeed planning to take over the world too.

The good thing about this fix for the tinymail Python bindings, is that I did it in a generic way this time. I created a filter.py script which you can pass a –topsrcdir=$(top_srcdir) option on its commandline. It will walk all folders searching for .h files that have GTypeInterfaces. The h2def.py of pygtk should also do this, but seems to fail on detecting whether or not it’s a GTypeInterface. I made the regex in filter.py a little bit less strict and piped the output of h2def through filter.py. The filter.py script also makes it possible to append extra defines to the resulting .defs file. That’s why you are seeing .defs.extra files in the directory with the Python binding of tinymail. The idea is that I wanted everything to be generated. No way I’m going to store .defs files in my repository and maintain that junk.

The API docs also had a little problem that my tny-shared.h file with its ugly forward typedefs obfuscated the real location of the type struct to the parser of gtk-doc. This has also been fixed by doing some #ifdef #endif tricks in the .h files of the types.

I also have this new idea for the mmap patch for Camel. Check the idea out if memory consumption interests you. Note that the defines for calculating the offsets are still wrong (even in the 3th attempt, I forgot the ‘* x’ after the last sizeof). Nevertheless should the idea itself be clear, no? If you wonder why this patch isn’t yet in upstream Camel and Evolution: ask the evolution team, not me. The patch is default in camel-lite-builder and will be used by tinymail releases.

At work, I know I don’t often talk about work and should do that more often in stead of only whining about tinymail, I started doing some kernel development. I’m implementing a reassembly algorithm for IP fragmented in ATM cells accompanied by a timestamp using sliding window. Funky stuff.

Tuesday, January 17th, 2006, I first mentioned that I was building this tinymail thing. Somewhere early that month I had announced the idea of implementing a better E-mail client for the Nokia 770. My idea was based around the proxy design pattern in combination with model view controller.

Today, almost nine months later, I proudly present you the birth of the API that actually really looks like what will be the API when I will release tinymail. I finally started caring enough about those “Iface” suffixes to remove them. I did the monkey work of renaming all the types that now conflicted with the interface name to have a detail prefix after the namespace thing. Now you have a naming scheme like this: TnyGpeDevice implements the interface TnyDevice which has a class struct called TnyDeviceIface. This is exactly like the API of GtkTreeModel interface which has GtkTreeModelIface as class struct and is implemented by GtkTreeModelSort, GtkTreeModelFilter, GtkTreeStore and GtkListStore. And in tinymail by TnyGtkAccountTreeModel, TnyGtkHeaderListModel and TnyGtkAttachListModel in libtinymailui-gtk

As a result of this monkey work, the gtk-doc API reference manual gets generated more clean AND people will actually stop whining about it. Which is great.

I only have one regression, and that is the Python bindings. The problem is that I generate the .defs files using some autotool tricks and the well known pygtk tools. The thing that reads the .h files doesn’t detect the difference between a GObject and a GTypeInterface it seems (I hinted the code generator using a regex that matched “.*Iface” on the type’s name, but that doesn’t work anymore of course). But I will probably fix this very soon (don’t worry, nor is the release ready yet anyway). Oh feel free to fix this for me if you are bored today. And here is a pointer to those bindings.

The following valgrind massif report was created using a standard tinymail from Subversion and a Camel build using camel-lite-builder from Subversion.

It’s using gslice (small leaks aren’t visible). It’s being ran on a x86 i386 architecture . This makes a significant difference if your pointer size isn’t four bytes: the type representing a summary-item — of which the visible subject, to and from headers are components — consumes a bunch of pointers per instance.

It’s using Gecko for displaying HTML E-mails which was first used around point C in the graph.

A detailed memory-analysis of tinymail can be found (and improved by you, as it’s a wiki) here. Note that the memory-analysis that you can find there, isn’t using gslice (and if you improve it, don’t use gslice for your tests).

Note that I indeed cropped the image a little bit. The removed pieces of the image aren’t (yet) very interesting.

Let’s iterate over the marked points:

• A is the start of the application. At this point aren’t the folders loaded, but the accounts are;
• B is when the folders are loaded and the demo-ui of tinymail is fully visible and usable for the user. Everything that doesn’t fully go back to this position can be considered being a leak. But be careful when thinking about leaks: during the graph several things are kept in memory. The worst memory consumer being kept in memory is Gecko which gets allocated at point C. Others are the folders, the accounts and Gtk+ components and caches.
• Between B and the start of C is the point where I opened my INBOX folder of ~ 1,000 messages.
• C is when I first clicked an E-mail that contains a “text/html” mime-part. As a result of that, a GtkMozEmbed widget (this is a Gtk+ wrapper for Gecko) is instantiated. The memory between C and D minus E is what Gecko consumes;
• What I did at D was that I clicked another folder. The HTML message and folder of B and C where unloaded. But because Gecko upon destruction keeps itself cached, you don’t see a major memory reduction here;
• The interesting part starts at E where I clicked a folder that contains no messages (My Drafts folder);
• F is when I reopened the first folder. You see all the memory go up and down completely. This is important of course (else I’m leaking things);
• At G/a I clicked on a NNTP folder. The folder was “C++” of the newsserver at digitalmars. What you are seeing between G/a and G/b is what I think is the Camel NNTP library being loaded and consuming some of the memory. But I can be wrong as I haven’t in-depth investigated this;
• H was when the folder was fully loaded;
• I was when I clicked on that Draft folder in my IMAP account;
• J was opening the “C++” folder of the digitalmars newsserver again;
• K is the Draft folder;

Tinymail is (slowly) getting more and more contributors and is increasingly becoming serious. I can’t open all details, but Nokia is indeed interested in tinymail. I decided to become semi self-employed for smaller projects and my full-time employer agreed to sell my consultancy for larger tinymail projects. Other companies that will help you with tinymail consultancy are Kernelconcepts and Collabora. If you or your company is interested to get on the support pages of tinymail, contact me. About politics: my philosophy about software doesn’t allow me to obey any employer in case he wouldn’t allow me to add your company on that page.

The license of tinymail will forever stay LGPL. Like its license will the spirit behind the project also always be a free software one. But I (its maintainer) isn’t dirty of selling consultancy around it. It’s okay when others do this too. I, however, encourage organizations (philosiphical and technical) to contribute their platform–specific code back into the LGPL project. If necessary (and only if necessary) I will dual license certain things for this. My opinion is that availability of the code and work is more important than religion.

Major (indeed major) memory reduction is still possible (mostly within Camel) and is being worked on. Support for non-Gtk+ platforms is on the agenda (and certainly isn’t impossible). Examples are WinCE and Qtopia. Support for those will, however, not be available at a first release. It’s simply too much work and it would delay such a first release significantly. It’s not that I’m politically not interested. I am technically interested and will put my own time in it if necessary. Also note that I don’t care about anti-Microsoft idiots who will condemn me when I’ll support WinCE (and I will). I propose them to write their own tinymail if that disturbs them.

That’s it for todays tinymail blog.

Some people worried that I was not blogging anymore. No, I have simply been very busy in my free time working on tinymail. Here is an update.

Since the last months, I’ve been proof of concept designing and implementing tinymail, a E-mail framework for mobile and embedded devices that also implements and defines lots of ready-to-use user interface components of most typical mail user agent. It does this in a highly adaptive and flexible way. Indeed using flash flash buzzword buzzword “design patterns” and other modern software development techniques like unit testing.

Since the last three weeks I feel that I started designing and refactoring tinymail for real use. I started getting the API (it’s updated indeed) and its design the way I really want it. I started to make sure that reference counting is absolutely correct. API and reference counting aren’t things which you want to change a lot once you started doing releases.

That’s why I have to make sure that both are right from the start, good. Some recent changes:

There used to be two APIs for getting folders from two types. They where not capable of running asynchronously. There was one API in a store-account type for getting the root folders. And there was one API in the folder type for getting child folders.

Every sane developer who cares a little bit about design, makes a folder-store interface and lets both types implement that interface. I did that for tinymail. I also added a query mechanism. Because sometimes you want folders that are subscribed, sometimes you don’t. Sometimes you only want folders that contain “comp.lang” (for example in case your account is for a NNTP server).

Just like when you are getting headers, I designed it by letting the developer pass his own list implementation. Returning a list is often the wrong thing to do: if you return a list, the implementation of that list is fixed defined by the framework (this isn’t adaptive nor flexible). In tinymail those lists are typically implemented by types that also implement GtkTreeModel. This means no silly copying items from one list-type to another list-type. And it means that the list itself can be the containing type that keeps a reference on its items. A doubly-linked list obviously can’t do that. A doubly-linked list is also a hell for language binding your framework to a higher programming language.

Think model view controller (you know, that *thing* that you learned at school during the Java course). The list instances that you will feed the tinymail framework with, will also be the list-models of your list-views. And you can implement those models yourself (the default libtinymailui library, libtinymailui-gtk, uses the GtkTreeModel API for this). The idea is that for example in .NET, you will use the IList type for this. The .NET bindings will indeed let the list-type binding proxies implement IList.

It’s now also possible to get folders asynchronously. You never know how slow your connection will be on a mobile device. Here’s a functional test that illustrates it. And this is a synchronous version of it. Both tests recursively walk all the folders and print them on your screen.

I finished most of it before I went on holiday, but there are now also fully functional Python bindings for tinymail. They make it possible to write a functional E-mail client in a few seconds. Three people are also working on .NET bindings.

Lots of work went into adding lots of unit tests. These are very simple to implement. Actually can everybody help with this. If you too want a great E-mail framework with standard user interface components for E-mail clients, join the mailing list and get in touch. Doing unit tests will of course get you familiar with the implementations. Most people have no idea how much a developer knows when a simple stupid test succeeds of fails.

I also added lots of pages to the tinymail development pages. They are wiki pages so you can change them if you have something interesting to add or if you want to correct something. A few interesting pages are: library image sizes, how much memory will tinymail use, debugging tips, memory testing and monitoring tinymail, features and facts, a list with todo items. New pages are added constantly. The webpages are the central communication tool of the developers (there are a few contributors indeed).

About a release. A month ago I promised some guys a release in two months or at the same second Nokia releases Modest, an E-mail client being build on top of tinymail. The iterations of changes are, however, going very fast. We’ll see. Not much normal users will care a lot about a release anyway. They are just waiting for an E-mail client that doesn’t suck. Tinymail is not their E-mail client. It’s a framework for building their E-mail client.

I don’t know how legal it is. But if you are going to Pukkelpop this year, and the Macromedia Flash print feature of the Pukkelpop website isn’t working, you can find a printable version of the line-ups here, here and here.

So there’s no more need to boot a Windows in your VMWare.

A few new pages on the tinymail development website: Development tips, Debugging tips, Memory testing, Refdbg and valgrind memory testing, Writing API reference documentation.

Feel invited to fix things like spelling errors. Or add interesting information.

These topics have recently been added to the development website of tinymail.

I’m going to take a break the next two weeks. So this is the perfect time to catch-up with what is current. By that I mean that the code is probably going to stay more or less stable for two weeks. As in: it will not change a lot.

You are of course invited to join the mailing list and discuss with the tinymail developers and contributors what you think the future of tinymail should look like.

Documentation

• Help building the marketing website
• Help building the development website
• Help writing documentation
• Help with release management

Design and coding

• Help by implementing unit tests
• Help by designing and implementing code
• Help by implementing language bindings

Redesign and refactoring plans

• Help by redesigning and implementing getting folders
• Help by renaming the interface types

Integration design and coding

• Help integrating tinymail with Mathusalem for progress indication
• Help integrating tinymail with Seahorse for PGP and authentication
• Help integrating tinymail with Telepathy and Galago for presence notification
• Help integrating tinymail with Dates and Contact

Testing

• Help by implementing unit tests
• Help my monitoring the reference counting
• Help by memory testing and debugging tinymail

The development website of tinymail is being shaped.

It has information about the mailing list, api reference documentation, how you can help, some samples, how contacting the developers, how to checkout and build the source code on your platform and device and some features and facts.

Feel free to check it out and make improvements.

This week will be a documentation week for tinymail. A normal XP iteration means meet, design, implement, document and test. I very often do the document part (mostly the API). I, however, never brought all that documentation together. There’s of course still a lot other things to document. For example making some sample code embedded in the documentation.

The API reference is updated and moved to its final location (this doesn’t mean that this is the final 1.0 API. It might still change until I actually do a 1.0 release). I have setup a trac website for tinymail and one for camel-lite-builder. These trac websites will function as the developers website for tinymail.

I discussed some ideas with Ivan Krstic about E-mail and the One Laptop Per Child project. The current idea is to make a local webserver that will serve webpages like, I guess, Roundcube does. But then much smaller in memory footprint. Which is why he’s interested in tinymail and its Python bindings. I will of course make sure that the tinymail framework can function perfectly in a web environment.

Christopher Blizzard then asked me to fill in this page on the OLPCWiki. It contains some information about tinymail on the OLPC laptop.

I hope people will help me this week with setting up all the wiki pages of the tinymail trac. Everybody is of course invited to improve things like the website (just wget -r it) or documentation or the trac infrastructure and its wiki pages. If you have an idea: please go ahead and edit it. I mean .. if I don’t like it, I can remove it right?

I also hope there’s going to be more people helping me with language bindings. Like improving the Python ones and implementing the .NET ones. The more of this stuff that I will have before making a first release, the better this first release will be. If somebody climbs the stage, I will definitely join him on this. If nobody does, it might be delayed until after my holiday: Tinne decided that I need a break next week.

A few days ago, on some mailing list, I wrote:

I actually but truly believe that running most GNOME programs under a garbage collector would actually save (a lot) more memory then the virtual machine would add.

In stead of just saying such crazy things, I should of course also prove it. It’s of course not easy to prove this. What I can do is show you two programs that do exactly the same. One program uses Python language bindings. The other doesn’t use language bindings but directly invokes the API in C using GObject.

The two applications loaded the same IMAP folder of ~1000 headers. And unloaded that folder.

The usage of Python bindings causes some Python proxy objects to be created. I think (but I’m not sure) that most of the extra memory that you will see is caused by these.

• The Python application
• The C application

In Python

In C

Consider that most desktop applications consume more than 20MB and that the test application (tinymail) is designed and developed with low memory consumption in mind, I still believe the overhead introduced by a higher programming language isn’t significant for most desktop applications. Not even for mobile devices with not-so-much memory.

On Mono, the results would probably differ a little bit. But probably not a lot. I indeed hope people will join me on creating good .NET bindings for tinymail.

I proudly present you, a fully working E-mail client that supports IMAP, Exchange, POP (and their SSL equivalents) written in Python.

It’s build on top of tinymail. It supports going online and viewing received information offline. It supports refreshing the folder asynchronous. It has a progressbar that shows the status of the worker thread that is getting the new message information. It can show E-mails in HTML. Proved using an enormous amount of valgrind testing and if you use a Camel build by camel-lite-builder, it consumes less then ten megabytes of ram when displaying a folder that is larger than 15,000 messages.

Its code (the Python one) is 64 lines long using only four event handlers: on folder select, on header select, on status update and on folder refreshed. This is proof that the tinymail API kicks ass. Updated gtk-doc reference API documentation will be online very soon.

The design of the tinymail framework allows for ultimate flexibility. You can reinvent and reimplement every tinymail type without even having to rebuild for example the Python bindings. Without losing ABI nor API compatibility (but there’s no release, so this doesn’t mean a lot yet. After a first release, it will mean a lot). You can do this for both underlying infrastructure (such as connectivity with the E-mail service) and user interface infrastructure (such as a component for viewing a message). For example: the design of tinymail allows you to (re)implement the user interface implementations using Qt or Win32.

A screenshot:

The bindings are working. Nevertheless I hope other people will be inspired to go ahead and make the Python bindings of tinymail even better. Because now there’s Python bindings, there’s a chance that your work would be used by the One Laptop Per Child project. For me personally, that alone is enough fuel. Once I know more, you know I will blog about it :-).

Words? who needs words when he has a picture?

This is indeed the 42 lines Python E-mail client on top of tinymail.

I hope this way I inspired Python developers to join the fun of making the Python bindings for tinymail the best humanly possible. My appreciation to all the people in #pygtk who helped me the last three days. Especially Gustavo J. A. M. Carneiro, James Livingston and Johan Dahlin. Thanks to the #pygtk people for not kicking me because I flooded their IRC channel with questions and pasted code.

I know I don’t make it my public secret that I’m not a Python evangelist. As a programming environment and language, I don’t like Python nor PyGtk+. It’s not my programming language. I’m more the .NET, D and Java developer when it comes to higher programming languages.

But that doesn’t mean I don’t love Python developers. I do! I decided to prove it by implementing the Python bindings for tinymail.

There’s still a few minor and solvable issues like that the binding generator doesn’t yet generate the stuff needed for a typedef to a function pointer. You know, like the GFunc for Gtk+. Sooner or later, I’ll figure out how to do that. Or maybe you, the experienced Python code warrior, can help me? The current resulting .so files also aren’t tested yet. There’s definitely still some work to do.

Other than that, are bindings for libtinymail, libtinymailui, libtinymailui-gtk and the platform library you selected being generated automatically. Just add –enable-python-bindings to the configure line.

You can find the overrides and makefile here.

So far, two people told me they are interested in doing the .NET bindings. They are of course encouraged to do so.

The Power Of Nightmares 1/3

The Power Of Nightmares 2/3

The Power Of Nightmares 3/3