Month: September 2006

Hey guys, howcome nobody told me about this one?

Cool. Thanks for the article Dirk-Jan.

No RedBull this time. Somebody more or less asked it. Poor you guys.

Returning a doubly linked lists is going to put your library at odds with higher programming languages like Java and .NET that don’t really have a corresponding type that can rapidly be converted in two directions from and to C and the higher programming language.

But who said that first of all, you need to return it. And second of all, that you can’t wrap it up in a GObject or GTypeInterface? I for sure didn’t.

So that is what I did for tinymail. I created both a TnyList and a TnyIterator interface. I’ve let my GtkTreeModel implementations also implement TnyList and created TnyIterator implementations for them.

The methods that give me a list of things, simply ask for the TnyList instance to fill up. Rather than return it (it could of course also return the one that it got from the parameters).

If now a language binding language binds the TnyList and TnyIterator API, just like other GTypeInterfaces and GObjects, everything will works just fine. Most language binding code generators do this automatically. If the proxy type being used by that binding implements the higher programming language’s list and iterator interfaces, the typical foreach and iterator patterns will work with it.

It’s one of those simple solutions for simple problems that very few people seem to tackle. Most people simply return a doubly linked list and that way burden the language binding creator with the task of supporting the method somehow in a typical C way internally in the proxy type.

A GObject really doesn’t consume a lot bytes. And you gain the freedom of how you’ll implement the list. You can still use a GList or GSList internally and enjoy the performance of that. You can also implement it as a btree or a skiplist perhaps. Depending on your mood, performance needs and the change that is a constant in the IT industry. Abstracting the list and iterator types using GTypeInterface makes your software more adaptive to changes. More flexible.

Oh and Jason, you are hereby kindly invited to join the development of tinymail. A framework for rapidly creating E-mail clients for mobile devices, embedded appliances and soon also web applications.

It having a focus on these targets doesn’t mean that with one extra developer, I can’t focus its development toward a framework (also known as the beginnings of) for a desktop E-mail client. In fact, Modest, a E-mail client being build on top of tinymail by Nokia, is probably going to be a desktop E-mail client for a lot people. After that (being under NDA with Nokia I can’t tell you more), the plan might be (hehe), who knows, to bring it to the Maemo platform too.

It’s likely (grin) that somebody is fixing it to work with the recent API changes which I have been introducing the last few months. These changes and my commitment not to release until all obvious API changes, documentation, unit testing and at least Python language bindings are fully working have been delaying such a release. A Modest release is also delaying that release, as the current idea is to in parallel with Modest, release the first tinymail version.

But then again, people tell me I code tinymail faster than my shadow. I even think that’s a direct quote from Florian (or it was something like that). I guess getting at a first version faster is therefore not really healthy for me anymore. By that I mean: I can use your help (especially if you want it faster).

The development pages of tinymail go in detail about what can be done and what should be done before a first release. They even explain development techniques and details. They also explain how to debug and measure memory usage correctly. In other words, there’s no excuse possible like: “but but, there’s no information about the code”. Lots of such information has been added recently. I often add more.

Oh and if you like Sylpheed, maybe you and me can investigate whether or not it’s doable to extract a libsylpheed out of Sylpheed and use that for implementing a libtinymail-sylpheed as a replacement for the libtinymail-camel? That would indeed be possible, yes. Or maybe a libtinymail-akonadi?

I stopped at a gaz station while I was in a traffic jam, bought myself a can of RedBull, and cause of that I basically opened my laptop and started hacking like a nut. Again.

Another consequence of the RedBull is this blog item.

Whenever we design or redesign a platform, be it a development platform or a user platform, both what GNOME is all about, we should always take into consideration that we aren’t really doing it for just ourselves. Nor are we doing it for just the current generation of people who are active as user or developer.

When redesigning or thinking about a new platform, we are also (and probably mostly) doing it for the next generation of people. The dudes and dudettes who are at this moment adolescents and who will in a near future (or have already) started to create a passion for computers and software development.

If we are going to decide now that a GNOME 3.0 should be as API stable with GNOME 2.x as possible, we are going to give them what is current today. Not what will be current when their tomorrow happens. Therefore I disagree with Johannes a little bit that we shouldn’t make (big) API changes.

We should make the platform of tomorrow. If that means breaking things, well, that’s why we need to increase the major version number. That way, people know that it’s different. Does this mean that we need to change everything like a nut would? No. However, in my opinion it also shouldn’t mean that we should be careful about not changing any API. If existing API is wrong or even a little bit suboptimal, then a GNOME 3.0 should fix that.

(In my opinion) The GNOME platform shouldn’t be called 3.0 for marketing reasons. If nobody is yet thinking about the platform of tomorrow, it’s my opinion that we should stick to the 2.x naming. I don’t think nobody is. Let those people use that 3.0 as place where they can put their experiments and innovative ideas. Let us not block them for another x years. The GNOME platform does need this young and innovative talent.

Let us keep GNOME young. And do as much as possible to attract young fresh minds and ideas. Of course, mix theirs with the existing experience. It will be a win-win situation. Which is, in my opinion, what free software is all about: sharing and passing knowledge. Experimenting. Etc.

I have very few to say about the quote. Except that any country that sends out such messages, is being ruled by sick individuals. That out of principle, I refuse to visit it.

I remember Mr. Bush recently told a reporter something like: “Men … my job is to protect you, to protect America”. What a piece of bullshit. “US bombs the world”, “To protect America”.

Maybe should the US start to “listen” in stead of “bombing”, “to protect America”. But don’t count on them listening to this advise. Words don’t sell, weapons do. Lives don’t matter, money and power does. Let’s make sheep people afraid of the damn damn terrorists! And call every country that isn’t 100% pro us, the terrorists. The so-called axis of evil! What a piece of bullshit.

In the eyes of Mr. Bush, I’m probably a terrorist. I blog terrorist things. Let’s bomb me. Terrorist terrorist terrorist. Everwhere terrorist. Ridiculous. I call the way how Mr. Bush calls everything and everybody a terrorists, a pure disrespect for the victims of for example events like 9/11. As if their death can be used to sell his agenda to frightened people. It’s actually embarrassing for many of the fine people who have to live under his rule.

Fear, it’s your only God. Don’t let some dude easily feed it. Don’t let some dude abuse your fears to gain more power. Teach your friends and family to independently think. To vote intelligent, in stead of steered by fears.

I would like to mirror a comment on my previous blog entry so that people in the US would at least see how their “democratic” decisions are having serious consequences on non-US peoples lives.

I wouldn’t agree if the commenter meant that US citizens are indeed fully responsible. I would, however, agree if he meant that US citizen are at least partially responsible for what is the result of US foreign policy.

It’s easy to throw the guilt at your government. But you are the people who should protest; You are the people who shouldn’t have voted for this government; You are the people that, while very few non-US people on this planet believed that it was possible, reelected this absurd current US government; You are the parents, brothers and sisters of soldiers that agree to fight absurd wars on behalf of their equally absurd governments; And YOU mister US-solder, are the person who pulls the trigger. In this case it where the Israel proxy soldiers pulling that trigger. I don’t see a significant difference. Both types of soldiers are murderers. Both get or got their highest orders from US politicians (don’t tell us the US had nothing to do with it, we are not idiots).

The comment:

As a Lebanese and an Arab having to deal daily with the results of the US foreign policy, I totally appreciate and salute your stance.

However, I feel that the US citizens (GNOMErs or otherwise) are responsible for the US foreign policy killing hundreds of thousands of people. They are responsible in the sense that they brought the current or previous governments to power and they have the power to change or protest their government’s actions since they live in a so called democracy.

Venezuela’s leader Hugo Chavez, said Mr. Bush promoted “a false democracy of the elite” and a “democracy of bombs”. I agree. Hugo Chavez said “He (Mr. Bush) came here talking as if he were the owner of the world”, I agree. “The UN in its current form doesn’t work”, he said. I agree.

I will not visit the Boston Summit because I’m against its country’s foreign policy. Try again when you guys have real and good leaders in Washington. I know the people organizing the summit aren’t responsible for American foreign policy. Their non-guilt isn’t the point. The point is philosophical: I can’t visit a country that doesn’t respect human rights, is directly responsible for thousands of innocent dead and of which its politicians think that they own the planet. They don’t. For me, visiting the country wouldn’t feel right.

My apologizes to them who wanted to talk with me. At GUADEC I hesitated because the group of people representing GNOME are great. It nearly overruled the problems I have with visiting the US. Recent happenings in the world, however, convinced me I shouldn’t visit the country. I simply can’t.

Xen might be very promising, yet this type of features I simply haven’t seen or successfully used with it:

The thing is that someday I want to try porting Tors Win32 Gtk+ work to WinCE. It’s definitely not a promise. However, I did start installing the environments that I would need for this. Among them are, for example, Visual eMbedded C++ 4.0, Visual Studio Embedded 5.0 and because you often also need it a Visual Studio .NET.

Surprisingly Microsoft gives away some of these products. The eMbedded stuff, for example, is free (not free as in free speech, only free as in priceless). For the others, like the operating system and VS.NET, I have access to volume licenses from my employer. Lucky me.

I first did a clean Windows XP Pro install in a VMWare Workstation 5.5.2 virtual machine instance, I upgraded the thing with SP2 and let the “Windows update” tool run over it a few times. This is indeed a few hours work just to get this done. We Ubuntu (Dapper) users are spoiled indeed (enjoying the wonders of apt-get install gnome-devel). First thing I did after I was finally done, was to create a snapshot of the VM instance.

Then I created a linked clone and called the clone “Windows XP pro with VS.NET”. I rebooted the clone, renamed the hostname, and installed VS.NET in the new instance. I created another linked clone and called that one “Windows XP pro with 2003 mobile”. I installed a bunch of eMbedded C++ 4.0 related tools in it. Like an SDK for PocketPC.

The result? The result is that I now have a setup where it’s easy to clone a fresh copy of a virtual machine that has everything I’ll need for testing and building my WinCE stuff. I can experiment, rollback, snapshot and clone stuff. I can archive them. I can very rapidly suspend and unsuspend the virtual machines (not the operating system, the entire virtual machine and its full state). And what is really great, is that it’s not consuming a lot disk space! Only ~10GB for all these installations together. That’s mostly because I used the “linked clone” feature of VMWare Workstation 5.5.2.

Using a COW device with LVM, something like this is probably also doable with Xen. But let’s agree that we don’t have these fancy ChipX86 wizards that do all the dirty stuff for me, for Xen, right? I also wouldn’t have the nice desktop integration. A terminal server client just doesn’t always cut it. For example, I want to resize the VMWare window and the guest OS should automatically adapt its resolution. Click on a button and have my USB stick bridged to the guest OS. Same for bridging the USB cabled connection with my mobile device so that Ms. ActiveSync which runs on the guest OS finds it. All that just works in VMWare. It’s amazing.

Anyway. That’s why I have to thank the people who are working at VMWare for this great piece of software. The software has already been worth my money, and will be in future. Nevertheless I’m of course also following up on Xen and other virtualization technologies. So far, however, VMWare hasn’t been beaten at the level of desktop integration and taking care of the type of features a software developer needs from virtualization.

Lots of respect for you guys at VMWare. An amazing piece of software you guys have made. Really impressive.

New tinymail development documentation (that might also be useful for software developers who are interested in contributing to GNOME related technologies, as most of the documentation explains the GObject type system which is also being used by a lot GNOME softwares):

• How to implement a unit test for tinymail
• How to implement a type (an interface) in tinymail
• How to implement a type by inheriting an existing one in tinymail

These are wiki pages which means that you (yes, you) are allowed to improve them. You can find more of this stuff here.

For all my non-pgo blog readers out there, check Federico’s blog about testing out. Very interesting papers he linked to.

Because I truly believe that documentation and unit testing is far more important than implementation, I have added micro samples and beautified the API reference manual of tinymail a lot. This makes tinymail not only a E-mail framework that helps you a lot with making sure you don’t consume a lot memory, it’s also thoroughly documented, has a nicely designed API, has a unit test for almost every significant type and has Python bindings.

And I’m definitely not done yet with the project. I will keep adding value to it as I’m not satisfied about it yet. It’s still missing something magic somewhere. I don’t know yet where, but I will find the magic and insert it in the project. I’m already thinking about supporting higher programming languages in all possible directions. Including making it possible and easy to implement types, that tinymail will consume, in your higher programming language. I’m already looking at Cilc and of course GAPI for this. The define-virtual of the Python language binding generator also does a nice job.

I’m also thinking about ways to implement query folders which will have features like a virtual folders in Evolution. The goal, however, is not to use as much memory consumption yet keep it possible to query all folders. Not just the active one. I’m also thinking a lot about reducing memory consumption even more. I’m considering implementing multiple libtinymail implementation libraries (at this moment there’s only a libtinymail-camel implementation).

And I’m thinking about supporting Qtopia, Windows CE and porting the dependencies and tinymail itself to targets like SymbianOS, QNX and eCos. Yes, I’m indeed planning to take over the world too.

The good thing about this fix for the tinymail Python bindings, is that I did it in a generic way this time. I created a filter.py script which you can pass a –topsrcdir=$(top_srcdir) option on its commandline. It will walk all folders searching for .h files that have GTypeInterfaces. The h2def.py of pygtk should also do this, but seems to fail on detecting whether or not it’s a GTypeInterface. I made the regex in filter.py a little bit less strict and piped the output of h2def through filter.py. The filter.py script also makes it possible to append extra defines to the resulting .defs file. That’s why you are seeing .defs.extra files in the directory with the Python binding of tinymail. The idea is that I wanted everything to be generated. No way I’m going to store .defs files in my repository and maintain that junk.

The API docs also had a little problem that my tny-shared.h file with its ugly forward typedefs obfuscated the real location of the type struct to the parser of gtk-doc. This has also been fixed by doing some #ifdef #endif tricks in the .h files of the types.

I also have this new idea for the mmap patch for Camel. Check the idea out if memory consumption interests you. Note that the defines for calculating the offsets are still wrong (even in the 3th attempt, I forgot the ‘* x’ after the last sizeof). Nevertheless should the idea itself be clear, no? If you wonder why this patch isn’t yet in upstream Camel and Evolution: ask the evolution team, not me. The patch is default in camel-lite-builder and will be used by tinymail releases.

At work, I know I don’t often talk about work and should do that more often in stead of only whining about tinymail, I started doing some kernel development. I’m implementing a reassembly algorithm for IP fragmented in ATM cells accompanied by a timestamp using sliding window. Funky stuff.

Tuesday, January 17th, 2006, I first mentioned that I was building this tinymail thing. Somewhere early that month I had announced the idea of implementing a better E-mail client for the Nokia 770. My idea was based around the proxy design pattern in combination with model view controller.

Today, almost nine months later, I proudly present you the birth of the API that actually really looks like what will be the API when I will release tinymail. I finally started caring enough about those “Iface” suffixes to remove them. I did the monkey work of renaming all the types that now conflicted with the interface name to have a detail prefix after the namespace thing. Now you have a naming scheme like this: TnyGpeDevice implements the interface TnyDevice which has a class struct called TnyDeviceIface. This is exactly like the API of GtkTreeModel interface which has GtkTreeModelIface as class struct and is implemented by GtkTreeModelSort, GtkTreeModelFilter, GtkTreeStore and GtkListStore. And in tinymail by TnyGtkAccountTreeModel, TnyGtkHeaderListModel and TnyGtkAttachListModel in libtinymailui-gtk

As a result of this monkey work, the gtk-doc API reference manual gets generated more clean AND people will actually stop whining about it. Which is great.

I only have one regression, and that is the Python bindings. The problem is that I generate the .defs files using some autotool tricks and the well known pygtk tools. The thing that reads the .h files doesn’t detect the difference between a GObject and a GTypeInterface it seems (I hinted the code generator using a regex that matched “.*Iface” on the type’s name, but that doesn’t work anymore of course). But I will probably fix this very soon (don’t worry, nor is the release ready yet anyway). Oh feel free to fix this for me if you are bored today. And here is a pointer to those bindings.

The following valgrind massif report was created using a standard tinymail from Subversion and a Camel build using camel-lite-builder from Subversion.

It’s using gslice (small leaks aren’t visible). It’s being ran on a x86 i386 architecture . This makes a significant difference if your pointer size isn’t four bytes: the type representing a summary-item — of which the visible subject, to and from headers are components — consumes a bunch of pointers per instance.

It’s using Gecko for displaying HTML E-mails which was first used around point C in the graph.

A detailed memory-analysis of tinymail can be found (and improved by you, as it’s a wiki) here. Note that the memory-analysis that you can find there, isn’t using gslice (and if you improve it, don’t use gslice for your tests).

Note that I indeed cropped the image a little bit. The removed pieces of the image aren’t (yet) very interesting.

Let’s iterate over the marked points:

• A is the start of the application. At this point aren’t the folders loaded, but the accounts are;
• B is when the folders are loaded and the demo-ui of tinymail is fully visible and usable for the user. Everything that doesn’t fully go back to this position can be considered being a leak. But be careful when thinking about leaks: during the graph several things are kept in memory. The worst memory consumer being kept in memory is Gecko which gets allocated at point C. Others are the folders, the accounts and Gtk+ components and caches.
• Between B and the start of C is the point where I opened my INBOX folder of ~ 1,000 messages.
• C is when I first clicked an E-mail that contains a “text/html” mime-part. As a result of that, a GtkMozEmbed widget (this is a Gtk+ wrapper for Gecko) is instantiated. The memory between C and D minus E is what Gecko consumes;
• What I did at D was that I clicked another folder. The HTML message and folder of B and C where unloaded. But because Gecko upon destruction keeps itself cached, you don’t see a major memory reduction here;
• The interesting part starts at E where I clicked a folder that contains no messages (My Drafts folder);
• F is when I reopened the first folder. You see all the memory go up and down completely. This is important of course (else I’m leaking things);
• At G/a I clicked on a NNTP folder. The folder was “C++” of the newsserver at digitalmars. What you are seeing between G/a and G/b is what I think is the Camel NNTP library being loaded and consuming some of the memory. But I can be wrong as I haven’t in-depth investigated this;
• H was when the folder was fully loaded;
• I was when I clicked on that Draft folder in my IMAP account;
• J was opening the “C++” folder of the digitalmars newsserver again;
• K is the Draft folder;

Tinymail is (slowly) getting more and more contributors and is increasingly becoming serious. I can’t open all details, but Nokia is indeed interested in tinymail. I decided to become semi self-employed for smaller projects and my full-time employer agreed to sell my consultancy for larger tinymail projects. Other companies that will help you with tinymail consultancy are Kernelconcepts and Collabora. If you or your company is interested to get on the support pages of tinymail, contact me. About politics: my philosophy about software doesn’t allow me to obey any employer in case he wouldn’t allow me to add your company on that page.

The license of tinymail will forever stay LGPL. Like its license will the spirit behind the project also always be a free software one. But I (its maintainer) isn’t dirty of selling consultancy around it. It’s okay when others do this too. I, however, encourage organizations (philosiphical and technical) to contribute their platform–specific code back into the LGPL project. If necessary (and only if necessary) I will dual license certain things for this. My opinion is that availability of the code and work is more important than religion.

Major (indeed major) memory reduction is still possible (mostly within Camel) and is being worked on. Support for non-Gtk+ platforms is on the agenda (and certainly isn’t impossible). Examples are WinCE and Qtopia. Support for those will, however, not be available at a first release. It’s simply too much work and it would delay such a first release significantly. It’s not that I’m politically not interested. I am technically interested and will put my own time in it if necessary. Also note that I don’t care about anti-Microsoft idiots who will condemn me when I’ll support WinCE (and I will). I propose them to write their own tinymail if that disturbs them.

That’s it for todays tinymail blog.