Author: admin

For all my non-pgo blog readers out there, check Federico’s blog about testing out. Very interesting papers he linked to.

Because I truly believe that documentation and unit testing is far more important than implementation, I have added micro samples and beautified the API reference manual of tinymail a lot. This makes tinymail not only a E-mail framework that helps you a lot with making sure you don’t consume a lot memory, it’s also thoroughly documented, has a nicely designed API, has a unit test for almost every significant type and has Python bindings.

And I’m definitely not done yet with the project. I will keep adding value to it as I’m not satisfied about it yet. It’s still missing something magic somewhere. I don’t know yet where, but I will find the magic and insert it in the project. I’m already thinking about supporting higher programming languages in all possible directions. Including making it possible and easy to implement types, that tinymail will consume, in your higher programming language. I’m already looking at Cilc and of course GAPI for this. The define-virtual of the Python language binding generator also does a nice job.

I’m also thinking about ways to implement query folders which will have features like a virtual folders in Evolution. The goal, however, is not to use as much memory consumption yet keep it possible to query all folders. Not just the active one. I’m also thinking a lot about reducing memory consumption even more. I’m considering implementing multiple libtinymail implementation libraries (at this moment there’s only a libtinymail-camel implementation).

And I’m thinking about supporting Qtopia, Windows CE and porting the dependencies and tinymail itself to targets like SymbianOS, QNX and eCos. Yes, I’m indeed planning to take over the world too.

The good thing about this fix for the tinymail Python bindings, is that I did it in a generic way this time. I created a filter.py script which you can pass a –topsrcdir=$(top_srcdir) option on its commandline. It will walk all folders searching for .h files that have GTypeInterfaces. The h2def.py of pygtk should also do this, but seems to fail on detecting whether or not it’s a GTypeInterface. I made the regex in filter.py a little bit less strict and piped the output of h2def through filter.py. The filter.py script also makes it possible to append extra defines to the resulting .defs file. That’s why you are seeing .defs.extra files in the directory with the Python binding of tinymail. The idea is that I wanted everything to be generated. No way I’m going to store .defs files in my repository and maintain that junk.

The API docs also had a little problem that my tny-shared.h file with its ugly forward typedefs obfuscated the real location of the type struct to the parser of gtk-doc. This has also been fixed by doing some #ifdef #endif tricks in the .h files of the types.

I also have this new idea for the mmap patch for Camel. Check the idea out if memory consumption interests you. Note that the defines for calculating the offsets are still wrong (even in the 3th attempt, I forgot the ‘* x’ after the last sizeof). Nevertheless should the idea itself be clear, no? If you wonder why this patch isn’t yet in upstream Camel and Evolution: ask the evolution team, not me. The patch is default in camel-lite-builder and will be used by tinymail releases.

At work, I know I don’t often talk about work and should do that more often in stead of only whining about tinymail, I started doing some kernel development. I’m implementing a reassembly algorithm for IP fragmented in ATM cells accompanied by a timestamp using sliding window. Funky stuff.

Tuesday, January 17th, 2006, I first mentioned that I was building this tinymail thing. Somewhere early that month I had announced the idea of implementing a better E-mail client for the Nokia 770. My idea was based around the proxy design pattern in combination with model view controller.

Today, almost nine months later, I proudly present you the birth of the API that actually really looks like what will be the API when I will release tinymail. I finally started caring enough about those “Iface” suffixes to remove them. I did the monkey work of renaming all the types that now conflicted with the interface name to have a detail prefix after the namespace thing. Now you have a naming scheme like this: TnyGpeDevice implements the interface TnyDevice which has a class struct called TnyDeviceIface. This is exactly like the API of GtkTreeModel interface which has GtkTreeModelIface as class struct and is implemented by GtkTreeModelSort, GtkTreeModelFilter, GtkTreeStore and GtkListStore. And in tinymail by TnyGtkAccountTreeModel, TnyGtkHeaderListModel and TnyGtkAttachListModel in libtinymailui-gtk

As a result of this monkey work, the gtk-doc API reference manual gets generated more clean AND people will actually stop whining about it. Which is great.

I only have one regression, and that is the Python bindings. The problem is that I generate the .defs files using some autotool tricks and the well known pygtk tools. The thing that reads the .h files doesn’t detect the difference between a GObject and a GTypeInterface it seems (I hinted the code generator using a regex that matched “.*Iface” on the type’s name, but that doesn’t work anymore of course). But I will probably fix this very soon (don’t worry, nor is the release ready yet anyway). Oh feel free to fix this for me if you are bored today. And here is a pointer to those bindings.

The following valgrind massif report was created using a standard tinymail from Subversion and a Camel build using camel-lite-builder from Subversion.

It’s using gslice (small leaks aren’t visible). It’s being ran on a x86 i386 architecture . This makes a significant difference if your pointer size isn’t four bytes: the type representing a summary-item — of which the visible subject, to and from headers are components — consumes a bunch of pointers per instance.

It’s using Gecko for displaying HTML E-mails which was first used around point C in the graph.

A detailed memory-analysis of tinymail can be found (and improved by you, as it’s a wiki) here. Note that the memory-analysis that you can find there, isn’t using gslice (and if you improve it, don’t use gslice for your tests).

Note that I indeed cropped the image a little bit. The removed pieces of the image aren’t (yet) very interesting.

Let’s iterate over the marked points:

• A is the start of the application. At this point aren’t the folders loaded, but the accounts are;
• B is when the folders are loaded and the demo-ui of tinymail is fully visible and usable for the user. Everything that doesn’t fully go back to this position can be considered being a leak. But be careful when thinking about leaks: during the graph several things are kept in memory. The worst memory consumer being kept in memory is Gecko which gets allocated at point C. Others are the folders, the accounts and Gtk+ components and caches.
• Between B and the start of C is the point where I opened my INBOX folder of ~ 1,000 messages.
• C is when I first clicked an E-mail that contains a “text/html” mime-part. As a result of that, a GtkMozEmbed widget (this is a Gtk+ wrapper for Gecko) is instantiated. The memory between C and D minus E is what Gecko consumes;
• What I did at D was that I clicked another folder. The HTML message and folder of B and C where unloaded. But because Gecko upon destruction keeps itself cached, you don’t see a major memory reduction here;
• The interesting part starts at E where I clicked a folder that contains no messages (My Drafts folder);
• F is when I reopened the first folder. You see all the memory go up and down completely. This is important of course (else I’m leaking things);
• At G/a I clicked on a NNTP folder. The folder was “C++” of the newsserver at digitalmars. What you are seeing between G/a and G/b is what I think is the Camel NNTP library being loaded and consuming some of the memory. But I can be wrong as I haven’t in-depth investigated this;
• H was when the folder was fully loaded;
• I was when I clicked on that Draft folder in my IMAP account;
• J was opening the “C++” folder of the digitalmars newsserver again;
• K is the Draft folder;

Tinymail is (slowly) getting more and more contributors and is increasingly becoming serious. I can’t open all details, but Nokia is indeed interested in tinymail. I decided to become semi self-employed for smaller projects and my full-time employer agreed to sell my consultancy for larger tinymail projects. Other companies that will help you with tinymail consultancy are Kernelconcepts and Collabora. If you or your company is interested to get on the support pages of tinymail, contact me. About politics: my philosophy about software doesn’t allow me to obey any employer in case he wouldn’t allow me to add your company on that page.

The license of tinymail will forever stay LGPL. Like its license will the spirit behind the project also always be a free software one. But I (its maintainer) isn’t dirty of selling consultancy around it. It’s okay when others do this too. I, however, encourage organizations (philosiphical and technical) to contribute their platform–specific code back into the LGPL project. If necessary (and only if necessary) I will dual license certain things for this. My opinion is that availability of the code and work is more important than religion.

Major (indeed major) memory reduction is still possible (mostly within Camel) and is being worked on. Support for non-Gtk+ platforms is on the agenda (and certainly isn’t impossible). Examples are WinCE and Qtopia. Support for those will, however, not be available at a first release. It’s simply too much work and it would delay such a first release significantly. It’s not that I’m politically not interested. I am technically interested and will put my own time in it if necessary. Also note that I don’t care about anti-Microsoft idiots who will condemn me when I’ll support WinCE (and I will). I propose them to write their own tinymail if that disturbs them.

That’s it for todays tinymail blog.

Some people worried that I was not blogging anymore. No, I have simply been very busy in my free time working on tinymail. Here is an update.

Since the last months, I’ve been proof of concept designing and implementing tinymail, a E-mail framework for mobile and embedded devices that also implements and defines lots of ready-to-use user interface components of most typical mail user agent. It does this in a highly adaptive and flexible way. Indeed using flash flash buzzword buzzword “design patterns” and other modern software development techniques like unit testing.

Since the last three weeks I feel that I started designing and refactoring tinymail for real use. I started getting the API (it’s updated indeed) and its design the way I really want it. I started to make sure that reference counting is absolutely correct. API and reference counting aren’t things which you want to change a lot once you started doing releases.

That’s why I have to make sure that both are right from the start, good. Some recent changes:

There used to be two APIs for getting folders from two types. They where not capable of running asynchronously. There was one API in a store-account type for getting the root folders. And there was one API in the folder type for getting child folders.

Every sane developer who cares a little bit about design, makes a folder-store interface and lets both types implement that interface. I did that for tinymail. I also added a query mechanism. Because sometimes you want folders that are subscribed, sometimes you don’t. Sometimes you only want folders that contain “comp.lang” (for example in case your account is for a NNTP server).

Just like when you are getting headers, I designed it by letting the developer pass his own list implementation. Returning a list is often the wrong thing to do: if you return a list, the implementation of that list is fixed defined by the framework (this isn’t adaptive nor flexible). In tinymail those lists are typically implemented by types that also implement GtkTreeModel. This means no silly copying items from one list-type to another list-type. And it means that the list itself can be the containing type that keeps a reference on its items. A doubly-linked list obviously can’t do that. A doubly-linked list is also a hell for language binding your framework to a higher programming language.

Think model view controller (you know, that *thing* that you learned at school during the Java course). The list instances that you will feed the tinymail framework with, will also be the list-models of your list-views. And you can implement those models yourself (the default libtinymailui library, libtinymailui-gtk, uses the GtkTreeModel API for this). The idea is that for example in .NET, you will use the IList type for this. The .NET bindings will indeed let the list-type binding proxies implement IList.

It’s now also possible to get folders asynchronously. You never know how slow your connection will be on a mobile device. Here’s a functional test that illustrates it. And this is a synchronous version of it. Both tests recursively walk all the folders and print them on your screen.

I finished most of it before I went on holiday, but there are now also fully functional Python bindings for tinymail. They make it possible to write a functional E-mail client in a few seconds. Three people are also working on .NET bindings.

Lots of work went into adding lots of unit tests. These are very simple to implement. Actually can everybody help with this. If you too want a great E-mail framework with standard user interface components for E-mail clients, join the mailing list and get in touch. Doing unit tests will of course get you familiar with the implementations. Most people have no idea how much a developer knows when a simple stupid test succeeds of fails.

I also added lots of pages to the tinymail development pages. They are wiki pages so you can change them if you have something interesting to add or if you want to correct something. A few interesting pages are: library image sizes, how much memory will tinymail use, debugging tips, memory testing and monitoring tinymail, features and facts, a list with todo items. New pages are added constantly. The webpages are the central communication tool of the developers (there are a few contributors indeed).

About a release. A month ago I promised some guys a release in two months or at the same second Nokia releases Modest, an E-mail client being build on top of tinymail. The iterations of changes are, however, going very fast. We’ll see. Not much normal users will care a lot about a release anyway. They are just waiting for an E-mail client that doesn’t suck. Tinymail is not their E-mail client. It’s a framework for building their E-mail client.

I don’t know how legal it is. But if you are going to Pukkelpop this year, and the Macromedia Flash print feature of the Pukkelpop website isn’t working, you can find a printable version of the line-ups here, here and here.

So there’s no more need to boot a Windows in your VMWare.

A few new pages on the tinymail development website: Development tips, Debugging tips, Memory testing, Refdbg and valgrind memory testing, Writing API reference documentation.

Feel invited to fix things like spelling errors. Or add interesting information.

These topics have recently been added to the development website of tinymail.

I’m going to take a break the next two weeks. So this is the perfect time to catch-up with what is current. By that I mean that the code is probably going to stay more or less stable for two weeks. As in: it will not change a lot.

You are of course invited to join the mailing list and discuss with the tinymail developers and contributors what you think the future of tinymail should look like.

Documentation

• Help building the marketing website
• Help building the development website
• Help writing documentation
• Help with release management

Design and coding

• Help by implementing unit tests
• Help by designing and implementing code
• Help by implementing language bindings

Redesign and refactoring plans

• Help by redesigning and implementing getting folders
• Help by renaming the interface types

Integration design and coding

• Help integrating tinymail with Mathusalem for progress indication
• Help integrating tinymail with Seahorse for PGP and authentication
• Help integrating tinymail with Telepathy and Galago for presence notification
• Help integrating tinymail with Dates and Contact

Testing

• Help by implementing unit tests
• Help my monitoring the reference counting
• Help by memory testing and debugging tinymail

The development website of tinymail is being shaped.

It has information about the mailing list, api reference documentation, how you can help, some samples, how contacting the developers, how to checkout and build the source code on your platform and device and some features and facts.

Feel free to check it out and make improvements.

This week will be a documentation week for tinymail. A normal XP iteration means meet, design, implement, document and test. I very often do the document part (mostly the API). I, however, never brought all that documentation together. There’s of course still a lot other things to document. For example making some sample code embedded in the documentation.

The API reference is updated and moved to its final location (this doesn’t mean that this is the final 1.0 API. It might still change until I actually do a 1.0 release). I have setup a trac website for tinymail and one for camel-lite-builder. These trac websites will function as the developers website for tinymail.

I discussed some ideas with Ivan Krstic about E-mail and the One Laptop Per Child project. The current idea is to make a local webserver that will serve webpages like, I guess, Roundcube does. But then much smaller in memory footprint. Which is why he’s interested in tinymail and its Python bindings. I will of course make sure that the tinymail framework can function perfectly in a web environment.

Christopher Blizzard then asked me to fill in this page on the OLPCWiki. It contains some information about tinymail on the OLPC laptop.

I hope people will help me this week with setting up all the wiki pages of the tinymail trac. Everybody is of course invited to improve things like the website (just wget -r it) or documentation or the trac infrastructure and its wiki pages. If you have an idea: please go ahead and edit it. I mean .. if I don’t like it, I can remove it right?

I also hope there’s going to be more people helping me with language bindings. Like improving the Python ones and implementing the .NET ones. The more of this stuff that I will have before making a first release, the better this first release will be. If somebody climbs the stage, I will definitely join him on this. If nobody does, it might be delayed until after my holiday: Tinne decided that I need a break next week.

A few days ago, on some mailing list, I wrote:

I actually but truly believe that running most GNOME programs under a garbage collector would actually save (a lot) more memory then the virtual machine would add.

In stead of just saying such crazy things, I should of course also prove it. It’s of course not easy to prove this. What I can do is show you two programs that do exactly the same. One program uses Python language bindings. The other doesn’t use language bindings but directly invokes the API in C using GObject.

The two applications loaded the same IMAP folder of ~1000 headers. And unloaded that folder.

The usage of Python bindings causes some Python proxy objects to be created. I think (but I’m not sure) that most of the extra memory that you will see is caused by these.

• The Python application
• The C application

In Python

In C

Consider that most desktop applications consume more than 20MB and that the test application (tinymail) is designed and developed with low memory consumption in mind, I still believe the overhead introduced by a higher programming language isn’t significant for most desktop applications. Not even for mobile devices with not-so-much memory.

On Mono, the results would probably differ a little bit. But probably not a lot. I indeed hope people will join me on creating good .NET bindings for tinymail.

I proudly present you, a fully working E-mail client that supports IMAP, Exchange, POP (and their SSL equivalents) written in Python.

It’s build on top of tinymail. It supports going online and viewing received information offline. It supports refreshing the folder asynchronous. It has a progressbar that shows the status of the worker thread that is getting the new message information. It can show E-mails in HTML. Proved using an enormous amount of valgrind testing and if you use a Camel build by camel-lite-builder, it consumes less then ten megabytes of ram when displaying a folder that is larger than 15,000 messages.

Its code (the Python one) is 64 lines long using only four event handlers: on folder select, on header select, on status update and on folder refreshed. This is proof that the tinymail API kicks ass. Updated gtk-doc reference API documentation will be online very soon.

The design of the tinymail framework allows for ultimate flexibility. You can reinvent and reimplement every tinymail type without even having to rebuild for example the Python bindings. Without losing ABI nor API compatibility (but there’s no release, so this doesn’t mean a lot yet. After a first release, it will mean a lot). You can do this for both underlying infrastructure (such as connectivity with the E-mail service) and user interface infrastructure (such as a component for viewing a message). For example: the design of tinymail allows you to (re)implement the user interface implementations using Qt or Win32.

A screenshot:

The bindings are working. Nevertheless I hope other people will be inspired to go ahead and make the Python bindings of tinymail even better. Because now there’s Python bindings, there’s a chance that your work would be used by the One Laptop Per Child project. For me personally, that alone is enough fuel. Once I know more, you know I will blog about it :-).

Words? who needs words when he has a picture?

This is indeed the 42 lines Python E-mail client on top of tinymail.

I hope this way I inspired Python developers to join the fun of making the Python bindings for tinymail the best humanly possible. My appreciation to all the people in #pygtk who helped me the last three days. Especially Gustavo J. A. M. Carneiro, James Livingston and Johan Dahlin. Thanks to the #pygtk people for not kicking me because I flooded their IRC channel with questions and pasted code.

I know I don’t make it my public secret that I’m not a Python evangelist. As a programming environment and language, I don’t like Python nor PyGtk+. It’s not my programming language. I’m more the .NET, D and Java developer when it comes to higher programming languages.

But that doesn’t mean I don’t love Python developers. I do! I decided to prove it by implementing the Python bindings for tinymail.

There’s still a few minor and solvable issues like that the binding generator doesn’t yet generate the stuff needed for a typedef to a function pointer. You know, like the GFunc for Gtk+. Sooner or later, I’ll figure out how to do that. Or maybe you, the experienced Python code warrior, can help me? The current resulting .so files also aren’t tested yet. There’s definitely still some work to do.

Other than that, are bindings for libtinymail, libtinymailui, libtinymailui-gtk and the platform library you selected being generated automatically. Just add –enable-python-bindings to the configure line.

You can find the overrides and makefile here.

So far, two people told me they are interested in doing the .NET bindings. They are of course encouraged to do so.

The Power Of Nightmares 1/3

The Power Of Nightmares 2/3

The Power Of Nightmares 3/3

Yesterday I told you guys that I didn’t yet implement a account store that doesn’t use GConf, right?

Here’s the one for OLPC that uses gkeyfile: libtinymail-olpc/tny-account-store.c

I also improved camel-lite-builder so that it no longer links with GnomeVFS (nor ORBit-2 nor Bonobo). I don’t know whether the final OLPC device will ship with GConf and the D-Bus version of GnomeVFS. It no longer matters for tinymail.

By the way, great blog post by Mukund where he explains how we, human species, are in fact nothing but tests that test whether some newer genetic instructions successfully work or not.

Guys, remember I promised that I was not going to post new photos on my blog until I have one with tinymail running on the OLPC mobo?

Well, I will need my blog-space for posting pictures soon. So here’s a picture of tinymail running on the OLPC mobo!

I haven’t yet implemented a account store that doesn’t use GConf for reading the account configuration. Nor have I already added a patch for Camel in camel-lite-builder that removes Camels GnomeVFS dependency. Since a month or so, Camel depends on GnomeVFS, which at this moment pulls things like Bonobo and ORBit-2.

So .. I don’t think it’s a good idea not to patch it out of Camel before starting to use Camel on devices like the OLPC laptop. The OLPC image indeed doesn’t seem to come with libraries like GConf, Bonobo and ORBit-2. Other than that, it’s a very easy device to target. It’s no surprise that tinymail runs on it.

There’s of course a gnome-vfs-dbus. Very interesting of course. Might also be a solution for this. Well, the current hack for getting it to run was simply removing the two files from Camels Makefile.am.

I just uploaded the new tinymail.org website.

The small size of the web-design is indeed a hint that the website can also be used on mobile devices, who are a target of the tinymail framework.

Some of the website content might change as some people will probably give me advises. The purpose of this website is to be a view for also the commercial world. The idea is indeed to encourage commercial activities around the LGPL tinymail framework.

I’m also planning to make a website with a much more technical point of view. Being a website for a development framework, it will include developer documentation like the API reference, class diagrams, demos and guidelines.

I hope people understand that I only have two hands and that making such documentation takes time. I am, however, committed to do it. There’s already an API reference which will of course very soon be updated. Using the source code from the Subversion repository you can of course rebuild the API reference yourself.

Things like unit tests and documentation have a high priority and some of it is already available. I will probably not release unless both are perfect.

I will of course welcome people who want to help me creating or improving developer documentation, unit tests, design, implementation, a first release and the website.

I hope you will hear about Modest very soon. And yes, I was told they are designing the Modest user interface in such a way that it will be possible to use it on both a desktop and on a mobile device.

It is my opinion that using our blogs and the Internet being our speech instruments, we should somehow start organizing a huge anti-war protest.

Israel is clearly targeting civilians and civilian infrastructure. I condemn these actions and will no longer feel any compassion whatsoever for Israeli soldiers that decide to fight this war for their government. I can not agree with Hezbollah for launching missiles on Israeli cities. I can understand, however, that Lebanese people (like Hezbollah) no longer agree with what Israel has been doing the past years and during the last 12 days.

This way Israel is creating new hatred and aggression against themselves. Question yourself: what would you do if another country bombs your village killing your woman, baby, kid and family members? Imagine finding your kids head and other body-parts in a bloody bombed street or building. Would you not hate the people that did this? Western people live behind their CNN screen. Enjoying “television war”. It actually makes me sick that so many of us, Westerns, have lost reality with what war really is (please, for once: look at it. Even if you are frightned of pictures showing dead children. Look at the reality. See it. Understand it).

Our grandparents fought World War I and II. They learned us, the generations after World War II, never again to go to war. They died for our peace. We, young people, might never understand what they have seen. But please respect them. War is never a solution. War against citizens, what in my opinion Israel is doing, is evil.

It is insanity. I refuse to call this human. We, Europeans, do not want this. If our European politicians would want this, I would tell them here and now that I no longer call myself a citizen of their European union. Luckily so far most of them have been condemning Israel and their recent actions. I hope this will continue. I hope everybody who makes himself guilty of war crimes (both Hezbollah and Israeli) will get severely punished for these crimes against the human race (almost 400 unarmed citizen after only 12 days of war, 350 of them Lebanese people).

I hope the Nuremberg trials will repeat themselves (but please read the interesting 5th comment of Max, I also agree with that).

People who are going to reply with the “terrorist” word in their comment, please read this first. Maybe you should stop being so naive and learn about indoctrination using fear. You might understand what this American president is really doing with his “terrorists are everywhere”-speeches.

To world leaders and the UN: if American leaders show no interest in peace, please no longer consider their opinion to be of any value. Proceed with your peace plans without American support.