Switching to multiple threads, with a non-thread-safe resource

Your application used to be single threaded and is consuming a resource that is not thread-safe. You’re splitting your application up into two or more threads. Both threads want to consume the non-thread-safe resource.

In this GNOME-Live item I explain how to use GThreadPool for this.

It’s a wiki so if you find any discrepancies in the sample and or text, just correct them. I’m subscribed so I’ll review it that way.

The GNOME-Live item is done in a similar way to the item about using asynchronous DBus bindings and the AsyncWorker item.

DBus using DBus.GLib.Async and dbus-binding-tool

While I was gathering some info about a DBus related task that I’m doing at this moment, I wrote down whatever I found about DBus’s glib bindings in tutorial format.

A few other people have done similar things in their blogs. This one explains how to use org.freedesktop.DBus.GLib.Async a little bit too.

If you find any mistakes in the document, it’s a wiki page so please just correct them.

Iterators and tree models! Shocking!

Iterators

Now that Murray has posted about the iterators idea I can no longer hide.

Together with Jürg, the Vala man and Murray, who before becoming a pregnant guy was and still is coding on things like Glom and gtkmm, I’ve been writing up a document. This document explains iterators like the ones you’ll find in .NET and Java.

This is related to the Iterators concept because the document about iterators could be part of a solution for this.

I once had to write a custom GtkTreeModel (I think I still suffer from vertigo) and therefore I sent treeview Kris an analysis about this.

There’s a rule that an interface should be a contract for solving exactly one problem.

Use interfaces to say “What objects can do” or “What can be done to an object”. You can let a class implement multiple interfaces. There’s absolutely no need to make huge interfaces. An interface should define that your class “Does One Thing”. Another interface defines that your class “Does Another Thing, too”, and a last interface defines that your class “Also Does This”. My class Z can do D, E and F. Therefore: class Z implements D, E and F.

Instead, the contract being GtkTreeModel requires you to solve five problems:

  • being iterable: you can go to the next node
  • being a container: you can add and remove things from it
  • being observable: it notifies a component about its changes
  • being a tree: sometimes items can parent other items
  • having columns: it’s something that describes columns

Summarizing: interface TreeModel means BEING a D, E, F, G and H at the same time. Correct would be if you specified that TreeView requires a model that does D, does E and perhaps also does F. In case treeview is showing the things as a tree, it requires that the model also does E and does H.

GtkTreeModel should have been five different interfaces.

Shocking! But in my opinion, true.

The consequence is that having to implement a GtkTreeModel has become a difficult task.

“You have to implement solutions for all five problems anyway!”, you might think. That’s true. However! If there were five interfaces, other people would have made default solutions for four other interfaces. Meanwhile “you” focus on “your” specialization. For example Rhythmbox and Banshee are specialized in huge models that contain a specific set of data that must be searchable. Preferably search and sort are done by a database instead of a model-filter.

Banshee and Rhythmbox developers:

  • … don’t have to care about the Columns problem
  • … don’t have to care at all about the Tree problem
  • … don’t have to care about the Container problem: they never need to add nor remove things from it because they set up their model instantly by passing a query to a data source. Maybe the feature “remove from playlist” requires solving the Container problem?
  • … unless they support inotify on the MP3 folder of the disk, they don’t even have to care about the observer problem

Yet with GtkTreeModel you have put the burden on application developers to solve all five the problems in their model. No wonder Banshee developers decide to code their own TreeView in .NET!

Reference counting

There’s nothing wrong with adding a reference to the GObjects that you’ll pass to a deferred call.

For example (also read GdkLock, for the gdk_threads_enter thing):

static gboolean
callback (gpointer user_data) {
  GObject *instance = user_data;
  gdk_threads_enter ();
  /* work with instance and don't worry */
  gdk_threads_leave ();
  return FALSE;
}

static void
MyMethod (GObject *instance) {
  g_timeout_add_full (G_PRIORITY_DEFAULT, 5*1000,
		callback, g_object_ref (instance),
		(GDestroyNotify) g_object_unref);
}

Don’t do this:

static gboolean
callback (gpointer user_data) {
  GObject *instance = user_data;
  /* work with instance AND worry */
  return FALSE;
}

static void
MyMethod (GObject *instance) {
  g_timeout_add_full (G_PRIORITY_DEFAULT, 5*1000,
		callback, instance, NULL);
}

The problem is that most applications have multiple contexts. Either are those contexts threads or multiple code-paths for ‘instance’ to become finalized. For example D-Bus handlers, threads (yes, some people use them) or callbacks caused by GtkWidgets. Your timeout’s callback depends on that ‘instance’, therefore it’s required that you add a reference. Just like how a garbage collected higher language would add a reference to make sure that the instance wont be garbage collected during ‘callback’.

When I look at a lot of GLib/Gtk+ consuming code, I conclude that some people seem to be afraid of g_object_ref. Some people seem to believe that overuse of it creates memory leaks. The reality is much more simple: one misuse of it, causes a memory leak. Using it when required, doesn’t. Just be careful for cyclic references: an instance ‘A’ that references another instance ‘B’, that itself references ‘A’. Sometimes there is a type in between ‘A’ and ‘B’, like a container type. Use weak references for those situations in stead. Do use references when they are required, though.

In the example above nothing guarantees you that callback wont be interrupted. Nothing even guarantees you that callback will be started with a valid instance in its user_data. I have seen people putting G_IS_OBJECT checks at the top of their callback implementation. This is wrong and doesn’t really make it any better. Right after that runtime check your callback can get interrupted and you can again find the instance becoming finalized while using it in ‘callback’. By adding a reference, you are sure about the reference that you added. You are also sure that g_timeout_add_full will make sure that the GDestroyNotify will eventually execute. So you are sure that the g_object_ref is not causing any reference leak.

Also note that code like this is fine, indeed. Sure the two g_object_ref calls are atomic increments and therefore a little bit slower than avoiding one of the two. But it’s consistent. When a new developer reads that code he’ll immediately recognize that my_thing_get_instance returns a reference. It’s my opinion that readability is more important than a micro optimization hack.

static gboolean
callback (gpointer user_data) {
  GObject *instance = user_data;
  gdk_threads_enter ();
  /* work with instance and don't worry */
  gdk_threads_leave ();
  return FALSE;
}

static void
MyMethod (void) {
  GObject *instance = my_thing_get_instance ();
  ...
  g_timeout_add_full (G_PRIORITY_DEFAULT, 5*1000,
		callback, g_object_ref (instance),
		(GDestroyNotify) g_object_unref);
  ...
  g_object_unref (instance);
}

Web 2.0 !!!

A few days ago I got this reply on one of my blog posts:

Phil:

Your post — and your work — miss the point entirely. Nobody cares how email works, they just want it to work.

Gmail (and most other webmail applications) makes everything else obsolete. I can’t imagine why Evolution is even shipped with Gnome anymore.

Web-based email clients are the standard.

-Anon

I just finished the E-mail client the guy wants. Here it is!

using GLib;
using Gtk;
using WebKit;

class Web20EmailClient : Window {
	WebView view;
	construct {
		view = new WebView ();
		view.open ("http://gmail.com");
		view.set_size_request (640, 480);
		add (view);
	}
	static void main (string[] args) {
		Gtk.init (ref args);
		var win = new Web20EmailClient ();
		win.show_all ();
		Gtk.main ();
	}
}

Mindstorm … s

You buy a bunch of Lego Mindstorms bricks and you start building a robot to remotely control your mobile devices.

Well, that’s the official explanation.

The actual explanation is that this is what happens when you are 26 years of age, your girlfriend tells you you are almost 30 and that when you are 30 it’s the end of your youth (although, people of that age usually tell me this ain’t true), you are a nerd of the type software developer (and quite addicted to this too), you have your own business and therefore your accountant asks to make some expenses (like .. buying a Mindstorms robot! No?).

I acknowledge it’s probably just an early midlife crisis. Boys want to make things, fiddle with stuff, put things together. Whereas girls, girls just wanna have fun. I’m totally guilty of being a boy. I know. (although, I’m sure a lot of girls enjoy making things too — before I get killed by a group of feminists –).

Now that the model itself is finished, I clearly see what I am becoming: an old lonely dude who plays with trains, electricity stuff and mostly breaks things just to put them back together. I’ll probably die getting electrocuted while trying to take apart a by that time old holographic 3D gesture recognizing display, as I’m trying to figure out whether some evil corporation is spying on its customers by using such electronic devices.

But, isn’t that cute? No? I mean, Tinne, seriously, now I must be ‘like’ a younger dude, no? I have been playing with toys for kids aged 11 to 16 (that’s what the Lego box’s age indicator says, so it must be true). Anyway, the only way that it can get worse now, is if I’ll start writing software for this Lego model. I’ll have a camera view on my screen where I can mouse-over so that the robot will follow my mouse pointer. With a library like GStreamer I can let that camera image go efficiently over a distance. Sending some commands over a socket ain’t very hard.

About the bot itself: it has three axis. One (the X one) uses normal wheels, two others (Y and Z) are built on top of the chassis. All axis are controlled by Mindstorms motors. The Mindstorms computer thing is integrated in the model, there’s a touch sensor on one of the axis (the Z one). I don’t yet have this software, that’s the next thing I’ll (try to) finish. I’ve spend ~ 450 euros on this thing (the normal Mindstorms package didn’t have enough bricks, but the programmable thing, the sensors and the motors are ~ 300 euros).

But hey, 450 euros for something that you could give to a little fellow as soon as you are done playing with it? That’s not much for multi functional and multi age toys! I mean, if I get bored of this thing, I can make another robot with it. If you have a son (or a technical minded daughter), you can let him (or her) play with the Lego bricks while watching his (her) brains grow! You can’t convince me that today’s computer games are better for training a kid’s brain than Lego.

After the kid is finished building the bot, you can make the software for it. Hah! Perfect father – son (or daughter) relationship. You actually help him make his toys, and you enjoy doing that! And … he’ll get interested in software development, join one of the many free software communities, he’ll find a job in IT as programmer, etc etc.

Lego rocks!

Three million rows in a GtkTreeView

Edit, the repository has since disappeared, you can find a Subversion Dump of it here:

Three million rows (the size per cell doesn’t matter a lot) in a treeview, and loading the treeview in four seconds. Is that doable? Sure! The treeview wil become very slow you think? Nope, it works as fast as any other (smaller) treeview. The amount of visible rows is what would slow it down. Since most screens can’t show more than 500 rows, and since showing more would be useless from a usability point of view, it’s fast.

I committed my performance tweaks to the demo repository. It includes using g_slice for allocating the real subject and replacing the GSList in the custom model with an implementation that uses a pointer position.

So I don’t have to depend on a slow linked list anymore. In stead I simply allocate a large block of proxy instances (three million proxy instances of 20 bytes each in a continuous allocation) and inject that as index in my custom treemodel.

Since those are proxy instances, they’ll each check whether their this->real property isn’t NULL when they are needed. When a row becomes visible, that instance is needed for the from, id and to properties. When it becomes invisible, it’s no longer needed (and should therefore be freed, but the unref_node thingy of gtktreeview doesn’t work perfectly — so when scrolling a little bit, around 200 instances are kept around for no reason, I’m going to try fixing that behaviour in gtktreeview soon).

Most of the time is spend in the loop that prepares the proxy instances (msg_header_proxy_new_alot). Bringing the (visible) items to the treeview doesn’t take a lot time, as the GtkTreeView is smart enough not to load everything in case fixed-row-height mode is on.

You don’t have to believe me, you can checkout the code here. Compile it (autotools) and try.

Anyway, I’m convinced GtkTreeView by itself isn’t slow. But that doesn’t mean that the way you use it can’t make it slow. I hope others will enjoy the demo as a starting point for getting their way of using the gtktreeview optimized. For most use-cases, the use of a GSList or GList is a better technique. A linked list makes it more easy to add new items to your model. Inserting and removing items would be a lot more difficult if you use the technique I used in the demo. That technique, however, is fast because you can allocate it as one large block and excercise your high-school pointer knowledge with.

Nevertheless, I swear the unref_node stuff isn’t working correctly! :-p. Or I misunderstood it’s purpose.