Category Archives: Science

Asynchronous commands

With asynchronous commands we have typical commands from the Model View ViewModel world that return asynchronously.

Whenever that happens we want result reporting and progress reporting. We basically want something like this in QML:

Item {
  id: container
  property ViewModel viewModel: ViewModel {}

  Connections {
    target: viewModel.asyncHelloCommand
    onExecuteProgressed: {
        progressBar.value = value
        progressBar.maximumValue = maximum
    }
  }
  ProgressBar {
     id: progressBar
  }
  Button {
    enabled: viewModel.asyncHelloCommand.canExecute
    onClicked: viewModel.asyncHelloCommand.execute()
  }
}

How do we do this? First we start with defining a AbstractAsyncCommand (impl. of protected APIs here):

class AbstractAsyncCommand : public AbstractCommand {
    Q_OBJECT
public:
    AbstractAsyncCommand(QObject *parent=0);

    Q_INVOKABLE virtual QFuture<void*> executeAsync() = 0;
    virtual void execute() Q_DECL_OVERRIDE;
signals:
    void executeFinished(void* result);
    void executeProgressed(int value, int maximum);
protected:
    QSharedPointer<QFutureInterface<void*>> start();
    void progress(QSharedPointer<QFutureInterface<void*>> fut, int value, int total);
    void finish(QSharedPointer<QFutureInterface<void*>> fut, void* result);
private:
    QVector<QSharedPointer<QFutureInterface<void*>>> m_futures;
};

After that we provide an implementation:

#include <QThreadPool>
#include <QRunnable>

#include <MVVM/Commands/AbstractAsyncCommand.h>

class AsyncHelloCommand: public AbstractAsyncCommand
{
    Q_OBJECT
public:
    AsyncHelloCommand(QObject *parent=0);
    bool canExecute() const Q_DECL_OVERRIDE { return true; }
    QFuture<void*> executeAsync() Q_DECL_OVERRIDE;
private:
    void* executeAsyncTaskFunc();
    QSharedPointer<QFutureInterface<void*>> current;
    QMutex mutex;
};

#include "asynchellocommand.h"

#include <QtConcurrent/QtConcurrent>

AsyncHelloCommand::AsyncHelloCommand(QObject* parent)
    : AbstractAsyncCommand(parent) { }

void* AsyncHelloCommand::executeAsyncTaskFunc()
{
    for (int i=0; i<10; i++) {
        QThread::sleep(1);
        qDebug() << "Hello Async!";
        mutex.lock();
        progress(current, i, 10);
        mutex.unlock();
    }
    return nullptr;
}

QFuture<void*> AsyncHelloCommand::executeAsync()
{
    mutex.lock();
    current = start();
    QFutureWatcher<void*>* watcher = new QFutureWatcher<void*>(this);
    connect(watcher, &QFutureWatcher<void*>::progressValueChanged, this, [=]{
        mutex.lock();
        progress(current, watcher->progressValue(), watcher->progressMaximum());
        mutex.unlock();
    });
    connect(watcher, &QFutureWatcher<void*>::finished, this, [=]{
        void* result=watcher->result();
        mutex.lock();
        finish(current, result);
        mutex.unlock();
        watcher->deleteLater();
    });
    watcher->setFuture(QtConcurrent::run(this, &AsyncHelloCommand::executeAsyncTaskFunc));
    QFuture<void*> future = current->future();
    mutex.unlock();

    return future;
}

You can find the complete working example here.

AbstractCommand Model View ViewModel techniques

In the .NET XAML world, you have the ICommand, the CompositeCommand and the DelegateCommand. You use these commands to in a declarative way bind them as properties to XAML components like menu items and buttons. You can find an excellent book on this titled Prism 5.0 for WPF.

The ICommand defines two things: a canExecute property and an execute() method. The CompositeCommand allows you to combine multiple commands together, the DelegateCommand makes it possible to pass two delegates (functors or lambda’s); one for the canExecute evaluation and one for the execute() method.

The idea here is that you want to make it possible to put said commands in a ViewModel and then data bind them to your View (so in QML that’s with Q_INVOKABLE and Q_PROPERTY). Meaning that the action of the component in the view results in execute() being called, and the component in the view being enabled or not is bound to the canExecute bool property.

In QML that of course corresponds to a ViewModel.cpp for a View.qml. Meanwhile you also want to make it possible to in a declarative way use certain commands in the View.qml without involving the ViewModel.cpp.

So I tried making exactly that. I’ve placed it on github in a project I plan to use more often to collect MVVM techniques I come up with. And in this article I’ll explain how and what. I’ll stick to the header files and the QML file.

We start with defining a AbstractCommand interface. This is very much like .NET’s ICommand, of course:

#include <QObject>

class AbstractCommand : public QObject {
    Q_OBJECT
    Q_PROPERTY(bool canExecute READ canExecute NOTIFY canExecuteChanged)
public:
    AbstractCommand(QObject *parent = 0):QObject(parent){}
    Q_INVOKABLE virtual void execute() = 0;
    virtual bool canExecute() const = 0;
signals:
    void canExecuteChanged(bool canExecute);
};

We will also make a command that is very easy to use in QML, the EmitCommand:

#include <MVVM/Commands/AbstractCommand.h>

class EmitCommand : public AbstractCommand
{
    Q_OBJECT
    Q_PROPERTY(bool canExecute READ canExecute WRITE setCanExecute NOTIFY privateCanExecuteChanged)
public:
    EmitCommand(QObject *parent=0):AbstractCommand(parent){}

    void execute() Q_DECL_OVERRIDE;
    bool canExecute() const Q_DECL_OVERRIDE;
public slots:
    void setCanExecute(bool canExecute);
signals:
    void executes();
    void privateCanExecuteChanged();
private:
    bool canExe = false;
};

We make a command that allows us to combine multiple commands together as one. This is the equivalent of .NET’s CompositeCommand, here you have our own:

#include <QSharedPointer>
#include <QQmlListProperty>

#include <MVVM/Commands/AbstractCommand.h>
#include <MVVM/Commands/ListCommand.h>

class CompositeCommand : public AbstractCommand {
    Q_OBJECT

    Q_PROPERTY(QQmlListProperty<AbstractCommand> commands READ commands NOTIFY commandsChanged )
    Q_CLASSINFO("DefaultProperty", "commands")
public:
    CompositeCommand(QObject *parent = 0):AbstractCommand (parent) {}
    CompositeCommand(QList<QSharedPointer<AbstractCommand> > cmds, QObject *parent=0);
    ~CompositeCommand();
    void execute() Q_DECL_OVERRIDE;
    bool canExecute() const Q_DECL_OVERRIDE;
    void remove(const QSharedPointer<AbstractCommand> &cmd);
    void add(const QSharedPointer<AbstractCommand> &cmd);

    void add(AbstractCommand *cmd);
    void clearCommands();
    QQmlListProperty<AbstractCommand> commands();

signals:
    void commandsChanged();
private slots:
    void onCanExecuteChanged(bool canExecute);
private:
    QList<QSharedPointer<AbstractCommand> > cmds;
    static void appendCommand(QQmlListProperty<AbstractCommand> *lst, AbstractCommand *cmd);
    static AbstractCommand* command(QQmlListProperty<AbstractCommand> *lst, int idx);
    static void clearCommands(QQmlListProperty<AbstractCommand> *lst);
    static int commandCount(QQmlListProperty<AbstractCommand> *lst);
};

We also make a command that looks a lot like ListElement in QML’s ListModel:

#include <MVVM/Commands/AbstractCommand.h>

class ListCommand : public AbstractCommand
{
    Q_OBJECT
    Q_PROPERTY(AbstractCommand *command READ command WRITE setCommand NOTIFY commandChanged)
    Q_PROPERTY(QString text READ text WRITE setText NOTIFY textChanged)
public:
    ListCommand(QObject *parent = 0):AbstractCommand(parent){}
    void execute() Q_DECL_OVERRIDE;
    bool canExecute() const Q_DECL_OVERRIDE;
    AbstractCommand* command() const;
    void setCommand(AbstractCommand *newCommand);
    void setCommand(const QSharedPointer<AbstractCommand> &newCommand);
    QString text() const;
    void setText(const QString &newValue);
signals:
    void commandChanged();
    void textChanged();
private:
    QSharedPointer<AbstractCommand> cmd;
    QString txt;
};

Let’s now also make the equivalent for QML’s ListModel, CommandListModel:

#include <QObject>
#include <QQmlListProperty>

#include <MVVM/Commands/ListCommand.h>

class CommandListModel:public QObject {
    Q_OBJECT
    Q_PROPERTY(QQmlListProperty<ListCommand> commands READ commands NOTIFY commandsChanged )
    Q_CLASSINFO("DefaultProperty", "commands")
public:
    CommandListModel(QObject *parent = 0):QObject(parent){}
    void clearCommands();
    int commandCount() const;
    QQmlListProperty<ListCommand> commands();
    void appendCommand(ListCommand *command);
    ListCommand* command(int idx) const;
signals:
    void commandsChanged();
private:
    static void appendCommand(QQmlListProperty<ListCommand> *lst, ListCommand *cmd);
    static ListCommand* command(QQmlListProperty<ListCommand> *lst, int idx);
    static void clearCommands(QQmlListProperty<ListCommand> *lst);
    static int commandCount(QQmlListProperty<ListCommand> *lst);

    QList<ListCommand* > cmds;
};

Okay, let’s now put all this together in a simple example QML:

import QtQuick 2.3
import QtQuick.Window 2.0
import QtQuick.Controls 1.2

import be.codeminded.mvvm 1.0

import Example 1.0 as A

Window {
    width: 360
    height: 360
    visible: true

    ListView {
        id: listView
        anchors.fill: parent

        delegate: Item {
            height: 20
            width: listView.width
            MouseArea {
                anchors.fill: parent
                onClicked: if (modelData.canExecute) modelData.execute()
            }
            Text {
                anchors.fill: parent
                text: modelData.text
                color: modelData.canExecute ? "black" : "grey"
            }
        }

        model: comsModel.commands

        property bool combineCanExecute: false

        CommandListModel {
            id: comsModel

            ListCommand {
                text: "C++ Lambda command"
                command:  A.LambdaCommand
            }

            ListCommand {
                text: "Enable combined"
                command: EmitCommand {
                    onExecutes: { console.warn( "Hello1");
                        listView.combineCanExecute=true; }
                    canExecute: true
                }
            }

            ListCommand {
                text: "Disable combined"
                command: EmitCommand {
                    onExecutes: { console.warn( "Hello2");
                        listView.combineCanExecute=false; }
                    canExecute: true
                }
            }

            ListCommand {
                text: "Combined emit commands"
                command: CompositeCommand {
                    EmitCommand {
                        onExecutes: console.warn( "Emit command 1");
                        canExecute: listView.combineCanExecute
                    }
                    EmitCommand {
                        onExecutes: console.warn( "Emit command 2");
                        canExecute: listView.combineCanExecute
                    }
                }
            }
        }
    }
}

I made a task-bug for this on Qt, here.

How to expose a QList<MyListClass> in a ViewModel to QML

MyPlugin/MyPlugin.cpp:

#include <ViewModels/MyListClass.h>
#include <ViewModels/DisplayViewModel.h>

qmlRegisterUncreatableType<MyListClass>( a_uri, 1, 0, "MyListClass",
         "Use access via DisplayViewModel instead");
qmlRegisterType<DisplayViewModel>( a_uri, 1, 0, "DisplayViewModel");

Utils/MyQMLListUtils.h

#define MY_DECLARE_QML_LIST(type, name, owner, prop) \
QQmlListProperty<type> name(){ \
   return QQmlListProperty<type>( \
               this, 0,&owner::count ## type ## For ## name ## List, \
               &owner::at ## type ## For ## name ## List); \
} \
static int count ## type ## For ## name ## List(QQmlListProperty<type>*property){ \
   owner *m = qobject_cast<owner *>(property->object); \
   return m->prop.size(); \
} \
static type *at ## type ## For ## name ## List( \
        QQmlListProperty<type>*property, int index){ \
   owner *m = qobject_cast<owner *>(property->object); \
   return m->prop[index]; \
}

ViewModels/DisplayViewModel.h

#ifndef DISPLAYVIEWMODEL_H
#define DISPLAYVIEWMODEL_H

#include <QObject>
#include <QtQml>
#include <ViewModels/MyListClass.h>
#include <Utils/MyQMLListUtils.h>

class DisplayViewModel : public QObject
{
    Q_OBJECT

    Q_PROPERTY(constQString title READ title WRITE setTitle NOTIFY titleChanged )
    Q_PROPERTY(constQList<MyListClass*> objects READ objects
                                          NOTIFY objectsChanged ) 
    Q_PROPERTY( QQmlListProperty<MyListClass> objectList READ objectList
                                              NOTIFY objectsChanged )
public:
    explicit DisplayViewModel( QObject *a_parent = nullptr );
    explicit DisplayViewModel( const QString &a_title,
                               QList<MyListClass*> a_objects,
                               QObject *a_parent = nullptr );
    const QString title()
        { return m_title; }
    void setTitle( const QString &a_title ); 
    const QList<MyListClass*> objects ()
        { return m_objects; } 
    Q_INVOKABLE void appendObject( MyListClass *a_object);
    Q_INVOKABLE void deleteObject( MyListClass *a_object);
    Q_INVOKABLE void reset( );

protected:
    MY_DECLARE_QML_LIST(MyListClass, objectList, DisplayViewModel, m_objects)

signals:
    void titleChanged();
    void objectsChanged();

private:
    QString m_title;
    QList<MyListObject*> m_objects;
};

#endif// DISPLAYVIEWMODEL_H

DisplayViewModel.cpp

#include "DisplayViewModel.h"

DisplayViewModel::DisplayViewModel( const QString &a_title,
                                    QList<MyListClass*> a_objects,
                                    QObject *a_parent )
    : QObject ( a_parent )
    , m_title ( a_title )
    , m_objects ( a_objects )
{
    foreach (MyListClass* mobject, m_objects) {
        mobject->setParent (this);
    }
}

void DisplayViewModel::setTitle (const QString &a_title )
{
    if ( m_title != a_title ) {
        m_title = a_title;
        emit titleChanged();
    }
}

void DisplayViewModel::reset( )
{
    foreach ( MyListClass *mobject, m_objects ) {
        mobject->deleteLater();
    }
    m_objects.clear();
    emit objectsChanged();
}

void DisplayViewModel::appendObject( MyListClass *a_object )
{
    a_object->setParent( this );
    m_objects.append( a_object );
    emit objectsChanged();
}

void DisplayViewModel::deleteObject( MyListClass *a_object )
{
    if (m_objects.contains( a_object )) {
        m_objects.removeOne( a_object );
        a_object->deleteLater();
        emit objectsChanged();
    }
}

Tester.cpp

#include <ViewModels/DisplayViewModel.h>
#include <ViewModels/MyListClass.h>

QList<MyListClass*> objectList;
for( int i = 0; i < 100 ; ++i ) {
    objectList.append ( new MyListClass (i) );
}
DisplayViewModel *viewModel = new DisplayViewModel (objectList);
viewModel->appendObject ( new MyListClass (101) );

Display.qml

import QtQuick 2.5
import MyPlugin 1.0

Repeater { 
    property DisplayViewModel viewModel: DisplayViewModel { } 
    model: viewModel.objectList
    delegate: Item {
        property MyListClass object: modelData
        Text {
            text: object.property
        }
    }
}

Beste VRT, over PGP

PGP, Pretty Good Privacy, is niet enkel te koop op Het Internet; het is zowel gratis als dat het open source is. Dat wil zeggen dat iedereen op deze planeet de broncode van PGP kan downloaden, compileren en aanpassen en dat iedereen het op een telefoon kan zetten.

Velen gebruiken dan ook zulke encryptiesoftware voor allerlei redenen. Vaak zonder dat ze het door hebben (https websites, de diensten van je bank, online aankopen, Whatsapp berichten, en zo verder). Soms hebben ze het wel door (dat zijn dan vaak techneuten).

Enkelingen doen dat om hun communicatie over criminele plannen te versleutelen. Maar velen anderen doen dat om hun bedrijfsgegevens, privacy, persoonlijke data en zo verder te beveiligen. Er zouden ook overheden zijn, naar het schijnt, zoals de onze, bijvoorbeeld, die het veelvuldig gebruiken. Volgens geruchten gebruiken alle militairen het. Ik ben er van overtuigd dat jullie werkgever, de VRT, het ook heel veel gebruikt en zelfs oplegt. Een beetje serieus journalist gebruikt het tegenwoordig ook voor, laat ik hopen, alles.

PGP is niet illegaal. Het versleutelen van berichten is niet illegaal. Dat de politie het daarom niet meer kan lezen, is op zichzelf niet illegaal. Er staat in onze Belgische wetgeving helemaal niets over het illegaal zijn van het versleutelen van berichten. Er staat in onze Belgische wetgeving helemaal niets over het illegaal zijn van bv. software zoals PGP.

Versleutelen van berichten bestaat ook al eeuwen. Nee, millennia. Misschien zelfs wel al tienduizenden jaren. Kinderen doen het en leren het. Misschien niet op al te professionele manier. Maar sommige kinderlijk eenvoudige encryptietechnieken zijn theoretisch onkraakbaar. Zoals bv. de one time pad

Sterker nog, onze Belgische universiteiten staan over de hele wereld bekend om haar wiskundigen die zich met cryptografie bezig houden. De wereldwijd belangrijkste encryptiestandaard, Rijndael, werd door Leuvense studenten ontwikkeld. Die PGP software gebruikt die encryptiestandaard, ook. Het staat tegenwoordig bekend als de Advanced Encryption Standard, AES, en wordt oa. militair vrijwel overal ingezet.

Dat een aantal mensen van het gerecht en de politie het lastig vinden dat een aantal criminelen gebruik maken van software zoals PGP, betekent op geen enkele manier dat het gebruik van en het voorzien van telefoons met PGP, op zichzelf, illegaal is. Helemaal niet. Het is dus vrij onnozel om in jullie journaal te laten uitschijnen als zou dat wel het geval zijn. Dat is het niet. Zeker in extreme tijden zoals nu is het gevaarlijk wanneer de media zoiets doet uitschijnen. Straks maakt men de gewone burger weer bang over allerlei dingen, en dan gaat de politiek domme eisen stellen zoals het verbieden van encryptie.

Het versleutelen van berichten is een erg belangrijk onderdeel, in de technologie, om die technologie veilig en betrouwbaar te laten werken. Het ontbreken van de mogelijkheid om berichten en gegevens te versleutelen zou werkelijk catastrofale gevolgen hebben.

Het zou ook geen enkele crimineel weerhouden toch één en ander te encrypteren. Encryptie verbieden zou letterlijk alleen maar de onschuldige burger schaden. De crimineel zou er net voordeel uit halen; de crimineel wordt het daardoor plots veel eenvoudiger gemaakt om bij onschuldige burgers (die geen encryptie meer kunnen gebruiken) digitaal in te breken.

Denk in deze extreme tijden goed na wat je in jullie mediakanaal, het nationale nieuws, doet uitschijnen. Als techneut vind ik het bijzonder onverantwoordelijk van jullie. OOK en vooral wanneer de bedoeling was om het nieuws zo dom mogelijk te brengen. Alsof onze burgers het niet zouden aankunnen om correct geïnformeerd te worden over deze materie.

Onlife, Hoe de digitale wereld je leven bepaalt

Moraal filosofe Katleen Gabriels presenteerde gisteren haar boekOnlife, Hoe de digitale wereld je leven bepaalt’. Ik ben dus maar eens gaan kijken. Ik was onder de indruk.

Haar uiteenzetting was gebalanceerd; ze klonk geïnformeerd. Het debat met oa. Sven Gatz, Pedro De Bruyckere en Karel Verhoeven was eigenlijk ook wel cava. Ik ben dus erg benieuwd naar het boek.

Na een Spinoza-kenner hebben we dus nu ook een moraal filosofe die zich met oa. Internet of Things dingen zal gaan bezig houden. Ik vind het dus wel goed dat de filosofie van’t land zich eindelijk eens komt moeien. De consument gelooft ons, techneuten, toch niet dat al die rommel die ze gekocht hebben dikke rotzooi is. Dus misschien dat ze wat gaan luisteren naar s’lands filosofen? Ik denk het niet. Maar slechter zal de situatie er ook niet van worden, hé?

Enfin. Medeneurdjes contacteer die Katleen en vertel over wat je zoal hebt meegemaakt wat onethisch is. Wie weet verwerkt ze je verhaal in een uiteenzetting of volgend boek? Je kan dat nooit weten he. Je moest trouwens toch eens van die zolder afkomen.

Geef vorm

We zijn goed. We tonen dat door ons respect voor privacy en veiligheid te combineren. Kennis is daar onontbeerlijk voor. Ik pleit voor investeren in techneuten die de twee beheersen.

Onze overheid moet niet alles investeren in miljoenen voor het bestrijden van computervredebreuk; wel ook investeren in betere software.

Belgische bedrijven maken soms software. Ze moeten aangemoedig worden, gestuurd, om het goede te doen.

Ik zou graag van ons centrum cybersecurity zien dat ze bedrijven aanmoedigt om goede en dus veilige software te maken. We moeten ook inzetten op repressie. Maar we moeten net zo veel inzetten op hoge kwaliteit.

Wij denken wel eens dat, ach, wij te klein zijn. Maar dat is niet waar. Als wij beslissen dat hier, in België, de software goed moet zijn: dan creërt dat een markt die zich zal aanpassen aan wat wij willen. Het is zaak standvastig te zijn.

Wanneer wij zeggen dat a – b hier welkom is, of niet, geven we vorm aan technologie.

Ik verwacht niet minder van mijn land. Geef vorm.

Putting an LRU in your code

For the ones who didn’t find the LRU in Tracker’s code (and for the ones who where lazy).

Let’s say we will have instances of something called a Statement. Each of those instances is relatively big. But we can recreate them relatively cheap. We will have a huge amount of them. But at any time we only need a handful of them.

The ones that are most recently used are most likely to be needed again soon.

First you make a structure that will hold some administration of the LRU:

typedef struct {
	Statement *head;
	Statement *tail;
	unsigned int size;
	unsigned int max;
} StatementLru;

Then we make the user of a Statement (a view or a model). I’ll be using a Map here. You can in Qt for example use QMap for this. Usually I want relatively fast access based on a key. You could also each time loop the stmt_lru to find the instance you want in useStatement based on something in Statement itself. That would rid yourself of the overhead of a map.

class StatementUser
{
	StatementUser();
	~StatementUser();
	void useStatement(KeyType key);
private:
	StatementLru stmt_lru;
	Map<KeyType, Statement*> stmts;
	StatementFactory stmt_factory;
}

Then we will add to the private fields of the Statement class the members prev and next: We’ll make a circular doubly linked list.

class Statement: QObject {
	Q_OBJECT
    ...
private:
	Statement *next;
	Statement *prev;
};

Next we initialize the LRU:

StatementUser::StatementUser() 
{
	stmt_lru.max = 500;
	stmt_lru.size = 0;		
}

Then we implement using the statements

void StatementUser::useStatement(KeyType key)
{
	Statement *stmt;

	if (!stmts.get (key, &stmt)) {

		stmt = stmt_factory.createStatement(key);

		stmts.insert (key, stmt);

		/* So the ring looks a bit like this: *
		 *                                    *
		 *    .--tail  .--head                *
		 *    |        |                      *
		 *  [p-n] -> [p-n] -> [p-n] -> [p-n]  *
		 *    ^                          |    *
		 *    `- [n-p] <- [n-p] <--------'    */

		if (stmt_lru.size >= stmt_lru.max) {
			Statement *new_head;

		/* We reached max-size of the LRU stmt cache. Destroy current
		 * least recently used (stmt_lru.head) and fix the ring. For
		 * that we take out the current head, and close the ring.
		 * Then we assign head->next as new head. */

			new_head = stmt_lru.head->next;
			auto to_del = stmts.find (stmt_lru.head);
			stmts.remove (to_del);
			delete stmt_lru.head;
			stmt_lru.size--;
			stmt_lru.head = new_head;
		} else {
			if (stmt_lru.size == 0) {
				stmt_lru.head = stmt;
				stmt_lru.tail = stmt;
			}
		}

	/* Set the current stmt (which is always new here) as the new tail
	 * (new most recent used). We insert current stmt between head and
	 * current tail, and we set tail to current stmt. */

		stmt_lru.size++;
		stmt->next = stmt_lru.head;
		stmt_lru.head->prev = stmt;

		stmt_lru.tail->next = stmt;
		stmt->prev = stmt_lru.tail;
		stmt_lru.tail = stmt;

	} else {
		if (stmt == stmt_lru.head) {

		/* Current stmt is least recently used, shift head and tail
		 * of the ring to efficiently make it most recently used. */

			stmt_lru.head = stmt_lru.head->next;
			stmt_lru.tail = stmt_lru.tail->next;
		} else if (stmt != stmt_lru.tail) {

		/* Current statement isn't most recently used, make it most
		 * recently used now (less efficient way than above). */

		/* Take stmt out of the list and close the ring */
			stmt->prev->next = stmt->next;
			stmt->next->prev = stmt->prev;

		/* Put stmt as tail (most recent used) */
			stmt->next = stmt_lru.head;
			stmt_lru.head->prev = stmt;
			stmt->prev = stmt_lru.tail;
			stmt_lru.tail->next = stmt;
			stmt_lru.tail = stmt;
		}

	/* if (stmt == tail), it's already the most recently used in the
	 * ring, so in this case we do nothing of course */
	}

	/* Use stmt */

	return;
}

In case StatementUser and Statement form a composition (StatementUser owns Statement, which is what makes most sense), don’t forget to delete the instances in the destructor of StatementUser. In the example’s case we used heap objects. You can loop the stmt_lru or the map here.

StatementUser::~StatementUser()
{
	Map<KeyType, Statement*>::iterator i;
    	for (i = stmts.begin(); i != stmts.end(); ++i) {
		delete i.value();
	}
}

Secretly reusing my own LRU code

Last week, I secretly reused my own LRU code in the model of the editor of a CNC machine (has truly huge files, needs a statement editor). I rewrote my own code, of course. It’s Qt based, not GLib. Wouldn’t work in original form anyway. But the same principle. Don’t tell Jürg who helped me write that, back then.

Extra points and free beer for people who can find it in Tracker’s code.

Ik heb nog eens m’n steentje bijgedrage

Heb dit geyoutubed gisterenavond toen m’n duikclub was komen vlees op een rooster en gloeiende koolen verbranden. Oh met den Hans Teeuwen en de Theo Maassen zijn we ook aan’t lachen geweest. Na één uur de Geubels vond ik dat we ook eens andere komieken moeten laten zien aan de wereld. Duikers zijn een leutige bende. Een beetje zoals ons, neurdjes. Ik raad alle andere computermannekens aan om ook te gaan duiken. Toffe mensen.

https://www.youtube.com/watch?v=yi1XwrPwz1k

Dit is niet de eerste keer dat ik dit filmpje blog. Maar ik zou graag meer Astronomers willen tegenkomen zodat we misschien effectief die planeet hier kunnen opkopen. En dan een gigantisch ruimteschip maken. Samen! Brand new world order 2.0. Or, shut the fuck up.

Daarnet nog een goed idee besproken met diene dat is blijven crashen. Ik wil hier in Rillaar toch ooit eens een zwembad zetten. Misschien moet ik er ééntje maken in de vorm van een schotel? Dan kunnen de duikers bij mij komen oefenen.

Gebruik maken van verbanden tussen metadata

Ik beweerde onlangs ergens dat een systeem dat verbanden (waar, wanneer, met wie, waarom) in plaats van louter metadata (titel, datum, auteur, enz.) over content verzamelt een oplossing zou kunnen bieden voor het probleem dat gebruikers van digitale media meer en meer zullen hebben; namelijk dat ze teveel materiaal gaan verzameld hebben om er ooit nog eens iets snel genoeg in terug te vinden.

Ik denk dat verbanden meer gewicht moeten krijgen dan louter de metadata omdat het door middel van verbanden is dat wij mensen in onze hersenen informatie onthouden. Niet door middel van feiten (titel, datum, auteur, enz.) maar wel door middel van verbanden (waar, wanneer, met wie, waarom) .

Ik gaf als hypothetisch voorbeeld dat ik een video wilde vinden die ik gekeken had met Erika toen ik op vakantie was met haar en die zij als super tof had gemarkeerd.

Wat zijn de verbanden die we moeten verzamelen? Dit is een eenvoudig oefeningetje in analyse: gewoon de zelfstandige naamwoorden onderlijnen en het probleem opnieuw uitschrijven:

  • Dat ik op vakantie was toen ik hem laatst zag. Dat is een point of interest (waar)
  • Dat het een film is (wat, is een feit over mijn te vinden onderwerp en dus geen verband. Maar we nemen dit mee)
  • Met wie ik de film gekeken heb en wanneer (met wie, wanneer)
  • Dat Erika, met wie ik de film gekeken heb, de film super tof vond (waarom)

Dus laat ik deze use-case eens in RDF gieten en oplossen met SPARQL. Dit moeten we verzamelen. Ik schrijf het in pseudo TTL. Bedenk er even bij dat deze ontology helemaal bestaat:

<erika> a Person ; name "Erika" .
<vakantiePlek> a PointOfInterest ; title "De vakantieplek" .
<filmA> a Movie ; lastSeenAt <vakantiePlek> ; sharedWith <erika>; title "The movie" .
<erika> likes <filmA> .

Dit is daarna de SPARQL query:

SELECT ?m { ?v a Movie ; title ?m . ?v lastSeenAt ?p . ?p title ?pt . ?v sharedWith <erika> . <erika> likes ?v . FILTER (?pt LIKE '%vakantieplek%') }

Ik laat het als een oefening aan de lezer om dit naar de ontology Nepomuk om te zetten (volgens mij kan het deze hele use-case aan). En dan kan je dat eens op je N9 of je standaard GNOME desktop testen met de tool tracker-sparql. Wedden dat het werkt. :-)

Het grote probleem is inderdaad de data aquisitie van de verbanden. De query maken is vrij eenvoudig. De ontology vastleggen en afspreken met alle partijen al wat minder. De informatie verzamelen is dé moeilijkheid.

Oh ja. En eens verzameld, de informatie veilig bijhouden zonder dat mijn privacy geschonden wordt. Dat lijkt tegenwoordig gewoonweg onmogelijk. Helaas.

Het is in ieder geval niet nodig dat een supercomputer of zo dit centraal moet oplossen (met AI en heel de gruwelijk complexe hype zooi van vandaag).

Ieder klein toestelletje kan dit soort use-cases zelfstandig oplossen. De bovenstaande inserts en query zijn eenvoudig op te lossen. SQLite doet dit in een paar milliseconden met een gedenormalizeerd schema. Uw fancy hipster NoSQL oplossing waarschijnlijk ook.

Dat is omdat het gewicht van data aquisitie op de verbanden ligt in plaats van op de feiten.

De dierentuin: geboortebeperking versus slachten

Michel Vandenbosch versus Dirk Draulans: ben ooit zo’n 15 jaar vegetariër geweest om vandaag tevreden te zijn over dit goed voorbereid en mooi gebalanceerd debat. Mijn dank aan de redactie van Terzake.

Ik was het eens met beide heren. Daarom was dit een waardig filosofische discussie: geboortebeperking versus het aan de leeuwen voeden van overbodige dieren plus het nut en de zin van goed gerunde dierenparken. Dat nut is me duidelijk: educatie voor de dwaze mens (z’n kinders, in de hoop dat de opvolging minder dwaas zal zijn)

Hoewel ik het eens was met beide ben ik momenteel zelf meer voor het aan de leeuwen voeden van overbodige dieren dan dat ik voor geboortebeperking van wel of niet bedreigde diersoorten ben. Dat leek me, met groot respect voor Vandenbosch’s, Draulans’ standpunt te zijn. Ethisch snapte ik Vandenbosch ook: is het niet beter om aan geboortebeperking te doen teneinde het leed van een slachting te vermijden?

Ik kies voor het standpunt van Draulans omdat dit het meeste de echte wereld nabootst. Ik vind het ook zeer goed dat het dierenpark de slachting van de giraffe aan de kinderen toonde. Want dit is de werkelijkheid. Onze kinderen moeten de werkelijkheid zien. We moeten met ons verstand de werkelijkheid nabootsen. Geen eufemisme zoals het doden van een giraffe een euthanasie noemen. Laten we onze kinders opvoeden met de werkelijkheid.

Rusland

De cultuur die heerst, leeft,  ploert en zweet op het grondgebied van Rusland zal niet verdwijnen. Zelfs niet na een nucleaire aanval tenzij die echt verschrikkelijk wreedaardig is. En dan zou ik ons verdommen dat we dat gedaan hebben zoals ik hoor te doen. Wij, Europeanen, moeten daar mee leven zoals zij met ons moeten leven.

Mr. Dillon; smartphone innovation in Europe ought to be about people’s privacy

Dear Mark,

Your team and you yourself are working on the Jolla Phone. I’m sure that you guys are doing a great job and although I think you’ve been generating hype and vaporware until we can actually buy the damn thing, I entrust you with leading them.

As their leader you should, I would like to, allow them to provide us with all of the device’s source code and build environments of their projects so that we can have the exact same binaries. With exactly the same I mean that it should be possible to use MD5 checksums. I’m sure you know what that means and you and I know that your team knows how to provide geeks like me with this. I worked with some of them together during Nokia’s Harmattan and Fremantle and we both know that you can easily identify who can make this happen.

The reason why is simple: I want Europe to develop a secure phone similar to how, among other open source projects, the Linux kernel can be trusted. By peer review of the source code.

Kind regards,

A former Harmattan developer who worked on a component of the Nokia N9 that stores the vast majority of user’s privacy.

ps. I also think that you should reunite Europe’s finest software developers and secure the funds to make this workable. But that’s another discussion which I’m eager to help you with.

IPC performance, the report

The Tracker team will be doing a codecamp this month. Among the subjects we will address is the IPC overhead of tracker-store, our RDF query service.

We plan to investigate whether a direct connection with our SQLite database is possible for clients. Jürg did some work on this. Turns out that due to SQLite not being MVCC we need to override some of SQLite’s VFS functions and perhaps even implement ourselves a custom page cache.

Another track that we are investigating involves using a custom UNIX domain socket and sending the data over in such a way that at either side the marshalling is cheap.

For that idea I asked Adrien Bustany, a computer sciences student who’s doing an internship at Codeminded, to develop three tests: A test that uses D-Bus the way tracker-store does (by using the DBusMessage API directly), a test that uses an as ideal as possible custom protocol and technique to get the data over a UNIX domain socket and a simple program that does the exact same query but connects to SQLite by itself.

Here’s the report:

Exposing a SQLite database remotely: comparison of various IPC methods

By Adrien Bustany
Computer Sciences student
National Superior School of Informatics and Applied Mathematics of Grenoble (ENSIMAG)

This study aims at comparing the overhead of an IPC layer when accessing a SQLite database. The two IPC methods included in this comparison are DBus, a generic message passing system, and a custom IPC method using UNIX sockets. As a reference, we also include in the results the performance of a client directly accessing the SQLite database, without involving any IPC layer.

Comparison methodology

In this section, we detail what the client and server are supposed to do during the test, regardless of the IPC method used.

The server has to:

  1. Open the SQLite database and listen to the client requests
  2. Prepare a query at the client’s request
  3. Send the resulting rows at the client’s request

Queries are only “SELECT” queries, no modification is performed on the database. This restriction is not enforced on server side though.

The client has to:

  1. Connect to the server
  2. Prepare a “SELECT” query
  3. Fetch all the results
  4. Copy the results in memory (not just fetch and forget them), so that memory pages are really used

Test dataset

For testing, we use a SQLite database containing only one table. This table has 31 columns, the first one is the identifier and the 30 others are columns of type TEXT. The table is filled with 300 000 rows, with randomly generated strings of 20 ASCII lowercase characters.

Implementation details

In this section, we explain how the server and client for both IPC methods were implemented.

Custom IPC (UNIX socket based)

In this case, we use a standard UNIX socket to communicate between the client and the server. The socket protocol is a binary protocol, and is detailed below. It has been designed to minimize CPU usage (there is no marshalling/demarshalling on strings, nor intensive computation to decode the message). It is fast over a local socket, but not suitable for other types of sockets, like TCP sockets.

Message types

There are two types of operations, corresponding to the two operations of the test: prepare a query, and fetch results.

Message format

All numbers are encoded in little endian form.

Prepare

Client sends:

Size Contents
4 bytes Prepare opcode (0×50)
4 bytes Size of the query (without trailing \0)
Query, in ASCII

Server answers:

Size Contents
4 bytes Return code of the sqlite3_prepare_v2 call

Fetch

Client sends:

Size Contents
4 bytes Fetch opcode (0×46)

Server sends rows grouped in fixed size buffers. Each buffer contains a variable number of rows. Each row is complete. If some padding is needed (when a row doesn’t fit in a buffer, but there is still space left in the buffer), the server adds an “End of Page” marker. The “End of page” marker is the byte 0xFF. Rows that are larger than the buffer size are not supported.

Each row in a buffer has the following format:

Size Contents
4 bytes SQLite return code. This is generally SQLITE_ROW (there is a row to read), or SQLITE_DONE (there are no more rows to read). When the return code is not SQLITE_ROW, the rest of the message must be ignored.
4 bytes Number of columns in the row
4 bytes Index of trailing \0 for first column (index is 0 after the “number of columns” integer, that is, index is equal to 0 8 bytes after the message begins)
4 bytes Index of trailing \0 for second column
4 bytes Index of trailing \0 for last column
Row data. All columns are concatenated together, and separated by \0

For the sake of clarity, we describe here an example row

100 4 1 7 13 19 1\0aaaaa\0bbbbb\0ccccc\0

The first 100 is the return code, in this case SQLITE_ROW. This row has 4 columns. The 4 following numbers are the offset of the \0 terminating each column in the row data. Finally comes the row data.

Memory usage

We try to minimize the calls to malloc and memcpy in the client and server. As we know the size of a buffer, we allocate the memory only once, and then use memcpy to write the results to it.

DBus

The DBus server exposes two methods, Prepare and Fetch.

Prepare

The Prepare method accepts a query string as a parameter, and returns nothing. If the query preparation fails, an error message is returned.

Fetch

Ideally, we should be able to send all the rows in one batch. DBus, however, puts a limitation on the message size. In our case, the complete data to pass over the IPC is around 220MB, which is more than the maximum size allowed by DBus (moreover, DBus marshalls data, which augments the message size a little). We are therefore obliged to split the result set.

The Fetch method accepts an integer parameter, which is the number of rows to fetch, and returns an array of rows, where each row is itself an array of columns. Note that the server can return less rows than asked. When there are no more rows to return, an empty array is returned.

Results

All tests are ran against the dataset described above, on a warm disk cache (the database is accessed several time before every run, to be sure the entire database is in disk cache). We use SQLite 3.6.22, on a 64 bit Linux system (kernel 2.6.33.3). All test are ran 5 times, and we use the average of the 5 intermediate results as the final number.

For the custom IPC, we test with various buffer sizes varying from 1 to 256 kilobytes. For DBus, we fetch 75000 rows with every Fetch call, which is close to the maximum we can fetch with each call (see the paragraph on DBus message size limitation).

The first tests were to determine the optimal buffer size for the UNIX socket based IPC. The following graph describes the time needed to fetch all rows, depending on the buffer size:

The graph shows that the IPC is the fastest using 64kb buffers. Those results depend on the type of system used, and might have to be tuned for different platforms. On Linux, a memory page is (generally) 4096 bytes, as a consequence buffers smaller than 4kB will use a full memory page when sent over the socket and waste memory bandwidth. After determining the best buffer size for socket IPC, we run tests for speed and memory usage, using a buffer size of 64kb for the UNIX socket based method.

Speed

We measure the time it takes for various methods to fetch a result set. Without any surprise, the time needed to fetch the results grows linearly with the amount of rows to fetch.

IPC method Best time
None (direct access) 2910 ms
UNIX socket 3470 ms
DBus 12300 ms

Memory usage

Memory usage varies greatly (actually, so much that we had to use a log scale) between IPC methods. DBus memory usage is explained by the fact that we fetch 75 000 rows at a time, and that it has to allocate all the message before sending it, while the socket IPC uses 64 kB buffers.

Conclusions

The results clearly show that in such a specialized case, designing a custom IPC system can highly reduce the IPC overhead. The overhead of a UNIX socket based IPC is around 19%, while the overhead of DBus is 322%. However, it is important to take into account the fact that DBus is a much more flexible system, offering far more features and flexibility than our socket protocol. Comparing DBus and our custom UNIX socket based IPC is like comparing an axe with a swiss knife: it’s much harder to cut the tree with the swiss knife, but it also includes a tin can opener, a ball pen and a compass (nowadays some of them even include USB keys).

The real conclusion of this study is: if you have to pass a lot of data between two programs and don’t need a lot of flexibility, then DBus is not the right answer, and never intended to be.

The code source used to obtain these results, as well as the numbers and graphs used in this document can be checked out from the following git repository: git://git.mymadcat.com/ipc-performance . Please check the various README files to see how to reproduce them and/or how to tune the parameters.

More introduction to RDF and SPARQL

Introduction

I plan to give an introduction to features like COUNT, FILTER REGEX and GROUP BY which are supported by Tracker‘s SPARQL engine. We support more such features but I have to start the introduction somewhere. And overloading people with introductions to all features wont help me much with explaining things.

Since my last introduction to RDF and SPARQL I have added a few relationships and actors to the game.

We have Morrel, Max and Sasha being dogs, Sheeba and Query are cats, Picca is still a parrot, Fred and John are contacts. Fred claims that John is his friend. I changed the ontology to allow friendships between the animals too: Sasha claims that Morrel and Max are her friends. Sheeba claims Query is her friend. John bought Query. Fred being inspired by John decided to also get some pets: Morrel, Sasha and Sheeba.

Ontology

Let’s put this story in Turtle:

<test:Picca> a test:Parrot, test:Pet ;
	test:name "Picca" .

<test:Max> a test:Dog, test:Pet ;
	test:name "Max" .

<test:Morrel> a test:Dog, test:Pet ;
	test:name "Morrel" ;
	test:hasFriend <test:Max> .

<test:Sasha> a test:Dog, test:Pet ;
	test:name "Sasha" ;
	test:hasFriend <test:Morrel> ;
	test:hasFriend <test:Max> .

<test:Sheeba> a test:Cat, test:Pet ;
	test:name "Sheeba" ;
	test:hasFriend <test:Query> .

<test:Query> a test:Cat, test:Pet ;
	test:name "Query" .

<test:John> a test:Contact ;
	test:owns <test:Max> ;
	test:owns <test:Picca> ;
	test:owns <test:Query> ;
	test:name "John" .

<test:Fred> a test:Contact ;
	test:hasFriend <test:John> ;
	test:name "Fred" ;
	test:owns <test:Morrel> ;
	test:owns <test:Sasha> ;
	test:owns <test:Sheeba> .

Querytime!

Let’s first start with all friend relationships:

SELECT ?subject ?friend
WHERE { ?subject test:hasFriend ?friend }

  test:Morrel, test:Max
  test:Sasha, test:Morrel
  test:Sasha, test:Max
  test:Sheeba, test:Query
  test:Fred, test:John

Just counting these is pretty simple. In SPARQL all selectable fields must have a variable name, so we add the “as c” here.

SELECT COUNT (?friend) AS c
WHERE { ?subject test:hasFriend ?friend }

  5

We counted friend relationships, of course. Let’s say we want to count how many friends each subject has. This is a more interesting query than the previous one.

SELECT ?subject COUNT (?friend) AS c
WHERE { ?subject test:hasFriend ?friend }
GROUP BY ?subject

  test:Fred, 1
  test:Morrel, 1
  test:Sasha, 2
  test:Sheeba, 1

Actually, we’re only interested in the human friends:

SELECT ?subject COUNT (?friend) AS c
WHERE { ?subject test:hasFriend ?friend .
        ?friend a test:Contact
} GROUP BY ?subject

  test:Fred, 1

No no, we are only interested in friends that are either cats or dogs:

SELECT ?subject COUNT (?friend) AS c
WHERE { ?subject test:hasFriend ?friend .
       ?friend a ?type .
       FILTER ( ?type = test:Dog || ?type = test:Cat)
} GROUP BY ?subject"

  test:Morrel, 1
  test:Sasha, 2
  test:Sheeba, 1

Now we are only interested in friends that are either a cat or a dog, but whose name starts with a ‘S’.

SELECT ?subject COUNT (?friend) as c
WHERE { ?subject test:hasFriend ?friend ;
                 test:name ?n .
       ?friend a ?type .
       FILTER ( ?type = test:Dog || ?type = test:Cat) .
       FILTER REGEX (?n, '^S', 'i')
} GROUP BY ?subject

  test:Sasha, 2
  test:Sheeba, 1

Conclusions

Should we stop talking about ontologies and start talking about searchboxes and user interfaces instead? Although I certainly agree more UI-stuff is needed, I’m not sure yet. RDF and SPARQL are also about relationships and roles. Not just about matching stuff. Whenever we explain the new Tracker to people, most are stuck with ‘matching’ in their mind. They don’t think about a lot of other use-cases.

Such a search is just one use-case starting point: user entered a random search string and gives zero other meaning about what he needs. Many more situations can be starting points: When I select a contact in a user interface designed to show an archive of messages that he once sent to me, the searchbox becomes much more narrow, much more helpful.

As soon as you have RDF and SPARQL, and with Tracker you do, an application developer can start taking into account relationships between resources: The relationship between a contact in Instant Messaging and the attachments in an E-mail that he as a person has sent to you. Why not combine it with friendship relationships synced from online services?

With a populated store you can make the relationship between a friend who joined you on a trip, and photos of a friend of your friend who suggested the holiday location.

With GeoClue integration we could link his photos up with actual location markers. You’d find these photos that came from the friend of your friend, and we could immediately feed the location markers to the GPS software on your phone.

I really hope application developers have more imagination than just global searchboxes.

And this is just a use-case that is technically already possible with today’s high-end phones.