RE: Bye Facebook

Wim made a stir in the land of the web. Good for Wim that he rid himself of the shackles of social media.

But how will we bring a generation of people, who are now more or less addicted to social media, to a new platform? And what should that platform look like?

I’m not a anthropologist, but I believe human nature of organizing around new concepts and techniques is that we, humans, start central and monolithic. Then we fine-tune it. We figure out that the central organization and monolithic implementation of it becomes a limiting factor. Then we decentralize it.

The next step for all those existing and potential so-called ‘online services’ is to become fully decentralized.

Every family or home should have its own IMAP and SMTP server. Should that be JMAP instead? Probably. But that ain’t the point. The fact that every family or home will have its own, is. For chat, XMPP’s s2s is like SMTP. Postfix is an implementation of SMTP like ejabberd is for XMPP’s s2s. We have Cyrus, Dovecot and others for IMAP, which is the c2s of course. And soon we’ll probably have JMAP, too. Addressability? IPv6.

Why not something like this for social media? For the next online appliance, too? Augmented reality worlds can be negotiated in a distributed fashion. Why must Second Life necessarily be centralized? Surely we can run Linden Lab’s server software, locally.

Simple, because money is not interested in anything non-centralized. Not yet.

In the other news, the Internet stopped working truly well ever since money became its driving factor.

ps. The surest way to corrupt a youth is to instruct him to hold in higher esteem those who think alike than those who think different. Quote by Friedrich Nietzsche.

Asynchronous undoable and redoable APIs

Combining QFuture with QUndoCommand made a lot of sense for us. The undo and the redo methods of the QUndoCommand can also be asynchronous, of course. We wanted to use QFuture without involving threads, because our asynchronosity is done through a process and IPC, and not a thread. It’s the design mistake of QtConcurrent‘s run method, in my opinion. That meant using QFutureInterface instead (which is undocumented, but luckily public – so it’ll remain with us until at least Qt’s 6.y.z releases).

So how do we make a QUndoCommand that has a undo, and that has a redo method that returns a asynchronous QFuture<ResultType>?

We just did that, today. I’m very satisfied with the resulting API and design. It might have helped if QUndoStack would be a QUndoStack<T> and QUndoCommand would have been a QUndoCommand<T> with undo and redo’s return type being T. Just an idea for the Qt 6.y.z developers.

The undoable editor that can open > 4 GB text files

We are making an editor for industrial uses at Heidenhain. This is to make big Klartext programs, editable. I’m sure other industries could also use that.

Nowadays these programs often come out of a conversion from a CAD-CAM format. Before you can mill and turn your pesky military secrets on one of the machines controlled by a Heidenhain set, you’ll have to tweak the program that you converted from your CAD-CAM product. We are making the editor for that.

I wrote on this blog how we will instantaneously open those >4GB files, ready for editing. It looks a lot like how I made the E-mail client modest open the headers instantaneously on the N900. Basically, having a partition or index table that gets mmapped.

We’re also making the overlaying (the changes made by the user) undoable. The APIs for that kinda look like this. All examples on my blog are amateur extracts of the real thing, of course.

I feel like it’s actually going to work out. Architecturally and organizationally the other developers in our team are getting at the right level of expertise and sense of wanting this.

That is most important for anything to make it happen.

It feels a bit like how Nokia was: I’m learning a lot about myself from techleading: how to propose a design, concept or idea; how to convince deeply technical people; how to push others to go further than what they can already do. How to make a team quit competing and start sharing a common goal. The infrastructure for that was provided to me by Nokia. At Heidenhain, I feel like having played a small role in it.

Making something that is ‘undoable editable’ with Qt

Among the problems we’ll face is that we want asynchronous APIs that are undoable and that we want to switch to read only, undoable editing, non-undoable editing and that QML doesn’t really work well with QFuture. At least not yet. We want an interface that is easy to talk with from QML. Yet we want to switch between complicated behaviors.

We will also want synchronous mode and asynchronous mode. Because I just invented that requirement out of thin air.

Ok, first the “design”. We see a lot of behaviors, for something that can do something. The behaviors will perform for that something, the actions it can do. That is the strategy design pattern, then. It’s the one about ducks and wing fly behavior and rocket propelled fly behavior and the ostrich that has a can’t fly behavior. For undo and redo, we have the command pattern. We have this neat thing in Qt for that. We’ll use it. We don’t reinvent the wheel. Reinventing the wheel is stupid.

Let’s create the duck. I mean, the thing-editor as I will use “Thing” for the thing that is being edited. We want copy (sync is sufficient), paste (must be aysnc), and edit (must be async). We could also have insert and delete, but those APIs would be just like edit. Paste is usually similar to insert, of course. Except that it can be a combined delete and insert when overwriting content. The command pattern allows you to make such combinations. Not the purpose of this example, though.

Enough explanation. Let’s start! The ThingEditor, is like the flying Duck in strategy. This is going to be more or less the API that we will present to the QML world. It could be your ViewModel, for example (ie. you could let your ThingViewModel subclass ThingEditor).

class ThingEditor : public QObject
{
    Q_OBJECT

    Q_PROPERTY ( ThingEditingBehavior* editingBehavior READ editingBehavior
                 WRITE setEditingBehavior NOTIFY editingBehaviorChanged )
    Q_PROPERTY ( Thing* thing READ thing WRITE setThing NOTIFY thingChanged )

public:
    explicit ThingEditor( QSharedPointer<Thing> &a_thing,
            ThingEditingBehavior *a_editBehavior,
            QObject *a_parent = nullptr );

    explicit ThingEditor( QObject *a_parent = nullptr );

    Thing* thing() const { return m_thing.data(); }
    virtual void setThing( QSharedPointer<Thing> &a_thing );
    virtual void setThing( Thing *a_thing );

    ThingEditingBehavior* editingBehavior() const { return m_editingBehavior.data(); }
    virtual void setEditingBehavior ( ThingEditingBehavior *a_editingBehavior );

    Q_INVOKABLE virtual void copyCurrentToClipboard ( );
    Q_INVOKABLE virtual void editCurrentAsync( const QString &a_value );
    Q_INVOKABLE virtual void pasteCurrentFromClipboardAsync( );

signals:
    void editingBehaviorChanged ();
    void thingChanged();
    void editCurrentFinished( EditCurrentCommand *a_command );
    void pasteCurrentFromClipboardFinished( EditCurrentCommand *a_command );

private slots:
    void onEditCurrentFinished();
    void onPasteCurrentFromClipboardFinished();

private:
    QScopedPointer<ThingEditingBehavior> m_editingBehavior;
    QSharedPointer<Thing> m_thing;
    QList<QFutureWatcher<EditCurrentCommand*> *> m_editCurrentFutureWatchers;
    QList<QFutureWatcher<EditCurrentCommand*> *> m_pasteCurrentFromClipboardFutureWatchers;
};

For the implementation of this class, I’ll only provide the non-obvious pieces. I’m sure you can do that setThing, setEditingBehavior and the constructor yourself. I’m also providing it only once, and also only for the EditCurrentCommand. The one about paste is going to be exactly the same.

void ThingEditor::copyCurrentToClipboard ( )
{
    m_editingBehavior->copyCurrentToClipboard( );
}

void ThingEditor::onEditCurrentFinished( )
{
    QFutureWatcher<EditCurrentCommand*> *resultWatcher
            = static_cast<QFutureWatcher<EditCurrentCommand*>*> ( sender() );
    emit editCurrentFinished ( resultWatcher->result() );
    if (m_editCurrentFutureWatchers.contains( resultWatcher )) {
        m_editCurrentFutureWatchers.removeAll( resultWatcher );
    }
    delete resultWatcher;
}

void ThingEditor::editCurrentAsync( const QString &a_value )
{
    QFutureWatcher<EditCurrentCommand*> *resultWatcher
            = new QFutureWatcher<EditCurrentCommand*>();
    connect ( resultWatcher, &QFutureWatcher<EditCurrentCommand*>::finished,
              this, &ThingEditor::onEditCurrentFinished, Qt::QueuedConnection );
    resultWatcher->setFuture ( m_editingBehavior->editCurrentAsync( a_value ) );
    m_editCurrentFutureWatchers.append ( resultWatcher );
}

For QUndo we’ll need a QUndoCommand. For each undoable action we indeed need to make such a command. You could add more state and pass it to the constructor. It’s common, for example, to pass Thing, or the ThingEditor or the behavior (this is why I used QSharedPointer for those: as long as your command lives in the stack, you’ll need it to hold a reference to that state).

class EditCurrentCommand: public QUndoCommand
{
public:
    explicit EditCurrentCommand( const QString &a_value,
                                 QUndoCommand *a_parent = nullptr )
        : QUndoCommand ( a_parent )
        , m_value ( a_value ) { }
    void redo() Q_DECL_OVERRIDE {
       // Perform action goes here
    }
    void undo() Q_DECL_OVERRIDE {
      // Undo what got performed goes here
    }
private:
    const QString &m_value;
};

You can (and probably should) also make this one abstract (and/or a so called pure interface), as you’ll usually want many implementations of this one (one for every kind of editing behavior). Note that it leaks the QUndoCommand instances unless you handle them (ie. storing them in a QUndoStack). That in itself is a good reason to keep it abstract.

class ThingEditingBehavior : public QObject
{
    Q_OBJECT

    Q_PROPERTY ( ThingEditor* editor READ editor WRITE setEditor NOTIFY editorChanged )
    Q_PROPERTY ( Thing* thing READ thing NOTIFY thingChanged )

public:
    explicit ThingEditingBehavior( ThingEditor *a_editor,
                                   QObject *a_parent = nullptr )
        : QObject ( a_parent )
        , m_editor ( a_editor ) { }

    explicit ThingEditingBehavior( QObject *a_parent = nullptr )
        : QObject ( a_parent ) { }

    ThingEditor* editor() const { return m_editor.data(); }
    virtual void setEditor( ThingEditor *a_editor );
    Thing* thing() const;

    virtual void copyCurrentToClipboard ( );
    virtual QFuture<EditCurrentCommand*> editCurrentAsync( const QString &a_value, bool a_exec = true );
    virtual QFuture<EditCurrentCommand*> pasteCurrentFromClipboardAsync( bool a_exec = true );

protected:
    virtual EditCurrentCommand* editCurrentSync( const QString &a_value, bool a_exec = true );
    virtual EditCurrentCommand* pasteCurrentFromClipboardSync( bool a_exec = true );

signals:
    void editorChanged();
    void thingChanged();

private:
    QPointer<ThingEditor> m_editor;
    bool m_synchronous = true;
};

That setEditor, the constructor, etc: these are too obvious to write here. Here are the non-obvious ones:

void ThingEditingBehavior::copyToClipboard ( )
{
}

EditCurrentCommand* ThingEditingBehavior::editCurrentSync( const QString &a_value, bool a_exec )
{
    EditCurrentCommand *ret = new EditCurrentCommand ( a_value );
    if ( a_exec )
        ret->redo();
    return ret;
}

QFuture<EditCurrentCommand*> ThingEditingBehavior::editCurrentAsync( const QString &a_value, bool a_exec )
{
    QFuture<EditCurrentCommand*> resultFuture =
            QtConcurrent::run( QThreadPool::globalInstance(), this,
                               &ThingEditingBehavior::editCurrentSync,
                               a_value, a_exec );
    if (m_synchronous)
        resultFuture.waitForFinished();
    return resultFuture;
}

And now we can make the whole thing undoable by making a undoable editing behavior. I’ll leave a non-undoable editing behavior as an exercise to the reader (ie. just perform redo() on the QUndoCommand, don’t store it in the QUndoStack and immediately delete or cmd->deleteLater() the instance).

Note that if m_synchronous is false, that (all access to) m_undoStack, and the undo and redo methods of your QUndoCommands, must be (made) thread-safe. The thread-safety is not the purpose of this example, though.

class UndoableThingEditingBehavior : public ThingEditingBehavior
{
    Q_OBJECT
public:
    explicit UndoableThingEditingBehavior( ThingEditor *a_editor,
                                           QObject *a_parent = nullptr );
protected:
    EditCellCommand* editCurrentSync( const QString &a_value, bool a_exec = true ) Q_DECL_OVERRIDE;
    EditCurrentCommand* pasteCurrentFromClipboardSync( bool a_exec = true ) Q_DECL_OVERRIDE;
private:
    QScopedPointer<QUndoStack> m_undoStack;
};

EditCellCommand* UndoableThingEditingBehavior::editCurrentSync( const QString &a_value, bool a_exec )
{
    Q_UNUSED(a_exec)
    EditCellCommand *undoable = ThingEditingBehavior::editCurrentSync(  a_value, false );
    m_undoStack->push( undoable );
    return undoable;
}

EditCellCommand* UndoableThingEditingBehavior::pasteCurrentFromClipboardSync( bool a_exec )
{
    Q_UNUSED(a_exec)
    EditCellCommand *undoable = ThingEditingBehavior::pasteCurrentFromClipboardSync( false );
    m_undoStack->push( undoable );
    return undoable;
}

Duck typing

Imagine you have a duck. Imagine you have a wall. Now imagine you throw the duck with a lot of force against a wall. Duck typing means that the duck hitting the wall quacks like a duck would.

ps. Replace wall with API and duck with ugly stupid script written by an idiot. You can leave quacks.

Binaries in git, release numbering, Git-Flow and Scrum at the CIA

Funny how even the software developers at the CIA have problems with idiots who want to put binaries in git. They also know about Git-Flow, my preferred git branching workflow. I kind of wonder how come, if they know about Git-Flow, we see so few leaked NSA and CIA tools with correct semver versioning. Sometimes it’s somewhat okayish, like you can see here. But v1.0-RC3 is not really semver if you see how they got there here. To start with, your alpha versions start with 0.0.x. So where are all those versions under 0.0.x that happened before release candidate 3? 1.0, 1.1-beta, 1.0-phase2, 1.0-beta1-, 1.0-beta-7. WTF guys. That’s not a good versioning scheme. Just call it 0.0.1, 0.0.2, 0.1.0, 0.1.1, 0.1.2 for the betas. And when the thing sees first usage, start calling it 1.0.0, 1.0.1, 1.0.2, 1.1.0, 1.1.1, 1.2.0 etc. What’s wrong with that? And how are all these words like alha-1, beta-2, phase2, etc any better? Maybe just fire your release maintainer! Admittedly for that 2.0.x series they at least tried to get it right.

The point is that packaging tools can be configured to let other packages depend on these version numbers. In x.y.z the x number has implications on API incompatibility whereas the y number can be used for compatible feature additions.

I can imagine that different malwares, exploits, rootkits, intrusion tools they develop would pose incompatibilities with each other, and that for example the NSA and CIA want to share libraries and link without having to recompile or repackage. So versioning to indicate ABI and API compatibility wouldn’t be a bad idea. Let’s hope in the next round of massive leaks we see them having learned good software development processes and practices.

They are doing Scrum sprints and do retrospectives, though. That’s not so bad.

How to expose a QList<MyListClass> in a ViewModel to QML

MyPlugin/MyPlugin.cpp:

#include <ViewModels/MyListClass.h>
#include <ViewModels/DisplayViewModel.h>

qmlRegisterUncreatableType<MyListClass>( a_uri, 1, 0, "MyListClass",
         "Use access via DisplayViewModel instead");
qmlRegisterType<DisplayViewModel>( a_uri, 1, 0, "DisplayViewModel");

Utils/MyQMLListUtils.h

#define MY_DECLARE_QML_LIST(type, name, owner, prop) \
QQmlListProperty<type> name(){ \
   return QQmlListProperty<type>( \
               this, 0,&owner::count ## type ## For ## name ## List, \
               &owner::at ## type ## For ## name ## List); \
} \
static int count ## type ## For ## name ## List(QQmlListProperty<type>*property){ \
   owner *m = qobject_cast<owner *>(property->object); \
   return m->prop.size(); \
} \
static type *at ## type ## For ## name ## List( \
        QQmlListProperty<type>*property, int index){ \
   owner *m = qobject_cast<owner *>(property->object); \
   return m->prop[index]; \
}

ViewModels/DisplayViewModel.h

#ifndef DISPLAYVIEWMODEL_H
#define DISPLAYVIEWMODEL_H

#include <QObject>
#include <QtQml>
#include <ViewModels/MyListClass.h>
#include <Utils/MyQMLListUtils.h>

class DisplayViewModel : public QObject
{
    Q_OBJECT

    Q_PROPERTY(constQString title READ title WRITE setTitle NOTIFY titleChanged )
    Q_PROPERTY(constQList<MyListClass*> objects READ objects
                                          NOTIFY objectsChanged ) 
    Q_PROPERTY( QQmlListProperty<MyListClass> objectList READ objectList
                                              NOTIFY objectsChanged )
public:
    explicit DisplayViewModel( QObject *a_parent = nullptr );
    explicit DisplayViewModel( const QString &a_title,
                               QList<MyListClass*> a_objects,
                               QObject *a_parent = nullptr );
    const QString title()
        { return m_title; }
    void setTitle( const QString &a_title ); 
    const QList<MyListClass*> objects ()
        { return m_objects; } 
    Q_INVOKABLE void appendObject( MyListClass *a_object);
    Q_INVOKABLE void deleteObject( MyListClass *a_object);
    Q_INVOKABLE void reset( );

protected:
    MY_DECLARE_QML_LIST(MyListClass, objectList, DisplayViewModel, m_objects)

signals:
    void titleChanged();
    void objectsChanged();

private:
    QString m_title;
    QList<MyListObject*> m_objects;
};

#endif// DISPLAYVIEWMODEL_H

DisplayViewModel.cpp

#include "DisplayViewModel.h"

DisplayViewModel::DisplayViewModel( const QString &a_title,
                                    QList<MyListClass*> a_objects,
                                    QObject *a_parent )
    : QObject ( a_parent )
    , m_title ( a_title )
    , m_objects ( a_objects )
{
    foreach (MyListClass* mobject, m_objects) {
        mobject->setParent (this);
    }
}

void DisplayViewModel::setTitle (const QString &a_title )
{
    if ( m_title != a_title ) {
        m_title = a_title;
        emit titleChanged();
    }
}

void DisplayViewModel::reset( )
{
    foreach ( MyListClass *mobject, m_objects ) {
        mobject->deleteLater();
    }
    m_objects.clear();
    emit objectsChanged();
}

void DisplayViewModel::appendObject( MyListClass *a_object )
{
    a_object->setParent( this );
    m_objects.append( a_object );
    emit objectsChanged();
}

void DisplayViewModel::deleteObject( MyListClass *a_object )
{
    if (m_objects.contains( a_object )) {
        m_objects.removeOne( a_object );
        a_object->deleteLater();
        emit objectsChanged();
    }
}

Tester.cpp

#include <ViewModels/DisplayViewModel.h>
#include <ViewModels/MyListClass.h>

QList<MyListClass*> objectList;
for( int i = 0; i < 100 ; ++i ) {
    objectList.append ( new MyListClass (i) );
}
DisplayViewModel *viewModel = new DisplayViewModel (objectList);
viewModel->appendObject ( new MyListClass (101) );

Display.qml

import QtQuick 2.5
import MyPlugin 1.0

Repeater { 
    property DisplayViewModel viewModel: DisplayViewModel { } 
    model: viewModel.objectList
    delegate: Item {
        property MyListClass object: modelData
        Text {
            text: object.property
        }
    }
}

Scrum is (best done) like a soccer team

As a freelancer I saw many companies, many situations and worked with many Project Managers, Product Owners, Scrum Masters and god knows what names the HR department came up with.

What is most important, in my experience, is that the people leading the team try to focus the people in the group on as few common goals as possible during one sprint. The more stories or goals the team has to finish, the more individualism and the fewer things will get done (with done being defined by your definition of done).

Differently put you should try to make your team work like how a soccer team plays. You try to make three, four or five goals per sprint. But you do this as a team.

Example: When a story isn’t finished at the end of the sprint; it’s the team’s fault. Not the fault of the one guy that worked on it. The team has to find a solution. If it’s always the same guy being lazy, that’s not the project’s or the team’s results. You or HR deals with that guy later. Doing so is outside of Scrum’s scope. It’s not for your retrospective discussion. But do sanction the team for not delivering.

Another example is not to put too much stories on the task board and yet to keep stories as small and/or well defined as possible. Too much stories or too large stories will result in every individual picking a different story (or subtask of the larger story) to be responsible for. At the end of the sprint none of these will be really finished. And the developers in the team won’t care about the other features being developed during the sprint. So they will dislike having to review other people’s features. They’ll have difficulty finding a reviewer. They won’t communicate with each other. They’ll become careless and dispassionate.

Your planning caused that. That makes you a bad team player. The coach is part of the team.

The Internet of crap – The Ioc. Now In Store!

Like I mentioned a few months ago, here we are again. More things equals more crap on the Internet. Not more utility. No. More crap. It’s only crap. With lower case ‘c’. The crap of the crap programmers of the crap is not worth wasting an expensive capital letter on.

Time to require CE marking for all that crap. Enough is enough.

Finally took the time to get certificates

Finally took the time to use certificates for codeminded.be and pvanhoof.be. Also the SMTP at mail.codeminded.be should now have a somewhat good TLS situation, too. But of course, whoever needed tell me something very secret … just met with me face to face. Duh.

My colleague in crime introduced me to keybase.io, which indeed looks quite nice and fantastic.

Congratulations to the Let’s encrypt initiative for making it really, really easy.

That certbot couldn’t parse my default-ssl in available-sites. No idea why. But it wasn’t in enabled-sites. After removing that original debian-package file, it all worked fine.

They probably also want to post a checksum of that “wget https://dl.eff.org/certbot-auto” thing there. When downloading and executing things on my server I usually do want to quickly and easily check and double-check it all.

The tool is also not super easy to use for anything that isn’t HTTPS. Especially SMTPS comes to mind.

The clouds

In the East there is a shark which is larger than all other fish. It changes into a bird whose wings are like clouds filling the sky. When this bird moves across the land, it brings a message from Corporate Headquarters. This message it drops into the midst of the programmers, like a seagull making its mark upon the beach. Then the bird mounts on the wind and, with the blue sky at its back, returns home.

Truly huge files and the problem of continuous virtual address space

As we all know does mmap, or even worse on Windows CreateFileMapping, need contiguous virtual address space for a given mapping size. That can become a problem when you want to load a file of a gigabyte with mmap.

The solution is of course to mmap the big file using multiple mappings. For example like adapting yesterday’s demo this way:

void FileModel::setFileName(const QString &fileName)
{
    ...
    if (m_file->open(QIODevice::ReadOnly)) {
        if (m_file->size() > MAX_MAP_SIZE) {
            m_mapSize = MAX_MAP_SIZE;
            m_file_maps.resize(1 + m_file->size() / MAX_MAP_SIZE, nullptr);
        } else {
            m_mapSize = static_cast(m_file->size());
            m_file_maps.resize(1, nullptr);
        }
        ...
    } else {
        m_index->open(QFile::ReadOnly);
        m_rowCount = m_index->size() / 4;
    }
    m_file_maps[0] = m_file->map(0, m_mapSize, QFileDevice::NoOptions);
    qDebug() << "Done loading " << m_rowCount << " lines";
    map_index = m_index->map(0, m_index->size(), QFileDevice::NoOptions);

    beginResetModel();
    endResetModel();
    emit fileNameChanged();
}

And in the data() function:

QVariant FileModel::data( const QModelIndex& index, int role ) const
{
    QVariant ret;
    ...
    quint32 mapIndex = pos_i / MAX_MAP_SIZE;
    quint32 map_pos_i = pos_i % MAX_MAP_SIZE;
    quint32 map_end_i = end_i % MAX_MAP_SIZE;
    uchar* map_file = m_file_maps[mapIndex];
    if (map_file == nullptr)
        map_file = m_file_maps[mapIndex] = m_file->map(mapIndex * m_mapSize, m_mapSize, QFileDevice::NoOptions);
    position = m_file_maps[mapIndex] + map_pos_i;
    if (position) {
            const int length = static_cast(end_i - pos_i);
            char *buffer = (char*) alloca(length+1);
            if (map_end_i >= map_pos_i)
                strncpy (buffer, (char*) position, length);
            else {
                const uchar *position2 = m_file_maps[mapIndex+1];
                if (position2 == nullptr) {
                    position2 = m_file_maps[mapIndex+1] = m_file->map((mapIndex+1) *
                         m_mapSize, m_mapSize, QFileDevice::NoOptions);
                }
                strncpy (buffer, (char*) position, MAX_MAP_SIZE - map_pos_i);
                strncpy (buffer + (MAX_MAP_SIZE - map_pos_i), (char*) position2, map_end_i);
            }
            buffer[length] = 0;
            ret = QVariant(QString(buffer));
        }
    }
    return ret;
}

You could also not use mmap for the very big source text file and use m_file.seek(map_pos_i) and m_file.read(buffer, length). The most important mapping is of course the index one, as the reading of the individual lines can also be done fast enough with normal read() calls (as long as you don’t have to do it for each and every line of the very big file and as long as you know in a O(1) way where the QAbstractListModel’s index.row()’s data is).

But you already knew that. Right?

Loading truly truly huge text files with a QAbstractListModel

Sometimes people want to do crazy stuff like loading a gigabyte sized plain text file into a Qt view that can handle QAbstractListModel. Like for example a QML ListView. You know, the kind of files you generate with this commando:

base64 /dev/urandom | head -c 100000000 > /tmp/file.txt

But, how do they do it?

FileModel.h

So we will make a custom QAbstractListModel. Its private member fields I will explain later:

#ifndef FILEMODEL_H
#define FILEMODEL_H

#include <QObject>
#include <QVariant>
#include <QAbstractListModel>
#include <QFile>

class FileModel: public QAbstractListModel {
    Q_OBJECT

    Q_PROPERTY(QString fileName READ fileName WRITE setFileName NOTIFY fileNameChanged )
public:
    explicit FileModel( QObject* a_parent = nullptr );
    virtual ~FileModel();

    int columnCount(const QModelIndex &parent) const;
    int rowCount( const QModelIndex& parent =  QModelIndex() ) const Q_DECL_OVERRIDE;
    QVariant data( const QModelIndex& index, int role = Qt::DisplayRole ) const  Q_DECL_OVERRIDE;
    QVariant headerData( int section, Qt::Orientation orientation,
                         int role = Qt::DisplayRole ) const  Q_DECL_OVERRIDE;
    void setFileName(const QString &fileName);
    QString fileName () const
        { return m_file->fileName(); }
signals:
    void fileNameChanged();
private:
    QFile *m_file, *m_index;
    uchar *map_file;
    uchar *map_index;
    int m_rowCount;
    void clear();
};

#endif// FILEMODEL_H

FileModel.cpp

We will basically scan the very big source text file for newline characters. We’ll write the offsets of those to a file suffixed with “.mmap”. We’ll use that new file as a sort of “partition table” for the very big source text file, in the data() function of QAbstractListModel. But instead of sectors and files, it points to newlines.

The reason why the scanner itself isn’t using the mmap’s address space is because apparently reading blocks of 4kb is faster than reading each and every byte from the mmap in search of \n characters. Or at least on my hardware it was.

You should probably do the scanning in small qEventLoop iterations (make sure to use nonblocking reads, then) or in a thread, as your very big source text file can be on a unreliable or slow I/O device. Plus it’s very big, else you wouldn’t be doing this (please promise me to just read the entire text file in memory unless it’s hundreds of megabytes in size: don’t micro optimize your silly homework notepad.exe clone).

Note that this is demo code with a lot of bugs like not checking for \r and god knows what memory leaks and stuff was remaining when it suddenly worked. I leave it to the reader to improve this. An example is that you should check for validity of the “.mmap” file: your very big source text file might have changed since the newline partition table was made.

Knowing that I’ll soon find this all over the place without any of its bugs fixed, here it comes ..

#include "FileModel.h"

#include <QDebug>

#include <stdio.h>
#include <stdlib.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <pthread.h>
#include <unistd.h>

FileModel::FileModel( QObject* a_parent )
    : QAbstractListModel( a_parent )
    , m_file (nullptr)
    , m_index(nullptr)
    , m_rowCount ( 0 ) { }

FileModel::~FileModel() { clear(); }

void FileModel::clear()
{
    if (m_file) {
        if (m_file->isOpen() && map_file != nullptr)
            m_file->unmap(map_file);
        delete m_file;
    }
    if (m_index) {
        if (m_index->isOpen() && map_index != nullptr)
            m_index->unmap(map_index);
        delete m_index;
    }
}

void FileModel::setFileName(const QString &fileName)
{
   clear();
   m_rowCount = 0;
   m_file = new QFile(fileName);
   int cur = 0;
   m_index = new QFile(m_file->fileName() + ".mmap");
   if (m_file->open(QIODevice::ReadOnly)) {
       if (!m_index->exists()) {
           char rbuffer[4096];
           m_index->open(QIODevice::WriteOnly);
           char nulbuffer[4];
           int idxnul = 0;
           memset( nulbuffer +0, idxnul >> 24 & 0xff, 1 );
           memset( nulbuffer +1, idxnul >> 16 & 0xff, 1 );
           memset( nulbuffer +2, idxnul >>  8 & 0xff, 1 );
           memset( nulbuffer +3, idxnul >>  0 & 0xff, 1 );
           m_index->write( nulbuffer, sizeof(quint32));
           qDebug() << "Indexing to" << m_index->fileName();
           while (!m_file->atEnd()) {
               int in = m_file->read(rbuffer, 4096);
               if (in == -1)
                   break;
               char *newline = (char*) 1;
               char *last = rbuffer;
               while (newline != 0) {
                   newline = strchr ( last, '\n');
                   if (newline != 0) {
                     char buffer[4];
                     int idx = cur + (newline - rbuffer);
                     memset( buffer +0, idx >> 24 & 0xff, 1 );
                     memset( buffer +1, idx >> 16 & 0xff, 1 );
                     memset( buffer +2, idx >>  8 & 0xff, 1 );
                     memset( buffer +3, idx >>  0 & 0xff, 1 );
                     m_index->write( buffer, sizeof(quint32));
                     m_rowCount++;
                     last = newline + 1;
                  }
               }
               cur += in;
           }
           m_index->close();
           m_index->open(QFile::ReadOnly);
           qDebug() << "done";
       } else {
           m_index->open(QFile::ReadOnly);
           m_rowCount = m_index->size() / 4;
       }
       map_file= m_file->map(0, m_file->size(), QFileDevice::NoOptions);
       qDebug() << "Done loading " << m_rowCount << " lines";
       map_index = m_index->map(0, m_index->size(), QFileDevice::NoOptions);
   }
   beginResetModel();
   endResetModel();
   emit fileNameChanged();
}

static quint32
read_uint32 (const quint8 *data)
{
    return data[0] << 24 |
           data[1] << 16 |
           data[2] << 8 |
           data[3];
}

int FileModel::rowCount( const QModelIndex& parent ) const
{
    Q_UNUSED( parent );
    return m_rowCount;
}

int FileModel::columnCount(const QModelIndex &parent) const
{
    Q_UNUSED( parent );
    return 1;
}

QVariant FileModel::data( const QModelIndex& index, int role ) const
{
    if( !index.isValid() )
        return QVariant();
    if (role == Qt::DisplayRole) {
        QVariant ret;
        quint32 pos_i = read_uint32(map_index + ( 4 * index.row() ) );
        quint32 end_i;
        if ( index.row() == m_rowCount-1 )
            end_i = m_file->size();
        else
            end_i = read_uint32(map_index + ( 4 * (index.row()+1) ) );
        uchar *position;
        position = map_file +  pos_i;
        uchar *end = map_file + end_i;
        int length = end - position;
        char *buffer = (char*) alloca(length +1);
        memset (buffer, 0, length+1);
        strncpy (buffer, (char*) position, length);
        ret = QVariant(QString(buffer));
        return ret;
    }
    return QVariant();
}

QVariant FileModel::headerData( int section, Qt::Orientation orientation, int role ) const
{
    Q_UNUSED(section);
    Q_UNUSED(orientation);
    if (role != Qt::DisplayRole)
           return QVariant();
    return QString("header");
}

main.cpp

#include <QGuiApplication>
#include <QQmlApplicationEngine>
#include <QtQml>// qmlRegisterType

#include "FileModel.h"

int main(int argc, char *argv[])
{
    QGuiApplication app(argc, argv);
    qmlRegisterType<FileModel>( "FileModel", 1, 0, "FileModel" );
    QQmlApplicationEngine engine;
    engine.load(QUrl(QStringLiteral("qrc:/main.qml")));
    return app.exec();
}

main.qml

import QtQuick 2.3
import QtQuick.Window 2.2
import FileModel 1.0

Window {
    visible: true

    FileModel { id: fileModel }
    ListView {
        id: list
        anchors.fill: parent
        delegate: Text { text: display }
        MouseArea {
            anchors.fill: parent
            onClicked: {
                list.model = fileModel
                fileModel.fileName = "/tmp/file.txt"
            }
        }
    }
}

profile.pro

TEMPLATE = app
QT += qml quick
CONFIG += c++11
SOURCES += main.cpp \
    FileModel.cpp
RESOURCES += qml.qrc
HEADERS += \
    FileModel.h

qml.qrc

<RCC>
    <qresource prefix="/">
        <file>main.qml</file>
    </qresource>
</RCC>

Composition and aggregation with QObject

Consider these rather simple relationships between classes

Continuing on this subject, here are some code examples.

Class1 & Class2: Composition
An instance of Class1 can not exist without an instance of Class2.

Example of composition is typically a Bicycle and its Wheels, Saddle and a HandleBar: without these the Bicycle is no longer a Bicycle but just a Frame.

It can no longer function as a Bicycle. Example of when you need to stop thinking about composition versus aggregation is whenever you say: without the other thing can’t in our software the first thing work.

Note that you must consider this in the context of Class1. You use aggregation or composition based on how Class2 exists in relation to Class1.

Class1 with QScopedPointer:

#ifndef CLASS1_H
#define CLASS1_H

#include <QObject>
#include <QScopedPointer>
#include <Class2.h>

class Class1: public QObject
{
    Q_PROPERTY( Class2* class2 READ class2 WRITE setClass2 NOTIFY class2Changed)
public:
    Class1( QObject *a_parent = nullptr )
        : QObject ( a_parent) {
        // Don't use QObject parenting on top here
        m_class2.reset (new Class2() );
    }
    Class2* class2() {
        return m_class2.data();
    }
    void setClass2 ( Class2 *a_class2 ) {
        Q_ASSERT (a_class2 != nullptr); // Composition can't set a nullptr!
        if ( m_class2.data() != a_class2 ) {
            m_class2.reset( a_class2 );
            emit class2Changed()
        }
    }
signals:
    void class2Changed();
private:
    QScopedPointer<Class2> m_class2;
};

#endif// CLASS1_H

Class1 with QObject parenting:

#ifndef CLASS1_H
#define CLASS1_H

#include <QObject>
#include <Class2.h>

class Class1: public QObject
{
    Q_PROPERTY( Class2* class2 READ class2 WRITE setClass2 NOTIFY class2Changed)
public:
    Class1( QObject *a_parent = nullptr )
        : QObject ( a_parent )
        , m_class2 ( nullptr ) {
        // Make sure to use QObject parenting here
        m_class2 = new Class2( this );
    }
    Class2* class2() {
        return m_class2;
    }
    void setClass2 ( Class2 *a_class2 ) {
         Q_ASSERT (a_class2 != nullptr); // Composition can't set a nullptr!
         if ( m_class2 != a_class2 ) {
             // Make sure to use QObject parenting here
             a_class2->setParent ( this );
             delete m_class2; // Composition can never be nullptr
             m_class2 = a_class2;
             emit class2Changed();
         }
    }
signals:
    void class2Changed();
private:
    Class2 *m_class2;
};

#endif// CLASS1_H

Class1 with RAII:

#ifndef CLASS1_H
#define CLASS1_H

#include <QObject>
#include <QScopedPointer>

#include <Class2.h>

class Class1: public QObject
{
    Q_PROPERTY( Class2* class2 READ class2 CONSTANT)
public:
    Class1( QObject *a_parent = nullptr )
        : QObject ( a_parent ) { }
    Class2* class2()
        { return &m_class2; }
private:
    Class2 m_class2;
};
#endif// CLASS1_H

Class3 & Class4: Aggregation

An instance of Class3 can exist without an instance of Class4. Example of composition is typically a Bicycle and its driver or passenger: without the Driver or Passenger it is still a Bicycle. It can function as a Bicycle.

Example of when you need to stop thinking about composition versus aggregation is whenever you say: without the other thing can in our software the first thing work.

Class3:

#ifndef CLASS3_H
#define CLASS3_H

#include <QObject>

#include <QPointer>
#include <Class4.h>

class Class3: public QObject
{
    Q_PROPERTY( Class4* class4 READ class4 WRITE setClass4 NOTIFY class4Changed)
public:
    Class3( QObject *a_parent = nullptr );
    Class4* class4() {
        return m_class4.data();
    }
    void setClass4 (Class4 *a_class4) {
         if ( m_class4 != a_class4 ) {
             m_class4 = a_class4;
             emit class4Changed();
         }
    }
signals:
    void class4Changed();
private:
    QPointer<Class4> m_class4;
};
#endif// CLASS3_H

Class5, Class6 & Class7: Shared composition
An instance of Class5 and-or an instance of Class6 can not exist without a instance of Class7 shared by Class5 and Class6. When one of Class5 or Class6 can and one can not exist without the shared instance, use QWeakPointer at that place.

Class5:

#ifndef CLASS5_H
#define CLASS5_H

#include <QObject>
#include <QSharedPointer>

#include <Class7.h>

class Class5: public QObject
{
    Q_PROPERTY( Class7* class7 READ class7 CONSTANT)
public:
    Class5( QObject *a_parent = nullptr, Class7 *a_class7 );
        : QObject ( a_parent )
        , m_class7 ( a_class7 ) { }
    Class7* class7()
        { return m_class7.data(); }
private:
    QSharedPointer<Class7> m_class7;
};

Class6:

#ifndef CLASS6_H
#define CLASS6_H

#include <QObject>
#include <QSharedPointer>

#include <Class7.h>

class Class6: public QObject
{
    Q_PROPERTY( Class7* class7 READ class7 CONSTANT)
public:
    Class6( QObject *a_parent = nullptr, Class7 *a_class7 )
        : QObject ( a_parent )
        , m_class7 ( a_class7 ) { }
    Class7* class7()
        { return m_class7.data(); }
private:
    QSharedPointer<Class7> m_class7;
};
#endif// CLASS6_H

Interfaces with QObject

FlyBehavior:

#ifndef FLYBEHAVIOR_H
#define FLYBEHAVIOR_H
#include <QObject>
// Don't inherit QObject here (you'll break multiple-implements)
class FlyBehavior {
    public:
        Q_INVOKABLE virtual void fly() = 0;
};
Q_DECLARE_INTERFACE(FlyBehavior , "be.codeminded.Flying.FlyBehavior /1.0") 
#endif// FLYBEHAVIOR_H

FlyWithWings:

#ifndef FLY_WITH_WINGS_H
#define FLY_WITH_WINGS_H
#include <QObject>  
#include <Flying/FlyBehavior.h>
// Do inherit QObject here (this is a concrete class)
class FlyWithWings: public QObject, public FlyBehavior
{
    Q_OBJECT
    Q_INTERFACES( FlyBehavior )
public:
    explicit FlyWithWings( QObject *a_parent = nullptr ): QObject ( *a_parent ) {}
    ~FlyWithWings() {}

    virtual void fly() Q_DECL_OVERRIDE;
}
#endif// FLY_WITH_WINGS_H

How about …

Next time, I show you how to turn that ViewModel into a Visitor? And then make a view that shows the syntax and outline of a parsed file’s language?

How about I turn my blog into the programmers’ equivalent to Dobbit magazine? :-)

Who knows what I’ll come up with next time. I guess I don’t know myself.

MVVM, Model View ViewModel, with Qt and QML

In the XAML world it’s very common to use the MVVM pattern. I will explain how to use the technique in a similar way with Qt and QML.

The idea is to not have too much code in the view component. Instead we have declarative bindings and move most if not all of our view code to a so called ViewModel. The ViewModel will sit in between the actual model and the view. The ViewModel typically has one to one properties for everything that the view displays. Manipulating the properties of the ViewModel alters the view through bindings. You typically don’t alter the view directly.

In our example we have two list-models, two texts and one button: available-items, accepted-items, available-count, accepted-count and a button. Pressing the button moves stuff from available to accepted. Should be a simple example.

First the ViewModel.h file. The class will have a property for ~ everything the view displays:

#ifndef VIEWMODEL_H
#define VIEWMODEL_H

#include <QAbstractListModel>
#include <QObject>

class ViewModel : public QObject
{
	Q_OBJECT

	Q_PROPERTY(QAbstractListModel* availableItems READ availableItems NOTIFY availableItemsChanged )
	Q_PROPERTY(QAbstractListModel* acceptedItems READ acceptedItems NOTIFY acceptedItemsChanged )
	Q_PROPERTY(int available READ available NOTIFY availableChanged )
	Q_PROPERTY(int accepted READ accepted NOTIFY acceptedChanged )
public:

	ViewModel( QObject *parent = 0 );
	~ViewModel() { }

	QAbstractListModel* availableItems()
		{ return m_availableItems; }

	QAbstractListModel* acceptedItems()
		{ return m_acceptedItems; }

	int available ()
		{ return m_availableItems->rowCount(); }

	int accepted ()
		{ return m_acceptedItems->rowCount(); }

	Q_INVOKABLE void onButtonClicked( int availableRow );

signals:
	void availableItemsChanged();
	void acceptedItemsChanged();
	void availableChanged();
	void acceptedChanged();

private:
	QAbstractListModel* m_availableItems;
	QAbstractListModel* m_acceptedItems;
};

#endif

The ViewModel.cpp implementation of the ViewModel. This is of course a simple example. The idea is that ViewModels can be quite complicated while the view.qml remains simple:

#include <QStringListModel>

#include "ViewModel.h"

ViewModel::ViewModel( QObject *parent ) : QObject ( parent )
{
	QStringList available;
	QStringList accepted;

	available << "Two" << "Three" << "Four" << "Five";
	accepted << "One";

	m_availableItems = new QStringListModel( available, this );
	emit availableItemsChanged();

	m_acceptedItems = new QStringListModel( accepted, this );
	emit acceptedItemsChanged();
}

void ViewModel::onButtonClicked(int availableRow)
{
	QModelIndex availableIndex = m_availableItems->index( availableRow, 0, QModelIndex() );
	QVariant availableItem = m_availableItems->data( availableIndex, Qt::DisplayRole );

	int acceptedRow = m_acceptedItems->rowCount();

	m_acceptedItems->insertRows( acceptedRow, 1 );

	QModelIndex acceptedIndex = m_acceptedItems->index( acceptedRow, 0, QModelIndex() );
	m_acceptedItems->setData( acceptedIndex, availableItem );
	emit acceptedChanged();

	m_availableItems->removeRows ( availableRow, 1, QModelIndex() );
	emit availableChanged();
}

The view.qml. We’ll try to have as few JavaScript code as possible; the idea is that coding itself is done in the ViewModel. The view should only be view code (styling, UI, animations, etc). The import url and version are defined by the use of qmlRegisterType in the main.cpp file, lower:

import QtQuick 2.0
import QtQuick.Controls 1.2

import be.codeminded.ViewModelExample 1.0

Rectangle {
    id: root
    width: 640; height: 320

	property var viewModel: ViewModel { }

	Rectangle {
		id: left
		anchors.left: parent.left
		anchors.top: parent.top
		anchors.bottom: button.top
		width: parent.width / 2
		ListView {
		    id: leftView
			anchors.left: parent.left
			anchors.right: parent.right
			anchors.top: parent.top
			anchors.bottom: leftText.top

			delegate: rowDelegate
		        model: viewModel.availableItems
		}
		Text {
			id: leftText
			anchors.left: parent.left
			anchors.right: parent.right
			anchors.bottom: parent.bottom
			height: 20
			text: viewModel.available
		}
	}

	Rectangle {
		id: right
		anchors.left: left.right
		anchors.right: parent.right
		anchors.top: parent.top
		anchors.bottom: button.top
		ListView {
		    id: rightView
			anchors.left: parent.left
			anchors.right: parent.right
			anchors.top: parent.top
			anchors.bottom: rightText.top

			delegate: rowDelegate
		        model: viewModel.acceptedItems
		}
		Text {
			id: rightText
			anchors.left: parent.left
			anchors.right: parent.right
			anchors.bottom: parent.bottom
			height: 20
			text: viewModel.accepted
		}
	}

	Component {
		id: rowDelegate
		Rectangle {
			width: parent.width
			height: 20
			color: ListView.view.currentIndex == index ? "red" : "white"
			Text { text: 'Name:' + display }
			MouseArea {
				anchors.fill: parent
				onClicked: parent.ListView.view.currentIndex = index
			}
		}
	}

	Button {
		id: button
		anchors.left: parent.left
		anchors.right: parent.right
		anchors.bottom: parent.bottom
		height: 20
	        text: "Accept item"
		onClicked: viewModel.onButtonClicked( leftView.currentIndex );
	}
}

A main.cpp example. The qmlRegisterType defines the url to import in the view.qml file:

#include <QGuiApplication>
#include <QQuickView>
#include <QtQml>
#include <QAbstractListModel>

#include "ViewModel.h"

int main(int argc, char *argv[])
{
	QGuiApplication app(argc, argv);
	QQuickView view;
	qRegisterMetaType<QAbstractListModel*>("QAbstractListModel*");
	qmlRegisterType<ViewModel>("be.codeminded.ViewModelExample", 1, 0, "ViewModel");
	view.setSource(QUrl("qrc:/view.qml"));
	view.show();
	return app.exec();
}

A project.pro file. Obviously should you use cmake nowadays. But oh well:

TEMPLATE += app
QT += quick
SOURCES += ViewModel.cpp main.cpp
HEADERS += ViewModel.h
RESOURCES += project.qrc

And a project.qrc file:

<!DOCTYPE RCC>
<RCC version="1.0">
<qresource prefix="/">
    <file>view.qml</file>
</qresource>
</RCC>

QML coding conventions checker that uses QML parser’s own abstract syntax tree

My colleague Henk Van Der Laak made a interesting tool that checks your code against the QML coding conventions. It uses the internal parser’s abstract syntax tree of Qt 5.6 and a visitor design.

It has a command line, but being developers ourselves we want an API too of course. Then we can integrate it in our development environments without having to use popen!

So this is how to use that API:

// Parse the code
QQmlJS::Engine engine;
QQmlJS::Lexer lexer(&engine);
QQmlJS::Parser parser(&engine);

QFileInfo info(a_filename);
bool isJavaScript = info.suffix().toLower() == QLatin1String("js");
lexer.setCode(code,  1, !isJavaScript);
bool success = isJavaScript ? parser.parseProgram() : parser.parse();
if (success) {
    // Check the code
    QQmlJS::AST::UiProgram *program = parser.ast();
    CheckingVisitor checkingVisitor(a_filename);
    program->accept(&checkingVisitor);
    foreach (const QString &warning, checkingVisitor.getWarnings()) {
        qWarning() << qPrintable(warning);
    }
}

Item isChild of another Item in QML

Damned, QML is inconsistent! Things have a content, data or children. And apparently they can all mean the same thing. So how do we know if something is a child of something else?

After a failed stackoverflow search I gave up on copy-paste coding and invented the damn thing myself.

function isChild( a_child, a_parent ) {
	if ( a_parent === null ) {
		return false
	}

	var tmp = ( a_parent.hasOwnProperty("content") ? a_parent.content
		: ( a_parent.hasOwnProperty("children") ? a_parent.children : a_parent.data ) )

	if ( tmp === null || tmp === undefined ) {
		return false
	}

	for (var i = 0; i < tmp.length; ++i) {

		if ( tmp[i] === a_child ) {
			return true
		} else {
			if ( isChild ( a_child, tmp[i] ) ) {
				return true
			}
		}
	}
	return false
}