Month: July 2010

Busy handling

I made an article about reporting busy status in Tracker before.

But then it wasn’t yet possible to queue a query while Tracker’s RDF store is busy. We’re making this possible following next unstable release. Yeah I know you guys hate that Tracker’s RDF store can be busy. But you tell us what else to do while restoring a backup, or while replaying a journal?

While we are replaying the journal, or restoring a backup, we’ll accept your result-hungry queries into our queue. Meanwhile you get progress and status indication over a DBus signal. Some documentation about this is available here.

SPARQL 1.1 Draft features: IN and NOT IN

We had a feature requests for supporting SPARQL IN and NOT IN. As usual, we’re ahead of the SPARQL Draft specification. But I don’t think IN and NOT IN will look much different in the end. Anyway, it was straightforward so I just implemented both.

It goes like this:

SELECT ?abc { ?abc a nie:InformationElement ;
                   nie:title ?title .
               FILTER (?title IN ('abc', 'def')) }

SELECT ?abc { ?abc a nie:InformationElement ;
                   nie:title ?title .
               FILTER (?title NOT IN ('xyz', 'def')) }

It’s particularly useful to get metadata about a defined set of resources (give me the author of this, this and that file)

Direct access

This work is progressing nicely. Most of the guys on the team are working on this, and it’s going to be awesome thanks to SQLite’s WAL journal mode. SQLite’s WAL mode is still under development and probably unstable here and there, but we’re trusting the SQLite guys with this anyway.

What is left to do for direct-access is cleaning up a bit, getting the small nasty things right. You know. The basics are all in place now.

We’re doing most of the library code in Vala, but clever people can easily imagine the C API valac makes from the .vala files here. That’s the abstract API that client developers will use. Unless you use a higher level API like libqttracker, QSparql, Hormiga or sparql-glib.

All of which still need to be adapted to the direct-access work that we’re doing. But we’re in close contact with all of the developers involved in those libraries. And they’re all thrilled to implement backends for the new stuff.

Plans

We plan to change the signals-on-changes or class-signals feature a bit so that the three signals are merged into one. The problem with three is that you can’t reliably identify a change-transaction this way (a rename of a file, for example).

Another thing on our list is merging Zeitgeist’s ontology. To the other team members at Tracker: guys, Zeitgeist has been waiting for three months now. Let’s just get this done!

Oh there are a lot of plans, to be honest.

I wonder when, if ever, we go in feature freeze. Hehe. I guess we’ll just have very short feature-freeze periods. Whatever, it’s fun.

MeeGo in cars

Hey BMW & co, if you guys want to learn how to write music players and playlists for car entertainment on MeeGo, get in touch! This Tracker that I’m talking about is on that MeeGo OS; being the Music’s metadata database is among its purposes.

I can’t wait to have a better music player playlist my car.

Or maybe some integration with the in-car GPS and the car owner’s appointments and meetings? With geo-tagged photos on the car owner’s phone? Automatic and instant synchronization with Nokia’s future phones? Sounds all very doable, even easy, to me. I’d want all that stuff. Use-cases!

Let’s talk!

The support for domain specific indexes is, awaiting review / finished. Although we can further optimize it now. More on that later in this post. Image that you have this ontology:

nie:InformationElement a rdfs:Class .

nie:title a rdf:Property ;
  nrl:maxCardinality 1 ;
  rdfs:domain nie:InformationElement ;
  rdfs:range xsd:string .

nmm:MusicPiece a rdfs:Class ;
  rdfs:subClassOf nie:InformationElement .

nmm:beatsPerMinute a rdf:Property ;
  nrl:maxCardinality 1 ;
  rdfs:domain nmm:MusicPiece ;
  rdfs:range xsd:integer .

With that ontology there are three tables called “Resource”, “nmo:MusicPiece” and “nie:InformationElement” in SQLite’s schema:

• The “Resource” table has ID and the subject string
• The “nie:InformationElement” has ID and “nie:title”
• The “nmm:MusicPiece” one has ID and “nmm:beatsPerMinute”

That’s fairly simple, right? The problem is that when you ORDER BY “nie:title” that you’ll cause a full table scan on “nie:InformationElement”. That’s not good, because there are less “nmm:MusicPiece” records than “nie:InformationElement” ones.

Imagine that we do this SPARQL query:

SELECT ?title WHERE {
   ?resource a nmm:MusicPiece ;
             nie:title ?title
} ORDER BY ?title

We translate that, for you, to this SQL on our schema:

SELECT   "title_u" FROM (
  SELECT "nmm:MusicPiece1"."ID" AS "resource_u",
         "nie:InformationElement2"."nie:title" AS "title_u"
  FROM   "nmm:MusicPiece" AS "nmm:MusicPiece1",
         "nie:InformationElement" AS "nie:InformationElement2"
  WHERE  "nmm:MusicPiece1"."ID" = "nie:InformationElement2"."ID"
  AND    "title_u" IS NOT NULL
) ORDER BY "title_u"

OK, so with support for domain indexes we change the ontology like this:

nmm:MusicPiece a rdfs:Class ;
  rdfs:subClassOf nie:InformationElement ;
  tracker:domainIndex nie:title .

Now we’ll have the three tables called “Resource”, “nmo:MusicPiece” and “nie:InformationElement” in SQLite’s schema. But they will look like this:

• The “Resource” table has ID and the subject string
• The “nie:InformationElement” has ID and “nie:title”
• The “nmm:MusicPiece” table now has three columns called ID, “nmm:beatsPerMinute” and “nie:title”

The same data, for titles of music pieces, will be in both “nie:InformationElement” and “nmm:MusicPiece”. We copy to the mirror column during ontology change coping, and when new inserts happen.

When now the rdf:type is known in the SPARQL query as a nmm:MusicPiece, like in the query mentioned earlier, we know that we can use the “nie:title” from the “nmm:MusicPiece” table in SQLite. That allows us to generate you this SQL query:

SELECT   "title_u" FROM (
  SELECT "nmm:MusicPiece1"."ID" AS "resource_u",
         "nmm:MusicPiece1"."nie:title" AS "title_u"
  FROM   "nmm:MusicPiece" AS "nmm:MusicPiece1"
  WHERE  "title_u" IS NOT NULL
) ORDER BY "title_u"

A remaining optimization is when you request a rdf:type that is a subclass of nmm:MusicPiece, like this:

SELECT ?title WHERE {
  ?resource a nmm:MusicPiece, nie:InformationElement ;
            nie:title ?title
} ORDER BY ?title

It’s still not as bad as now the “nie:title” is still taken from the “nmm:MusicPiece” table. But the join with “nie:InformationElement” is still needlessly there (we could just do the earlier SQL query in this case):

SELECT   "title_u" FROM (
  SELECT "nmm:MusicPiece1"."ID" AS "resource_u",
         "nmm:MusicPiece1"."nie:title" AS "title_u"
  FROM   "nmm:MusicPiece" AS "nmm:MusicPiece1",
         "nie:InformationElement" AS "nie:InformationElement2"
  WHERE  "nmm:MusicPiece1"."ID" = "nie:InformationElement2"."ID"
  AND    "title_u" IS NOT NULL
) ORDER BY "title_u"

We will probably optimize this specific use-case further later this week.