At six in the morning the girl who was working her first day at the bar, after she and the bartender and owner got a Leffe from me, played Ne me quitte pas by Jacques Brel. I didn’t request it, they even asked me to shut the fuck up for the song.
I realized that although I’m coming from that insignificant country, insignificance doesn’t matter.
And this is why I like Europe’s many cultures. Why I like Switzerland, too. We don’t but we do understand each other.
Every (good) developer knows that copying of memory and boxing, especially when dealing with a large amount of pieces like members of collections and the cells in a table, are a bad thing for your performance.
More experienced developers also know that novice developers tend to focus on just their algorithms to improve performance, while often the single biggest bottleneck is needless boxing and allocating. Experienced developers come up with algorithms that avoid boxing and copying; they master clever pragmatical engineering and know how to improve algorithms. A lot of newcomers use virtual machines and script languages that are terrible at giving you the tools to control this and then they start endless religious debates about how great their programming language is (as if it matters). (Anti-.NET people don’t get on your horses too soon: if you know what you are doing, C# is actually quite good here).
Once Jürg and Carlos have finished the work on parallelizing SELECT queries we plan to let the code that walks the SQLite statement fill in the DBusMessage directly without any memory copying or boxing (for marshalling to DBus). We found the get_reply and send_reply functions; they sound useful for this purpose.
I still don’t really like DBus as IPC for data transfer of Tracker’s RDF store’s query results. Personally I think I would go for a custom Unix socket here. But Jürg so far isn’t convinced. Admittedly he’s probably right; he’s always right. Still, DBus to me doesn’t feel like a good IPC for this data transfer..
We know about the requests to have direct access to the SQLite database from your own process. I explained in the bug that SQLite3 isn’t MVCC and that this means that your process will often get blocked for a long time on our transaction. A longer time than any IPC overhead takes.
It used to be in Tracker that you couldn’t just change the ontology. When you did, you had to reboot the database. This means loosing all the non-embedded data. For example your tags or other such information that’s uniquely stored in Tracker’s RDF store.
This was of course utterly unacceptable and this was among the reasons why we kept 0.8 from being released for so long: we were afraid that we would need to make ontology changes during the 0.8 series.
So during 0.7 I added support for what I call modest ontology changes. This means adding a class, adding a property. But just that. Not changing an existing property. This was sufficient for 0.8 because now we could at least do some changes like adding a property to a class, or adding a new class. You know, making implementing the standard feature requests possible.
Last two weeks I worked on supporting more intrusive ontology changes. The branch that I’m working on currently supports changing tracker:notify for the signals on changes feature, tracker:writeback for the writeback features and tracker:indexed which controls the indexes in the SQLite tables.
But also certain range changes are supported. For example integer to string, double and boolean. String to integer, double and boolean. Double to integer, string and boolean. Range changes will sometimes of course mean data loss.
Plenty of code was also added to detect an unsupported ontology change and to ensure that we just abort the process and don’t do any changes in that case.
It’s all quite complex so it might take a while before the other team members have tested and reviewed all this. It should probably take even longer before it hits the stable 0.8 branch.
We wont yet open the doors to custom ontologies. Several reasons:
We want more testing on the support for ontology changes. We know that once we open the doors to custom ontologies that we’ll see usage of this rather sooner than later.
We don’t yet support removing properties and classes. This would be easy (drop the table and columns away and log the event in the journal) but it’s not yet supported. Mostly because we don’t need it ourselves (which is a good reason).
We don’t want you to meddle with the standard ontologies (we’ll do that, don’t worry). So we need a bit of ontology management code to also look in other directories, etc.
The error handling of unsupported ontology changes shouldn’t abort like mentioned above. Another piece of software shouldn’t make Tracker unusable just because they install junk ontologies.
We actually want to start using OSCAF’s ontology format. Perhaps it’s better that we wait for this instead of later asking everybody to convert their custom ontologies?
We’re a bunch of pussies who are afraid of the can of worms that you guys’ custom ontologies will open.
But yes, you could say that the basics are being put in place as we speak.
MSNBC: You have more tapes like this? Julian Assange: Yes we do. Assange: I won’t go into the precise number. But there was a rumor that the tape that we were about to release was about a similar incident in Afghanistan, where 97 people were bombed in May last year. We euhm, have that video. MSNBC: Do you intent to release that video as well? Assange: Yes, as soon as we have finished our analysis, we will release it.
Thank you Wikileaks. Thank you Julian Assange. You are bringing Wikileak’s perspective calm and clear in the media. You’re an example to all whistleblowers. Julian, you’re doing a great job.
I understand more people are involved in this leak; thanks everybody. You’re being respected.
Information technology is all about information. Information for humanity.
Don’t you guys stop believing in this! We now believe in you. Many people like me are highly focused and when intelligence services want a battle: we’ll listen. People like me are prepared to act.
I understand you guys like Belgium’s law that protects journalist’ sources. As the owner of a Belgian Ltd. maybe I can help?
I’m not often proud about my country. Last week I told my Swiss friends here in Zürich that I have about 3000 reasons to leave Belgium and a 1000 reasons to come to Switzerland. I wasn’t exaggerating.
I’m a guy with principles and ethics. So thank you.
Today after I brought Tinne to the airport I drove around Zürichsee. She can’t stay in Switzerland the entire month; she has to go back to school on Monday.
While driving on the Seestrasse I started counting luxury cars. After I reached two for Lamborgini and three for Ferrari I started thinking: Zimmerberg Sihltal and Pfannenstiel must be expensive districts too… And yes, they are.
I was lucky today that it was nice weather. But wow, what a nice view on the mountain tops when you look south over Zürichsee. People from Zürich, you guys are so lucky! Such immense calming feeling the view gives me! For me, it beats sauna. And I’m a real sauna fan.
I’m thinking to check it out south of Zürich. But not the canton. I think the house prices are just exaggerated high in the canton of Zürich. I was thinking Sankt Gallen, Toggenburg. I’ve never been there; I’ll check it out tomorrow.
Hmmr, meteoswiss gives rain for tomorrow. Doesn’t matter.
Actually, when I came back from the airport the first thing I really did was fix coping with property changes in ontologies for Tracker. Yesterday it wasn’t my day, I think. I couldn’t find this damn problem in my code! And in the evening I lost three chess games in a row against Tinne. That’s really a bad score for me. Maybe after two weeks of playing chess almost every evening, she got better than me? Hmmrr, that’s a troubling idea.
Anyway, so when I got back from the airport I couldn’t resist beating the code problem that I didn’t find on Friday. I found it! It works!
I guess I’m both a dreamer and a realist programmer. But don’t tell my customers that I’m such a dreamer.
Today Tinne and I visited Switzerland’s capital, Bern.
We were really surprised; we’d never imagined that a capital city could offer so much peace and calm. It felt good to be there.
The fountains, the old houses, the river and the snowy mountain peaks give the city an idyllic image.
Standing on the bridge, you see the roofs of all these lovely small houses.
The bear is the symbol of Bern. Near the House of Parliament there was this statue of a bear. Tinne just couldn’t resist to give it a hug. Bern has also got real bears. Unfortunately, Tinne was not allowed to cuddle those bears.
The House of Parliament is a truly impressive building. It looks over the snowy mountains, its people and its treasury, the National Bank of Switzerland.
As you can imagine, the National Bank building is a master piece as well. And even more impressive; it issues a world leading currency.
On the market square in Oerlikon we first saw this chess board on the street; black and white stones and giant chess pieces. In Bern there was also a giant chess board in the backyard of the House of Parliament. Tinne couldn’t resist to challenge me for a game of chess. (*edit*, Armin noted in a comment that the initial position of knight and bishop are swapped. And OMG, he’s right!)
And she won!
At the House of Parliament you get a stunning, idyllic view on the mountains of Switzerland.
Congratulations to Mr. Van Rompuy for helping the EU powers to find a compromise.
Diplomats credit him with a shrewd sense of deal-making and a determination that is belied by his quiet anti-charisma, and he has already begun to win plaudits from Paris, Berlin and other capitals.
When the RDF store is replaying its journal at startup and when the RDF store is restoring a backup it can be in busy state. This means that we can’t handle your DBus requests during that time; your DBus method will be returned late.
Because that’s not very nice from a UI perspective (the uh, what is going on?? -syndrome kicks in) we’re adding a signal emission that emits the progression and status. You can also ask it using DBus methods GetProgress and GetStatus.
The miners already had something like this, so I kept the API more or less the same.
The original SPARQL regex support of Tracker is using a custom SQLite function. But of course back when we wrote it we didn’t yet think much about optimizing. As a result, we were using g_regex_match_simple which of course recompiles the regular expression each time.
Today Jürg and me found out about sqlite3_get_auxdata and sqlite3_set_auxdata which allows us to cache a compiled value for a specific custom SQLite function for the duration of the query.
Today we improved journal replaying from 1050s for my test of 25249 resources to 58s.
Journal replaying happens when your cache database gets corrupted. Also when you restore a backup: restore uses the same code the journal replaying uses, backup just makes a copy of your journal.
During the performance improvements we of course found other areas related to data entry. It looks like we’re entering a period of focus on performance, as we have a few interesting ideas for next week already. The ideas for next week will focus on performance of some SPARQL functions like regex.
I think the first pieces of the RSS- and the other web miners will start becoming available in this week’s unstable 0.7 release. Martyn is still reviewing the branches of the guys, but we’re very lucky with such good software developers as contributors. Very nice work Michele, Roberto and Adrien!
Nonetheless, I think the European community should do it just to strengthen Europe’s economy. I’m not satisfied by Europe’s economic strength: I want it to be undefeatable.
We must not let the IMF solve our problems. Europe might be a political dwarf, but we Europeans should show that we will solve our own problems. We’re an adult composition of cultures with vast amounts of experience. We know how to solve any imaginable problem. And let’s not, in our defeatism, pretend we don’t.
A EMF is a commitment to future member states: Europe often asks them fundamental changes; economic strength is what Europe offers in return. This needs to come at a highest price: Greece will have to fix their deficit problem. Even if their entire population goes on strike. Greece will be an example for countries like my own: Belgium has to fix a serious deficit problem, too.
An EMF comes at an equally high price, and that frightens me a bit: I don’t want the ECB to go as ballistic on money creation as the FED has been last two years. I want the EURO to be the strongest relevant currency mankind has ever created. No matter how insane the rest of the world thinks that ambition is: I believe that keeping the EURO’s M3 in check is a key to creating a wealthy society in Europe.
Politically I want European nations to negotiate more and more often. The European Union is a political dwarf only because finding agreement is hard. But in the long run will our solution be the most negotiated, most tested on this planet.
Together we can deal with anything. That doesn’t mean it’ll be easy; it has never been easy: just seventy years ago we were still killing each other. We’re all guilty of that one way or another. And before that it wasn’t any better. Today, not that many people still care: “it wasn’t me”, right? So stop being a bitch about it, then.
It’s time to let it be. It’s time to start a new European century that will be better. With respect for all European cultures, languages, nations, nationalities, values, borders and interests.
But also a European century with economic responsibilities for each member. It’s our strength: we figured out how to keep our population wealthy: let’s continue doing so in the future.
It was the dawn of the 1970s, at the height of worldwide student protests against the Vietnam War, and a librarian stationed at a U.S. Information Agency post abroad had received bad news: A student group was threatening to burn down her library.
But the librarian had friends among the group of student activists who made the threat. Her response on first glance might seem either naïve or foolhardy — or both: She invited the group to use the library facilities for some of their meetings.
But she also brought Americans living in the country there to listen to them — and so engineered a dialogue instead of a confrontation.
In doing so, she was capitalizing on her personal relationship with the handful of student leaders she knew well enough to trust — and for them to trust her. The tactic opened new channels of mutual understanding, and it strengthened her friendship with the student leaders. The library was never touched.
– Daniel Goleman, Working With Emotional Intelligence, Competencies of the stars. 1998
In Working with Emotional Intelligence, Daniel Goleman explains several practical methods to improve the social skills of people. Before I bought this book a year or two ago, I read Daniel’s first book Emotional Intelligence. This weekend I finally started reading Working With.
I recommend the section Some Misconceptions. Regretfully ain’t this section available for display in the flash preview widget. Instead of violating copyright laws by typing it down here, I’m recommending to just buy the book.
You can find audiobooks online. The section about misconceptions is at track three. Track five talks about two computer programmers, which is very illustrative for many of my blog’s readers (and possibly myself). I hope you wont illegally download using torrents. Instead, buy the material.
What distinguishes Daniel Goleman from old line proponents of positive thinking, however, is his grounding in psychology and neuroscience. Armed with a Ph.D in psychology from Harvard and a first-grade journalism background at the New York Times, Dr. Goleman has authored half a dozen books that explore the physical and chemical workings on the brain and their relationship with what we experience as everyday life.
– Peter Allen, director of Google university, introduction to Daniel Goleman. August 3, 2007
I hope readers of my blog will shun away from pseudo science when it comes to emotional and social intelligence, but instead read and learn from authors like Daniel Goleman. I also (still) recommend the books available at The Moral Brain by for example Dr. Jan Verplaetse.
psst. I have inside information that I might not be allowed to share that 1.2 is being prepared already, and will have bodystructure and envelope summary fetch. And it’ll fetch E-mail body content per requested MIME part, instead of always entire E-mails. Whoohoo!
You know about those guys that use your software against huge datasets like their entire filesystem, with thousands of files?
We do. His name is Tshepang Lekhonkhobe and we owe him a few beers for reporting to us many scalability issues.
Today we found and fixed such a scalability issue: the update query to reset the availability of file resources (this is for support for removable media) was causing at least a linear increase of VmRss usage per amount of file resources. For Tshepang’s situation that meant 600 MB of VmRss. Jürg reduced this to 30 MB of peak VmRss in the same use-case, and a performance improvement from minutes to a second or two, three. Without memory fragmentation as glibc is returning almost all of the VmRss back to the kernel.
Thursday is our usual release day. I invite all of the 0.7 pioneers to test us with your huge filesystems, just like Tshepang always does.
So long and thanks for all the testing, Tshepang! I’m glad we finally found it.
We would rather suffer the visible costs of a few bad decisions than incur the many invisible costs that come from decisions made too slowly - or not at all - because of a stifling bureaucracy.
– Letter by Warren E. Buffett to the shareholders of Berkshire, February 26, 2010
Perhaps we should put a damp rag like the one he mentions in his mouth next time he opens it?
Nigel Farage, you’re an disgrace to yourself. The European parliament is no place for personal attacks, and you aren’t fit to carry the title Member of the European Parliament. Please keep the honour to yourself and resign.
Every sensible person outside of the U.K. thinks you should. Even the Euro skeptics do. You’re an embarrassment for your country and its culture, so I hope for the people in the U.K. that they’ll kick you out of politics.
I fear you’re just playing the populist card, and that you’ll even get votes for this from other morons.
I don’t decide about Tracker’s release. The team of course does.
But when you look at our roadmap you notice one remaining ‘big feature’. That’s coping with modest ontology changes.
Right now if we’d make even a small ontology change all of our users would have to recreate their databases. We also don’t support restoring a backup of your metadata over a modified ontology.
This is about to change. This week I started working in a branch on supporting class and property ontology additions.
I finished it today and it appears to be working. The patches obviously need a thorough review by the other team members, and then testing of course. I invite all the contributors and people who have been testing Tracker 0.7’s releases to tryout the branch. It only supports additions, so don’t try to change properties or classes, or remove them. You can only add new ones. You might have noticed the nao:deprecated property in the ontology files? That’s what we do with deleted properties.
Anyway
Meanwhile are Martyn and Carlos working on a bugfix in the miner about duplicate entries for file resources and on a timeout for the extractor so that extraction of large or complicated documents doesn’t block the entire filesystem miner.
