Tired of the fact that my N9 had few battery time I decided to “as a developer” investigate my device a little bit. Last time I did that I was still contracted by Nokia and a few days later I had to fly to Helsinki to help fix a Tracker in combination with contactsd bug. I’m btw. no longer working for Nokia since a few months. So this time I can’t fix it for everyone. Lemme write it here instead.
It’s pretty funny what is going on: I installed Battery-Icon at some point. The software is writing periodically to /usr/share/applications/battery-icon.desktop. Having been a developer at Nokia for the metadata subsystem I know that tracker-miner-fs will reindex .desktop files that change. You don’t really need to be a developer to know that: Tracker’s FS miner is, among other things, responsible for keeping up to date a list of known applications.
Because of Battery-Icon, which people are probably installing to monitor their battery, tracker-miner-fs wakes up to update the metadata. That in turn wakes up tracker-store to store the metadata. That in turn wakes up smartsearch which will fetch from Tracker some textual data. All three will consume power periodically because of this .desktop file write trigger. I’m guessing the power consumption is triggering Battery-Icon to update the .desktop file. And circular power consumption was born.
I guess I should file a bug on Battery-Icon and tell its author to update the .desktop file less often. I think he could for example wait ten minutes before doing that write. Or is the user really interested in accurate battery information each and every second? Looks like Battery-Icon is even writing to the file more frequent every hour. Interesting behavior for a tool monitoring battery to do things in a way that influences power consumption significantly.
Btw, while it’s not fixed: devel-su (enable developer mode, install terminal and password for devel-su is rootme) on your N9 and chmod -x /usr/bin/smartsearch, reboot, then uninstall Battery-Icon and your battery will last longer. I know the guys who were or are on the smartsearch team are going to hate me for that advice. Sorry guys.
Do you have any advice for the gallery problem? No photos appear in Gallery, just some junk entry which does not correspond to images, and when opening says the file is corrupted. I removed all pictures, and hard-reset tracker, but the problem persists. I will flash EMMC now, let’s see.
@Kis Zoltan: I know that there can be … a … reason why corruption of Tracker’s SQLite database can happen, and that making a backup will just copy the corruption with it. So even restoring the backup you’ll still get your corruption in the SQLite database nicely back (this is because of how sqlite3_backup works).
A way to fix it is to sacrifice your data and flash the device (or to remove Tracker’s database files, but you can’t manually do that, at least not easily, due to Aegis security on ~/.cache/tracker).
About that reason, lemme put it at: use dmesg with grep and your memories about a very interesting bug on the bugzilla. Note that it’s not caused by Tracker itself.
Tracker is actually SQLite too? Good news because it means at least it could be dumped and repaired generic way (I hope). My fist concern with tracker database is that it is extremely easy to damage it and then EVERYTHING on your phone goes wrong at once.
Start a terminal on your N9 phone and issue this commando:
tracker-control –backup=file.sqlite
And afterward
tracker-control –restore=file.sqlite
You can open file.sqlite (on your desktop) with a standard SQLite client. You can even shut down tracker-store on your desktop, then put the backup-file of your N9 in $HOME/.cache/tracker as meta.db on your desktop, and then restart tracker-store on your desktop. It should work just fine. Tracker uses the sqlite3_backup API for this, by the way (and, it’s all open source on GNOME’s git).
Note that the N9’s standard backup and restore software does the same thing. It just also puts this file into an archive, but you can open the archive and take the metadata DB out and then open it with SQLite. There is nothing special about all this on your N9.
Enjoy.
SQLite databases do not get damaged easily at all – assuming that safety measures such as fsync are not disabled, of course. Tracker on the N9 has fsync enabled and our endurance tests didn’t reveal any issues in SQLite.
If you do see database corruption, your phone hardware may be defective. If you see this repeatedly, you should send it to your local dealer / Nokia for exchange.
No, as stated below, it is purely software and sadly reproduceable issue :-(
https://harmattan-bugs.nokia.com/show_bug.cgi?id=245
this is one simple way
and sync may lead to similar situation as well
So why on earth does the battery icon dare to touch its own .desktop file? Desktop files are supposed to be persistent and not contain any cache data, no?
So the fix would be to make sure the battery icon doesn’t suck and that’s about it?
Because the battery-icon needs to update the icon displayed in those.those homescreen-application so it reflects the changed battery state. The homescreen-application provides only one way to update the icon, that is to touch the desktopfile. The homescreen-application has on each desktopfile it reads and displays the icon for a fileSystemWatcher and rereads the desktopfile+updates what is displayed if the deaktopfile changes.
So, an alternate way to solve that is to add a proper dbus-method to the homescreen-app and bypass all the desktopfile changes. But we cannot get such fixes in any longer and it seems there will be no 1.4. What could be done is to offer a fixed homescreen-app and let it replace the default one (Nokia forgot? to apply aegis-patches to X11 and so apps can register as desktop and effectively replace the homescreen-app that way at runtime – have working prof of concept if there is interest).
Author of the software can just delay the write to the .desktop file using Harmattan’s Heartbeat API, and all would be ok. The frequency of the battery event is not guaranteed and so you shouldn’t write to .desktop files in that event handler. So postpone writing to the file using Harmattan’s Heartbeat API.
Of cause that would be the easiest solution and should be done imho anyways. Independent of that when someone some day has interest to extend the homescreen-app: http://meego.gitorious.org/meegotouch/meegotouch-home
Guess with X11 being “open” some more interesting thinks can be done with harmattan.
I think it could indeed for example listen for a signal msg on D-Bus and then applications like Battery-Icon could just emit that message over D-Bus, so much like what you suggested (just that I would use a signal instead of a method, because multiple users can then connect and act on it instead of just one – ie. HomeScreenNeedsUpdate).
But you’re right that we cannot get such fixes in any longer. A fixed homescreen app that replaces the stock one, done as an app, would actually not be a bad idea if you can guarantee that it won’t brick people’s phones.
Who is ‘we’ here btw? Mer or Jolla? or? ;-)
Ideally (one of) those groups stop trying to replace the whole OS and start supporting MeeGo Harmattan by providing replacements and/or fixed or improved packages of the opensource stuff. Most is available in gitorious and/or has the Nokia work and patches been committed upstream. And then we can still see what we can do about the closed parts. Like rewriting them, or asking Nokia to open the source code (you never know). I would probably join that group, whereas I see the efforts to rewrite most of the whole OS (by naming it MeeGo it doesn’t mean it’s not a rewrite) as doomed from the start. That’s also actually what N9 owners are asking for. Most people are not asking to replace the whole damn OS, or to try to get MeeGo (not MeeGo Harmattan) on par with MeeGo Harmattan on an N9. Which an almost impossible task if it is to be done as a community effort, if you ask me: the sync between Nokia’s and Intel’s platforms was absolutely not finished at the time MeeGo and MeeGo Harmattan defacto splitted. It was mostly PR blab bla.
> A fixed homescreen app that replaces the stock one, done as an app, would actually not be a bad idea if you can guarantee that it won’t brick people’s phones.
I am workin on that. Protoype works fine but needs some more things so the original homescreen app continues to work like before when replacement is quit. Will push to gitorious once done and publish in Ovi later.
> Who is ‘we’ here btw?
Anyone who likes to make fixes to parts that are Nokia aegis signed and/or would need a PR1.4 to reach users. I would love to extend homescreen-app (sparetime like you not being contracted by Nokia any longer).
> Ideally (one of) those groups stop trying to replace the whole OS and start supporting MeeGo Harmattan
I could not agree more. That would be the fastest and most clever way to move forward.
> I would probably join that group
So would I. Its all there, the revolution was done and finished. From here on what id needed most is evolution, polishing, improving and extending and not start from scratch once more (what applies not only to Mer but to the whole stack).
p.s. thanks for investigation+fix+blog. Great work!
@sebsauer:
Thanks and no problem. I had an itch to scratch for my own phone and after scratching it was writing down a solution just a small effort.
Good to know that there are people willing to do this. Problem with Aegis signatures is that even the developer certificates that we got on our prototypes won’t work on the PR devices sold to people.
So not even in an illicit way can we break into the platform (nor would we, mind that we all had very very strict NDAs for this).
There are probably plenty of exploitable holes, though (again, try finding them yourself).
I would very much prefer a deal with Nokia allowing us to legitimately take over maintenance of some of the N9’s stack. I’m sure there are still people employed at Nokia with the right mindset who can make this happen. But I don’t know how much the new top management at Nokia have screwed things to the point of something like this being impossible to ever happen.
Only thing I saw was a top management trying to kill the N9. But perhaps I’m too much into conspiracy theories? It could also have been pure incompetence of course. I don’t know. I only know it was sad to see it happening.
The idea was ~ that Nokia itself would lead that community and allow the community to provide patches that they would manage and push to the public releases through the software update mechanisms. This is why gitorious was used as a partner for the purpose of inviting the communities of many open source projects. Kinda like the Maemo and Fremantle community stuff but done better.
Anyway … I can cry and moan about what did and didn’t happen for the rest of my life or I can try doing something about it. (or I can move on, which is what I’m doing now more than trying to do something about it. I admit).
> But I don’t know how much the new top management at Nokia have screwed things to the point of something like this being impossible to ever happen.
I think right now its near to impossible. Harmattan is still an investment they may try to make into cash somehow, eg selling it, and the last thing they would support right now is that someone is able to pick up there aborted path and make it a success. Some may even fear it could end in showing how successful that strategy would have been if applied. I fear its just easier to keep things like they are now rather then doing anything.
But I see that as huge possibility to make a Harmattan that is 100% free. Its the perfect situation to build up something that really is and stays open. That allows any startup to pickup, put on hardware and sell.
Btw. Is Mathias or somebody else still working on contactsd? Whoever is working, if anyone, on that should make contactsd burst its updates as soon as the device goes out of dimmed mode.
Right now while the device is dimmed, contactsd is causing wakeups of tracker-store and smartsearch needlessly. It could just collect the changes in its own state, and burst them to tracker-store as soon as the user undims his device.
According to powertop and logging of tracker-store.log is this combo of processes among the top reasons for power consumption while idle and dimmed on my N9. And it should be pretty easy to fix this if I remember contactsd’s code correctly.
I don’t really understand why availability of my contacts must be accurately up to date while my device is dimmed …
@Benjamin Otte: yeah. I didn’t find a bugzilla or something for the app so I wrote an E-mail to the author. Let’s hope he fixes it soon. Note that it’s not a standard Nokia app, but it is available in Ovi Store (Nokia should IMHO perform better at quality control, like doing a power consumption test run after installing an app, before allowing apps in Ovi Store). Perhaps it shouldn’t have been possible to write to the .desktop files from your process in the first place, but only by the package manager. Dunno.
Same situation as you. Cannot push fixes anymore.
Hmm, and I guess such a change can’t be done as a plugin or something? Bleh, Nokia shouldn’t have stopped Harmattan. Poor N9 owners. Can’t we start a repository for Harmattan and explain people how to add it? And then just let it overwrite contactsd somehow. Not sure if that’s doable due to Aegis stuff …
That would be awesome :)
Developer mode.
Is anything of contactsd closed? Like a plugin? Else we could indeed tell people to first enable developer mode, then add a repo, then enjoy updates and get a community fixed contactsd. No?
Great info, thanks. What exactly does smartsearch provide in the n9?
thanks again,
The “Search” app uses data provided by the smartsearch service. If you remove smartsearch’s x permission, it’s likely that “Search” wont work correctly any more. I never used it myself, to be honest.
So it was a third party application responsible for the battery drain. Well that’s kind of good, glad to hear it wasn’t messageserver (as then I would feel responsible).
Personally I haven’t had a problem with corrupted data myself, and I actually really like the use of sqlite wal in Harmattan, but I’ve seen some bug reports about corrupted data. Such a hard problem to prevent. I’ve advised users to factory reset (and not restore from backup) to fix it, hopefully I haven’t been misinforming them.
Such a tragic situation. If only Nokia had the strength and vision to believe in Harmattan then how much different and better things could have been.
SQLite WAL also isn’t the problem. It’s not about any software. We actually never saw any serious bug or problem in SQLite WAL, even though we started using it when it was still experimental.
I indeed didn’t see messageserver in an abusive way in Tracker’s logs nor in a significant way in powertop’s output during my tests of yesterday.
Indeed, it’s a tragic situation. Harmattan was ready, if it just would have gotten better support and a few months of bugfixing then its latest 1.2 upgrades would probably by now have fixed each and every major issue and by now the platform would probably have gotten a few new top features extra.
I just hope the N9 customers won’t blame the technical team behind Harmattan for the silly decisions Nokia’s management is taking.
Other processes that consume a lot of battery are skyhost and presenced. You can turn off skyhost by setting your Availability to offline for your Skype accounts (but I understand this isn’t an option for a lot of people). Skyhost is used by the Skype support. Presenced is used for setting presence status of your IM accounts.
First of all there is NO battery drain caused by the Battery Icon app. Just because it updates the .desktop file when the battery level changes that doesn’t mean there is a battery drain. The updates are too small and too infrequent to cause a battery drain.
Sorry to read that you think that way: I measured it, I’m sure there is battery drain caused by this. Also when uninstalling your software, the battery lasts 5 hours longer (this, I also measured). I enabled Tracker’s logging and I saw battery-icon.desktop being the file causing it to wake up. Your process writes to it frequently.
I guess that means your tool won’t be fixed. Sad. The problem btw. isn’t the amount of data that you write, but the fact that you write (to a .desktop file). As writing to the .desktop file triggers waking up of processes (like the metadata services) having to reindex the file (so the draining might not be caused by your process, but by a trigger triggered by your process). A way to fix it is by doing it less frequently. For example use Harmattan’s Heartbeat API which I E-mailed you. At most Heartbeat triggers the processes that your trigger wakes up are awake anyway. Because of that your activity won’t influence power consumption a lot anymore (if you’d use the Heartbeat API).
I don’t know how the frequency is determined, but I saw it being > 1 times per second, on my device yesterday. Everything faster than let’s say a minute shouldn’t happen when the device is in idle, as such activity is interactive activity while the user is actively using the device (never the case with Battery-Icon). Anything slower should use the Heartbeat API.
Well if you have some proper measuring setup and tools I would certainly like to hear about them, but I also ran extensive testing with the app and I detected no drain. There is though a s small chance that there is some bug which causes a drain in your specific case. So if you can provide me with more details would be great. But most likely something else causes the drain if there is a drain at all.
I am not using the Heartbeat API for this because I need to update the battery level right away when it changes, not a few minutes later. And these updates definitely do not happen every second, more like once every 30 min if your battery lasts 2 days or every 15 min if your battery lasts 1 day.
Ok, so you used an event that you get from the battery.
Since you don’t control the frequency of that event, you shouldn’t write to files in an event handler of it, as writing to certain files (and in particular to .desktop files) triggers new events. Those new events can trigger your event again (logical loop danger).
In this particular case it is causing three other processes to wake up and start performing activity (being tracker-miner-fs, tracker-store and smartsearch). Those three processes’ activity will result in yet again power consumption.
And that power consumption likely is triggering the same event that you are listening for. And I guess that in my scenario I had a logical circle, causing battery drain. Since this should just never happen, it’s a bug (admittedly could the event sources break the cycle, but since that won’t happen you’ll need to fix Battery-Icon instead).
I would suggest to continue using the event to read the value but to queue its values in a queue or to overwrite a “last percentage” static variable. Then use the Heartbeat API to periodically read that queue or value and write to your .desktop file at Heartbeat’s frequencies. Don’t write to your .desktop file in the battery’s event.
You asked for proper tools. The tools I used was a) enabling Tracker’s logging (devel-su, develsh, set Verbosity to 3 in /home/user/.config/tracker/tracker-store.cfg and tracker-miner-fs.cfg, cd /home/user/.local/share/tracker/data and read or tail tracker-store.log and tracker-miner-fs.log after a reboot), b) powertop (apt-get install powertop) and c) just normal top and htop.
I saw tracker-miner-fs, tracker-store and smartsearch in powertop’s output and in top and htop. Then I started following tracker-store.log and noticed that it was continuously getting data for desktop-icon.desktop by tracker-miner-fs (this explained tracker-miner-fs and tracker-store in powertop’s output). I also saw smartsearch but that process gets woken up because of tracker-store signalling it that new textual data is available for fetching. Your .desktop file write causes such textual data to be needed to be updated by smartsearch. So smartsearch is only indirectly related, but it itself was the top power consumer in the cycle of four involved processes.
Then I investigated which process is doing the writes to desktop-icon.desktop, and it’s the app Desktop-Icon itself doing this. In itself is writing to that .desktop file not a big issue. It’s a big issue that the frequency of writing to the .desktop file ends up being uncontrolled (a faster and faster frequency, in my scenario).
Cutting a long explanation short: use the Heartbeat API for the writes and don’t write to the .desktop file in the battery event handler. If the battery’s event handler is the only way to get the value, continue using it but let the write to the .desktop file be for later (again, ie. at Heartbeat’s predefined and platform-wide frequencies).
I didn’t mean to imply wal is a problem. Quite the opposite, I’m saying I think using wal is a great choice.
In my experience, working with sqlite wal, database corruption has been due to bad device memory or logic errors in relatively high level application/middleware code that have put bad data in the database and entered a confused state. I’ve never found a database corruption bug in sqlite. It’s great software. (I don’t have experience with tracker but I think it is built on a solid foundation).
I love using my N9, think it is a great product, and it has got bugfixing with the PR releases. But of course I agree with you that it could have been even better with continued development by the team. In fact personally I think that if Nokia had committed to Harmattan then they could have more or less retained market share in the smart phone market.
But I have! Just try to merge “wrong” contacts and it will go into the infinite loop. Address book is barely usable on Harmattan.
@ArkanoiD I doubt merging ‘wrong’ contacts is triggering an sqlite bug. It’s more probably an application/middleware bug.
Sadly AFAIK there is no longer a dev team to look into the root cause of the issue.
I really liked the concept of this app and thank Cristi for his efforts in writing it.
I downloaded the app from Nokia Store very soon after it was first released and I wanted to write a great review of it.
I am not developer and so I am not even going to try to debate how the app was written. However, it was very noticeable after 24 hours of installing Battery Icon that my device’s battery was draining, fast. Battery Analyser even identified the app.
So sadly, I had to uninstall. Battery comsumption returned to normal rates.
I think (from a lay person’s perspective) that the suggestions outlined above are worth investigating. I hope that Cristi considers this course. I would be happy to hear any updates from him should he decide to make some tweaks to the app.
Regards to all
John
I also have battery icon installed but I have no battery drain. My average idle consumption is most of the time around 10mA. I can easily get 2 days of battery time, even when showing battery status on the stand-by screen.
But … I do have a massive battery drain at the office caused by WiFi connection (> 300mA idle & active consumption). It’s as if my phone continuousely processes the incoming data packets and stays awake.
I have no active IM accounts and my two mail accounts only update once every 30 minutes, so those can’t be the cause.
When I disconnect the WiFi connection, idle consumption goes back to 10mA. (Android & WP7 phones don’t seem to have this problem)
My best guess is that you have an app installed that is keeping your wifi hw alive needlessly. You could try using powertop and top or htop to detect which one.