Replicating memes

Federico, I’m not going to make yet another long and technical blog post about this, because it’s rather obvious how it works. But you can make a db-cursor or a cursor-like API for a query that is solved at the service and kept in a session.

My opinion is also that 484 bytes actually is a lot of memory per item. Wrap that in a proxy instance and lazy load the proxy, and you reduce that to an actual average of 50 bytes per item. It wouldn’t even matter if when parsed the size exploded. Only the items that are needed right now would be exploded in memory, in stead of all items (which is now the case). You could even cache the strings that are needed to render an overview of the items in the proxy, and avoid creating the real this way. This would give you a solution that could scale to 100,000 items (although that’s probably stupid to render in a view, sure).

Note that for this proxy concept, you don’t even need a cursor-like API. Just an API to fetch one real item, and a way to fetch just the minimal info for your proxies, so that they know how to fetch their real.

About the structures that tend to explode in size when you parse them: is the library using GStringChunk to store all the tiny little strings? Else I kinda know already where all that heap-admin is going to. It’s a lot, if you measure it. That’s because all of them are stored in parsed structures, right? Each mini string needs four bytes for the pointer, at least four bytes for heap admin, etc. I can also imagine that this data contains a lot of duplicate strings, perhaps take a look at how camel_pstring_add works? If GStringChunk is sufficient, then go with that, of course.

Or, even more simple:

If you’d replace all \n characters with \0 characters in the original VCard string, you could also just make them point to an offset in the original buffer. Note sure about line wrapping and whether the format could support this easily (not my field of expertise).

Also note that the prevailing model for E-mail might be that you display a usually huge linear list of messages, but that the market is increasingly demanding a search-based model. Especially for mobiles with small screens.

Clarification:

• A proxy here is an instance that has the same interface as the actual thing. To fulfill that interface it might have to get a real first, and then proxy the command to the real. A proxy usually only contains the things that are necessary to fetch a real from the service. It can also contain extra data (I call this a micro cache) so that it can avoid having to fetch the real for common strings. For example strings that are displayed in an overview screen. A proxy must be small.
• A real is an instance with (all its) data fetched from the service. Perhaps not pixbuf image data if in this case the card contains image data. Perhaps you want to lazy load that data, in the real, too. A real can be really, really large.

This entry was posted on Thursday, March 20th, 2008 at 12:46 pm and is filed under Informatics and programming, english, maemo, meego. You can follow any responses to this entry through the RSS 2.0 feed. You can skip to the end and leave a response. Pinging is currently not allowed.