Oh and Jason, you are hereby kindly invited to join the development of tinymail. A framework for rapidly creating E-mail clients for mobile devices, embedded appliances and soon also web applications.
It having a focus on these targets doesn’t mean that with one extra developer, I can’t focus its development toward a framework (also known as the beginnings of) for a desktop E-mail client. In fact, Modest, a E-mail client being build on top of tinymail by Nokia, is probably going to be a desktop E-mail client for a lot people. After that (being under NDA with Nokia I can’t tell you more), the plan might be (hehe), who knows, to bring it to the Maemo platform too.
It’s likely (grin) that somebody is fixing it to work with the recent API changes which I have been introducing the last few months. These changes and my commitment not to release until all obvious API changes, documentation, unit testing and at least Python language bindings are fully working have been delaying such a release. A Modest release is also delaying that release, as the current idea is to in parallel with Modest, release the first tinymail version.
But then again, people tell me I code tinymail faster than my shadow. I even think that’s a direct quote from Florian (or it was something like that). I guess getting at a first version faster is therefore not really healthy for me anymore. By that I mean: I can use your help (especially if you want it faster).
The development pages of tinymail go in detail about what can be done and what should be done before a first release. They even explain development techniques and details. They also explain how to debug and measure memory usage correctly. In other words, there’s no excuse possible like: “but but, there’s no information about the code”. Lots of such information has been added recently. I often add more.
Oh and if you like Sylpheed, maybe you and me can investigate whether or not it’s doable to extract a libsylpheed out of Sylpheed and use that for implementing a libtinymail-sylpheed as a replacement for the libtinymail-camel? That would indeed be possible, yes. Or maybe a libtinymail-akonadi?
There is a libSylph already, but it’s the Sylpheed library, which is very different from Sylpheed-Claws. See http://sylpheed.sraoss.jp/en/news.html
Well, I looked at the .h files very quickly, especially the folder.h. The api of libsylph doesn’t look extremely different. The implementation itself, however, isn’t the same. With libsylph, for example, it will be more difficult to mmap the data which I in tinymail and the Evolution people in Camel call summary information (the displayable headers, I think it’s going to be called in sylpheed?). In tinymail this is indeed the TnyHeader type.
By the way, my current idea is to have one interface but two implementations. One TnyCamelWritableHeader and one TnyCamelDisplayableHeader. Where the displayable one isn’t ever going to be instantiated by the developer who’ll use the tinymail framework, and the writable one is to be instantiated if the developer wants to create a reply or compose a new message.
The current TnyCamelHeader is a little bit misdesigned. Which will of course be fixed sooner or later.
Anyway. I don’t see any big hurdles that make a libtinymail-sylpheed impossible at this moment. But I also don’t yet see a very good reason to switch from the current Camel implementation to libsylph. Libsylph’s memory consumption will not be drastrically better. It looks like most of the memory consumption reduction done by libsylph happens at it’s user interface (not much bloat). Whereas, for a mobile E-mail client .. the memory consumption is never going to be at the user interface. But in stead directly at the “displayable headers”.
But I really first need to get some introduction to the technique being used and/or read the code myself. Perhaps it’s using some clever way. Like storing only the really visible information.
The current problem with Camel (with the mmap patch) is that it keeps a massive amount of pointers around per each such “displayable header” or “summary item”. The problem with a normal Camel is that it not only keeps those pointers around, it also stores “everything” in malloc() allocated buffers.
How the mmap patch, and therefore camel-lite(-builder), solves it is by keepinge data itself on-disk in a on-demand-page-loaded mmap. But the pointers to that information (four bytes each) are still real allocated memory.
Maybe the sylpheed author and me should sit together and share ideas. Are you the developer, Colin? You didn’t leave your E-mail address in the comment, so I don’t know.
Here the sylpheed developer :
(Hiroyuki Yamamoto)
http://sylpheed.sraoss.jp/en/
And here is the list of sylpheed-claws developers :
http://claws.sylpheed.org/theteam.php
In fact, with Colin, we saw that memory use of tinymail was comparable to sylpheed-claws memory use.
http://tinymail.org/trac/tinymail/wiki/MemoryStats
(I consider mmap things useless since once you touched all memory used by messages, this will remain in physical memory until more physical memory is reclaimed by an other application or by the application itself).
If you really want to restrict memory usage of tinymail, you’ll have to implement a real memory manager inside the application that won’t let the headers use more than xxx megabytes memory, swap data between memory and disk.
@hoa: But, first of all, you don’t touch all memory. And second of all, on a mobile device with few memory resources, other applications *are* going to ask for more memory. And since an mmap is demand paging, the tinymail process will not keep the mmaped memory in real memory unless it’s needed. So I don’t consider it useless. Rather important for the target of tinymail: mobile devices.
In stead of such a memory manager, it would be more simple to use the virtual proxy pattern and the gtktreemodels _unref function pointer to know when a proxies real instance can be cleared.
Tinymail already supports this. But doesn’t use it with the current camel implementation. It did used to use it before the mmap. With the mmap, that technique is useless as all data of the file must be mmaped anyway.
I measured (in kernel) that it’s better to mmap the file, than to use an aggressive allocation and deallocation scheme using the virtual proxy pattern. First because memory allocation and deallocation (even with a slice or magazine allocator like gslice) isn’t “nice” for your mobile device’s battery. Second because it’s slower (a lot slower) than what the kernel makes of the mmap demand paging. Thirth because mmap is rather clever by itself and that therefore the real amount of memory is depending on the real amount of free available memory. Which is better than letting the tinymail process choose (the kernel knows better, because the kernel manages ALL applications). And fourth because it reduces complexity in the E-mail application, which the kernel implements with mmap anyway.
About putting soms sort of memory manager in tinymail. No, that doesn’t sound like a good idea at all to me. In design I don’t see a reason why that would be needed. As I said, a virtual proxy pattern solves the same problem in a MUCH more transparant way.
Note that if you sort the summary, that indeed in that case you will touch all memory. But then again, most of the times are the items in a more or less sorted way stored in the summary file already. Therefore also displayed in that order. Most people (using mobile devices) will not sort it. Some developers also told me that sorting isn’t really always needed (depends very much on the mobile device and the concepts of the ui of the mobile application: imagine a cell phone with a little E-mail application applet on it).
In those cases, only the first page of the summary file will be swapped in memory. I measured that (depending on the amount of available memory and kernel tweaking) that with the on-demand paging and hinting the mmap that it’s a sequential mmap, very few memory is needed (mostly not more than two or three pages). It does depend on your kernel and its fs’s mmap implementation.