While developing Tinymail, a library for writing E-mail clients, I was convinced that the storage of the summary was something Tinymail itself must handle. Back then there was, even pragmatically, nothing that could cope with the requirements of E-mail on mobile devices for this task.
Meanwhile I got the opportunity to work on the Tracker project. Using the Nepomuk Ontology, I made sure that the message ontology that Tracker uses can actually handle these requirements. I believe that the adoption of Nepomuk and SPARQL evolved Tracker from something that isn’t useful for E-mail software to something that should be involved when writing a desktop service for E-mail today.
Ryan, a pioneer in experimenting E-mail as a desktop service, advised me to be careful with my bias for RDF and SPARQL. I’ll keep it in mind! However…
I believe such a desktop service for E-mail should:
- Download metadata by getting and parsing ENVELOPE and BODYSTRUCTURE using the FETCH programme of an IMAP server. As explained in this document.
- Give priority to downloading metadata of those E-mails nearby the user’s scroll position.
- Use IMAP’s pipelining. It gives the user the feeling that his technology operates faster than his human brains, even when on high latency connections like GPRS.
- Cache the information, using Tracker’s Nepomuk Message Ontology as schema.
- Make it possible to fetch just one particular MIME part and not the entire messages.
- Enable it to create a new message, consisting of individual MIME parts.
- Make it possible for those MIME parts to have their source in existing messages on an IMAP server when creating a message. When the IMAP server supports CATENATE, it should be used for this purpose.
- Make applications use SPARQL with Tracker’s NMO to query metadata about E-mails.
- Provide a stream API to get access to the the actual data of individual MIME parts. If not cached, the service should download the MIME part on demand. The DBus Stream API should look like GInputStream. Except for read(): I think for the transfer of the chunks of data that Unix Sockets or named pipes are better than using D-Bus.
To this I would like to add that although many people falsely believe that E-mails are like files, E-mails are more like recursive directories (container MIME parts) with items: the E-mail’s MIME parts. Any API that doesn’t admit this, is incorrectly designed.
This goes all the way up to the protocol, where you fetch per MIME part. You don’t fetch entire messages. You can indeed do that but that doesn’t mean it isn’t wrong. IMAP is not POP3. It’s also better to design for IMAP, than to use IMAP as a POP3 service. Better have hacks to support POP3 in your model (I’m serious).
Please don’t make the same mistake nearly every newcomer of E-mail solutions makes. There’s plenty of rubbish already, seriously.
Mail as a desktop service was done very well by the BeOS, using a simple multi-protocol mail agent that saved incoming messages as plain text files, and all the header information as filesystem attributes. (The BeOS file system had a tracker-style indexing service built in.)
You browsed mail using standard file manager windows. On the BeOS, though, you could create ‘live queries’ as folders (think evolution vfolders), and the file manager was more of an ‘object manager’. You could set up a file manager with columns for any custom attribute on a file type. So for reading a folder full of mail, you’d set up a file manager window with columns for the from address attribute, the subject attribute, etc.
The only other ‘mail application’ was a simple app that showed a single mail composer window. When you clicked Send, it would write a text file in the outgoing folder, and the daemon would pick it up and send the mail.
MIME parts were stored as filesystem attributes as well, I believe. I could be wrong on that front.
When Be, Inc. died, some folks wrote an improved mail daemon. http://bemaildaemon.sourceforge.net/
BeOS was way ahead of its time. I last used it nearly 8 years ago, and in a lot of ways GNOME still doesn’t come close to matching it for ease of use, simplicity, and pure usable elegance.
A bit off topic I know, but you triggered some memories. Good post.
Isn’t this what Akonadi is for? According to http://trueg.wordpress.com/2009/02/19/akonepomuk-neponadi-friendly-takeover-or-real-love-marriage/ Nepomuk and Akonadi will kinda merge anyway
or you can use Google Gears
the last point should really be:
* provide a gvfs backand to get access to the mails
a mailbox is noting different than a directory or filesystem and all the stuff you writing about is already implemented in gvfs.
some time ago i started implementing a gvfs backend that mounts a Maildir, just to experiment how such a fs could look like:
http://github.com/initcrash/gvfs/blob/fb02e2838207d439c38177680cf98eb0fc9b4135/daemon/gvfsbackendmail.c
Fairly interesting insights (which I’m guessing the first commentor doesn’t really get the technical merit of, or is just trolling). I like the focus on IMAP and also the ideas given for use of it. These focus on the client side aspects of it, so what problems do you think implementations might face on the server side?
@JP: The IMAP servers, you mean? IMAP servers should implement QRESYNC & CONDSTORE, NOTIFY, COMPRESS or STARTTLS (with compression over SSL), THREAD, SORT, UIDPLUS and BINARY.
The DBus E-mail desktop service should focus its implementation on having those capabilities available. It should implement alternatives for when the capability isn’t available, but it should be designed “as if” all IMAP servers have it.
By that I mean that if designing such that it’s a performance problem to “prefer to assume availability of a capability”, when in fact it’s not, then it should be a performance problem. The design should not lean towards IMAP servers that don’t offer mentioned capabilities.
For the simple reason that IMAP servers that don’t, should be punished for keeping the IMAP protocol old and archaic.
IMAP today should be RFC 3501 plus QRESYNC & CONDSTORE, NOTIFY, COMPRESS or STARTTLS (with compression over SSL), THREAD, SORT, UIDPLUS and BINARY. No matter what Mark Crispin has been saying last few months.
I personally think he’s wrong. He apparently believes that because big vendors don’t implement the fancy capabilities, because in fact they don’t want IMAP to succeed in the short term future, that it’s impossible to define a better IMAP than RFC3501 is. I disagree. Flat out and philosophically.
Instead we should ignore more the ones that don’t implement the important extra capabilities. Like Google’s IMAP server.
And we should congratulate more the ones that do or that have developers who’s intention it is to improve their product by adding IMAP capabilities that are useful. Like the ones I listed in this reply.
@Markus: The Akonadi team currently makes one huge mistake: they want to use IMAP as IPC between the applications and the desktop service. Because, so they say, “there are existing E-mail clients”.
This is a very, very bad argument. As I believe applications that will consume E-mail materials will in future be fundamentally different designed and written than today’s E-mail clients are (most of which suck really hard, and the authors of the E-mail clients know this – hi guys!- ).
The desktop E-mail client is archaic and also is its typical UI paradigm, wrong. Especially for mobile. A lot of applications want to integrate with the E-mail metadata, and materials. And no one of those application developers wants to talk IMAP with a local service. Nor do they want to link with some strange library that does it for them. IMAP is also designed for non-local connections. Not for local 2 local connections. And IMAP’s SEARCH, even its new ESEARCH, capability is simply put: inadequate. It’s too simple. It’s not powerful enough. It doesn’t even compare with SPARQL.
I don’t know what Akonadi are planning, but if they are planning to marriage SPARQL with IMAP, then they might as well just use DBus instead. At least for the interactive part of the protocol, and a much more simple named pipe or unix socket for what IMAP’s FETCH BODY.PEEK does. That FETCH BODY.PEEK is what the service must do, not what each and every application must do. Application developers want to talk to a Stream API. Like GInputStream in GLib, IO.Stream in .NET and similar in Java.
A reason why I’d like to use for example named pipes, is because then app developers could for example use fdopen() for this. But right now that’s just an idea. Anything is better than teaching all applications about IMAP’s FETCH programme.
What I do like about Akonadi is that they too are planning to use Nepomuk’s Message Ontology. I hope they too soon find out that they’ll need to adapt it to store the message’s MIME structure, and that they’ll probably need to adopt the adaptations that we did for Tracker. But don’t worry too much, we (the Tracker team) are in contact with some of the Akonadi people. Mostly, at this moment, about Nepomuk and other such ontology issues. I’m also pretty sure that at some point they’ll see the light and abandon that foolish idea of wanting to use IMAP as IPC. If they do, then I think whatever they’ll come up with will look a lot like what I’m proposing here, indeed. By that time, who knows, both Akonadi and Tracker will be talking Xesam 2.0 with SPARQL as query language and Nepomuk as ontology. And we’ll just be competing at the level of implementation. Which would be great, btw.
Also note, and this might shock some KDE fans, I know KDE fans are easily shocked, that Akonadi is about as much vaporware as the idea in this blog post is. Sure it’s thrown together. But I would absolutely not say that it’s “ready for use”. It’s still in experimental phase. A lot. Notwithstanding is it good that experiments are taking place. Experiments don’t always look exactly like the future. They help defining it, though.
@Gabe: I just looked at the code of that project, and at first sight I have a bunch of serious concerns about the rather limited API. But you are right that it was ahead of its time, yes. I can go into those concerns, but that would be a few new posts by themselves. However, I’ll study it further, as I want to learn from past successes and mistakes. Thanks.
Any chance of re-architecting the tinymail UI to be more like sup?
http://sup.rubyforge.org/
(sup is made of pure awesome)
Unfortunately the evolution maintainers rejected the idea of making evolution more like sup.
@pvanhoof:
“The Akonadi team currently makes one huge mistake: they want to use IMAP as IPC between the applications and the desktop service.”
This is a misunderstanding, a very common one actually :)
The protocol used on the data connection is somewhat similar to IMAP in the sense that it allows for pipelining, fetching individual parts of data items, etc.
It is not like one would connect with an IMAP capable client and try to squeeze all necessary operations into the capabilities offered by IMAP ;)
“And no one of those application developers wants to talk IMAP with a local service.”
Exactly, especially those which are not interested in e-mail at all, but perhaps on contacts, events, todos, bookmarks, etc.
“I don’t know what Akonadi are planning, but if they are planning to marriage SPARQL with IMAP, then they might as well just use DBus instead.”
Well, aside from not planning to marry SPARQL and IMAP, we are in fact using D-Bus for the control connection, e.g. change notifications using D-Bus signals, etc.
“Application developers want to talk to a Stream API. Like GInputStream in GLib, IO.Stream in .NET and similar in Java.”
Exactly, or in the case of KDE developers a job based API.
There is often a weird misconception by some developers it will always be better to have the same API across stacks, while it is usually a lot better to have an API specifically designed for a stack so it matches all expectations of developers on that stack.
Good example is D-Bus, i.e. how the Java and Mono folks implemented it natively in their stack to look what their developers expect it to look like.
“By that time, who knows, both Akonadi and Tracker will be talking Xesam 2.0 with SPARQL as query language and Nepomuk as ontology”
Indeed. Our understanding is that it won’t matter if the system running Akonadi is running Nepomuk or Tracker as the query service as both understand SPARQL and use the same ontologies.
From our point of view there is no gain in depending on one specific implementation of a query service.
“…that Akonadi is about as much vaporware as the idea in this blog post is.”
Well, no, it isn’t.
It is not as fully integrated with traditional KDE applications yet as we would have hoped and we are missing some backends like POP3, but if one would want to do e-mail based on Akonadi using local storage and IMAP (and NNTP IIRC) that is possible.
Of course having the well known titles like KMail/Kontact using Akonadi will make that more obvious.
@Kevin: okay. So my proposal is to start with agreeing on the adaptations in Nepomuk’s Message Ontology and SPARQL as query language. I assume we want to use Xesam’s mailinglist and IRC channels for that.
The actual IPC that will be used is still undecided, but if DBus is going to be used for interactive requests then I’m pretty sure we can easily agree on this as soon as the prototypes are made (a GSoc student is making a proto of this proposal, we’ll see how far he gets of course).
Note that we are not planning to implement a mini nor big IMAP server, nor are we planning to use any IMAPish at all between applications and the local service. We simply don’t think that IMAP is a good IPC for this. All interactive commands will simply be DBus methods instead. We don’t think DBus’s latency is high enough to need pipelining between local service and application. The latency per call is probably much lower than the amount of time needed to spin a mainloop once. So also lower than redrawing the UI. Pipelining between local service and the UI app sounds unnecessary to me for that reason.
For the streaming of the actual content of the MIME parts, I don’t know yet. My idea, as you might have noticed already, is either a named pipe or a unix socket. Not sure what you guys will use (apparently something that smells and looks like IMAP).
@foo: Tinymail is not a UI, Tinymail is a library for writing an E-mail client. You can use Tinymail to write an E-mail client like Sup. Pls don’t confuse “library” with “E-mail client”.
@pvanhoof
“So my proposal is to start with agreeing on the adaptations in Nepomuk’s Message Ontology and SPARQL as query language.”
You’ll have to talk to the Nepomuk guys for that.
Akonadi is basically a consumer of a SPARQL capable query service, we assume that the query service can return a list of URIs identifying the search results, ideally capable of live updates.
More or less a matter of developer convenience. The application developers could do the query themselves and then get whatever data they want from Akonadi. Doing it through Akonadi and getting the result as a nice virtual collection is probably a matter of taste.
The second approach has the advantage that results are accessed the same way independent whether the local or remote query service reports them.
“Note that we are not planning to implement a mini nor big IMAP server…”
Right, there are already enough IMAP servers available, no need to implement another one.
Makes it kind of funny to see this “urban ledgend” style rumor getting around so far ;)
“…nor are we planning to use any IMAPish at all between applications and the local service.”
Yeah, probably on overkill for a query service. We have to handle a lot of data and a lot of data formats, parallel requests and potential for out-of-order executions are quite nice there.
“My idea, as you might have noticed already, is either a named pipe or a unix socket. Not sure what you guys will use”
We use named pipes on Windows and Unix sockets on, well, Unix.
I have no idea what kind of transport is being used when making data accesible to Nepomuk right now (probably D-Bus), but assuming a similar approach in a system with Tracker as the query service (i.e. using feeder agents), we can adapt to whatever technology you’d like to get the data with.