I already E-mailed the author of the Maildir standard about this, but I didn’t get a reply. Other people who asked the same question told me they got the same reaction: none. Of course I don’t know for sure whether they actually tried.
Nowadays people use FAT32 a lot for storing their data on flash disk and USB sticks. They buy these at a computer shop or supermarket, plug it in their mobile device and they expect it to just work. They don’t want us to require them to reformat the device’s partition. Quite often do these people already have actual (thus, important) data on those storage devices.
I know that some people live in a imaginary Utopical world where FAT32 is a piece of shit filesystem where we not only puke on it, we also ignore it and religiously hate its authors. The problem is that I live in the realistic and pragmatic world and that I ignore people who try to force me to live in their imaginary Utopical worlds. I really have no time for the religious anti Microsoft rebellions who want me to become one of their crusaders.
In stead, I try to use the time nature has given to me to do things that can actually be done. You see, trying to get Microsoft to support the ‘:’ character on FAT32 is not among those goals of mine. Although, sure, I too don’t know why they don’t support it. I too dislike the fact that they don’t support it. I too wish live was different. In fact, I think I remember that I wanted to be born as an ant with gigantic colourful butterfly wings living in a delicious and fresh cow .. shed. In stead, they created me as a normal human.
It’s not fair!
Nevertheless, this is what happened.
So I noticed that one person decided to ignore the Maildir standard and replace the vicious ‘:’ character with ‘!’. I propose to make the Maildir standard more meaningful by changing this ‘:’ character to ‘!’. This way we can allow our users to put the E-mail data on standard USB sticks, flash cards and other media that comes with the evil, satanic, bad, ugly and patented FAT32 partition type.
I know people will tell me to use a different format for FAT32 partitions. Well, Maildir is not really bad for slow storages and it gives you easy atomic operations: rename() is, as far as I know, atomic on all (and if not most) platforms. The kernel is good at doing its own indexing for filesystems, so it’s not slow either. The idea behind Maildir is not a bad one. Most of the other local formats are kinda broken in different ways too. I could create my own standard, of course, but then I’ll get that other group of anti NIH religious rebellions who’ll start armed protest marches in front of my house.
First of all: I agree that Maildir/ is a good format. As far as I’m concerned, it’s the only relevant format for storing email and the only one I’ve so far not managed to corrupt completely in my perfectly simple setup: I am allergic to complicated things like IMAP. I like simple things like NFS.
You’ll never get Microsoft to change FAT32. But I doubt there’s something to stop you from fixing it on relevant operating systems? What does the Microsoft implementation do when it finds a filename with a : in it? I’m aware of the fact that you’ve not got the source (why people will insist on buying products without source is beyond me, but that aside) but I’m sure it can’t be too difficult to fabricate a test? Presumably, you merely want to support the filesystem because people are too lazy to format sticks (point!) and you don’t expect things to actually work on the Microsoft implementation (hopeless)? If it doesn’t completely (further) clobber Microsoft setups, just go for it. Par for the course and all that.
Most USB sticks these days come with wear-leveling magic that’s particularly FAT32-oriented. A nice way to take advantage of this, and to work around people not wanting to format their sticks with a sensible filesystem would be to put a vnode-backed filesystem inside the FAT32. This complicates direct access to the Maildir/ (which incidently means people don’t fiddle with it directly and whine when they break it) but it gets around the limitations of FAT32 in a rather elegant way and it allows you to keep the advantage of the wear-levelling. For bonus points, you can add compression (email compresses rather nicely) and encryption without effort. You might have heard that in the FAT-based world, people attach significance to the last three characters of the filename, so you could do MAIL.DIR for a normal vnode-backed ext2 Maildir, MAILD.IRZ for a compressed one and MAILD.IRC for encrypted.
Trying to change the Maildir/ spec is fighting windmills, I think. DJB is one of those people than himself can have relevant ideas. Most Maildir/ implementations I’ve encountered seem to deal fine with any ‘weird but not :’ character as flags sepparator though. Admittedly, my exposure is limited.
There are multiple problems with putting a binary like that on a FAT32 partition:
Although it circumvents this limitation, it also circumvents one of the reasons why Maildir has been so successfully for your case: it can’t easily get corrupted. A single (large) file would (could) if it gets a single defect, make your entire E-mail store unreadable. Notice that a lot of people will without correcetly unmounting the device first, unplug it. Each time they’ll do that, the filesystem in that big file can easily be corrupted. Whereas with single-file-per-E-mail concept, it’s far more likely that at least some got written out correctly to the storage device. It’s even likely that the majority got written correctly. Regretfully does your proposal therefore make it effectively less robust.
Another reason why Maildir is a successful format is that it’s easy to move its content to other solutions (other devices, other E-mail clients, other E-mail servers). There’s no (or not much) converting needed. Well, of course now there is a small converting step needed: renaming the files from ‘!’ to ‘:’ ones :-(. Although I agree that mounting that big file on a UNIX-like, like Linux, with a loopback device is not difficult for migration; using a tool like an FTP client or WinSCP is still more easy for the typical Windows administrator, though.
The rename-for-migrating-the-content problem is the one I’d like to address here. Technically I indeed don’t mind that I’ve implemented it with ‘!’ rather than ‘:’. It’s the exact same code, really (with one byte different in both the source and binary).
Other than all that is the implementation you propose a lot more difficult to technically achieve than to just cope with changing the ‘:’ character to ‘!’, including a tool for all imaginable platforms to convert the filenames from one standard to another.
Correct: I forgot about the case of people blindly unplugging filesystems. Isn’t it amazing how as technology advances, people seem to want to regress? Not only is formatting a stick to difficult but even simply unmounting it! But I digress… You’ve reminded me why userspace scares me. :-)
You’re quite right that the mail-per-file paradigm needs to stay. So you’re indeed pretty much stuck with “some flags sepparator which doesn’t break anything else” (remember that ‘!’ is the event selector in many Unix shells so you need to escape it when accessing that way ;-)). I just instinctively feel it’s “Wrong[tm]” to change a standard to work around a troublesome environment. Not even a truoblesome implementation either. Must be my upbringing, I’m sure. :-)
Compared to what I’ve suggested, changing the filenames is indeed much easier. Go for it. :-) I think it will be easier to approach the implementors of Maildir/ than the author. More interestingly, I think you’ll find most implementors have already done what you’ve proposed one way or another. Either taking ! or other ‘abnormal characters’ as flag sepparators on read or providing a fairly simple option of setting a non-: sepparator. Though I’ll bet qmail won’t play nice.
Wouldn’t it be nice to just fix FAT? Or to get factories to format sticks with jffs2. *dream*
Well, if I could pick what stick-vendors ship with, I’d vote for LogFS over jffs2.
This why I’m indeed trying to contact the implementors of E-mail solutions that utilize either the Maildir or the Maildir++ format to be aware that some E-mail clients (or, solutions) must use another character for the ‘:’ one. And if we are going to cope with those, let’s standardize on ‘!’ in stead of all picking our own favorite character.
Which of course would complicate the implementations and make multiple ones incompatible with each other.
So, if you know somebody who wrote an E-mail client or server, or some sort of script that copes with a Maildir, a spam filter thingy or a replacement for Maildrop (or Maildrop itself): please let him join this discussion (or at least tell him about this problem).
your blog login system is highly annoying.
i think the correct solution to your problem is to change the vfat filesystem driver in the linux kernel to replace : ? * and all the other “special[1]” characters with something like an underscore. make open() of a file with a “:” in the name search for one with “_”.
then make your maildir implementation write : but tolerate reading back a _ in a directory listing.
this would fix the general problem that we have when trying to copy files containing “special[1]” characters over to vfat…
[1] seriously… wtf?
Hmm, convincing the Linux kernel folks on this is going to be hard … and it ain’t fixes the problem for the day when I’ll port my library to WinCE.
2 remarks:
– no need to convince the linux devs: you can trivially make a fuse module that stores ‘:’ chars in filenames as another character. If you look in the existing fuse filesystems, you’ll find quite a lot of similar examples (case-changing fs, read-only fs, etc etc)
– since you probably won’t hav any luck with the author of Maildir, you might try and contact the Maildir++ guy. Although I doubt that they’ll be willing to break compatibility in that way. (And I also remember you mentioning you didn’t like courier-imap, don’t know why though)
o. FUSE is not available on most mobile devices
o. Well, because Courier is not an IMAP server but an SMAP server (Sam’s Message Access Protocol). A server that promises to support the IMAP protocol but does certain things completely different, should not clutter the IMAP space or at least be hounest about it and not call itself IMAP.
The author or Courier frequently disagrees with the people who define the IMAP standard. In stead of giving his view on this during the draft stage of the proposed RFC, he rarely wants to discuss the problems and ends up implementing his certain pieces about the protocol the way he thinks it should be.
Well, as a E-mail client developer my opinion is that either he implements IMAP and calls his server IMAP, or he doesn’t call his server IMAP and does what the hack he wants (but DON’T call it IMAP then).
I would indeed really love to meet this guy and explain him how many hours or frustration he created for me with his own point of view on IMAP.
In my opinion should people who want to use Courier be aware that they are NOT using IMAP, but another protocol invented by the author of Courier. Those people should NOT expect things to work with a normal IMAP client.
Regretfully, it doesn’t work that way. The way it works is that the E-mail client developer gets all the blame.
See the problem is that a lot of people somehow decided they want to use it as their IMAP server. I can’t imagine why, though.
Also, the things that he did do different make everything for the E-mail client developer A LOT more difficult. Not only does Courier return incorrect IMAP responses, it also behaves incorrect and inconsistent.
Some versions have broken mime parsers, some versions give broken BODYSTRUCTURE reponses, some versions do funny things with the UIDs, some versions don’t respect the UIDNEXT standard, some versions wont unsubscribe a mailbox that you just deleted and keep the non-existing folder in the LSUB list.
Other versions have significant security problems. And yet other versions make the imapd process hang if just selecting a very large folder. But those are not so much my problem, as a client developer, really. Things become my problem if I end up having to implement two or three versions of a method, just because one guy didn’t agree with the standard. This is absolutely NOT making it more easy for the E-mail client developer.
I don’t agree with all that you can find in IMAP’s RFC either. It wouldn’t make a lot of sense if I’d ignore it and do my E-mail client’s IMAP communication the way I wanted it, right? Well, so how does this make sense for Courier then? In my opinion should a server respect the standard even more than the client. Well, both must respect the standard, of course. That’s the point of a protocol, right?
That’s very interesting. I can tell you why I decided to not only use courier-imap but also courier-mta on my mail server: it’s very easy to set up, partly because it has very good defaults, and pretty well integrated.
In Debian it’s literally apt-get install courier-imap and that’s it – no configuration needed.
Compared to the hell I had to go through with Cyrus and the weird and funky issues I encountered (messages duplicating for no good reason, to the point they even had a duplicate message detection feature (wtf?), etc.) courier was a lot less hassle.
That being said courier-imap has serious problems too: the problems it has with huge folders are not unfamiliar to me. A folder I have with >200000 messages is barely usable. TLS/SSL behaviour breakage happened more than once, and then there’s the eternal all folders are subfolders of INBOX annoyance.
So, do you have a recommendation for an IMAP server that uses Maildir? I remember trying Bincimap and Dovecot at some point long time ago but I believe they were a bit too immature at that point. What would you use?
Dovecot is very easy to setup and behaves a lot better than Courier. Dovecot also uses Maildir++ so you can migrate with few hassle from Courier.
Due to the Thunderbird developpers fixation on windows, the only hope to get maildir support in Thunderbird is to define a convention that works as-is under Windows, so all the fuse/vnode/etc tricks are out
https://bugzilla.mozilla.org/show_bug.cgi?id=58308