Camel (a library being used by Evolution and a heavily hacked version of it by tinymail) does something like this when you for the first time downloaded 30,000 message headers from an IMAP service’s folder:
The Camel in tinymail does something like this (after this patch, as I have to hack a few other providers to support this hack first):
Anyway, this hack together with the mmap one (which has already been applied to tinymail’s camel) makes it possible to use very large folders (more than 20,000 headers) on devices with ~10M RAM. Like maybe the cellphones of tomorrow?
How? Simple: I dumped the headers to the summary file that will be mmapped each 1000th header, store some status and continue fetching more headers from the IMAP server.
This also improves canceling. Because it now dumps each 1000th header, you can cancel the operation yet not lose per 1000th header the ones that were already received (like what happens if you would cancel such a download in Evolution).