You know about those guys that use your software against huge datasets like their entire filesystem, with thousands of files?
We do. His name is Tshepang Lekhonkhobe and we owe him a few beers for reporting to us many scalability issues.
Today we found and fixed such a scalability issue: the update query to reset the availability of file resources (this is for support for removable media) was causing at least a linear increase of VmRss usage per amount of file resources. For Tshepang’s situation that meant 600 MB of VmRss. Jürg reduced this to 30 MB of peak VmRss in the same use-case, and a performance improvement from minutes to a second or two, three. Without memory fragmentation as glibc is returning almost all of the VmRss back to the kernel.
Thursday is our usual release day. I invite all of the 0.7 pioneers to test us with your huge filesystems, just like Tshepang always does.
So long and thanks for all the testing, Tshepang! I’m glad we finally found it.