Compressed caching to RAM

[jg]

Nitin Gupta, a Google Summer of Code student, is working on compressed caching, with Rik van Riel as supervisor.

If this gets mainlined into Linux and appears a net win, we'll want to incorporate it into our builds.

[bluefoxicy]

Nitin has stated on https://wiki.ubuntu.com/CompressedMemory:

"I will be glad to continue its development if provided funding by Ubuntu. In case of funding, I will make my best attempt to get it to mainline (which is the only possible way of keeping this project alive/meaningful). If you want any details on current implementation status and roadmap please contact me"

I believe there is benefit to getting this into mainline; among these is a broadened base of further research into this particular application of compression algorithms and improved usage of those algorithms. I believe that initially this will be a win, but a small one; continued development will improve the state of the art and increase the win.

Some limiting factors are the size of memory and the behavior of memory access. You can't just make all but 64M of RAM a compressed cache without compression thrashing (i.e. CPU activity, performance drop). You can probably get away with the top 16-32MiB, but at 40% compression this will give you 6.4-12.8MiB added effective memory (5-10% increase). Still, it's difficult to argue with what is effectively "free" (cheaper performance and hardware wise than swapping; better than running out of memory).

I encourage you to look to the future. The state of the art can only improve once this is polished and in mainline; and the OLPC will be revised into more powerful versions as feasible. In the future 256MiB of RAM will be feasible at very low cost, and the area devoted to compression will increase, probably adaptively by then. The effective gain of 5-10% would then be 12.8-25.6MiB, but the gains would likely be up around 10-20%.

[blizzard]

Over to Marcelo, the proper owner.

[nitin]

Replying to bluefoxicy:

Nitin has stated on https://wiki.ubuntu.com/CompressedMemory: "I will be glad to continue its development if provided funding by Ubuntu. In case of funding, I will make my best attempt to get it to mainline (which is the only possible way of keeping this project alive/meaningful). If you want any details on current implementation status and roadmap please contact me"

I no longer depend on funding but I am finding it very hard to get time for this project along with my current job. If anyone is considering continuing its development, I will be glad to help him out (Approx time to get it stable: 3 months).

[bluefoxicy]

Nitin seems to have moved on to aiming for the embedded field, while the XO more closely relates to the desktop field. I am not sure the real impact of this change, but the project here will be slow unless someone knowledgeable about the kernel's memory systems gets together with him.

Another person has started ANOTHER memory compression project as part of Google's Summer of Code 2008 now. http://code.google.com/p/cclinux/

[nitin]

Replying to bluefoxicy:

Nitin seems to have moved on to aiming for the embedded field, while the XO more closely relates to the desktop field. I am not sure the real impact of this change, but the project here will be slow unless someone knowledgeable about the kernel's memory systems gets together with him.

The only difference in new development is that we no longer compress page-cache (filesystem backed) pages since reading from flash is fast. Since XO is also using flash for storage (I think), so this is applicable here also.

With this change, the code is significantly simplified and requires no kernel patching at all - hence more likely to go mainline.

FYI, new dev home: http://code.google.com/p/ccache/

Another person has started ANOTHER memory compression project as part of Google's Summer of Code 2008 now. http://code.google.com/p/cclinux/

I have talked to the developer of this project and it seems only purpose here is to learn Linux VM. We are also planning to merge these efforts later.

[nitin]

I have rewritten ccaching feature with new approach that is much cleaner and does not require any kernel patching. You can download it from: http://code.google.com/p/ccache/downloads/list

This mail gives more information: http://lists.laptop.org/pipermail/linux-mm-cc/2008-February/000195.html

I have tested ccache-0.1 on Linux kernel 2.6.23.x and 2.6.25-rc2 (x86 only) and is very stable.

[nitin]

Replying to nitin:

I have rewritten ccaching feature with new approach that is much cleaner and does not require any kernel patching. You can download it from: http://code.google.com/p/ccache/downloads/list

Project has now moved to: http://code.google.com/p/compcache/

This was done to avoid confusion with: http://ccache.samba.org/ which has nothing to do with this project.

[nitin]

Just FYI, the project is now in release 0.3. In all testing by various people it worked without any problems with significant performance improvement. I does not require any kernel patching and all components are separate modules. So, its easy to evaluate on OLPC systems. I will also try to submit this to LKML for possible inclusion in mainline. Its available for download here:

http://code.google.com/p/compcache/downloads/list

- Nitin

[mtd]

I'm using this now on joyride -- it seems to help in low memory situations. No idea about stability. Some initial reactions / installation instructions at http://www.xades.com

[mtd]

So far it's pretty darn stable for my (not light, not heavy) usage pattern.

[mtd]

Seems to work fine in olpc3/F9 joyride.

[mtd]

A new version - compcache-0.4 - has been released and seems to work fine.

If anyone is interested in trying this out, this part of my post-olpc-update script might be useful:

yum -y install make autotools gcc patch

if ! `grep -q compcache /etc/rc.local` ; then
    cd /home/olpc/src
    wget http://dev.laptop.org/~dilinger/{stable,testing}/kernel-devel-`uname -r`.i586.rpm
    rpm -ivh kernel-devel-`uname -r`.i586.rpm
    cp -af /boot/* /versions/boot/current/boot/
    wget http://compcache.googlecode.com/files/compcache-0.4.tar.gz
    tar xzf compcache-0.4.tar.gz
    cd compcache-0.4
    make
    ./use_compcache.sh
    echo "(cd /home/olpc/src/compcache-0.4 ; ./use_compcache.sh)" >> /etc/rc.local
fi