Ticket #28 (assigned enhancement)

Opened 8 years ago

Last modified 6 years ago

Compressed caching to RAM

Reported by: jg Owned by: nitin
Priority: high Milestone: Future Release
Component: kernel Version:
Keywords: Cc: nitingupta910@…, mtd, mstone, ridderman, unk.nown@…, toyj@…, sayamindu, dsaxena, shenki
Action Needed: never set Verified: no
Deployments affected: Blocked By:
Blocking:

Description

Nitin Gupta, a Google Summer of Code student, is working on compressed caching, with Rik van Riel as supervisor.

If this gets mainlined into Linux and appears a net win, we'll want to incorporate it into our builds.

Change History

  Changed 8 years ago by krstic

  • cc nitingupta.mail@… added
  • priority changed from normal to high
  • milestone set to rev1 final

follow-up: ↓ 4   Changed 8 years ago by bluefoxicy

Nitin has stated on https://wiki.ubuntu.com/CompressedMemory:

"I will be glad to continue its development if provided funding by Ubuntu. In case of funding, I will make my best attempt to get it to mainline (which is the only possible way of keeping this project alive/meaningful). If you want any details on current implementation status and roadmap please contact me"

I believe there is benefit to getting this into mainline; among these is a broadened base of further research into this particular application of compression algorithms and improved usage of those algorithms. I believe that initially this will be a win, but a small one; continued development will improve the state of the art and increase the win.

Some limiting factors are the size of memory and the behavior of memory access. You can't just make all but 64M of RAM a compressed cache without compression thrashing (i.e. CPU activity, performance drop). You can probably get away with the top 16-32MiB, but at 40% compression this will give you 6.4-12.8MiB added effective memory (5-10% increase). Still, it's difficult to argue with what is effectively "free" (cheaper performance and hardware wise than swapping; better than running out of memory).

I encourage you to look to the future. The state of the art can only improve once this is polished and in mainline; and the OLPC will be revised into more powerful versions as feasible. In the future 256MiB of RAM will be feasible at very low cost, and the area devoted to compression will increase, probably adaptively by then. The effective gain of 5-10% would then be 12.8-25.6MiB, but the gains would likely be up around 10-20%.

  Changed 8 years ago by blizzard

  • owner changed from blizzard to marcelo

Over to Marcelo, the proper owner.

in reply to: ↑ 2   Changed 8 years ago by nitin

Replying to bluefoxicy:

Nitin has stated on https://wiki.ubuntu.com/CompressedMemory: "I will be glad to continue its development if provided funding by Ubuntu. In case of funding, I will make my best attempt to get it to mainline (which is the only possible way of keeping this project alive/meaningful). If you want any details on current implementation status and roadmap please contact me"

I no longer depend on funding but I am finding it very hard to get time for this project along with my current job. If anyone is considering continuing its development, I will be glad to help him out (Approx time to get it stable: 3 months).

  Changed 7 years ago by nitin

  • cc nitingupta910@… added; nitingupta.mail@… removed

  Changed 7 years ago by jg

  • milestone changed from First Deployment, V1.0 to V1.1

  Changed 7 years ago by jg

  • milestone changed from V1.1 to Future Release

  Changed 7 years ago by jg

  • owner changed from marcelo to jg

follow-up: ↓ 9   Changed 7 years ago by bluefoxicy

Nitin seems to have moved on to aiming for the embedded field, while the XO more closely relates to the desktop field. I am not sure the real impact of this change, but the project here will be slow unless someone knowledgeable about the kernel's memory systems gets together with him.

Another person has started ANOTHER memory compression project as part of Google's Summer of Code 2008 now. http://code.google.com/p/cclinux/

in reply to: ↑ 8   Changed 7 years ago by nitin

Replying to bluefoxicy:

Nitin seems to have moved on to aiming for the embedded field, while the XO more closely relates to the desktop field. I am not sure the real impact of this change, but the project here will be slow unless someone knowledgeable about the kernel's memory systems gets together with him.

The only difference in new development is that we no longer compress page-cache (filesystem backed) pages since reading from flash is fast. Since XO is also using flash for storage (I think), so this is applicable here also.

With this change, the code is significantly simplified and requires no kernel patching at all - hence more likely to go mainline.

FYI, new dev home: http://code.google.com/p/ccache/

Another person has started ANOTHER memory compression project as part of Google's Summer of Code 2008 now. http://code.google.com/p/cclinux/

I have talked to the developer of this project and it seems only purpose here is to learn Linux VM. We are also planning to merge these efforts later.

in reply to: ↑ description ; follow-up: ↓ 11   Changed 6 years ago by nitin

I have rewritten ccaching feature with new approach that is much cleaner and does not require any kernel patching. You can download it from: http://code.google.com/p/ccache/downloads/list

This mail gives more information: http://lists.laptop.org/pipermail/linux-mm-cc/2008-February/000195.html

I have tested ccache-0.1 on Linux kernel 2.6.23.x and 2.6.25-rc2 (x86 only) and is very stable.

in reply to: ↑ 10   Changed 6 years ago by nitin

Replying to nitin:

I have rewritten ccaching feature with new approach that is much cleaner and does not require any kernel patching. You can download it from: http://code.google.com/p/ccache/downloads/list

Project has now moved to: http://code.google.com/p/compcache/

This was done to avoid confusion with: http://ccache.samba.org/ which has nothing to do with this project.

  Changed 6 years ago by nitin

  • owner changed from jg to nitin
  • status changed from new to assigned

Just FYI, the project is now in release 0.3. In all testing by various people it worked without any problems with significant performance improvement. I does not require any kernel patching and all components are separate modules. So, its easy to evaluate on OLPC systems. I will also try to submit this to LKML for possible inclusion in mainline. Its available for download here:

http://code.google.com/p/compcache/downloads/list

- Nitin

  Changed 6 years ago by mtd

  • cc mtd added

  Changed 6 years ago by mtd

I'm using this now on joyride -- it seems to help in low memory situations. No idea about stability. Some initial reactions / installation instructions at http://www.xades.com

  Changed 6 years ago by mstone

  • cc mstone added

  Changed 6 years ago by mtd

So far it's pretty darn stable for my (not light, not heavy) usage pattern.

  Changed 6 years ago by ridderman

  • cc ridderman added

  Changed 6 years ago by mtd

  • next_action set to never set

Seems to work fine in olpc3/F9 joyride.

  Changed 6 years ago by mtd

  • cc unk.nown@… added

A new version - compcache-0.4 - has been released and seems to work fine.

If anyone is interested in trying this out, this part of my post-olpc-update script might be useful:

yum -y install make autotools gcc patch

if ! `grep -q compcache /etc/rc.local` ; then
    cd /home/olpc/src
    wget http://dev.laptop.org/~dilinger/{stable,testing}/kernel-devel-`uname -r`.i586.rpm
    rpm -ivh kernel-devel-`uname -r`.i586.rpm
    cp -af /boot/* /versions/boot/current/boot/
    wget http://compcache.googlecode.com/files/compcache-0.4.tar.gz
    tar xzf compcache-0.4.tar.gz
    cd compcache-0.4
    make
    ./use_compcache.sh
    echo "(cd /home/olpc/src/compcache-0.4 ; ./use_compcache.sh)" >> /etc/rc.local
fi

  Changed 6 years ago by mtd

  • cc toyj@… added

  Changed 6 years ago by sayamindu

  • cc sayamindu added

  Changed 6 years ago by dsaxena

  • cc dsaxena added

  Changed 6 years ago by shenki

  • cc shenki added
Note: See TracTickets for help on using tickets.