Opened 8 years ago

Last modified 6 years ago

#28 assigned enhancement

Compressed caching to RAM

Reported by: jg Owned by: nitin
Priority: high Milestone: Future Release
Component: kernel Version:
Keywords: Cc: nitingupta910@…, mtd, mstone, ridderman, unk.nown@…, toyj@…, sayamindu, dsaxena, shenki
Blocked By: Blocking:
Deployments affected: Action Needed: never set
Verified: no

Description

Nitin Gupta, a Google Summer of Code student, is working on compressed caching, with Rik van Riel as supervisor.

If this gets mainlined into Linux and appears a net win, we'll want to incorporate it into our builds.

Change History (24)

comment:1 Changed 8 years ago by krstic

  • Cc nitingupta.mail@… added
  • Milestone set to rev1 final
  • Priority changed from normal to high

comment:2 follow-up: Changed 8 years ago by bluefoxicy

Nitin has stated on https://wiki.ubuntu.com/CompressedMemory:

"I will be glad to continue its development if provided funding
by Ubuntu. In case of funding, I will make my best attempt to
get it to mainline (which is the only possible way of keeping
this project alive/meaningful). If you want any details on
current implementation status and roadmap please contact me"

I believe there is benefit to getting this into mainline; among
these is a broadened base of further research into this particular application of
compression algorithms and improved usage of those algorithms.
I believe that initially this will be a win, but a small one;
continued development will improve the state of the art and
increase the win.

Some limiting factors are the size of memory and the behavior
of memory access. You can't just make all but 64M of RAM a
compressed cache without compression thrashing (i.e. CPU activity,
performance drop). You can probably get away with the top
16-32MiB, but at 40% compression this will give you 6.4-12.8MiB
added effective memory (5-10% increase). Still, it's difficult
to argue with what is effectively "free" (cheaper performance and hardware wise than swapping; better than running out of memory).

I encourage you to look to the future. The state of the art can
only improve once this is polished and in mainline; and the OLPC
will be revised into more powerful versions as feasible. In the
future 256MiB of RAM will be feasible at very low cost, and the
area devoted to compression will increase, probably adaptively by
then. The effective gain of 5-10% would then be 12.8-25.6MiB,
but the gains would likely be up around 10-20%.

comment:3 Changed 8 years ago by blizzard

  • Owner changed from blizzard to marcelo

Over to Marcelo, the proper owner.

comment:4 in reply to: ↑ 2 Changed 8 years ago by nitin

Replying to bluefoxicy:

Nitin has stated on https://wiki.ubuntu.com/CompressedMemory:

"I will be glad to continue its development if provided funding
by Ubuntu. In case of funding, I will make my best attempt to
get it to mainline (which is the only possible way of keeping
this project alive/meaningful). If you want any details on
current implementation status and roadmap please contact me"

I no longer depend on funding but I am finding it very hard to get time for this project along with my current job. If anyone is considering continuing its development, I will be glad to help him out (Approx time to get it stable: 3 months).

comment:5 Changed 7 years ago by nitin

  • Cc nitingupta910@… added; nitingupta.mail@… removed

comment:6 Changed 7 years ago by jg

  • Milestone changed from First Deployment, V1.0 to V1.1

comment:7 Changed 7 years ago by jg

  • Milestone changed from V1.1 to Future Release

comment:7 Changed 7 years ago by jg

  • Owner changed from marcelo to jg

comment:8 follow-up: Changed 7 years ago by bluefoxicy

Nitin seems to have moved on to aiming for the embedded field, while the XO more closely relates to the desktop field. I am not sure the real impact of this change, but the project here will be slow unless someone knowledgeable about the kernel's memory systems gets together with him.

Another person has started ANOTHER memory compression project as part of Google's Summer of Code 2008 now. http://code.google.com/p/cclinux/

comment:9 in reply to: ↑ 8 Changed 7 years ago by nitin

Replying to bluefoxicy:

Nitin seems to have moved on to aiming for the embedded field, while the XO more closely relates to the desktop field. I am not sure the real impact of this change, but the project here will be slow unless someone knowledgeable about the kernel's memory systems gets together with him.

The only difference in new development is that we no longer compress page-cache (filesystem backed) pages since reading from flash is fast. Since XO is also using flash for storage (I think), so this is applicable here also.

With this change, the code is significantly simplified and requires no kernel patching at all - hence more likely to go mainline.

FYI, new dev home: http://code.google.com/p/ccache/

Another person has started ANOTHER memory compression project as part of Google's Summer of Code 2008 now. http://code.google.com/p/cclinux/

I have talked to the developer of this project and it seems only purpose here is to learn Linux VM. We are also planning to merge these efforts later.

comment:10 in reply to: ↑ description ; follow-up: Changed 7 years ago by nitin

I have rewritten ccaching feature with new approach that is much cleaner and does not require any kernel patching. You can download it from: http://code.google.com/p/ccache/downloads/list

This mail gives more information:
http://lists.laptop.org/pipermail/linux-mm-cc/2008-February/000195.html

I have tested ccache-0.1 on Linux kernel 2.6.23.x and 2.6.25-rc2 (x86 only) and is very stable.

comment:11 in reply to: ↑ 10 Changed 7 years ago by nitin

Replying to nitin:

I have rewritten ccaching feature with new approach that is much cleaner and does not require any kernel patching. You can download it from: http://code.google.com/p/ccache/downloads/list

Project has now moved to: http://code.google.com/p/compcache/

This was done to avoid confusion with: http://ccache.samba.org/ which has nothing to do with this project.

comment:12 Changed 7 years ago by nitin

  • Owner changed from jg to nitin
  • Status changed from new to assigned

Just FYI, the project is now in release 0.3. In all testing by various people it worked without any problems with significant performance improvement. I does not require any kernel patching and all components are separate modules. So, its easy to evaluate on OLPC systems. I will also try to submit this to LKML for possible inclusion in mainline. Its available for download here:

http://code.google.com/p/compcache/downloads/list

  • Nitin

comment:13 Changed 7 years ago by mtd

  • Cc mtd added

comment:14 Changed 7 years ago by mtd

I'm using this now on joyride -- it seems to help in low memory situations. No idea about stability. Some initial reactions / installation instructions at http://www.xades.com

comment:15 Changed 7 years ago by mstone

  • Cc mstone added

comment:16 Changed 7 years ago by mtd

So far it's pretty darn stable for my (not light, not heavy) usage pattern.

comment:17 Changed 7 years ago by ridderman

  • Cc ridderman added

comment:18 Changed 6 years ago by mtd

  • Action Needed set to never set

Seems to work fine in olpc3/F9 joyride.

comment:19 Changed 6 years ago by mtd

  • Cc unk.nown@… added

A new version - compcache-0.4 - has been released and seems to work fine.

If anyone is interested in trying this out, this part of my post-olpc-update script might be useful:

yum -y install make autotools gcc patch

if ! `grep -q compcache /etc/rc.local` ; then
    cd /home/olpc/src
    wget http://dev.laptop.org/~dilinger/{stable,testing}/kernel-devel-`uname -r`.i586.rpm
    rpm -ivh kernel-devel-`uname -r`.i586.rpm
    cp -af /boot/* /versions/boot/current/boot/
    wget http://compcache.googlecode.com/files/compcache-0.4.tar.gz
    tar xzf compcache-0.4.tar.gz
    cd compcache-0.4
    make
    ./use_compcache.sh
    echo "(cd /home/olpc/src/compcache-0.4 ; ./use_compcache.sh)" >> /etc/rc.local
fi

comment:20 Changed 6 years ago by mtd

  • Cc toyj@… added

comment:21 Changed 6 years ago by sayamindu

  • Cc sayamindu added

comment:22 Changed 6 years ago by dsaxena

  • Cc dsaxena added

comment:23 Changed 6 years ago by shenki

  • Cc shenki added
Note: See TracTickets for help on using tickets.