[OE-core] sstate compression

[Richard Purdie]

On Wed, 2012-01-04 at 16:41 +0000, Phil Blundell wrote:
> On Wed, 2012-01-04 at 09:32 -0700, Chris Larson wrote:
> > Agreed. Sstate can get truly massive, being able to opt-into xz or
> > something would be lovely for us as well.
> 
> Yes, that would be nice.  xz does seem to compress nearly twice as well
> as "gzip -9", though it takes about six times as long to do it.  (16M
> and 55 seconds vs 28M and 9.5 seconds for my webkit testcase.)
> Personally I don't care about the disk space as much as the build time,
> but I can see that sstate could start to become quite unwieldy if you
> have a lot of packages in there.
> 
> Alternatively, maybe we could have sstate.bbclass accept multiple
> compression types when reading (i.e. look for .tar.xz, .tar.gz, .tar.lzo
> etc in turn), make it use the fastest reasonable compression when
> generating the archives in the first place, and then folks who want to
> either put them into long-term storage or send them over slow links can
> post-process the sstate-cache by transcoding them into .xz or whatever
> format.

Just to note that looking for multiple versions can cause a fair bit of
network traffic as for http:// mirror urls it will have to wget each in
turn. Better would be one file name and dynamic detection of the
compression format I guess.

Cheers,

Richard