[OE-core] sstate compression

[McClintock Matthew-B29882]

On Wed, Jan 4, 2012 at 10:47 AM, Richard Purdie
<richard.purdie at linuxfoundation.org> wrote:
> On Wed, 2012-01-04 at 16:41 +0000, Phil Blundell wrote:
>> On Wed, 2012-01-04 at 09:32 -0700, Chris Larson wrote:
>> > Agreed. Sstate can get truly massive, being able to opt-into xz or
>> > something would be lovely for us as well.
>>
>> Yes, that would be nice.  xz does seem to compress nearly twice as well
>> as "gzip -9", though it takes about six times as long to do it.  (16M
>> and 55 seconds vs 28M and 9.5 seconds for my webkit testcase.)
>> Personally I don't care about the disk space as much as the build time,
>> but I can see that sstate could start to become quite unwieldy if you
>> have a lot of packages in there.
>>
>> Alternatively, maybe we could have sstate.bbclass accept multiple
>> compression types when reading (i.e. look for .tar.xz, .tar.gz, .tar.lzo
>> etc in turn), make it use the fastest reasonable compression when
>> generating the archives in the first place, and then folks who want to
>> either put them into long-term storage or send them over slow links can
>> post-process the sstate-cache by transcoding them into .xz or whatever
>> format.
>
> Just to note that looking for multiple versions can cause a fair bit of
> network traffic as for http:// mirror urls it will have to wget each in
> turn. Better would be one file name and dynamic detection of the
> compression format I guess.

The patch I submitted just looks for the type you have selected, so
there is no looking for gzip or xz sstate-cache. It just checks one,
like we currently do.

-M