[OE-core] [PATCH 2/4] ccache.bbclass: Refactor it to make it more reliable

Tue Dec 11 09:35:14 UTC 2018

On 12/11/18 12:16 AM, Richard Purdie wrote:
> On Sun, 2018-12-09 at 18:24 -0800, Robert Yang wrote:
>> The previous ccache.bbclass has the following problems:
>> - It uses host's ccache for native recipes, but this may not work on some
>>    hosts, for example, it nerver works on my Ubuntu 14.04.4, there are always
>>    build failures (m4-native failed at do_configure, and others will also be
>>    failed if I disable CCACHE for m4-native)
>>
>> - native/nativesdk/cross/crosssdk recipes use host's ccahe, but target uses
>>    ccache-native, this may confuse user.
>>
>> - The target recipes may use both host's ccache and ccache-native, this may
>>    cause unexpected problems and hard to debug. This is because ccache-native is
>>    in SIGGEN_EXCLUDE_SAFE_RECIPE_DEPS, so ccache-native may not be present when
>>    rebuild target recipes, and then it would use hosttools/ccache, but the
>>    previous ccache files were generated by ccache-native.
>>
>> - Target recipes can't use ccache when no ccahe is installed on the host:
>>    CCACHE = "${@bb.utils.which(d.getVar('PATH'), 'ccache') and 'ccache '}"
>>
>> After refactored:
>> All types recipes (native, target and others) will use ccache-native except
>> ccache-native's dependencies, host's cache won't be used any more. It is more
>> reliable now, which will work everywhere when ccache-native can be built.
>>
>> And now we need use "CCACHE_DISABLE = '1'" to disable ccahe for the recipe
>> rather than "CCACHE = ''" since we set CCACHE in anonymous function, and
>> d.getVar('CCACHE') works after "CCACHE ??=" which is set in bitbake.conf, so we
>> can't check whether CCACHE is set or not in anonymous function since it is
>> always set. Use CCACHE_DISABLE to disable it would be more clear.
>>
>> Signed-off-by: Robert Yang <liezhi.yang at windriver.com>
>> ---
>>   meta/classes/ccache.bbclass             | 25 ++++++++++++++++++++++---
>>   meta/conf/bitbake.conf                  | 21 ++++++++++++++++-----
>>   meta/conf/layer.conf                    |  1 -
>>   meta/lib/oe/utils.py                    |  3 +++
>>   meta/recipes-devtools/ccache/ccache.inc |  2 ++
>>   5 files changed, 43 insertions(+), 9 deletions(-)
>>
>> diff --git a/meta/classes/ccache.bbclass b/meta/classes/ccache.bbclass
>> index 9609020..f4eefce 100644
>> --- a/meta/classes/ccache.bbclass
>> +++ b/meta/classes/ccache.bbclass
>> @@ -1,4 +1,14 @@
>> -CCACHE = "${@bb.utils.which(d.getVar('PATH'), 'ccache') and 'ccache '}"
>> +#
>> +# Usage:
>> +# - Enable ccache
>> +#   Add the following line to a conffile such as conf/local.conf:
>> +#   INHERIT += "ccache"
>> +#
>> +# - Disable ccache for a recipe
>> +#   Add the following line to the recipe if it can't be built with ccache:
>> +#   CCACHE_DISABLE = '1'
>> +#
>> +
>>   export CCACHE_DIR ?= "${TMPDIR}/ccache/${MULTIMACH_TARGET_SYS}/${PN}"
>>   
>>   # We need to stop ccache considering the current directory or the
>> @@ -7,5 +17,14 @@ export CCACHE_DIR ?= "${TMPDIR}/ccache/${MULTIMACH_TARGET_SYS}/${PN}"
>>   # ${PV} or ${PR} change.
>>   export CCACHE_NOHASHDIR ?= "1"
>>   
>> -DEPENDS_append_class-target = " ccache-native"
>> -DEPENDS[vardepvalueexclude] = " ccache-native"
>> +python() {
>> +    """
>> +    Enable ccache for the recipe if is not a dependency of ccache-native
>> +    """
>> +    ccache_native_deps = d.getVar('CCACHE_NATIVE_DEPENDS') or ''
>> +    ccache_native_deps += 'ccache-native'
>> +    if not (d.getVar('PN') in ccache_native_deps.split() or \
>> +            bb.utils.to_boolean(d.getVar('CCACHE_DISABLE'))):
>> +        d.appendVar('DEPENDS', ' ccache-native')
>> +        d.setVar('CCACHE', 'ccache ')
>> +}
>> diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
>> index 6480062..cb1dba8 100644
>> --- a/meta/conf/bitbake.conf
>> +++ b/meta/conf/bitbake.conf
>> @@ -492,7 +492,7 @@ HOSTTOOLS += " \
>>   HOSTTOOLS += "${@'ip ping ps scp ssh stty' if (bb.utils.contains_any('IMAGE_CLASSES', 'testimage testsdk', True, False, d) or any(x in (d.getVar("BBINCLUDED") or "") for x in ["testimage.bbclass", "testsdk.bbclass"])) else ''}"
>>   
>>   # Link to these if present
>> -HOSTTOOLS_NONFATAL += "aws ccache gcc-ar gpg ld.bfd ld.gold nc pigz sftp socat ssh sudo"
>> +HOSTTOOLS_NONFATAL += "aws gcc-ar gpg ld.bfd ld.gold nc pigz sftp socat ssh sudo"
>>   
>>   # Temporary add few more detected in bitbake world
>>   HOSTTOOLS_NONFATAL += "join nl size yes zcat"
>> @@ -503,11 +503,22 @@ HOSTTOOLS_NONFATAL += "bzr"
>>   # Used by ssh fetcher
>>   HOSTTOOLS_NONFATAL += "scp"
>>   
>> +# This is used by both ccache.bbclass and ccahe-native recipe, these
>> +# recipes can't use ccache since ccache-native depends on them,
>> +# otherwise, there would be loop dependencies.
>> +CCACHE_NATIVE_DEPENDS ?= " \
>> +    gnu-config-native \
>> +    autoconf-native \
>> +    quilt-native \
>> +    zlib-native \
>> +    gettext-minimal-native \
>> +    texinfo-dummy-native \
>> +    m4-native \
>> +    libtool-native \
>> +    automake-native \
>> +    xz-native \
>> +"
>>   CCACHE ??= ""
>> -# ccache < 3.1.10 will create CCACHE_DIR on startup even if disabled, and
>> -# autogen sets HOME=/dev/null so in certain situations builds can fail.
>> -# Explicitly export CCACHE_DIR until we can assume ccache >3.1.10 on the host.
>> -export CCACHE_DIR ??= "${@os.getenv('HOME')}/.ccache"
> 
> Hardcoding a list of ccache-native dependencies is horrible and I do
> not want to do that.

Yes, I don't like this either:-). I've fixed it as:

* Remove all ccache-native's dependencies.
   ccache-native has two dependencies: zlib-native and autotools-native, we can
   use --with-bundled-zlib do get rid of zlib-native. And use
   INHIBIT_AUTOTOOLS_DEPS to get rid of autotools-native, and add a simple
   do_configure function:

do_configure_class-native() {
     oe_runconf
}

So that ccache-native can be depended by any recipe except quilt-native:

So now the code in ccache.bbclass is very simple:
python() {
     """
     Enable ccache for the recipe
     """
     pn = d.getVar('PN')
     if not pn in ('quilt-native', 'ccache-native'):
             d.appendVar('DEPENDS', ' ccache-native')
             d.setVar('CCACHE', 'ccache ')
}

We can also set PATCHTOOL = "patch" in ccache to make quilt-native can use
ccache too, but I don't think that we have too.

I will send a V2 after more testing.

// Robert

> 
> Also, I want to stop the build up of unneeded 'stuff' in bitbake.conf.
> We should start making better use of inc files.
> 
> Please also ensure that the oe-selftest ccache tests continue to work
> after these changes. I haven't checked specifically here but we've had
> a lot of trouble with that before.
> 
> Cheers,
> 
> Richard
>   
> 
> 