[OE-core] [PATCH 2/6] ccache.bbclass: Refactor it to make it more reliable

[Robert Yang]

On 12/12/18 2:59 PM, Robert Yang wrote:
> The previous ccache.bbclass has the following problems:
> - It uses host's ccache for native recipes, but this may not work on some
>    hosts, for example, it nerver works on my Ubuntu 14.04.4, there are always
>    build failures (m4-native failed at do_configure, and others will also be
>    failed if I disable CCACHE for m4-native)
> 
> - native/nativesdk/cross/crosssdk recipes use host's ccahe, but target uses
>    ccache-native, this may confuse user.
> 
> - The target recipes may use both host's ccache and ccache-native, this may
>    cause unexpected problems and hard to debug. This is because ccache-native is
>    in SIGGEN_EXCLUDE_SAFE_RECIPE_DEPS, so ccache-native may not be present when
>    rebuild target recipes, and then it would use hosttools/ccache, but the
>    previous ccache files were generated by ccache-native.
> 
> - Target recipes can't use ccache when no ccahe is installed on the host:
>    CCACHE = "${@bb.utils.which(d.getVar('PATH'), 'ccache') and 'ccache '}"
> 
> After refactored:
> All types recipes (native, target and others) will use ccache-native except
> ccache-native itself, host's cache won't be used any more. It is more
> reliable now, which will work everywhere when ccache-native can be built.
> 
> And now we need use "CCACHE_DISABLE = '1'" to disable ccahe for the recipe
> rather than "CCACHE = ''" since we set CCACHE in anonymous function, and
> d.getVar('CCACHE') works after "CCACHE ??=" which is set in bitbake.conf, so we
> can't check whether CCACHE is set or not in anonymous function since it is
> always set. Use CCACHE_DISABLE to disable it would be more clear.
> 
> Signed-off-by: Robert Yang <liezhi.yang at windriver.com>
> ---
>   meta/classes/ccache.bbclass | 25 ++++++++++++++++++++++---
>   meta/conf/bitbake.conf      |  6 +-----
>   meta/conf/layer.conf        |  1 -
>   meta/lib/oe/utils.py        |  3 ++-
>   4 files changed, 25 insertions(+), 10 deletions(-)
> 
> diff --git a/meta/classes/ccache.bbclass b/meta/classes/ccache.bbclass
> index 9609020..59e1022 100644
> --- a/meta/classes/ccache.bbclass
> +++ b/meta/classes/ccache.bbclass
> @@ -1,4 +1,14 @@
> -CCACHE = "${@bb.utils.which(d.getVar('PATH'), 'ccache') and 'ccache '}"
> +#
> +# Usage:
> +# - Enable ccache
> +#   Add the following line to a conffile such as conf/local.conf:
> +#   INHERIT += "ccache"
> +#
> +# - Disable ccache for a recipe
> +#   Add the following line to the recipe if it can't be built with ccache:
> +#   CCACHE_DISABLE = '1'
> +#
> +
>   export CCACHE_DIR ?= "${TMPDIR}/ccache/${MULTIMACH_TARGET_SYS}/${PN}"
>   
>   # We need to stop ccache considering the current directory or the
> @@ -7,5 +17,14 @@ export CCACHE_DIR ?= "${TMPDIR}/ccache/${MULTIMACH_TARGET_SYS}/${PN}"
>   # ${PV} or ${PR} change.
>   export CCACHE_NOHASHDIR ?= "1"
>   
> -DEPENDS_append_class-target = " ccache-native"
> -DEPENDS[vardepvalueexclude] = " ccache-native"
> +python() {
> +    """
> +    Enable ccache for the recipe
> +    """
> +    pn = d.getVar('PN')
> +    # quilt-native doesn't need ccache since no c files
> +    if not (pn in ('ccache-native', 'quilt-native') or
> +            bb.utils.to_boolean(d.getVar('CCACHE_DISABLE'))):
> +        d.appendVar('DEPENDS', ' ccache-native')
> +        d.setVar('CCACHE', 'ccache ')
> +}
> diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
> index 6480062..c88d66c 100644
> --- a/meta/conf/bitbake.conf
> +++ b/meta/conf/bitbake.conf
> @@ -492,7 +492,7 @@ HOSTTOOLS += " \
>   HOSTTOOLS += "${@'ip ping ps scp ssh stty' if (bb.utils.contains_any('IMAGE_CLASSES', 'testimage testsdk', True, False, d) or any(x in (d.getVar("BBINCLUDED") or "") for x in ["testimage.bbclass", "testsdk.bbclass"])) else ''}"
>   
>   # Link to these if present
> -HOSTTOOLS_NONFATAL += "aws ccache gcc-ar gpg ld.bfd ld.gold nc pigz sftp socat ssh sudo"
> +HOSTTOOLS_NONFATAL += "aws gcc-ar gpg ld.bfd ld.gold nc pigz sftp socat ssh sudo"
>   
>   # Temporary add few more detected in bitbake world
>   HOSTTOOLS_NONFATAL += "join nl size yes zcat"
> @@ -504,10 +504,6 @@ HOSTTOOLS_NONFATAL += "bzr"
>   HOSTTOOLS_NONFATAL += "scp"
>   
>   CCACHE ??= ""
> -# ccache < 3.1.10 will create CCACHE_DIR on startup even if disabled, and
> -# autogen sets HOME=/dev/null so in certain situations builds can fail.
> -# Explicitly export CCACHE_DIR until we can assume ccache >3.1.10 on the host.
> -export CCACHE_DIR ??= "${@os.getenv('HOME')}/.ccache"
>   
>   TOOLCHAIN_OPTIONS = " --sysroot=${STAGING_DIR_TARGET}"
>   
> diff --git a/meta/conf/layer.conf b/meta/conf/layer.conf
> index c67dec4..4141d6c 100644
> --- a/meta/conf/layer.conf
> +++ b/meta/conf/layer.conf
> @@ -49,7 +49,6 @@ SIGGEN_EXCLUDE_SAFE_RECIPE_DEPS += " \
>     *->quilt-native \
>     *->subversion-native \
>     *->git-native \
> -  *->ccache-native \
>     *->icecc-create-env-native \
>     gcc-cross-${TARGET_ARCH}->virtual/${TARGET_PREFIX}libc-for-gcc \
>     gcc-cross-${TARGET_ARCH}->linux-libc-headers \
> diff --git a/meta/lib/oe/utils.py b/meta/lib/oe/utils.py
> index 8a584d6..a64f16c 100644
> --- a/meta/lib/oe/utils.py
> +++ b/meta/lib/oe/utils.py
> @@ -362,7 +362,8 @@ def host_gcc_version(d, taskcontextonly=False):
>       if taskcontextonly and d.getVar('BB_WORKERCONTEXT') != '1':
>           return
>   
> -    compiler = d.getVar("BUILD_CC")
> +    # Get rid of ccache since it is not present when parsing.
> +    compiler = d.getVar("BUILD_CC").lstrip("ccache-")

Sorry, here should be:

     compiler = d.getVar("BUILD_CC")
     # Get rid of ccache since it is not present when parsing.
     if compiler.startswith('ccache '):
         compiler = compiler[7:]


I updated it in the repo.

// Robert

>       try:
>           env = os.environ.copy()
>           env["PATH"] = d.getVar("PATH")
>