[bitbake-devel] [PATCH] layerindexlib: Fix parsing of recursive layer dependencies

[Mark Hatle]

Ping.  I haven't seen any feedback on this and it doesn't appear to have been
included in -next.

--Mark

On 8/1/19 3:36 PM, Mark Hatle wrote:
> [YOCTO #13447]
> 
> When running bitbake-layers layerindex-fetch from 'master', there is a
> circular dependency between meta-oe and meta-python.  This triggered a maximum
> recursion depth exception.
> 
> To fix the exception, as we walk down a branch (depth first search), we track
> the layers we've already seen.  If we are about to recurse into a layer we've
> already seen we report a warning and then stop recursion.
> 
> Signed-off-by: Mark Hatle <mark.hatle at windriver.com>
> ---
>  lib/layerindexlib/__init__.py | 21 +++++++++++++++++++--
>  1 file changed, 19 insertions(+), 2 deletions(-)
> 
> diff --git a/lib/layerindexlib/__init__.py b/lib/layerindexlib/__init__.py
> index d231cf6a..77196b40 100644
> --- a/lib/layerindexlib/__init__.py
> +++ b/lib/layerindexlib/__init__.py
> @@ -376,7 +376,7 @@ layerBranches set.  If not, they are effectively blank.'''
>                  invalid.append(name)
>  
>  
> -        def _resolve_dependencies(layerbranches, ignores, dependencies, invalid):
> +        def _resolve_dependencies(layerbranches, ignores, dependencies, invalid, processed=None):
>              for layerbranch in layerbranches:
>                  if ignores and layerbranch.layer.name in ignores:
>                      continue
> @@ -388,6 +388,13 @@ layerBranches set.  If not, they are effectively blank.'''
>                      if ignores and deplayerbranch.layer.name in ignores:
>                          continue
>  
> +                    # Since this is depth first, we need to know what we're currently processing
> +                    # in order to avoid infinite recursion on a loop.
> +                    if processed and deplayerbranch.layer.name in processed:
> +                        # We have found a recursion...
> +                        logger.warning('Circular layer dependency found: %s -> %s' % (processed, deplayerbranch.layer.name))
> +                        continue
> +
>                      # This little block is why we can't re-use the LayerIndexObj version,
>                      # we must be able to satisfy each dependencies across layer indexes and
>                      # use the layer index order for priority.  (r stands for replacement below)
> @@ -411,7 +418,17 @@ layerBranches set.  If not, they are effectively blank.'''
>  
>                      # New dependency, we need to resolve it now... depth-first
>                      if deplayerbranch.layer.name not in dependencies:
> -                        (dependencies, invalid) = _resolve_dependencies([deplayerbranch], ignores, dependencies, invalid)
> +                        # Avoid recursion on this branch.
> +                        # We copy so we don't end up polluting the depth-first branch with other
> +                        # branches.  Duplication between individual branches IS expected and
> +                        # handled by 'dependencies' processing.
> +                        if not processed:
> +                            local_processed = []
> +                        else:
> +                            local_processed = processed.copy()
> +                        local_processed.append(deplayerbranch.layer.name)
> +
> +                        (dependencies, invalid) = _resolve_dependencies([deplayerbranch], ignores, dependencies, invalid, local_processed)
>  
>                      if deplayerbranch.layer.name not in dependencies:
>                          dependencies[deplayerbranch.layer.name] = [deplayerbranch, layerdependency]
>