[OE-core] [PATCH] bitbake: gitsm: download submodules

[Paul Barker]

On Wed, 4 Mar 2020 12:30:10 +0000
"Freihofer, Adrian" <adrian.freihofer at siemens.com> wrote:

> -----Original Message-----
> From: Paul Barker <pbarker at konsulko.com>
> To: "Freihofer, Adrian" <adrian.freihofer at siemens.com>
> Cc: openembedded-core at lists.openembedded.org <
> openembedded-core at lists.openembedded.org>
> Subject: Re: [OE-core] [PATCH] bitbake: gitsm: download submodules
> Date: Wed, 04 Mar 2020 09:59:44 +0000
> 
> On Wed, 4 Mar 2020 08:12:27 +0000
> "Freihofer, Adrian" <adrian.freihofer at siemens.com> wrote:
> 
> > The unpack function failed because the submodules were not
> > downloaded.
> > Calling download before unpack for each submodule solves this issue.
> > 
> > Signed-off-by: Adrian Freihofer <adrian.freihofer at siemens.com>
> > ---
> >  bitbake/lib/bb/fetch2/gitsm.py | 1 +
> >  1 file changed, 1 insertion(+)
> > 
> > diff --git a/bitbake/lib/bb/fetch2/gitsm.py
> > b/bitbake/lib/bb/fetch2/gitsm.py
> > index c622771d21..3715e9824f 100644
> > --- a/bitbake/lib/bb/fetch2/gitsm.py
> > +++ b/bitbake/lib/bb/fetch2/gitsm.py
> > @@ -184,6 +184,7 @@ class GitSM(Git):
> >  
> >              try:
> >                  newfetch = Fetch([url], d, cache=False)
> > +                newfetch.download()
> >                  newfetch.unpack(root=os.path.dirname(os.path.join(re
> > po
> > _conf, 'modules', module)))
> >              except Exception as e:
> >                  logger.error('gitsm: submodule unpack failed: %s %s'
> > %
> > (type(e).__name__, str(e)))  
> 
> You shouldn't be trying to download submodules in the do_unpack step.
> If
> they're missing at this stage it probably indicates a fetch issue.
> 
> Basically true, but the information about which submodules need to be
> downloaded is not available before the top level repo has been
> unpacked. I guess the solution must be something like:
> 
> fetch top-level
> unpack top-level
> foreach submodule:
>   fetch submodule
>   unpack submodule
> 
> What's the exact error that you're seeing?
> 
> Don't remeber exactly. But when I debugged the flow, it became obvious,
> that the submodules are not downloaded.
> 
> This could be related to the issue I saw when the fetcher uses git
> shallow
> tarballs from a mirror - if that's the case I'm planning to get that
> fixed
> this weekend.
> 
> Yes, the problem occurs with git shallows. The patch solves it with and
> without shallows for us.
> 
> Also, your email client seems to have chewed the patch up. It's best to
> send
> patches using `git send-email` only.
> 
> Sorry, we switched to a new e-mail solution. I have to adapt my setup.

The problem with your patch is that it would require network access during
do_unpack which then breaks our ability to do offline builds.

Do you only see this issue when fetching from git shallow mirror tarballs? If
so, the mirror tarballs need to be extracted during do_fetch but not placed
in the usual 'git2' directory (as future fetches can't cope with shallow
clones in that directory).

Fixing this also has some impact on archiver.bbclass as that also needs to be
able to walk the tree of submodules correctly.

-- 
Paul Barker
Konsulko Group