[OE-core] [PATCH] bitbake: gitsm: download submodules

[Freihofer, Adrian]

On Thu, 2020-03-05 at 09:42 +0000, Paul Barker wrote:
> On Wed, 4 Mar 2020 12:30:10 +0000
> "Freihofer, Adrian" <adrian.freihofer at siemens.com> wrote:
> 
> > -----Original Message-----
> > From: Paul Barker <pbarker at konsulko.com>
> > To: "Freihofer, Adrian" <adrian.freihofer at siemens.com>
> > Cc: openembedded-core at lists.openembedded.org <
> > openembedded-core at lists.openembedded.org>
> > Subject: Re: [OE-core] [PATCH] bitbake: gitsm: download submodules
> > Date: Wed, 04 Mar 2020 09:59:44 +0000
> > 
> > On Wed, 4 Mar 2020 08:12:27 +0000
> > "Freihofer, Adrian" <adrian.freihofer at siemens.com> wrote:
> > 
> > > The unpack function failed because the submodules were not
> > > downloaded.
> > > Calling download before unpack for each submodule solves this
> > > issue.
> > > 
> > > Signed-off-by: Adrian Freihofer <adrian.freihofer at siemens.com>
> > > ---
> > >  bitbake/lib/bb/fetch2/gitsm.py | 1 +
> > >  1 file changed, 1 insertion(+)
> > > 
> > > diff --git a/bitbake/lib/bb/fetch2/gitsm.py
> > > b/bitbake/lib/bb/fetch2/gitsm.py
> > > index c622771d21..3715e9824f 100644
> > > --- a/bitbake/lib/bb/fetch2/gitsm.py
> > > +++ b/bitbake/lib/bb/fetch2/gitsm.py
> > > @@ -184,6 +184,7 @@ class GitSM(Git):
> > >  
> > >              try:
> > >                  newfetch = Fetch([url], d, cache=False)
> > > +                newfetch.download()
> > >                  newfetch.unpack(root=os.path.dirname(os.path.joi
> > > n(re
> > > po
> > > _conf, 'modules', module)))
> > >              except Exception as e:
> > >                  logger.error('gitsm: submodule unpack failed: %s
> > > %s'
> > > %
> > > (type(e).__name__, str(e)))  
> > 
> > You shouldn't be trying to download submodules in the do_unpack
> > step.
> > If
> > they're missing at this stage it probably indicates a fetch issue.
> > 
> > Basically true, but the information about which submodules need to
> > be
> > downloaded is not available before the top level repo has been
> > unpacked. I guess the solution must be something like:
> > 
> > fetch top-level
> > unpack top-level
> > foreach submodule:
> >   fetch submodule
> >   unpack submodule
> > 
> > What's the exact error that you're seeing?
> > 
> > Don't remeber exactly. But when I debugged the flow, it became
> > obvious,
> > that the submodules are not downloaded.
> > 
> > This could be related to the issue I saw when the fetcher uses git
> > shallow
> > tarballs from a mirror - if that's the case I'm planning to get
> > that
> > fixed
> > this weekend.
> > 
> > Yes, the problem occurs with git shallows. The patch solves it with
> > and
> > without shallows for us.
> > 
> > Also, your email client seems to have chewed the patch up. It's
> > best to
> > send
> > patches using `git send-email` only.
> > 
> > Sorry, we switched to a new e-mail solution. I have to adapt my
> > setup.
> 
> The problem with your patch is that it would require network access
> during
> do_unpack which then breaks our ability to do offline builds.
Fetching in the unpack task is probably really wrong, I agree.
However, it also works on our CI infrastructure which runs without
Internet connectivity.
> 
> Do you only see this issue when fetching from git shallow mirror
> tarballs?

On our CI which works with gitshallow archives, fetching did not work
at all. On machines with Internet access we also had problems, but I did not completely analyze what happened. The ovmf package from poky was causing the troubles in our case.

> If so, the mirror tarballs need to be extracted during do_fetch but
> not placed in the usual 'git2' directory (as future fetches can't
> cope with shallow clones in that directory).

Do you already have an idea how this could be implemented?

> 
> Fixing this also has some impact on archiver.bbclass as that also
> needs to be able to walk the tree of submodules correctly.
Yes.
>