[OE-core] Long delays with latest bitbake (was: [PATCH 1/7] insane.bbclass: in file-rdeps do not look into RDEPENDS recursively)

[Martin Jansa]

On Fri, Aug 16, 2019 at 04:54:48PM +0100, richard.purdie at linuxfoundation.org wrote:
> On Fri, 2019-08-16 at 17:00 +0200, Martin Jansa wrote:
> > > Will try to bump BB_NUMBER_THREADS from 8 to 72.
> > 
> > I've tried to remove icecc.bbclass inherit (because it does things
> > while parsing and RP probably doesn't have it inherited), but that
> > didn't save much time.
> > 
> > All 3 tests were with bitbake
> > 18d4a31fdcec1f0e5d2199d6142f0ce833fca1a7
> > 94m19.081s  8 BB_NUMBER_THREADS and icecc
> > 82m59.574s  8 BB_NUMBER_THREADS no icecc
> > 68m3.556s    72 BB_NUMBER_THREADS no icecc
> > 
> > Still don't know how to get to sub 10min world runs RP was seeing,
> > but at least it's as slow as it was before runqeueu changes (or even
> > a bit faster in lastest master).
> 
> Just thinking out loud, other things which can influence timings:
> 
> Is SSTATE_DIR on NFS or local disk?

SSTATE_DIR is empty directory on local disk
/dev/mapper/vg00-jenkins on /jenkins type ext4 (rw,noatime,nobarrier,commit=6000,stripe=128,data=ordered)

> Are sstate mirrors configured?

None, normally I have SSTATE_MIRRORS over sshfs in this case, but I've
removed it before any performance testing.

> Is there an existing build or not, if so, how much is valid?

Nothing, I remove whole TMPDIR and cache before each run. Then let it
recreate the cache before starting the profile:
bitbake -p; time bitbake world -P -n

> Underlying filesystem of the build?

ext4, everything is pretty much generic Ubuntu 18.04

There is plenty of ram, I'll try to test this from tmpfs as well.

> Your build seems especially slow at executing through the tasks which
> is effectively a test on how fast the system can fork() and return in
> some ways. It would be interesting to block dry-run on the server side,
> skip the fork and see how the numbers compare.

As discussed on IRC, it's slower than yours (8 minutes from 68), but the
most significant chunk of time is lost somewhere else.

> My build manages some parts of the tasklist faster than others, perhaps
> because there are more "free" tasks to execute at some points in the
> task graph than others.
> 
> Also, I have some patches which improve performance a bit which I'm
> still testing.

Thanks for all the work on this!

Cheers,
-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa at gmail.com
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 195 bytes
Desc: Digital signature
URL: <http://lists.openembedded.org/pipermail/openembedded-core/attachments/20190816/6573acdc/attachment.sig>