[OE-core] Long delays with latest bitbake (was: [PATCH 1/7] insane.bbclass: in file-rdeps do not look into RDEPENDS recursively)

[Martin Jansa]

On Fri, Aug 16, 2019 at 07:22:55PM +0200, Martin Jansa wrote:
> On Fri, Aug 16, 2019 at 04:54:48PM +0100, richard.purdie at linuxfoundation.org wrote:
> > On Fri, 2019-08-16 at 17:00 +0200, Martin Jansa wrote:
> > > > Will try to bump BB_NUMBER_THREADS from 8 to 72.
> > > 
> > > I've tried to remove icecc.bbclass inherit (because it does things
> > > while parsing and RP probably doesn't have it inherited), but that
> > > didn't save much time.
> > > 
> > > All 3 tests were with bitbake
> > > 18d4a31fdcec1f0e5d2199d6142f0ce833fca1a7
> > > 94m19.081s  8 BB_NUMBER_THREADS and icecc
> > > 82m59.574s  8 BB_NUMBER_THREADS no icecc
> > > 68m3.556s    72 BB_NUMBER_THREADS no icecc
> > > 
> > > Still don't know how to get to sub 10min world runs RP was seeing,
> > > but at least it's as slow as it was before runqeueu changes (or even
> > > a bit faster in lastest master).
> > 
> > Just thinking out loud, other things which can influence timings:
> > 
> > Is SSTATE_DIR on NFS or local disk?
> 
> SSTATE_DIR is empty directory on local disk
> /dev/mapper/vg00-jenkins on /jenkins type ext4 (rw,noatime,nobarrier,commit=6000,stripe=128,data=ordered)
> 
> > Are sstate mirrors configured?
> 
> None, normally I have SSTATE_MIRRORS over sshfs in this case, but I've
> removed it before any performance testing.
> 
> > Is there an existing build or not, if so, how much is valid?
> 
> Nothing, I remove whole TMPDIR and cache before each run. Then let it
> recreate the cache before starting the profile:
> bitbake -p; time bitbake world -P -n
> 
> > Underlying filesystem of the build?
> 
> ext4, everything is pretty much generic Ubuntu 18.04
> 
> There is plenty of ram, I'll try to test this from tmpfs as well.

I did the same "bitbake -p; time bitbake world -P -n" on the same build with thud, warrior and zeus:

thud/profile.log.processed:         2803894450 function calls (2803191143 primitive calls) in 3340.784 seconds
thud/profile-worker.log.processed:         27005515 function calls (26944030 primitive calls) in 3243.139 seconds
warrior/profile.log.processed:         2804294486 function calls (2803446296 primitive calls) in 3604.226 seconds
warrior/profile-worker.log.processed:         27649679 function calls (27591220 primitive calls) in 3503.772 seconds
zeus/profile.log.processed:         2327031298 function calls (2326396186 primitive calls) in 4433.851 seconds
zeus/profile-worker.log.processed:         28829453 function calls (28771730 primitive calls) in 4331.257 seconds

thud/time:real    55m41.595s
warrior/time:real    60m4.954s
zeus/time:real    72m23.053s

with multilib disabled I got
zeus-no-multilib/profile.log.processed:         1232798107 function calls (1232447521 primitive calls) in 1773.164 seconds
zeus-no-multilib/profile-worker.log.processed:         15561565 function calls (15555044 primitive calls) in 1716.234 seconds
real    29m33.658s

Which seems reasonable as the number of tasks was cut in half.

Let me know if it's worth uploading these profiles somewhere.

Cheers,

> > Your build seems especially slow at executing through the tasks which
> > is effectively a test on how fast the system can fork() and return in
> > some ways. It would be interesting to block dry-run on the server side,
> > skip the fork and see how the numbers compare.
> 
> As discussed on IRC, it's slower than yours (8 minutes from 68), but the
> most significant chunk of time is lost somewhere else.
> 
> > My build manages some parts of the tasklist faster than others, perhaps
> > because there are more "free" tasks to execute at some points in the
> > task graph than others.
> > 
> > Also, I have some patches which improve performance a bit which I'm
> > still testing.
> 
> Thanks for all the work on this!
> 
> Cheers,
> -- 
> Martin 'JaMa' Jansa     jabber: Martin.Jansa at gmail.com



-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa at gmail.com
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 195 bytes
Desc: Digital signature
URL: <http://lists.openembedded.org/pipermail/openembedded-core/attachments/20190819/58e0aee0/attachment.sig>