[OE-core] Hash Equivalency - What this means for developer productivity

[richard.purdie at linuxfoundation.org]

On Thu, 2019-08-08 at 10:18 +0000, Jacob Kroon wrote:
> Hi,
> 
> On 8/2/19 5:53 PM, Richard Purdie wrote:
> > This has been talked about in a lot of the project meetings but I
> > wanted to ensure its on the mailing list too.
> > 
> > master-next (and hopefully soon, master) now contain patches which
> > enable "hash equivalency" support in runqueue in bitbake. That
> > doesn't
> > sound so interesting until I explain what this means.
> > 
> <snip>
> > I think this should be a "big deal" for many developers, reducing
> > unneded rebuilds and hence speeding up development.
> > 
> > As with any new feature, there are likely going to be some some
> > teething issues but I'm very opitmistic about some of the benefits
> > this
> > should bring!
> > 
> 
> Very exciting see this feature being worked on! I have high hopes it
> is
> gonna reduce my build times.
> 
> I just updated to latest bitbake/oe-core master and started an
> incremental build of my distro with hash equivalency enabled, and ran
> into one problem:
> 
> ERROR: core-image-minimal-1.0-r0 do_rootfs: Unable to install
> packages.
> <snip>
> Collected errors:
>   * Solver encountered 2 problem(s):
>   * Problem 1/2:
>   *   - nothing provides eudev needed by <snip>
> <snip>
>   * Problem 2/2:
>   *   - nothing provides udev needed by <snip>
> <snip>
> 
> Checking the build log I see:
> 
> ...
> NOTE: Task
> /home/jkroon/Projects/<snip>/build/../openembedded-core/meta/recipes-
> core/glib-2.0/glib-2.0_2.60.6.bb:do_populate_sysroot
> unihash changed to
> b88f1b01421cd7501b290888b783b057b92a7fb7a20f37fd08a56f2eaf8c56ed
> NOTE: Setscene task
> /home/jkroon/Projects/<snip>/build/../openembedded-core/meta/recipes-
> core/udev/eudev_3.2.8.bb:do_populate_sysroot
> now valid and being rerun
> ...
> NOTE: Setscene tasks completed
> 
> NOTE: Task
> /home/jkroon/Projects/<snip>/build/../openembedded-core/meta/recipes-
> core/udev/eudev_3.2.8.bb:do_package_qa
> unihash changed to
> 607d4eb7859bf6c725add7592a2c8bd64464afb259264281cbf2a3ad4e17e8d2
> ...
> NOTE: Task
> /home/jkroon/Projects/<snip>/build/../openembedded-core/meta/recipes-
> core/glib-2.0/glib-2.0_2.60.6.bb:do_package_qa
> unihash changed to
> 89413c6afc25b390471f0cebb6193a226e20a744ac86b473c0b11f7432b5165a
> ...
> NOTE: Setscene tasks completed
> 
> ...
> 
> Checking for built ipk packages I see:
> 
> [jkroon at manatarms-greyskull <snip>]$ find
> /tmp/ramdisk/<snip>/deploy/ipk/|grep eudev
> /tmp/ramdisk/<snip>/deploy/ipk/i686/eudev-dbg_3.2.8-r0_i686.ipk
> /tmp/ramdisk/<snip>/deploy/ipk/i686/eudev-dev_3.2.8-r0_i686.ipk
> 
> If I then do:
> 
> bitbake -c cleansstate eudev
> bitbake eudev
> 
> then I get:
> 
> [jkroon at manatarms-greyskull <snip>]$ find
> /tmp/ramdisk/<snip>/deploy/ipk/|grep eudev
> /tmp/ramdisk/<snip>/deploy/ipk/i686/eudev-dev_3.2.8-r0_i686.ipk
> /tmp/ramdisk/<snip>/deploy/ipk/i686/eudev-dbg_3.2.8-r0_i686.ipk
> /tmp/ramdisk/<snip>/deploy/ipk/i686/eudev-hwdb_3.2.8-r0_i686.ipk
> /tmp/ramdisk/<snip>/deploy/ipk/i686/eudev_3.2.8-r0_i686.ipk
> /tmp/ramdisk/<snip>/deploy/ipk/i686/eudev-src_3.2.8-r0_i686.ipk
> 
> and then I can complete building the image.
> 
> I did not have hash equivalency enabled in the beginning, I've just
> recently enabled it, without wiping tmp-build/ or sstate in between.

That sounds like some kind of rebuild problem with eudev rather than a
problem with the new runqueue code. It looks like eudev rebuilt but
only produces some subset of the packages it should have done.

If you still had the logs for eudev it would be have been interesting
to see if the install/package logs looked correct.

FWIW, enabling this on an existing build should work fine, it may just
take an initial rebuild of a given recipe for it to have something
compare against so it wouldn't work quite as effectively until after a
few builds where things have rebuilt.

Cheers,

Richard