[OE-core] [PATCH v2 3/3] rm_work.bbclass: clean up sooner

[Mike Crowe]

On Thursday 09 February 2017 at 17:24:39 +0100, Patrick Ohly wrote:
> On Wed, 2017-02-08 at 13:48 +0000, Mike Crowe wrote:
> > On Wednesday 08 February 2017 at 14:04:42 +0100, Patrick Ohly wrote:
> > > On Wed, 2017-02-08 at 11:50 +0000, Mike Crowe wrote:
> > > > On Friday 13 January 2017 at 15:52:33 +0100, Patrick Ohly wrote:
> > > > > The right solution is to inject do_rm_work before do_build and after
> > > > > all tasks of the recipe. Achieving that depends on the new bitbake
> > > > > bb.event.RecipeTaskPreProcess and bb.build.preceedtask().
> > > > 
> > > > We've run into trouble with this change. We have a number of custom
> > > > ancillary tasks that are used to generate source release files and run
> > > > package tests. No other tasks (including do_build) depend on these tasks
> > > > since they are run explicitly when required using bitbake -c; either
> > > > directly or via a recrdeptask.
> > > > 
> > > > Running a single task continues to work correctly - presumably this is
> > > > because the do_build task is not being run, so its dependencies (including
> > > > rm_work) aren't run either.
> > > > 
> > > > Running via the recrdeptask fails. This is because for any particular
> > > > recipe we end up depending on both do_build and the source release tasks.
> > > > There's nothing to stop do_rm_work running before (or even during!) one of
> > > > the source release tasks.
> > > 
> > > Can you show how you use recrdeptask and how you call bitbake to trigger
> > > those extra tasks, just for my understanding?
> > 
> > Certainly, we have a bbclass globally in INHERIT that contains:
> > 
> >  addtask source_release # potential fix here
> >  do_source_release() {
> >      :
> >  }
> > 
> >  addtask all_source_releases
> >  xx_do_all_source_releases() {
> >      :
> >  }
> > 
> >  do_all_source_releases[nostamp] = "1"
> >  do_all_source_releases[recrdeptask] += "do_all_source_releases do_source_release"
> > 
> >  addtask husk_recipe before do_source_release
> >  python xx_do_husk_recipe() {
> >     ...
> >  }
> > 
> >  (there's also another task similar to do_husk_recipe)
> > 
> > and in the particular recipe that has trouble with racing against rm_work:
> > 
> >  do_husk_recipe() {
> >     # do stuff in ${WORKDIR}
> >  }
> >  addtask husk_recipe after do_populate_sysroot before do_source_release
> > 
> > there's also a source-release-world recipe that contains:
> > 
> >  DEPENDS = "our-image"
> > 
> > and we run:
> > 
> >  bitbake -c all_source_releases source-release-world
> 
> I tried to replicate that with Poky master (= 226a508da), but without
> luck:
> 
> /work/poky$ cat meta/classes/release-source.bbclass 
> addtask source_release # potential fix here
> do_source_release() {
>     :
> }
> 
> addtask all_source_releases
> do_all_source_releases() {
>     :
> }
> 
> do_all_source_releases[nostamp] = "1"
> do_all_source_releases[recrdeptask] += "do_all_source_releases do_source_release"
> 
> addtask husk_recipe before do_source_release
> python do_husk_recipe() {
>     pass
> }
> 
> /work/poky$ cat meta/recipes-core/husk/husk.bb 
> LICENSE = "custom"
> 
> do_husk_recipe() {
>     # do stuff in ${WORKDIR}
>     :
> }
> addtask husk_recipe after do_populate_sysroot before do_source_release
> 
> /work/poky$ cat meta/recipes-core/husk/source-release-world.bb
> LICENSE = "custom"
> DEPENDS = "core-image-sato"
> 
> /work/poky/build$ tail -1 conf/local.conf
> INHERIT += "rm_work release-source"

The part I'd missed is the all-important line in source-release-world.bb:

 do_source_release[depends] += "core-image-sato:do_build"

We have this to ensure that the source release includes everything that is
required to build the rootfs itself. (For example, elfutils-native is only
included if this line is present.)

> /work/poky/build$ bitbake --dry-run -c all_source_releases source-release-world
> ...
> 
> That last command does not trigger the do_rm_work task and thus there is
> also no race.

That was my experience without the magic extra line too. Sorry I omitted
that.

> It seems unsafe to have tasks that are not properly ordered and just
> rely on not activating them in the same build, but without understanding
> the problem better it is too early to look for a solution.

Thanks for investigating. If you're still having trouble then I have a
single patch on top of current oe-core master that reproduces it for me
that I can send.

Mike.