[OE-core] [PATCH v2 3/3] rm_work.bbclass: clean up sooner

[Patrick Ohly]

On Wed, 2017-02-08 at 13:48 +0000, Mike Crowe wrote:
> On Wednesday 08 February 2017 at 14:04:42 +0100, Patrick Ohly wrote:
> > On Wed, 2017-02-08 at 11:50 +0000, Mike Crowe wrote:
> > > On Friday 13 January 2017 at 15:52:33 +0100, Patrick Ohly wrote:
> > > > The right solution is to inject do_rm_work before do_build and after
> > > > all tasks of the recipe. Achieving that depends on the new bitbake
> > > > bb.event.RecipeTaskPreProcess and bb.build.preceedtask().
> > > 
> > > We've run into trouble with this change. We have a number of custom
> > > ancillary tasks that are used to generate source release files and run
> > > package tests. No other tasks (including do_build) depend on these tasks
> > > since they are run explicitly when required using bitbake -c; either
> > > directly or via a recrdeptask.
> > > 
> > > Running a single task continues to work correctly - presumably this is
> > > because the do_build task is not being run, so its dependencies (including
> > > rm_work) aren't run either.
> > > 
> > > Running via the recrdeptask fails. This is because for any particular
> > > recipe we end up depending on both do_build and the source release tasks.
> > > There's nothing to stop do_rm_work running before (or even during!) one of
> > > the source release tasks.
> > 
> > Can you show how you use recrdeptask and how you call bitbake to trigger
> > those extra tasks, just for my understanding?
> 
> Certainly, we have a bbclass globally in INHERIT that contains:
> 
>  addtask source_release # potential fix here
>  do_source_release() {
>      :
>  }
> 
>  addtask all_source_releases
>  xx_do_all_source_releases() {
>      :
>  }
> 
>  do_all_source_releases[nostamp] = "1"
>  do_all_source_releases[recrdeptask] += "do_all_source_releases do_source_release"
> 
>  addtask husk_recipe before do_source_release
>  python xx_do_husk_recipe() {
>     ...
>  }
> 
>  (there's also another task similar to do_husk_recipe)
> 
> and in the particular recipe that has trouble with racing against rm_work:
> 
>  do_husk_recipe() {
>     # do stuff in ${WORKDIR}
>  }
>  addtask husk_recipe after do_populate_sysroot before do_source_release
> 
> there's also a source-release-world recipe that contains:
> 
>  DEPENDS = "our-image"
> 
> and we run:
> 
>  bitbake -c all_source_releases source-release-world

I tried to replicate that with Poky master (= 226a508da), but without
luck:

/work/poky$ cat meta/classes/release-source.bbclass 
addtask source_release # potential fix here
do_source_release() {
    :
}

addtask all_source_releases
do_all_source_releases() {
    :
}

do_all_source_releases[nostamp] = "1"
do_all_source_releases[recrdeptask] += "do_all_source_releases do_source_release"

addtask husk_recipe before do_source_release
python do_husk_recipe() {
    pass
}

/work/poky$ cat meta/recipes-core/husk/husk.bb 
LICENSE = "custom"

do_husk_recipe() {
    # do stuff in ${WORKDIR}
    :
}
addtask husk_recipe after do_populate_sysroot before do_source_release

/work/poky$ cat meta/recipes-core/husk/source-release-world.bb
LICENSE = "custom"
DEPENDS = "core-image-sato"

/work/poky/build$ tail -1 conf/local.conf
INHERIT += "rm_work release-source"

/work/poky/build$ bitbake --dry-run -c all_source_releases source-release-world
...

That last command does not trigger the do_rm_work task and thus there is
also no race.

I had a hunch that it is related to populating the sysroot and how it
might depend on do_build in the dependencies, but that's not it. I added
DEPENDS="m4" to the husk.bb above and it has an effect:

/work/poky/build$ bitbake -g -c all_source_releases husk
Loading cache: 100% |
#################################################################################| Time: 0:00:00
Loaded 1324 entries from dependency cache.
NOTE: Resolving any missing task queue dependencies
NOTE: PN build list saved to 'pn-buildlist'
NOTE: PN dependencies saved to 'pn-depends.dot'
NOTE: Package dependencies saved to 'package-depends.dot'
NOTE: Task dependencies saved to 'task-depends.dot'
/work/poky/build$ grep -e 'husk.*->.*m4' task-depends.dot 
"husk.do_all_source_releases" -> "m4.do_source_release"
"husk.do_all_source_releases" -> "m4-native.do_source_release"
"husk.do_prepare_recipe_sysroot" -> "m4.do_populate_sysroot"

But it does not cause m4.do_build and thus m4.do_rm_work to run.

> > > It seems that we need to ensure that do_rm_work also needs to depend on our
> > > ancillary tasks too, but only if they are being built. I'm unsure how this
> > > can be done though. :(
> > 
> > How do you determine whether the tasks need to run? Does it depend on
> > how bitbake is invoked or does it depend on specific properties of the
> > recipe?
> 
> Running bitbake as above.
> 
> I think that I could fix this by changing:
> 
>  addtask source_release
> 
> to
> 
>  addtask source_release before do_rm_work
> 
> (and also fixing any other tasks that suffer from this problem)
> 
> but I was hoping that there was a better fix to rm_work.bbclass itself.

It seems unsafe to have tasks that are not properly ordered and just
rely on not activating them in the same build, but without understanding
the problem better it is too early to look for a solution.

-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.