[OE-core] [PATCH] package: Fix PACKAGELOCK handling

[Martin Jansa]

Now with pseudo log files a bit smaller this package-output.lock seems to
be most common "issue" reported with do_qa_pseudo.

e.g.:
libao-1.2.0: This /work/i586-oe-linux-musl/libao/1.2.0-r0/pseudo/pseudo.log
indicates 16 errors [qa-pseudo]
see grep -e '^creat ignored for existing file' -e '^creat for.*replaces
existing'
/home/jenkins/oe/world/shr-core/tmpfs/work/i586-oe-linux-musl/libao/1.2.0-r0/pseudo/pseudo.log
or grep -v '^path mismatch'
/home/jenkins/oe/world/shr-core/tmpfs/work/i586-oe-linux-musl/libao/1.2.0-r0/pseudo/pseudo.log:
debug_logfile: fd 2
pid 4165 [parent 4160], doing new pid setup and server start
Setup complete, sending SIGUSR1 to pid 4160.
db cleanup for server shutdown, 01:43:56.829
memory-to-file backup complete, 01:43:56.830.
db cleanup finished, 01:43:56.830
debug_logfile: fd 2
pid 11266 [parent 11265], doing new pid setup and server start
Setup complete, sending SIGUSR1 to pid 11265.
inode mismatch:
'/home/jenkins/oe/world/shr-core/tmpfs/sysroots/package-output.lock' ino
346594 in db, 347812 in request.
creat ignored for existing file
'/home/jenkins/oe/world/shr-core/tmpfs/sysroots/package-output.lock'.
inode mismatch:
'/home/jenkins/oe/world/shr-core/tmpfs/sysroots/package-output.lock' ino
346594 in db, 347812 in request.
inode mismatch:
'/home/jenkins/oe/world/shr-core/tmpfs/sysroots/package-output.lock' ino
346594 in db, 347812 in request.
inode mismatch:
'/home/jenkins/oe/world/shr-core/tmpfs/sysroots/package-output.lock' ino
346594 in db, 347812 in request.
inode mismatch:
'/home/jenkins/oe/world/shr-core/tmpfs/sysroots/package-output.lock' ino
346594 in db, 347812 in request.
inode mismatch:
'/home/jenkins/oe/world/shr-core/tmpfs/sysroots/package-output.lock' ino
346594 in db, 347812 in request.
 [qa-pseudo]

shown in last world build:
http://jenkins.nas-admin.org/view/OE/job/oe_world_qemux86/760/console

I have no idea how to resolve this one as it needs to be global and shared
between many pseudo databases.

Any idea?

On Fri, Apr 20, 2018 at 9:25 AM, Richard Purdie <
richard.purdie at linuxfoundation.org> wrote:

> PACKAGELOCK is there to protect readers of PKGDATA_DIR from writes and
> files
> changing whilst they're being read. With various changes to the codebase,
> the lock code has become confused as the files are now written by the
> sstate
> code in do_packagedata, not in do_package directly any longer.
>
> This change cleans up the code so read sites take the shared lock
> (anything in
> do_package), write sites take the full lock (do_packagedata sstate).
>
> The lock from do_package sstate is no longer needed since it doesn't write
> outside WORKDIR.
>
> Signed-off-by: Richard Purdie <richard.purdie at linuxfoundation.org>
> ---
>  meta/classes/package.bbclass | 11 ++++++-----
>  1 file changed, 6 insertions(+), 5 deletions(-)
>
> diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass
> index 0436d91..e03d450 100644
> --- a/meta/classes/package.bbclass
> +++ b/meta/classes/package.bbclass
> @@ -1615,7 +1615,7 @@ python package_do_shlibs() {
>      shlibswork_dir = d.getVar('SHLIBSWORKDIR')
>
>      # Take shared lock since we're only reading, not writing
> -    lf = bb.utils.lockfile(d.expand("${PACKAGELOCK}"))
> +    lf = bb.utils.lockfile(d.expand("${PACKAGELOCK}"), True)
>
>      def linux_so(file, needed, sonames, renames, pkgver):
>          needs_ldconfig = False
> @@ -1900,7 +1900,7 @@ python package_do_pkgconfig () {
>                                  pkgconfig_needed[pkg] += exp.replace(',',
> ' ').split()
>
>      # Take shared lock since we're only reading, not writing
> -    lf = bb.utils.lockfile(d.expand("${PACKAGELOCK}"))
> +    lf = bb.utils.lockfile(d.expand("${PACKAGELOCK}"), True)
>
>      for pkg in packages.split():
>          pkgs_file = os.path.join(shlibswork_dir, pkg + ".pclist")
> @@ -2233,11 +2233,9 @@ do_package[dirs] = "${SHLIBSWORKDIR} ${PKGDESTWORK}
> ${D}"
>  do_package[vardeps] += "${PACKAGEBUILDPKGD} ${PACKAGESPLITFUNCS}
> ${PACKAGEFUNCS} ${@gen_packagevar(d)}"
>  addtask package after do_install
>
> -PACKAGELOCK = "${STAGING_DIR}/package-output.lock"
>  SSTATETASKS += "do_package"
>  do_package[cleandirs] = "${PKGDEST} ${PKGDESTWORK}"
>  do_package[sstate-plaindirs] = "${PKGD} ${PKGDEST} ${PKGDESTWORK}"
> -do_package[sstate-lockfile-shared] = "${PACKAGELOCK}"
>  do_package_setscene[dirs] = "${STAGING_DIR}"
>
>  python do_package_setscene () {
> @@ -2252,9 +2250,12 @@ do_packagedata () {
>  addtask packagedata before do_build after do_package
>
>  SSTATETASKS += "do_packagedata"
> +# PACKAGELOCK protects readers of PKGDATA_DIR against writes
> +# whilst code is reading in do_package
> +PACKAGELOCK = "${STAGING_DIR}/package-output.lock"
>  do_packagedata[sstate-inputdirs] = "${PKGDESTWORK}"
>  do_packagedata[sstate-outputdirs] = "${PKGDATA_DIR}"
> -do_packagedata[sstate-lockfile-shared] = "${PACKAGELOCK}"
> +do_packagedata[sstate-lockfile] = "${PACKAGELOCK}"
>  do_packagedata[stamp-extra-info] = "${MACHINE_ARCH}"
>
>  python do_packagedata_setscene () {
> --
> 2.7.4
>
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core at lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openembedded.org/pipermail/openembedded-core/attachments/20180420/3a7b58e6/attachment-0002.html>