[OE-core] [PATCH v2 4/4] reproducible: try to ensure reproducible xz archives
André Draszik
git at andred.net
Tue Mar 3 16:08:03 UTC 2020
On Tue, 2020-03-03 at 16:05 +0000, André Draszik wrote:
> xz suffers from a reproducibility problem when not using multi-
> threaded mode:
> a) archives are created differently in single- vs multi-threaded
> modes
> b) xz will scale down the compression level so as to be able to
> work within any memory limit given to it when being launched
> in single-threaded mode.
>
> Thus, for reproducible xz archives we need to launch xz with
> at least two threads.
>
> Add a little sanity test, and error out otherwise, so as to
> guarantee no difference due this fact.
>
> Assumptions:
> * The thread limit should be set via XZ_THREADS, not via
> modifying XZ_DEFAULTS or XZ_OPTS, or any other way
> * The thread limit should not be set to xz's magic value
> zero (0), as that will lead to single-threaded mode on
> single-core systems
>
> This patch here doesn't prevent people from shooting themselves
> into the foot by changing XZ_DEFAULTS to change the number
> of threads directly, but it's can serve as a hint at least.
I don't know if this patch is useful, feel free to drop it.
In an ideal world, it'd parse the output of xz --verbose --verbose, to catch
all possible ways people might be adjusting the thread limit, but that's
non-trivial.
Cheers,
Andre'
>
> Signed-off-by: André Draszik <git at andred.net>
> ---
> meta/classes/reproducible_build.bbclass | 5 +++++
> 1 file changed, 5 insertions(+)
>
> diff --git a/meta/classes/reproducible_build.bbclass b/meta/classes/reproducible_build.bbclass
> index 750eb950f2..e07bef87d8 100644
> --- a/meta/classes/reproducible_build.bbclass
> +++ b/meta/classes/reproducible_build.bbclass
> @@ -35,6 +35,7 @@
> # SOURCE_DATE_EPOCH is set for all tasks that might use it (do_configure, do_compile, do_package, ...)
>
> BUILD_REPRODUCIBLE_BINARIES ??= '1'
> +BUILD_REPRODUCIBLE_XZ_ARCHIVES ??= '1'
> inherit ${@oe.utils.ifelse(d.getVar('BUILD_REPRODUCIBLE_BINARIES') == '1', 'reproducible_build_simple', '')}
>
> SDE_DIR ="${WORKDIR}/source-date-epoch"
> @@ -198,4 +199,8 @@ BB_HASHBASE_WHITELIST += "SOURCE_DATE_EPOCH"
> python () {
> if d.getVar('BUILD_REPRODUCIBLE_BINARIES') == '1':
> d.appendVarFlag("do_unpack", "postfuncs", " do_create_source_date_epoch_stamp")
> +
> + if d.getVar('BUILD_REPRODUCIBLE_XZ_ARCHIVES') == '1':
> + if int(d.getVar('XZ_THREADS')) < 2:
> + bb.fatal("Can not build reproducible XZ archives without threading")
> }
More information about the Openembedded-core
mailing list