[OE-core] [RFC PATCH 1/2] cross-localedef-native: Add hardlink resolver from util-linux

[Jason Wessel]

On 8/16/19 4:50 PM, richard.purdie at linuxfoundation.org wrote:
> On Fri, 2019-08-16 at 14:29 -0700, Khem Raj wrote:
>> On Fri, Aug 16, 2019 at 2:25 PM Richard Purdie
>> <richard.purdie at linuxfoundation.org> wrote:
>>> This was discussed on IRC. We really really want to avoid that as a
>>> dependency if we care about build performance. Its really easy to
>>> just
>>> add that in but the implications are significant :(
>>>
>> A captured version of this utility would also mean that we have to
>> keep this in sync for bug fixes and security fixes etc. which could
>> also be considered as expense, so how much do we lose if util-linux-
>> native is brought early into dependency chain?
> The tool is being used for a specific purpose at build time. Its not
> exposed on target and its use is extremely constrained to a specific
> set of files where its behaviour should be deterministic.
>
> As such, yes there are questions about bug fixes and about security but
> I don't think its going to be a serious burden. We'd need to carry this
> until such times as the host systems we support have hardlink which is
> probably a few years out but will arrive.
>
> The impact of having util-linux-native in the dependency graph at this
> point is significant. I say that as someone who has done a lot of work
> in trimming down the times of our bootstrap process. This is exactly
> the kind of dependency I've removed and avoided.
>
> glibc is special in that its the worst choke point on the dependency
> graph. Nothing to do with target binaries happens until it is ready and
> then gcc-runtime can build. Whilst the locales are split from glibc,
> they are still connected in the task graph for packaging purposes so
> this is important.
>
> Its been pointed out there are other places where util-linux-native has
> crept in, in particular python3-native. My preference would be to
> minimise these dependencies rather than use them as a way to justify
> adding it here too.


So what is the consensus here?   Can we require the host system to have hardlink?   It is available as a stand alone binary on Ubuntu 14.04 and up and on Fedora systems, but isn't installed by default, and we'd have to check the version because the arguments sadly are slightly different.   We could certainly make util-linux a dependency, but that is why this patch existed, in order to avoid the dependency.

I did find a python version at https://raw.githubusercontent.com/akaihola/hardlinkpy/master/hardlink.py. I made a one line modification to it and it works, but it isn't nearly as efficient.  The proof is below, using the same locale-tree each time.  It is important to look at both the real and user time.  The difference gets more significant when you run it under pseudo, where we expect it to run 100% of the time.  It is roughly 10x slower to do this in python.


time recipe-sysroot-native/usr/bin/cross-localedef-hardlink -c -v locale-tree
Directories:         982
Objects:            7441
Regular files:      6441
Comparisons:        3331
Linked:             3109
Saved:         754675712

real    0m0.641s
user    0m0.120s
sys     0m0.518s

time python3 hardlink.py -v 0 -c locale-tree


Hard linking Statistics:
Directories           : 982
Regular files         : 6441
Comparisons           : 31884
Hardlinked this run   : 3109
Total hardlinks       : 3109
Bytes saved this run  : 744089843 (709.619 mebibytes)
Total bytes saved     : 744089843 (709.619 mebibytes)
Total run time        : 2.124422073364258 seconds

real    0m2.180s
user    0m1.220s
sys     0m0.953s


============Now lets run it under pseudo ================

time recipe-sysroot-native/usr/bin/cross-localedef-hardlink -c -v locale-tree
Directories:         982
Objects:            7441
Regular files:      6441
Comparisons:        3331
Linked:             3109
Saved:         754675712

real    0m2.505s
user    0m0.373s
sys     0m1.286s

time python3 hardlink.py -v 0 -c locale-tree


Hard linking Statistics:
Directories           : 982
Regular files         : 6441
Comparisons           : 31884
Hardlinked this run   : 3109
Total hardlinks       : 3109
Bytes saved this run  : 744089843 (709.619 mebibytes)
Total bytes saved     : 744089843 (709.619 mebibytes)
Total run time        : 12.971824884414673 seconds

real    0m13.046s
user    0m3.633s
sys     0m5.192s