[OE-core] [RFC PATCH 0/2] RFC: Implement deterministic uid/gid

[Mark Hatle]

On 1/9/14, 5:19 PM, Martin Jansa wrote:
> On Thu, Jan 09, 2014 at 01:49:28PM -0600, Mark Hatle wrote:
>> I have updated the git://git.yoctoproject.org/poky-contrib mhatle/uidgid to the
>> latest oe-core master.
>>
>> I haven't seen any comments on this RFC yet.  Does anyone have any opinion
>> either way on the code referenced here?
>
> I wanted to do more tests and find some smaller reproducer, but I'm
> interested in this series.
>
> What I'm seeing in our builds is that when you compare
> files-in-image.txt report from independent (not reusing sstate-cache)
> builds with exactly the same metadata, group owners aren't
> deterministic.

Yes, this is partially to address that.  The issue is that the package install 
order itself is not deterministic.  The only thing that is, is the resulting set 
of files, modes, owner and group -names-.

Since the order the scripts are run is determines the effective uid/gid, the 
non-deterministic ordering causes these issues.

This patch set deals with that by allowing the user to specify -the- one and 
true deterministic set which is then reused.

> Random files in image getting owned by "messagebus" group etc., these
> issues went away when useradd class creating these ad-hoc groups had
> parameter for fixed gid or when I've added these groups directly to
> base-passwd group.master.
>
> Is this the same problem you're addressing in this patchset?

I have never seen a problem where specific files uname/gname are wrong.  It 
might be a related problem, but that shouldn't be happening in any case.

The -names- should always be right compared to the contents of the passwd/group 
files located within the constructed image.

I know there is an open bug where there were a few cases (in the sysroot) where 
things used a uname/gname from the default passwd/group file, but it had not yet 
been written.  So it fell back to the host's passwd/group file which had 
different numbers.  This is certainly incorrect behavior, and I think it's fixed 
by either inheriting the adduser class in those cases (even if they don't add 
users) or requiring the base-passwd in those specific package depends... but 
I've been looking for a reproducer to work though that.

--Mark

>> I'm confident patch 01/02 should be added to master.
>>
>> The patch 02/02 works properly in all of my testing and does implement support
>> for 'deterministic uid/gid' with dynamic passwd/group file construction during
>> package install.  The code is only activated if USERADD_REWRITE_PARAMS is set to
>> '1', so it's low risk -- but it does add a fairly large chunk of code to be
>> maintained over time.
>>
>> --Mark
>>
>> On 12/10/13, 12:31 PM, Mark Hatle wrote:
>>> The following series implements the deterministic uid/gid setting for a
>>> distribution.  Currently when a filesystem is generated the uid/gid values
>>> are generally set at install time, so the install order determines what
>>> the actual uid/gid values become.  In order to create a deterministic uid/gid
>>> set, that still dynamically constructs the passwd/group file, we add an
>>> option to read a special passwd/group file to allow the system to determine
>>> the values.
>>>
>>> It uses the existing parameters, and the values from the special passwd/group
>>> files to reconstruct the parameter set to ensure these items are fully
>>> defined with static values.
>>>
>>> The first patch (01/02) is generally applicable.  It fixes a real bug in
>>> the way the user/group adds occur today within the system.
>>>
>>> Patch 02/02 implements the new functionality.
>>>
>>>
>>> The following changes since commit 8505f0fa48cc79d51616b923c6e2c778c4b46a44:
>>>
>>>     libmatchbox: use PACKAGECONFIG (2013-12-10 14:13:01 +0000)
>>>
>>> are available in the git repository at:
>>>
>>>     git://git.yoctoproject.org/poky-contrib mhatle/uidgid
>>>     http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=mhatle/uidgid
>>>
>>> Mark Hatle (2):
>>>     useradd.bbclass: Fix build time install issues
>>>     useradd.bbclass: Add ability to select a static uid/gid automatically
>>>
>>>    meta/classes/useradd.bbclass         | 247 ++++++++++++++++++++++++++++++++++-
>>>    meta/conf/local.conf.sample.extended |  24 ++++
>>>    2 files changed, 265 insertions(+), 6 deletions(-)
>>>
>>
>> _______________________________________________
>> Openembedded-core mailing list
>> Openembedded-core at lists.openembedded.org
>> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>