Page MenuHomePhabricator

Restrict GIDs for system users to 499 as the upper boundary
Open, MediumPublic

Description

This was found when a puppet master reimage was failing: T235067#5562514

Users/groups created by packages are added with s the default tool in Debian for this (adduser). We use the default Debian config for adduser (due to some dpkg limitation back in 2001 (https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=541620 it gets created in the adduser.postinst instead of being shipped as a conffile). The default config specifies FIRST_SYSTEM_GID=100 and LAST_SYSTEM_GID=999, i.e. an ID from that pool is used when creating a user with the --system flag.

That's not a very good default for us to begin with: After all we're specifying various GIDs in the range >= 500 in data.yaml.

We should puppetise adduser.conf and reduce LAST_SYSTEM_GID to 499

There are various system users created during initial d-i base install, so it needs to be investigated if that change also needs to be applied to early install in some way or not.

Details

ProjectBranchLines +/-Subject
operations/puppetproduction+1 -7
operations/puppetproduction+1 -7
operations/puppetproduction+1 -1
operations/puppetproduction+1 -7
operations/puppetproduction+1 -9
operations/puppetproduction+2 -0
operations/puppetproduction+5 -3
operations/puppetproduction+6 -8
operations/puppetproduction+2 -0
operations/puppetproduction+2 -0
operations/puppetproduction+2 -0
operations/puppetproduction+2 -0
operations/puppetproduction+1 -0
operations/puppetproduction+31 -0
operations/puppetproduction+87 -39
operations/puppetproduction+8 -1
operations/puppetproduction+1 -0
operations/puppetproduction+7 -0
operations/puppetproduction+430 -0
Show related patches Customize query in gerrit

Event Timeline

jijiki triaged this task as Medium priority.Oct 14 2019, 3:32 PM

Change 542983 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] adduser: create module to manage /etc/adduser.conf

https://gerrit.wikimedia.org/r/542983

Change 542984 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] profile::base: add adduser module to profile:base

https://gerrit.wikimedia.org/r/542984

Change 542983 merged by Jbond:
[operations/puppet@production] adduser: create module to manage /etc/adduser.conf

https://gerrit.wikimedia.org/r/542983

Change 542984 merged by Jbond:
[operations/puppet@production] profile::base: add adduser module to profile:base

https://gerrit.wikimedia.org/r/542984

Change 547738 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Enable new adduser base class on spare hosts

https://gerrit.wikimedia.org/r/547738

Change 547738 merged by Muehlenhoff:
[operations/puppet@production] Enable new adduser base class on spare hosts

https://gerrit.wikimedia.org/r/547738

Change 548269 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Bump system user UID range in enforce-users-groups.sh

https://gerrit.wikimedia.org/r/548269

Change 548269 merged by Muehlenhoff:
[operations/puppet@production] Bump system user UID range in enforce-users-groups.sh

https://gerrit.wikimedia.org/r/548269

Change 573990 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] admin: add support for system users and groups

https://gerrit.wikimedia.org/r/573990

Change 574412 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] admin: add CI checks to ensure users and group shave the correct gid/uid

https://gerrit.wikimedia.org/r/574412

Change 573990 merged by Jbond:
[operations/puppet@production] admin: add support for system users and groups

https://gerrit.wikimedia.org/r/573990

Change 574412 merged by Jbond:
[operations/puppet@production] admin: add CI checks to ensure users and group have the correct gid/uid

https://gerrit.wikimedia.org/r/574412

Change 599032 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Enable adduser/sysusers config for role(test)

https://gerrit.wikimedia.org/r/599032

Change 599032 merged by Muehlenhoff:
[operations/puppet@production] Enable adduser/sysusers config for role(test)

https://gerrit.wikimedia.org/r/599032

Change 599042 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Enable managed adduser config for ulsfo

https://gerrit.wikimedia.org/r/599042

Change 599042 merged by Muehlenhoff:
[operations/puppet@production] Enable managed adduser config for ulsfo

https://gerrit.wikimedia.org/r/599042

Change 599358 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Enable managed adduser config for codfw

https://gerrit.wikimedia.org/r/599358

Change 599359 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Enable managed adduser config fleet-wide

https://gerrit.wikimedia.org/r/599359

Change 599358 merged by Muehlenhoff:
[operations/puppet@production] Enable managed adduser config for codfw

https://gerrit.wikimedia.org/r/599358

Change 601703 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Make systemd::sysuser require systemd class

https://gerrit.wikimedia.org/r/601703

Change 601703 merged by Muehlenhoff:
[operations/puppet@production] Make systemd::sysuser require systemd class

https://gerrit.wikimedia.org/r/601703

Change 601730 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Enable managed adduser config for codfw

https://gerrit.wikimedia.org/r/601730

Change 601730 merged by Muehlenhoff:
[operations/puppet@production] Enable managed adduser config for codfw

https://gerrit.wikimedia.org/r/601730

Change 601743 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Ship the sysusers default config via systemd::sysuser

https://gerrit.wikimedia.org/r/601743

Change 601743 merged by Muehlenhoff:
[operations/puppet@production] Ship the sysusers default config via systemd::sysuser

https://gerrit.wikimedia.org/r/601743

Change 602046 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] systemd-sysuser: Use /bin/systemd-sysusers and skip config on jessie

https://gerrit.wikimedia.org/r/602046

Change 602046 merged by Muehlenhoff:
[operations/puppet@production] systemd-sysuser: Use /bin/systemd-sysusers and skip config on jessie

https://gerrit.wikimedia.org/r/602046

Change 602067 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Enable managed adduser config for codfw, take three

https://gerrit.wikimedia.org/r/602067

Change 602067 merged by Muehlenhoff:
[operations/puppet@production] Enable managed adduser config for codfw, take three

https://gerrit.wikimedia.org/r/602067

Change 599359 merged by Muehlenhoff:
[operations/puppet@production] Enable managed adduser config fleet-wide

https://gerrit.wikimedia.org/r/599359

Change 602286 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Enable managed adduser/sysusers config also for WMCS

https://gerrit.wikimedia.org/r/602286

Change 602288 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Remove Hiera option to enable adduser config

https://gerrit.wikimedia.org/r/602288

Change 602288 merged by Jbond:
[operations/puppet@production] Remove Hiera option to enable adduser config

https://gerrit.wikimedia.org/r/602288

Change 644808 had a related patch set uploaded (by Jbond; owner: Jbond):
[operations/puppet@production] Remove Hiera option to enable adduser config

https://gerrit.wikimedia.org/r/644808

Change 602286 merged by Muehlenhoff:
[operations/puppet@production] Enable managed adduser/sysusers config also for WMCS

https://gerrit.wikimedia.org/r/602286

Status update: The puppetised adduser.conf is rolled out, what remains is to fix some of the mismatched UIDs (most of them should be in debmonitor, which will get fixed via a new package using systemd-sysuser).

Change 657770 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Enable managed adduser.conf unconditionally

https://gerrit.wikimedia.org/r/657770

Change 657770 merged by Muehlenhoff:
[operations/puppet@production] Enable managed adduser.conf unconditionally

https://gerrit.wikimedia.org/r/657770

Mentioned in SAL (#wikimedia-operations) [2021-06-03T08:37:31Z] <moritzm> upgrading codfw to debmonitor-client 0.3.0 (along with deleting/recreating system user within 100-499 range) T235162

Mentioned in SAL (#wikimedia-operations) [2021-06-03T09:37:26Z] <moritzm> upgrading eqiad to debmonitor-client 0.3.0 (along with deleting/recreating system user within 100-499 range) T235162

Status update: The puppetised adduser.conf is rolled out, what remains is to fix some of the mismatched UIDs (most of them should be in debmonitor, which will get fixed via a new package using systemd-sysuser).

debmonitor 0.3.0 has been rolled out along with a removal of the pre-existing sysuser , so that the subsequent update recreates the sysuser within the correct range. For buster there's https://phabricator.wikimedia.org/T256098 which had to be worked around by switching back to adduser. I've done a fleet-wide cumin run to validate that all debmonitor sysusers are now in the correct range (there were a few stuck old systemd timers stuck (since they were started before the urllib fix, some dating back to March) which made deluser fail, but I cleaned those up via cumin as well). Now that we are consistently on 0.3.0 we should run into no further issues like that.

Now that debmonitor is fixed, the final thing to do is to audit whether there are other sysusers outrange the allowed range.

Now that debmonitor is fixed, the final thing to do is to audit whether there are other sysusers outrange the allowed range.

I ran a fleet-wide check on UIDs within the 500 (current upper limit for system users) and 999 (human users start with 1000), along with filtering out the centrally mapped UIDs we maintain in data.yaml (currently 900-911):

  • We have a few users created a long time ago which are in the range: mark/531, ariel/543, awjrichards/552, tstarling/501, aaron/544, catrope/546, daniel/545, ezachte/523
  • 325 system users for mcrouter within the wrong range
  • 1267 system users for systemd-coredump within the wrong range

And then there's a few remaining cases:
On host alert1001.wikimedia.org: user karma has UID 997
On host alert1001.wikimedia.org: user kthxbye has UID 996
On host alert1001.wikimedia.org: user phalerts has UID 995
On host alert2001.wikimedia.org: user karma has UID 997
On host alert2001.wikimedia.org: user kthxbye has UID 996
On host alert2001.wikimedia.org: user phalerts has UID 995
On host an-coord1001.eqiad.wmnet: user mysql has UID 998
On host an-coord1002.eqiad.wmnet: user mysql has UID 998
On host an-test-coord1001.eqiad.wmnet: user mysql has UID 998
On host an-tool1007.eqiad.wmnet: user turnilo has UID 998
On host an-tool1009.eqiad.wmnet: user hue has UID 997
On host dborch1001.wikimedia.org: user orchestrator has UID 997
On host sodium.wikimedia.org: user acme has UID 998
On host sodium.wikimedia.org: user mirror has UID 997
On host thorium.eqiad.wmnet: user stats has UID 999

I'll have a closer look how they are created and will create sub tasks.