Page MenuHomePhabricator
Create Task
Maniphest T235162

Restrict GIDs for system users to 499 as the upper boundary
Open, MediumPublic
Actions

Description

This was found when a puppet master reimage was failing: T235067#5562514

Users/groups created by packages are added with s the default tool in Debian for this (adduser). We use the default Debian config for adduser (due to some dpkg limitation back in 2001 (https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=541620 it gets created in the adduser.postinst instead of being shipped as a conffile). The default config specifies FIRST_SYSTEM_GID=100 and LAST_SYSTEM_GID=999, i.e. an ID from that pool is used when creating a user with the --system flag.

That's not a very good default for us to begin with: After all we're specifying various GIDs in the range >= 500 in data.yaml.

We should puppetise adduser.conf and reduce LAST_SYSTEM_GID to 499

There are various system users created during initial d-i base install, so it needs to be investigated if that change also needs to be applied to early install in some way or not.

Details

ProjectBranchLines +/-Subject
operations/puppetproduction+1 -7Remove Hiera option to enable adduser config
operations/puppetproduction+1 -7Enable managed adduser.conf unconditionally
operations/puppetproduction+1 -1Enable managed adduser/sysusers config also for WMCS
operations/puppetproduction+1 -7Remove Hiera option to enable adduser config
operations/puppetproduction+1 -9Enable managed adduser config fleet-wide
operations/puppetproduction+2 -0Enable managed adduser config for codfw, take three
operations/puppetproduction+5 -3systemd-sysuser: Use /bin/systemd-sysusers and skip config on jessie
operations/puppetproduction+6 -8Ship the sysusers default config via systemd::sysuser
operations/puppetproduction+2 -0Enable managed adduser config for codfw
operations/puppetproduction+2 -0Make systemd::sysuser require systemd class
operations/puppetproduction+2 -0Enable managed adduser config for codfw
operations/puppetproduction+2 -0Enable managed adduser config for ulsfo
operations/puppetproduction+1 -0Enable adduser/sysusers config for role(test)
operations/puppetproduction+31 -0admin: add CI checks to ensure users and group have the correct gid/uid
operations/puppetproduction+87 -39admin: add support for system users and groups
operations/puppetproduction+8 -1Bump system user UID range in enforce-users-groups.sh
operations/puppetproduction+1 -0Enable new adduser base class on spare hosts
operations/puppetproduction+7 -0profile::base: add adduser module to profile:base
operations/puppetproduction+430 -0adduser: create module to manage /etc/adduser.conf
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

MoritzMuehlenhoff created this task.Oct 10 2019, 10:11 AM
ayounsi added a project: SRE.Oct 10 2019, 10:14 AM
ayounsi removed a subscriber: SRE.
Peachey88 updated the task description. (Show Details)Oct 10 2019, 10:14 AM
jijiki triaged this task as Medium priority.Oct 14 2019, 3:32 PM
gerritbot added a comment.Oct 14 2019, 4:05 PM

Change 542983 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] adduser: create module to manage /etc/adduser.conf

https://gerrit.wikimedia.org/r/542983

gerritbot added a project: Patch-For-Review.Oct 14 2019, 4:05 PM

Change 542984 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] profile::base: add adduser module to profile:base

https://gerrit.wikimedia.org/r/542984

gerritbot added a comment.Oct 16 2019, 7:59 AM

Change 542983 merged by Jbond:
[operations/puppet@production] adduser: create module to manage /etc/adduser.conf

https://gerrit.wikimedia.org/r/542983

gerritbot added a comment.Oct 21 2019, 1:26 PM

Change 542984 merged by Jbond:
[operations/puppet@production] profile::base: add adduser module to profile:base

https://gerrit.wikimedia.org/r/542984

Maintenance_bot removed a project: Patch-For-Review.Oct 21 2019, 2:11 PM
jbond added a project: User-jbond.Oct 30 2019, 5:53 PM
jbond moved this task from Unsorted 💣 to Back Burner 🏛️ on the User-jbond board.Oct 30 2019, 6:05 PM
gerritbot added a comment.Nov 1 2019, 2:51 PM

Change 547738 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Enable new adduser base class on spare hosts

https://gerrit.wikimedia.org/r/547738

gerritbot added a project: Patch-For-Review.Nov 1 2019, 2:51 PM
gerritbot added a comment.Nov 4 2019, 1:19 PM

Change 547738 merged by Muehlenhoff:
[operations/puppet@production] Enable new adduser base class on spare hosts

https://gerrit.wikimedia.org/r/547738

Maintenance_bot removed a project: Patch-For-Review.Nov 4 2019, 2:11 PM
gerritbot added a comment.Nov 4 2019, 2:18 PM

Change 548269 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Bump system user UID range in enforce-users-groups.sh

https://gerrit.wikimedia.org/r/548269

gerritbot added a project: Patch-For-Review.Nov 4 2019, 2:18 PM
gerritbot added a comment.Nov 18 2019, 10:00 AM

Change 548269 merged by Muehlenhoff:
[operations/puppet@production] Bump system user UID range in enforce-users-groups.sh

https://gerrit.wikimedia.org/r/548269

Maintenance_bot removed a project: Patch-For-Review.Nov 18 2019, 10:10 AM
gerritbot added a comment.Feb 21 2020, 12:52 PM

Change 573990 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] admin: add support for system users and groups

https://gerrit.wikimedia.org/r/573990

gerritbot added a project: Patch-For-Review.Feb 21 2020, 12:52 PM
gerritbot added a comment.Feb 24 2020, 11:10 AM

Change 574412 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] admin: add CI checks to ensure users and group shave the correct gid/uid

https://gerrit.wikimedia.org/r/574412

gerritbot added a comment.Feb 26 2020, 11:04 AM

Change 573990 merged by Jbond:
[operations/puppet@production] admin: add support for system users and groups

https://gerrit.wikimedia.org/r/573990

gerritbot added a comment.Feb 26 2020, 12:08 PM

Change 574412 merged by Jbond:
[operations/puppet@production] admin: add CI checks to ensure users and group have the correct gid/uid

https://gerrit.wikimedia.org/r/574412

Maintenance_bot removed a project: Patch-For-Review.Feb 26 2020, 12:11 PM
gerritbot added a comment.May 27 2020, 2:00 PM

Change 599032 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Enable adduser/sysusers config for role(test)

https://gerrit.wikimedia.org/r/599032

gerritbot added a project: Patch-For-Review.May 27 2020, 2:00 PM
gerritbot added a comment.May 27 2020, 2:08 PM

Change 599032 merged by Muehlenhoff:
[operations/puppet@production] Enable adduser/sysusers config for role(test)

https://gerrit.wikimedia.org/r/599032

Maintenance_bot removed a project: Patch-For-Review.May 27 2020, 2:11 PM
gerritbot added a comment.May 27 2020, 2:21 PM

Change 599042 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Enable managed adduser config for ulsfo

https://gerrit.wikimedia.org/r/599042

gerritbot added a project: Patch-For-Review.May 27 2020, 2:21 PM
gerritbot added a comment.May 28 2020, 8:53 AM

Change 599042 merged by Muehlenhoff:
[operations/puppet@production] Enable managed adduser config for ulsfo

https://gerrit.wikimedia.org/r/599042

Maintenance_bot removed a project: Patch-For-Review.May 28 2020, 9:10 AM
gerritbot added a comment.May 28 2020, 3:27 PM

Change 599358 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Enable managed adduser config for codfw

https://gerrit.wikimedia.org/r/599358

gerritbot added a project: Patch-For-Review.May 28 2020, 3:27 PM

Change 599359 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Enable managed adduser config fleet-wide

https://gerrit.wikimedia.org/r/599359

MoritzMuehlenhoff claimed this task.May 29 2020, 8:43 AM
gerritbot added a comment.Jun 2 2020, 11:08 AM

Change 599358 merged by Muehlenhoff:
[operations/puppet@production] Enable managed adduser config for codfw

https://gerrit.wikimedia.org/r/599358

gerritbot added a comment.Jun 2 2020, 11:56 AM

Change 601703 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Make systemd::sysuser require systemd class

https://gerrit.wikimedia.org/r/601703

gerritbot added a comment.Jun 2 2020, 1:13 PM

Change 601703 merged by Muehlenhoff:
[operations/puppet@production] Make systemd::sysuser require systemd class

https://gerrit.wikimedia.org/r/601703

gerritbot added a comment.Jun 2 2020, 1:28 PM

Change 601730 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Enable managed adduser config for codfw

https://gerrit.wikimedia.org/r/601730

gerritbot added a comment.Jun 2 2020, 1:35 PM

Change 601730 merged by Muehlenhoff:
[operations/puppet@production] Enable managed adduser config for codfw

https://gerrit.wikimedia.org/r/601730

gerritbot added a comment.Jun 2 2020, 2:30 PM

Change 601743 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Ship the sysusers default config via systemd::sysuser

https://gerrit.wikimedia.org/r/601743

gerritbot added a comment.Jun 3 2020, 10:32 AM

Change 601743 merged by Muehlenhoff:
[operations/puppet@production] Ship the sysusers default config via systemd::sysuser

https://gerrit.wikimedia.org/r/601743

gerritbot added a comment.Jun 3 2020, 10:51 AM

Change 602046 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] systemd-sysuser: Use /bin/systemd-sysusers and skip config on jessie

https://gerrit.wikimedia.org/r/602046

gerritbot added a comment.Jun 3 2020, 11:58 AM

Change 602046 merged by Muehlenhoff:
[operations/puppet@production] systemd-sysuser: Use /bin/systemd-sysusers and skip config on jessie

https://gerrit.wikimedia.org/r/602046

gerritbot added a comment.Jun 3 2020, 12:19 PM

Change 602067 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Enable managed adduser config for codfw, take three

https://gerrit.wikimedia.org/r/602067

gerritbot added a comment.Jun 3 2020, 12:35 PM

Change 602067 merged by Muehlenhoff:
[operations/puppet@production] Enable managed adduser config for codfw, take three

https://gerrit.wikimedia.org/r/602067

gerritbot added a comment.Jun 4 2020, 8:06 AM

Change 599359 merged by Muehlenhoff:
[operations/puppet@production] Enable managed adduser config fleet-wide

https://gerrit.wikimedia.org/r/599359

Maintenance_bot removed a project: Patch-For-Review.Jun 4 2020, 8:11 AM
gerritbot added a comment.Jun 4 2020, 8:19 AM

Change 602286 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Enable managed adduser/sysusers config also for WMCS

https://gerrit.wikimedia.org/r/602286

gerritbot added a project: Patch-For-Review.Jun 4 2020, 8:19 AM
gerritbot added a comment.Jun 4 2020, 8:35 AM

Change 602288 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Remove Hiera option to enable adduser config

https://gerrit.wikimedia.org/r/602288

gerritbot added a comment.Dec 2 2020, 12:21 PM

Change 602288 merged by Jbond:
[operations/puppet@production] Remove Hiera option to enable adduser config

https://gerrit.wikimedia.org/r/602288

gerritbot added a comment.Dec 2 2020, 12:31 PM

Change 644808 had a related patch set uploaded (by Jbond; owner: Jbond):
[operations/puppet@production] Remove Hiera option to enable adduser config

https://gerrit.wikimedia.org/r/644808

gerritbot added a comment.Jan 11 2021, 9:59 AM

Change 602286 merged by Muehlenhoff:
[operations/puppet@production] Enable managed adduser/sysusers config also for WMCS

https://gerrit.wikimedia.org/r/602286

MoritzMuehlenhoff added a comment.Jan 15 2021, 3:36 PM

Status update: The puppetised adduser.conf is rolled out, what remains is to fix some of the mismatched UIDs (most of them should be in debmonitor, which will get fixed via a new package using systemd-sysuser).

gerritbot added a comment.Jan 22 2021, 8:23 AM

Change 657770 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Enable managed adduser.conf unconditionally

https://gerrit.wikimedia.org/r/657770

gerritbot added a comment.Jan 26 2021, 10:22 AM

Change 657770 merged by Muehlenhoff:
[operations/puppet@production] Enable managed adduser.conf unconditionally

https://gerrit.wikimedia.org/r/657770

Stashbot added a comment.Jun 3 2021, 8:37 AM

Mentioned in SAL (#wikimedia-operations) [2021-06-03T08:37:31Z] <moritzm> upgrading codfw to debmonitor-client 0.3.0 (along with deleting/recreating system user within 100-499 range) T235162

Stashbot added a comment.Jun 3 2021, 9:37 AM

Mentioned in SAL (#wikimedia-operations) [2021-06-03T09:37:26Z] <moritzm> upgrading eqiad to debmonitor-client 0.3.0 (along with deleting/recreating system user within 100-499 range) T235162

MoritzMuehlenhoff added a comment.Jun 3 2021, 10:27 AM
In T235162#6751520, @MoritzMuehlenhoff wrote:

Status update: The puppetised adduser.conf is rolled out, what remains is to fix some of the mismatched UIDs (most of them should be in debmonitor, which will get fixed via a new package using systemd-sysuser).

debmonitor 0.3.0 has been rolled out along with a removal of the pre-existing sysuser , so that the subsequent update recreates the sysuser within the correct range. For buster there's https://phabricator.wikimedia.org/T256098 which had to be worked around by switching back to adduser. I've done a fleet-wide cumin run to validate that all debmonitor sysusers are now in the correct range (there were a few stuck old systemd timers stuck (since they were started before the urllib fix, some dating back to March) which made deluser fail, but I cleaned those up via cumin as well). Now that we are consistently on 0.3.0 we should run into no further issues like that.

Now that debmonitor is fixed, the final thing to do is to audit whether there are other sysusers outrange the allowed range.

MoritzMuehlenhoff added a comment.Jun 3 2021, 11:31 AM
In T235162#7131437, @MoritzMuehlenhoff wrote:

Now that debmonitor is fixed, the final thing to do is to audit whether there are other sysusers outrange the allowed range.

I ran a fleet-wide check on UIDs within the 500 (current upper limit for system users) and 999 (human users start with 1000), along with filtering out the centrally mapped UIDs we maintain in data.yaml (currently 900-911):

  • We have a few users created a long time ago which are in the range: mark/531, ariel/543, awjrichards/552, tstarling/501, aaron/544, catrope/546, daniel/545, ezachte/523
  • 325 system users for mcrouter within the wrong range
  • 1267 system users for systemd-coredump within the wrong range

And then there's a few remaining cases:
On host alert1001.wikimedia.org: user karma has UID 997
On host alert1001.wikimedia.org: user kthxbye has UID 996
On host alert1001.wikimedia.org: user phalerts has UID 995
On host alert2001.wikimedia.org: user karma has UID 997
On host alert2001.wikimedia.org: user kthxbye has UID 996
On host alert2001.wikimedia.org: user phalerts has UID 995
On host an-coord1001.eqiad.wmnet: user mysql has UID 998
On host an-coord1002.eqiad.wmnet: user mysql has UID 998
On host an-test-coord1001.eqiad.wmnet: user mysql has UID 998
On host an-tool1007.eqiad.wmnet: user turnilo has UID 998
On host an-tool1009.eqiad.wmnet: user hue has UID 997
On host dborch1001.wikimedia.org: user orchestrator has UID 997
On host sodium.wikimedia.org: user acme has UID 998
On host sodium.wikimedia.org: user mirror has UID 997
On host thorium.eqiad.wmnet: user stats has UID 999

I'll have a closer look how they are created and will create sub tasks.

Aklapper edited projects, added Patch-Needs-Improvement; removed Patch-For-Review.Feb 21 2023, 10:10 PM
gerritbot added a comment.Feb 22 2023, 12:50 PM

Change 644808 abandoned by Jbond:

[operations/puppet@production] Remove Hiera option to enable adduser config

Reason:

no longer required

https://gerrit.wikimedia.org/r/644808

jbond edited projects, added Puppet, Infrastructure Security; removed Patch-Needs-Improvement, SRE.Feb 22 2023, 12:51 PM
Restricted Application added a project: Infrastructure-Foundations. · View Herald TranscriptFeb 22 2023, 12:51 PM
Aklapper removed MoritzMuehlenhoff as the assignee of this task.Tue, Mar 21, 9:46 AM

@MoritzMuehlenhoff: Removing task assignee as this open task has been assigned for more than two years - See the email sent to task assignee on Feburary 22nd, 2023.
Please assign this task to yourself again if you still realistically [plan to] work on this task - it would be welcome! :)
If this task has been resolved in the meantime, or should not be worked on by anybody ("declined"), please update its task status via "Add Action… 🡒 Change Status".
Also see https://www.mediawiki.org/wiki/Bug_management/Assignee_cleanup for tips how to best manage your individual work in Phabricator. Thanks!

MoritzMuehlenhoff claimed this task.Tue, Mar 21, 12:03 PM
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL