Page MenuHomePhabricator

Wikimedia deployers audit
Closed, ResolvedPublic

Description

The Security-Team recently reviewed membership of the Wikimedia deployment group. We validated a few things such as affiliation (WMF, WMDE) a valid NDA on file with WMF-Legal and the last date an individual performed a deploy according to the SAL and archives. These results are partially generated by a python script (soon to live within wikimedia/security/tooling) and exist within a protected Google sheet. The following is a subset of the aforementioned results and includes users who have not performed a deploy within the last two years. The Security-Team believes that if these individuals do not regularly perform deploys and are unlikely to do so any time in the future, we should remove their deployment rights, given the sensitivity and power of such rights. Some of these users very recently received these rights and should not be considered within this discussion.

Updated 2020-02-25:

Shell usernameNameWMFWMDEWMF Legal NDA?Last DeployedDate Updated
audeKatie Filbert (former WMDE)001No deploys last 2 years2020-02-25 21:06
awjrichardsArthur Richards101No deploys last 2 years2020-02-25 21:06
danielDaniel Kinzler101No deploys last 2 years2020-02-25 21:06
demonChad Horohoe (former WMF)001No deploys last 2 years2020-02-25 21:06
derecksonSébastien Santoro001No deploys last 2 years2020-02-25 21:06
dr0ptp4ktAdam Baso101No deploys last 2 years2020-02-25 21:06
dsharpeDavid Sharpe101No deploys last 2 years2020-02-25 21:06
esandersEd Sanders101No deploys last 2 years2020-02-25 21:06
jdlJason David Linehan101No deploys last 2 years2020-02-25 21:06
jfishbackJames Fishback101No deploys last 2 years2020-02-25 21:06
kaldariRyan Kaldari101No deploys last 2 years2020-02-25 21:06
mattflaschenMatt Flaschen (former WMF)001No deploys last 2 years2020-02-25 21:06
mstylesMaryam Styles101No deploys last 2 years2020-02-25 21:06
musikanimalLeon Ziemba101No deploys last 2 years2020-02-25 21:06
niedzielskiSteven Niedzielski001No deploys last 2 years2020-02-25 21:06
phuedxSam Smith101No deploys last 2 years2020-02-25 21:06
samwilsonSam Wilson101No deploys last 2 years2020-02-25 21:06
wmde-leszekLeszek Manicki011No deploys last 2 years2020-02-25 21:06
zpapierskiZbyszko Papierski101No deploys last 2 years2020-02-25 21:06

@MoritzMuehlenhoff - do you or anyone else in SRE have any opinions on this? The Security-Team would like to perform this audit and make any necessary access adjustments on a quarterly basis.

Event Timeline

sbassett created this task.Nov 7 2019, 11:08 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 7 2019, 11:08 PM
sbassett claimed this task.Nov 7 2019, 11:09 PM
sbassett triaged this task as Medium priority.
sbassett added a project: Security-Team.
sbassett moved this task from Incoming to Watching on the Security-Team board.
sbassett edited subscribers, added: Reedy, JBennett, chasemp, Dsharpe; removed: sbassett.

Sure, no objections. Access to the deployers group is overseen by @greg, so just sync up with him on removing people. When you have a final list of people to remove from the group, just open a Phab task, tag it "Operations" and "SRE-Access-Requests" and whoever is on SRE Clinic Duty the week will deal with removing the access.

sbassett updated the task description. (Show Details)Feb 25 2020, 9:08 PM
sbassett updated the task description. (Show Details)Feb 25 2020, 9:29 PM
sbassett changed the visibility from "Custom Policy" to "Public (No Login Required)".

Change 574869 had a related patch set uploaded (by SBassett; owner: SBassett):
[operations/puppet@production] Deployment group audit

https://gerrit.wikimedia.org/r/574869

@MoritzMuehlenhoff - patch is up, see above. I wanted to make this task public and give those being removed of their deployment rights a chance to respond if 1) they still have good reason to keep deployment 2) my audit script made a mistake combing the SAL.

sbassett updated the task description. (Show Details)Feb 25 2020, 9:39 PM
sbassett moved this task from Backlog to In Progress on the user-sbassett board.
Dzahn added a subscriber: Dzahn.Feb 25 2020, 9:49 PM

So should we add those users as subscribers here on the ticket so that they get notified?

So should we add those users as subscribers here on the ticket so that they get notified?

Done.

sbassett updated the task description. (Show Details)Feb 25 2020, 10:55 PM
kaldari added a comment.EditedFeb 26 2020, 8:26 PM

For myself, I don't do deployments any more (now that I'm managing). I suspect the same is true for Adam.

kaldari updated the task description. (Show Details)Feb 26 2020, 8:27 PM
kaldari updated the task description. (Show Details)

I am in a similar boat as Kaldari. It has been many years since I accidentally took Wikipedia offline :p Please feel free to remove me as a deployer.

I've recently stepped sideways into development from management and will likely be doing more deployments in the future. I can't give an expected date of when I'll next be deploying though.

Your stats are wrong if they think it was more than 2 years since I've deployed:) And I'm not in the deployers group anymore either. Anyway, I was under the impressionthat my account was reenabled to keep me in the WMF-NDA LDAP group, as requested in T238960.

Anyway, I was under the impressionthat my account was reenabled to keep me in the WMF-NDA LDAP group, as requested in T238960.

That's correct, Max is currently only in the cn=nda LDAP group and doesn't have any shell access.

Dzahn added a comment.Feb 27 2020, 4:08 PM

That's correct, Max is currently only in the cn=nda LDAP group and doesn't have any shell access.

But when he still had deployment access he deployed and that is not over 2 years ago. So something went wrong here.

https://gerrit.wikimedia.org/r/q/project:operations%252Fmediawiki-config+owner:maxsem.wiki%2540gmail.com

sbassett added a comment.EditedFeb 27 2020, 4:44 PM

But when he still had deployment access he deployed and that is not over 2 years ago. So something went wrong here.

https://gerrit.wikimedia.org/r/q/project:operations%252Fmediawiki-config+owner:maxsem.wiki%2540gmail.com

Yes, thanks all for the feedback on this. I've removed MaxSem from the table within the task description. I'm guessing this was due in part to a merge of the data when this audit was initially run last November and, as noted, the automated searching of the SAL being a bit buggy. Hence why this task was made public and why I tried to add everyone whose access was to change as a result to this task and the patch.

sbassett updated the task description. (Show Details)Feb 27 2020, 4:44 PM

I've recently stepped sideways into development from management and will likely be doing more deployments in the future. I can't give an expected date of when I'll next be deploying though.

This is fine, I'll add you back on the patch.

I do not need or have a strong desire for deployment rights, however I would like to keep terbium access if possible. Could I be moved to the "restricted" group? My deployer rights were added way back when I was working on IP range contributions and needed to run queries on a new table (T161181). Incidentally, I'm back in the same position with our ongoing work on Expiring-Watchlist-Items. Pinging @aezell in case you need manager approval. Thanks!

sbassett added a comment.EditedFeb 27 2020, 4:51 PM

For myself, I don't do deployments any more (now that I'm managing). I suspect the same is true for Adam.

I am in a similar boat as Kaldari. It has been many years since I accidentally took Wikipedia offline :p Please feel free to remove me as a deployer.

Noted, thanks!

I do not need or have a strong desire for deployment rights, however I would like to keep terbium access if possible. Could I be moved to the "restricted" group?

Sounds reasonable to me, done in PS3.

Dzahn added a comment.Feb 27 2020, 6:50 PM

however I would like to keep terbium access if possible. Could I be moved to the "restricted" group?

FYI, terbium does not exist anymore. use "mwmaint1001.eqiad.wmnet" and "mwmaint2001.codfw.wmnet"

Yes, the restricted group will give you access to that just like deployment does.

@sbassett Hey, thanks for this. Does the script also provide when they acquired their rights? That'd be useful for me to review these from my end. Thanks!

@sbassett Hey, thanks for this.

You're welcome.

Does the script also provide when they acquired their rights? That'd be useful for me to review these from my end.

Not at the moment, but I've just filed T246388.

I've not deployed for a long time, and not often ever, however as the other deployers on my team (Community-Tech) are losing their rights or are generally feeling like they've got too much to do, we think it'd be good for me to re-train for deployments and keep the rights. Is that okay?

I've not deployed for a long time, and not often ever, however as the other deployers on my team (Community-Tech) are losing their rights or are generally feeling like they've got too much to do, we think it'd be good for me to re-train for deployments and keep the rights. Is that okay?

Added you back in PS4.

kaldari removed a subscriber: kaldari.Feb 29 2020, 12:24 AM
chasemp added a subscriber: sbassett.

I'll review the current and merge if things seem up and up, assigning to me so I don't forget

Change 574869 merged by Rush:
[operations/puppet@production] Deployment group audit

https://gerrit.wikimedia.org/r/574869

1Info: Using configured environment 'production'
2Info: Retrieving pluginfacts
3Info: Retrieving plugin
4Info: Retrieving locales
5Info: Loading facts
6Info: Caching catalog for deploy1001.eqiad.wmnet
7Info: Applying configuration version '(ab7b86bead) Rush - Deployment group audit'
8Info: Computing checksum on file /etc/ssh/userkeys/aude
9Info: /Stage[main]/Ssh::Server/File[/etc/ssh/userkeys/aude]: Filebucketed /etc/ssh/userkeys/aude to puppet with sum 8fdeabe0e72af264970cbe66b4cf8486
10Notice: /Stage[main]/Ssh::Server/File[/etc/ssh/userkeys/aude]/ensure: removed
11Info: Computing checksum on file /etc/ssh/userkeys/awjrichards
12Info: /Stage[main]/Ssh::Server/File[/etc/ssh/userkeys/awjrichards]: Filebucketed /etc/ssh/userkeys/awjrichards to puppet with sum 22a2bde854ab1e97f1ab893532000ed1
13Notice: /Stage[main]/Ssh::Server/File[/etc/ssh/userkeys/awjrichards]/ensure: removed
14Info: Computing checksum on file /etc/ssh/userkeys/daniel
15Info: /Stage[main]/Ssh::Server/File[/etc/ssh/userkeys/daniel]: Filebucketed /etc/ssh/userkeys/daniel to puppet with sum c2839f6c641f40d0ebc184b7b2c8673d
16Notice: /Stage[main]/Ssh::Server/File[/etc/ssh/userkeys/daniel]/ensure: removed
17Info: Computing checksum on file /etc/ssh/userkeys/dereckson
18Info: /Stage[main]/Ssh::Server/File[/etc/ssh/userkeys/dereckson]: Filebucketed /etc/ssh/userkeys/dereckson to puppet with sum 4c1ae16c9722911b583fb588676eb303
19Notice: /Stage[main]/Ssh::Server/File[/etc/ssh/userkeys/dereckson]/ensure: removed
20Info: Computing checksum on file /etc/ssh/userkeys/dr0ptp4kt
21Info: /Stage[main]/Ssh::Server/File[/etc/ssh/userkeys/dr0ptp4kt]: Filebucketed /etc/ssh/userkeys/dr0ptp4kt to puppet with sum 8bbc35a7df0f04cce76bfe5b219c297e
22Notice: /Stage[main]/Ssh::Server/File[/etc/ssh/userkeys/dr0ptp4kt]/ensure: removed
23Info: Computing checksum on file /etc/ssh/userkeys/dsharpe
24Info: /Stage[main]/Ssh::Server/File[/etc/ssh/userkeys/dsharpe]: Filebucketed /etc/ssh/userkeys/dsharpe to puppet with sum 4d924d1447bbc0da6cc694855781b945
25Notice: /Stage[main]/Ssh::Server/File[/etc/ssh/userkeys/dsharpe]/ensure: removed
26Info: Computing checksum on file /etc/ssh/userkeys/esanders
27Info: /Stage[main]/Ssh::Server/File[/etc/ssh/userkeys/esanders]: Filebucketed /etc/ssh/userkeys/esanders to puppet with sum a34f3096ad1f33e56aad3899059a003c
28Notice: /Stage[main]/Ssh::Server/File[/etc/ssh/userkeys/esanders]/ensure: removed
29Info: Computing checksum on file /etc/ssh/userkeys/jdl
30Info: /Stage[main]/Ssh::Server/File[/etc/ssh/userkeys/jdl]: Filebucketed /etc/ssh/userkeys/jdl to puppet with sum d2d4f5bc42ba50e95901220598c26d4e
31Notice: /Stage[main]/Ssh::Server/File[/etc/ssh/userkeys/jdl]/ensure: removed
32Info: Computing checksum on file /etc/ssh/userkeys/jfishback
33Info: /Stage[main]/Ssh::Server/File[/etc/ssh/userkeys/jfishback]: Filebucketed /etc/ssh/userkeys/jfishback to puppet with sum a42bf9bf8c515a771353b91244577d66
34Notice: /Stage[main]/Ssh::Server/File[/etc/ssh/userkeys/jfishback]/ensure: removed
35Info: Computing checksum on file /etc/ssh/userkeys/kaldari
36Info: /Stage[main]/Ssh::Server/File[/etc/ssh/userkeys/kaldari]: Filebucketed /etc/ssh/userkeys/kaldari to puppet with sum ecc334a7ee563e863aa215f108fbf5f3
37Notice: /Stage[main]/Ssh::Server/File[/etc/ssh/userkeys/kaldari]/ensure: removed
38Info: Computing checksum on file /etc/ssh/userkeys/mattflaschen
39Info: /Stage[main]/Ssh::Server/File[/etc/ssh/userkeys/mattflaschen]: Filebucketed /etc/ssh/userkeys/mattflaschen to puppet with sum aa5a55e32b05184a9f8a8d74469225f8
40Notice: /Stage[main]/Ssh::Server/File[/etc/ssh/userkeys/mattflaschen]/ensure: removed
41Info: Computing checksum on file /etc/ssh/userkeys/musikanimal
42Info: /Stage[main]/Ssh::Server/File[/etc/ssh/userkeys/musikanimal]: Filebucketed /etc/ssh/userkeys/musikanimal to puppet with sum 3665b28ca17c2e4f81b3e3cb419c2fba
43Notice: /Stage[main]/Ssh::Server/File[/etc/ssh/userkeys/musikanimal]/ensure: removed
44Info: Computing checksum on file /etc/ssh/userkeys/wmde-leszek
45Info: /Stage[main]/Ssh::Server/File[/etc/ssh/userkeys/wmde-leszek]: Filebucketed /etc/ssh/userkeys/wmde-leszek to puppet with sum b8ee1749697d6718faf0948ba72839d2
46Notice: /Stage[main]/Ssh::Server/File[/etc/ssh/userkeys/wmde-leszek]/ensure: removed
47Notice: /Stage[main]/Admin/Admin::Groupmembers[deployment]/Exec[deployment_ensure_members]/returns: executed successfully
48Notice: /Stage[main]/Admin/Exec[enforce-users-groups-cleanup]/returns: /usr/local/sbin/enforce-users-groups removing user/id: dr0ptp4kt/2962
49Notice: /Stage[main]/Admin/Exec[enforce-users-groups-cleanup]/returns: /usr/local/sbin/enforce-users-groups removing user/id: aude/1185
50Notice: /Stage[main]/Admin/Exec[enforce-users-groups-cleanup]/returns: /usr/local/sbin/enforce-users-groups removing user/id: esanders/2875
51Notice: /Stage[main]/Admin/Exec[enforce-users-groups-cleanup]/returns: /usr/local/sbin/enforce-users-groups removing user/id: kaldari/1271
52Notice: /Stage[main]/Admin/Exec[enforce-users-groups-cleanup]/returns: /usr/local/sbin/enforce-users-groups removing user/id: mattflaschen/2662
53Notice: /Stage[main]/Admin/Exec[enforce-users-groups-cleanup]/returns: /usr/local/sbin/enforce-users-groups removing user/id: dereckson/2362
54Notice: /Stage[main]/Admin/Exec[enforce-users-groups-cleanup]/returns: /usr/local/sbin/enforce-users-groups removing user/id: musikanimal/11106
55Notice: /Stage[main]/Admin/Exec[enforce-users-groups-cleanup]/returns: /usr/local/sbin/enforce-users-groups removing user/id: jdl/20295
56Notice: /Stage[main]/Admin/Exec[enforce-users-groups-cleanup]/returns: /usr/local/sbin/enforce-users-groups removing user/id: wmde-leszek/12300
57Notice: /Stage[main]/Admin/Exec[enforce-users-groups-cleanup]/returns: /usr/local/sbin/enforce-users-groups removing user/id: dsharpe/20828
58Notice: /Stage[main]/Admin/Exec[enforce-users-groups-cleanup]/returns: /usr/local/sbin/enforce-users-groups removing user/id: jfishback/21257
59Notice: /Stage[main]/Admin/Exec[enforce-users-groups-cleanup]/returns: executed successfully
60Notice: Applied catalog in 92.07 seconds

back to scott for confirmation and whatever else :)

sbassett closed this task as Resolved.Mar 2 2020, 7:19 PM
sbassett moved this task from In Progress to Our Part Is Done on the Security-Team board.

@chasemp - LGTM, thanks for the deploy! I'll plan to track any access restoration requests that might trickle in.

sbassett moved this task from In Progress to Done on the user-sbassett board.Mar 2 2020, 7:19 PM

@chasemp - LGTM, thanks for the deploy! I'll plan to track any access restoration requests that might trickle in.

@sbassett There is one at T258119

@chasemp - LGTM, thanks for the deploy! I'll plan to track any access restoration requests that might trickle in.

@sbassett There is one at T258119

Thanks, @Dzahn - looks like that has now been resolved.