Page MenuHomePhabricator

Onboarding Mathew Onipe
Closed, ResolvedPublic

Description

  • add to wmf LDAP group
  • add to ops LDAP group
  • gerrit login
  • +2 on operations/puppet
  • phabricator login (@Mathew.onipe)
  • phabricator permissions to see NDA and Ops restricted tickets
  • shell user (connecting to bastions)
  • server root shell (on relforge/elasticsearch hosts)
  • add to private IRC channels
  • add to ops mailing lists
  • add to exim mail aliases
  • icinga login (this is auto-solved by the "add to wmf LDAP group" part which also gives a lot more logins listed at https://wikitech.wikimedia.org/wiki/LDAP/Groups#Specific_groups)
  • icinga user and permissions (icinga commands, paging/notifications)
  • access to pwstore

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 24 2018, 7:02 AM
Gehel renamed this task from Add Mathew.onipe to #wmf-nda to Onboarding Mathew Onipe.Aug 24 2018, 7:58 AM
Gehel triaged this task as Medium priority.
Gehel edited projects, added Operations; removed WMF-NDA-Requests.
Gehel updated the task description. (Show Details)
Gehel updated the task description. (Show Details)Aug 24 2018, 9:04 AM
Gehel added a comment.Aug 24 2018, 9:10 AM

Note: @Mathew.onipe does not have an @wikimedia.org email yet. Some of the checklist items above would make more sense with an @wikimedia.org email (like exim email aliases), so those might be delayed a bit.

Volans added a subscriber: Volans.Aug 27 2018, 4:47 PM
Gehel added a comment.Aug 27 2018, 4:51 PM

It is not entirely clear what access we want to give @Mathew.onipe at this point.

Constraints:

  • Matt is a contractor with a more junior profile than our usual Opsen
  • We need Matt to be able to work on at least elasticsearch and wdqs clusters, including the reimaging those clusters
  • Our current flat model is reaching its limit, is now the right time to think about changing it (without blocking Matt from working while we change it)

It is not entirely clear what access we want to give @Mathew.onipe at this point.

Constraints:

  • Matt is a contractor with a more junior profile than our usual Opsen
  • We need Matt to be able to work on at least elasticsearch and wdqs clusters, including the reimaging those clusters
  • Our current flat model is reaching its limit, is now the right time to think about changing it (without blocking Matt from working while we change it)

We have used wmcs-roots to grant a subset of roles to particular users who really only operate within the context of cloud services and it has seemed to work well. If we created a search-roots (or something) and applied it to a handful of roles it seems like this may be sufficient? This isn't me objecting to adding this user to the broader ops group, but it sounds like some smaller scope may be appropriate if it's reasonably practical to achieve in this case for now. It would be fairly quick/easy to create.

Gehel added a comment.Aug 29 2018, 1:49 PM

Summarizing a few back channel conversations here:

  • the current thinking is to start by giving @Mathew.onipe a few already existing roles (elasticsearch-roots, wdqs-admins)
  • there are a few operations where we don't have good ways to restrict access by cluster (reimaging servers, access to remote management consoles, ...), so at some point we'll need to provide larger accesses
  • while @Mathew.onipe has less experience than most our SRE he has experience working on production infrastructure
Dzahn added a subscriber: Dzahn.Aug 29 2018, 2:12 PM

(reimaging servers, access to remote management consoles, ...), so at some point we'll need to provide larger accesses

Access to remote management consoles only needs bastion access and having the mgmt password.

The datacenter-ops admin group allows running puppet and install-console which allows for reimaging servers.

Dzahn added a comment.Aug 29 2018, 2:17 PM

Some of the checklist items above would make more sense with an @wikimedia.org email (like exim email aliases), so those might be delayed a bit.

It's basically all of them. Onboarding always starts with an email address provided by OIT. Gerrit/Phab/LDAP/shell/mailing lists/icinga all rely on that.

If we expect that at some point Matt will do general ops work for us (example: clinic duty), we want to make sure there is a path forward that provides him with the access to do that. Or decide to give such access now. I don't think the matter of his being 'junior' is at issue; do we trust him to be careful as a root user? Has he had root on a production cluster in his previous ops-related work? I understand the answer to the last question is 'yes'. Does he have the requisite knowledge to be careful? Again my understanding from irc chats is that the answer to that question is yes.

It's a week later; do we want to talk about this at the SRE meeting or can we come to some sort of agreement here on the task?

Dzahn added a comment.Sep 6 2018, 3:07 PM

What about the email address? Are we still waiting for that?

EBjune added a subscriber: EBjune.Sep 7 2018, 5:54 PM

@Dzahn the email address was granted last week, thanks for checking up on that!

Dzahn updated the task description. (Show Details)Sep 7 2018, 6:09 PM

Thanks @EBjune ! I fixed the mailing list situation first. subscribed to ops and ops-private with the wikimedia.org address (and removed the former one where it existed to avoid duplicates). Also talking to Matt on IRC about it.

Dzahn updated the task description. (Show Details)Sep 7 2018, 6:11 PM

Mentioned in SAL (#wikimedia-operations) [2018-09-07T18:12:34Z] <mutante> LDAP: added user 'monipe' to group 'wmf' (T202708)

Mentioned in SAL (#wikimedia-operations) [2018-09-07T18:20:21Z] <mutante> LDAP: correction, 'monipe' replaced with 'onimisionipe' in wmf group (T202708)

Dzahn added a comment.Sep 7 2018, 6:23 PM

Added to "wmf" LDAP group, which gives these permissions, including Icinga:

https://wikitech.wikimedia.org/wiki/LDAP/Groups#Specific_groups

"ops" LDAP group is separate though since that needs to match the shell access group which is still pending, hence i made it a separate checkbox

Dzahn updated the task description. (Show Details)Sep 7 2018, 6:35 PM

Logstash, Tendril, Graphite, Grafana, Icinga, Piwik should all work now

Dzahn updated the task description. (Show Details)Sep 7 2018, 6:40 PM

SSH Keys
Production:
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCzc2xF3S5UnIFMnK0gbGBhYvsN+xjiJejtEZUnGb23vJTX7N955C7dBdHkTHpcV7+yWqpzzWkJCpnRs5Q0P+JyQ5hOikv7WrKcsjIODuUpMzkRIlWzmGwuA6fXvJqAcyqSdXeQAAczUnlItMl0BB0L0LsB5xY7aqkt0atm1CPkQcFKBc90KJiW8Tkuh5MiYIXe18o+mCI/Q+yPfUxaqQZ0rp9pmFk1L021D3BL7YNpTZYSwbulnxc/y++VD1Ot/2kmCX2HhB9APVP36VZwqDcb+Ik2ZMsrtxIBz2qQjsbUnZvS9rqqJrF8MQhLmfe/M1gK8pbR0yV8DXHSSG3uJ1Db matt@matt

cloud:
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/YXPi5kmu067j+YF5sSDkdH7+YDtu4aT5I58IREQmWqMzOj5zpIdbIGsEeP5wGJimqE9SQD9vokS8jO5H/d+3ZSZz1qU2lU1cmx6CHDm7vL/tOPgBgscn4TipvX1XkQ8CfEBddVe5jL9Y3GLX5EgxuNFXsMmpE8KpHQ/nSmOoxVIDAeqFgfNJSHccw7PyqVjZIdxmmU4hYq4KtcmnrETHJ6SPFigzR+J4kMAq97/9+LxY68vwzbhQF1O2yHmTLASrxIg2vQaI07d8+5N6lewmwXPt++Bkg8pT4IZOdNn24YtiBceK3Zi0w1Yki5H3tPTkppfk2GSaxROmyvFdoWt matt@matt

Change 458877 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] admins: create shell user for Mathew Onipe

https://gerrit.wikimedia.org/r/458877

Gehel added a comment.Sep 7 2018, 7:50 PM

Thanks @Dzahn to move this forward! I was stalling this for too long.

For the next steps:

  • I'll propose adding @Mathew.onipe to elasticsearch-roots and wdqs-admins during next ops weekly
  • I propose to wait until @Mathew.onipe has worked on a few puppet patches before giving him +2 on puppet
  • I propose to wait until we have an actual need to give him access to pwstore

@Mathew.onipe is not blocked by this in his current work, so let's not rush things more than we need.

Change 459556 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] admins: add Mathew Onipe as member of elasticsearch-roots and wdqs-admins

https://gerrit.wikimedia.org/r/459556

Gehel added a comment.Sep 10 2018, 4:52 PM

Shell access and membership to elasticsearch-roots and wdqs-admins has been approved in weekly SRE meeting.

Change 458877 merged by Gehel:
[operations/puppet@production] admins: create shell user for Mathew Onipe

https://gerrit.wikimedia.org/r/458877

Change 459556 merged by Gehel:
[operations/puppet@production] admins: add Mathew Onipe as member of elasticsearch-roots and wdqs-admins

https://gerrit.wikimedia.org/r/459556

@Mathew.onipe has access to the elastic and wdqs clusters, which is what we need at the moment. We'll reopen specific tasks for specific access as needed.

Dzahn added a comment.Sep 18 2018, 4:19 PM

@Mathew.onipe Let's meet on IRC and finish the Icinga part together if you like.

Change 461166 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] icinga: give privs to run commands to Matt Onipe

https://gerrit.wikimedia.org/r/461166

Change 461166 merged by Dzahn:
[operations/puppet@production] icinga: give privs to run commands to Matt Onipe

https://gerrit.wikimedia.org/r/461166

Dzahn updated the task description. (Show Details)Sep 18 2018, 6:37 PM
Dzahn closed this task as Resolved.Sep 18 2018, 6:49 PM

To the best of my knowledge this is resolved now.

We went through the Icinga part and added permissions and confirmed they work.

I updated the checkboxes that were already done and clarified (ssh to bastions, ssh and root on elastic/relforge). Matt confirmed he can get root on elastic and relforge machines.

The remaining boxes have been declined for not being needed for now.

The former comments from Gehel also confirm this and i see it was moved to a "Done" column, so resolving it here as well. Let me know if i'm missing anything. Matt also said he is good for now and would open a new ticket if he runs into something blocking him.