Page MenuHomePhabricator

NDA, MOU and LDAP (analytics cluster) for Shilad Sen
Closed, ResolvedPublic

Description

Dr. Sen has offered to do the work to implement a productionized workflow for Clickstream data and Navigation vectors. This task is to get him the requisite access to be able to start work on that. I've talked to @Nuria and she's interested in having Analytics work with Dr. Sen once @JAllemandou is back to work in mid-August.

This task is done when: we have a MOU and signed NDA for @Shilad and he has access to the analytics cluster hosts.

Target duration: 3 months

  • 1 month for primary work (if we're very lucky)
  • 2 months for ongoing maintenance and incidental work

Point of contact: @Halfak

Shilad's wikitech user: https://wikitech.wikimedia.org/wiki/Shell_Request/Shilad_Sen
Groups: researchers statistics-privatedata-users statistics-users

  • MOU and NDA drafted
  • Official documents sent out for signature
  • Documents signed and filed
  • shell access set up for accessing stat100* (specifically so Dr. Sen can work with hive/set up oozie jobs)

ops access request checklist

  • - @Shilad has reviewed and signed L3
  • - has nda on file with WMF legal (@RobH confirmed this via checking the NDA spreadsheet on 2017-08-17. User's NDA is valid until 2018-02-17. when access patchset is created, set expiry to that date, and set to email @Halfak
  • - @Shilad provides us with a preferred shell username, email address, and a public SSH key that is dedicated ONLY to WMF production access. This needs to be a different ssh key than used in labs.
    • username: shiladsen
    • email: shilad at gmail.com
    • public ssh key: T171988#3532244
  • - WMF manager approval (this is granted via the fact the manager @Halfak filed the task)
  • - patchset created with the above and groups researchers statistics-privatedata-users statistics-users
  • - 3 business day wait passes without complaint (ends on 2017-08-22)

Event Timeline

Halfak created this task.Jul 28 2017, 7:36 PM
Halfak created this object in space Restricted Space.
Halfak created this object with visibility "acl*research_collaborations_policy_admins (Project)".
Restricted Application added a project: Research-collaborations. · View Herald TranscriptJul 28 2017, 7:36 PM
Halfak shifted this object from the Restricted Space space to the S1 Public space.
Halfak removed subscribers: ellery, Tbayer, Ladsgroup.
Halfak removed a subscriber: Shilad.
Halfak added a subscriber: srodlund.
Halfak updated the task description. (Show Details)Jul 28 2017, 9:33 PM
DarTar moved this task from Staged to In Progress on the Research board.Jul 31 2017, 6:27 PM
Nuria added a comment.Aug 7 2017, 4:15 PM

Please have in mind that this work is quite low priority at this time for the many other things we need to do, it is likely that we have availability for this a bit later, September. Let's please make sure NDA and MOU have an expiration date.

Nuria moved this task from Incoming to Radar on the Analytics board.Aug 7 2017, 4:15 PM
Nuria added a comment.Aug 7 2017, 5:25 PM

Approved on our end.

Halfak added a comment.Aug 7 2017, 6:21 PM

@Nuria, you'd originally told me mid-Aug for a start. What pushed this back to mid-Sept and would it be OK if I worked with @Shilad to get started in mid-Aug anyway?

Nuria added a comment.Aug 7 2017, 7:31 PM

@Halfak : starting with @Shilad mid august on your end sounds fine

We are not so precise as to have everything working exactly as expected at all times,
unexpected stuff gets added to our backlog and a 2 week slide on dates is not unheard of.
Having two people out makes it even harder to plan for,

Some details: besides some operational stuff there are couple projects that have higher priority than this one that I thought they will be on their way by this point but due to some issues are not:

Halfak renamed this task from NDA and MOU for Shilad Sen to NDA, MOU and LDAP (analytics cluster) for Shilad Sen.Aug 17 2017, 6:47 PM
Halfak added a project: LDAP-Access-Requests.
Halfak updated the task description. (Show Details)Aug 17 2017, 6:53 PM
Halfak updated the task description. (Show Details)
Halfak updated the task description. (Show Details)Aug 17 2017, 6:57 PM
Halfak updated the task description. (Show Details)Aug 17 2017, 8:21 PM

The NDA/MOU documents have been signed and filed. We're ready to get LDAP set up.

Halfak changed the visibility from "Custom Policy" to "All Users".Aug 17 2017, 8:25 PM
Halfak updated the task description. (Show Details)
Halfak added a subscriber: Shilad.

@RStallman-legalteam can you confirm the NDA status for this user Shilad Sen?

I approve this. Let's set the end date to Nov. 30th. Email me (ahalfaker at wikimedia.org) when the end date is reached.

RobH added a comment.Aug 17 2017, 8:35 PM

@RStallman-legalteam can you confirm the NDA status for this user Shilad Sen?

revoked my request, i see him on the sheet you shared with me before!

Restricted Application added a project: Operations. · View Herald TranscriptAug 17 2017, 8:40 PM
Halfak updated the task description. (Show Details)Aug 17 2017, 8:41 PM
Halfak updated the task description. (Show Details)Aug 17 2017, 8:44 PM
RobH updated the task description. (Show Details)Aug 17 2017, 8:48 PM
RobH removed a subscriber: RStallman-legalteam.
Halfak changed the visibility from "All Users" to "Public (No Login Required)".Aug 17 2017, 8:54 PM
RobH assigned this task to Shilad.Aug 17 2017, 8:58 PM

Assigned to @Shilad for them to sign L3, and provide preferred shell username, wikitech username, and a production ssh pub key. Once that is done, please feel free to assign back to me and I'll prepare the patchset and watch for objection/merge window next tuesday.

Thanks!

@RobH, I've signed the L3. My wikitech username is "Shilad Sen" and my preferred shell username is shiladsen. I created a new public SSH key for the production environment, and it is below I think that should be everything!

ssh-dss AAAAB3NzaC1kc3MAAACBAKBspbywXptKB4djp8jYjfk0fAQUAhsEM03zvRhuCpIwB5BYQl2mIeIwADHqM5DA0plGtFZLLwZvFR/LpHIiK3zcDuvz5N6LBkTulKQ5TrjnMkAeTk1SA900u6jCoKitF7j6ZO3Q4diLgFSY5F4EJI80GiWkOx+JAnzhS3kHbkibAAAAFQDzFcnzFRA7bawBb0ZVhCYDU2v+2wAAAIBzWSGg2rEvV0UT+cDzGZMl6LGWT+3oC1pJviW8vilOhIKvdbXYeQeGpqpJjxZToN/5Ok+P0kAMNTacdPWyYiDDepb+zgB9tbW+DPB3HgH2y6u7SMNWnOXK+C9VAT62LEX4zQsD41NC3kMijDjLuAzAkyKPAVmgtFWCXpYDDU/+zgAAAIAW66EVt/6tp7o6GlfU3TS3JnYLA3cFzWqmbuHuV2dFhW3h7OAbmCRivhOVuhJuu56C/AJeKdGzIA10p/eo39YXUX3iOjUTO8/YFFAAnh9m4Fb1YDTMG3JzwBi8jT6r8iOm9414ITX48y9zzD3smXku3o3At/w5Up6rl/lDeywI4g== a558989@600308a4c4c6

Shilad reassigned this task from Shilad to RobH.Aug 17 2017, 9:36 PM
Halfak updated the task description. (Show Details)Aug 18 2017, 9:38 PM
Halfak updated the task description. (Show Details)Aug 18 2017, 9:40 PM
herron added a subscriber: herron.Aug 21 2017, 5:17 PM
RobH reassigned this task from RobH to herron.Aug 21 2017, 6:10 PM

So @herron is on clinic duty this week, and expressed an interest in taking care of this request. He'll prepare a patchset for review and merge tomorrow (if no objections are noted, none are as of yet.)

Change 373115 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] Add shiladsen shell account

https://gerrit.wikimedia.org/r/373115

Change 373115 merged by Herron:
[operations/puppet@production] Add shiladsen shell account

https://gerrit.wikimedia.org/r/373115

Shell account shiladsen has been added to puppet and deployed to stat systems:

stat1003:~$ id shiladsen
uid=10339(shiladsen) gid=500(wikidev) groups=500(wikidev),726(statistics-users),714(researchers),725(statistics-privatedata-users)

@Shilad could you please confirm that the new shell account is working as expected?

Thanks in advance!

herron updated the task description. (Show Details)Aug 22 2017, 8:52 PM

Thanks @herron.

@Shilad: Aaron is currently traveling, but let me know if you need any assistance. If we have an entry on Meta for this project I can link it from here. I guess we could just reuse and expand the previous Meta pages, what do you think?

@herron I am having some trouble logging in. I can get to bastion but not beyond. I'm suspicious that the key I gave you is a DSS key, not a DSA key. I requested a DSA key, but the public key starts with ssh-dss instead.

Here's the relevant part of my .ssh_config:

Host bast1001.wikimedia.org
  # Direct connection for the bastion host
  ProxyCommand none
  ControlMaster auto

Host *.wikimedia.org *.wmnet !gerrit.wikimedia.org !git-ssh.wikimedia.org
  User shiladsen
  # Everything else goes via bastion acting as a proxy
  ProxyCommand ssh -a -W %h:%p bast1001.wikimedia.org
  # Do not offer other identities loaded in ssh-agent
  IdentitiesOnly yes
  IdentityFile ~/.ssh/wmf_prod

## Internal Zones
Host *.wmnet
    User shiladsen
    IdentitiesOnly yes
    IdentityFile ~/.ssh/wmf_prod

Host *.eqiad.wmnet
    ProxyCommand ssh -a -W %h:%p bast1001.wikimedia.org

And the log for ssh -v:

$ ssh -v shiladsen@stat1005.eqiad.wmnet 
OpenSSH_7.4p1, LibreSSL 2.5.0
debug1: Reading configuration data /Users/a558989/.ssh/config
debug1: /Users/a558989/.ssh/config line 1: Applying options for *
debug1: /Users/a558989/.ssh/config line 18: Applying options for *.wmnet
debug1: /Users/a558989/.ssh/config line 27: Applying options for *.wmnet
debug1: /Users/a558989/.ssh/config line 32: Applying options for *.eqiad.wmnet
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Executing proxy command: exec ssh -a -W stat1005.eqiad.wmnet:22 bast1001.wikimedia.org
debug1: identity file /Users/a558989/.ssh/wmf_prod type 2
debug1: key_load_public: No such file or directory
debug1: identity file /Users/a558989/.ssh/wmf_prod-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.4
debug1: permanently_drop_suid: 233839612
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.4p1 Debian-10+deb9u1
debug1: match: OpenSSH_7.4p1 Debian-10+deb9u1 pat OpenSSH* compat 0x04000000
debug1: Authenticating to stat1005.eqiad.wmnet:22 as 'shiladsen'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256@libssh.org
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:Sf1aEKXiaKLrmTkAWZbn+8N66xstSQx51mGBmYN9HRw
debug1: Host 'stat1005.eqiad.wmnet' is known and matches the ECDSA host key.
debug1: Found key in /Users/a558989/.ssh/known_hosts:343
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,keyboard-interactive
debug1: Next authentication method: publickey
debug1: Offering DSA public key: /Users/a558989/.ssh/wmf_prod
debug1: Authentications that can continue: publickey,keyboard-interactive
debug1: Next authentication method: keyboard-interactive

I think it may be worth changing my key to the following new RSA key (which I just newly created). Does that seem reasonable to you? If so, how can I go about doing this?

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrSCbT7GL3eYqdIISBaajhusTPvXhd/sK3ANdU46qzIbvRADGJjB1/OAIbq8NlWChv8RV2ahA409tphVJl2cKmu6a8++V4s1K73yLuASJR3t5q6M3vfZqHEncR6nPLc6aO5S8Vc76cV4DFFfeW83M5F4d43AvunRnzXakFODdWwwFtyqRHyc5eHUf++X2E8ghZ66MKOQ+9Q0RoBL+9d2vQvk5IFDR7FahIFX8eR5jNIQOQ9nf517IDQ8c2dt5+3MrZC+rux384txKsFV3ytJmctxlqaDJeLFRHweumNrE8pSYif+MfZNM6QLPW8kSjifZBekrWL2smdk5RncORbd5j a558989@600308a4c4c6

Sorry for the trouble, and thanks for your help!

Change 373177 had a related patch set uploaded (by Jeremyb; owner: Jeremyb):
[operations/puppet@production] shiladsen shell: try RSA key instead, add expiry

https://gerrit.wikimedia.org/r/373177

that's maybe exactly the problem.

your debug log says

debug1: match: OpenSSH_7.4p1 Debian-10+deb9u1 pat OpenSSH* compat 0x04000000
debug1: Authenticating to stat1005.eqiad.wmnet:22 as 'shiladsen'

cf. bast1001:

$ nc -v bast1001.wikimedia.org 22
bast1001.wikimedia.org [208.80.154.149] 22 (ssh) open
SSH-2.0-OpenSSH_6.7p1 Debian-5+deb8u3

different OpenSSH server versions, more recent disallows that algorithm. I think. see http://www.openssh.com/legacy.html

here's a patch with your new key (not merged yet): https://gerrit.wikimedia.org/r/373177

Dzahn added a subscriber: Dzahn.Aug 23 2017, 4:46 AM

I can confirm this. The reason is the key type DSS. From auth.log on stat1005:

85335 Aug 23 03:10:05 stat1005 sshd[28673]: userauth_pubkey: key type ssh-dss not in PubkeyAcceptedKeyTypes [preauth]

So that RSA key should do it.

Please don't add new DSA keys, we're down to two keys of that kind and I'm planning to remove server-side support at some point not too far away as well.

Change 373326 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] Change shiladsen ssh key

https://gerrit.wikimedia.org/r/373326

Change 373326 merged by Herron:
[operations/puppet@production] Change shiladsen ssh key

https://gerrit.wikimedia.org/r/373326

Hi @Shilad, your ssh key has been updated. Are you able to log in?

Aug 23 17:54:50 stat1005 puppet-agent[4323]: (/Stage[main]/Admin/Admin::Hashuser[shiladsen]/Admin::User[shiladsen]/Ssh::Userkey[shiladsen]/File[/etc/ssh/userkeys/shiladsen]/content) content changed '{md5}7683d27500c565f522b16ab551d10783' to '{md5}4b0c57645187d5d5d908586041818d55'

Yes! I updated Help:SSH to indicate that DSA is being phased out.

Thanks for your help, @herron!

herron closed this task as Resolved.Aug 23 2017, 8:36 PM

Great! Glad to hear it!

One follow-up: The Navigation Vectors project uses Hive queries, so I think I also need the analytics-privatedata-users role. Is this correct? If so, should I start a new ticket, or can that also be added to this ticket?

Thanks for your help, everybody!

Restricted Application added a subscriber: jeblad. · View Herald TranscriptAug 25 2017, 9:06 AM

Change 373177 abandoned by Dzahn:
shiladsen shell: try RSA key instead, add expiry

Reason:
this is a duplicate now. already done.

https://gerrit.wikimedia.org/r/373177

Dzahn reopened this task as Open.Aug 26 2017, 4:08 AM

@RobH @herron Please see above, reopened because of the follow-up questions from Shilad.

Ya, pretty sure this will need analytics-privatedata-users.

I'm on clinic duty now, this has already been approved for other groups, and the intention of this ticket is to work on clickstream productionization, which I'm pretty sure is a Hadoopy task. I'm going to go ahead and merge analytics-privatedata-users access.

Change 374379 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Add shiladsen to analytics-privatedata-users

https://gerrit.wikimedia.org/r/374379

Change 374379 merged by Ottomata:
[operations/puppet@production] Add shiladsen to analytics-privatedata-users

https://gerrit.wikimedia.org/r/374379

Shilad closed this task as Resolved.Aug 28 2017, 8:41 PM

Everything looks good now! Thanks for your quick help, @Ottomata! I'm going to close this ticket and get to work :)

Added the user shiladsen to the nda LDAP group to allow yarn/pivot/etc.. access.

Indeed, I now have Yarn access! Thanks @elukey!