Request creation of labs-vpn labs project
Open, Stalled, Needs TriagePublic

Description

Project Name: labs-vpn
Purpose: Set up a VPN service for Labs users to use to access Labs-internal hosts easier and quicker. Discussed in Cloud-Services on IRC.
Wikitech Username of requestor: Tom29739

Note: will need a public IP (at some point, but would be useful to have to test).

See https://wikitech.wikimedia.org/wiki/Labs_VPN_proposal for additional details and discussion

tom29739 created this task.Aug 25 2016, 9:40 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 25 2016, 9:40 PM
tom29739 updated the task description. (Show Details)Aug 25 2016, 9:45 PM

Discussed in... what? Your text cut off there @tom29739 :)

tom29739 updated the task description. (Show Details)Aug 25 2016, 9:52 PM
AlexMonk-WMF added a comment.EditedAug 25 2016, 10:10 PM

Logs in http://bots.wmflabs.org/~wm-bot/logs/%23wikimedia-labs/20160825.txt from 21:17:49 to 21:44:10
Key things:

  • Would be for windows users who can't easily do simple SSH tunnelling?
  • Would need to be restricted to only routing connections to Wikimedia IPs (10.0.0.0/8, maybe 208.80.155.128/25?) only, no bouncing outside.
  • Would we need to restrict who can be an admin / sudo in this project, like bastion? I imagine things like MySQL credentials would be going through it without end-to-end encryption.

To recap some of the arguments from Cloud-Services:

  • ProxyCommand is complex for windows users; a VPN would allow a much more user-friendly connection (directly connecting to e.g. tools-exec-1201.eqiad.wmflabs from putty)
  • Corollary: currently bastions need to have shell access to allow windows users to more easily jump through. This could be removed.
  • Developing locally becomes easier due to direct access to replicas (instead of ssh tunneling and connecting to localhost:someport, connect to enwiki.labsdb)

Main things that would need to be figured out:

  • What VPN software?
  • Sane configuration, not allowing traffic through labs (e.g. user -> labs -> google)
  • Authentication (some form of public/private key? preferrably linked to existing ssh keys)
tom29739 added a comment.EditedAug 25 2016, 10:19 PM

OpenVPN

There is other VPN software available of course, but I have had a good experience with OpenVPN.

  • Would only carry traffic to *.wmflabs hosts, not to internet.

You'd need labsdb/wmnet for database replica access. I'm not quite sure how DNS would work actually - the client would need to be able to resolve these, but then all their own DNS lookups would be going through labs?

You'd need labsdb/wmnet for database replica access. I'm not quite sure how DNS would work actually - the client would need to be able to resolve these, but then all their own DNS lookups would be going through labs?

The OpenVPN client would need to be configured to only route certain traffic through the VPN, otherwise the user's internet connection would not work. We could provide .ovpn files for this purpose.

Sane configuration, not allowing traffic through labs (e.g. user -> labs -> google)

Note that it is possible to configure openvpn so it only provides routes to a subrange (ie. labs hosts). It could be blocked anyway¹ but that would require the user misconfigurating his routes.

¹ Not sure if it is worth it, since labs users already can access the internet through their instances.

Sane configuration, not allowing traffic through labs (e.g. user -> labs -> google)

Note that it is possible to configure openvpn so it only provides routes to a subrange (ie. labs hosts). It could be blocked anyway¹ but that would require the user misconfigurating his routes.

¹ Not sure if it is worth it, since labs users already can access the internet through their instances.

https://wikitech.wikimedia.org/wiki/Wikitech:Labs_Terms_of_use#What_uses_of_Labs_do_we_not_like.3F - #5

Sane configuration, not allowing traffic through labs (e.g. user -> labs -> google)

Note that it is possible to configure openvpn so it only provides routes to a subrange (ie. labs hosts). It could be blocked anyway¹ but that would require the user misconfigurating his routes.

¹ Not sure if it is worth it, since labs users already can access the internet through their instances.

We don't want Labs turning into a free VPN service (explicitly prohibited by the Labs ToU).

OpenVPN server can be configured to only provide routes to certain IP ranges/hosts

"can" as in "are technically able to"

So, if it is hard to misconfigure the vpn, we could simply trust our users to do the right thing™

OpenVPN server can be configured to only provide routes to certain IP ranges/hosts

That's the configuration I was describing

Doesn't the ssh of Cygwin / MinGW / MSYS (not an expert here) work?

Doesn't the ssh of Cygwin / MinGW / MSYS (not an expert here) work?

Yes, but it's a big pain to set up + it doesn't work reliably.

bd808 updated the task description. (Show Details)Sep 9 2016, 3:10 PM

Some considerations if you want to go down the VPN road:

  • VPN should be at least as secure as our current SSH setup or better
  • We're talking about user vpn's here, not site-to-site vpn's
  • VPN's are generally either ipsec or ssl based.
  • Ipsec is generally more open standard: different vendors and different programs can communicate with each other (see https://en.wikipedia.org/wiki/IPsec)
  • SSL based VPN's are usually proprietary. They use open standard (SSL), but you need their client to communicate with their server. So you might end up having multiple clients on your machine if you use multiple services.
  • VPN should be split tunnel (see https://en.wikipedia.org/wiki/Split_tunneling): Only push the rang(es) you want users to access directly
  • Pick the right cypher(s): Plenty of outdated insecure stuff out there
  • Terminate the VPN on a firewall: Deny everything and only allow the kind of traffic you want to allow.
  • Don't even consider starting this without two-factor authentication.
  • I would integrate it with the central LDAP to keep users in one place. Having multiple LDAP instances is a PITA and a security incident waiting to happen. AFAIK the central LDAP setup already provides two-factor authentication
  • I would integrate it with the central LDAP to keep users in one place. Having multiple LDAP instances is a PITA and a security incident waiting to happen. AFAIK the central LDAP setup already provides two-factor authentication

Nope, just password-based auth. The 2FA you see on wikitech and horizon comes from the wikitech database on silver, which won't be accessible by this.

chasemp changed the task status from Open to Stalled.Sep 12 2016, 3:55 PM
chasemp added a subscriber: chasemp.

Please make a task for the discussion of https://wikitech.wikimedia.org/wiki/Labs_VPN_proposal and then when it's agreed we can move forward this ticket would be applicable. We'll need to run it by legal among other criteria.

scfc added a subscriber: scfc.Nov 24 2016, 1:23 PM

To recap some of the arguments from Cloud-Services:

  • ProxyCommand is complex for windows users; a VPN would allow a much more user-friendly connection (directly connecting to e.g. tools-exec-1201.eqiad.wmflabs from putty) […]

Is the configuration really that complex? https://wikitech.wikimedia.org/wiki/Help:Access_to_instances_with_PuTTY_and_WinSCP#How_to_set_up_PuTTY_for_proxying_through_bastion.wmflabs.org_to_your_instance looks tedious to repeat for every instance, but fairly simple.

To recap some of the arguments from Cloud-Services:

  • ProxyCommand is complex for windows users; a VPN would allow a much more user-friendly connection (directly connecting to e.g. tools-exec-1201.eqiad.wmflabs from putty) […]

Is the configuration really that complex? https://wikitech.wikimedia.org/wiki/Help:Access_to_instances_with_PuTTY_and_WinSCP#How_to_set_up_PuTTY_for_proxying_through_bastion.wmflabs.org_to_your_instance looks tedious to repeat for every instance, but fairly simple.

I have tried this configuration before, and it works, but not reliably. The connection can cut out unexpectedly and often PuTTY is unable to find the plink.exe file. This configuration also means that there is a plink.exe process running on the computer for every SSH connection to Labs machines, and these SSH sessions are slower than SSH to bastion, SSH to other machine in Labs sessions.

scfc added a comment.Nov 26 2016, 1:29 PM

plink is a (rudimentary) VPN, just like ssh's -W option in ProxyCommand: It needs to run, it will encrypt the session once again and it can end unexpectedly, so these are not arguments against it.