Page MenuHomePhabricator

Provide read-only access to OpenStack APIs from WMF IP space
Closed, ResolvedPublic

Description

We're moving more and more OpenStack data out of ldap and into proper OpenStack services. That's good, but it means that public ldap credentials are no longer adequate to do things like enumerate all instances in all projects. This has broken several tools -- for example, watroles.

There are a few hacks to provide this information, most obviously the 'wikitech API'. The right solution, though, is just to provide access to OpenStack APIs from within labs.

  • implement login ACLs so that labs machines can get tokens only for select users (for the time being, just novaobserver)
  • Open up firewalls, routes, etc so that labs instances can hit the non-admin Nova, Glance and Keystone APIs. (Actually the Nova api is already reachable from labs, although useless without keystone tokens)
  • Insert a 'novaobserver' account into every project
  • Provide PUBLIC username/password credentials for the novaobserver account

With this setup users will also be able to use their horizon login to manipulate projects from within Labs. I'm pretty sure that's a feature and not a bug, although it will provide a way to bypass 2fa for such operations. /Maybe/ we want to enforce 2fa for all users that aren't novaadmin or novaobserver.

Thanks to the ACLs, this means that individual user logins are still only useful when logging in from horizon or wikitech; they'll still be blocked from within Labs (and firewalled _and_ blocked from the rest of the internet). The only account that will work is novaobserver, which will not have any explicit roles set on any projects. That means that 'observer' creds can only be used to do things that are entirely unrestricted by nova or keystone policy.

Details

SubjectRepoBranchLines +/-
operations/puppetproduction+167 -9
operations/puppetproduction+1 -0
operations/puppetproduction+62 -34
operations/puppetproduction+1 -0
operations/puppetproduction+19 -0
labs/privatemaster+6 -0
labs/privatemaster+2 -0
operations/puppetproduction+26 -26
operations/puppetproduction+169 -9
operations/puppetproduction+1 -1
mediawiki/extensions/OpenStackManagermaster+9 -1
operations/puppetproduction+0 -24
operations/puppetproduction+7 -0
operations/puppetproduction+24 -0
operations/puppetproduction+19 -0
operations/puppetproduction+18 -7
operations/puppetproduction+2 -2
operations/puppetproduction+2 -2
operations/puppetproduction+0 -4
operations/puppetproduction+6 -0
operations/puppetproduction+92 -0
Show related patches Customize query in gerrit

Event Timeline

cc'ing everyone because this has huge security implications and I want to make sure I'm not missing obvious problems.

I'm pretty sure the Nova half of #1 is unnecessary, instances can already hit the nova API (it runs on labnet), they just can't use it due to lack of Keystone access:

krenair@bastion-01:~$ curl http://labnet1001.eqiad.wmnet:8774/v2
Authentication required
krenair@bastion-01:~$ curl -v http://labcontrol1001.wikimedia.org:5000/v3
* Hostname was NOT found in DNS cache
*   Trying 208.80.154.92...

You're right about other services, I don't have use cases for them to be hitting Glance or Designate.

We should probably disallow logins from instance IPs on usernames other than whitelisted, special-purpose accounts like novaobserver. I'm pretty sure we do NOT want individual user's LDAP passwords being exposed to labs instances (anyone's accounts, but also in a lot of cases user's LDAP passwords provide access to production sensitive data via the wmf/nda/ops LDAP groups)

We should probably disallow logins from instance IPs on usernames other than whitelisted, special-purpose accounts like novaobserver. I'm pretty sure we do NOT want individual user's LDAP passwords being exposed to labs instances (anyone's accounts, but also in a lot of cases user's LDAP passwords provide access to production sensitive data via the wmf/nda/ops LDAP groups)

Actually, I'll go further: We should lock (via changing the password to something random) any user other than a whitelisted account that managed to get a successful username+password match from a labs instance IP. Then automatically notify operations/security of a potential breach along with the username and a list of their groups.

Can you talk me through how that would help? As I understand it, you still need the initial password in order to get the token... would we need to create a separate special-purpose service to generate delegated tokens?

And, of course, for those tokens to work we would still need to open up the firewall for API ports, right?

We should probably disallow logins from instance IPs on usernames other than whitelisted, special-purpose accounts like novaobserver. I'm pretty sure we do NOT want individual user's LDAP passwords being exposed to labs instances (anyone's accounts, but also in a lot of cases user's LDAP passwords provide access to production sensitive data via the wmf/nda/ops LDAP groups)

Actually, I'll go further: We should lock (via changing the password to something random) any user other than a whitelisted account that managed to get a successful username+password match from a labs instance IP. Then automatically notify operations/security of a potential breach along with the username and a list of their groups.

@Krenair, do you have in mind a facility for blocking account access by name and IP?

We should probably disallow logins from instance IPs on usernames other than whitelisted, special-purpose accounts like novaobserver. I'm pretty sure we do NOT want individual user's LDAP passwords being exposed to labs instances (anyone's accounts, but also in a lot of cases user's LDAP passwords provide access to production sensitive data via the wmf/nda/ops LDAP groups)

Actually, I'll go further: We should lock (via changing the password to something random) any user other than a whitelisted account that managed to get a successful username+password match from a labs instance IP. Then automatically notify operations/security of a potential breach along with the username and a list of their groups.

@Krenair, do you have in mind a facility for blocking account access by name and IP?

Can probably usurp Keystone's own password authentication plugin, subclass it, add a check against context.remote_addr (?) in the authenticate method as well as a username whitelist, call the super method, and then add the same check to wmtotp?

Can you talk me through how that would help? As I understand it, you still need the initial password in order to get the token... would we need to create a separate special-purpose service to generate delegated tokens?

I haven't looked to see if there is any web frontend for OAuth that can be easily plugged into horizon for creating access tokens, but in general the point of OAuth would be use easily revoked tokens for authentication requests coming from any external application/script rather than LDAP passwords. The "With this setup users will also be able to use their horizon login to manipulate projects from within Labs." statement sounds like using LDAP passwords from within Labs which has always been highly discouraged do to the various theoretical attacks that could capture a password stored or transmitted across the Labs internal network.

And, of course, for those tokens to work we would still need to open up the firewall for API ports, right?

Yes, network access to the nova/keystone api endpoints would still be needed.

in general the point of OAuth would be use easily revoked tokens for
authentication requests coming from any external application/script rather than
LDAP passwords.

So... I'm sorry, I still not sure I understand. In this model, would a tool developer request that a Labs admin issue a special-purpose token, and the token would be effectively immortal until explicitly revoked?

Can probably usurp Keystone's own password authentication plugin, subclass
it, add a check against context.remote_addr (?) in the authenticate method as
well as a username whitelist, call the super method, and then add the same
check to wmtotp?

I recognize the wisdom of this suggestion, but I also really hate the idea of second-guessing (and rewriting) keystone's existing security model, since the whole point of keystone is to be the secure auth layer.

in general the point of OAuth would be use easily revoked tokens for
authentication requests coming from any external application/script rather than
LDAP passwords.

So... I'm sorry, I still not sure I understand. In this model, would a tool developer request that a Labs admin issue a special-purpose token, and the token would be effectively immortal until explicitly revoked?

Ideally, Labs admins would not be involved. With a web interface (maybe in horizon?) a user should be able to self-issue a token with a controlled scope of rights. MediaWiki's OAuth has this functionality via Special:OAuthConsumerRegistration/propose's "owner only consumers". In the case of MediaWiki the tokens have no automatic expiration, but can be easily revoked via Special:OAuthManageMyGrants. It looks like the Keystone tokens may additionally have the ability to limit the token lifetime at the time of issuance although I'm not sure if using this would actually be desired by a typical consumer.

Change 320706 had a related patch set uploaded (by Andrew Bogott):
Keystone: Limit password auth to certain hosts and users.

https://gerrit.wikimedia.org/r/320706

Update -- documentation for keystone/oauth is pretty poor, and keystone has a habit of leaving partially-completed features to rot so I'm inclined to not rely on it.

The attached patch limits logins to certain accounts and ip ranges, which I think makes opening up the firewalls fairly safe. It compiles properly for labtestcontrol but not for labcontrol, which I'm not yet clear on if that's a real problem or a puppet compiler problem.

It compiles properly for labtestcontrol but not for labcontrol, which I'm not yet clear on if that's a real problem or a puppet compiler problem.

http://puppet-compiler.wmflabs.org/4575/labcontrol1001.wikimedia.org/change.labcontrol1001.wikimedia.org.err

Error: Failed to parse template openstack/liberty/keystone/keystone.conf.erb:
  Filepath: /mnt/jenkins-workspace/puppet-compiler/4575/change/src/modules/openstack/templates/liberty/keystone/keystone.conf.erb
  Line: 418
  Detail: undefined method `each' for nil:NilClass
 at /mnt/jenkins-workspace/puppet-compiler/4575/change/src/modules/openstack/manifests/keystone/service.pp:25 on node labcontrol1001.wikimedia.org

https://gerrit.wikimedia.org/r/#/c/320706/2/modules/openstack/templates/liberty/keystone/keystone.conf.erb

Maybe it needs a require network::constants before the template() or something?

Maybe it needs a require network::constants before the template() or something?

Yep, fixed by changing the erb lookup.

Change 320787 had a related patch set uploaded (by Andrew Bogott):
Keystone: open up firewall for public keystone API

https://gerrit.wikimedia.org/r/320787

Change 320791 had a related patch set uploaded (by Andrew Bogott):
Check password/ip whitelist for wmtotp.

https://gerrit.wikimedia.org/r/320791

Change 320825 had a related patch set uploaded (by Andrew Bogott):
Keystone: remove explicit observer rights

https://gerrit.wikimedia.org/r/320825

Change 320826 had a related patch set uploaded (by Andrew Bogott):
Keystone: Make the project list public

https://gerrit.wikimedia.org/r/320826

Change 320827 had a related patch set uploaded (by Andrew Bogott):
Make compute:get fully public

https://gerrit.wikimedia.org/r/320827

Change 320830 had a related patch set uploaded (by Andrew Bogott):
Labs: Add observerenv.sh, helper script for read-only creds

https://gerrit.wikimedia.org/r/320830

Change 320706 merged by Andrew Bogott:
Keystone: Limit password auth to certain hosts and users.

https://gerrit.wikimedia.org/r/320706

Change 320791 merged by Andrew Bogott:
Check password/ip whitelist for wmtotp.

https://gerrit.wikimedia.org/r/320791

Andrew updated the task description. (Show Details)

Change 320825 merged by Andrew Bogott:
Keystone: remove explicit observer rights

https://gerrit.wikimedia.org/r/320825

Change 320826 merged by Andrew Bogott:
Keystone: Make the project list public

https://gerrit.wikimedia.org/r/320826

Change 320827 merged by Andrew Bogott:
Make compute:get fully public

https://gerrit.wikimedia.org/r/320827

Change 320787 merged by Andrew Bogott:
Keystone: open up firewall to allow labs access to keystone API

https://gerrit.wikimedia.org/r/320787

Change 320830 merged by Andrew Bogott:
Labs: Add observerenv.sh, helper script for read-only creds

https://gerrit.wikimedia.org/r/320830

chasemp renamed this task from Provide public access to OpenStack APIs to Provide instance level ro access to OpenStack APIs.Dec 2 2016, 4:51 PM
chasemp renamed this task from Provide instance level ro access to OpenStack APIs to Provide read-only access to OpenStack APIs from WMF IP space.Dec 2 2016, 4:56 PM

Change 324963 had a related patch set uploaded (by Andrew Bogott):
Keystone: add 'observer' domain

https://gerrit.wikimedia.org/r/324963

Change 325371 had a related patch set uploaded (by Andrew Bogott):
Labs ldap: Hide the novaobserver account from everyone but keystone

https://gerrit.wikimedia.org/r/325371

Change 324963 abandoned by Andrew Bogott:
Keystone: add 'observer' domain

https://gerrit.wikimedia.org/r/324963

Change 325371 merged by Andrew Bogott:
Labs ldap: Hide the novaobserver account from everyone but keystone

https://gerrit.wikimedia.org/r/325371

Change 325633 had a related patch set uploaded (by Andrew Bogott):
wmfkeystonehooks: Don't bother adding novaobserver to projects

https://gerrit.wikimedia.org/r/325633

Change 325633 abandoned by Andrew Bogott:
wmfkeystonehooks: Don't bother adding novaobserver to projects

Reason:
This works only if novaobserver is in the 'admin' project, which seems like a bad idea

https://gerrit.wikimedia.org/r/325633

Change 325643 had a related patch set uploaded (by Andrew Bogott):
Novaobserver: novaobserver isn't in the admin project.

https://gerrit.wikimedia.org/r/325643

Change 325717 had a related patch set uploaded (by Andrew Bogott):
Add $wgOpenStackHiddenUsernames global array

https://gerrit.wikimedia.org/r/325717

Change 325717 merged by jenkins-bot:
Add $wgOpenStackHiddenUsernames global array

https://gerrit.wikimedia.org/r/325717

Change 325643 merged by Andrew Bogott:
Novaobserver: novaobserver isn't in the admin project.

https://gerrit.wikimedia.org/r/325643

Change 325828 had a related patch set uploaded (by Andrew Bogott):
Add clientlib.pp and mwopenstackclients.py

https://gerrit.wikimedia.org/r/325828

Change 325830 had a related patch set uploaded (by Andrew Bogott):
Add clientlib.pp and mwopenstackclients.py

https://gerrit.wikimedia.org/r/325830

Change 325994 had a related patch set uploaded (by Andrew Bogott):
Designate policy: Add public read-only access

https://gerrit.wikimedia.org/r/325994

Change 325830 merged by Andrew Bogott:
Add clientlib.pp and mwopenstackclients.py

https://gerrit.wikimedia.org/r/325830

Change 325994 merged by Andrew Bogott:
Designate policy: Add public read-only access

https://gerrit.wikimedia.org/r/325994

Change 326979 had a related patch set uploaded (by Andrew Bogott):
Keystone: Publish credentials for novaobserver account

https://gerrit.wikimedia.org/r/326979

Change 326979 merged by Andrew Bogott:
Keystone: Publish credentials for novaobserver account

https://gerrit.wikimedia.org/r/326979

Change 327230 had a related patch set uploaded (by Andrew Bogott):
Add novaconfig to labs.yaml

https://gerrit.wikimedia.org/r/327230

Change 327231 had a related patch set uploaded (by Andrew Bogott):
Add labs.yaml to labs/private

https://gerrit.wikimedia.org/r/327231

Change 327232 had a related patch set uploaded (by Andrew Bogott):
Labs hiera: Include private labs.yaml in hiera search

https://gerrit.wikimedia.org/r/327232

Change 327231 merged by Andrew Bogott:
Add labs.yaml to labs/private

https://gerrit.wikimedia.org/r/327231

Change 327230 merged by Andrew Bogott:
Add novaconfig to labs.yaml

https://gerrit.wikimedia.org/r/327230

Change 327232 merged by Andrew Bogott:
Labs hiera: Include private labs.yaml in hiera search

https://gerrit.wikimedia.org/r/327232

Change 327290 had a related patch set uploaded (by Andrew Bogott):
Keystone: refactor observerenv.sh

https://gerrit.wikimedia.org/r/327290

Change 327290 merged by Andrew Bogott:
Keystone: refactor observerenv.sh

https://gerrit.wikimedia.org/r/327290

Change 327787 had a related patch set uploaded (by Andrew Bogott):
Add export OS_INTERFACE=public to observerenv

https://gerrit.wikimedia.org/r/327787

Change 327787 merged by Andrew Bogott:
Add export OS_INTERFACE=public to observerenv

https://gerrit.wikimedia.org/r/327787

I can now do 'openstack project list' on a labs Jessie machine with addition of openstack::clientlib.

Change 325828 abandoned by Andrew Bogott:
Add clientlib.pp and mwopenstackclients.py

https://gerrit.wikimedia.org/r/325828