OpenStack Queens added support for "Application Credentials" (a.k.a. API tokens) that allows software to make API calls without exposing user/password.
This necessary to support use cases where project owners want to make use of automation software (e.g. Terraform) to deploy their infrastructure. Currently, this has to be done in a point-and-click manner through the Horizon WebUI.
More about it here:
Right now AFAIK, the APIs are only available internally within labs/prod, and logins from within labs are (basically) restricted to the read-only user. People aren't going to be running terraform against labs from within prod. Without changing the network restrictions to allow the outside world I don't see the use case.
Correct. The use case I think we should enable is for project owners running Terraform (or other tools) from their laptops remotely. For that we need App Credentials so the user/pass is not exposed unnecessarily (and they could be scoped as well).
Once this functionality is enabled, then we can work on the networking changes. It seems fair to require people to open an SSH tunnel through the bastions to reach the API server but that's an implementation detail.
Why couldn't people use Terraform against the API with user+password, instead of user+password through their browser+Horizon? Are you worried that people might end up storing their password in an insecure manner, but that tokens would be stored securely and be scoped appropriately?
Or would the idea be to permit labs network traffic to authenticate using tokens but not username+passwords, outside traffic via horizon to use username+passwords but not tokens (?), and hope nobody shows a labs instance a token that can do anymore more than read?
We're not just talking about a single API end point, they'd probably want to set up a SOCKS proxy into the labs network and have Terraform use that, so it can talk to each of the different OpenStack services.
Btw, one thing that would be missing in all of this would be the ability to deal with our dynamicproxy setup. IIRC it's a custom API that doesn't check tokens and instead relies upon trusting the Horizon IP? But something for another ticket.
Yes, Terraform against the API directly with App Credentials instead of user+password (so people don't have to store their dev account credentials in plain text somewhere). And yes, scoped tokens would be preferred as well, so impact is limited in case of a compromise.
Or would the idea be to permit labs network traffic to authenticate using tokens but not username+passwords, outside traffic via horizon to use username+passwords but not tokens (?), and hope nobody shows a labs instance a token that can do anymore more than read?
My initial idea is to permit people to bootstrap infrastructure in OpenStack without using Horizon directly (which isn't easily automatable). I don't think Horizon would be accepting App Credentials (those would be created through the Horizon interface and stored by Keystone but not used to login to Horizon itself, which would continue to require their dev acccount and 2FA).
Once App Credentials are enabled I think it doesn't matter much if they are used by someone from their laptops or stored in a Cloud VPS instance. People might decide to start a "control VM" manually and use that as a launching point for the rest of the infrastructure (this would actually be a better way to work for certain teams, so they have a centralized place to manage things).
We're not just talking about a single API end point, they'd probably want to set up a SOCKS proxy into the labs network and have Terraform use that, so it can talk to each of the different OpenStack services.
Here's an example that would work if the bastions were allowed to connect to the OpenStack API.
example.tf:
provider "openstack" {
version = "~> 1.14"
user_name = "username"
tenant_name = "project"
password = "app_credential"
auth_url = "http://localhost:5000/v3"
region = "eqiad1-r"
}
resource "openstack_compute_instance_v2" "test-server-1010" {
name = "test-server-1010"
image_id = "d620d77c-c023-41ae-944c-2f10063bfc77"
flavor_id = "2"
}terminalA$ ssh -L 5000:cloudcontrol1003.wikimedia.org:5000 bastion.wmflabs.org terminalB$ terraform plan
Btw, one thing that would be missing in all of this would be the ability to deal with our dynamicproxy setup. IIRC it's a custom API that doesn't check tokens and instead relies upon trusting the Horizon IP? But something for another ticket.
Yes, good point. That seems like something that could be replaced by LBaaS but I agree it's something for another ticket.
I'm not familiar with the workings of that particular client but can't people just insert their username+password once and get their (normal, current system) token, and store the token in memory for use like a session cookie would for Horizon? Is there any need for the user to store their username+password locally?
I think it does matter. People should never be permitted to create a token that runs things under their own account and then show that token to (let alone store it on) a Cloud VPS instance.
It also sounds like this might remove the current audit-ability of who shut down/started what instance when if actions can take place under a group user.
I wonder if we should instead open the APIs to the outside world with the appropriate credentials and continue with the current authentication restrictions for connections coming from the labs network.
Would it? There's a tunnel to Keystone but how would one talk to Nova, Neutron, Designate, Glance, etc.? Wouldn't Keystone just return API URLs that the clients can't actually talk to?
It would be important that such a thing has support for restricting the remote IP address from being seen by backend instances.
They could but is that's less secure than a scope API token. Usually, systems use API tokens for automation rather than requiring user/password to be entered every time. I think this is a common pattern everywhere and I don't see what's specific constrain in Cloud VPS that wouldn't permit to adopt this way of working.
Once App Credentials are enabled I think it doesn't matter much if they are used by someone from their laptops or stored in a Cloud VPS instance. People might decide to start a "control VM" manually and use that as a launching point for the rest of the infrastructure (this would actually be a better way to work for certain teams, so they have a centralized place to manage things).
I think it does matter. People should never be permitted to create a token that runs things under their own account and then show that token to (let alone store it on) a Cloud VPS instance.
Why? There's nothing in Amazon AWS, DigitalOcean, Azure or Google Cloud preventing people from doing that. What's special about us in this regard?
It also sounds like this might remove the current audit-ability of who shut down/started what instance when if actions can take place under a group user.
Is that something we should concern ourselves with? The project owner is responsible for their project. If they choose to share an App Credential in a way that auditability within their project is diminished, isn't it their problem?
I wonder if we should instead open the APIs to the outside world with the appropriate credentials and continue with the current authentication restrictions for connections coming from the labs network.
At our current OpenStack version, I wouldn't expose anything to the outside world :)
Why is the external world more trusted than the labs network in this proposal?
That's a relatively new thing around here (in MediaWiki which got by for years without it). I think we could change the network rules to allow such API clients without first requiring any new special API tokens system. On the other hand as mentioned below I think it's reasonable to defer that until after we run a reasonably up-to-date version of OpenStack, so we might get that for free anyway.
In AWS you never put your own credentials inside an instance, you attach roles to the instance with the relevant permissions. The instance must not be permitted to operate under the name of a particular user when other users are able to log into the instance and run commands.
It's a cross-Wikimedia problem to be solved centrally. I don't want deployment-prep users to be able to create application tokens, upload one somewhere, and then deny responsibility for actions taken under their account because their own token was shared with others and we allowed such sharing. People should treat their personal tokens as they do personal passwords, on their own personal devices.
Yes, this is all under the assumption that we move to a non-EOL version of OpenStack.
Labs would continue to have extra restrictions to discourage inappropriate credential sharing, given it would be the easiest pre-existing place for people to do that. People sharing credentials incorrectly outside is still possible but less likely imo.
For reference, this is the error in keystone while attempting to use the dev account's password with terraform:
(keystone.middleware.auth): 2019-02-05 17:53:54,527 WARNING RBAC: Invalid token (keystone.common.wsgi): 2019-02-05 17:53:54,528 WARNING The request you have made requires authentication.
Exporting the RC file through the Horizon interface doesn't export any passwords.
Closing this because we'll automatically get it once we update to a modern OpenStack version.