Page MenuHomePhabricator

OpenStack API account to control `contintcloud` labs project
Closed, ResolvedPublic

Description

To spawn disposable VMs in the contintcloud labs project (T47499), we would need credentials with OpenStack API to be able to spawn instances. The service we are going to use (nodepool) also maintains/refresh disk images to boot instances from them. I am not sure how we could restrict those images to the contintcloud project and avoid cluttering the rest of the labs projects.

Event Timeline

hashar created this task.Jan 8 2015, 2:04 PM
hashar updated the task description. (Show Details)
hashar raised the priority of this task from to Needs Triage.
hashar added a subscriber: hashar.
hashar added a subscriber: Cloud-Services.

Dear #Labs-Team , do you have any idea how to provide OpenStack API credentials for the contintcloud project ? Would use it to programmatically spawn/delete instances to be consumed by Jenkins.

Where are you going to call the API from? From inside labs or from inside production?

chasemp triaged this task as Normal priority.Jan 8 2015, 5:03 PM
chasemp set Security to None.

I have created a first draft of the architecture at https://www.mediawiki.org/wiki/Continuous_integration/Architecture/Isolation .

In that proposal, the NodePool system (which interacts with the OpenStack API) is on a server in the labs subnet, much like labmon. I am hoping that would make it easy to interact with the API on one of the virt** machine.

coren moved this task from Triage to Backlog on the Cloud-Services board.Feb 10 2015, 9:20 PM

Adding @chasemp . We talked about nodepool user/credentials today. The task description probably needs to be enhanced but in short the user needs:

  • actions to be limited to contintcloud project, we dont want nodepool to start deleting random instances in labs
  • ability to upload images to glance

From labnodepool I tried to authenticate using the user/pass I have created via wikitech. The password is in /etc/nodepool/experiment.yaml

OS_AUTH_URL=http://virt1000.wikimedia.org:35357/v2.0/ \
OS_TENANT_NAME=contintcloud \
OS_USERNAME=nodepoolmanager \
OS_PASSWORD=XXXXXXXXX \
nova --debug flavor-list
REQ: curl -i 'http://virt1000.wikimedia.org:35357/v2.0/tokens' -X POST -H "Accept: application/json" -H "Content-Type: application/json" -H "User-Agent: python-novaclient" -d '{"auth": {"tenantName": "contintcloud", "passwordCredentials": {"username": "nodepoolmanager", "password": "XXXXXXXXX"}}}'
INFO (connectionpool:188) Starting new HTTP connection (1): virt1000.wikimedia.org
DEBUG (connectionpool:362) "POST /v2.0/tokens HTTP/1.1" 401 87
RESP: [401] {'content-length': '87', 'vary': 'X-Auth-Token', 'x-distribution': 'Ubuntu', 'connection': 'keep-alive', 'date': 'Mon, 20 Apr 2015 20:12:26 GMT', 'content-type': 'application/json', 'www-authenticate': 'Keystone uri="http://virt1000.wikimedia.org:35357"'}
RESP BODY: {"error": {"message": "Invalid user / password", "code": 401, "title": "Unauthorized"}}

DEBUG (shell:802) Invalid OpenStack Nova credentials.
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/novaclient/shell.py", line 799, in main
    OpenStackComputeShell().main(argv)
  File "/usr/lib/python2.7/dist-packages/novaclient/shell.py", line 700, in main
    raise exc.CommandError(_("Invalid OpenStack Nova credentials."))
CommandError: Invalid OpenStack Nova credentials.
ERROR (CommandError): Invalid OpenStack Nova credentials.

So we need some more magic :)

Here's a test... can you log in to https://horizon.wikimedia.org/ as yourself? (I can). And, can you log in there as nodepoolmanager? (I can't.)

That makes me think that the password in /etc/nodepool/experiment.yaml is wrong, maybe?

It works now, nodepoolmanager still had a temporary password on wikitech. I have set a new password and I can now login in horizon. Nodepool is now able to interact with the OpenStack API!

I have some more work to do though since nodepoold complains with:

2015-04-21 08:49:15,095 DEBUG urllib3.connectionpool: "POST /v2.0/tokens HTTP/1.1" 200 5413
Could not find any suitable endpoint. Correct region?
                                      ^^^^^^^^^^^^^^^
2015-04-21 08:49:15,097 DEBUG nodepool.ProviderManager: Manager wmflabs-eqiad ran task <nodepool.provider_manager.ListFlavorsTask object at 0x7fa042f11d10> in 1.19026207924s
2015-04-21 08:49:15,097 ERROR nodepool.SnapshotImageUpdater: Exception updating image trusty-medium in wmflabs-eqiad:
Traceback (most recent call last):

  File "novaclient/client.py", line 515, in _extract_service_catalog
    volume_service_name=self.volume_service_name,)
  File "novaclient/service_catalog.py", line 84, in url_for
    raise novaclient.exceptions.EndpointNotFound()
EndpointNotFound

I have adjusted the nodepool setting file and it managed to creates its first ever instance trusty-medium-1429608367.eqiad.wmflabs. Nodepool has to ssh to it but the authentication does not work yet. Will figure it out eventually.

hashar claimed this task.
hashar closed this task as Resolved.Apr 21 2015, 1:37 PM

Nodepool can access the OpenStack API just fine now \o/ Thank you @Andrew