To spawn disposable VMs in the contintcloud labs project (T47499), we would need credentials with OpenStack API to be able to spawn instances. The service we are going to use (nodepool) also maintains/refresh disk images to boot instances from them. I am not sure how we could restrict those images to the contintcloud project and avoid cluttering the rest of the labs projects.
Description
Description
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | hashar | T60772 common gating job for mediawiki core and extensions | |||
| Resolved | hashar | T69216 Have unit tests of all wmf deployed extensions pass when installed together, in both PHP-Zend and HHVM (tracking) | |||
| Invalid | Ryasmeen | T90647 Create Jenkins builds for Editing across repositories (MobileFrontend, VisualEditor etc) | |||
| Declined | None | T50407 Jenkins: Setup Vagrant for some jobs (tracking) | |||
| Declined | None | T45266 Write and implement tests for Wikimedia's Apache configuration (redirects.conf, etc.) | |||
| Resolved | hashar | T47499 [EPIC] Run CI jobs in disposable VMs | |||
| Resolved | None | T109914 [keyresult] subset of jobs run in disposable instances | |||
| Resolved | hashar | T109913 [keyresult] boot instances from OpenStack API | |||
| Resolved | hashar | T86170 OpenStack API account to control `contintcloud` labs project | |||
| Resolved | yuvipanda | T86167 Create labs project for CI disposables instances |
Event Timeline
Comment Actions
Dear #Labs-Team , do you have any idea how to provide OpenStack API credentials for the contintcloud project ? Would use it to programmatically spawn/delete instances to be consumed by Jenkins.
Comment Actions
Where are you going to call the API from? From inside labs or from inside production?
Comment Actions
I have created a first draft of the architecture at https://www.mediawiki.org/wiki/Continuous_integration/Architecture/Isolation .
In that proposal, the NodePool system (which interacts with the OpenStack API) is on a server in the labs subnet, much like labmon. I am hoping that would make it easy to interact with the API on one of the virt** machine.
Comment Actions
Adding @chasemp . We talked about nodepool user/credentials today. The task description probably needs to be enhanced but in short the user needs:
- actions to be limited to contintcloud project, we dont want nodepool to start deleting random instances in labs
- ability to upload images to glance
Comment Actions
From labnodepool I tried to authenticate using the user/pass I have created via wikitech. The password is in /etc/nodepool/experiment.yaml
OS_AUTH_URL=http://virt1000.wikimedia.org:35357/v2.0/ \ OS_TENANT_NAME=contintcloud \ OS_USERNAME=nodepoolmanager \ OS_PASSWORD=XXXXXXXXX \ nova --debug flavor-list
REQ: curl -i 'http://virt1000.wikimedia.org:35357/v2.0/tokens' -X POST -H "Accept: application/json" -H "Content-Type: application/json" -H "User-Agent: python-novaclient" -d '{"auth": {"tenantName": "contintcloud", "passwordCredentials": {"username": "nodepoolmanager", "password": "XXXXXXXXX"}}}'
INFO (connectionpool:188) Starting new HTTP connection (1): virt1000.wikimedia.org
DEBUG (connectionpool:362) "POST /v2.0/tokens HTTP/1.1" 401 87
RESP: [401] {'content-length': '87', 'vary': 'X-Auth-Token', 'x-distribution': 'Ubuntu', 'connection': 'keep-alive', 'date': 'Mon, 20 Apr 2015 20:12:26 GMT', 'content-type': 'application/json', 'www-authenticate': 'Keystone uri="http://virt1000.wikimedia.org:35357"'}
RESP BODY: {"error": {"message": "Invalid user / password", "code": 401, "title": "Unauthorized"}}
DEBUG (shell:802) Invalid OpenStack Nova credentials.
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/novaclient/shell.py", line 799, in main
OpenStackComputeShell().main(argv)
File "/usr/lib/python2.7/dist-packages/novaclient/shell.py", line 700, in main
raise exc.CommandError(_("Invalid OpenStack Nova credentials."))
CommandError: Invalid OpenStack Nova credentials.
ERROR (CommandError): Invalid OpenStack Nova credentials.So we need some more magic :)
Comment Actions
Here's a test... can you log in to https://horizon.wikimedia.org/ as yourself? (I can). And, can you log in there as nodepoolmanager? (I can't.)
That makes me think that the password in /etc/nodepool/experiment.yaml is wrong, maybe?
Comment Actions
It works now, nodepoolmanager still had a temporary password on wikitech. I have set a new password and I can now login in horizon. Nodepool is now able to interact with the OpenStack API!
I have some more work to do though since nodepoold complains with:
2015-04-21 08:49:15,095 DEBUG urllib3.connectionpool: "POST /v2.0/tokens HTTP/1.1" 200 5413
Could not find any suitable endpoint. Correct region?
^^^^^^^^^^^^^^^
2015-04-21 08:49:15,097 DEBUG nodepool.ProviderManager: Manager wmflabs-eqiad ran task <nodepool.provider_manager.ListFlavorsTask object at 0x7fa042f11d10> in 1.19026207924s
2015-04-21 08:49:15,097 ERROR nodepool.SnapshotImageUpdater: Exception updating image trusty-medium in wmflabs-eqiad:
Traceback (most recent call last):
File "novaclient/client.py", line 515, in _extract_service_catalog
volume_service_name=self.volume_service_name,)
File "novaclient/service_catalog.py", line 84, in url_for
raise novaclient.exceptions.EndpointNotFound()
EndpointNotFoundComment Actions
I have adjusted the nodepool setting file and it managed to creates its first ever instance trusty-medium-1429608367.eqiad.wmflabs. Nodepool has to ssh to it but the authentication does not work yet. Will figure it out eventually.