The backups failed running with unauthorized error, looking:
Feb 08 14:00:02 cloudvirt1023 wmcs-backup[58653]: keystoneauth1.exceptions.http.Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-8783bac7-2fa3-437d-9402-e66e9e65d38c)
The request that is getting unauthorized is:
INFO:[2021-02-08 14:15:51,975] Request to POST http://openstack.eqiad1.wikimediacloud.org:5000/v3/auth/tokens returned failure status: 401
The user that is trying to do it is novaobserver, weird
This is failing also on other cloudcontrols now, so this will trigger a bunch
of alerts, looking.
Awesome, it seems that I was missing some extra auth on the new project xd,
manually added some logs to the wmopenstackclients.py:
root@cloudvirt1023:~# wmcs-backup instances summary a44cbe037af045819cba4deff482035c ...
root@cloudvirt1023:~# openstack project show a44cbe037af045819cba4deff482035c +-------------+--------------------------------------------------------------+ | Field | Value | +-------------+--------------------------------------------------------------+ | description | This project is used to test development of the pki project. | | domain_id | default | | enabled | True | | id | a44cbe037af045819cba4deff482035c | | is_domain | False | | name | pki | | parent_id | default | | tags | [] | +-------------+--------------------------------------------------------------+
Something weird is going on anyhow...
It seems that in order to list the servers for a single project you need admin
privileges:
root@cloudvirt1023:~# openstack server list --help | grep '\-\-project '
[--all-projects] [--project <project>]
--project <project> Search by project (admin only) (name or ID)On the cli, it will list all servers no matter what, but on the api it seems to
fail with unauthorized.
The funny thing is that this was working last week :/
Looking...
So the issue is that openstack now generates a hash ID for the project instead of re-using the name as the project ID (which is what our authentication house of cards depends on). The reason is, you only see the names in a project that you actually are in. You can always see IDs.
So some scripts get the name and then search for the project name as the ID. This can be worked around by getting the ID instead in our scripting. However, I worry this will have unintended consequences elsewhere (since we didn't plan to change this, lol)
I found confirmation of this theory here: https://logstash.wikimedia.org/app/dashboards#/view/49ce6bf0-67d2-11eb-ba38-efc6fae89eca?_g=h@865c245&_a=h@f450cee
(which won't restore state, but you can search for *pki* on the OpenStack eqiad1 dashboard to get the actual strings)
Also:
[bstorm@cloudcontrol1003]:~ $ sudo wmcs-openstack project list +----------------------------------+--------------------------------+ | ID | Name | +----------------------------------+--------------------------------+ | a44cbe037af045819cba4deff482035c | pki | | account-creation-assistance | account-creation-assistance | | admin | admin | | admin-monitoring | admin-monitoring | | analytics | analytics | | annotation | annotation | | antiharassment | antiharassment | | automation-framework | automation-framework | | bastion | bastion | | centralnotice-staging | centralnotice-staging | | chat | chat | | clouddb-services | clouddb-services | | cloudinfra | cloudinfra | | cloudstore | cloudstore | | cloudvirt-canary | cloudvirt-canary | | codereview | codereview | | codesearch | codesearch | | collection-alt-renderer | collection-alt-renderer | | commonsarchive | commonsarchive | | commtech | commtech | | community-labs-monitoring | community-labs-monitoring | | cvn | cvn | | cyberbot | cyberbot | | dashiki | dashiki | | deployment-prep | deployment-prep | | devtools | devtools | | discourse | discourse | | download | download | | dumps | dumps | | dwl | dwl | | entity-detection | entity-detection | | etytree | etytree | | eventmetrics | eventmetrics | | extdist | extdist | | fa-wp | fa-wp | | fastcci | fastcci | | getstarted | getstarted | | gitlab-test | gitlab-test | | glampipe | glampipe | | globalcu | globalcu | | globaleducation | globaleducation | | google-api-proxy | google-api-proxy | | graphql | graphql | | gratitude | gratitude | | hashtags | hashtags | | huggle | huggle | | huwiki-dev | huwiki-dev | | iiab | iiab | | incubator | incubator | | integration | integration | | k8splay | k8splay | | language | language | | library-upgrader | library-upgrader | | linkwatcher | linkwatcher | | logging | logging | | lta-tracker | lta-tracker | | mailman | mailman | | maps | maps | | maps-experiments | maps-experiments | | mariadb104-test | mariadb104-test | | math | math | | matrix | matrix | | mediawiki-vagrant | mediawiki-vagrant | | meet | meet | | metricsinfra | metricsinfra | | mix-n-match | mix-n-match | | mobile | mobile | | monitoring | monitoring | | mwoffliner | mwoffliner | | mwstake | mwstake | | mwv-apt | mwv-apt | | observer | observer | | ocrtoy | ocrtoy | | openocr | openocr | | openrefine | openrefine | | openstack | openstack | | ores | ores | | ores-staging | ores-staging | | osmit | osmit | | packaging | packaging | | packagist-mirror | packagist-mirror | | paws | paws | | petscan | petscan | | pipelinelib-experimental | pipelinelib-experimental | | pluggableauth | pluggableauth | | privpol-captcha | privpol-captcha | | project-proxy | project-proxy | | puppet-diffs | puppet-diffs | | push | push | | quarry | quarry | | rcm | rcm | | reading-web-staging | reading-web-staging | | recommendation-api | recommendation-api | | redirects | redirects | | redwarn | redwarn | | sciencesource | sciencesource | | search | search | | security-tools | security-tools | | sentry | sentry | | services | services | | shiny-r | shiny-r | | signwriting | signwriting | | snuggle | snuggle | | soweego | soweego | | sre-sandbox | sre-sandbox | | srwiki-dev | srwiki-dev | | sso | sso | | striker | striker | | suggestbot | suggestbot | | swift | swift | | testlabs | testlabs | | thumbor | thumbor | | toolhub | toolhub | | tools | tools | | toolsbeta | toolsbeta | | traffic | traffic | | twl | twl | | utrs | utrs | | video | video | | videocuttool | videocuttool | | videowiki | videowiki | | visualeditor | visualeditor | | wcdo | wcdo | | webperf | webperf | | wikiapiary | wikiapiary | | wikibase-registry | wikibase-registry | | wikicommunityhealth | wikicommunityhealth | | wikidata-autodesc | wikidata-autodesc | | wikidata-dev | wikidata-dev | | wikidata-history-query-service | wikidata-history-query-service | | wikidata-query | wikidata-query | | wikidata-realtime-dumps | wikidata-realtime-dumps | | wikidocumentaries | wikidocumentaries | | wikidumpparse | wikidumpparse | | wikilabels | wikilabels | | wikilink | wikilink | | wikiloop | wikiloop | | wikipathways | wikipathways | | wikisource | wikisource | | wikispeech | wikispeech | | wikispore | wikispore | | wikistats | wikistats | | wikitextexp | wikitextexp | | wildcat | wildcat | | wm-bot | wm-bot | | wmde-dashboards | wmde-dashboards | | wmde-templates-alpha | wmde-templates-alpha | | wmf-research-tools | wmf-research-tools | | wmflabsdotorg | wmflabsdotorg | | xtools | xtools | +----------------------------------+--------------------------------+ [bstorm@cloudcontrol1003]:~ $ sudo wmcs-openstack project list | grep pki | a44cbe037af045819cba4deff482035c | pki | [bstorm@cloudcontrol1003]:~ $ https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Projects_lifecycle#Using_CLI^C [bstorm@cloudcontrol1003]:~ $ sudo wmcs-openstack project create --enable --description "a test project to make sure keystone is good" create-test +-------------+----------------------------------------------+ | Field | Value | +-------------+----------------------------------------------+ | description | a test project to make sure keystone is good | | domain_id | default | | enabled | True | | id | c72d1283026d4ccfa0209492c0732a62 | | is_domain | False | | name | create-test | | parent_id | default | | tags | [] | +-------------+----------------------------------------------+
So we've somehow broken our setup during Openstack upgrades or something. (T261134)
I seem to recall this happening before. We can fix it in the database, if we have to.
The "bad" aka "standard" projects are now deleted. I suspect it should start working now, maybe with a restart?
Mentioned in SAL (#wikimedia-cloud) [2021-02-08T18:44:57Z] <bstorm> restarted the backup_vms.service on cloudvirt1027 T274144
Mentioned in SAL (#wikimedia-cloud) [2021-02-08T18:50:30Z] <bstorm> enabled puppet on cloudvirt1023 for now T274144
This is solved now, though until T274165 is fixed any new project will blow it up again.