Change 868814 had a related patch set uploaded (by FNegri; author: Andrew Bogott):

[operations/software/cumin@master] add domain param to openstack backend

https://gerrit.wikimedia.org/r/868814

@fnegri thanks for the task. Have you checked on the openstack logs if there is anything explicit on what the missing permission was?

From the cumin logs:

2023-09-19 13:23:26,434 [DEBUG 282873 keystoneauth.identity.v3.base.get_auth_ref] Making authentication request to https://openstack.eqiad1.wikimediacloud.org:25000/v3/auth/tok
ens
2023-09-19 13:23:26,436 [DEBUG 282873 urllib3.connectionpool._new_conn] Starting new HTTPS connection (1): openstack.eqiad1.wikimediacloud.org:25000
2023-09-19 13:23:27,023 [DEBUG 282873 urllib3.connectionpool._make_request] https://openstack.eqiad1.wikimediacloud.org:25000 "POST /v3/auth/tokens HTTP/1.1" 401 None
2023-09-19 13:23:27,026 [DEBUG 282873 keystoneauth.session.request] Request returned failure status: 401

But a previous call to the same endpoint was successful at the start of the same run:

2023-09-19 13:23:25,348 [DEBUG 282873 urllib3.connectionpool._make_request] https://openstack.eqiad1.wikimediacloud.org:25000 "POST /v3/auth/tokens HTTP/1.1" 201 None

And those were the only 2 calls made to that api:

$ sudo grep "POST /v3/auth/tokens" /var/log/cumin/cumin.log
2023-09-19 13:23:25,348 [DEBUG 282873 urllib3.connectionpool._make_request] https://openstack.eqiad1.wikimediacloud.org:25000 "POST /v3/auth/tokens HTTP/1.1" 201 None
2023-09-19 13:23:27,023 [DEBUG 282873 urllib3.connectionpool._make_request] https://openstack.eqiad1.wikimediacloud.org:25000 "POST /v3/auth/tokens HTTP/1.1" 401 None

So I'm wondering if this was some sort of re-authentication from keystoneauth because of expired token or because it had to authenticate for another domain or similar and that failed because of lack of authorization.

That said, yes I know there are the 2 pending patches and I should really find the time to debug/test/work on them. This is also the end of the quarter and I'm fairly busy finishing things, do you think it could wait the beginning of October?

@Volans October is fine, we'll make sure that the CI is passing in the meantime.

wondering if this was some sort of re-authentication from keystoneauth because of expired token or because it had to authenticate for another domain

I think this is because of the domain, but I'm not 100% sure.