As part of a planned audit/analysis of the existing backups for WMCS resources, Riccardo discovered that the backups of cinder volumes (Openstack additional disks attached to VMs, not the root disks of the VMs) were failing to be backed up since March, due to a change in firewall rules that required to use the HTTP proxies to connect to the Openstack APIs, used by the backup jobs. Those are cross-site backups.
Incident doc: https://docs.google.com/document/d/17h-QBQ74bjIJkRsdayveG5PiTBVTnxFR3KY13fHSwsY/edit?usp=drive_link
----
While working on my backup audit/analysis, I've discovered that the cinder volume backups taken on `cloudbackup2*` hosts are broken (`backup_cinder_volumes` unit):
```
Jun 11 07:00:06 cloudbackup2004 systemd[1]: Starting backup_cinder_volumes.service - backup cinder volumes...
Jun 11 07:04:36 cloudbackup2004 wmcs-backup[182725]: WARNING:[2026-06-11 07:04:36,369] Retrying mwopenstackclients.Clients.allvolumes in 13.062859424353473 seconds as it raised ConnectTimeout: Request to https://openstack.eqiad1.wikimediacloud.org:25357/v3/auth/tokens timed out.
Jun 11 07:09:18 cloudbackup2004 wmcs-backup[182725]: WARNING:[2026-06-11 07:09:18,993] Retrying mwopenstackclients.Clients.allvolumes in 11.129678384689344 seconds as it raised ConnectTimeout: Request to https://openstack.eqiad1.wikimediacloud.org:25357/v3/auth/tokens timed out.
Jun 11 07:13:59 cloudbackup2004 wmcs-backup[182725]: WARNING:[2026-06-11 07:13:59,569] Retrying mwopenstackclients.Clients.allvolumes in 8.372975600660839 seconds as it raised ConnectTimeout: Request to https://openstack.eqiad1.wikimediacloud.org:25357/v3/auth/tokens timed out.
Jun 11 07:18:38 cloudbackup2004 wmcs-backup[182725]: WARNING:[2026-06-11 07:18:38,097] Retrying mwopenstackclients.Clients.allvolumes in 11.525299719816571 seconds as it raised ConnectTimeout: Request to https://openstack.eqiad1.wikimediacloud.org:25357/v3/auth/tokens timed out.
Jun 11 07:23:18 cloudbackup2004 wmcs-backup[182725]: WARNING:[2026-06-11 07:23:18,673] Retrying mwopenstackclients.Clients.allvolumes in 6.168648336640929 seconds as it raised ConnectTimeout: Request to https://openstack.eqiad1.wikimediacloud.org:25357/v3/auth/tokens timed out.
Jun 11 07:27:55 cloudbackup2004 wmcs-backup[182725]: WARNING:[2026-06-11 07:27:55,153] Retrying mwopenstackclients.Clients.allvolumes in 5.576154738960626 seconds as it raised ConnectTimeout: Request to https://openstack.eqiad1.wikimediacloud.org:25357/v3/auth/tokens timed out.
Jun 11 07:32:29 cloudbackup2004 wmcs-backup[182725]: WARNING:[2026-06-11 07:32:29,585] Retrying mwopenstackclients.Clients.allvolumes in 12.37777738170948 seconds as it raised ConnectTimeout: Request to https://openstack.eqiad1.wikimediacloud.org:25357/v3/auth/tokens timed out.
Jun 11 07:37:12 cloudbackup2004 wmcs-backup[182725]: WARNING:[2026-06-11 07:37:12,209] Retrying mwopenstackclients.Clients.allvolumes in 7.052115883403381 seconds as it raised ConnectTimeout: Request to https://openstack.eqiad1.wikimediacloud.org:25357/v3/auth/tokens timed out.
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: Traceback (most recent call last):
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 198, in _new_conn
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: sock = connection.create_connection(
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: (self._dns_host, self.port),
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: ...<2 lines>...
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: socket_options=self.socket_options,
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: )
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 85, in create_connection
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: raise err
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 73, in create_connection
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: sock.connect(sa)
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: ~~~~~~~~~~~~^^^^
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: TimeoutError: [Errno 110] Connection timed out
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: The above exception was the direct cause of the following exception:
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: Traceback (most recent call last):
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 787, in urlopen
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: response = self._make_request(
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: conn,
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: ...<10 lines>...
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: **response_kw,
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: )
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 488, in _make_request
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: raise new_e
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 464, in _make_request
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: self._validate_conn(conn)
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: ~~~~~~~~~~~~~~~~~~~^^^^^^
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 1093, in _validate_conn
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: conn.connect()
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: ~~~~~~~~~~~~^^
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 704, in connect
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: self.sock = sock = self._new_conn()
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: ~~~~~~~~~~~~~~^^
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 207, in _new_conn
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: raise ConnectTimeoutError(
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: ...<2 lines>...
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: ) from e
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: urllib3.exceptions.ConnectTimeoutError: (<urllib3.connection.HTTPSConnection object at 0x7f4e456b82d0>, 'Connection to openstack.eqiad1.wikimediacloud.org timed out. (connect timeout=None)')
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: The above exception was the direct cause of the following exception:
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: Traceback (most recent call last):
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: File "/usr/lib/python3/dist-packages/requests/adapters.py", line 667, in send
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: resp = conn.urlopen(
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: method=request.method,
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: ...<9 lines>...
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: chunked=chunked,
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: )
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 841, in urlopen
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: retries = retries.increment(
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: method, url, error=new_e, _pool=self, _stacktrace=sys.exc_info()[2]
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: )
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 519, in increment
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type]
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='openstack.eqiad1.wikimediacloud.org', port=25357): Max retries exceeded with url: /v3/auth/tokens (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f4e456b82d0>, 'Connection to openstack.eqiad1.wikimediacloud.org timed out. (connect timeout=None)'))
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: During handling of the above exception, another exception occurred:
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: Traceback (most recent call last):
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 1169, in _send_request
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: resp = self.session.request(method, url, **kwargs)
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: File "/usr/lib/python3/dist-packages/requests/sessions.py", line 589, in request
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: resp = self.send(prep, **send_kwargs)
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: File "/usr/lib/python3/dist-packages/requests/sessions.py", line 703, in send
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: r = adapter.send(request, **kwargs)
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: File "/usr/lib/python3/dist-packages/requests/adapters.py", line 688, in send
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: raise ConnectTimeout(e, request=request)
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: requests.exceptions.ConnectTimeout: HTTPSConnectionPool(host='openstack.eqiad1.wikimediacloud.org', port=25357): Max retries exceeded with url: /v3/auth/tokens (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f4e456b82d0>, 'Connection to openstack.eqiad1.wikimediacloud.org timed out. (connect timeout=None)'))
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: During handling of the above exception, another exception occurred:
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: Traceback (most recent call last):
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: File "/usr/local/sbin/wmcs-backup", line 2190, in <module>
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: args.func()
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: ~~~~~~~~~^^
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: File "/usr/local/sbin/wmcs-backup", line 2041, in <lambda>
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: func=lambda: get_current_volumes_state(from_cache=args.from_cache).delete_expired(
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: File "/usr/local/sbin/wmcs-backup", line 1647, in get_current_volumes_state
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: volume_id_to_volume_dict = get_volumes_info(from_cache)
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: File "/usr/local/sbin/wmcs-backup", line 1623, in get_volumes_info
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: volume_id_to_volume_info = {volume.id: volume.to_dict() for volume in clients.allvolumes()}
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: ~~~~~~~~~~~~~~~~~~^^
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: File "/usr/lib/python3/dist-packages/tenacity/__init__.py", line 332, in wrapped_f
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: return self(f, *args, **kw)
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: File "/usr/lib/python3/dist-packages/tenacity/__init__.py", line 469, in __call__
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: do = self.iter(retry_state=retry_state)
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: File "/usr/lib/python3/dist-packages/tenacity/__init__.py", line 370, in iter
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: result = action(retry_state)
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: File "/usr/lib/python3/dist-packages/tenacity/__init__.py", line 412, in exc_check
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: raise retry_exc.reraise()
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: ~~~~~~~~~~~~~~~~~^^
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: File "/usr/lib/python3/dist-packages/tenacity/__init__.py", line 185, in reraise
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: raise self.last_attempt.result()
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: ~~~~~~~~~~~~~~~~~~~~~~~~^^
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: File "/usr/lib/python3.13/concurrent/futures/_base.py", line 449, in result
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: return self.__get_result()
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: ~~~~~~~~~~~~~~~~~^^
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: File "/usr/lib/python3.13/concurrent/futures/_base.py", line 401, in __get_result
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: raise self._exception
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: File "/usr/lib/python3/dist-packages/tenacity/__init__.py", line 472, in __call__
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: result = fn(*args, **kwargs)
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: File "/usr/lib/python3/dist-packages/mwopenstackclients.py", line 410, in allvolumes
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: return self.cinderclient(projectid).volumes.list(search_opts=search_params)
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: File "/usr/lib/python3/dist-packages/cinderclient/v3/volumes_base.py", line 222, in list
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: return self._list(url, resource_type, limit=limit)
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: File "/usr/lib/python3/dist-packages/cinderclient/base.py", line 78, in _list
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: resp, body = self.api.client.get(url)
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: ~~~~~~~~~~~~~~~~~~~^^^^^
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: File "/usr/lib/python3/dist-packages/cinderclient/client.py", line 220, in get
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: return self._cs_request(url, 'GET', **kwargs)
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: File "/usr/lib/python3/dist-packages/cinderclient/client.py", line 211, in _cs_request
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: return self.request(url, method, **kwargs)
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: File "/usr/lib/python3/dist-packages/cinderclient/client.py", line 192, in request
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: resp, body = super(SessionClient, self).request(*args,
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: raise_exc=False,
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: ^^^^^^^^^^^^^^^^
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: **kwargs)
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: ^^^^^^^^^
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 657, in request
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: resp = self._request(url, method, **kwargs)
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 294, in _request
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: return self.session.request(url, method, **kwargs)
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 901, in request
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: auth_headers = self.get_auth_headers(auth)
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 1387, in get_auth_headers
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: return auth.get_headers(self)
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: ~~~~~~~~~~~~~~~~^^^^^^
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: File "/usr/lib/python3/dist-packages/keystoneauth1/plugin.py", line 124, in get_headers
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: token = self.get_token(session)
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: File "/usr/lib/python3/dist-packages/keystoneauth1/identity/base.py", line 91, in get_token
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: return self.get_access(session).auth_token
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: ~~~~~~~~~~~~~~~^^^^^^^^^
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: File "/usr/lib/python3/dist-packages/keystoneauth1/identity/base.py", line 139, in get_access
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: self.auth_ref = self.get_auth_ref(session)
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: ~~~~~~~~~~~~~~~~~^^^^^^^^^
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: File "/usr/lib/python3/dist-packages/keystoneauth1/identity/v3/base.py", line 240, in get_auth_ref
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: resp = session.post(
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: token_url,
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: ...<4 lines>...
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: **rkwargs,
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: )
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 1334, in post
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: return self.request(url, 'POST', **kwargs)
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 1057, in request
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: resp = send(**kwargs)
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 1175, in _send_request
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: raise exceptions.ConnectTimeout(msg)
Jun 11 07:41:48 cloudbackup2004 wmcs-backup[182725]: keystoneauth1.exceptions.connection.ConnectTimeout: Request to https://openstack.eqiad1.wikimediacloud.org:25357/v3/auth/tokens timed out
Jun 11 07:41:48 cloudbackup2004 systemd[1]: backup_cinder_volumes.service: Control process exited, code=exited, status=1/FAILURE
Jun 11 07:41:48 cloudbackup2004 systemd[1]: backup_cinder_volumes.service: Failed with result 'exit-code'.
Jun 11 07:41:48 cloudbackup2004 systemd[1]: Failed to start backup_cinder_volumes.service - backup cinder volumes.
```
This is probably the same issue we got with T419996 for cumin. I would have expected also the other openstack backups to fail (root disks and glance images) but the root ones are unclear (see T428865 ) while the glance images seems to be backing up correctly.
I'm working to add support for http proxies to `mwopenstackclients` that is the one used by `wmcs-backup.py` to connect to openstack.