Page MenuHomePhabricator

swift-repl failing due to auth failures
Closed, ResolvedPublic

Description

currently there is this alert in Icinga:

https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=ms-fe1005&service=Check+systemd+state

Initial thought was that this was T162122, but it's actually an authentication error:

2022-01-10T08:00:01.713061 Traceback (most recent call last):
2022-01-10T08:00:01.713243   File "./swiftrepl.py", line 528, in <module>
2022-01-10T08:00:01.713266     srcconn = srcconnpool.get()
2022-01-10T08:00:01.713280   File "/usr/lib/python2.7/dist-packages/cloudfiles/connection.py", line 479, in get
2022-01-10T08:00:01.720405     connobj = Connection(**self.connargs)
2022-01-10T08:00:01.720464   File "/usr/lib/python2.7/dist-packages/cloudfiles/connection.py", line 85, in __init__
2022-01-10T08:00:01.720491     self._authenticate()
2022-01-10T08:00:01.720506   File "./swiftrepl.py", line 34, in https_authenticate
2022-01-10T08:00:01.720519     (url, self.cdn_url, self.token) = self.auth.authenticate()
2022-01-10T08:00:01.720535   File "/usr/lib/python2.7/dist-packages/cloudfiles/authentication.py", line 74, in authenticate
2022-01-10T08:00:01.720701     raise AuthenticationFailed()
2022-01-10T08:00:01.720738 cloudfiles.errors.AuthenticationFailed
2022-01-10T08:00:01.725680 Command exited with non-zero status 1
2022-01-10T08:00:01.725755 0.04user 0.01system 0:00.11elapsed 46%CPU (0avgtext+0avgdata 14340maxresident)k
2022-01-10T08:00:01.725773 312inputs+0outputs (0major+2289minor)pagefaults 0swaps

Event Timeline

This is a consequence of two things:

  1. T296767
  2. /srv/software/swiftrepl/swiftrepl.conf being hand-edited

I've confirmed that swiftrepl.conf has the previous mw:media eqiad key in.

I've updated manually the relevant key on ms-fe1005 and swiftrepl is now going. I've also done the same edit on ms-fe2005.

We should probably puppetize swiftrepl.conf.

Cool, thanks for this and already creating T299125