Page MenuHomePhabricator

swift-account-stats: Max retries exceeded with url: /auth/v1.0
Closed, ResolvedPublic

Description

swift-account-stats sometimes fails when running via cron with the following error:

 Traceback (most recent call last):                                                                                                                                                             
  File "/usr/local/bin/swift-account-stats", line 76, in <module>                                                                                                                              
    sys.exit(main())                                                                                                                                                                           
  File "/usr/local/bin/swift-account-stats", line 49, in main                                                                                                                                  
    headers = connection.head_account()                                                                                                                                                        
  File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1229, in head_account                                                                                                    
    return self._retry(None, head_account)                                                                                                                                                     
  File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1181, in _retry                                                                                                          
    self.url, self.token = self.get_auth()                                                                                                                                                     
  File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1155, in get_auth                                                                                                        
    insecure=self.insecure)                                                                                                                                                                    
  File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 318, in get_auth                                                                                                         
    insecure=insecure)                                                                                                                                                                         
  File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 228, in get_auth_1_0                                                                                                     
    {'X-Auth-User': user, 'X-Auth-Key': key})                                                                                                                                                  
  File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 188, in request                                                                                                          
    files=files, **self.requests_args)                                                                                                                                                         
  File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 177, in _request                                                                                                         
    return requests.request(*arg, **kwarg)                                                                                                                                                     
  File "/usr/lib/python2.7/dist-packages/requests/api.py", line 44, in request                                                                                                                 
    return session.request(method=method, url=url, **kwargs)                                                                                                                                   
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 383, in request                                                                                                           
    resp = self.send(prep, **send_kwargs)                                                                                                                                                      
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 486, in send                                                                                                              
    r = adapter.send(request, **kwargs)                                                                                                                                                        
  File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 378, in send                                                                                                              
    raise ConnectionError(e)                                                                                                                                                                   
requests.exceptions.ConnectionError: HTTPConnectionPool(host='ms-fe.svc.codfw.wmnet', port=80): Max retries exceeded with url: /auth/v1.0 (Caused by <class 'socket.error'>: [Errno 111] Connection refused)

Sometimes this only triggers a few cron emails, in other situations such as on January 6th we got > 100 messages.

Event Timeline

fgiunchedi claimed this task.
fgiunchedi subscribed.

I'm assuming this happened on ms-fe2001 today? It was during decomission and a legitimate error, I'll tentatively resolve