Page MenuHomePhabricator

/usr/local/sbin/wikireplica_dns timeouts
Closed, ResolvedPublic

Description

While doing one of the last steps for T184805: Move some wikis to s5 which is https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Wiki_Replica_DNS

I got a timeout I tried it twice and got the same issue:

root@labcontrol1001:~# source <(sudo cat ~root/novaenv.sh)
root@labcontrol1001:~# /usr/local/sbin/wikireplica_dns --aliases --shard s5
Traceback (most recent call last):
  File "/usr/local/sbin/wikireplica_dns", line 127, in <module>
    main()
  File "/usr/local/sbin/wikireplica_dns", line 90, in main
    r = dns.zones(name=zone)
  File "/usr/lib/python2.7/dist-packages/mwopenstackclients.py", line 245, in zones
    r = self._get('/v2/zones', params=params)
  File "/usr/lib/python2.7/dist-packages/mwopenstackclients.py", line 227, in _get
    return self._req('GET', *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/mwopenstackclients.py", line 220, in _req
    r = map[verb.upper()](*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/api.py", line 67, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/api.py", line 53, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 468, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 576, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 437, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='cloudservices1003.wikimedia.org', port=9001): Max retries exceeded with url: /v2/zones?name=analytics.db.svc.eqiad.wmflabs. (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f068d0fdc10>: Failed to establish a new connection: [Errno 110] Connection timed out',))

We need to run those aliases for s3 and s5, as we moved some wikis from s3 to s5.

Event Timeline

@Marostegui this needs to run from cloudcontrol1003 now. I've updated the documentation and ran it one time (/usr/local/sbin/wikireplica_dns --aliases --shard s5).

Thanks for the heads up.

GTirloni claimed this task.

It needs to also run for S3 shard.

Thanks a lot :-)

No problem! Script executed for s5 and s3 (this one takes way longer).

Yeah, S3 has like 90 wikis!
Thank you!
Feel free to resolve this ticket if his is fully done from your side!
Thanks again!