Page MenuHomePhabricator

/usr/local/sbin/wikireplica_dns timeouts
Closed, ResolvedPublic

Description

While doing one of the last steps for T184805: Move some wikis to s5 which is https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Wiki_Replica_DNS

I got a timeout I tried it twice and got the same issue:

root@labcontrol1001:~# source <(sudo cat ~root/novaenv.sh)
root@labcontrol1001:~# /usr/local/sbin/wikireplica_dns --aliases --shard s5
Traceback (most recent call last):
  File "/usr/local/sbin/wikireplica_dns", line 127, in <module>
    main()
  File "/usr/local/sbin/wikireplica_dns", line 90, in main
    r = dns.zones(name=zone)
  File "/usr/lib/python2.7/dist-packages/mwopenstackclients.py", line 245, in zones
    r = self._get('/v2/zones', params=params)
  File "/usr/lib/python2.7/dist-packages/mwopenstackclients.py", line 227, in _get
    return self._req('GET', *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/mwopenstackclients.py", line 220, in _req
    r = map[verb.upper()](*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/api.py", line 67, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/api.py", line 53, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 468, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 576, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 437, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='cloudservices1003.wikimedia.org', port=9001): Max retries exceeded with url: /v2/zones?name=analytics.db.svc.eqiad.wmflabs. (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f068d0fdc10>: Failed to establish a new connection: [Errno 110] Connection timed out',))

We need to run those aliases for s3 and s5, as we moved some wikis from s3 to s5.

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 10 2018, 12:09 PM
Marostegui triaged this task as High priority.Oct 10 2018, 12:09 PM

@Marostegui this needs to run from cloudcontrol1003 now. I've updated the documentation and ran it one time (/usr/local/sbin/wikireplica_dns --aliases --shard s5).

Thanks for the heads up.

GTirloni closed this task as Resolved.Oct 15 2018, 5:13 PM
GTirloni claimed this task.
Marostegui reopened this task as Open.Oct 15 2018, 5:31 PM

It needs to also run for S3 shard.

Thanks a lot :-)

This comment was removed by GTirloni.

No problem! Script executed for s5 and s3 (this one takes way longer).

Yeah, S3 has like 90 wikis!
Thank you!
Feel free to resolve this ticket if his is fully done from your side!
Thanks again!

GTirloni closed this task as Resolved.Oct 15 2018, 6:25 PM