Page MenuHomePhabricator
Create Task
Maniphest T95163

Browser tests running against beta all failing because of mw-api-siteinfo.py
Closed, ResolvedPublic
Actions

Description

Started happening consistently on the 4th. Examples:

https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce/557/console
https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce/558/console
https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce/559/console

08:12:09 Traceback (most recent call last):
08:12:09 File "/srv/deployment/integration/slave-scripts/bin/mw-api-siteinfo.py", line 92, in <module>
08:12:09 main()
08:12:09 File "/srv/deployment/integration/slave-scripts/bin/mw-api-siteinfo.py", line 77, in main
08:12:09 response = requests.get(mw_api_url, params=API_QUERY)
08:12:09 File "/usr/lib/python2.7/dist-packages/requests/api.py", line 55, in get
08:12:09 return request('get', url, kwargs)
08:12:09 File "/usr/lib/python2.7/dist-packages/requests/api.py", line 44, in request
08:12:09 return session.request(method=method, url=url, kwargs)
08:12:09 File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 455, in request
08:12:09 resp = self.send(prep, send_kwargs)
08:12:09 File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 558, in send
08:12:09 r = adapter.send(request, kwargs)
08:12:09 File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 378, in send
08:12:09 raise ConnectionError(e)
08:12:09 requests.exceptions.ConnectionError: HTTPConnectionPool(host='en.wikipedia.beta.wmflabs.org', port=80): Max retries exceeded with url: /w/api.php?action=query&meta=siteinfo&siprop=general&format=json (Caused by <class 'socket.error'>: [Errno 110] Connection timed out)

The URL being requested works fine right now: http://en.wikipedia.beta.wmflabs.org/w/api.php?action=query&meta=siteinfo&siprop=general&format=json

Maybe the format changed and the python script keeps retrying and not finding what it needs? (I haven't looked at the source yet)

Related Objects
Search...

Event Timeline

• Gilles created this task.Apr 6 2015, 11:09 AM
• Gilles raised the priority of this task from to High.
• Gilles updated the task description. (Show Details)
• Gilles added a project: Browser-Tests-Infrastructure.
• Gilles subscribed.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 6 2015, 11:09 AM
• Gilles added a project: Release-Engineering-Team.Apr 6 2015, 11:09 AM
• Gilles added a comment.Apr 6 2015, 11:15 AM

Browser tests are definitely in bad shape across the board but beta is probably to blame. The only ones that stayed green are running against non-beta sites.

• Gilles renamed this task from Browser test failing because of mw-api-siteinfo.py to Browser tests failing because of mw-api-siteinfo.py.Apr 6 2015, 11:15 AM
• Gilles renamed this task from Browser tests failing because of mw-api-siteinfo.py to Browser tests running against beta all failing because of mw-api-siteinfo.py.
• Gilles raised the priority of this task from High to Unbreak Now!.
• Gilles set Security to None.
• Gilles added a comment.Apr 6 2015, 11:25 AM

Tried to run a test manually and the issue is still happening.

Tried hitting the API url in a SauceLabs interactive session and it works fine:
https://saucelabs.com/tests/566c6261360e4c9c8f74f8cf09745f2c

Although, if I follow the script correctly, the http request probably runs on the integration-slave* machines and not on saucelabs, which comes into play later.

• Gilles added a comment.Apr 6 2015, 11:36 AM

If someone who's an admin on labs for the "Integration" project could add me to it, that'd be great. Right now I can't SSH into the integrattion-slave* machines to troubleshoot this further.

Requesting the beta API url from an unrelated labs instance works fine.

• Gilles added a comment.Apr 6 2015, 11:40 AM

The issue, which seems to be a network/connectivity problem, seems to have actually started on April 3rd, and affects other urls as well: https://integration.wikimedia.org/ci/job/UploadWizard-api-commons.wikimedia.beta.wmflabs.org/1729/console

greg subscribed.Apr 9 2015, 3:29 PM

Sorry about the radio silence here.

In T95163#1182020, @Gilles wrote:

If someone who's an admin on labs for the "Integration" project could add me to it, that'd be great. Right now I can't SSH into the integrattion-slave* machines to troubleshoot this further.

Should now be done. You've even now got sudo :)

greg added subscribers: zeljkofilipin, dduvall.Apr 9 2015, 3:37 PM
greg added a subscriber: hashar.Apr 9 2015, 3:44 PM
greg moved this task from Inbox to Next on the Browser-Tests-Infrastructure board.Apr 9 2015, 3:46 PM
hashar added a comment.Apr 16 2015, 8:14 AM

The root cause is that the CI instances have been migrated to a new DNS resolver which for *.beta.wmflabs.org replied with the public IP address instead of the private IP of the instance. Since labs public IP are not reacheable from labs due to NAT, the beta cluster was no more reacheable.

References:

It is all good now :)

hashar closed this task as Resolved.Apr 16 2015, 8:15 AM
hashar claimed this task.
hashar added a subtask: T95273: integration labs project DNS resolver improperly switched to openstack-designate.
greg moved this task from Next to Done on the Browser-Tests-Infrastructure board.Apr 16 2015, 4:14 PM
greg moved this task from INBOX to Done on the Release-Engineering-Team board.May 23 2015, 2:14 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits