Fastly CDN has some transient issue (was: pypi.python.org having transient issues (503 errors) )
Closed, ResolvedPublic
Actions

Assigned To

None

Authored By

	hashar
	Jun 28 2017, 2:05 PM

Description

Jenkins jobs relying on python / tox might have issue as python CDN has some issues
Example:

HTTPSConnectionPool(host='pypi.python.org', port=443): Max retries exceeded with url: /simple/setuptools/ (Caused by ResponseError('too many 503 error responses',))

Acknowledged by Pypi and Fastly.

On https://status.fastly.com/ :

Identified - The issue has been identified and a fix is being implemented.
Jun 28, 14:05 UTC
Investigating - Fastly is reporting a global event on our network as this time; multiple teams are responding and investigating.
Jun 28, 14:00 UTC

On https://status.python.org/ :

Investigating - We are currently investigating issues affecting HTTP services fronted by our CDN.
Jun 28, 14:02 UTC

Identified - We have confirmation from our CDN provider that there is a "global event" on their network, and are tracking the issue waiting for resolution.
Jun 28, 14:03 UTC

pypi.python.org uses Fastly CDN.

Event Timeline

hashar created this task.Jun 28 2017, 2:05 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 28 2017, 2:05 PM

hashar added a project: Upstream.Jun 28 2017, 2:06 PM

Mentioned in SAL (#wikimedia-releng) [2017-06-28T14:06:16Z] <hashar> pypi.python.org has an issue with its CDN . That would affect any CI jobs relying on tox/python - See https://status.python.org for updates and T169091

Mentioned in SAL (#wikimedia-operations) [2017-06-28T14:06:19Z] <hashar> pypi.python.org has an issue with its CDN . That would affect any CI jobs relying on tox/python - See https://status.python.org for updates and T169091

$ dig +short ANY pypi.python.org
prod.python.map.fastly.net.

@Paladox mentioned a similar error with apt-get:

[16:06:30]  <paladox>	Err:5 http://cdn-fastly.deb.debian.org/debian stretch Release                  
[16:06:30]  <paladox>	  503  Maximum threads for service reached [IP: 151.101.88.204 80]

Paladox added a comment.Jun 28 2017, 2:10 PM

This comment was removed by Paladox.

hashar renamed this task from pypi.python.org having transient issues (503 errors) to Fastly CDN has some transient issue (was: pypi.python.org having transient issues (503 errors) ).Jun 28 2017, 2:10 PM

hashar updated the task description. (Show Details)

hashar updated the task description. (Show Details)Jun 28 2017, 2:14 PM

FIXED

https://status.fastly.com/

Jun 28, 14:40 UTC Resolved - This incident has been resolved.
Jun 28, 14:16 UTC Monitoring - A fix has been implemented and we are monitoring the results.
Jun 28, 14:05 UTC Identified - The issue has been identified and a fix is being implemented.
Jun 28, 14:00 UTC Investigating - Fastly is reporting a global event on our network as this time; multiple teams are responding and investigating.

Mentioned in SAL (#wikimedia-operations) [2017-06-28T14:42:58Z] <hashar> pypi.python.org is back again - T169091

Mentioned in SAL (#wikimedia-releng) [2017-06-28T14:43:07Z] <hashar> pypi.python.org is back again - T169091

Fastly CDN has some transient issue (was: pypi.python.org having transient issues (503 errors) )Closed, ResolvedPublicActions

Description

Event Timeline

Fastly CDN has some transient issue (was: pypi.python.org having transient issues (503 errors) )
Closed, ResolvedPublic
Actions