Page MenuHomePhabricator

Investigate today's EU SWAT scap & infrastructure issues
Closed, ResolvedPublic

Description

Today's EU SWAT had to be cancelled due to scap and other infrastructure issues.

Event Timeline

TheresNoTime renamed this task from Investigate todays EU SWAT scap & infraestructure issues to Investigate todays EU SWAT scap & infrastructure issues.Sep 11 2017, 2:11 PM
TheresNoTime updated the task description. (Show Details)
TheresNoTime subscribed.
MarcoAurelio renamed this task from Investigate todays EU SWAT scap & infrastructure issues to Investigate today's EU SWAT scap & infrastructure issues.Sep 11 2017, 2:13 PM
MarcoAurelio updated the task description. (Show Details)
greg claimed this task.
greg moved this task from Active investigation to Active Situation on the Wikimedia-Incident board.
greg subscribed.

This was due to two issues:

  1. The "too many auth failures" from scap/ssh: T172333. This is addressed.
  1. CI being temporarily overloaded. The nodepool CI max number of instances was reduced and the API rate was increased this weekend due to an WMC outage. Then a large number of (known to be slow) wikidata changes were submitted and +2 causing a long delay. The nodepool limits were lifted earlier today.