Varnish rate limiting has broken beta
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Reedy
	Nov 11 2015, 1:30 AM

Description

http://meta.wikimedia.beta.wmflabs.org/wiki/Special:UserRights

Error: 429, Request Rate Exceeded at Wed, 11 Nov 2015 01:10:36 GMT

Details

	Subject	Repo	Branch	Lines +/-
	Revert "varnish: misspass limiter"	operations/puppet	production	+0 -34

Customize query in gerrit

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		Krenair	T118486 [Regression pre-wmf.7] Images for musical scores, formulæ, heiroglyphics, thumbnails are returning 429s in the Beta Cluster when using VE (and other times?)
		Resolved		• GWicke	T118362 Varnish rate limiting has broken beta

Event Timeline

Reedy created this task.Nov 11 2015, 1:30 AM

Reedy raised the priority of this task from to Needs Triage.

Reedy updated the task description. (Show Details)

Reedy added projects: Beta-Cluster-Infrastructure, SRE, Blocked-on-Operations.

Reedy added subscribers: Reedy, BBlack.

Restricted Application added subscribers: StudiesWorld, Aklapper. · View Herald TranscriptNov 11 2015, 1:30 AM

Reedy triaged this task as High priority.Nov 11 2015, 1:31 AM

Reedy set Security to None.

Other victims are RESTBase integration tests, from either the office or travis. About 40 API requests distributed across a ~50 second test run.

Maybe oauth from labs is affected. Getting reports that login screens like:

https://tools.wmflabs.org/wikidata-game/

using the "widar" tool (https://tools.wmflabs.org/widar/) are broken:

< Reedy> The URI you have requested, /widar/index.php?action=authorize, appears to be non-functional at this time

edit: made a separate ticket for that: T118363

In case it helps with debugging, there seems to have been a multi-hour delay between the deploy (around 13:46 UTC) & blocking starting in the office. I noticed the first full block when running RESTBase tests just before 00:56 UTC. The same tests had been working okay about 40 minutes earlier.

Change 252385 had a related patch set uploaded (by Faidon Liambotis):
Revert "varnish: misspass limiter"

https://gerrit.wikimedia.org/r/252385

gerritbot added a project: Patch-For-Review.Nov 11 2015, 2:08 AM

This works again http://deployment.wikimedia.beta.wmflabs.org/wiki/Main_Page after Faidon restarted varnish in beta.

Confirmed resolved for RESTBase tests as well. Most of those are actually hitting labs these days, so they were affected by the labs-specific outage.

• GWicke added a subscriber: • mobrovac.Nov 11 2015, 2:56 AM

http://en.m.wikipedia.beta.wmflabs.org/ is broken

Jdforrester-WMF added a parent task: T118486: [Regression pre-wmf.7] Images for musical scores, formulæ, heiroglyphics, thumbnails are returning 429s in the Beta Cluster when using VE (and other times?).Nov 12 2015, 11:09 PM

<Krenair> I took a look through deployment-cache-text04:/root/.bash_history and found the command they used
<Krenair> Ran it on deployment-cache-mobile04
<Reedy> what was it?
<Krenair> #1447207753
<Krenair> service  varnish restart
<Krenair> #1447207761
<Krenair> service  varnish-frontend restart

Krenair mentioned this in T118486: [Regression pre-wmf.7] Images for musical scores, formulæ, heiroglyphics, thumbnails are returning 429s in the Beta Cluster when using VE (and other times?).Nov 12 2015, 11:22 PM

Change 252385 abandoned by Faidon Liambotis:
Revert "varnish: misspass limiter"

Reason:
Everything seems to be working fine so far.