Huge increase in cache_upload 404s due to buggy client-side code from graphiq.com
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	• ema
	Nov 23 2016, 10:40 AM

Description

We've noticed a very significant increase in 404s on cache_upload starting on 2016-11-22.

The requested file is https://upload.wikimedia.org/wikipedia/commons/8/80/KrisKobach.jpg, and from the referal we've found out that graphiq.com is responsible for those requests.

The reason for such a significant increase seems to be a bug in graphiq.com's javascript code:

<img class='ew-image ' src='https://upload.wikimedia.org/wikipedia/commons/thumb/8/80/KrisKobach.jpg/250px-KrisKobach.jpg' width='180px' height=180px onerror='if(this.errr){$(this).remove();}this.errr=1;this.src="https://upload.wikimedia.org/wikipedia/commons/8/80/KrisKobach.jpg";'/>

Neither of the two URLs in the snippet above can be found on upload, resulting in a endless attempt to load the second one.

We might want to deploy a workaround (eg: returning 401s instead of 404s for such requests) in case the situation gets worse.

Details

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	cache_upload: stop graphiq.com buggy javascript	operations/puppet	production	+7 -0

Customize query in gerrit

Related Objects

Mentioned In: T151067: Implement rate limiter in Thumbor

Event Timeline

• ema created this task.Nov 23 2016, 10:40 AM

Restricted Application added a project: SRE. · View Herald TranscriptNov 23 2016, 10:40 AM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

• ema triaged this task as Medium priority.Nov 23 2016, 10:42 AM

Change 323135 had a related patch set uploaded (by Ema):
cache_upload: stop graphiq.com buggy javascript

https://gerrit.wikimedia.org/r/323135

gerritbot added a project: Patch-For-Review.Nov 23 2016, 10:45 AM

fgiunchedi mentioned this in T151067: Implement rate limiter in Thumbor.Nov 23 2016, 8:35 PM

• ema moved this task from Backlog to Caching on the Traffic board.Nov 24 2016, 1:28 PM

The amount of upload 404s decreased significantly, we're almost back to normal: https://grafana.wikimedia.org/dashboard/db/varnish-aggregate-client-status-codes?panelId=3&fullscreen&var-site=All&var-cache_type=upload&var-status_type=4&from=1479683011958&to=1480222523270

Leaving this ticket open though given that they still haven't fixed the javascript bug. The decrease in 404s is probably due to the fact that people are not opening the Kris Kobach page as much as they did last week apparently.

404 rate back to normal, closing.

Change 323135 abandoned by Ema:
cache_upload: stop graphiq.com buggy javascript

Reason:
graphiq.com's bug seems fixed, 404 rate back to normal. Abandoning.