Implement PoolCounter support in Thumbor
Closed, ResolvedPublic

Gilles created this task.Nov 18 2016, 4:42 PM
Gilles lowered the priority of this task from "High" to "Normal".Nov 18 2016, 6:20 PM
Gilles changed the title from "Implement PoolCounter support" to "Implement PoolCounter support in Thumbor".Nov 23 2016, 12:14 PM
Gilles moved this task from Inbox to Next-up on the Performance-Team board.Nov 23 2016, 8:40 PM
elukey removed a subscriber: elukey.Nov 24 2016, 11:31 AM
Gilles moved this task from Next-up to Doing on the Performance-Team board.Dec 22 2016, 4:00 PM
Gilles raised the priority of this task from "Normal" to "Needs Triage".
Gilles triaged this task as "Normal" priority.Dec 22 2016, 4:11 PM

Change 330157 had a related patch set uploaded (by Gilles):
Add poolcounter to thumbor role

https://gerrit.wikimedia.org/r/330157

Change 330157 merged by jenkins-bot:
Add poolcounter to thumbor role

https://gerrit.wikimedia.org/r/330157

Gilles moved this task from Doing to Blocked on the Performance-Team board.Jan 3 2017, 10:49 AM

Change 330646 had a related patch set uploaded (by Gilles):
Pass the filtered request headers to Thumbor

https://gerrit.wikimedia.org/r/330646

Change 330646 merged by Filippo Giunchedi:
Pass the filtered request headers to Thumbor

https://gerrit.wikimedia.org/r/330646

Gilles moved this task from Blocked to Doing on the Performance-Team board.Jan 6 2017, 12:36 PM

Change 330870 had a related patch set uploaded (by Gilles):
Enable Thumbor PoolCounter support

https://gerrit.wikimedia.org/r/330870

Change 330870 merged by jenkins-bot:
Enable Thumbor PoolCounter support

https://gerrit.wikimedia.org/r/330870

Gilles moved this task from Doing to Blocked on the Performance-Team board.Jan 9 2017, 10:14 AM
Gilles added a comment.EditedJan 20 2017, 10:30 AM

Issue when deployed, the X-Forwarded-For header in production is a list of IPs, which might explain why the throttle kicks in incorectly for every request:

Jan 20 10:26:35 thumbor1002 thumbor@8836[32389]: 2017-01-20 10:26:35,054 8836 thumbor:ERROR [ImagesHandler] Throttled by PoolCounter: thumbor-ip-x.x.x.x, 10.64.32.81, 10.64.32.81, 10.64.0.167 {'workers': 32, 'maxqueue': 500, 'timeout': 8}
Jan 20 10:26:01 thumbor1002 thumbor@8836[32389]: 2017-01-20 10:26:01,925 8836 thumbor:ERROR [ImagesHandler] Throttled by PoolCounter: thumbor-ip-x:x:x:x:x:x:x:x, 10.128.0.113, 10.128.0.113, 10.128.0.115, 10.192.32.116, 10.64.0.167 {'workers': 32, 'maxqueue': 500, 'timeout': 8}

We should only be looking at the client IP, which I imagine is the first in the list.

Change 333228 had a related patch set uploaded (by Gilles):
Remove broken Thumbor IP throttling from configuration

https://gerrit.wikimedia.org/r/333228

Change 333228 merged by Filippo Giunchedi:
Remove broken Thumbor IP throttling from configuration

https://gerrit.wikimedia.org/r/333228

Change 334251 had a related patch set uploaded (by Gilles):
Upgrade to 0.1.33

https://gerrit.wikimedia.org/r/334251

Change 334251 merged by Filippo Giunchedi:
Upgrade to 0.1.33

https://gerrit.wikimedia.org/r/334251

Mentioned in SAL (#wikimedia-operations) [2017-01-30T09:44:07Z] <godog> upgrade to thumbor 0.1.33 - T151066

Gilles closed this task as "Resolved".Feb 1 2017, 7:47 PM