⚓ T151066 Implement PoolCounter support in Thumbor

Subject	Repo	Branch	Lines +/-
Upgrade to 0.1.33	operations/debs/python-thumbor-wikimedia	master	+21 -5
Remove broken Thumbor IP throttling from configuration	operations/puppet	production	+0 -8
Enable Thumbor PoolCounter support	mediawiki/vagrant	master	+36 -1
Pass the filtered request headers to Thumbor	operations/puppet	production	+3 -0
Add poolcounter to thumbor role	mediawiki/vagrant	master	+1 -0

Status	Assigned	Task
Resolved	• Gilles	T121388 Service-based thumbnailing re-architecture in production with Thumbor
Resolved	• Gilles	T151066 Implement PoolCounter support in Thumbor
Resolved	• Gilles	T169312 Implement poolcounter failover in Thumbor
Resolved	• Gilles	T169313 Investigate poolcounter failure leading to thumbor failing to generate thumbs

• Gilles created this task.Nov 18 2016, 4:42 PM

• Gilles lowered the priority of this task from High to Medium.Nov 18 2016, 6:20 PM

• Gilles edited parent tasks, added: T139606: add thumbor to production infrastructure; removed: T150745: Reimplement various rate-limiting mechanisms in Thumbor.Nov 23 2016, 12:08 PM

• Gilles renamed this task from Implement PoolCounter support to Implement PoolCounter support in Thumbor.Nov 23 2016, 12:14 PM

• Gilles moved this task from Inbox, needs triage to To-do: Goals prioritized current Quarter on the Performance-Team board.Nov 23 2016, 8:40 PM

elukey unsubscribed.Nov 24 2016, 11:31 AM

• Gilles edited parent tasks, added: T121388: Service-based thumbnailing re-architecture in production with Thumbor; removed: T139606: add thumbor to production infrastructure.Dec 6 2016, 2:32 PM

fgiunchedi mentioned this in T151851: Thumbor resource consumption is spiky.Dec 12 2016, 7:16 PM

• Gilles raised the priority of this task from Medium to Needs Triage.Dec 22 2016, 4:00 PM

• Gilles moved this task from To-do: Goals prioritized current Quarter to Doing (old) on the Performance-Team board.

• Gilles triaged this task as Medium priority.Dec 22 2016, 4:11 PM

Nemo_bis awarded a token.Dec 26 2016, 9:16 AM

Change 330157 had a related patch set uploaded (by Gilles):
Add poolcounter to thumbor role

https://gerrit.wikimedia.org/r/330157

Change 330157 merged by jenkins-bot:
Add poolcounter to thumbor role

https://gerrit.wikimedia.org/r/330157

• Gilles added a revision: Restricted Differential Revision.Jan 3 2017, 10:49 AM

• Gilles moved this task from Doing (old) to Blocked (old) on the Performance-Team board.Jan 3 2017, 10:49 AM

Change 330646 had a related patch set uploaded (by Gilles):
Pass the filtered request headers to Thumbor

https://gerrit.wikimedia.org/r/330646

Change 330646 merged by Filippo Giunchedi:
Pass the filtered request headers to Thumbor

https://gerrit.wikimedia.org/r/330646

• Gilles added a commit: rTHMBREXT6cf4b802f799: Add PoolCounter support.Jan 6 2017, 12:10 PM

• Gilles moved this task from Blocked (old) to Doing (old) on the Performance-Team board.Jan 6 2017, 12:36 PM

Change 330870 had a related patch set uploaded (by Gilles):
Enable Thumbor PoolCounter support

https://gerrit.wikimedia.org/r/330870

Change 330870 merged by jenkins-bot:
Enable Thumbor PoolCounter support

https://gerrit.wikimedia.org/r/330870

• Gilles moved this task from Doing (old) to Blocked (old) on the Performance-Team board.Jan 9 2017, 10:14 AM

Issue when deployed, the X-Forwarded-For header in production is a list of IPs, which might explain why the throttle kicks in incorectly for every request:

Jan 20 10:26:35 thumbor1002 thumbor@8836[32389]: 2017-01-20 10:26:35,054 8836 thumbor:ERROR [ImagesHandler] Throttled by PoolCounter: thumbor-ip-x.x.x.x, 10.64.32.81, 10.64.32.81, 10.64.0.167 {'workers': 32, 'maxqueue': 500, 'timeout': 8}
Jan 20 10:26:01 thumbor1002 thumbor@8836[32389]: 2017-01-20 10:26:01,925 8836 thumbor:ERROR [ImagesHandler] Throttled by PoolCounter: thumbor-ip-x:x:x:x:x:x:x:x, 10.128.0.113, 10.128.0.113, 10.128.0.115, 10.192.32.116, 10.64.0.167 {'workers': 32, 'maxqueue': 500, 'timeout': 8}

We should only be looking at the client IP, which I imagine is the first in the list.

Change 333228 had a related patch set uploaded (by Gilles):
Remove broken Thumbor IP throttling from configuration

https://gerrit.wikimedia.org/r/333228

Change 333228 merged by Filippo Giunchedi:
Remove broken Thumbor IP throttling from configuration

https://gerrit.wikimedia.org/r/333228

Change 334251 had a related patch set uploaded (by Gilles):
Upgrade to 0.1.33

https://gerrit.wikimedia.org/r/334251

Change 334251 merged by Filippo Giunchedi:
Upgrade to 0.1.33

https://gerrit.wikimedia.org/r/334251

Mentioned in SAL (#wikimedia-operations) [2017-01-30T09:44:07Z] <godog> upgrade to thumbor 0.1.33 - T151066

• Gilles closed this task as Resolved.Feb 1 2017, 7:47 PM

akosiaris mentioned this in T160692: Use poolcounter to limit number of connections to ores uwsgi.Mar 20 2017, 6:56 PM