Page MenuHomePhabricator
Create Task
Maniphest T151851

Thumbor resource consumption is spiky
Closed, ResolvedPublic
Actions

Description

https://grafana.wikimedia.org/dashboard/db/prometheus-machine-stats?var-server=thumbor1001:9100&var-datasource=eqiad%20prometheus%2Fops&from=now-60d&to=now

Especially since the beginning of November we can see that the system load spikes have gotten bigger.

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
swift: enable back shadow traffic to thumborFilippo Giunchedioperations/puppetproduction+17 -1
swift: drain thumbor trafficFilippo Giunchedioperations/puppetproduction+1 -17
Customize query in gerrit

Related Objects

Event Timeline

• Gilles created this task.Nov 29 2016, 8:48 AM
• Gilles moved this task from Inbox, needs triage to Backlog: Maintenance, non-prioritized on the Performance-Team board.Dec 1 2016, 9:55 PM
• Gilles moved this task from Backlog: Maintenance, non-prioritized to To-do: Goals prioritized current Quarter on the Performance-Team board.Dec 6 2016, 12:23 AM
• Gilles added a comment.Dec 6 2016, 1:32 PM

Seems to have gotten better since I fixed the big memory leak, but we'll have to wait and see.

• Gilles removed a parent task: T139606: add thumbor to production infrastructure.Dec 6 2016, 2:34 PM
• Gilles claimed this task.Dec 7 2016, 5:25 PM
fgiunchedi added a comment.Dec 12 2016, 7:16 PM

The spikes in load average on thumbor machines keep happening, it correlates with high iowait too. The iowait doesn't seem related to swap activity, rather to application writes overloading the disks.

e.g.

$ sudo find /srv/thumbor/tmp/ -type f -ls | grep -i magi
11796587 169460 -rw-------   1 thumbor  thumbor  173522752 Dec 12 18:52 /srv/thumbor/tmp/thumbor@8801/magick-2fGojBNXq9JkK
11796607 83192 -rw-------   1 thumbor  thumbor  168633504 Dec 12 19:00 /srv/thumbor/tmp/thumbor@8801/magick-2KuUwLZ7ceX3S
11796621 169416 -rw-------   1 thumbor  thumbor  173475840 Dec 12 18:54 /srv/thumbor/tmp/thumbor@8816/magick-2JlKQIpqDTGEW
11796740 33604 -rw-------   1 thumbor  thumbor  171488096 Dec 12 18:59 /srv/thumbor/tmp/thumbor@8816/magick-2QT-pK5zCsNP4
11796692 167544 -rw-------   1 thumbor  thumbor  171488096 Dec 12 19:00 /srv/thumbor/tmp/thumbor@8808/magick-2yWb4SZ2tyeuo
11796756 163748 -rw-------   1 thumbor  thumbor  172824288 Dec 12 19:00 /srv/thumbor/tmp/thumbor@8837/magick-208oPYfR0ouIa
11796578 169480 -rw-------   1 thumbor  thumbor  173462016 Dec 12 18:58 /srv/thumbor/tmp/thumbor@8837/magick-2RRNpe2roDP2f
11796667 164688 -rw-------   1 thumbor  thumbor  168633504 Dec 12 18:57 /srv/thumbor/tmp/thumbor@8824/magick-2iwYvowKJWTk4
11796630 167628 -rw-------   1 thumbor  thumbor  171639936 Dec 12 18:54 /srv/thumbor/tmp/thumbor@8833/magick-2Tuz8W3BuocOt
11796864 156624 -rw-------   1 thumbor  thumbor  168633504 Dec 12 18:59 /srv/thumbor/tmp/thumbor@8834/magick-2kpWvkp4i18ip
11796803 166616 -rw-------   1 thumbor  thumbor  170607424 Dec 12 18:52 /srv/thumbor/tmp/thumbor@8836/magick-2fDmqwCc8SCsl
11796577 29548 -rw-------   1 thumbor  thumbor  171488096 Dec 12 19:00 /srv/thumbor/tmp/thumbor@8812/magick-2C_D1xW7n8Jou
root@thumbor1001:~# find /srv/thumbor/tmp/ -type f -ls | grep -i magi
19398878 28004 -rw-------   1 thumbor  thumbor  171488096 Dec 12 18:55 /srv/thumbor/tmp/thumbor@8824/magick-29wQc9DEvgSIL
19399037 298436 -rw-------   1 thumbor  thumbor  338000000 Dec 12 19:03 /srv/thumbor/tmp/thumbor@8832/magick-2IDwM7M1WfPea
19399001 167544 -rw-------   1 thumbor  thumbor  171488096 Dec 12 19:03 /srv/thumbor/tmp/thumbor@8812/magick-2lh7Sok65bIaf
19398818 31188 -rw-------   1 thumbor  thumbor  170607424 Dec 12 18:57 /srv/thumbor/tmp/thumbor@8833/magick-2p03qSUENZauW
19398826 169728 -rw-------   1 thumbor  thumbor  173765696 Dec 12 18:55 /srv/thumbor/tmp/thumbor@8833/magick-27DfV-Xr96G2j
19399324 55664 -rw-------   1 thumbor  thumbor  171488096 Dec 12 19:03 /srv/thumbor/tmp/thumbor@8806/magick-2kJhQlpFwIt2U
19399216 167520 -rw-------   1 thumbor  thumbor  171488096 Dec 12 19:01 /srv/thumbor/tmp/thumbor@8808/magick-2z7Dr_YRDN1Sa
19398987 84432 -rw-------   1 thumbor  thumbor  171488096 Dec 12 18:57 /srv/thumbor/tmp/thumbor@8826/magick-2t3QhovEAkzA2
19399257 136336 -rw-------   1 thumbor  thumbor  170607424 Dec 12 18:59 /srv/thumbor/tmp/thumbor@8816/magick-23LPqFwpOMRD5
19398815 30332 -rw-------   1 thumbor  thumbor  170607424 Dec 12 19:03 /srv/thumbor/tmp/thumbor@8829/magick-24XGeTlH6KdXE
19398752 133640 -rw-------   1 thumbor  thumbor  171488096 Dec 12 19:03 /srv/thumbor/tmp/thumbor@8835/magick-2dtnKbkyHaF3J
19399326 167496 -rw-------   1 thumbor  thumbor  171488096 Dec 12 19:01 /srv/thumbor/tmp/thumbor@8801/magick-28b9tez3e8qRj
19398783 50672 -rw-------   1 thumbor  thumbor  171488096 Dec 12 19:03 /srv/thumbor/tmp/thumbor@8836/magick-2Rak7B42JrLPj
19399533 167488 -rw-------   1 thumbor  thumbor  171488096 Dec 12 18:54 /srv/thumbor/tmp/thumbor@8836/magick-2921OB0phUD68
19399266 110592 -rw-------   1 thumbor  thumbor  171488096 Dec 12 19:01 /srv/thumbor/tmp/thumbor@8805/magick-2JnOydF4QvNHf
19399099 168780 -rw-------   1 thumbor  thumbor  172824288 Dec 12 18:53 /srv/thumbor/tmp/thumbor@8834/magick-2QLDfkVTqZnwA
19399033 169416 -rw-------   1 thumbor  thumbor  173475840 Dec 12 18:54 /srv/thumbor/tmp/thumbor@8822/magick-2C5cp7R7batVF
19398973 29540 -rw-------   1 thumbor  thumbor  171488096 Dec 12 19:03 /srv/thumbor/tmp/thumbor@8822/magick-2UYm55iFKJmqw
19399535 122528 -rw-------   1 thumbor  thumbor  171488096 Dec 12 19:03 /srv/thumbor/tmp/thumbor@8837/magick-2Nzfg6Tjq7Zic

I haven't confirmed it but I'm assuming this traffic might be related to a single big file being requested. Things like T151066: Implement PoolCounter support in Thumbor or T151067: Implement rate limiter in Thumbor would help mitigating this I'd say

gerritbot added a comment.Dec 22 2016, 6:52 PM

Change 328713 had a related patch set uploaded (by Filippo Giunchedi):
swift: drain thumbor traffic

https://gerrit.wikimedia.org/r/328713

gerritbot added a project: Patch-For-Review.Dec 22 2016, 6:52 PM
gerritbot added a comment.Dec 22 2016, 6:59 PM

Change 328713 merged by Filippo Giunchedi:
swift: drain thumbor traffic

https://gerrit.wikimedia.org/r/328713

• Gilles moved this task from To-do: Goals prioritized current Quarter to Doing (old) on the Performance-Team board.Jan 6 2017, 12:17 PM
gerritbot added a comment.Jan 20 2017, 9:53 AM

Change 333222 had a related patch set uploaded (by Filippo Giunchedi):
swift: enable back shadow traffic to thumbor

https://gerrit.wikimedia.org/r/333222

gerritbot added a comment.Jan 20 2017, 9:57 AM

Change 333222 merged by Filippo Giunchedi:
swift: enable back shadow traffic to thumbor

https://gerrit.wikimedia.org/r/333222

• Gilles mentioned this in T150746: Nginx timeouts on Thumbor.Jan 24 2017, 9:16 AM
• Gilles removed a project: Patch-For-Review.EditedJan 30 2017, 10:22 AM

@fgiunchedi please open read rights for me on the nginx access logs on thumbor100*

This way I can look for some kind of pattern in the requests around the time the iowait spikes begin.

• Gilles closed this task as Resolved.Mar 9 2017, 11:52 AM

I'm closing this, as the load spikes have considerably lowered and now look reasonable compared to the amount of cores. Memory consumption halved as well since we deployed the subprocess-based IM engine.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits