Page MenuHomePhabricator
Create Task
Maniphest T116767

limit the impact of heavy/large graphite queries
Closed, ResolvedPublic
Actions

Description

as discovered in a recent graphite outage, heavy/large queries can occupy all uwsgi workers, resulting in 502s. we should seek how to limit the impact of such queries, ideally with timeouts at the graphite-web level

Details

ProjectBranchLines +/-Subject
operations/puppetproduction+32 -13graphite: uwsgi workers: set timeouts + max RSS
Customize query in gerrit

Related Objects

Event Timeline

fgiunchedi created this task.Oct 27 2015, 3:35 PM
fgiunchedi raised the priority of this task from to Medium.
fgiunchedi updated the task description. (Show Details)
fgiunchedi added projects: SRE, observability.
fgiunchedi added a subscriber: fgiunchedi.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 27 2015, 3:35 PM
fgiunchedi mentioned this in T155872: graphite1003 short of available RAM.Jan 27 2017, 10:24 AM

See also T155872: graphite1003 short of available RAM for a case where heavy queries were not impacting uwsgi but carbon-cache instead using a lot of memory.

fgiunchedi added a comment.Jan 27 2017, 10:32 AM

Updated wikitech Graphite troubleshooting on how to identify such queries.

fgiunchedi merged tasks: T155872: graphite1003 short of available RAM, T155876: Increased load on graphite1003, carbon-cache not autorestarting when killed by OOM.Jan 27 2017, 10:33 AM
fgiunchedi added subscribers: Volans, • ema, Stashbot, gerritbot.
fgiunchedi mentioned this in T155876: Increased load on graphite1003, carbon-cache not autorestarting when killed by OOM.Jan 27 2017, 10:36 AM
• Phabricator_maintenance moved this task from Backlog to Acknowledged on the SRE board.Jan 26 2019, 8:23 PM
gerritbot added a comment.Mar 5 2019, 11:20 PM

Change 494620 had a related patch set uploaded (by CDanis; owner: CDanis):
[operations/puppet@production] add uwsgi worker timeouts + max RSS for graphite

https://gerrit.wikimedia.org/r/494620

gerritbot added a project: Patch-For-Review.Mar 5 2019, 11:20 PM
gerritbot added a comment.Mar 6 2019, 2:44 PM

Change 494620 merged by CDanis:
[operations/puppet@production] graphite: uwsgi workers: set timeouts + max RSS

https://gerrit.wikimedia.org/r/494620

CDanis closed this task as Resolved.Mar 7 2019, 4:35 PM
CDanis claimed this task.
CDanis added a subscriber: CDanis.

Just saw the new timeout work -- query returned a 500 status after ~60 seconds. Boldly going to call this resolved; of course reopen if there's still more to be done.

Content licensed under Creative Commons Attribution-ShareAlike 4.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL