Page MenuHomePhabricator
Create Task
Maniphest T119721

500 errors from graphite shouldn't be retried by varnish
Closed, ResolvedPublic
Actions

Description

see also https://wikitech.wikimedia.org/wiki/Incident_documentation/20151126-graphite-grafana

Event Timeline

fgiunchedi created this task.Nov 26 2015, 3:48 PM
fgiunchedi claimed this task.
fgiunchedi raised the priority of this task from to Medium.
fgiunchedi updated the task description. (Show Details)
fgiunchedi added projects: SRE, Grafana.
fgiunchedi added subscribers: Aklapper, fgiunchedi.
fgiunchedi added a comment.Nov 27 2015, 3:58 PM

retry sample from latest outage logs

graphite.wikimedia.org:80 10.64.0.107 - - [25/Nov/2015:14:49:00 +0000] "GET /render?format=json&from=-1hour&target=sumSeries%28carbon.agents.graphite1001-%2A.creates%29 HTTP/1.1" 502 379 "-" "check_graphite/1.0"
graphite.wikimedia.org:80 10.64.0.107 - - [25/Nov/2015:14:49:00 +0000] "GET /render?format=json&from=-1hour&target=sumSeries%28carbon.agents.graphite1001-%2A.creates%29 HTTP/1.1" 502 379 "-" "check_graphite/1.0"
graphite.wikimedia.org:80 10.64.0.107 - - [25/Nov/2015:14:49:00 +0000] "GET /render?format=json&from=-1hour&target=sumSeries%28carbon.agents.graphite1001-%2A.creates%29 HTTP/1.1" 502 379 "-" "check_graphite/1.0"
graphite.wikimedia.org:80 10.64.0.107 - - [25/Nov/2015:14:49:00 +0000] "GET /render?format=json&from=-1hour&target=sumSeries%28carbon.agents.graphite1001-%2A.creates%29 HTTP/1.1" 502 379 "-" "check_graphite/1.0"
graphite.wikimedia.org:80 10.64.0.107 - - [25/Nov/2015:14:49:00 +0000] "GET /render?format=json&from=-1hour&target=sumSeries%28carbon.agents.graphite1001-%2A.creates%29 HTTP/1.1" 502 379 "-" "check_graphite/1.0"
graphite.wikimedia.org:80 10.64.0.107 - - [25/Nov/2015:14:49:00 +0000] "GET /render?format=json&from=-1hour&target=sumSeries%28carbon.agents.graphite1001-%2A.creates%29 HTTP/1.1" 502 379 "-" "check_graphite/1.0"
graphite.wikimedia.org:80 10.64.0.107 - - [25/Nov/2015:14:49:00 +0000] "GET /render?format=json&from=-1hour&target=sumSeries%28carbon.agents.graphite1001-%2A.creates%29 HTTP/1.1" 502 379 "-" "check_graphite/1.0"
graphite.wikimedia.org:80 10.64.0.107 - - [25/Nov/2015:14:49:00 +0000] "GET /render?format=json&from=-1hour&target=sumSeries%28carbon.agents.graphite1001-%2A.creates%29 HTTP/1.1" 502 379 "-" "check_graphite/1.0"
graphite.wikimedia.org:80 10.64.0.107 - - [25/Nov/2015:14:49:00 +0000] "GET /render?format=json&from=-1hour&target=sumSeries%28carbon.agents.graphite1001-%2A.creates%29 HTTP/1.1" 502 379 "-" "check_graphite/1.0"
graphite.wikimedia.org:80 10.64.0.107 - - [25/Nov/2015:14:49:00 +0000] "GET /render?format=json&from=-1hour&target=sumSeries%28carbon.agents.graphite1001-%2A.creates%29 HTTP/1.1" 502 379 "-" "check_graphite/1.0"
graphite.wikimedia.org:80 10.64.0.107 - - [25/Nov/2015:14:49:00 +0000] "GET /render?format=json&from=-1hour&target=sumSeries%28carbon.agents.graphite1001-%2A.creates%29 HTTP/1.1" 502 379 "-" "check_graphite/1.0"
graphite.wikimedia.org:80 10.64.0.107 - - [25/Nov/2015:14:49:00 +0000] "GET /render?format=json&from=-1hour&target=sumSeries%28carbon.agents.graphite1001-%2A.creates%29 HTTP/1.1" 502 379 "-" "check_graphite/1.0"
graphite.wikimedia.org:80 10.64.0.107 - - [25/Nov/2015:14:49:00 +0000] "GET /render?format=json&from=-1hour&target=sumSeries%28carbon.agents.graphite1001-%2A.creates%29 HTTP/1.1" 502 379 "-" "check_graphite/1.0"
graphite.wikimedia.org:80 10.64.0.107 - - [25/Nov/2015:14:49:00 +0000] "GET /render?format=json&from=-1hour&target=sumSeries%28carbon.agents.graphite1001-%2A.creates%29 HTTP/1.1" 502 379 "-" "check_graphite/1.0"
graphite.wikimedia.org:80 10.64.0.107 - - [25/Nov/2015:14:49:00 +0000] "GET /render?format=json&from=-1hour&target=sumSeries%28carbon.agents.graphite1001-%2A.creates%29 HTTP/1.1" 502 379 "-" "check_graphite/1.0"
graphite.wikimedia.org:80 10.64.0.107 - - [25/Nov/2015:14:49:00 +0000] "GET /render?format=json&from=-1hour&target=sumSeries%28carbon.agents.graphite1001-%2A.creates%29 HTTP/1.1" 502 379 "-" "check_graphite/1.0"
fgiunchedi added a comment.Nov 27 2015, 4:20 PM

see also https://gerrit.wikimedia.org/r/#/c/255706/

fgiunchedi closed this task as Resolved.Nov 30 2015, 9:17 AM

fixed by https://gerrit.wikimedia.org/r/#/c/255706/

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL