Change Details

03:25:57 <+icinga-wm> PROBLEM - LVS wdqs eqiad port 80/tcp - Wikidata Query Service IPv4 #page on wdqs.svc.eqiad.wmnet is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems 03:27:43 <+icinga-wm> RECOVERY - LVS wdqs eqiad port 80/tcp - Wikidata Query Service IPv4 #page on wdqs.svc.eqiad.wmnet is OK: HTTP OK: HTTP/1.1 200 OK - 448 bytes in 0.016 second response time https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems @rzl noticed that in all metrics, it appears wdqs1012 went to 0 or disappeared per https://grafana.wikimedia.org/d/000000489/wikidata-query-service?orgId=1&from=1629775199997&to=1629775911861&var-cluster_name=wdqs I found some exceptions related to the prometheus exporters, restarted both to no success (see P17064). @rzl has depooled wdqs1012 for now until it can be investigated by a WDQS expedrt since it does appear like something is wrong with it.

03:25:57 <+icinga-wm> PROBLEM - LVS wdqs eqiad port 80/tcp - Wikidata Query Service IPv4 #page on wdqs.svc.eqiad.wmnet is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems 03:27:43 <+icinga-wm> RECOVERY - LVS wdqs eqiad port 80/tcp - Wikidata Query Service IPv4 #page on wdqs.svc.eqiad.wmnet is OK: HTTP OK: HTTP/1.1 200 OK - 448 bytes in 0.016 second response time https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems @rlazarus noticed that in all metrics, it appears wdqs1012 went to 0 or disappeared per https://grafana.wikimedia.org/d/000000489/wikidata-query-service?orgId=1&from=1629775199997&to=1629775911861&var-cluster_name=wdqs I found some exceptions related to the prometheus exporters, restarted both to no success (see P17064). @rlazarus has depooled wdqs1012 for now until it can be investigated by a WDQS expedrt since it does appear like something is wrong with it.

03:25:57 <+icinga-wm> PROBLEM - LVS wdqs eqiad port 80/tcp - Wikidata Query Service IPv4 #page on wdqs.svc.eqiad.wmnet is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems 03:27:43 <+icinga-wm> RECOVERY - LVS wdqs eqiad port 80/tcp - Wikidata Query Service IPv4 #page on wdqs.svc.eqiad.wmnet is OK: HTTP OK: HTTP/1.1 200 OK - 448 bytes in 0.016 second response time https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems @rzl@rlazarus noticed that in all metrics, it appears wdqs1012 went to 0 or disappeared per https://grafana.wikimedia.org/d/000000489/wikidata-query-service?orgId=1&from=1629775199997&to=1629775911861&var-cluster_name=wdqs I found some exceptions related to the prometheus exporters, restarted both to no success (see P17064). @rzl@rlazarus has depooled wdqs1012 for now until it can be investigated by a WDQS expedrt since it does appear like something is wrong with it.