List of steps to reproduce (step by step, including full links if applicable):
What happens?:
Returns 502 Bad Gateway error.
(confirmed in Toolforge and PAWS, and my local computer via 2620:0:863:ed1a::1)
amsa@C382:~$ curl --resolve query.wikidata.org:443 "https://query.wikidata.org/sparql?query=prefix%20schema:%20%3Chttp://schema.org/%3E%20SELECT%20*%20WHERE%20%7B%3Chttp://www.wikidata.org%3E%20schema:dateModified%20?y%7D&nocache=27178056" <?xml version='1.0' encoding='UTF-8'?> <sparql xmlns='http://www.w3.org/2005/sparql-results#'> <head> <variable name='y'/> </head> <results> <result> <binding name='y'> <literal datatype='http://www.w3.org/2001/XMLSchema#dateTime'>2021-09-03T15:50:38Z</literal> </binding> </result> </results> </sparql>
I can confirm via simply viewing https://query.wikidata.org/sparql?query=prefix%20schema:%20%3Chttp://schema.org/%3E%20SELECT%20*%20WHERE%20%7B%3Chttp://www.wikidata.org%3E%20schema:dateModified%20?y%7D&nocache=27178056 on my computer, where query.wikidata.org resolves to 2620:0:863:ed1a::1
Note there are no problem to just access to https://query.wikidata.org/, it only return 502 when a query is executed.
We are experiencing overload issues on the WDQS cluster in codfw. We suspect some specific queries are taking blazegraph down, but we haven't been able to identify (or block) those yet. The best workaround at the moment seems to be to regularly restart blazegraph (every hour).
See @Zbyszko's email to the wikidata mailing list: https://lists.wikimedia.org/hyperkitty/list/wikidata@lists.wikimedia.org/thread/SO7JKEGEO4AT7EDUYGYTVRGFHRLU6SB2/
Mentioned in SAL (#wikimedia-operations) [2021-09-03T16:10:13Z] <gehel> blazegraph (public cofdfw cluster) will now restart every hour - T290330
Change 717494 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):
[operations/puppet@production] wdqs: temp mitigation => restart hourly w random
Change 717494 merged by Ryan Kemper:
[operations/puppet@production] wdqs: temp mitigation => restart hourly w random
Change 717508 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):
[operations/puppet@production] wdqs: temp mitigation => restart hourly w random
Change 717508 merged by Ryan Kemper:
[operations/puppet@production] wdqs: temp mitigation => restart hourly w random
Mentioned in SAL (#wikimedia-operations) [2021-09-03T17:17:55Z] <ryankemper> T290330 Deployed https://gerrit.wikimedia.org/r/c/operations/puppet/+/717508 across wdqs fleet; codfw wdqs hosts will restart on average once per hour now to address ongoing availability issues for wdqs codfw
Mentioned in SAL (#wikimedia-operations) [2021-09-03T19:04:43Z] <ryankemper> T290330 ryankemper@cumin1001:~$ sudo -E cumin 'P{wdqs2*}' 'sudo rm -fv /etc/cron.hourly/restart-blazegraph' (Cleaned up manually created crons now that we have [somewhat hacky] systemd timers doing the same job)
Change 720102 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):
[operations/puppet@production] wdqs: remove codfw hourly restarts
Change 720102 merged by Ryan Kemper:
[operations/puppet@production] wdqs: remove codfw hourly restarts
Mentioned in SAL (#wikimedia-operations) [2021-09-17T01:47:59Z] <ryankemper> T290330 [Remove WDQS codfw ~hourly restarts] sudo cumin 'C:query_service::crontasks' 'sudo disable-puppet "Stop doing wdqs codfw ~hourly restarts - T290330"'
Mentioned in SAL (#wikimedia-operations) [2021-09-17T01:55:10Z] <ryankemper> T290330 [Remove WDQS codfw ~hourly restarts] Testing on arbitrary codfw host: ryankemper@wdqs2001:~$ sudo run-puppet-agent
Mentioned in SAL (#wikimedia-operations) [2021-09-17T02:22:00Z] <ryankemper> T290330 [Remove WDQS codfw ~hourly restarts] wdqs2001 and wdqs2004 look fine after running sudo systemctl reset-failed wdqs-restart-hourly-w-random-delay.timer to clean up dangling timer
Mentioned in SAL (#wikimedia-operations) [2021-09-17T02:28:39Z] <ryankemper> T290330 [Remove WDQS codfw ~hourly restarts] Successfully rolled out to rest of fleet sudo cumin 'C:query_service::crontasks' 'sudo run-puppet-agent --force && sudo systemctl reset-failed wdqs-restart-hourly-w-random-delay.timer'
We're no longer doing hourly restarts; we should monitor availability over the weekend to make sure that we're capable of handling current load without the crutch of the hourly restarts. We still have the useragent we banned in the last full outage (dailymotion) banned, so I'd think we'll be okay on load, but I'll be checking Grafana over the next couple days.
We can/should unban dailymotion early next week - perhaps tuesday - if wdqs has been stable over the weekend and during monday's load
Change 803336 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):
[operations/puppet@production] query service: clean up absented resources
Change 803336 merged by Ryan Kemper:
[operations/puppet@production] query service: clean up absented resources