localhost:600...": (curl error: 28) Timeout was reached
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Description

Error

MediaWiki version: 1.36.0-wmf.10

message

Error fetching URL "http://localhost:6007/echoseen/v1/global%3Aecho%3Aseen%3Aalert%3Atime%3A59877388": (curl error: 28) Timeout was reached

Impact

Lots of short bursts of errors like the above in the mwdebug2001 logs, starting yesterday:

https://logstash.wikimedia.org/goto/8a0050058a844886a3e27727735f7b96

(Noticed while deploying something unrelated.)

All appear to be timeouts to various services via Envoy. Recent occurrences appear to be limited to mwdebug2001.

Notes

Nothing in /var/log/envoy/*.log which seems to correspond with the errors in logstash.

Details

Request ID: f2eaf5da-2950-4717-b99c-ef4a7431ed7f
Request URL: https://meta.wikimedia.org/w/index.php?title=MediaWiki:Wikiminiatlas.js&action=raw&ctype=text/javascript

Stack Trace

exception.trace

n/a

Event Timeline

dpifke created this task.Oct 14 2020, 10:19 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 14 2020, 10:19 PM

Seems to affect both echo and sessionstore requests to Envoy / Kask.

Krinkle moved this task from Untriaged to Oct2020/1.36.0-wmf.11+ on the Wikimedia-production-error board.Oct 14 2020, 10:59 PM

Clarakosi triaged this task as High priority.Oct 20 2020, 8:32 PM

Clarakosi edited projects, added Platform Team Workboards (Clinic Duty Team); removed Platform Engineering.

Clarakosi moved this task from Inbox to Later on the Platform Team Workboards (Clinic Duty Team) board.

No more of these in the log for the past 90 days. Maybe this was transient or already fixed under another task? Closing task, please reopen if this is seen again.

Error fetching URL "http://localhost:600...": (curl error: 28) Timeout was reachedClosed, ResolvedPublicPRODUCTION ERRORActions