Page MenuHomePhabricator
Paste P8748

Logstash is down (T228089)
ActivePublic

Authored by Urbanecm on Jul 15 2019, 6:28 PM.
[19:00:54] <Daimona> Hello, is it already known that channels aren't working anymore on logstash since 12:30? See e.g. https://logstash.wikimedia.org/app/kibana#/discover?_g=(refreshInterval:(display:Off,pause:!f,value:0),time:(from:now-24h,mode:quick,to:now))&_a=(columns:!(_source),index:'logstash-*',interval:auto,query:(query_string:(query:'channel:StashEdit')),sort:!('@timestamp',desc))
[19:01:31] <wikibugs> (^C03PS2^O) ^C10^BRobH^O: puppetmaster1003 production dns entries [dns] - ^C10https://gerrit.wikimedia.org/r/523231^O (https://phabricator.wikimedia.org/T201342)
[19:01:54] <wikibugs> (^C03CR^O) ^C10^Bjerkins-bot^O: [V: ^C04-1^O] puppetmaster1003 production dns entries [dns] - ^C10https://gerrit.wikimedia.org/r/523231^O (https://phabricator.wikimedia.org/T201342) (owner:
^C10^BRobH^O)
[19:02:15] <Daimona> And it seems not to be about channels only
[19:02:32] <wikibugs> (^C03PS3^O) ^C10^BCwhite^O: prometheus: wire up prometheus-varnishkafka-exporter for deploy [puppet] - ^C10https://gerrit.wikimedia.org/r/522556^O (https://phabricator.wikimedia.org/T196066
)
[19:03:17] <icinga-wm> PROBLEM - Host mr1-eqsin.oob IPv6 is DOWN: PING CRITICAL - Packet loss = 100%
[19:03:30] <robh> hrmm
[19:03:31] <wikibugs> (^C03PS3^O) ^C10^BRobH^O: puppetmaster1003 production dns entries [dns] - ^C10https://gerrit.wikimedia.org/r/523231^O (https://phabricator.wikimedia.org/T201342)
[19:03:54] <jynus> robh: see eqsin mails
[19:03:57] *** Joins: yurik__ (~yurik@mediawiki/Yurik)
[19:04:06] <robh> ahh
[19:04:25] <robh> jynus: thx!
[19:04:26] <jynus> it is still not great, but someone is working on it
[19:04:35] <robh> its oob so as long as its known its fine
[19:04:38] <robh> =]
[19:04:46] <robh> i just didnt realize that was what that email thread was for
[19:05:00] <jynus> I asked too, to be fair
[19:05:03] <Daimona> https://logstash.wikimedia.org/app/kibana#/discover?_g=(refreshInterval:(display:Off,pause:!f,value:0),time:(from:now-24h,mode:quick,to:now))&_a=(columns:!(_source),index:'logstash-*',interv
al:auto,query:(query_string:(query:'type:mediawiki')),sort:!('@timestamp',desc)) wtf?
[19:05:04] <wikibugs> (^C03CR^O) ^C10^BRobH^O: [C: ^C03^B+2^O] puppetmaster1003 production dns entries [dns] - ^C10https://gerrit.wikimedia.org/r/523231^O (https://phabricator.wikimedia.org/T201342) (owner: ^C10
^BRobH^O)
[19:05:41] <Urbanecm> Daimona, please use the short url feature, if you can :)
[19:06:02] <Urbanecm> at least I have to copy the url to access it
[19:06:10] <Daimona> Urbanecm: Huh, thanks, I've never noticed that little link right there
[19:06:11] <wikibugs> ^C10Operations^O: rack/setup/install puppetmaster1003.eqiad.wmnet - https://phabricator.wikimedia.org/T201342 (^C10RobH^O)
[19:06:19] <Urbanecm> yw Daimona