Page MenuHomePhabricator

Beta Cluster ("deployment-prep") logstash has gone silent since 2019-01-02Z01:14:14
Closed, DuplicatePublic

Description

15:36:28 <SMalyshev> anybody knows what access logstash-beta.wmflabs.org requires?
15:36:41 <SMalyshev> the password for regular logstash does not work
15:40:46 <James_F> SMalyshev: HTTP Auth permissions are on office wiki – see https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/Log_Files – but it's not showing anything today.
15:41:16 <SMalyshev> James_F: ahh, thanks!
15:42:21 <SMalyshev> James_F: yep, looks like completely barren and no file logging is happening as far as I can see...
15:43:16 <James_F> Most recent event is 2019-01-02Z01:14:14 FWICS.
15:49:49 <SMalyshev> James_F: ok, I found the logs on fluorine, so it's enough for me, I think I know what causes the problems there
15:50:10 <James_F> Oh, neat.
16:02:27 <+bd808> SMalyshev, James_F: logstash-beta being empty looks like a bug to me. Probably something to poke godog about related to his logging pipeline changes
16:03:02 <James_F> bd808: Ah, yeah, new logging changes happened last week, good point.
16:03:44 <+bd808> if you open the 30 day view you can see several stairstep drops in volume and then noting after the 2nd
16:03:51 <+bd808> *nothing

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 8 2019, 12:05 AM

AFAICT not immediately related to pipeline changes, though see also T211984. I've restarted logstash on deployment-logstash2 now though. To set expectations I don't think we'll be able to look into logstash beta "soon".

Ah, right, thanks!