Page MenuHomePhabricator
Create Task
Maniphest T350786

No entries at all in beta-logs.wmcloud.org since 2023-11-06 Z 12:15:39
Closed, ResolvedPublic
Actions

Description

Noticed whilst debugging T350777. Usual traffic is ~ 130k per three hours.

Details

SubjectRepoBranchLines +/-
logstash: beta-logs to use current w3creportingapi templateoperations/puppetproduction+1 -1
Customize query in gerrit

Related Objects

Event Timeline

Jdforrester-WMF created this task.Nov 8 2023, 2:55 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 8 2023, 2:55 PM
Nikerabbit mentioned this in T351131: Creating a Wikistory on English betacluster results in a 502 error.Nov 14 2023, 10:27 AM
TheresNoTime mentioned this in T351193: RESTbase broken on beta cluster.Nov 14 2023, 11:34 AM
kostajh subscribed.Nov 14 2023, 4:36 PM
kostajh added a comment.Nov 16 2023, 1:12 PM

In the past, restarting Logstash (T274593: Logstash beta is not getting any events) got things working again.

kostajh triaged this task as High priority.Nov 16 2023, 1:16 PM
kostajh added a project: Release-Engineering-Team.

I don't know if this should be considered a train blocking task (tagging Release-Engineering-Team for triaging) but not having visibility into MediaWiki errors in beta cluster would increase the likelihood of us finding out about issues only in group0 deployment, if not later.

jbond subscribed.Nov 16 2023, 1:32 PM

i have logged into logging-logstash-02.logging.eqiad1.wikimedia.cloud and ran systemctl restart logstash.service hopefully that has fixed this
Add Comment

kostajh added a comment.Nov 16 2023, 1:42 PM

@jbond discovered that Logstash is in a crash loop, see

P53517 (An Untitled Masterwork)
1Nov 16 13:39:03 logging-logstash-02 systemd[1]: Started logstash.
2Nov 16 13:39:03 logging-logstash-02 logstash[2669694]: Using LS_JAVA_HOME defined java: /usr/lib/jvm/java-11-openjdk-amd64.
3Nov 16 13:39:03 logging-logstash-02 logstash[2669694]: WARNING: Using LS_JAVA_HOME while Logstash distribution comes with a bundled JDK.
4Nov 16 13:39:12 logging-logstash-02 logstash[2669694]: Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
5Nov 16 13:39:18 logging-logstash-02 logstash[2669694]: warning: thread "Converge PipelineAction::Create<main>" terminated with exception (report_on_exception is true):
6Nov 16 13:39:18 logging-logstash-02 logstash[2669694]: LogStash::Error: Don't know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create<main>`
7Nov 16 13:39:18 logging-logstash-02 logstash[2669694]: create at org/logstash/execution/ConvergeResultExt.java:135
8Nov 16 13:39:18 logging-logstash-02 logstash[2669694]: add at org/logstash/execution/ConvergeResultExt.java:60
9Nov 16 13:39:18 logging-logstash-02 logstash[2669694]: converge_state at /usr/share/logstash/logstash-core/lib/logstash/agent.rb:396
10Nov 16 13:39:18 logging-logstash-02 systemd[1]: logstash.service: Main process exited, code=exited, status=1/FAILURE
11Nov 16 13:39:18 logging-logstash-02 systemd[1]: logstash.service: Failed with result 'exit-code'.
12Nov 16 13:39:18 logging-logstash-02 systemd[1]: logstash.service: Consumed 51.043s CPU time.

gerritbot added a comment.Nov 16 2023, 3:23 PM

Change 974635 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: beta-logs to use current w3creportingapi template

https://gerrit.wikimedia.org/r/974635

gerritbot added a project: Patch-For-Review.Nov 16 2023, 3:23 PM
gerritbot added a comment.Nov 16 2023, 3:26 PM

Change 974635 merged by Cwhite:

[operations/puppet@production] logstash: beta-logs to use current w3creportingapi template

https://gerrit.wikimedia.org/r/974635

Maintenance_bot removed a project: Patch-For-Review.Nov 16 2023, 3:30 PM
colewhite closed this task as Resolved.Nov 16 2023, 3:31 PM
colewhite claimed this task.
colewhite added projects: Observability-Logging, SRE Observability (FY2023/2024-Q2).
colewhite added subscribers: herron, colewhite.

Logstash was crashlooping because it was attempting to load a template that did not exist on the host anymore. Now that it is using the right template, logs are flowing again.

Jdforrester-WMF added a comment.Nov 16 2023, 4:03 PM
In T350786#9337868, @colewhite wrote:

Logstash was crashlooping because it was attempting to load a template that did not exist on the host anymore. Now that it is using the right template, logs are flowing again.

Thank you!

kostajh added a comment.Nov 16 2023, 4:11 PM

@colewhite thank you! Would you mind adding notes about how you found and fixed the issue, either here or in https://wikitech.wikimedia.org/wiki/Logs#Beta_cluster?

kostajh reopened this task as Open.Nov 17 2023, 4:29 PM
kostajh added a project: Quality-and-Test-Engineering-Team.

I am still not seeing MediaWiki application events in beta-logs.wmcloud.org. Maybe that is worth tracking separately; but for now I am re-opening this one. I'm also tagging Quality-and-Test-Engineering-Team, seems relevant to them.

colewhite closed this task as Resolved.Nov 17 2023, 4:45 PM

Definitely a different problem.

Restarted rsyslog on deployment-mediawiki11 and I saw some logs come through. Going back through syslog, it seems the kafka output plugin crashed and rsyslog never recovered.

It makes me suspicious about the state of rsyslog on the rest of the deployment hosts. Went ahead and restarted rsyslog on all of deployment prep to see if that would restore things.

Optimistically resolving, but please let us know if you find anything we missed!

kostajh added a comment.Nov 17 2023, 5:00 PM
In T350786#9341449, @colewhite wrote:

Definitely a different problem.

Restarted rsyslog on deployment-mediawiki11 and I saw some logs come through. Going back through syslog, it seems the kafka output plugin crashed and rsyslog never recovered.

It makes me suspicious about the state of rsyslog on the rest of the deployment hosts. Went ahead and restarted rsyslog on all of deployment prep to see if that would restore things.

Optimistically resolving, but please let us know if you find anything we missed!

@colewhite thanks, looks good at the moment

image.png (1×2 px, 267 KB)

lmata moved this task from Inbox to Done on the SRE Observability (FY2023/2024-Q2) board.Jan 26 2024, 1:08 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL