Page MenuHomePhabricator
Create Task
Maniphest T337818

apache2 cpu-stuck on logstash hosts causes kafka logging lag
Closed, ResolvedPublic
Actions

Assigned To
fgiunchedi
Authored By
fgiunchedi
May 31 2023, 8:08 AM
Tags
Referenced Files
F43269110: Screenshot 2024-03-24 at 18-31-03 Kafka Consumer Lag - Dashboards - Grafana.png
Mar 25 2024, 12:33 AM
F43267292: Screenshot 2024-03-24 at 18-07-05 Kafka Consumer Lag - Dashboards - Grafana.png
Mar 25 2024, 12:08 AM
F43267110: Screenshot 2024-03-24 at 18-03-00 Logstash - Dashboards - Grafana.png
Mar 25 2024, 12:08 AM
Subscribers
Aklapper
andrea.denisse
colewhite
fgiunchedi
herron

Description

LogstashKafkaConsumerLag alert was firing since ~6h this EU morning, upon investigation from https://grafana.wikimedia.org/d/000000561/logstash I noticed logstash1032 wasn't consuming messages; turns out apache2 process was stuck on CPU (on futexes) and there was no CPU left for logstash/elasticsearch. I wanted to capture a backtrace with symbols, unfortunately installing apache2-bin-dbgsym also restarted apache2, so the problem went away.

Filing a task for tracking purposes, however I don't think we run into this bug very often (i.e. will be hard to reproduce)

Details

SubjectRepoBranchLines +/-
opensearch: switch dashboards to sso authoperations/puppetproduction+1 -9
opensearch: set vhost and issuer url for dashboards sso testoperations/puppetproduction+4 -0
opensearch: move apache-auth-sso.erb to the right locationoperations/puppetproduction+0 -0
opensearch: use Sensitive[String] for sso secretsoperations/puppetproduction+2 -2
opensearch: fix sso supportoperations/puppetproduction+5 -1
hieradata: test sso for opensearch-dashboards in cloud vpsoperations/puppetproduction+3 -0
Use oauth2-proxy for opensearch dashboardsoperations/puppetproduction+36 -12
hieradata: add logstash_oidc clientoperations/puppetproduction+17 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

fgiunchedi created this task.May 31 2023, 8:08 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 31 2023, 8:08 AM
colewhite triaged this task as Low priority.May 31 2023, 3:26 PM
colewhite subscribed.

👀

colewhite moved this task from Inbox to Watching on the Observability-Logging board.Jun 15 2023, 9:12 PM
colewhite added a comment.EditedJan 13 2024, 10:25 PM

This occurred again today on logstash1030 at 21:44Z.

Stashbot added a comment.Jan 18 2024, 11:21 AM

Mentioned in SAL (#wikimedia-operations) [2024-01-18T11:21:02Z] <godog> bounce apache2 on logstash1025 / logstash1031 - T337818

fgiunchedi added a comment.Jan 18 2024, 11:22 AM

This was happening again on two logstash hosts in eqiad, interestingly enough I was able to capture the backtraces via gdb with debug symbols (in /root, I had to sanitize users and passwords so beware) and it didn't seem to cause any active impact since apache2 has been like that since multiple days AFAICS (!)

Stashbot added a comment.Mar 20 2024, 11:10 AM

Mentioned in SAL (#wikimedia-operations) [2024-03-20T11:10:56Z] <godog> bounce apache2 on logstash1031 - T337818

fgiunchedi renamed this task from apache2 cpu-stuck on logstash1032 causes kafka logging lag to apache2 cpu-stuck on logstash hosts causes kafka logging lag.Mar 20 2024, 11:14 AM
herron subscribed.Mar 20 2024, 1:54 PM
herron added a comment.Mar 20 2024, 2:13 PM

IMO it'd be worth considering a split of the collector role into independent dashboard, logstash and potentially logs-api roles to provide additional isolation between services.

andrea.denisse subscribed.Mar 24 2024, 11:50 PM
Stashbot added a comment.Mar 24 2024, 11:59 PM

Mentioned in SAL (#wikimedia-operations) [2024-03-24T23:59:14Z] <denisse> restarting apache2 on logstash1023 - T337818

andrea.denisse added a comment.EditedMar 25 2024, 12:08 AM

I think this issue triggered again today, I noticed that the affected host was logstash1023, I restarted the apache2 service.

Here are relevant graphs at the time of the incident:

Screenshot 2024-03-24 at 18-03-00 Logstash - Dashboards - Grafana.png (258×907 px, 39 KB)

I can see a drop in the Logstash input rate for logstash1023 that correlates with an increase in 'tripped in_flight_requests' errors on logstash1023.
https://grafana.wikimedia.org/goto/cHYbV9JIz

Screenshot 2024-03-24 at 18-07-05 Kafka Consumer Lag - Dashboards - Grafana.png (530×1 px, 123 KB)

https://grafana.wikimedia.org/goto/KD04S9JSz

andrea.denisse added a comment.Mar 25 2024, 12:33 AM

After restarting apache2 on logstash1023 I notice that the consumer group lag graph shows that the number of events is decreasing, I think the issue may be contained but we still need a proper fix for this issue.

Screenshot 2024-03-24 at 18-31-03 Kafka Consumer Lag - Dashboards - Grafana.png (537×1 px, 110 KB)

https://grafana.wikimedia.org/goto/z9u1N9JIz

fgiunchedi raised the priority of this task from Low to Medium.Mar 27 2024, 2:28 PM
fgiunchedi added a project: SRE Observability (FY2023/2024-Q4).
gerritbot added a comment.Tue, Apr 2, 9:12 AM

Change #1015045 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] Use oauth2-proxy for opensearch dashboards

https://gerrit.wikimedia.org/r/1015045

gerritbot added a project: Patch-For-Review.Tue, Apr 2, 9:12 AM

Change #1016301 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] hieradata: add logstash_oidc client

https://gerrit.wikimedia.org/r/1016301

colewhite moved this task from Inbox to In progress on the SRE Observability (FY2023/2024-Q4) board.Wed, Apr 3, 11:10 PM
gerritbot added a comment.Thu, Apr 4, 8:39 AM

Change #1016301 merged by Filippo Giunchedi:

[operations/puppet@production] hieradata: add logstash_oidc client

https://gerrit.wikimedia.org/r/1016301

colewhite added a parent task: T246998: Enable SSO for Kibana.Tue, Apr 9, 9:09 PM
colewhite mentioned this in T246998: Enable SSO for Kibana.
gerritbot added a comment.Wed, Apr 10, 8:53 AM

Change #1015045 merged by Filippo Giunchedi:

[operations/puppet@production] Use oauth2-proxy for opensearch dashboards

https://gerrit.wikimedia.org/r/1015045

gerritbot added a comment.Wed, Apr 10, 9:07 AM

Change #1018647 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] hieradata: test sso for opensearch-dashboards in cloud vps

https://gerrit.wikimedia.org/r/1018647

gerritbot added a comment.Wed, Apr 10, 9:57 AM

Change #1018647 merged by Filippo Giunchedi:

[operations/puppet@production] hieradata: test sso for opensearch-dashboards in cloud vps

https://gerrit.wikimedia.org/r/1018647

gerritbot added a comment.Wed, Apr 10, 10:02 AM

Change #1018654 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] opensearch: fix sso support

https://gerrit.wikimedia.org/r/1018654

gerritbot added a comment.Wed, Apr 10, 10:06 AM

Change #1018654 merged by Filippo Giunchedi:

[operations/puppet@production] opensearch: fix sso support

https://gerrit.wikimedia.org/r/1018654

gerritbot added a comment.Wed, Apr 10, 10:14 AM

Change #1018657 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] opensearch: use Sensitive[String] for sso secrets

https://gerrit.wikimedia.org/r/1018657

gerritbot added a comment.Wed, Apr 10, 10:14 AM

Change #1018657 merged by Filippo Giunchedi:

[operations/puppet@production] opensearch: use Sensitive[String] for sso secrets

https://gerrit.wikimedia.org/r/1018657

gerritbot added a comment.Wed, Apr 10, 10:22 AM

Change #1018659 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] opensearch: move apache-auth-sso.erb to the right location

https://gerrit.wikimedia.org/r/1018659

gerritbot added a comment.Wed, Apr 10, 10:22 AM

Change #1018659 merged by Filippo Giunchedi:

[operations/puppet@production] opensearch: move apache-auth-sso.erb to the right location

https://gerrit.wikimedia.org/r/1018659

gerritbot added a comment.Wed, Apr 10, 10:38 AM

Change #1018667 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] opensearch: set vhost and issuer url for dashboards sso test

https://gerrit.wikimedia.org/r/1018667

gerritbot added a comment.Wed, Apr 10, 12:27 PM

Change #1018667 merged by Filippo Giunchedi:

[operations/puppet@production] opensearch: set vhost and issuer url for dashboards sso test

https://gerrit.wikimedia.org/r/1018667

gerritbot added a comment.Thu, Apr 11, 7:13 AM

Change #1018872 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] opensearch: switch dashboards to sso auth

https://gerrit.wikimedia.org/r/1018872

gerritbot added a comment.Mon, Apr 15, 8:35 AM

Change #1018872 merged by Filippo Giunchedi:

[operations/puppet@production] opensearch: switch dashboards to sso auth

https://gerrit.wikimedia.org/r/1018872

fgiunchedi closed this task as Resolved.Mon, Apr 15, 8:55 AM
fgiunchedi claimed this task.

I'm optimistically resolving this since we no longer use mod auth ldap

Maintenance_bot removed a project: Patch-For-Review.Fri, Apr 26, 7:03 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL