Page MenuHomePhabricator
Create Task
Maniphest T233739

MediaWiki log spam during row D blip / rack D2 unavailable
Closed, DeclinedPublic
Actions

Description

[Related but different than T233623: MediaWiki log spam during master unavailable / read only, this one is about spam during https://wikitech.wikimedia.org/wiki/Incident_documentation/20190920-d2_switch_failure]

We've observed a large amount of logs sent by MW during the incident above, namely a peak of ~130k msg/s (see below) with the biggest chuck being from memcache channel:

e.g. Memcached error for key "WANCache:t:ptwiki:messages:pt" on server "127.0.0.1:11213": SERVER ERROR (https://logstash.wikimedia.org/goto/8c177e65b10582aaf15a904b91977751)

while the rest was mostly dbconnection / dbreplication channels: https://logstash.wikimedia.org/goto/006c2fa6f31a1d6b7138e673cb077a20

There should be some logging rate limit (per mw channel + level?) so that similar situations are handled more gracefully

2019-09-24-183428_1075x1185_scrot.png (1×1 px, 192 KB)

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
logstash: add id to drop action in filter-throttle-errorsoperations/puppetproduction+5 -1
logstash: throttle duplicate normalized_message with level:ERR*operations/puppetproduction+46 -0
Customize query in gerrit

Related Objects

Event Timeline

fgiunchedi created this task.Sep 24 2019, 4:40 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 24 2019, 4:40 PM
gerritbot added a comment.Sep 25 2019, 4:33 PM

Change 538931 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] logstash: throttle duplicate normalized_message with level:ERR*

https://gerrit.wikimedia.org/r/538931

gerritbot added a project: Patch-For-Review.Sep 25 2019, 4:33 PM
fgiunchedi added a comment.Sep 26 2019, 9:10 AM

Another type of spam that we have observed, php-generated but comes through by means of syslog + apache: https://logstash.wikimedia.org/goto/c5afb82f07a0249524d66b74c82e55c4

2019-09-24T11:35:36	mw1341	NOTICE	PHP Notice:  Undefined variable: wmgFileExporterBetaFeature in /srv/mediawiki/wmf-config/CommonSettings.php on line 3321
2019-09-24T11:35:35	mw1341	NOTICE	PHP Notice:  Undefined variable: wmgFileExporterBetaFeature in /srv/mediawiki/wmf-config/CommonSettings.php on line 3321
2019-09-24T11:35:35	mw1341	NOTICE	PHP Notice:  Undefined variable: wmgFileExporterBetaFeature in /srv/mediawiki/wmf-config/CommonSettings.php on line 3321
2019-09-24T11:35:35	mw1341	NOTICE	PHP Notice:  Undefined variable: wmgFileExporterBetaFeature in /srv/mediawiki/wmf-config/CommonSettings.php on line 3321
2019-09-24T11:35:35	mw1341	NOTICE	PHP Notice:  Undefined variable: wmgFileExporterBetaFeature in /srv/mediawiki/wmf-config/CommonSettings.php on line 3321
2019-09-24T11:35:35	mw1341	NOTICE	PHP Notice:  Undefined variable: wmgFileExporterBetaFeature in /srv/mediawiki/wmf-config/CommonSettings.php on line 3321
2019-09-24T11:35:35	mw1341	NOTICE	PHP Notice:  Undefined variable: wmgFileExporterBetaFeature in /srv/mediawiki/wmf-config/CommonSettings.php on line 3321
2019-09-24T11:35:35	mw1341	NOTICE	PHP Notice:  Undefined variable: wmgFileExporterBetaFeature in /srv/mediawiki/wmf-config/CommonSettings.php on line 3321
2019-09-24T11:35:35	mw1341	NOTICE	PHP Notice:  Undefined variable: wmgFileExporterBetaFeature in /srv/mediawiki/wmf-config/CommonSettings.php on line 3321
2019-09-24T11:35:35	mw1341	NOTICE	PHP Notice:  Undefined variable: wmgFileExporterBetaFeature in /srv/mediawiki/wmf-config/CommonSettings.php on line 3321
gerritbot added a comment.Oct 2 2019, 6:18 PM

Change 538931 merged by Herron:
[operations/puppet@production] logstash: throttle duplicate normalized_message with level:ERR*

https://gerrit.wikimedia.org/r/538931

gerritbot added a comment.Oct 2 2019, 7:08 PM

Change 540465 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] logstash: add id to drop action in filter-throttle-errors

https://gerrit.wikimedia.org/r/540465

gerritbot added a comment.Oct 2 2019, 7:15 PM

Change 540465 merged by Herron:
[operations/puppet@production] logstash: add id to drop action in filter-throttle-errors

https://gerrit.wikimedia.org/r/540465

Maintenance_bot removed a project: Patch-For-Review.Oct 2 2019, 8:10 PM
Krinkle moved this task from Active investigation to Follow-up prevention on the Wikimedia-Incident board.Dec 17 2019, 10:19 PM
Krinkle edited projects, added Sustainability (Incident Followup); removed Wikimedia-Incident.Apr 28 2020, 9:50 PM
fgiunchedi added a project: observability.Apr 20 2021, 8:43 AM
• lmata moved this task from Inbox to Backlog on the observability board.Jun 9 2021, 4:23 AM
• lmata edited projects, added SRE Observability; removed observability.Jul 12 2021, 2:21 AM
Maintenance_bot added a project: observability.Jul 12 2021, 2:46 AM
• lmata moved this task from Inbox to Backlog on the SRE Observability board.Jul 15 2021, 4:09 AM
• lmata edited projects, added Observability-Logging; removed SRE Observability.Aug 9 2021, 2:56 AM
Maintenance_bot edited projects, added SRE Observability; removed Observability-Logging.Aug 9 2021, 3:46 AM
• lmata edited projects, added Observability-Logging; removed SRE Observability.Aug 9 2021, 3:48 AM
Maintenance_bot edited projects, added SRE Observability; removed Observability-Logging.Aug 9 2021, 4:45 AM
• lmata edited projects, added Observability-Logging; removed SRE Observability.Jan 17 2022, 11:29 PM
Volans edited projects, added SRE Observability, SRE-Sprint-Week-Sustainability-March2023; removed Observability-Logging.Mar 21 2023, 11:23 AM
Volans moved this task from Backlog to E_TOO_BIG_MAYBE_OKR? on the SRE-Sprint-Week-Sustainability-March2023 board.
• lmata edited projects, added Observability-Logging; removed SRE Observability.Mar 28 2023, 10:22 PM
• lmata subscribed.

Adding back Observability-Logging which is a component tag within SRE Observability; when this gets selected, SRE Observability gets untagged. This keeps the intake tag less crowded and separate from the backlog and triaged flows.

colewhite closed this task as Declined.Jan 10 2025, 1:19 AM
colewhite subscribed.

Boldly declining in favor of focusing effort on T331879: Investigate methods to rate-limit/discard excessive log messages closer to the producer. Feel free to reopen if this is in error.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits