Page MenuHomePhabricator
Create Task
Maniphest T233739

MediaWiki log spam during row D blip / rack D2 unavailable
Open, Needs TriagePublic
Actions

Description

[Related but different than T233623: MediaWiki log spam during master unavailable / read only, this one is about spam during https://wikitech.wikimedia.org/wiki/Incident_documentation/20190920-d2_switch_failure]

We've observed a large amount of logs sent by MW during the incident above, namely a peak of ~130k msg/s (see below) with the biggest chuck being from memcache channel:

e.g. Memcached error for key "WANCache:t:ptwiki:messages:pt" on server "127.0.0.1:11213": SERVER ERROR (https://logstash.wikimedia.org/goto/8c177e65b10582aaf15a904b91977751)

while the rest was mostly dbconnection / dbreplication channels: https://logstash.wikimedia.org/goto/006c2fa6f31a1d6b7138e673cb077a20

There should be some logging rate limit (per mw channel + level?) so that similar situations are handled more gracefully

2019-09-24-183428_1075x1185_scrot.png (1×1 px, 192 KB)

Details

ProjectBranchLines +/-Subject
operations/puppetproduction+5 -1logstash: add id to drop action in filter-throttle-errors
operations/puppetproduction+46 -0logstash: throttle duplicate normalized_message with level:ERR*
Customize query in gerrit

Related Objects

Event Timeline

fgiunchedi created this task.Sep 24 2019, 4:40 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 24 2019, 4:40 PM
gerritbot added a comment.Sep 25 2019, 4:33 PM

Change 538931 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] logstash: throttle duplicate normalized_message with level:ERR*

https://gerrit.wikimedia.org/r/538931

gerritbot added a project: Patch-For-Review.Sep 25 2019, 4:33 PM
fgiunchedi added a comment.Sep 26 2019, 9:10 AM

Another type of spam that we have observed, php-generated but comes through by means of syslog + apache: https://logstash.wikimedia.org/goto/c5afb82f07a0249524d66b74c82e55c4

2019-09-24T11:35:36	mw1341	NOTICE	PHP Notice:  Undefined variable: wmgFileExporterBetaFeature in /srv/mediawiki/wmf-config/CommonSettings.php on line 3321
2019-09-24T11:35:35	mw1341	NOTICE	PHP Notice:  Undefined variable: wmgFileExporterBetaFeature in /srv/mediawiki/wmf-config/CommonSettings.php on line 3321
2019-09-24T11:35:35	mw1341	NOTICE	PHP Notice:  Undefined variable: wmgFileExporterBetaFeature in /srv/mediawiki/wmf-config/CommonSettings.php on line 3321
2019-09-24T11:35:35	mw1341	NOTICE	PHP Notice:  Undefined variable: wmgFileExporterBetaFeature in /srv/mediawiki/wmf-config/CommonSettings.php on line 3321
2019-09-24T11:35:35	mw1341	NOTICE	PHP Notice:  Undefined variable: wmgFileExporterBetaFeature in /srv/mediawiki/wmf-config/CommonSettings.php on line 3321
2019-09-24T11:35:35	mw1341	NOTICE	PHP Notice:  Undefined variable: wmgFileExporterBetaFeature in /srv/mediawiki/wmf-config/CommonSettings.php on line 3321
2019-09-24T11:35:35	mw1341	NOTICE	PHP Notice:  Undefined variable: wmgFileExporterBetaFeature in /srv/mediawiki/wmf-config/CommonSettings.php on line 3321
2019-09-24T11:35:35	mw1341	NOTICE	PHP Notice:  Undefined variable: wmgFileExporterBetaFeature in /srv/mediawiki/wmf-config/CommonSettings.php on line 3321
2019-09-24T11:35:35	mw1341	NOTICE	PHP Notice:  Undefined variable: wmgFileExporterBetaFeature in /srv/mediawiki/wmf-config/CommonSettings.php on line 3321
2019-09-24T11:35:35	mw1341	NOTICE	PHP Notice:  Undefined variable: wmgFileExporterBetaFeature in /srv/mediawiki/wmf-config/CommonSettings.php on line 3321
gerritbot added a comment.Oct 2 2019, 6:18 PM

Change 538931 merged by Herron:
[operations/puppet@production] logstash: throttle duplicate normalized_message with level:ERR*

https://gerrit.wikimedia.org/r/538931

gerritbot added a comment.Oct 2 2019, 7:08 PM

Change 540465 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] logstash: add id to drop action in filter-throttle-errors

https://gerrit.wikimedia.org/r/540465

gerritbot added a comment.Oct 2 2019, 7:15 PM

Change 540465 merged by Herron:
[operations/puppet@production] logstash: add id to drop action in filter-throttle-errors

https://gerrit.wikimedia.org/r/540465

Maintenance_bot removed a project: Patch-For-Review.Oct 2 2019, 8:10 PM
Krinkle moved this task from Active investigation to Follow-up prevention on the Wikimedia-Incident board.Dec 17 2019, 10:19 PM
Krinkle edited projects, added Sustainability (Incident Followup); removed Wikimedia-Incident.Apr 28 2020, 9:50 PM
fgiunchedi added a project: observability.Apr 20 2021, 8:43 AM
lmata moved this task from Inbox to Backlog on the observability board.Jun 9 2021, 4:23 AM
lmata edited projects, added SRE Observability; removed observability.Jul 12 2021, 2:21 AM
Maintenance_bot added a project: observability.Jul 12 2021, 2:46 AM
lmata moved this task from Inbox to Backlog on the SRE Observability board.Jul 15 2021, 4:09 AM
lmata edited projects, added Observability-Logging; removed SRE Observability.Aug 9 2021, 2:56 AM
Maintenance_bot edited projects, added SRE Observability; removed Observability-Logging.Aug 9 2021, 3:46 AM
lmata edited projects, added Observability-Logging; removed SRE Observability.Aug 9 2021, 3:48 AM
Maintenance_bot edited projects, added SRE Observability; removed Observability-Logging.Aug 9 2021, 4:45 AM
lmata edited projects, added Observability-Logging; removed SRE Observability.Jan 17 2022, 11:29 PM
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL