TL;DR: This was a reoccurrence of T198279 because the fix for it had not been backported to 1.32.0-wmf.10 when mcrouter-writes were enabled for all wikis.
Details at https://wikitech.wikimedia.org/wiki/Incident_documentation/20180710-MediaWiki.
Description
Description
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Joe | T192370 Deploy mcrouter to production as a wancache backend | |||
| Resolved | aaron | T198239 Rollout use of mcrouter for MediaWiki in production | |||
| Resolved | Krinkle | T199237 Unable to submit edit due to Exception "Key contains invalid characters" from MemcachedBagOStuff |
Event Timeline
Comment Actions
One user says it happen while he was creating a new article, preview works, but not saving the article.
Reported articles no:Rydöbruk and no:Måker.
Comment Actions
Setting this as "high" as it might not be obvious to users that they can log out and back in to solve the problem. It might also block users from saving on-going edits.
Comment Actions
The error message was reported as the new article after publishing (save), that is as the new article text (not in any dialog box or similar).
Comment Actions
An other user reports that he checked in both wikitext (old wikitoolbar?) and VisualEditor. The first one reported the error message where the article should be, while he can't remember how VisualøEditor reported the error.
Comment Actions
Could it be related to T199207?
Like an error on Wikipedia is related to an error in Phabricator?
Comment Actions
Details from the private mw-log/exception log:
2018-07-10 11:54:16 [W0SeaApAAEAAAGGOaZoAAAAF] mw1269 nowiki 1.32.0-wmf.10 exception ERROR: [W0SeaApAAEAAAGGOaZoAAAAF] /w/index.php?title=***&action=submit
Exception from line 156 of /srv/mediawiki/php-1.32.0-wmf.10/includes/libs/objectcache/MemcachedBagOStuff.php:
Key contains invalid characters: nowiki:blacklist:spam:pass:***
{ *** }
[Exception Exception] (/srv/mediawiki/php-1.32.0-wmf.10/includes/libs/objectcache/MemcachedBagOStuff.php:156) Key contains invalid characters: nowiki:blacklist:spam:pass:***
#0 /srv/mediawiki/php-1.32.0-wmf.10/includes/libs/objectcache/MemcachedPeclBagOStuff.php(154): MemcachedBagOStuff->validateKeyEncoding(string)
#1 /srv/mediawiki/php-1.32.0-wmf.10/includes/libs/objectcache/MemcachedBagOStuff.php(56): MemcachedPeclBagOStuff->getWithToken(string, NULL, integer)
#2 /srv/mediawiki/php-1.32.0-wmf.10/includes/libs/objectcache/BagOStuff.php(197): MemcachedBagOStuff->doGet(string, integer)
#3 /srv/mediawiki/php-1.32.0-wmf.10/includes/libs/objectcache/ReplicatedBagOStuff.php(80): BagOStuff->get(string, integer)
#4 /srv/mediawiki/php-1.32.0-wmf.10/includes/libs/objectcache/BagOStuff.php(197): ReplicatedBagOStuff->doGet(string, integer)
#5 /srv/mediawiki/php-1.32.0-wmf.10/extensions/SpamBlacklist/includes/SpamBlacklist.php(78): BagOStuff->get(string)
#6 /srv/mediawiki/php-1.32.0-wmf.10/extensions/SpamBlacklist/includes/SpamBlacklistHooks.php(42): SpamBlacklist->filter(array, Title)
#7 /srv/mediawiki/php-1.32.0-wmf.10/includes/Hooks.php(174): SpamBlacklistHooks::filterMergedContent(RequestContext, WikitextContent, Status, string, User, boolean)
#8 /srv/mediawiki/php-1.32.0-wmf.10/includes/Hooks.php(202): Hooks::callHook(string, array, array, NULL)
#9 /srv/mediawiki/php-1.32.0-wmf.10/includes/EditPage.php(1702): Hooks::run(string, array)
#10 /srv/mediawiki/php-1.32.0-wmf.10/includes/EditPage.php(2146): EditPage->runPostMergeFilters(WikitextContent, Status, User)
#11 /srv/mediawiki/php-1.32.0-wmf.10/includes/EditPage.php(1529): EditPage->internalAttemptSave(NULL, boolean)
#12 /srv/mediawiki/php-1.32.0-wmf.10/includes/EditPage.php(662): EditPage->attemptSave(NULL)
#13 /srv/mediawiki/php-1.32.0-wmf.10/includes/actions/EditAction.php(60): EditPage->edit()
#14 /srv/mediawiki/php-1.32.0-wmf.10/includes/actions/SubmitAction.php(38): EditAction->show()
#15 /srv/mediawiki/php-1.32.0-wmf.10/includes/MediaWiki.php(500): SubmitAction->show()
#16 /srv/mediawiki/php-1.32.0-wmf.10/includes/MediaWiki.php(294): MediaWiki->performAction(Article, Title)
#17 /srv/mediawiki/php-1.32.0-wmf.10/includes/MediaWiki.php(867): MediaWiki->performRequest()
#18 /srv/mediawiki/php-1.32.0-wmf.10/includes/MediaWiki.php(524): MediaWiki->main()
#19 /srv/mediawiki/php-1.32.0-wmf.10/index.php(42): MediaWiki->run()
#20 /srv/mediawiki/w/index.php(3): include(string)
#21 {main}Comment Actions
Nutshell:
This is the same issue we observed on Beta Cluster last week (see T198279 and T198280 for details). This was fixed in master and in wmf.12, but was not back-ported to wmf.10. This meant that when mcrouter was activated on all wikis earlier today (see T198239 and https://gerrit.wikimedia.org/r/444932), it caused two problems:
- Certain Memcached add() operations to consistently fail. This mainly affected process locks and certain merge() calls from WANObjectCache.
- Any Memcached keys containing special characters (instead of using page IDs or hashed titles) no longer worked and would throw a fatal error.
Both of these were fixed within about 10-15min of it being noticed. Aaron or myself will follow-up tomorrow with a proper incident report and actionables.