Page MenuHomePhabricator

Unable to submit edit due to Exception "Key contains invalid characters" from MemcachedBagOStuff
Closed, ResolvedPublic

Description

TL;DR: This was a reoccurrence of T198279 because the fix for it had not been backported to 1.32.0-wmf.10 when mcrouter-writes were enabled for all wikis.

Details at https://wikitech.wikimedia.org/wiki/Incident_documentation/20180710-MediaWiki.

Reported at no:Wikipedia:Torget#Feilmelding under redigering and no:Wikipedia:Administratorenes oppslagstavle#Intern feil.

The reports does not contain much information except it reads [W0SeaApAAEAAAGGOaZoAAAAF] 2018-07-10 11:54:16: Uhåndterlig unntak av typen «Exception», in English that should be some thing like "Unsolvable exception of type 'Exception'". It is not clear which editing interface the reporting users use.

Quick fix seems to be to log out and log back in.

Could it be related to T199207?

Event Timeline

jeblad created this task.Jul 10 2018, 3:45 PM
Restricted Application added subscribers: Danmichaelo, Aklapper. · View Herald TranscriptJul 10 2018, 3:45 PM

One user says it happen while he was creating a new article, preview works, but not saving the article.

Reported articles no:Rydöbruk and no:Måker.

jeblad renamed this task from rror during editing to Error during editing.Jul 10 2018, 3:50 PM
jeblad updated the task description. (Show Details)
jeblad updated the task description. (Show Details)
jeblad triaged this task as High priority.Jul 10 2018, 3:55 PM

Setting this as "high" as it might not be obvious to users that they can log out and back in to solve the problem. It might also block users from saving on-going edits.

jeblad added a comment.EditedJul 10 2018, 4:11 PM

The error message was reported as the new article after publishing (save), that is as the new article text (not in any dialog box or similar).

An other user reports that he checked in both wikitext (old wikitoolbar?) and VisualEditor. The first one reported the error message where the article should be, while he can't remember how VisualøEditor reported the error.

Could it be related to T199207?

Like an error on Wikipedia is related to an error in Phabricator?

Could it be related to T199207?

Like an error on Wikipedia is related to an error in Phabricator?

My bad.

jeblad updated the task description. (Show Details)Jul 10 2018, 6:24 PM
jeblad updated the task description. (Show Details)Jul 10 2018, 6:27 PM
jeblad updated the task description. (Show Details)
Aklapper renamed this task from Error during editing to Exception during editing.Jul 10 2018, 8:08 PM

Details from the private mw-log/exception log:

2018-07-10 11:54:16 [W0SeaApAAEAAAGGOaZoAAAAF] mw1269 nowiki 1.32.0-wmf.10 exception ERROR: [W0SeaApAAEAAAGGOaZoAAAAF] /w/index.php?title=***&action=submit

Exception from line 156 of /srv/mediawiki/php-1.32.0-wmf.10/includes/libs/objectcache/MemcachedBagOStuff.php:
Key contains invalid characters: nowiki:blacklist:spam:pass:***

{ *** }

[Exception Exception] (/srv/mediawiki/php-1.32.0-wmf.10/includes/libs/objectcache/MemcachedBagOStuff.php:156) Key contains invalid characters: nowiki:blacklist:spam:pass:***
  #0 /srv/mediawiki/php-1.32.0-wmf.10/includes/libs/objectcache/MemcachedPeclBagOStuff.php(154): MemcachedBagOStuff->validateKeyEncoding(string)
  #1 /srv/mediawiki/php-1.32.0-wmf.10/includes/libs/objectcache/MemcachedBagOStuff.php(56): MemcachedPeclBagOStuff->getWithToken(string, NULL, integer)
  #2 /srv/mediawiki/php-1.32.0-wmf.10/includes/libs/objectcache/BagOStuff.php(197): MemcachedBagOStuff->doGet(string, integer)
  #3 /srv/mediawiki/php-1.32.0-wmf.10/includes/libs/objectcache/ReplicatedBagOStuff.php(80): BagOStuff->get(string, integer)
  #4 /srv/mediawiki/php-1.32.0-wmf.10/includes/libs/objectcache/BagOStuff.php(197): ReplicatedBagOStuff->doGet(string, integer)
  #5 /srv/mediawiki/php-1.32.0-wmf.10/extensions/SpamBlacklist/includes/SpamBlacklist.php(78): BagOStuff->get(string)
  #6 /srv/mediawiki/php-1.32.0-wmf.10/extensions/SpamBlacklist/includes/SpamBlacklistHooks.php(42): SpamBlacklist->filter(array, Title)
  #7 /srv/mediawiki/php-1.32.0-wmf.10/includes/Hooks.php(174): SpamBlacklistHooks::filterMergedContent(RequestContext, WikitextContent, Status, string, User, boolean)
  #8 /srv/mediawiki/php-1.32.0-wmf.10/includes/Hooks.php(202): Hooks::callHook(string, array, array, NULL)
  #9 /srv/mediawiki/php-1.32.0-wmf.10/includes/EditPage.php(1702): Hooks::run(string, array)
  #10 /srv/mediawiki/php-1.32.0-wmf.10/includes/EditPage.php(2146): EditPage->runPostMergeFilters(WikitextContent, Status, User)
  #11 /srv/mediawiki/php-1.32.0-wmf.10/includes/EditPage.php(1529): EditPage->internalAttemptSave(NULL, boolean)
  #12 /srv/mediawiki/php-1.32.0-wmf.10/includes/EditPage.php(662): EditPage->attemptSave(NULL)
  #13 /srv/mediawiki/php-1.32.0-wmf.10/includes/actions/EditAction.php(60): EditPage->edit()
  #14 /srv/mediawiki/php-1.32.0-wmf.10/includes/actions/SubmitAction.php(38): EditAction->show()
  #15 /srv/mediawiki/php-1.32.0-wmf.10/includes/MediaWiki.php(500): SubmitAction->show()
  #16 /srv/mediawiki/php-1.32.0-wmf.10/includes/MediaWiki.php(294): MediaWiki->performAction(Article, Title)
  #17 /srv/mediawiki/php-1.32.0-wmf.10/includes/MediaWiki.php(867): MediaWiki->performRequest()
  #18 /srv/mediawiki/php-1.32.0-wmf.10/includes/MediaWiki.php(524): MediaWiki->main()
  #19 /srv/mediawiki/php-1.32.0-wmf.10/index.php(42): MediaWiki->run()
  #20 /srv/mediawiki/w/index.php(3): include(string)
  #21 {main}
Krinkle renamed this task from Exception during editing to Unable to submit edit due to Exception "Key contains invalid characters" from MemcachedBagOStuff.Jul 11 2018, 3:39 AM
Krinkle added a project: Performance-Team.
Krinkle added a project: Wikimedia-Incident.

Nutshell:

This is the same issue we observed on Beta Cluster last week (see T198279 and T198280 for details). This was fixed in master and in wmf.12, but was not back-ported to wmf.10. This meant that when mcrouter was activated on all wikis earlier today (see T198239 and https://gerrit.wikimedia.org/r/444932), it caused two problems:

  • Certain Memcached add() operations to consistently fail. This mainly affected process locks and certain merge() calls from WANObjectCache.
  • Any Memcached keys containing special characters (instead of using page IDs or hashed titles) no longer worked and would throw a fatal error.

Both of these were fixed within about 10-15min of it being noticed. Aaron or myself will follow-up tomorrow with a proper incident report and actionables.

Krinkle updated the task description. (Show Details)Jul 12 2018, 2:44 AM
Krinkle closed this task as Resolved.
Krinkle claimed this task.