Page MenuHomePhabricator
Create Task
Maniphest T209429

memcached error: A BAD KEY WAS PROVIDED/CHARACTERS OUT OF RANGE
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Description

Spotted some errors in logstash, easily found via https://logstash.wikimedia.org/app/kibana#/dashboard/memcached

Memcached error for key "{memcached-key}" on server "{memcached-server}":
A BAD KEY WAS PROVIDED/CHARACTERS OUT OF RANGE

That seems related to 1.33.0-wmf.4 (T206658). Some related to jobrunner.discovery.wmnet / /rpc/RunSingleJob.php , but I had at least one from an api call.

The memcached key seems to be an empty string which would explain the error. I could not find stacktraces in logstash :/

Details

SubjectRepoBranchLines +/-
JobQueue: Actually return the value from getRootJobCacheKey()mediawiki/corewmf/1.33.0-wmf.4+1 -1
JobQueue: Actually return the value from getRootJobCacheKey()mediawiki/coremaster+1 -1
Add trace to debug memcached bad key errormediawiki/corewmf/1.33.0-wmf.4+4 -0
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedReleasehasharT206658 1.33.0-wmf.4 deployment blockers
 ResolvedPRODUCTION ERRORAnomieT209429 memcached error: A BAD KEY WAS PROVIDED/CHARACTERS OUT OF RANGE

Event Timeline

hashar created this task.Nov 13 2018, 10:56 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 13 2018, 10:56 PM
hashar updated the task description. (Show Details)Nov 13 2018, 10:58 PM
hashar updated the task description. (Show Details)Nov 13 2018, 11:00 PM
hashar added a parent task: T206658: 1.33.0-wmf.4 deployment blockers.
hashar mentioned this in T206658: 1.33.0-wmf.4 deployment blockers.Nov 13 2018, 11:02 PM
hashar added a comment.Nov 13 2018, 11:05 PM

Does not seem to harmful, I will attempt to find the root cause tomorrow.

xSavitar subscribed.Nov 14 2018, 2:33 PM
elukey subscribed.Nov 14 2018, 2:36 PM
zeljkofilipin triaged this task as Unbreak Now! priority.Nov 14 2018, 3:36 PM
zeljkofilipin subscribed.

Train blockers are UBN.

Restricted Application added subscribers: Liuxinyu970226, TerraCodes. · View Herald TranscriptNov 14 2018, 3:36 PM
Anomie subscribed.Nov 14 2018, 4:05 PM
This comment was removed by Anomie.
gerritbot subscribed.Nov 14 2018, 4:05 PM

Change 473551 had a related patch set uploaded (by Anomie; owner: Anomie):
[mediawiki/core@wmf/1.33.0-wmf.4] Add trace to debug memcached bad key error

https://gerrit.wikimedia.org/r/473551

gerritbot added a project: Patch-For-Review.Nov 14 2018, 4:05 PM
Lucas_Werkmeister_WMDE subscribed.Nov 14 2018, 4:07 PM
Nikerabbit subscribed.Nov 14 2018, 5:27 PM
gerritbot added a comment.Nov 14 2018, 5:55 PM

Change 473551 merged by jenkins-bot:
[mediawiki/core@wmf/1.33.0-wmf.4] Add trace to debug memcached bad key error

https://gerrit.wikimedia.org/r/473551

Stashbot subscribed.Nov 14 2018, 5:58 PM

Mentioned in SAL (#wikimedia-operations) [2018-11-14T17:58:57Z] <hashar@deploy1001> Started scap: php-1.33.0-wmf.4/includes/libs/objectcache/MemcachedPeclBagOStuff.php Add trace to debug memcached bad key error - T209429

ReleaseTaggerBot added a project: MW-1.33-notes (1.33.0-wmf.4; 2018-11-13).Nov 14 2018, 6:00 PM
Stashbot added a comment.Nov 14 2018, 6:33 PM

Mentioned in SAL (#wikimedia-operations) [2018-11-14T18:33:04Z] <hashar@deploy1001> Finished scap: php-1.33.0-wmf.4/includes/libs/objectcache/MemcachedPeclBagOStuff.php Add trace to debug memcached bad key error - T209429 (duration: 34m 07s)

Anomie edited projects, added MediaWiki-Core-JobQueue; removed MW-1.33-notes (1.33.0-wmf.4; 2018-11-13), Patch-For-Review.Nov 14 2018, 6:34 PM

Looks like it's probably something to do with rMW51945dbca359: Use DB domain in JobQueueGroup and make WikiMap domain ID methods stricter. I'm continuing to investigate.

A representative trace:

2018-11-14 18:31:06 [W@xp6QpAAEAAAIGhifMAAACC] mw1269 mediawikiwiki 1.33.0-wmf.4 memcached ERROR: Memcached error for key "" on server ":": A BAD KEY WAS PROVIDED/CHARACTERS OUT OF RANGE {"memcached-server":":","memcached-key":null} 
[Exception Exception] (/srv/mediawiki/php-1.33.0-wmf.4/includes/libs/objectcache/MemcachedPeclBagOStuff.php:234) A BAD KEY WAS PROVIDED/CHARACTERS OUT OF RANGE
  #0 /srv/mediawiki/php-1.33.0-wmf.4/includes/libs/objectcache/MemcachedPeclBagOStuff.php(167): MemcachedPeclBagOStuff->checkResult(NULL, boolean)
  #1 /srv/mediawiki/php-1.33.0-wmf.4/includes/jobqueue/JobQueue.php(491): MemcachedPeclBagOStuff->set(NULL, string, integer)
  #2 /srv/mediawiki/php-1.33.0-wmf.4/includes/jobqueue/JobQueue.php(464): JobQueue->doDeduplicateRootJob(HTMLCacheUpdateJob)
  #3 /srv/mediawiki/php-1.33.0-wmf.4/includes/jobqueue/JobQueue.php(345): JobQueue->deduplicateRootJob(HTMLCacheUpdateJob)
  #4 /srv/mediawiki/php-1.33.0-wmf.4/includes/jobqueue/JobQueue.php(310): JobQueue->batchPush(array, integer)
  #5 /srv/mediawiki/php-1.33.0-wmf.4/includes/jobqueue/JobQueueGroup.php(158): JobQueue->push(array)
  #6 /srv/mediawiki/php-1.33.0-wmf.4/includes/deferred/JobQueueEnqueueUpdate.php(57): JobQueueGroup->push(array)
  #7 /srv/mediawiki/php-1.33.0-wmf.4/includes/deferred/DeferredUpdates.php(270): JobQueueEnqueueUpdate->doUpdate()
  #8 /srv/mediawiki/php-1.33.0-wmf.4/includes/deferred/DeferredUpdates.php(216): DeferredUpdates::runUpdate(JobQueueEnqueueUpdate, Wikimedia\Rdbms\LBFactoryMulti, string, integer)
  #9 /srv/mediawiki/php-1.33.0-wmf.4/includes/deferred/DeferredUpdates.php(140): DeferredUpdates::execute(array, string, integer)
  #10 /srv/mediawiki/php-1.33.0-wmf.4/includes/MediaWiki.php(905): DeferredUpdates::doUpdates(string)
  #11 /srv/mediawiki/php-1.33.0-wmf.4/includes/MediaWiki.php(728): MediaWiki->restInPeace(string, boolean)
  #12 [internal function]: Closure$MediaWiki::doPostOutputShutdown()
  #13 {main}

All the traces so far seem to go through JobQueue::deduplicateRootJob().

hashar added a comment.Nov 14 2018, 6:37 PM

I ran a full sync for deployment, hence the delay :(

We have traces now, an example:

#0 /srv/mediawiki/php-1.33.0-wmf.4/includes/libs/objectcache/MemcachedPeclBagOStuff.php(156): MemcachedPeclBagOStuff->checkResult(NULL, boolean)
#1 /srv/mediawiki/php-1.33.0-wmf.4/includes/libs/objectcache/MemcachedBagOStuff.php(56): MemcachedPeclBagOStuff->getWithToken(NULL, NULL, integer)
#2 /srv/mediawiki/php-1.33.0-wmf.4/includes/libs/objectcache/BagOStuff.php(193): MemcachedBagOStuff->doGet(NULL, integer)
#3 /srv/mediawiki/php-1.33.0-wmf.4/includes/jobqueue/JobQueue.php(485): BagOStuff->get(NULL)
#4 /srv/mediawiki/php-1.33.0-wmf.4/includes/jobqueue/JobQueue.php(464): JobQueue->doDeduplicateRootJob(RefreshLinksJob)
#5 /srv/mediawiki/php-1.33.0-wmf.4/includes/jobqueue/JobQueue.php(345): JobQueue->deduplicateRootJob(RefreshLinksJob)
#6 /srv/mediawiki/php-1.33.0-wmf.4/includes/jobqueue/JobQueue.php(310): JobQueue->batchPush(array, integer)
#7 /srv/mediawiki/php-1.33.0-wmf.4/includes/jobqueue/JobQueueGroup.php(158): JobQueue->push(array)
#8 /srv/mediawiki/php-1.33.0-wmf.4/includes/deferred/LinksUpdate.php(363): JobQueueGroup->push(array)
#9 /srv/mediawiki/php-1.33.0-wmf.4/includes/deferred/LinksUpdate.php(316): LinksUpdate::queueRecursiveJobsForTable(Title, string, string, string)
#10 /srv/mediawiki/php-1.33.0-wmf.4/includes/deferred/LinksUpdate.php(299): LinksUpdate->queueRecursiveJobs()
#11 /srv/mediawiki/php-1.33.0-wmf.4/includes/deferred/LinksUpdate.php(175): LinksUpdate->doIncrementalUpdate()
#12 /srv/mediawiki/php-1.33.0-wmf.4/includes/deferred/DeferredUpdates.php(270): LinksUpdate->doUpdate()
#13 /srv/mediawiki/php-1.33.0-wmf.4/includes/deferred/DeferredUpdates.php(216): DeferredUpdates::runUpdate(LinksUpdate, Wikimedia\Rdbms\LBFactoryMulti, string, integer)
#14 /srv/mediawiki/php-1.33.0-wmf.4/includes/deferred/DeferredUpdates.php(140): DeferredUpdates::execute(array, string, integer)
#15 /srv/mediawiki/php-1.33.0-wmf.4/includes/MediaWiki.php(905): DeferredUpdates::doUpdates(string)
#16 /srv/mediawiki/php-1.33.0-wmf.4/includes/MediaWiki.php(728): MediaWiki->restInPeace(string, boolean)
#17 [internal function]: Closure$MediaWiki::doPostOutputShutdown()
#18 {main}
gerritbot added a comment.Nov 14 2018, 6:41 PM

Change 473578 had a related patch set uploaded (by Anomie; owner: Anomie):
[mediawiki/core@master] JobQueue: Actually return the value from getRootJobCacheKey()

https://gerrit.wikimedia.org/r/473578

gerritbot added a project: Patch-For-Review.Nov 14 2018, 6:41 PM

Change 473579 had a related patch set uploaded (by Hashar; owner: Anomie):
[mediawiki/core@wmf/1.33.0-wmf.4] JobQueue: Actually return the value from getRootJobCacheKey()

https://gerrit.wikimedia.org/r/473579

gerritbot added a comment.Nov 14 2018, 7:03 PM

Change 473578 merged by jenkins-bot:
[mediawiki/core@master] JobQueue: Actually return the value from getRootJobCacheKey()

https://gerrit.wikimedia.org/r/473578

gerritbot added a comment.Nov 14 2018, 7:09 PM

Change 473579 merged by jenkins-bot:
[mediawiki/core@wmf/1.33.0-wmf.4] JobQueue: Actually return the value from getRootJobCacheKey()

https://gerrit.wikimedia.org/r/473579

Stashbot added a comment.Nov 14 2018, 7:14 PM

Mentioned in SAL (#wikimedia-operations) [2018-11-14T19:14:23Z] <hashar@deploy1001> Synchronized php-1.33.0-wmf.4/includes/jobqueue/JobQueue.php: Actually return the value from getRootJobCacheKey() - T209429 (duration: 00m 53s)

hashar closed this task as Resolved.Nov 14 2018, 7:21 PM
hashar assigned this task to Anomie.

Thank you Brad. There is no more errors in logstash.

Sorry for the long time it took to add the stack trace to the log context, I ran a full scap instead of syncing just the file.

Anomie added a comment.Nov 14 2018, 7:38 PM
In T209429#4747803, @hashar wrote:

Sorry for the long time it took to add the stack trace to the log context, I ran a full scap instead of syncing just the file.

No problem, things happen sometimes.

ReleaseTaggerBot added a project: MW-1.33-notes (1.33.0-wmf.6; 2018-11-27).Nov 14 2018, 8:01 PM
Aklapper awarded a token.Nov 14 2018, 10:14 PM
Liuxinyu970226 awarded a token.Nov 19 2018, 1:28 PM
Liuxinyu970226 unsubscribed.
• mmodell changed the subtype of this task from "Task" to "Production Error".Aug 28 2019, 11:07 PM
Aklapper removed a subscriber: Anomie.Oct 16 2020, 5:38 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL