Page MenuHomePhabricator

periodic spike of MW exceptions "DB connection was already closed or the connection dropped."
Closed, ResolvedPublic

Description

icinga periodically reports high MW exceptions, these in particular seems to affect jobrunners mostly, on de / fr / no / commons wikis, last 24h logstash: https://logstash.wikimedia.org/goto/56ac3eabbb84953b3d92e69b3ca56f60

browsing through phabricator for a similar error message it looks like we've seen this before, e.g. T67263

{
  "file": "/srv/mediawiki/php-1.28.0-wmf.13/includes/db/DatabaseMysqli.php",
  "line": 307,
  "function": "getBindingHandle",
  "class": "DatabaseMysqlBase",
  "type": "->",
  "args": []
},
{
  "file": "/srv/mediawiki/php-1.28.0-wmf.13/includes/db/DatabaseMysqlBase.php",
  "line": 573,
  "function": "mysqlRealEscapeString",
  "class": "DatabaseMysqli",
  "type": "->",
  "args": [
    "string"
  ]
},
{
  "file": "/srv/mediawiki/php-1.28.0-wmf.13/includes/db/Database.php",
  "line": 1969,
  "function": "strencode",
  "class": "DatabaseMysqlBase",
  "type": "->",
  "args": [
    "string"
  ]
},
{
  "file": "/srv/mediawiki/php-1.28.0-wmf.13/includes/Title.php",
  "line": 4386,
  "function": "addQuotes",
  "class": "DatabaseBase",
  "type": "->",
  "args": [
    "string"
  ]
},
{
  "file": "/srv/mediawiki/php-1.28.0-wmf.13/includes/deferred/AutoCommitUpdate.php",
  "line": 40,
  "function": "Closure$Title::invalidateCache",
  "args": [
    "DatabaseMysqli",
    "string"
  ]
},
{
  "file": "/srv/mediawiki/php-1.28.0-wmf.13/includes/deferred/DeferredUpdates.php",
  "line": 163,
  "function": "doUpdate",
  "class": "AutoCommitUpdate",
  "type": "->",
  "args": []
},
{
  "file": "/srv/mediawiki/php-1.28.0-wmf.13/includes/deferred/DeferredUpdates.php",
  "line": 84,
  "function": "execute",
  "class": "DeferredUpdates",
  "type": "::",
  "args": [
    "array",
    "string"
  ]
},
{
  "file": "/srv/mediawiki/php-1.28.0-wmf.13/includes/MediaWiki.php",
  "line": 776,
  "function": "doUpdates",
  "class": "DeferredUpdates",
  "type": "::",
  "args": [
    "string"
  ]
},
{
  "file": "/srv/mediawiki/rpc/RunJobs.php",
  "line": 51,
  "function": "restInPeace",
  "class": "MediaWiki",
  "type": "->",
  "args": []
}

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 4 2016, 9:30 AM
aaron claimed this task.Aug 8 2016, 7:58 PM
aaron added a comment.Aug 8 2016, 9:53 PM

Only seems to happen from ChangeNotification jobs...

Change 303969 had a related patch set uploaded (by Aaron Schulz):
Make Database disconnect and error suppression more robust

https://gerrit.wikimedia.org/r/303969

aaron triaged this task as Normal priority.Aug 11 2016, 12:05 AM

Change 303969 merged by jenkins-bot:
Make Database disconnect and error suppression more robust

https://gerrit.wikimedia.org/r/303969

aaron closed this task as Resolved.Aug 11 2016, 5:12 PM

Closing for now. If this comes back next cycle it can be reopened.