Page MenuHomePhabricator

rpcjob cirrusSearchLinksUpdatePrioritized is causing Call to a member function getNamespace() on a non-object (NULL)
Closed, ResolvedPublic

Description

I think since the latest deployment, we have an increasing number of this exception (600/hour):

{
  "_index": "logstash-2015.08.04",
  "_type": "mediawiki",
  "_id": "AU752OI2f2hURINuxq6I",
  "_score": null,
  "_source": {
    "message": "Call to a member function getNamespace() on a non-object (NULL)",
    "@version": 1,
    "@timestamp": "2015-08-04T17:53:37.182Z",
    "type": "mediawiki",
    "host": "mw1008",
    "level": "ERROR",
    "tags": [
      "syslog",
      "es",
      "es",
      "exception-json",
      "normalized_message_untrimmed"
    ],
    "channel": "exception",
    "url": "/rpc/RunJobs.php?wiki=eswiki&type=cirrusSearchLinksUpdatePrioritized&maxtime=30&maxmem=300M",
    "ip": "127.0.0.1",
    "http_method": "POST",
    "server": "127.0.0.1",
    "referrer": null,
    "uid": "53b5de8",
    "process_id": 14712,
    "wiki": "eswiki",
    "private": true,
    "file": "/srv/mediawiki/php-1.26wmf16/includes/jobqueue/Job.php",
    "line": 199,
    "code": 0,
    "backtrace": [
      {
        "file": "/srv/mediawiki/php-1.26wmf16/includes/jobqueue/JobQueueFederated.php",
        "line": 221,
        "function": "getDeduplicationInfo",
        "class": "Job",
        "type": "->",
        "args": []
      },
      {
        "file": "/srv/mediawiki/php-1.26wmf16/includes/jobqueue/JobQueueFederated.php",
        "line": 196,
        "function": "tryJobInsertions",
        "class": "JobQueueFederated",
        "type": "->",
        "args": [
          "array",
          "HashRing",
          "integer"
        ]
      },
      {
        "file": "/srv/mediawiki/php-1.26wmf16/includes/jobqueue/JobQueue.php",
        "line": 324,
        "function": "doBatchPush",
        "class": "JobQueueFederated",
        "type": "->",
        "args": [
          "array",
          "integer"
        ]
      },
      {
        "file": "/srv/mediawiki/php-1.26wmf16/includes/jobqueue/JobQueue.php",
        "line": 296,
        "function": "batchPush",
        "class": "JobQueue",
        "type": "->",
        "args": [
          "array",
          "integer"
        ]
      },
      {
        "file": "/srv/mediawiki/php-1.26wmf16/includes/jobqueue/JobQueueGroup.php",
        "line": 129,
        "function": "push",
        "class": "JobQueue",
        "type": "->",
        "args": [
          "array"
        ]
      },
      {
        "file": "/srv/mediawiki/php-1.26wmf16/extensions/CirrusSearch/includes/Job/ElasticaWrite.php",
        "line": 64,
        "function": "push",
        "class": "JobQueueGroup",
        "type": "->",
        "args": [
          "array"
        ]
      },
      {
        "file": "/srv/mediawiki/php-1.26wmf16/extensions/CirrusSearch/includes/Job/Job.php",
        "line": 57,
        "function": "doJob",
        "class": "CirrusSearch\\Job\\ElasticaWrite",
        "type": "->",
        "args": []
      },
      {
        "file": "/srv/mediawiki/php-1.26wmf16/extensions/CirrusSearch/includes/Updater.php",
        "line": 243,
        "function": "run",
        "class": "CirrusSearch\\Job\\Job",
        "type": "->",
        "args": []
      },
      {
        "file": "/srv/mediawiki/php-1.26wmf16/extensions/CirrusSearch/includes/Updater.php",
        "line": 83,
        "function": "deletePages",
        "class": "CirrusSearch\\Updater",
        "type": "->",
        "args": [
          "array",
          "array",
          "integer"
        ]
      },
      {
        "file": "/srv/mediawiki/php-1.26wmf16/extensions/CirrusSearch/includes/Job/LinksUpdate.php",
        "line": 49,
        "function": "updateFromTitle",
        "class": "CirrusSearch\\Updater",
        "type": "->",
        "args": [
          "Title"
        ]
      },
      {
        "file": "/srv/mediawiki/php-1.26wmf16/extensions/CirrusSearch/includes/Job/Job.php",
        "line": 57,
        "function": "doJob",
        "class": "CirrusSearch\\Job\\LinksUpdate",
        "type": "->",
        "args": []
      },
      {
        "file": "/srv/mediawiki/php-1.26wmf16/includes/jobqueue/JobRunner.php",
        "line": 183,
        "function": "run",
        "class": "CirrusSearch\\Job\\Job",
        "type": "->",
        "args": []
      },
      {
        "file": "/srv/mediawiki/rpc/RunJobs.php",
        "line": 42,
        "function": "run",
        "class": "JobRunner",
        "type": "->",
        "args": [
          "array"
        ]
      }
    ],
    "exception_id": "f142f4be",
    "class": "mediawiki",
    "message_checksum": "8a717dd6209a3239ad5a8efe906a7e11",
    "normalized_message": "Call to a member function getNamespace() on a non-object (NULL)"
  },
  "sort": [
    1438710817182
  ]
}

This is on all wikis but only on jobs requests related to CirrusSearch.

Event Timeline

jcrespo created this task.Aug 4 2015, 6:02 PM
jcrespo updated the task description. (Show Details)
jcrespo raised the priority of this task from to Needs Triage.
jcrespo added a subscriber: jcrespo.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 4 2015, 6:02 PM
Legoktm triaged this task as High priority.Aug 4 2015, 6:03 PM
Legoktm set Security to None.
Restricted Application added a project: Discovery. · View Herald TranscriptAug 4 2015, 6:03 PM

Looks like this happens because updateFromTitle() calls deletePages() with empty array, but deletePages() calls Job\ElasticaWrite() with Title argument of reset($titles) which when applied to empty array produces null. So title for the job is null, which of course breaks when getDeduplicationInfo() tries to use that title.

Change 259629 had a related patch set uploaded (by Smalyshev):
Use fake title if no real ones were supplied

https://gerrit.wikimedia.org/r/259629

Change 259629 merged by jenkins-bot:
Use fake title if no real ones were supplied

https://gerrit.wikimedia.org/r/259629

Smalyshev closed this task as Resolved.Dec 17 2015, 1:50 AM
Smalyshev claimed this task.