logspam from mediawiki - LoadBalancer::{closure}: found writes/callbacks pending.
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Description

This appears to be a new error since I deployed 1.28.0-wmf.23 to group 2.

Raw log json:

{
  "_index": "logstash-2016.10.27",
  "_type": "mediawiki",
  "_id": "AVgHoAgojKHLXmxCbu3m",
  "_score": null,
  "_source": {
    "message": "LoadBalancer::{closure}: found writes/callbacks pending.",
    "@version": 1,
    "@timestamp": "2016-10-27T19:31:17.000Z",
    "type": "mediawiki",
    "host": "mw1167",
    "level": "ERROR",
    "tags": [
      "syslog",
      "es",
      "es"
    ],
    "channel": "DBQuery",
    "normalized_message": "LoadBalancer::{closure}: found writes/callbacks pending.",
    "url": "/rpc/RunJobs.php?wiki=cawiki&type=refreshLinks&maxtime=30&maxmem=300M",
    "ip": "127.0.0.1",
    "http_method": "POST",
    "server": "127.0.0.1",
    "referrer": null,
    "wiki": "cawiki",
    "mwversion": "1.28.0-wmf.23",
    "reqId": "0bbe9732ac71d1d1d75f25ba"
  }

Details

	Subject	Repo	Branch	Lines +/-
	Downgrade 'found writes/callbacks pending.' from error to info	mediawiki/core	master	+1 -1
	Downgrade 'found writes/callbacks pending.' from error to info	mediawiki/core	wmf/1.30.0-wmf.7	+1 -1

Customize query in gerrit

Related Objects

Mentioned Here: rMW148faa9f54aa: Downgrade 'found writes/callbacks pending.' from error to info

Event Timeline

• mmodell created this task.Oct 27 2016, 7:36 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 27 2016, 7:36 PM

• mmodell added a parent task: T147517: MW-1.28.0-wmf.23 deployment blockers.Oct 27 2016, 7:37 PM

Ok after giving it more time the spike in error frequency was just an anomaly around the time of deployment. It went back down to fairly low levels now. So this is apparently not directly related to wmf.23

• mmodell removed a parent task: T147517: MW-1.28.0-wmf.23 deployment blockers.Oct 27 2016, 7:43 PM

While this might be useful for debugging and catching issues with db transactions, maybe it should be downgraded to a warning in production to avoid spamming the error logs? This kind of thing is just a distraction during deployments when we need to be able to make quick judgements about the state of the branch. Some of the only signals we have are fatalmonitor and mediawiki-errors in logstash, so spam in the logs is a fairly big problem.

While this might be useful for debugging and catching issues with db transactions

Can someone exaplain what this is? If it is what I think it says, this should be an INFO or DEBUG, not ERROR. It spams other errors on the dbquery log, so +1 on mmodell comment.

• demon moved this task from Untriaged to Dec2019/1.35.wmf.10+ on the Wikimedia-production-error board.Jan 10 2017, 10:55 PM

Reception123 subscribed.Jun 23 2017, 4:22 PM

Change 361481 had a related patch set uploaded (by 20after4; owner: 20after4):
[mediawiki/core@master] Downgrade 'found writes/callbacks pending.' from error to info

https://gerrit.wikimedia.org/r/361481

gerritbot added a project: Patch-For-Review.Jun 26 2017, 4:29 PM

@jcrespo: I've submitted the patch to do exactly that - rMW148faa9f54aa: Downgrade 'found writes/callbacks pending.' from error to info

• mmodell claimed this task.Jun 26 2017, 4:44 PM

Restricted Application added a project: Release-Engineering-Team (Kanban). · View Herald TranscriptJun 26 2017, 4:44 PM

• mmodell moved this task from Backlog to In-progress on the Release-Engineering-Team (Kanban) board.Jun 26 2017, 4:44 PM

Nikerabbit subscribed.Jun 27 2017, 5:23 PM

Change 361481 merged by jenkins-bot:
[mediawiki/core@master] Downgrade 'found writes/callbacks pending.' from error to info

https://gerrit.wikimedia.org/r/361481

Change 361733 had a related patch set uploaded (by 20after4; owner: 20after4):
[mediawiki/core@wmf/1.30.0-wmf.7] Downgrade 'found writes/callbacks pending.' from error to info

https://gerrit.wikimedia.org/r/361733

• mmodell closed this task as Resolved.Jun 27 2017, 7:52 PM

ReleaseTaggerBot added a project: MW-1.30-release-notes (WMF-deploy-2017-07-11_(1.30.0-wmf.9)).Jun 27 2017, 8:00 PM

Change 361733 merged by jenkins-bot:
[mediawiki/core@wmf/1.30.0-wmf.7] Downgrade 'found writes/callbacks pending.' from error to info

https://gerrit.wikimedia.org/r/361733

ReleaseTaggerBot edited projects, added MW-1.30-release-notes (WMF-deploy-2017-06-27_(1.30.0-wmf.7)); removed MW-1.30-release-notes (WMF-deploy-2017-07-11_(1.30.0-wmf.9)).Jun 27 2017, 9:00 PM

• Phabricator_maintenance edited projects, added RelEng-Archive-FY201718-Q1; removed Release-Engineering-Team (Kanban).Sep 26 2017, 11:46 PM

Reedy removed a project: Patch-For-Review.May 17 2019, 9:47 PM

• mmodell changed the subtype of this task from "Task" to "Production Error".Aug 28 2019, 11:11 PM

logspam from mediawiki - LoadBalancer::{closure}: found writes/callbacks pending.Closed, ResolvedPublicPRODUCTION ERRORActions

Description

Details

Related Objects

Event Timeline

logspam from mediawiki - LoadBalancer::{closure}: found writes/callbacks pending.
Closed, ResolvedPublicPRODUCTION ERROR
Actions