Page MenuHomePhabricator

New logspam from ResourceLoader
Closed, ResolvedPublic

Description

ResourceLoader has gotten very spammy in the last couple days with the roll out of rMW976943c9913b: resourceloader: Bump severity of style queue violation to Warning. Where last week we are seeing in the low 10's of logs per 3 hours, in the last couple days i'm seeing 5M logs per 3 hours. Last week at this time the entire cluster only generated 1.5-2M logs per 5 hours, so this is a pretty significant jump.

They mostly seem to say the same thing, although with a different module each time: Unexpected general module "mobile.messageBox" in styles queue.

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 12 2017, 3:08 PM

Closing as invalid / duplicate.

The severity of this particular log message was intentionally raised 1.5 week ago as part of T92459. Just happens to (roughly) match the timeline of the Logstash upgrade.

None of these module queue violations have become more common either, it was just less visible before because we only logged it to debug (Jenkins/Beta/Vagrant/mwdebug) not to warning (production). As we're making progress with T92459 and fixed all the obvious issues, I decided to raise the severity to help weed out the remnant violations.

This is an incredible increase in the amount of log messages the logging subsystem has to handle, and is going to degrade performance for everyone using logstash.wikimedia.org. Could we at least sample the messages to 1 in 100 or something?

Change 353581 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[mediawiki/core@master] resourceloader: Temporarily lower styles queue warning severity

https://gerrit.wikimedia.org/r/353581

Krinkle reopened this task as Open.May 12 2017, 5:08 PM
Krinkle claimed this task.

Change 353582 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[mediawiki/core@wmf/1.30.0-wmf.1] resourceloader: Temporarily lower styles queue warning severity

https://gerrit.wikimedia.org/r/353582

Krinkle renamed this task from New logspam from Resource Loader to New logspam from ResourceLoader.May 12 2017, 5:10 PM
Krinkle triaged this task as High priority.
Krinkle updated the task description. (Show Details)
Krinkle added a project: Performance-Team.

Okay. Bringing down severity back to how it was before rMW976943c9913b: resourceloader: Bump severity of style queue violation to Warning.

Change 353582 merged by jenkins-bot:
[mediawiki/core@wmf/1.30.0-wmf.1] resourceloader: Temporarily lower styles queue warning severity

https://gerrit.wikimedia.org/r/353582

Krinkle closed this task as Resolved.May 12 2017, 9:43 PM

Change 353581 abandoned by Krinkle:
resourceloader: Temporarily lower styles queue warning severity

https://gerrit.wikimedia.org/r/353581