Page MenuHomePhabricator

Direct Parsoid/PHP scandium logs to a different channel
Closed, ResolvedPublic

Description

Currently, all Parsoid/PHP fatal errors clog the production wiki fatal monitor channel. Given that Parsoid/PHP will run a lot and at least intially have a lot of errors, we should probably direct these logs to a different channel.

Need to look up what config to tweak where, and could possible go into tests/RTTestSettings.php once we figure that out.

Event Timeline

ssastry created this task.Wed, Sep 4, 8:35 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptWed, Sep 4, 8:35 PM
ssastry triaged this task as High priority.Wed, Sep 4, 8:35 PM
ssastry moved this task from Backlog to Testing / QA on the Parsoid-PHP board.
ssastry claimed this task.Fri, Sep 6, 8:01 PM

Change 534889 had a related patch set uploaded (by Subramanya Sastry; owner: Subramanya Sastry):
[operations/mediawiki-config@master] Direct Parsoid/PHP rt-testing log events to a different target

https://gerrit.wikimedia.org/r/534889

Change 534889 merged by jenkins-bot:
[operations/mediawiki-config@master] Direct Parsoid/PHP rt-testing log events to a different target

https://gerrit.wikimedia.org/r/534889

Mentioned in SAL (#wikimedia-operations) [2019-09-10T17:58:17Z] <jforrester@deploy1001> Synchronized wmf-config/logging.php: T232042 Direct Parsoid/PHP rt-testing log events to a different target (duration: 01m 02s)

ssastry closed this task as Resolved.Tue, Sep 10, 5:59 PM

Will verify when we start a new test run.

ssastry reopened this task as Open.Wed, Sep 11, 6:00 PM
ssastry added subscribers: fgiunchedi, hashar.

So, it looks like these logs disappeared from the "mediawiki" type, but they did not make an appearance under the "parsoid-tests" type in logstash (I searched in Kibana). Based on IRC conversations on MediaWiki-General, I learnt that there is puppet code involved in directing the log events from mediawiki to logstash. So, it looks like we'll need some puppet updates to these filters to ensure these events end up in logstash / kibana.

@fgiunchedi Something you can help us with?

ssastry added a comment.EditedWed, Sep 11, 8:05 PM

Looks like adding this code at the end of modules/profile/files/logstash/filter-mediawiki.conf in the puppet repo might do the trick if we revert my mediawiki-config change patch?

# Scandium is a Parsoid test server and treat errors here as
# parsoid-tests errors and don't clog the main mediawiki channel
if [host] == "scandium" {
  mutate {
    replace => [ "type", "parsoid-tests" ]
  }
}

Looks like adding this code at the end of modules/profile/files/logstash/filter-mediawiki.conf in the puppet repo might do the trick if we revert my mediawiki-config change patch?

Yes sth like that should work, I'm not in love with hardcoding hostnames but overall it is best to have it in puppet than mediawiki-config when we'll replace scandium, so +1 from me!

Change 536208 had a related patch set uploaded (by Subramanya Sastry; owner: Subramanya Sastry):
[operations/puppet@production] Take #2: Redirect Parsoid/PHP rt-testing log events to "parsoid-tests"

https://gerrit.wikimedia.org/r/536208

Change 536208 merged by Giuseppe Lavagetto:
[operations/puppet@production] Take #2: Redirect Parsoid/PHP rt-testing log events to "parsoid-tests"

https://gerrit.wikimedia.org/r/536208

ssastry closed this task as Resolved.Fri, Sep 13, 2:18 PM

Joe deployed the revert and the puppet patch and I confirmed that this now directs the logs to 'parsoid-tests' type in logstash!