@bd808, how hard would it be to add alerting for this log bucket, like you did for memcached?

The memcached monitor is setup in role::graphite::production::alerts based on the data that Logstash adds to logstash.rate.mediawiki.memcached.ERROR.sum.

We have a logstash.rate.mediawiki.exception.ERROR.sum graphite metric for exceptions in general, but nothing more granular than that for this use case. To use the existing alerting tools we would need to either setup a new log channel for these on the MediaWiki side and start feeding it to Logstash or add a special rule to our Logstash config to count this subset of the exception channel traffic in graphite.

Adding counting to logstash.rate.mediawiki.exception-CAS.ERROR.sum would look something like this in ::role::logstash:

   logstash::output::statsd { 'MW_exception_CAS_rate':
       host            => $statsd_host,
       guard_condition => '[type] == "mediawiki" and [channel] == "exception" and [message] =~ "CAS"',
       namespace       => 'logstash.rate',
       sender          => 'mediawiki',
       increment       => [ 'exception-CAS.%{level}' ],
}

Per https://logstash.wikimedia.org/goto/ee0b4bf7c773c18981cae04d43a3e2d9 there is currently about 100 hits per day of exceptions with "CAS update failed"

A few examples:

• /wiki/Special:ConfirmEmail/** MWException from line 4045 of .../User.php: CAS update failed on user_touched for user ID '**' (read from replica);
• /w/api.php MWException from line 4045 of .../User.php: CAS update failed on user_touched for user ID '**' (read from master);
• /wiki/Special:Enroll/** MWException from line 4045 of .../User.php: CAS update failed on user_touched for user
• /wiki/Special:ContentTranslation?page=**&from=**&to=**&targettitle=&campaign=interlanguagelink MWException from line 4045 of
• /wiki/**?veaction=edit MWException from line 4045 of .../User.php: CAS update failed on user_touched for user ID '**' (read from replica);

What's the current status on this?

Looks like it's still an on-going issue: https://logstash.wikimedia.org/app/kibana#/dashboard/default?_g=(refreshInterval%3A(display%3AOff%2Cpause%3A!f%2Cvalue%3A0)%2Ctime%3A(from%3Anow-24h%2Cmode%3Aquick%2Cto%3Anow))

I'm well aware of that. I mean what's the status on somebody actually caring and doing something about it.