- Logging looks good on beta now in that we're getting plenty of report-only data there, according to beta-logs.wmcloud.org/mwlog01.deployment-prep.
- Over the past week (ending 3/17/2022) we're averaging about 6,671 filter trips per day. Which... is possibly a lot for the beta cluster? But we won't really have a great sense of the true false positive rate until we set to enforce and people start to complain.
- Just looking at the code again, there's probably some logspam/redundancy as this logging technically isn't guarded by the report-only global. And basically the same thing will now also get logged here.
- Let's merge this on a Monday?
Description
Description
Details
Details
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T273220 Deploy StopForumSpam extension to production | |||
| Resolved | sbassett | T304111 Set StopForumSpam to enforce on beta cluster |
Event Timeline
Comment Actions
Change 771696 had a related patch set uploaded (by SBassett; author: SBassett):
[operations/mediawiki-config@master] Set StopForumSpam to enforce on the beta cluster
Comment Actions
Change 771696 merged by jenkins-bot:
[operations/mediawiki-config@master] Set StopForumSpam to enforce on the beta cluster
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2022-03-21T15:56:35Z] <reedy@deploy1002> Synchronized wmf-config/CommonSettings-labs.php: T304111 (duration: 00m 50s)
Comment Actions
Revert was pulled to deployment but not synced (2a37e70fc7ba63209264b890aa36b5b6c19e028d)
Comment Actions
It looks like SFS enforce mode might have worked fine, but that it either broke logs being sent to beta-logs.wmcloud.org, or that coincidentally broke at about the same time as this config patch was deployed. There are definitely logs from today (2022-03-22) on deployment-mwlog01:/srv/mw-log and likely were yesterday. But I was using beta-logs to monitor and didn't think to check on the server. And channel:StopForumSpam is still not reporting anything on beta-logs.
Anyhow, I still want to refactor Hooks::onGetUserPermissionsErrorsExpensive - basically reduce redundant logging and better organize and log around the early returns. That'll be a new bug that I'll file soon. Then we can likely re-enable SFS enforce mode and deal with the beta-logs issue later.
Comment Actions
Upon further review of the StopForumSpam beta logs (/srv/mw-log) for 2022-03-01, about 841 out of 8,490 lines look to be enforcing actions (i.e. not "user tripped...") So this did appear to work properly, but for some reason it also broke channel:StopForumSpam in beta-logs, where it still appears to be broken.
Comment Actions
Change 773340 had a related patch set uploaded (by SBassett; author: SBassett):
[operations/mediawiki-config@master] Set StopForumSpam to enforce on the beta cluster
Comment Actions
Change 773340 merged by jenkins-bot:
[operations/mediawiki-config@master] Set StopForumSpam to enforce on the beta cluster
Comment Actions
Ok, this seems to be working well and we don't appear to have a coincidental beta-logs outage either :) Stats since yesterday's config sync:
- 2,299 total channel:StopForumSpam events on the various *.beta.wmflabs.org sites
- 2,207 active blocks of users/IPs
- 92 exemptions from either sfsblock-bypass or autoblock exemptions
Comment Actions
As of 2022-05-09:
- 71,590 blocked user actions on beta cluster over the past 30 days
- 21,569 IP-based exemptions from SFS blocking actions over the past 30 days
- 6 cases where an IP address was not able to be obtained (not sure how/why this happens)
Comment Actions
From 2022-05-18 to 2022-06-03, according to beta-logs:
- 91,735 blocked user actions on beta cluster
- 20,267 IP-based exemptions from SFS blocking actions
- 6 cases where an IP address was not able to be obtained (not sure how/why this happens)