To achieve T351421: Create a Grafana chart to plot the number of checked and unchecked images per wiki, a maintenance script that publishes to statsd various metrics about the number of unscanned and scanned images is needed.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T351200 [EPIC] Start automated scanning | |||
| Resolved | None | T351420 [Sub EPIC] Create Grafana charts, and Logstash filters | |||
| Resolved | STran | T351421 Create a Grafana chart to plot the number of checked and unchecked images per wiki | |||
| Resolved | kostajh | T353703 Implement daily runs of updateMetrics on WMF wikis | |||
| Resolved | Dreamy_Jazz | T353074 Create a maintenance script to publish to statsd metrics about the mediamoderation_scan table |
Change 981634 had a related patch set uploaded (by Dreamy Jazz; author: Dreamy Jazz):
[mediawiki/extensions/MediaModeration@master] Add updateMetrics.php maintenance script along with statds support
Change 981634 merged by jenkins-bot:
[mediawiki/extensions/MediaModeration@master] Add updateMetrics.php maintenance script along with statsd support
Change 982492 had a related patch set uploaded (by Dreamy Jazz; author: Dreamy Jazz):
[mediawiki/extensions/MediaModeration@master] Add ScannedImagesMetric to MetricsFactory::METRICS
Change 982492 merged by jenkins-bot:
[mediawiki/extensions/MediaModeration@master] Add ScannedImagesMetric to MetricsFactory::METRICS
This ticket can be partly QA'd, though some of the behaviour added (sending data to statsd) is difficult to effectively QA at this stage. This part should be able to be QA'd in T351421: Create a Grafana chart to plot the number of checked and unchecked images per wiki which is dependent on the code written here.
To test the rest of this change you can do the following (on betawikis or local environment):
MediaModeration.ScanTable.TotalCount is X. MediaModeration.ScanTable.Scanned is 0. MediaModeration.ScanTable.Unscanned is X. MediaModeration.ScanTable.UnscannedWithLastCheckedDefined is 0.
SELECT COUNT(*) FROM mediamoderation_scan;
SELECT COUNT(*) FROM mediamoderation_scan WHERE mms_is_match IS NULL;
Per the QA Steps, I was able to verify the number of scanned and unscanned images for the Media Moderation_Scan table.
@Dreamy_Jazz great work once again!!! Appreciate the detailed QA Steps, Thank you sir.
$ mwscript MediaModeration:updateMetrics.php --wiki enwiki --verbose MediaModeration.ScanTable.TotalCount is 138. MediaModeration.ScanTable.Scanned is 0. MediaModeration.ScanTable.Unscanned is 138. MediaModeration.ScanTable.UnscannedWithLastCheckedDefined is 0. wikiadmin@172.16.4.172(enwiki)> SELECT COUNT(*) FROM mediamoderation_scan; +----------+ | COUNT(*) | +----------+ | 138 | +----------+ wikiadmin@172.16.4.172(enwiki)> SELECT COUNT(*) FROM mediamoderation_scan WHERE mms_is_match IS NULL; +----------+ | COUNT(*) | +----------+ | 138 | +----------+ __________________________________________________________________________________________________ $ mwscript MediaModeration:updateMetrics.php --wiki viwiki --verbose MediaModeration.ScanTable.TotalCount is 3. MediaModeration.ScanTable.Scanned is 0. MediaModeration.ScanTable.Unscanned is 3. MediaModeration.ScanTable.UnscannedWithLastCheckedDefined is 0. wikiadmin@172.16.4.172(viwiki)> SELECT COUNT(*) FROM mediamoderation_scan; +----------+ | COUNT(*) | +----------+ | 3 | +----------+ wikiadmin@172.16.4.172(viwiki)> SELECT COUNT(*) FROM mediamoderation_scan WHERE mms_is_match IS NULL; +----------+ | COUNT(*) | +----------+ | 3 | +----------+ __________________________________________________________________________________________________ $ mwscript MediaModeration:updateMetrics.php --wiki commonswiki --verbose MediaModeration.ScanTable.TotalCount is 49835. MediaModeration.ScanTable.Scanned is 0. MediaModeration.ScanTable.Unscanned is 49835. MediaModeration.ScanTable.UnscannedWithLastCheckedDefined is 0. wikiadmin@172.16.4.172(commonswiki)> SELECT COUNT(*) FROM mediamoderation_scan; +----------+ | COUNT(*) | +----------+ | 49835 | +----------+ wikiadmin@172.16.4.172(commonswiki)> SELECT COUNT(*) FROM mediamoderation_scan WHERE mms_is_match IS NULL; +----------+ | COUNT(*) | +----------+ | 49835 | +----------+ __________________________________________________________________________________________________ $ mwscript MediaModeration:updateMetrics.php --wiki fawiki --verbose MediaModeration.ScanTable.TotalCount is 4. MediaModeration.ScanTable.Scanned is 0. MediaModeration.ScanTable.Unscanned is 4. MediaModeration.ScanTable.UnscannedWithLastCheckedDefined is 0. wikiadmin@172.16.4.172(fawiki)> SELECT COUNT(*) FROM mediamoderation_scan; +----------+ | COUNT(*) | +----------+ | 4 | +----------+ wikiadmin@172.16.4.172(fawiki)> SELECT COUNT(*) FROM mediamoderation_scan WHERE mms_is_match IS NULL; +----------+ | COUNT(*) | +----------+ | 4 | +----------+ __________________________________________________________________________________________________ $ mwscript MediaModeration:updateMetrics.php --wiki dewiki --verbose MediaModeration.ScanTable.TotalCount is 104. MediaModeration.ScanTable.Scanned is 0. MediaModeration.ScanTable.Unscanned is 104. MediaModeration.ScanTable.UnscannedWithLastCheckedDefined is 0. wikiadmin@172.16.4.172(dewiki)> SELECT COUNT(*) FROM mediamoderation_scan; +----------+ | COUNT(*) | +----------+ | 104 | +----------+ wikiadmin@172.16.4.172(dewiki)> SELECT COUNT(*) FROM mediamoderation_scan WHERE mms_is_match IS NULL; +----------+ | COUNT(*) | +----------+ | 104 | +----------+ __________________________________________________________________________________________________ $ mwscript MediaModeration:updateMetrics.php --wiki zhwiki --verbose MediaModeration.ScanTable.TotalCount is 12. MediaModeration.ScanTable.Scanned is 0. MediaModeration.ScanTable.Unscanned is 12. MediaModeration.ScanTable.UnscannedWithLastCheckedDefined is 0. wikiadmin@172.16.4.172(zhwiki)> SELECT COUNT(*) FROM mediamoderation_scan; +----------+ | COUNT(*) | +----------+ | 12 | +----------+ wikiadmin@172.16.4.172(zhwiki)> SELECT COUNT(*) FROM mediamoderation_scan WHERE mms_is_match IS NULL; +----------+ | COUNT(*) | +----------+ | 12 | +----------+ __________________________________________________________________________________________________ $ mwscript MediaModeration:updateMetrics.php --wiki metawiki --verbose MediaModeration.ScanTable.TotalCount is 1. MediaModeration.ScanTable.Scanned is 0. MediaModeration.ScanTable.Unscanned is 1. MediaModeration.ScanTable.UnscannedWithLastCheckedDefined is 0. wikiadmin@172.16.4.172(metawiki)> SELECT COUNT(*) FROM mediamoderation_scan; +----------+ | COUNT(*) | +----------+ | 1 | +----------+ wikiadmin@172.16.4.172(metawiki)> SELECT COUNT(*) FROM mediamoderation_scan WHERE mms_is_match IS NULL; +----------+ | COUNT(*) | +----------+ | 1 | +----------+ __________________________________________________________________________________________________ $ mwscript MediaModeration:updateMetrics.php --wiki testwiki --verbose MediaModeration.ScanTable.TotalCount is 15. MediaModeration.ScanTable.Scanned is 0. MediaModeration.ScanTable.Unscanned is 15. MediaModeration.ScanTable.UnscannedWithLastCheckedDefined is 0. wikiadmin@172.16.4.172(testwiki)> SELECT COUNT(*) FROM mediamoderation_scan; +----------+ | COUNT(*) | +----------+ | 15 | +----------+ wikiadmin@172.16.4.172(testwiki)> SELECT COUNT(*) FROM mediamoderation_scan WHERE mms_is_match IS NULL; +----------+ | COUNT(*) | +----------+ | 15 | +----------+