I don't have an immediate explanation for these spikes - there was a bad instance of Thumbor over the weekend for a few hours that would have caused an increased rate of bad responses, but that was cleaned up relatively quickly and also wouldn't have responded with 404s. I can't really imagine why the migrated job would trigger a somewhat increased rate of failures, as if the job wasn't performing correctly I'd expect a massive spike and/or complete failure to scan. For the short term should we migrate the job back to metal and see if the rate changes?

Perhaps we could try switching back to bare metal?

Just to note that this increased spike in scanning failures has not stopped:

However, the metrics generated by UpdateMetrics have seemed to recover suggesting that this was a temporary problem.

Having looked at the queries we use, we do not average out the metrics per-wiki. Doing this it actually removes the spike, which suggests to me that the metrics script was run more than once in a day for some of the wikis (that then skewed the overall metric). Adding an average to the queries fixes the issue.

Change #1143501 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/puppet@production] Revert "mw::maintenance: migrate mediamoderation-hourlyScan to k8s"

https://gerrit.wikimedia.org/r/1143501

Change #1143501 merged by Hnowlan:

[operations/puppet@production] Revert "mw::maintenance: migrate mediamoderation-hourlyScan to k8s"

https://gerrit.wikimedia.org/r/1143501

In T393236#10803638, @Dreamy_Jazz wrote:

Perhaps we could try switching back to bare metal?

Just to note that this increased spike in scanning failures has not stopped:

---

However, the metrics generated by UpdateMetrics have seemed to recover suggesting that this was a temporary problem.

Having looked at the queries we use, we do not average out the metrics per-wiki. Doing this it actually removes the spike, which suggests to me that the metrics script was run more than once in a day for some of the wikis (that then skewed the overall metric). Adding an average to the queries fixes the issue.

I've reverted for the time being. Please let me know how things look after the next run.

With the graphs being tallied per day, it will be hard to see a change until we've let this sit for a day or two. I'll check back then if that's okay?

FWIW, I'm not sure that we will see a change because the script on k8s on WMF production just queues jobs. This should mean that nothing would change if the place where the job is queued from does not affect the running of the job. There may be some latency involved, and perhaps bare metal takes longer to queue the jobs?

One immediate change I saw was these graphs:

It seems that after the deployment of the above change the scanning is now starting to use the source image more often (when the thumbnail does not meet the requirements). This also shows in the "Thumbnail and source image lookup failed" graph, where the decreased number means that we are now able to get more images to scan.

Unfortunately, due to T393894: New version of Grafana makes it not possible to remove option in long list of values, I cannot filter out commonswiki to better verify that the changes in the graphs are from the hourly script run. When I try to load the page using the URL generated by pressing "Selected", the URL itself is too long and Grafana refuses to respond. To get a good picture, I will need to check all wikis but commonswiki.

The number of 404 errors from Thumbor hasn't changed though, however, due to T393894 I can't verify if these are just from commonswiki or also elsewhere:

Based on the idea in T393894#10811739, I've been able to unselect all wikis but commonswiki. Having done this, it appears that the issues seem to be with commonswiki.

Therefore, it seems that the reverting of the hourly scan change did not fix the issue and it should be fine to undo the revert. cc @hnowlan.

Thanks for reverting the change to allow me to check if it was the cause.

Change #1144582 had a related patch set uploaded (by Dreamy Jazz; author: Dreamy Jazz):

[operations/puppet@production] Revert^2 "mw::maintenance: migrate mediamoderation-hourlyScan to k8s"

https://gerrit.wikimedia.org/r/1144582

In T393236#10811776, @Dreamy_Jazz wrote:

Based on the idea in T393894#10811739, I've been able to unselect all wikis but commonswiki. Having done this, it appears that the issues seem to be with commonswiki.

Therefore, it seems that the reverting of the hourly scan change did not fix the issue and it should be fine to undo the revert. cc @hnowlan.

Thanks for reverting the change to allow me to check if it was the cause.

Great, thanks for clearing that up and filing the revert!

Change #1144582 merged by Hnowlan:

[operations/puppet@production] Revert^2 "mw::maintenance: migrate mediamoderation-hourlyScan to k8s"

https://gerrit.wikimedia.org/r/1144582