I've ruled out any structural failure (e.g. the script looking in the wrong directory) because it does still reliably do what it's supposed to do: Create SVG Flame Graphs files for performance.wikimedia.org.

It is most likely a race condition because of old log files being removed on rotation.

Unless it causes the script to stop early and it happens every time, then it has no impact on the application because the purpose of arclamp-generate-svgs is to ensure SVG files exist for each of the incoming log files. If a log file no longer exists, it also doesn't need an SVG any more, and the next run it will get to the other files (if it caused it to stop early).

Either way, we should fix it by making sure the error is caught, ignored and let the script move on.

Assuming no impact on actual functionality, this could have lower priority, not have the Wikimedia-production-error and be lower priority, just be kept (and eventually be solved or workarounded) to try to reduce root cron spam.

I consider log spam as impact. It's okay, we'll get it fixed. We only have one other open prod-error and I'd rather we not get in the habit of building up a backlog.

Thanks, let us know how we can help.

Change 568117 had a related patch set uploaded (by Dave Pifke; owner: Dave Pifke):
[operations/puppet@production] Fix log spam from arclamp-generate-svgs

https://gerrit.wikimedia.org/r/568117

Change 568117 merged by Jcrespo:
[operations/puppet@production] Fix log spam from arclamp-generate-svgs

https://gerrit.wikimedia.org/r/568117