Page MenuHomePhabricator
Create Task
Maniphest T257620

Beta cluster changeprop job queue down since 2020-06-18
Closed, ResolvedPublic
Actions

Description

It appears that that at least some jobs on Beta halted on 2020-06-18, when the old deployment-changeprop and deployment-cpjobqueue instances were shut down as a result of T255068.

https://grafana-labs.wikimedia.org/d/000000034/jobqueue-eventbus?orgId=1&from=now-60d&to=now

I don't think this is merely a matter of a dashboard needing to be updated, because fetchGoogleCloudVisionAnnotations jobs are actually no longer running since then. (Some jobs do still seem to be running, as reflected on deployment-fluorine02:/srv/mw-log/JobExecutor.log; I don't know how some job types are being processed and not others.)

Related Objects

Event Timeline

• Mholloway created this task.Jul 9 2020, 7:54 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 9 2020, 7:54 PM
hnowlan claimed this task.Jul 10 2020, 10:21 AM
hnowlan added a comment.Jul 10 2020, 10:39 AM

This is a side effect of the migration to Docker - I think the docker instance either didn't pick up the correct config or needed to be restarted to take the new config. Metrics weren't being sent and while some jobs were being executed, it looked like none were. I'm seeing more metrics appearing now and I'll keep an eye to see if actual events are being processed

BPirkle triaged this task as Medium priority.Jul 14 2020, 8:42 PM
BPirkle moved this task from Inbox to Tracking/Watching on the Platform Engineering board.
hnowlan closed this task as Resolved.May 25 2021, 4:34 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL