Page MenuHomePhabricator
Create Task
Maniphest T198220

Stop and remove old job runners
Closed, ResolvedPublic
Actions

Description

All the jobs were completely transitioned to the Kafka queue for all projects, so old job runner service processes and job queue Redis instances are not used anymore. These should be stopped and removed, and puppet cleaned up.

Details

SubjectRepoBranchLines +/-
site.pp: move slave redises to system::spareoperations/puppetproduction+9 -7
jobqueue_redis: Purge role jobqueue_redisoperations/puppetproduction+0 -187
Purge the last references to jobqueue redis.operations/mediawiki-configmaster+0 -32
redis: remove now-useless specific classesoperations/puppetproduction+0 -36
redis: remove cronjob for restarts on slaves as welloperations/puppetproduction+3 -15
profile::jobrunner: remove references to mediawiki::jobrunneroperations/puppetproduction+0 -104
profile::mediawiki::jobrunner: remove the old jobrunner serviceoperations/puppetproduction+29 -219
Customize query in gerrit

Related Objects
Search...

Event Timeline

• Pchelolo triaged this task as Medium priority.Jun 26 2018, 2:39 PM
• Pchelolo created this task.
Restricted Application added a project: Analytics. · View Herald TranscriptJun 26 2018, 2:39 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
• mobrovac edited parent tasks, added: T157088: [EPIC] Develop a JobQueue backend based on EventBus; removed: T190327: FY17/18 Q4 Program 8 Services Goal: Complete the JobQueue transition to EventBus.Jun 26 2018, 2:43 PM
• mobrovac added a project: User-Joe.
Joe claimed this task.Jul 2 2018, 1:11 PM
Joe moved this task from Backlog to Doing on the User-Joe board.
gerritbot subscribed.Jul 2 2018, 1:12 PM

Change 443420 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/mediawiki-config@master] Purge the last references to jobqueue redis.

https://gerrit.wikimedia.org/r/443420

gerritbot added a project: Patch-For-Review.Jul 2 2018, 1:12 PM
gerritbot added a comment.Jul 2 2018, 2:33 PM

Change 443426 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] profile::mediawiki::jobrunner: remove the old jobrunner service

https://gerrit.wikimedia.org/r/443426

gerritbot added a comment.Jul 2 2018, 2:33 PM

Change 443427 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] profile::jobrunner: remove references to mediawiki::jobrunner

https://gerrit.wikimedia.org/r/443427

gerritbot added a comment.Jul 2 2018, 3:52 PM

Change 443426 merged by Giuseppe Lavagetto:
[operations/puppet@production] profile::mediawiki::jobrunner: remove the old jobrunner service

https://gerrit.wikimedia.org/r/443426

Stashbot subscribed.Jul 2 2018, 3:55 PM

Mentioned in SAL (#wikimedia-operations) [2018-07-02T15:55:55Z] <_joe_> stopping jobrunner, jobchron across all jobrunners T198220

• fdans moved this task from Incoming to Radar on the Analytics board.Jul 2 2018, 4:18 PM
gerritbot added a comment.Jul 3 2018, 7:28 AM

Change 443427 merged by Giuseppe Lavagetto:
[operations/puppet@production] profile::jobrunner: remove references to mediawiki::jobrunner

https://gerrit.wikimedia.org/r/443427

gerritbot added a comment.Jul 4 2018, 8:30 AM

Change 443772 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] redis: remove cronjob for restarts on slaves as well

https://gerrit.wikimedia.org/r/443772

gerritbot added a comment.Jul 4 2018, 8:30 AM

Change 443773 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] redis: remove now-useless specific classes

https://gerrit.wikimedia.org/r/443773

gerritbot added a comment.Jul 4 2018, 8:30 AM

Change 443789 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] site.pp: move slave redises to system::spare

https://gerrit.wikimedia.org/r/443789

gerritbot added a comment.Jul 4 2018, 8:40 AM

Change 443772 merged by Giuseppe Lavagetto:
[operations/puppet@production] redis: remove cronjob for restarts on slaves as well

https://gerrit.wikimedia.org/r/443772

gerritbot added a comment.Jul 4 2018, 9:03 AM

Change 443773 merged by Giuseppe Lavagetto:
[operations/puppet@production] redis: remove now-useless specific classes

https://gerrit.wikimedia.org/r/443773

Krinkle edited projects, added WMF-JobQueue; removed MediaWiki-Core-JobQueue.Jul 11 2018, 2:50 AM
Krinkle mentioned this in WMF-JobQueue.Jul 11 2018, 3:02 AM
Krinkle moved this task from Untriaged to Legacy infra on the WMF-JobQueue board.
Krinkle mentioned this in T160146: jobrunner/jobchron services fail in codfw .Jul 11 2018, 3:05 AM
Krinkle mentioned this in T146039: redisJobRunnerService uses sh to run curl commands and shows defunct sh process.
Krinkle mentioned this in T87398: Job runner tight loop when job class does not exist.
Krinkle mentioned this in T172479: Collect error logs from jobchron/jobrunner services in Logstash.Jul 14 2018, 1:56 AM
Joe moved this task from Doing to Blocking others on the User-Joe board.Aug 3 2018, 9:54 AM
gerritbot added a comment.Nov 7 2018, 2:58 PM

Change 443420 merged by jenkins-bot:
[operations/mediawiki-config@master] Purge the last references to jobqueue redis.

https://gerrit.wikimedia.org/r/443420

jijiki added a project: User-jijiki.Nov 7 2018, 3:20 PM
jijiki subscribed.
Stashbot added a comment.Nov 7 2018, 3:29 PM

Mentioned in SAL (#wikimedia-operations) [2018-11-07T15:29:02Z] <jiji@deploy1001> Synchronized wmf-config/ProductionServices.php: Remove jobqueue_redis references, T198220 (duration: 00m 54s)

gerritbot added a comment.Nov 12 2018, 4:05 PM

Change 473029 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] jobqueue_redis: Purge role jobqueue_redis

https://gerrit.wikimedia.org/r/473029

gerritbot added a comment.Nov 14 2018, 3:19 PM

Change 473029 merged by Effie Mouzeli:
[operations/puppet@production] jobqueue_redis: Purge role jobqueue_redis

https://gerrit.wikimedia.org/r/473029

jijiki moved this task from Inbox 🐅 to In Progress 🏋️‍♀️ on the User-jijiki board.Nov 14 2018, 6:44 PM
jijiki added a comment.Nov 19 2018, 9:15 AM

@Pchelolo/@mobrovac jobqueue_redis instances have been removed from prod and we have cleaned up any puppet and mediawiki-config references . Should we mark this issue as Resolved?

• mobrovac closed this task as Resolved.Nov 19 2018, 10:33 AM
• mobrovac edited projects, added Services (done); removed Patch-For-Review, Services (watching).

Indeed @jijiki ! Thanks!

gerritbot added a comment.Nov 23 2018, 3:21 PM

Change 443789 abandoned by Giuseppe Lavagetto:
site.pp: move slave redises to system::spare

Reason:
not needed anymore

https://gerrit.wikimedia.org/r/443789

Aklapper edited projects, added Analytics-Radar; removed Analytics.Jun 10 2020, 6:44 AM
Aklapper mentioned this in T135113: Rationalize our jobqueues redis topology.Apr 25 2021, 7:44 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL