Page MenuHomePhabricator

Separate off ChangePropagation for JobQueue as a new deployment
Closed, ResolvedPublic

Description

As part of the effort to move the JobQueue transport to the EventBus sub-system, we have to create a separate deployment repository for the ChangePropagation instance that is going to be handling the jobs. This will allow us to (i) easily move the processing elsewhere if resources start being a problem; and (ii) separate the processing of jobs from the processing of asynchronous updates.

This instance should initially be placed on SCB. The deploy repo is change-propagation/jobqueue-deploy.

Details

Related Gerrit Patches:
mediawiki/services/change-propagation/jobqueue-deploy : masterBug fix: The service's name is cpjobqueue, not cp-jobqueue
operations/puppet : productionCP-JobQueue: Add the service to SCB
mediawiki/services/change-propagation/jobqueue-deploy : masterMove the repo over from cp-jobqueue
operations/puppet : productionChangeProp-JobQueue: Fix the repo location and name
operations/puppet : productionAdd the cp-jobqueue profile

Event Timeline

mobrovac created this task.Sep 7 2017, 2:44 PM
Restricted Application edited projects, added Services, Analytics; removed Services (doing). · View Herald TranscriptSep 7 2017, 2:44 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Change 376527 had a related patch set uploaded (by Mobrovac; owner: Mobrovac):
[mediawiki/services/cp-jobqueue@master] Initial add of the Scap3 configuration

https://gerrit.wikimedia.org/r/376527

mobrovac updated the task description. (Show Details)Sep 7 2017, 2:49 PM

Change 376527 merged by Mobrovac:
[mediawiki/services/cp-jobqueue@master] Initial add of the Scap3 configuration

https://gerrit.wikimedia.org/r/376527

Change 376707 had a related patch set uploaded (by Mobrovac; owner: Mobrovac):
[operations/puppet@production] Add the cp-jobqueue profile

https://gerrit.wikimedia.org/r/376707

Change 376707 merged by Alexandros Kosiaris:
[operations/puppet@production] Add the cp-jobqueue profile

https://gerrit.wikimedia.org/r/376707

Change 376724 had a related patch set uploaded (by Mobrovac; owner: Mobrovac):
[operations/puppet@production] ChangeProp-JobQueue: Fix the repo location and name

https://gerrit.wikimedia.org/r/376724

Change 376724 merged by Alexandros Kosiaris:
[operations/puppet@production] ChangeProp-JobQueue: Fix the repo location and name

https://gerrit.wikimedia.org/r/376724

Change 376730 had a related patch set uploaded (by Mobrovac; owner: Mobrovac):
[mediawiki/services/change-propagation/jobqueue-deploy@master] Move the repo over from cp-jobqueue

https://gerrit.wikimedia.org/r/376730

Change 376730 merged by Mobrovac:
[mediawiki/services/change-propagation/jobqueue-deploy@master] Move the repo over from cp-jobqueue

https://gerrit.wikimedia.org/r/376730

mobrovac updated the task description. (Show Details)Sep 8 2017, 2:50 PM

The repo has been set up and cloned on tin and the ops/puppet profile created and merged. Left to do is to add the profile to SCB's role, which is scheduled to happen on Monday, 2017-09-08.

Change 377213 had a related patch set uploaded (by Mobrovac; owner: Mobrovac):
[operations/puppet@production] CP-JobQueue: Add the service to SCB

https://gerrit.wikimedia.org/r/377213

Change 377213 merged by Giuseppe Lavagetto:
[operations/puppet@production] CP-JobQueue: Add the service to SCB

https://gerrit.wikimedia.org/r/377213

Change 377221 had a related patch set uploaded (by Mobrovac; owner: Mobrovac):
[mediawiki/services/change-propagation/jobqueue-deploy@master] Bug fix: The service's name is cpjobqueue, not cp-jobqueue

https://gerrit.wikimedia.org/r/377221

Change 377221 merged by Mobrovac:
[mediawiki/services/change-propagation/jobqueue-deploy@master] Bug fix: The service's name is cpjobqueue, not cp-jobqueue

https://gerrit.wikimedia.org/r/377221

Mentioned in SAL (#wikimedia-operations) [2017-09-11T09:39:34Z] <mobrovac@tin> Started deploy [cpjobqueue/deploy@e73a10f]: Initial deploy of cpjobqueue - T175281

Mentioned in SAL (#wikimedia-operations) [2017-09-11T09:39:55Z] <mobrovac@tin> Finished deploy [cpjobqueue/deploy@e73a10f]: Initial deploy of cpjobqueue - T175281 (duration: 00m 20s)

mobrovac closed this task as Resolved.Sep 11 2017, 9:46 AM
mobrovac edited projects, added Services (done); removed Patch-For-Review, Services.

Everything is set up now, and the cpjobqueue service is live in production on the SCB cluster (currently idling pending the resolution of T175210: Select candidate jobs for transferring to the new infrastucture). Calling this done!