Page MenuHomePhabricator
Create Task
Maniphest T231194

Increase concurrency of the cirrusCheckerJob
Closed, ResolvedPublic
Actions

Description

As we introduced a new elastic cluster to write to (cloudelastic) the CheckerJob has now more work to do.
We tuned the cloudelastic cluster to keep up with the synchronous writes but we are still behind on the checker job.

Details

SubjectRepoBranchLines +/-
[cirrus] Reenable sanity checksoperations/mediawiki-configmaster+1 -2
[cirrus] Stop generating new cirrusSearchChecker jobsoperations/mediawiki-configmaster+2 -2
cirrusSearchLinksUpdate: Increase concurrency to 150mediawiki/services/change-propagation/jobqueue-deploymaster+1 -1
[TEMP] cirusSearchCheckerJob: Increase concurrency to 20mediawiki/services/change-propagation/jobqueue-deploymaster+1 -0
Customize query in gerrit

Event Timeline

• dcausse created this task.Aug 26 2019, 11:34 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 26 2019, 11:34 AM
• dcausse moved this task from Incoming to Waiting on the Discovery-Search (Current work) board.Aug 26 2019, 11:35 AM
gerritbot added a comment.Aug 26 2019, 11:38 AM

Change 532357 had a related patch set uploaded (by Mobrovac; owner: Mobrovac):
[mediawiki/services/change-propagation/jobqueue-deploy@master] [TEMP] cirusSearchCheckerJob: Increase concurrency to 20

https://gerrit.wikimedia.org/r/532357

gerritbot added a project: Patch-For-Review.Aug 26 2019, 11:38 AM
• mobrovac triaged this task as Medium priority.Aug 26 2019, 11:39 AM
• mobrovac edited projects, added Services (doing), Platform Team Workboards (Clinic Duty Team); removed Services.
gerritbot added a comment.Aug 26 2019, 11:40 AM

Change 532357 merged by Mobrovac:
[mediawiki/services/change-propagation/jobqueue-deploy@master] [TEMP] cirusSearchCheckerJob: Increase concurrency to 20

https://gerrit.wikimedia.org/r/532357

Stashbot added a comment.Aug 26 2019, 11:55 AM

Mentioned in SAL (#wikimedia-operations) [2019-08-26T11:55:43Z] <mobrovac@deploy1001> Started deploy [cpjobqueue/deploy@e742ecf]: Increase the concurrency of cirusSearchCheckerJobs to 20 - T231194

Stashbot added a comment.Aug 26 2019, 11:57 AM

Mentioned in SAL (#wikimedia-operations) [2019-08-26T11:57:13Z] <mobrovac@deploy1001> Finished deploy [cpjobqueue/deploy@e742ecf]: Increase the concurrency of cirusSearchCheckerJobs to 20 - T231194 (duration: 01m 31s)

Maintenance_bot removed a project: Patch-For-Review.Aug 26 2019, 12:10 PM
• mobrovac changed the task status from Open to Stalled.Aug 26 2019, 12:10 PM
• mobrovac moved this task from Inbox to Blocked Externally on the Platform Team Workboards (Clinic Duty Team) board.

Stalling until the backlog has been cleared. Once that happens, we'll likely want to decrease the concurrency slightly.

gerritbot added a comment.Aug 27 2019, 8:03 AM

Change 532556 had a related patch set uploaded (by Mobrovac; owner: Mobrovac):
[mediawiki/services/change-propagation/jobqueue-deploy@master] cirrusSearchLinksUpdate: Increase concurrency to 150

https://gerrit.wikimedia.org/r/532556

gerritbot added a project: Patch-For-Review.Aug 27 2019, 8:03 AM
gerritbot added a comment.Aug 27 2019, 9:08 AM

Change 532556 merged by Mobrovac:
[mediawiki/services/change-propagation/jobqueue-deploy@master] cirrusSearchLinksUpdate: Increase concurrency to 150

https://gerrit.wikimedia.org/r/532556

Stashbot added a comment.Aug 27 2019, 9:09 AM

Mentioned in SAL (#wikimedia-operations) [2019-08-27T09:09:56Z] <mobrovac@deploy1001> Started deploy [cpjobqueue/deploy@c2bc1a3]: Increase cirrusSearchLinksUpdate concurrency to 150 - T231194

Maintenance_bot removed a project: Patch-For-Review.Aug 27 2019, 9:10 AM
Stashbot added a comment.Aug 27 2019, 9:11 AM

Mentioned in SAL (#wikimedia-operations) [2019-08-27T09:11:05Z] <mobrovac@deploy1001> Finished deploy [cpjobqueue/deploy@c2bc1a3]: Increase cirrusSearchLinksUpdate concurrency to 150 - T231194 (duration: 01m 09s)

gerritbot added a comment.Aug 27 2019, 11:40 AM

Change 532694 had a related patch set uploaded (by DCausse; owner: DCausse):
[operations/mediawiki-config@master] [cirrus] Stop generating new cirrusSearchChecker jobs

https://gerrit.wikimedia.org/r/532694

gerritbot added a project: Patch-For-Review.Aug 27 2019, 11:40 AM
gerritbot added a comment.Aug 27 2019, 11:46 AM

Change 532694 merged by jenkins-bot:
[operations/mediawiki-config@master] [cirrus] Stop generating new cirrusSearchChecker jobs

https://gerrit.wikimedia.org/r/532694

Stashbot added a comment.Aug 27 2019, 11:49 AM

Mentioned in SAL (#wikimedia-operations) [2019-08-27T11:49:24Z] <dcausse@deploy1001> Synchronized wmf-config/CirrusSearch-production.php: T231194 [cirrus] Stop generating new cirrusSearchChecker jobs (duration: 00m 45s)

Maintenance_bot removed a project: Patch-For-Review.Aug 27 2019, 12:10 PM
gerritbot added a comment.Sep 2 2019, 8:20 AM

Change 533842 had a related patch set uploaded (by DCausse; owner: DCausse):
[operations/mediawiki-config@master] [cirrus] Reenable sanity checks

https://gerrit.wikimedia.org/r/533842

gerritbot added a project: Patch-For-Review.Sep 2 2019, 8:20 AM
gerritbot added a comment.Sep 4 2019, 11:40 AM

Change 533842 merged by jenkins-bot:
[operations/mediawiki-config@master] [cirrus] Reenable sanity checks

https://gerrit.wikimedia.org/r/533842

Stashbot added a comment.Sep 4 2019, 11:49 AM

Mentioned in SAL (#wikimedia-operations) [2019-09-04T11:49:34Z] <dcausse@deploy1001> Synchronized wmf-config/CirrusSearch-production.php: T231194: [cirrus] Reenable sanity checks (duration: 00m 56s)

Maintenance_bot removed a project: Patch-For-Review.Sep 4 2019, 12:10 PM
• dcausse closed this task as Resolved.Sep 23 2019, 8:56 AM

Thanks @mobrovac, we re-enabled the sanitizer and the topics were stable. closing.

• WDoranWMF moved this task from Blocked Externally to Done on the Platform Team Workboards (Clinic Duty Team) board.Oct 29 2019, 3:27 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits