Page MenuHomePhabricator
Create Task
Maniphest T190327

FY17/18 Q4 Program 8 Services Goal: Complete the JobQueue transition to EventBus
Closed, ResolvedPublic
Actions

Description

For FY17/18 Q4, the Services team will:

  • Port the remaining jobs over to EventBus
  • Enable support for private wikis

Details

SubjectRepoBranchLines +/-
Switch all jobs to the new queue and clean up the old queue configs.operations/mediawiki-configmaster+1 -132
Remove unused jobrunners.operations/puppetproduction+14 -72
Disable redis queue for videoscaler jobs.operations/mediawiki-configmaster+1 -3
Enable videoscaler jobs in the new queue.mediawiki/services/change-propagation/jobqueue-deploymaster+23 -2
Specify videoscalers uri in hiera/changeprop manifest.operations/puppetproduction+6 -0
Switch all job apart from exceptions for everything.operations/mediawiki-configmaster+5 -22
Enable all jobs apart from exceptions for everything.mediawiki/services/change-propagation/jobqueue-deploymaster+3 -45
Switch all jobs for everything except wikipedia, commons and wikidata.operations/mediawiki-configmaster+5 -7
Switch all jobs for everything except wikipedia, commons and wikiata.mediawiki/services/change-propagation/jobqueue-deploymaster+2 -3
Switch cross-wiki posting jobs for everything.operations/mediawiki-configmaster+8 -0
[config] Switch cross-wiki posting jobs for everything.mediawiki/services/change-propagation/jobqueue-deploymaster+7 -0
Disable Redis queue for most of jobs for test wikis.operations/mediawiki-configmaster+22 -8
Enable all jobs for test, test2, testwikidata and mediawiki.mediawiki/services/change-propagation/jobqueue-deploymaster+15 -5
Validate that events are JSON-serializble and log and error if not.mediawiki/extensions/EventBusmaster+37 -0
Switch second bulk of low-traffic jobs for all wikis.operations/mediawiki-configmaster+9 -18
Switch second bulk of low-traffic jobs for all wikis.mediawiki/services/change-propagation/jobqueue-deploymaster+7 -5
Disable bulk number 2 jobs in redis.operations/mediawiki-configmaster+26 -0
Switch a bulk of jobs for test wikis.mediawiki/services/change-propagation/jobqueue-deploymaster+47 -4
Clean up the config for high traffic jobs.operations/mediawiki-configmaster+6 -13
Switch high traffic jobs to kafka for all wikis.operations/mediawiki-configmaster+12 -22
Switch high traffic jobs to kafka for all wikis.mediawiki/services/change-propagation/jobqueue-deploymaster+5 -25
Switch remaining high traffic jobs for test wikis.operations/mediawiki-configmaster+22 -0
Switch remaining high traffic jobs for test wikis.mediawiki/services/change-propagation/jobqueue-deploymaster+23 -0
Show related patches Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 Resolved• PcheloloT157088 [EPIC] Develop a JobQueue backend based on EventBus
 Resolved• PcheloloT190327 FY17/18 Q4 Program 8 Services Goal: Complete the JobQueue transition to EventBus
 Resolved• PcheloloT175210 Select candidate jobs for transferring to the new infrastucture
 Resolved• PcheloloT191238 Add support for catch-all rule in ChangeProp
 Resolved• PcheloloT189137 Migrate CirrusSearch jobs to Kafka queue
 ResolvedEBernhardsonT190958 CirrusSearchCheckerJob should have a title
 ResolvedEBernhardsonT191024 Exception thrown while running DataSender::sendData in cluster codfw: Data should be a Document, a Script or an array containing Documents and/or Scripts
 Resolved• PcheloloT191464 Enable CP4JQ support for private wikis
 ResolvedOttomataT192005 Disable MirrorMaker for job queue events
 Resolved• PcheloloT192198 Wikidata doesn't update recentchanges
 Resolved• PcheloloT192363 The .meta.domain is incorrect in EventBus when other wiki is used
 ResolvedSBissonT192945 Make EchoNotification job JSON-serializable
ResolvedPRODUCTION ERRORSBissonT194848 Fatal error: $this is null in Echo/includes/model/Event.php on line 345
 ResolvedmatthiasmullieT192946 Make gwtoolsetUploadMediafileJob JSON-serializable
 ResolvedLadsgroupT192085 PHP Fatal in AffectedPagesFinder::getChangedAspects
 ResolvedPRODUCTION ERROR• mobrovacT193254 Global renames get stuck at metawiki
 InvalidNoneT193471 JobQueueGroup's singletons using the wrong wgJobTypeConf
 Resolved• PcheloloT188947 Create an LVS endpoint for jobrunners on videoscalers
 ResolvedNikerabbitT192107 Unable to mark pages for translation in Meta
 ResolvedNikerabbitT192111 Make TranslationsUpdateJob JSON-serializable
 Resolved• mobrovacT195500 Global mass message delivered on meta but not on other wikis?
 Resolved• PcheloloT195858 Make JobExecutor debug-log to mwlog
 ResolvedfgiunchediT196067 Clean up cpjobqueue metrics
 ResolvedLadsgroupT196076 ORESFetchScoreJob fails quite a lot

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
• mobrovac closed subtask T192111: Make TranslationsUpdateJob JSON-serializable as Resolved.May 8 2018, 3:35 PM
Etonkovidova closed subtask T192945: Make EchoNotification job JSON-serializable as Resolved.May 12 2018, 12:39 AM
• mobrovac reopened subtask T192945: Make EchoNotification job JSON-serializable as Open.May 12 2018, 7:22 AM
gerritbot added a comment.May 22 2018, 10:08 AM

Change 434466 had a related patch set uploaded (by Ppchelko; owner: Ppchelko):
[operations/mediawiki-config@master] Switch cross-wiki posting jobs for everything.

https://gerrit.wikimedia.org/r/434466

gerritbot added a comment.May 22 2018, 10:14 AM

Change 434467 had a related patch set uploaded (by Ppchelko; owner: Ppchelko):
[mediawiki/services/change-propagation/jobqueue-deploy@master] [config] Switch cross-wiki posting jobs for everything.

https://gerrit.wikimedia.org/r/434467

gerritbot added a comment.May 22 2018, 10:26 AM

Change 434467 merged by Mobrovac:
[mediawiki/services/change-propagation/jobqueue-deploy@master] [config] Switch cross-wiki posting jobs for everything.

https://gerrit.wikimedia.org/r/434467

gerritbot added a comment.May 22 2018, 10:28 AM

Change 434466 merged by jenkins-bot:
[operations/mediawiki-config@master] Switch cross-wiki posting jobs for everything.

https://gerrit.wikimedia.org/r/434466

Stashbot added a comment.May 22 2018, 10:32 AM

Mentioned in SAL (#wikimedia-operations) [2018-05-22T10:32:03Z] <mobrovac@tin> Synchronized wmf-config/jobqueue.php: Switch cross-wiki posting jobs to EventBus - T190327 (duration: 01m 18s)

• Pchelolo added a subtask: T188947: Create an LVS endpoint for jobrunners on videoscalers.May 22 2018, 10:47 AM
• mobrovac added a subtask: T192107: Unable to mark pages for translation in Meta.May 22 2018, 10:48 AM
gerritbot added a comment.May 24 2018, 9:32 AM

Change 429983 merged by Mobrovac:
[mediawiki/services/change-propagation/jobqueue-deploy@master] Switch all jobs for everything except wikipedia, commons and wikiata.

https://gerrit.wikimedia.org/r/429983

gerritbot added a comment.May 24 2018, 9:33 AM

Change 429980 merged by jenkins-bot:
[operations/mediawiki-config@master] Switch all jobs for everything except wikipedia, commons and wikidata.

https://gerrit.wikimedia.org/r/429980

Stashbot added a comment.May 24 2018, 9:37 AM

Mentioned in SAL (#wikimedia-operations) [2018-05-24T09:37:25Z] <mobrovac@tin> Synchronized wmf-config/jobqueue.php: Switch all jobs to EventBus for everything except wikipedia, commons and wikidata, file 1/2 - T190327 (duration: 01m 09s)

Stashbot added a comment.May 24 2018, 9:53 AM

Mentioned in SAL (#wikimedia-operations) [2018-05-24T09:53:42Z] <mobrovac@tin> Synchronized wmf-config/jobqueue.php: Switch all jobs to EventBus for everything except wikipedia, commons and wikidata, file 1/2, take #2 - T190327 (duration: 01m 08s)

Stashbot added a comment.May 24 2018, 9:54 AM

Mentioned in SAL (#wikimedia-operations) [2018-05-24T09:54:25Z] <ppchelko@tin> Started deploy [cpjobqueue/deploy@b537fa1]: Switch all non-special jobs for everything except wikipedia, commons and wikidata T190327

Stashbot added a comment.May 24 2018, 9:55 AM

Mentioned in SAL (#wikimedia-operations) [2018-05-24T09:55:07Z] <ppchelko@tin> Finished deploy [cpjobqueue/deploy@b537fa1]: Switch all non-special jobs for everything except wikipedia, commons and wikidata T190327 (duration: 00m 42s)

Stashbot added a comment.May 24 2018, 9:55 AM

Mentioned in SAL (#wikimedia-operations) [2018-05-24T09:55:32Z] <mobrovac@tin> Synchronized wmf-config/InitialiseSettings.php: Switch all jobs to EventBus for everything except wikipedia, commons and wikidata, file 2/2 - T190327 (duration: 01m 06s)

gerritbot added a comment.May 24 2018, 9:58 AM

Change 434891 had a related patch set uploaded (by Ppchelko; owner: Ppchelko):
[operations/mediawiki-config@master] Switch all job apart from exceptions for everything.

https://gerrit.wikimedia.org/r/434891

gerritbot added a comment.May 24 2018, 10:05 AM

Change 434893 had a related patch set uploaded (by Ppchelko; owner: Ppchelko):
[mediawiki/services/change-propagation/jobqueue-deploy@master] Enable all jobs apart from exceptions for everything.

https://gerrit.wikimedia.org/r/434893

Legoktm mentioned this in T195500: Global mass message delivered on meta but not on other wikis?.May 28 2018, 11:36 PM
• mobrovac added a subtask: T195500: Global mass message delivered on meta but not on other wikis?.May 29 2018, 9:40 AM
gerritbot added a comment.May 29 2018, 9:58 AM

Change 434893 merged by Mobrovac:
[mediawiki/services/change-propagation/jobqueue-deploy@master] Enable all jobs apart from exceptions for everything.

https://gerrit.wikimedia.org/r/434893

gerritbot added a comment.May 29 2018, 10:02 AM

Change 434891 merged by jenkins-bot:
[operations/mediawiki-config@master] Switch all job apart from exceptions for everything.

https://gerrit.wikimedia.org/r/434891

Stashbot added a comment.May 29 2018, 10:04 AM

Mentioned in SAL (#wikimedia-operations) [2018-05-29T10:04:48Z] <ppchelko@tin> Started deploy [cpjobqueue/deploy@c6dc83d]: Enable all jobs apart from exceptions for everything. T190327

Stashbot added a comment.May 29 2018, 10:05 AM

Mentioned in SAL (#wikimedia-operations) [2018-05-29T10:05:46Z] <ppchelko@tin> Finished deploy [cpjobqueue/deploy@c6dc83d]: Enable all jobs apart from exceptions for everything. T190327 (duration: 00m 58s)

Stashbot added a comment.May 29 2018, 10:06 AM

Mentioned in SAL (#wikimedia-operations) [2018-05-29T10:06:26Z] <mobrovac@tin> Synchronized wmf-config/jobqueue.php: Switch all jobs to EventBus file 1/2 - T190327 T195500 (duration: 01m 39s)

Stashbot added a comment.May 29 2018, 10:09 AM

Mentioned in SAL (#wikimedia-operations) [2018-05-29T10:09:33Z] <mobrovac@tin> Synchronized wmf-config/InitialiseSettings.php: Switch all jobs to EventBus file 2/2 - T190327 T195500 (duration: 01m 47s)

• mobrovac closed subtask T195500: Global mass message delivered on meta but not on other wikis? as Resolved.May 29 2018, 10:11 AM
Anomie subscribed.May 29 2018, 12:08 PM

Did this break logging of jobs to runJobs.log on mwlog1001, and detection of jobs via maintenance/showJobs.php?

I was trying to look into T195397: {{PAGESINCATEGORY}} returns incorrect value on en-wiki Category:Candidates for speedy deletion and couldn't find any record of links update jobs running at all.

• Pchelolo added a comment.May 29 2018, 12:13 PM

@Anomie

Did this break logging of jobs to runJobs.log on mwlog1001

The runJobs.log contains the logs for the old queue. The kafka-based queue logs can be found either in JobExecutor.log file on mwlog, or there's a more comprehensive logstash dashboard that contains all the logs related to the new system.

And detection of jobs via maintenance/showJobs.php?

That one still needs to be updated to fetch things from kafka in case the kafka queue is used.

Anomie added a comment.May 29 2018, 2:58 PM
In T190327#4238643, @Pchelolo wrote:

The kafka-based queue logs can be found either in JobExecutor.log file on mwlog,

That appears to contain only errors.

or there's a more comprehensive logstash dashboard that contains all the logs related to the new system.

That link takes me to a dashboard that does no filtering at all, so it shows every log message whether job-related or not.

• Pchelolo added a comment.May 29 2018, 3:04 PM

That link takes me to a dashboard that does no filtering at all, so it shows every log message whether job-related or not.

That's weird, perhaps a copy-paste error, or logstash bug. Here's a new link: https://logstash.wikimedia.org/goto/ccd5e2517591489ad88bc66922f8311c

That appears to contain only errors.

Created T195858 to make it DEBUG-log same as the JobRunner did.

Anomie added a comment.May 29 2018, 4:10 PM

The new link works. Although I don't see any messages in there about jobs being run, just errors and messages about jobs being deduplicated. And an occasional "Processed event sample" with no indication of what that means.

• Pchelolo added a comment.May 29 2018, 4:11 PM

The new link works. Although I don't see any messages in there about jobs being run, just errors and messages about jobs being deduplicated. And an occasional "Processed event sample" with no indication of what that means.

Ye, we don't send all the jobs into logstash, that would be overwhelming for it. I'm adding debug logging about jobs being run to log into mwlog1001. Will be done soon.

gerritbot added a comment.May 31 2018, 4:00 PM

Change 436574 had a related patch set uploaded (by Ppchelko; owner: Ppchelko):
[operations/puppet@production] Remove unused jobrunners.

https://gerrit.wikimedia.org/r/436574

Anomie unsubscribed.May 31 2018, 4:05 PM
Joe closed subtask T188947: Create an LVS endpoint for jobrunners on videoscalers as Resolved.Jun 4 2018, 3:10 PM
gerritbot added a comment.Jun 4 2018, 4:01 PM

Change 437281 had a related patch set uploaded (by Ppchelko; owner: Ppchelko):
[operations/puppet@production] Specify videoscalers uri in hiera/changeprop manifest.

https://gerrit.wikimedia.org/r/437281

gerritbot added a comment.Jun 4 2018, 4:26 PM

Change 437285 had a related patch set uploaded (by Ppchelko; owner: Ppchelko):
[mediawiki/services/change-propagation/jobqueue-deploy@master] Enable videoscaler jobs in the new queue.

https://gerrit.wikimedia.org/r/437285

gerritbot added a comment.Jun 4 2018, 4:29 PM

Change 437286 had a related patch set uploaded (by Ppchelko; owner: Ppchelko):
[operations/mediawiki-config@master] Disable redis queue for videoscaler jobs.

https://gerrit.wikimedia.org/r/437286

gerritbot added a comment.Jun 5 2018, 8:38 AM

Change 437281 merged by Giuseppe Lavagetto:
[operations/puppet@production] Specify videoscalers uri in hiera/changeprop manifest.

https://gerrit.wikimedia.org/r/437281

gerritbot added a comment.Jun 5 2018, 8:42 AM

Change 437285 merged by Ppchelko:
[mediawiki/services/change-propagation/jobqueue-deploy@master] Enable videoscaler jobs in the new queue.

https://gerrit.wikimedia.org/r/437285

gerritbot added a comment.Jun 5 2018, 8:49 AM

Change 437286 merged by jenkins-bot:
[operations/mediawiki-config@master] Disable redis queue for videoscaler jobs.

https://gerrit.wikimedia.org/r/437286

Stashbot added a comment.Jun 5 2018, 8:51 AM

Mentioned in SAL (#wikimedia-operations) [2018-06-05T08:51:13Z] <ppchelko@deploy1001> Started deploy [cpjobqueue/deploy@63b30a6]: Enable videoscaler jobs in kafka T190327

Stashbot added a comment.Jun 5 2018, 8:52 AM

Mentioned in SAL (#wikimedia-operations) [2018-06-05T08:52:02Z] <ppchelko@deploy1001> Finished deploy [cpjobqueue/deploy@63b30a6]: Enable videoscaler jobs in kafka T190327 (duration: 00m 49s)

Stashbot added a comment.Jun 5 2018, 8:52 AM

Mentioned in SAL (#wikimedia-operations) [2018-06-05T08:52:07Z] <mobrovac@deploy1001> Synchronized wmf-config/jobqueue.php: Switch video scaling jobs to EventBus - T190327 (duration: 00m 52s)

gerritbot added a comment.Jun 5 2018, 10:45 AM

Change 436574 merged by Giuseppe Lavagetto:
[operations/puppet@production] Remove unused jobrunners.

https://gerrit.wikimedia.org/r/436574

Stashbot added a comment.Jun 5 2018, 11:21 AM

Mentioned in SAL (#wikimedia-operations) [2018-06-05T11:21:05Z] <ppchelko@deploy1001> Started deploy [cpjobqueue/deploy@aa5e94b]: Enable cirrus jobs in kafka for everything except wikipedia, wikidata and commons T190327

Stashbot added a comment.Jun 5 2018, 11:21 AM

Mentioned in SAL (#wikimedia-operations) [2018-06-05T11:21:46Z] <ppchelko@deploy1001> Finished deploy [cpjobqueue/deploy@aa5e94b]: Enable cirrus jobs in kafka for everything except wikipedia, wikidata and commons T190327 (duration: 00m 41s)

• Pchelolo closed subtask T175210: Select candidate jobs for transferring to the new infrastucture as Resolved.Jun 5 2018, 11:41 AM
Stashbot added a comment.Jun 6 2018, 12:22 PM

Mentioned in SAL (#wikimedia-operations) [2018-06-06T12:22:14Z] <ppchelko@deploy1001> Started deploy [cpjobqueue/deploy@c8d62da]: Enable cirrus for everything T190327

Stashbot added a comment.Jun 6 2018, 12:23 PM

Mentioned in SAL (#wikimedia-operations) [2018-06-06T12:23:07Z] <ppchelko@deploy1001> Finished deploy [cpjobqueue/deploy@c8d62da]: Enable cirrus for everything T190327 (duration: 00m 47s)

• mobrovac closed subtask T196076: ORESFetchScoreJob fails quite a lot as Resolved.Jun 6 2018, 12:34 PM
• mobrovac closed subtask T195858: Make JobExecutor debug-log to mwlog as Resolved.
gerritbot added a comment.Jun 6 2018, 2:24 PM

Change 437767 had a related patch set uploaded (by Ppchelko; owner: Ppchelko):
[operations/mediawiki-config@master] Switch all jobs to the new queue and clean up the old queue configs.

https://gerrit.wikimedia.org/r/437767

• Pchelolo closed subtask T192946: Make gwtoolsetUploadMediafileJob JSON-serializable as Resolved.Jun 12 2018, 10:20 AM
fgiunchedi closed subtask T196067: Clean up cpjobqueue metrics as Resolved.Jun 15 2018, 9:49 AM
• Pchelolo closed subtask T192945: Make EchoNotification job JSON-serializable as Resolved.Jun 25 2018, 2:19 PM
gerritbot added a comment.Jun 26 2018, 8:42 AM

Change 437767 merged by Mobrovac:
[operations/mediawiki-config@master] Switch all jobs to the new queue and clean up the old queue configs.

https://gerrit.wikimedia.org/r/437767

Stashbot added a comment.Jun 26 2018, 8:46 AM

Mentioned in SAL (#wikimedia-operations) [2018-06-26T08:46:33Z] <ppchelko@deploy1001> Started deploy [cpjobqueue/deploy@7d9a1aa]: Enable all jobs in kafka queue T190327

• Pchelolo closed subtask T189137: Migrate CirrusSearch jobs to Kafka queue as Resolved.Jun 26 2018, 8:47 AM
Stashbot added a comment.Jun 26 2018, 8:47 AM

Mentioned in SAL (#wikimedia-operations) [2018-06-26T08:47:31Z] <ppchelko@deploy1001> Finished deploy [cpjobqueue/deploy@7d9a1aa]: Enable all jobs in kafka queue T190327 (duration: 00m 58s)

Stashbot added a comment.Jun 26 2018, 8:47 AM

Mentioned in SAL (#wikimedia-operations) [2018-06-26T08:47:33Z] <mobrovac@deploy1001> Synchronized wmf-config/CommonSettings.php: Switch the last remaining jobs to EventBus, only CommonSettings.php now - T190327 (duration: 00m 58s)

Stashbot added a comment.Jun 26 2018, 8:48 AM

Mentioned in SAL (#wikimedia-operations) [2018-06-26T08:48:07Z] <mobrovac@deploy1001> Started scap: Switch the last remaining jobs to EventBus, full scap sync for clean-up - T190327

Stashbot added a comment.Jun 26 2018, 9:03 AM

Mentioned in SAL (#wikimedia-operations) [2018-06-26T09:03:12Z] <mobrovac@deploy1001> Started scap: Switch the last remaining jobs to EventBus, full scap sync, take #2 - T190327

Stashbot added a comment.Jun 26 2018, 9:11 AM

Mentioned in SAL (#wikimedia-operations) [2018-06-26T09:11:47Z] <mobrovac@deploy1001> Finished scap: Switch the last remaining jobs to EventBus, full scap sync, take #2 - T190327 (duration: 08m 34s)

• mobrovac removed a subtask: T182274: Create custom per-job metric reporters capability.Jun 26 2018, 9:22 AM
Stashbot added a comment.Jun 26 2018, 9:28 AM

Mentioned in SAL (#wikimedia-operations) [2018-06-26T09:28:28Z] <mobrovac@deploy1001> Started scap: Switch the last remaining jobs to EventBus, full scap sync with HHVM restarts, take #3 - T190327

Stashbot added a comment.Jun 26 2018, 9:33 AM

Mentioned in SAL (#wikimedia-operations) [2018-06-26T09:33:36Z] <mobrovac@deploy1001> Finished scap: Switch the last remaining jobs to EventBus, full scap sync with HHVM restarts, take #3 - T190327 (duration: 05m 07s)

• mobrovac removed a subtask: T198220: Stop and remove old job runners.Jun 26 2018, 2:43 PM
• mobrovac closed this task as Resolved.Jun 29 2018, 3:41 PM
• mobrovac assigned this task to • Pchelolo.
• mobrovac edited projects, added Services (done); removed Patch-For-Review, Services (doing).

We are done, mesdames et messieurs. \o/ \o/ \o/

Eevans awarded a token.Jun 29 2018, 3:52 PM
• Vvjjkkii reopened subtask T196067: Clean up cpjobqueue metrics as Open.Jul 1 2018, 1:06 AM
• Vvjjkkii reopened subtask T196076: ORESFetchScoreJob fails quite a lot as Open.
• Vvjjkkii reopened subtask T195858: Make JobExecutor debug-log to mwlog as Open.
• Vvjjkkii reopened subtask T195500: Global mass message delivered on meta but not on other wikis? as Open.
• Vvjjkkii reopened subtask T193471: JobQueueGroup's singletons using the wrong wgJobTypeConf as Open.Jul 1 2018, 1:12 AM
• Vvjjkkii reopened subtask T193254: Global renames get stuck at metawiki as Open.
• Vvjjkkii reopened subtask T192946: Make gwtoolsetUploadMediafileJob JSON-serializable as Open.
• Vvjjkkii reopened subtask T192945: Make EchoNotification job JSON-serializable as Open.
1997kB closed subtask T193254: Global renames get stuck at metawiki as Resolved.Jul 1 2018, 2:41 AM
1997kB closed subtask T193471: JobQueueGroup's singletons using the wrong wgJobTypeConf as Invalid.Jul 1 2018, 2:47 AM
• Community_Tech_bot closed subtask T196076: ORESFetchScoreJob fails quite a lot as Resolved.Jul 1 2018, 6:21 AM
Mainframe98 closed subtask T192945: Make EchoNotification job JSON-serializable as Resolved.Jul 1 2018, 7:54 AM
Mainframe98 closed subtask T192946: Make gwtoolsetUploadMediafileJob JSON-serializable as Resolved.Jul 1 2018, 8:01 AM
• mobrovac reopened subtask T196076: ORESFetchScoreJob fails quite a lot as Open.Jul 1 2018, 9:48 AM
• mobrovac closed subtask T195500: Global mass message delivered on meta but not on other wikis? as Resolved.Jul 1 2018, 11:05 AM
• mobrovac closed subtask T195858: Make JobExecutor debug-log to mwlog as Resolved.Jul 1 2018, 11:11 AM
• mobrovac closed subtask T196067: Clean up cpjobqueue metrics as Resolved.Jul 1 2018, 11:17 AM
CommunityTechBot closed subtask T196076: ORESFetchScoreJob fails quite a lot as Resolved.Jul 3 2018, 3:25 AM
• mobrovac reopened subtask T196076: ORESFetchScoreJob fails quite a lot as Open.Jul 3 2018, 1:00 PM
• mobrovac closed subtask T196076: ORESFetchScoreJob fails quite a lot as Resolved.Jul 5 2018, 5:53 AM
Aklapper edited projects, added Analytics-Radar; removed Analytics.Jun 10 2020, 6:44 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL