Page MenuHomePhabricator
Create Task
Maniphest T209587

Migrate ores celery configs to celery 4
Closed, ResolvedPublic
Actions

Description

Right now, celery 4 supports upper case setting names but it's officially deprecated and we need to use lower case settings and they will be dropped soon. See http://docs.celeryproject.org/en/v4.1.0/whatsnew-4.0.html#v400-upgrade-settings
One big reason is that you can't mix old and new configs so if we want to add settings for sentinel (added in celery4), it might gets complicated. Also since some of our configs come from puppet and some from the config files. There is no easy way to handle this. The only thing I can think of is to move all of configs to puppet, deploy, apply the changes to puppet and then re-add migrated configs to ores repos and remove them from puppet.

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
ores: labs celery4 config updateoperations/puppetproduction+8 -8
Followup fix for ores: Change configs to celery4operations/puppetproduction+2 -2
ores: Change configs to celery4 onesoperations/puppetproduction+13 -13
Bump ORES to HEADmediawiki/services/ores/deploymaster+1 -10
ores: Move all of celery configs to puppetoperations/puppetproduction+15 -6
ores: Drop old b/c configsoperations/puppetproduction+0 -14
Customize query in gerrit

Related Objects
Search...

Event Timeline

Ladsgroup created this task.Nov 15 2018, 2:16 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 15 2018, 2:16 PM
Ladsgroup claimed this task.Nov 15 2018, 6:28 PM
Restricted Application added a project: User-Ladsgroup. · View Herald TranscriptNov 15 2018, 6:28 PM
gerritbot subscribed.Nov 16 2018, 11:25 AM

Change 474157 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[operations/puppet@production] ores: Drop old b/c configs

https://gerrit.wikimedia.org/r/474157

gerritbot added a project: Patch-For-Review.Nov 16 2018, 11:25 AM
gerritbot added a comment.Nov 16 2018, 11:54 AM

Change 474158 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[operations/puppet@production] ores: Move all of celery configs to puppet

https://gerrit.wikimedia.org/r/474158

gerritbot added a comment.Nov 16 2018, 12:06 PM

Change 474157 merged by Alexandros Kosiaris:
[operations/puppet@production] ores: Drop old b/c configs

https://gerrit.wikimedia.org/r/474157

Ladsgroup mentioned this in rORES35fc735f06c9: Remove all of celery configs.Nov 16 2018, 6:25 PM
Ladsgroup mentioned this in rORES0fcfc3a5e8c6: Remove all of celery configs.Nov 16 2018, 6:40 PM
gerritbot added a comment.Nov 19 2018, 1:34 PM

Change 474158 merged by Alexandros Kosiaris:
[operations/puppet@production] ores: Move all of celery configs to puppet

https://gerrit.wikimedia.org/r/474158

gerritbot added a comment.Nov 19 2018, 2:32 PM

Change 474694 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[operations/puppet@production] ores: Change configs to celery4 ones

https://gerrit.wikimedia.org/r/474694

gerritbot added a comment.Nov 19 2018, 3:02 PM

Change 474703 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[mediawiki/services/ores/deploy@master] Bump ORES to HEAD

https://gerrit.wikimedia.org/r/474703

Ladsgroup added a comment.Nov 19 2018, 3:04 PM

https://github.com/wikimedia/ores/pull/291

gerritbot added a comment.Nov 19 2018, 3:05 PM

Change 474703 merged by Ladsgroup:
[mediawiki/services/ores/deploy@master] Bump ORES to HEAD

https://gerrit.wikimedia.org/r/474703

Ladsgroup mentioned this in rORESDEPLOYe957b244afaf: Bump ORES to HEAD.Nov 19 2018, 3:05 PM
Ladsgroup mentioned this in rORESefcf36df6f5e: Bring back celery configs.
Stashbot subscribed.Nov 19 2018, 3:28 PM

Mentioned in SAL (#wikimedia-operations) [2018-11-19T15:28:24Z] <ladsgroup@deploy1001> Started deploy [ores/deploy@e957b24]: T209587 T170950

Stashbot mentioned this in T170950: ORES deployment finish "successfully" even when uwsgi and celery fail to successfully start up.Nov 19 2018, 3:28 PM
Ladsgroup added a parent task: T178441: Upgrade celery to 4.1.0 for ORES.Nov 19 2018, 3:31 PM
Stashbot added a comment.Nov 19 2018, 3:45 PM

Mentioned in SAL (#wikimedia-operations) [2018-11-19T15:45:33Z] <ladsgroup@deploy1001> Finished deploy [ores/deploy@e957b24]: T209587 T170950 (duration: 17m 09s)

Ladsgroup moved this task from Parked to Review on the Machine-Learning-Team (Active Tasks) board.Nov 19 2018, 5:21 PM
gerritbot added a comment.Nov 21 2018, 11:13 AM

Change 474694 merged by Alexandros Kosiaris:
[operations/puppet@production] ores: Change configs to celery4 ones

https://gerrit.wikimedia.org/r/474694

gerritbot added a comment.Nov 21 2018, 11:39 AM

Change 475084 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/puppet@production] Followup fix for ores: Change configs to celery4

https://gerrit.wikimedia.org/r/475084

gerritbot added a comment.Nov 21 2018, 11:45 AM

Change 475084 merged by Alexandros Kosiaris:
[operations/puppet@production] Followup fix for ores: Change configs to celery4

https://gerrit.wikimedia.org/r/475084

Ladsgroup moved this task from Review to Completed on the Machine-Learning-Team (Active Tasks) board.Nov 21 2018, 12:25 PM
gerritbot added a comment.Nov 28 2018, 12:17 PM

Change 476255 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[operations/puppet@production] ores: labs celery4 config update

https://gerrit.wikimedia.org/r/476255

gerritbot added a comment.Nov 28 2018, 12:32 PM

Change 476255 merged by Alexandros Kosiaris:
[operations/puppet@production] ores: labs celery4 config update

https://gerrit.wikimedia.org/r/476255

Ladsgroup closed this task as Resolved.Dec 17 2018, 12:39 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits