Page MenuHomePhabricator

Shorten/Simplify MW train deploy cadence to Tu->W->Th
Closed, ResolvedPublic

Description

Current situation:
cadence is at: https://wikitech.wikimedia.org/w/index.php?title=Deployments/One_week&oldid=156468

  • Worst case to get code to English Wikipedia (merging code into master right after a new branch cut): 14 days
  • Best case to get code to enwiki (merging right before new branch cut): 7 days

Proposal:
Make the group0 -> 1 -> 2 days line up with Tues -> Wed -> Thu

  1. Tuesday branch cut - master gets branched and then deployed to group0 (Mediawiki.org and test wikis)
  2. Wednesday the branch gets deployed to group1 (everything but wikipedia)
  3. Thursday the branch gets deployed everywhere.

Time to get code to English Wikipedia:

  • Worst case: 10 days
  • Best case: 3 days

Drawbacks of proposal:

  • shortened testing period for new code on production before hitting enwiki (from 7 days to 3 days)
  • 3 deployment windows instead of 2

Benefits of proposal:

  • code gets to users faster
  • process is simpler/easier to understand

Event Timeline

greg created this task.Apr 29 2015, 6:00 PM
greg raised the priority of this task from to Normal.
greg updated the task description. (Show Details)
greg added a project: Release-Engineering-Team.
greg added subscribers: greg, mmodell, Jdlrobson, demon.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 29 2015, 6:00 PM
greg added a comment.Apr 29 2015, 6:02 PM

(NB: This proposal is basically how FB does their deploy cadence, other than how the gradual rollout happens. Essentially weekly starting on Monday.)

greg renamed this task from Shorten/Simply MW train deploy cadence to M->Tu->W to Shorten/Simplify MW train deploy cadence to M->Tu->W.Apr 29 2015, 6:06 PM
greg set Security to None.
demon added a comment.Apr 29 2015, 6:16 PM

If we did this we should automate the cutting (and testing) of the new branch like Sunday night.

Why MTuW instead of TuWTh? (deploying on monday means you're rushed to fix any bugs that might have been discovered over the weekend)

greg added a comment.Apr 29 2015, 6:28 PM

If we did this we should automate the cutting (and testing) of the new branch like Sunday night.

+1

Why MTuW instead of TuWTh? (deploying on monday means you're rushed to fix any bugs that might have been discovered over the weekend)

Not a bad point.

mmodell added a subscriber: aude.Apr 29 2015, 6:54 PM

We really need to automate the branching stuff anyway - it's really time consuming and error prone. the way it is now wastes not just my time but @aude's and anyone else who works on extensions regularly.

There is no sane reason to cut new deployment branches manually every week. In many cases nothing even changes between branches.

I'd also suggest TuWTh so that the preponderance of holiday Mondays doesn't massively disrupt schedules. (As well as @demon's, @mmodell's and @Legoktm's points.)

greg renamed this task from Shorten/Simplify MW train deploy cadence to M->Tu->W to Shorten/Simplify MW train deploy cadence to Tu->W->Th.Apr 29 2015, 6:58 PM
greg updated the task description. (Show Details)

I'm confused at how the best case would be 3 days but worst case 10 days?

greg added a comment.Apr 29 2015, 7:41 PM

I'm confused at how the best case would be 3 days but worst case 10 days?

If you merge into master right after the new branch which happens on say Tuesday May 5th. You're commit will go out in the next branch which is cut on Tuesday May 12th, which goes to English Wikipedia on Thu May 14th. 9 or 10 days depending on how you count Tuesday the 5th.

If you merge right before the branch cut on Tuesday May 5th then it'll go to English Wikipedia on Thur May 7th. 2 or 3 days depending on how you count.

This comment was removed by mmodell.
Tgr added a subscriber: Tgr.Apr 30 2015, 2:30 PM
greg added a comment.May 24 2015, 7:47 AM

@mmodell: The person this most affects day-to-day is you. I'm fine with making the change (with giving people ~2 weeks of warning) at any point now. What are your thoughts on that?

@greg: I'm ok with it. I'll have to adjust my scripts a bit. Will we use the same time slot for the Thursday deployment window as we do on Tuesday and Wednesday?

greg added a comment.May 25 2015, 9:49 AM

@mmodell yeah, 11am on all those days. Shall we decree to start doing this the week of .... /me looks at a calendar....June 8th?

ok

So cut new branch and deploy to group 0 on tuseday, deploy to group 1 on wednesday, deploy to everything else on thursday, rinse and repeat?

mmodell updated the task description. (Show Details)May 25 2015, 3:28 PM

I updated the description to be a little more clear. I'll add it to my calendar for the 8th.

Florian added a subscriber: Florian.
Elitre added a subscriber: Elitre.May 27 2015, 9:31 PM
bd808 added a subscriber: bd808.May 28 2015, 4:05 PM
hashar assigned this task to greg.May 29 2015, 10:35 AM
hashar added a subscriber: hashar.

Seems @greg is taking the lead on this. He announced the new cadence at Wed May 27 20:19:38 UTC 2015 on https://lists.wikimedia.org/pipermail/wikitech-l/2015-May/081863.html

I have troubles understanding the word "simplify". What's simpler? The difference from quickest to slowest, which is currently 100 %, will become 233 %. Therefore people will need to check their calendars and calculate time intervals much more carefully.

greg added a comment.Jun 1 2015, 5:21 PM

I have added T437: Track state of browser tests before new wmfXX branch cut as a blocker since we want to run smoke browser tests at some point.

This isn't a blocker for this change, but is still something we should do.

greg closed this task as Resolved.Jun 1 2015, 5:21 PM

This is announced, scheduled, and on it's way for next week. Resolving now.

In the announcement email, @greg wrote:

Transition

Transitions from one cadence to another are hard. Here's how we'll be
doing this transition:
Week of June 1st (next week):

  • We'll complete the wmf8 rollout on June 3rd
  • However, we won't be cutting wmf9 on June 3rd

Week of June 8th (in two weeks):

  • We'll begin the new cadence with wmf9 on Tuesday June 9th

So I won't be branching wmf9 today.

Change 215679 had a related patch set uploaded (by 20after4):
All wikis to 1.26wmf8, no new branch today per T97553

https://gerrit.wikimedia.org/r/215679

Change 215679 merged by jenkins-bot:
All wikis to 1.26wmf8, no new branch today per T97553

https://gerrit.wikimedia.org/r/215679