Page MenuHomePhabricator

Automate the recurring management of wikitech:Deployments and phab:#train_deployments
Closed, ResolvedPublic

Description

What (all) it would need to do:

Deployment calendar

  • Have a config somewhere for "what does the basic week skeleton look like"
    • This can live in code or on wiki, I don't care
  • Add that skeleton for two weeks out on Friday
    • eg: on Fri Oct 2nd 2015, add the skeleton for the week of Oct 12th
  • Archive previous weeks on Sunday night
    • eg: archive content for the week of Sept 28th, 2015 on October 4th
    • Archive layout currently is: wikitech:Deployments/Archive/YYYY/MM (with each week keeping it's heading level (===) on that month)

Deployment Blockers

Batch creation / editing of the task series which we use for tracking release-blocker bugs with each weekly deployment train. These tasks can be seen in the Train Deployments project and they are annoying to maintain manually. Automation will enable the following conveniences:

  • Batch creation and editing of these tasks instead of spending 10 minutes on manual copy/paste every couple of weeks.
  • Generate standardized markup for the task descriptions from a text file with template markup plus a few variables which are computed dynamically for each weekly deployment
    • Compute the week beginning day for "The week of $date"
    • Link each task to the previous and next task within a series which is difficult to do currently due to a circular dependency: you have to know the task ID of next week's task before you can link to it. So we currently are forced to edit each task twice in order to get them all linked up properly. Automation will eliminate this problem.

Tags and Milestones

We have at least two more periodic objects in phabricator which are managed somewhat manually:

MediaWiki Release Notes

These are series' of tags and milestones that also represent each weekly mediawiki train iteration and are grouped within each major mediawiki version, e.g. MW-1.30-release-notes. AFAIK these are manually created by @Jdforrester-WMF

Phabricator Release Notes

These are milestones within the Phabricator project which are used to track merges from upstream phabricator when they are deployed to Wikimedia's fork of Phabricator. An example can be seen in Phabricator (2017-06-01). These are maintained by myself (@mmodell) and I have already automated their creation using the same code which I intend to apply to the other processes mentioned above.

Event Timeline

greg raised the priority of this task from to Medium.
greg updated the task description. (Show Details)
greg added a project: Release-Engineering-Team.
greg moved this task to INBOX on the Release-Engineering-Team board.
greg added a subscriber: greg.

@mmodell is working on something like this.

mmodell raised the priority of this task from Medium to High.Jun 7 2017, 5:33 PM
mmodell renamed this task from Bot to handle recurring tasks to manage wikitech:Deployments to Bot to handle recurring tasks to manage wikitech:Deployments and phab:#train_deployments.Jun 7 2017, 6:50 PM
mmodell updated the task description. (Show Details)
mmodell added a subscriber: Jdforrester-WMF.
mmodell renamed this task from Bot to handle recurring tasks to manage wikitech:Deployments and phab:#train_deployments to Automate the recurring management of wikitech:Deployments and phab:#train_deployments.Jun 23 2017, 4:29 AM
mmodell lowered the priority of this task from High to Medium.
mmodell raised the priority of this task from Medium to High.Jul 5 2017, 11:52 PM

This is still progressing: Current status is I got down a rabbit hole learning lua + scribunto.

mmodell lowered the priority of this task from High to Medium.Aug 7 2017, 4:39 PM

It's been proposed to use phabricator events for SWATs and that's really the way I'd like to do it. If we do it that way I can write code to periodically update the markup for the deployments wiki page.

@greg ^ how do you feel about doing a trial to evaluate whether Phabricator events will work well for this? I suspect that it will work well since it's worked reasonably well for organizing the code review office hours (which has a format that is somewhat similar to SWATs)

mmodell raised the priority of this task from Medium to High.Aug 28 2017, 7:26 AM

@greg ^ how do you feel about doing a trial to evaluate whether Phabricator events will work well for this? I suspect that it will work well since it's worked reasonably well for organizing the code review office hours (which has a format that is somewhat similar to SWATs)

Just reiterating what I said in our team meeting this morning so it's on record :)

Let's (read: Mukunda ;) ) do a mock-up of an example SWAT window in the propose system; see how it works etc.

Not finished yet but here's an example: {E705}

The way I'd imagine it is having a section at the top of the page which collects together any gerrit URLs or git hashes which are mentioned in the comments for the event.

E673 is actually a good example, just imagine having the table in the description update automatically.

mmodell updated the task description. (Show Details)

Status update: This should finally get some more attention this week.

No progress per-se as I was shaving yaks all week.

mmodell lowered the priority of this task from High to Medium.Oct 11 2017, 4:38 PM
mmodell moved this task from In-progress to Backlog on the Release-Engineering-Team (Kanban) board.

This has unfortunately been on the back burner as I've been busy with scap and other phabricator work.

greg removed mmodell as the assignee of this task.Oct 23 2017, 4:56 PM
greg assigned this task to mmodell.
mmodell lowered the priority of this task from Medium to Low.Jan 11 2018, 11:03 PM
mmodell changed the task status from Open to Stalled.Mar 29 2018, 6:49 PM
Alroilim set Due Date to Feb 1 2019, 9:00 PM.
Alroilim updated the task description. (Show Details)
Restricted Application changed the subtype of this task from "Task" to "Deadline". · View Herald TranscriptFeb 2 2019, 7:15 PM
Aklapper removed Due Date.
Aklapper updated the task description. (Show Details)
Restricted Application changed the subtype of this task from "Deadline" to "Task". · View Herald TranscriptFeb 23 2019, 6:12 AM

New idea: watch the gerrit stream-events feed and update the wiki page whenever a hashtag is added to a patch in gerrit.

mmodell changed the task status from Stalled to Open.Jul 30 2019, 5:59 PM
mmodell moved this task from Working on it to Soon on the User-MModell board.

Change 530006 had a related patch set uploaded (by 20after4; owner: Thcipriani):
[mediawiki/tools/release@master] make-deployment-calendar: Create wikitech:Deployments

https://gerrit.wikimedia.org/r/530006

Change 530006 merged by jenkins-bot:
[mediawiki/tools/release@master] make-deployment-calendar: Create wikitech:Deployments

https://gerrit.wikimedia.org/r/530006

I made a script to generate the calendar from the train task: https://github.com/wikimedia/mediawiki-tools-release/tree/master/make-deployment-calendar#readme-for-make-deployments-calendar

It's cron'd to run via the toolforge gridengine weekly every Saturday at 2am UTC: https://admin.toolforge.org/tool/deployment-calendar

MediaWiki notes have been automated a while. I still manually create train tasks. I think that's fine, the train tasks are the canonical source of information.

Calling this done from my perspective.