Page MenuHomePhabricator

Streamline MW security release process
Open, MediumPublic

Description

From Chad's "I'm passing the torch on MW releases" parting thought on https://gerrit.wikimedia.org/r/#/c/437867/ :

I think we should *stop* the process now wherein we generate tarballs *before* we push the patches to Gerrit. I think the proper order should be

Fix in production
Fix in master & supported branches
Tag new releases
Send the announcement e-mail with links to
* The patches themselves / the tags
* The soon-to-be-live tarballs
Tarball

It's the most streamlined process, and would allow us to get to a point of one security release per patch. This has the triple benefits of making the upgrade delta easier to understand, keeping production patches to a minimum. and finally getting rid of the GIANT backlog of stuff we throw at Zuul/Jenkins all at once when we do multiple patches in a release. With the simplification of the build process in this overall change (drop composer, vendor as submodule, removing the entire patch-an-untagged-release crap) there's no reason the script can't be run quickly (nearly automatic, I'd say) to churn out the tarballs *minutes* from being announced.

I'd kinda hinted to folks this is where I was taking things, but I don't think I've laid them out as such yet. 1.31 is clearly the last release with my name on it, but here's where I was gonna go with 1.32 and beyond and I'd like to see someone take up the torch.

@JBennett / @Reedy: Thoughts?

Event Timeline

greg created this task.Jun 7 2018, 6:23 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 7 2018, 6:23 AM
greg added a comment.Jun 7 2018, 9:20 PM
20:53:41 no_justification | https://github.com/Kentzo/git-archive-all                                                                                     
20:53:43 no_justification | BAM                                                                                                                           
20:53:59          legoktm | oooh                                                                                                                          
20:54:25 no_justification | So then, our exclusion lists go into .gitattributes (which allows bundled extensions to exclude extra things without having to
                          | maintain a big meta-list)                                                                                                     
20:55:04 no_justification | And release becomes:                                                                                                          
20:55:04 no_justification | 1) Clone recursively                                                                                                          
20:55:04 no_justification | 2) `git-archive-all`                                                                                                          
20:55:04 no_justification | 3) Sign the releases                                                                                                          
20:55:09 no_justification | Is it really this easy?
Vvjjkkii renamed this task from Streamline MW security release process to 5hbaaaaaaa.Jul 1 2018, 1:05 AM
Vvjjkkii triaged this task as High priority.
Vvjjkkii updated the task description. (Show Details)
Vvjjkkii removed a subscriber: Aklapper.
CommunityTechBot renamed this task from 5hbaaaaaaa to Streamline MW security release process.Jul 2 2018, 3:13 PM
CommunityTechBot raised the priority of this task from High to Needs Triage.
CommunityTechBot updated the task description. (Show Details)
CommunityTechBot added a subscriber: Aklapper.
Reedy added a comment.Jul 7 2018, 5:15 PM

Should we be still sending a pre-announce email so people are aware? I'm guessing ~24H before pushing the fixes to master/branches?

Fix in production
Send pre-release announcement
Fix in master & supported branches
Tag new releases
Send the announcement e-mail with links to

  • The patches themselves / the tags
  • The soon-to-be-live tarballs

Tarball

Paladox added a subscriber: Paladox.Jul 7 2018, 5:23 PM

git-archive-all looks extremely promising from my testing so far. I was able to recreate a functionally identical tarball from the 1.31.0 git tag. I'll split my notes/suggestions w/r to that into a separate task soon.

I don't think we should move to a model of pushing to git before releasing the tarball, mostly because I don't think it actually fixes the problems of lack of CI before a release. I think the pieces are mostly in place now (quibble/docker) that we can run proper CI against security patches.

There are also some older notes from T156445: Streamline/automate MW tarball security release process that need to be merged into any future planning/release work.

I think the most relevant comment/suggestion to go forwards is still T156445#3088982.

chasemp triaged this task as Medium priority.Dec 9 2019, 5:01 PM
chasemp added a project: Security-Team.