Page MenuHomePhabricator

Design m8s deployment workflows and tooling
Open, Needs TriagePublic

Description

Now that we have jobs capable of building deployable images complete with multiple versions of MediaWiki, configuration, security patches, private settings, we need to deploy them to both k8s and legacy servers. Let's sketch out what that could look like from a deployer's perspective.

We (@dancy, @jeena, and @dduvall) brainstormed during our meeting today and came up with the following. It focuses on backport deployments. We also need to define commands/workflows for security patch updates, private settings updates, and train.

deployment commands

scap backport [--list]
  1. query gerrit for unmerged patchsets on relevant wmf/ branches
  2. user selects which to backport/deploy
  3. delegates to scap backport change_url (below)
scap backport change_url [change_url]
  1. Validate that the specified changes are suitable (e.g., acceptable repository, acceptable branch)
  2. +2 on changes
  3. trigger pipeline job on releases-jenkins to build new wmf branch image w/ given changes and new multiversion image
  4. listen for promote patchset on deployment-charts
    • tag images with something identifying patch for use by final image build and deployment-charts(?)
  5. +2 deployment-charts ps (bump image tag in helmfile.d/service/values.yaml), wait for merge
  6. (LEGACY) unpack image (based on the image tag in helmfile.d/service/values.yaml) into /srv/mediawiki-staging
  7. runs helmfile apply
  8. (LEGACY) syncs to legacy servers using scap sync-world (or optimized equivalent)
scap rollback

(Note that rollbacks are an emergency event that require coordination on further deployments.)

  1. revert deployment-charts commit for mediawiki
  2. +2 on revert
  3. need to update legacy deployment, which looks at deployment-charts helmfile values
  4. (steps 6-8 of scap backport)
scap backport --revert change_url [change_url]
  1. create reverts in gerrit for given changes
  2. (steps 2-8 of scap backport)

Event Timeline

scap backport --revert change_id [change_id]

One quick point here, a change_id can refer to more than 1 change: https://gerrit.wikimedia.org/r/q/If04177f2a9f681002be5eafd97cbc5800aed2e27

I'm not sure if that's a problem or not, but something to be aware of in the design

scap backport --revert change_id [change_id]

One quick point here, a change_id can refer to more than 1 change: https://gerrit.wikimedia.org/r/q/If04177f2a9f681002be5eafd97cbc5800aed2e27

I'm not sure if that's a problem or not, but something to be aware of in the design

I've changed the spec to take change_url as a parameter which could be either a change number or a full url (the contains the number), the important thing being that we should reference change numbers which are unique and not IDs which, as you mentioned, are not.