Page MenuHomePhabricator
Create Task
Maniphest T313229

Production Dispatch Infrastructure
Closed, DeclinedPublic
Actions

Description

Checklist for deployment of initial "production" dispatch infrastructure

  • Obtain dedicated WMCS horizon project space/quota (we'll use onfire project)
  • Containers (or puppetized service with VMs)
  • Postgres backend
  • DNS entries
  • Service/host monitoring T326842
  • Data protection (e.g. data exported and backed up) T326843
  • HA/maintenance considerations
20220823

Status update: with https://gerrit.wikimedia.org/r/c/operations/puppet/+/824449 and https://gerrit.wikimedia.org/r/c/operations/puppet/+/824448 we can stand up a database primary and dispatch's frontend (behind SSO) and scheduler. Items pending:

  • Write a dispatch plugin to read username from a request header.
  • Ship said plugin with dispatch container
  • Write a sync script between LDAP and dispatch API to set user permissions based on group (e.g. ops is dispatch admins)
  • Make sure an admin user is created for API access for the script above

Details

SubjectRepoBranchLines +/-
dispatch: manage config.js locallyoperations/docker-images/production-imagesmaster+532 -1
dispatch: upgrade to 20221110 and build with local config.jsoperations/docker-images/production-imagesmaster+531 -2
dispatch: add apache redirect from default org to wikimedia orgoperations/puppetproduction+11 -0
dispatch: sync user role and info from LDAPoperations/puppetproduction+413 -0
hieradata: set dispatch-be2001 as replicaoperations/puppetproduction+2 -1
bacula: Add missing fileset definition dispatch-postgresoperations/puppetproduction+5 -0
dispatch: refactor/simplify db profileoperations/puppetproduction+22 -37
dispatch: move frontend to its own moduleoperations/puppetproduction+83 -41
dispatch: configure http header auth provideroperations/puppetproduction+7 -5
dispatch: add ipython for 'dispatch server shell'operations/docker-images/production-imagesmaster+7 -1
dispatch: run wrapper with interactive/tty supportoperations/puppetproduction+1 -1
wikimedia.org: add dispatch.w.ooperations/dnsmaster+2 -0
hieradata: don't monitor /run/docker on alerting_hostoperations/puppetproduction+4 -0
dispatch: run the scheduler on active host onlyoperations/puppetproduction+9 -0
dispatch: enforce ssl for dispatch DB useroperations/puppetproduction+2 -0
hieradata: fix Prometheus IDP entryoperations/puppetproduction+3 -1
hieradata: add dispatch.w.o to IDPoperations/puppetproduction+7 -0
idp: fix vhost_settings for dispatch_portoperations/puppetproduction+2 -2
hieradata: add dispatch db_hostnameoperations/puppetproduction+1 -0
alerting_host: include dispatch profileoperations/puppetproduction+2 -0
dispatch: introduce profileoperations/puppetproduction+159 -0
dispatch: update to latest upstreamoperations/docker-images/production-imagesmaster+10 -1
dispatch: assign backend roleoperations/puppetproduction+1 -1
dispatch: add backend roleoperations/puppetproduction+222 -0
dispatch-be1001: apply role::insetupoperations/puppetproduction+5 -0
install_server: add dhcp/netboot records for dispatch-be1001operations/puppetproduction+6 -0
wmflib: introduce pythonloglevel typeoperations/puppetproduction+24 -2
postgresql: default to autodetecting pg versionoperations/cookbooksmaster+2 -2
service: use --env-file for dockeroperations/puppetproduction+11 -3
docker: use ExecStartPre to implement --pull=alwaysoperations/puppetproduction+4 -1
postgresql: resync_replica improvements and fixesoperations/puppetproduction+11 -13
pontoon: generate en_US.UTF-8 localeoperations/puppetproduction+14 -0
dispatch: add containeroperations/docker-images/production-imagesmaster+64 -0
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
gerritbot added a comment.Aug 18 2022, 12:49 PM

Change 824447 merged by Filippo Giunchedi:

[operations/puppet@production] postgresql: resync_replica improvements and fixes

https://gerrit.wikimedia.org/r/824447

gerritbot added a comment.Aug 18 2022, 12:51 PM

Change 824486 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/cookbooks@master] postgresql: default to autodetecting pg version

https://gerrit.wikimedia.org/r/824486

gerritbot added a comment.Aug 19 2022, 8:42 AM

Change 824450 merged by Filippo Giunchedi:

[operations/puppet@production] docker: use ExecStartPre to implement --pull=always

https://gerrit.wikimedia.org/r/824450

gerritbot added a comment.Aug 19 2022, 8:45 AM

Change 824451 merged by Filippo Giunchedi:

[operations/puppet@production] service: use --env-file for docker

https://gerrit.wikimedia.org/r/824451

gerritbot added a comment.Aug 19 2022, 8:49 AM

Change 824486 merged by Filippo Giunchedi:

[operations/cookbooks@master] postgresql: default to autodetecting pg version

https://gerrit.wikimedia.org/r/824486

gerritbot added a comment.Aug 22 2022, 9:13 AM

Change 825253 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] wmflib: introduce pythonloglevel type

https://gerrit.wikimedia.org/r/825253

fgiunchedi updated the task description. (Show Details)Aug 22 2022, 9:41 AM
gerritbot added a comment.Aug 22 2022, 3:21 PM

Change 825253 merged by Filippo Giunchedi:

[operations/puppet@production] wmflib: introduce pythonloglevel type

https://gerrit.wikimedia.org/r/825253

fgiunchedi updated the task description. (Show Details)Aug 23 2022, 8:56 AM
fgiunchedi updated the task description. (Show Details)Aug 23 2022, 9:02 AM
Volans subscribed.Sep 7 2022, 7:55 AM
gerritbot added a comment.Sep 14 2022, 1:37 PM

Change 832263 had a related patch set uploaded (by Herron; author: Herron):

[operations/puppet@production] install_server: add dhcp/netboot records for dispatch-be1001

https://gerrit.wikimedia.org/r/832263

gerritbot added a comment.Sep 14 2022, 1:41 PM

Change 832263 merged by Herron:

[operations/puppet@production] install_server: add dhcp/netboot records for dispatch-be1001

https://gerrit.wikimedia.org/r/832263

gerritbot added a comment.Sep 15 2022, 2:44 PM

Change 832505 had a related patch set uploaded (by Herron; author: Herron):

[operations/puppet@production] dispatch-be1001: apply role::insetup

https://gerrit.wikimedia.org/r/832505

gerritbot added a comment.Sep 15 2022, 2:59 PM

Change 832505 merged by Herron:

[operations/puppet@production] dispatch-be1001: apply role::insetup

https://gerrit.wikimedia.org/r/832505

gerritbot added a comment.Oct 24 2022, 8:47 AM

Change 848191 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] dispatch: assign backend role

https://gerrit.wikimedia.org/r/848191

gerritbot added a comment.Oct 24 2022, 8:53 AM

Change 824448 merged by Filippo Giunchedi:

[operations/puppet@production] dispatch: add backend role

https://gerrit.wikimedia.org/r/824448

gerritbot added a comment.Oct 24 2022, 9:26 AM

Change 848228 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/docker-images/production-images@master] dispatch: update to latest upstream

https://gerrit.wikimedia.org/r/848228

gerritbot added a comment.Oct 24 2022, 1:29 PM

Change 848191 merged by Filippo Giunchedi:

[operations/puppet@production] dispatch: assign backend role

https://gerrit.wikimedia.org/r/848191

gerritbot added a comment.Oct 25 2022, 7:04 AM

Change 848228 merged by Filippo Giunchedi:

[operations/docker-images/production-images@master] dispatch: update to latest upstream

https://gerrit.wikimedia.org/r/848228

gerritbot added a comment.Oct 25 2022, 8:44 AM

Change 849021 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] alerting_host: include dispatch profile

https://gerrit.wikimedia.org/r/849021

gerritbot added a comment.Oct 27 2022, 2:36 PM

Change 824449 merged by Filippo Giunchedi:

[operations/puppet@production] dispatch: introduce profile

https://gerrit.wikimedia.org/r/824449

gerritbot added a comment.Oct 27 2022, 2:41 PM

Change 849021 merged by Filippo Giunchedi:

[operations/puppet@production] alerting_host: include dispatch profile

https://gerrit.wikimedia.org/r/849021

gerritbot added a comment.Oct 27 2022, 2:53 PM

Change 850178 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] hieradata: add dispatch db_hostname

https://gerrit.wikimedia.org/r/850178

gerritbot added a comment.Oct 27 2022, 2:55 PM

Change 850178 merged by Filippo Giunchedi:

[operations/puppet@production] hieradata: add dispatch db_hostname

https://gerrit.wikimedia.org/r/850178

lmata edited projects, added SRE Observability (FY2022/2023-Q2); removed SRE Observability (FY2022/2023-Q1).Oct 28 2022, 6:28 PM
gerritbot added a comment.Oct 31 2022, 9:21 AM

Change 850993 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] hieradata: don't monitor /run/docker on alerting_host

https://gerrit.wikimedia.org/r/850993

gerritbot added a comment.Oct 31 2022, 10:04 AM

Change 850999 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] dispatch: enforce ssl for dispatch DB user

https://gerrit.wikimedia.org/r/850999

gerritbot added a comment.Oct 31 2022, 10:12 AM

Change 851000 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] dispatch: run the scheduler on active host only

https://gerrit.wikimedia.org/r/851000

gerritbot added a comment.Oct 31 2022, 10:16 AM

Change 851001 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] idp: fix vhost_settings for dispatch_port

https://gerrit.wikimedia.org/r/851001

gerritbot added a comment.Oct 31 2022, 10:26 AM

Change 851001 merged by Filippo Giunchedi:

[operations/puppet@production] idp: fix vhost_settings for dispatch_port

https://gerrit.wikimedia.org/r/851001

gerritbot added a comment.Oct 31 2022, 10:47 AM

Change 851003 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] hieradata: add dispatch.w.o to IDP

https://gerrit.wikimedia.org/r/851003

gerritbot added a comment.Oct 31 2022, 10:47 AM

Change 851004 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] hieradata: fix Prometheus IDP entry

https://gerrit.wikimedia.org/r/851004

gerritbot added a comment.Oct 31 2022, 11:02 AM

Change 851003 merged by Filippo Giunchedi:

[operations/puppet@production] hieradata: add dispatch.w.o to IDP

https://gerrit.wikimedia.org/r/851003

gerritbot added a comment.Oct 31 2022, 11:04 AM

Change 851004 merged by Filippo Giunchedi:

[operations/puppet@production] hieradata: fix Prometheus IDP entry

https://gerrit.wikimedia.org/r/851004

gerritbot added a comment.Oct 31 2022, 2:38 PM

Change 850999 merged by Filippo Giunchedi:

[operations/puppet@production] dispatch: enforce ssl for dispatch DB user

https://gerrit.wikimedia.org/r/850999

gerritbot added a comment.Oct 31 2022, 2:41 PM

Change 851000 merged by Filippo Giunchedi:

[operations/puppet@production] dispatch: run the scheduler on active host only

https://gerrit.wikimedia.org/r/851000

gerritbot added a comment.Nov 1 2022, 8:32 AM

Change 850993 merged by Filippo Giunchedi:

[operations/puppet@production] hieradata: don't monitor /run/docker on alerting_host

https://gerrit.wikimedia.org/r/850993

gerritbot added a comment.Nov 1 2022, 12:54 PM

Change 851619 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/docker-images/production-images@master] dispatch: add ipython for 'dispatch server shell'

https://gerrit.wikimedia.org/r/851619

gerritbot added a comment.Nov 1 2022, 12:57 PM

Change 851620 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] dispatch: run wrapper with interactive/tty support

https://gerrit.wikimedia.org/r/851620

gerritbot added a comment.Nov 1 2022, 1:40 PM

Change 851632 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/dns@master] wikimedia.org: add dispatch.w.o

https://gerrit.wikimedia.org/r/851632

gerritbot added a comment.Nov 1 2022, 1:48 PM

Change 851632 merged by Filippo Giunchedi:

[operations/dns@master] wikimedia.org: add dispatch.w.o

https://gerrit.wikimedia.org/r/851632

gerritbot added a comment.Nov 1 2022, 2:45 PM

Change 851645 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] dispatch: configure http header auth provider

https://gerrit.wikimedia.org/r/851645

gerritbot added a comment.Nov 1 2022, 2:46 PM

Change 851620 merged by Filippo Giunchedi:

[operations/puppet@production] dispatch: run wrapper with interactive/tty support

https://gerrit.wikimedia.org/r/851620

gerritbot added a comment.Nov 1 2022, 2:48 PM

Change 851619 merged by Filippo Giunchedi:

[operations/docker-images/production-images@master] dispatch: add ipython for 'dispatch server shell'

https://gerrit.wikimedia.org/r/851619

gerritbot added a comment.Nov 1 2022, 2:56 PM

Change 851645 merged by Filippo Giunchedi:

[operations/puppet@production] dispatch: configure http header auth provider

https://gerrit.wikimedia.org/r/851645

gerritbot added a comment.Nov 1 2022, 4:21 PM

Change 851672 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] dispatch: move frontend to its own module

https://gerrit.wikimedia.org/r/851672

gerritbot added a comment.Nov 1 2022, 5:44 PM

Change 851693 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] dispatch: refactor/simplify db profile

https://gerrit.wikimedia.org/r/851693

fgiunchedi updated the task description. (Show Details)Nov 2 2022, 1:55 PM
gerritbot added a comment.Nov 2 2022, 2:18 PM

Change 851672 merged by Filippo Giunchedi:

[operations/puppet@production] dispatch: move frontend to its own module

https://gerrit.wikimedia.org/r/851672

gerritbot added a comment.Nov 3 2022, 9:10 AM

Change 851693 merged by Filippo Giunchedi:

[operations/puppet@production] dispatch: refactor/simplify db profile

https://gerrit.wikimedia.org/r/851693

gerritbot added a comment.Nov 3 2022, 6:42 PM

Change 852992 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] dispatch: sync user role and info from LDAP

https://gerrit.wikimedia.org/r/852992

gerritbot added a comment.Nov 4 2022, 11:58 AM

Change 853268 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] bacula: Add missing fileset definition dispatch-postgres

https://gerrit.wikimedia.org/r/853268

gerritbot added a comment.Nov 4 2022, 1:05 PM

Change 853268 merged by Jcrespo:

[operations/puppet@production] bacula: Add missing fileset definition dispatch-postgres

https://gerrit.wikimedia.org/r/853268

fgiunchedi added a comment.Nov 7 2022, 10:29 AM

I have started https://wikitech.wikimedia.org/wiki/Dispatch with some initialization steps, the page will need expanding of course!

fgiunchedi added a subtask: T322556: Site: codfw VM %request for dispatch-be2001.Nov 7 2022, 4:23 PM
fgiunchedi closed subtask T322556: Site: codfw VM %request for dispatch-be2001 as Resolved.Nov 8 2022, 9:22 AM
gerritbot added a comment.Nov 8 2022, 11:02 AM

Change 854486 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] hieradata: set dispatch-be2001 as replica

https://gerrit.wikimedia.org/r/854486

gerritbot added a comment.Nov 8 2022, 11:09 AM

Change 854486 merged by Filippo Giunchedi:

[operations/puppet@production] hieradata: set dispatch-be2001 as replica

https://gerrit.wikimedia.org/r/854486

gerritbot added a comment.Nov 14 2022, 12:59 PM

Change 852992 merged by Filippo Giunchedi:

[operations/puppet@production] dispatch: sync user role and info from LDAP

https://gerrit.wikimedia.org/r/852992

fgiunchedi updated the task description. (Show Details)Nov 14 2022, 1:14 PM
gerritbot added a comment.Nov 14 2022, 4:53 PM

Change 856612 had a related patch set uploaded (by Herron; author: Herron):

[operations/puppet@production] dispatch: add apache redirect from default org to wikimedia org

https://gerrit.wikimedia.org/r/856612

gerritbot added a comment.Nov 16 2022, 2:57 PM

Change 856612 merged by Herron:

[operations/puppet@production] dispatch: add apache redirect from default org to wikimedia org

https://gerrit.wikimedia.org/r/856612

gerritbot added a comment.Nov 16 2022, 9:43 PM

Change 857781 had a related patch set uploaded (by Herron; author: Herron):

[operations/docker-images/production-images@master] dispatch: upgrade to 20221110 and build with local config.js

https://gerrit.wikimedia.org/r/857781

MoritzMuehlenhoff subscribed.Nov 17 2022, 1:01 PM

Is there a reason these hosts are still being setup under Buster even though it's a new service? The designated end date for Buster is September 2023, so let's not build a new service with an OS which is at the end of the support life cycle?

gerritbot added a comment.Nov 17 2022, 7:45 PM

Change 857781 merged by Herron:

[operations/docker-images/production-images@master] dispatch: upgrade to 20221110 and build with local config.js

https://gerrit.wikimedia.org/r/857781

herron added a comment.Nov 17 2022, 8:11 PM
In T313229#8401979, @MoritzMuehlenhoff wrote:

Is there a reason these hosts are still being setup under Buster even though it's a new service? The designated end date for Buster is September 2023, so let's not build a new service with an OS which is at the end of the support life cycle?

The hosts are paired with the alerting_host frontends which are Buster. Agreed though, I'll have a closer look at running the backends on Bullseye. And the alert frontends will have to be upgraded in the relatively near future too.

gerritbot added a comment.Nov 18 2022, 3:43 PM

Change 858616 had a related patch set uploaded (by Herron; author: Herron):

[operations/docker-images/production-images@master] dispatch: manage config.js locally

https://gerrit.wikimedia.org/r/858616

gerritbot added a comment.Nov 18 2022, 4:59 PM

Change 858616 merged by Herron:

[operations/docker-images/production-images@master] dispatch: manage config.js locally

https://gerrit.wikimedia.org/r/858616

fgiunchedi mentioned this in rCCKB4b123b98bb53: postgresql: default to autodetecting pg version.Dec 14 2022, 3:31 PM
herron added a project: User-herron.Jan 5 2023, 5:58 PM
fgiunchedi removed a project: User-fgiunchedi.Jan 12 2023, 8:51 AM
lmata removed a project: SRE Observability (FY2022/2023-Q2).Jan 12 2023, 6:54 PM
lmata added a project: SRE Observability (FY2022/2023-Q3).
lmata mentioned this in T308467: implementing an incident response workflow automation tool for SRE.Jan 12 2023, 8:21 PM
herron updated the task description. (Show Details)Jan 13 2023, 2:33 PM
herron updated the task description. (Show Details)
lmata edited projects, added Incident Tooling; removed SRE Observability (FY2022/2023-Q3).May 2 2023, 1:46 PM
Maintenance_bot removed a project: Patch-For-Review.May 2 2023, 2:11 PM
lmata moved this task from Inbox to Backlog on the Incident Tooling board.May 16 2023, 3:31 AM
lmata moved this task from Backlog to Prioritized on the Incident Tooling board.
BCornwall subscribed.Aug 23 2023, 7:36 PM

Should this be closed since AFAIK dispatch has been ruled out?

lmata added a comment.Aug 23 2023, 9:00 PM

@BCornwall I'm good with that. Do you want to do the honors?

BCornwall closed this task as Declined.Aug 24 2023, 4:27 PM

Closing as dispatch has been ruled out as an option: See T308467 for follow-up discussion of where we're going.

MoritzMuehlenhoff reopened this task as Open.Aug 24 2023, 4:33 PM

We still have two dispatch-be* hosts running in production these should be properly decommissioned if that's not the solution we'll end up using?

herron added a comment.EditedAug 24 2023, 4:36 PM

Agree, although let's create a decom task for that as there are some services on the alert hosts to clean up as well

I've created T344937 to track decom efforts

herron closed this task as Resolved.Aug 24 2023, 4:40 PM
herron changed the task status from Resolved to Declined.
BCornwall closed subtask T326842: Service/host monitoring for dispatch as Declined.Aug 24 2023, 4:55 PM
BCornwall closed subtask T340772: Dispatch: enable gmail plugin as Declined.Aug 24 2023, 4:57 PM
BCornwall closed subtask T326843: Dispatch Data protection (e.g. data exported and backed up) as Declined.
herron closed subtask T344937: Decom dispatch infrastructure as Resolved.Sep 25 2023, 5:11 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits