Page MenuHomePhabricator
Create Task
Maniphest T344937

Decom dispatch infrastructure
Closed, ResolvedPublic
Actions

Description

Tracking task for dispatch decom/cleanup, high level todo:

  • Remove dispatch instances from alert hosts
  • Clean up docker and idm remnants on alert hosts
  • Decom dispatch-be hosts
  • Clean up dispatch DNS entries
  • Clean up dispatch from puppet repository
  • Clean up cloud instances in onfire horizon project
  • Clean up containers
  • Clean up documentation
  • Remove gcp credentials/roles
  • Drop dispatch credentials entry from pwstore

Details

Due Date
Dec 30 2023, 11:00 PM
Related Changes in Gerrit:
SubjectRepoBranchLines +/-
modules: cleanup last dispatch renmantsoperations/puppetproduction+0 -73
dispatch: remove puppetizationoperations/puppetproduction+0 -800
remove dispatch dns recordoperations/dnsmaster+0 -2
dispatch::web: correct /usr/local/bin/dispatch ensureoperations/puppetproduction+1 -1
dispatch::web: add ensure param and ensure => absentoperations/puppetproduction+5 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

herron created this task.Aug 24 2023, 4:39 PM
herron mentioned this in T313229: Production Dispatch Infrastructure.
herron added a comment.Aug 24 2023, 4:53 PM

@lmata could you please confirm if/when ready to proceed with decom of dispatch infra?

BCornwall mentioned this in T326842: Service/host monitoring for dispatch.Aug 24 2023, 4:55 PM
BCornwall mentioned this in T313230: Dispatch IRC Integration.
BCornwall mentioned this in T313232: Dispatch Documentation.
BCornwall mentioned this in T313233: Dispatch Splunk On-call / VictorOps Integration.
BCornwall mentioned this in T313234: Dispatch Phabricator Integration.
BCornwall mentioned this in T317192: implement on call shift hand-off / hand-overs in dispatch.
BCornwall mentioned this in T340772: Dispatch: enable gmail plugin.Aug 24 2023, 4:57 PM
BCornwall mentioned this in T326843: Dispatch Data protection (e.g. data exported and backed up).
BCornwall mentioned this in T317316: Fine-tune dispatch SRE incident template.
herron updated the task description. (Show Details)Aug 30 2023, 1:59 PM
• lmata added a comment.Aug 31 2023, 10:46 PM

Sounds good to me, I've posed an ask from the SRE-OnFire team also to confirm we are no longer interested in dispatch. Maybe allow for another week for us to meet, and we can move forward.

gerritbot added a comment.Sep 14 2023, 3:06 PM

Change 957749 had a related patch set uploaded (by Herron; author: Herron):

[operations/puppet@production] dispatch::web: add ensure param and ensure => absent

https://gerrit.wikimedia.org/r/957749

gerritbot added a project: Patch-For-Review.Sep 14 2023, 3:06 PM
gerritbot added a comment.Sep 14 2023, 3:25 PM

Change 957756 had a related patch set uploaded (by Herron; author: Herron):

[operations/puppet@production] dispatch: remove puppetization

https://gerrit.wikimedia.org/r/957756

gerritbot added a comment.Sep 14 2023, 8:07 PM

Change 957799 had a related patch set uploaded (by Herron; author: Herron):

[operations/dns@master] remove dispatch dns record

https://gerrit.wikimedia.org/r/957799

gerritbot added a comment.Sep 20 2023, 1:31 PM

Change 957749 merged by Herron:

[operations/puppet@production] dispatch::web: add ensure param and ensure => absent

https://gerrit.wikimedia.org/r/957749

herron updated the task description. (Show Details)Sep 20 2023, 1:36 PM
gerritbot added a comment.Sep 20 2023, 1:39 PM

Change 959220 had a related patch set uploaded (by Herron; author: Herron):

[operations/puppet@production] dispatch::web: correct /usr/local/bin/dispatch ensure

https://gerrit.wikimedia.org/r/959220

herron updated the task description. (Show Details)Sep 20 2023, 1:41 PM
gerritbot added a comment.Sep 20 2023, 1:43 PM

Change 959220 merged by Herron:

[operations/puppet@production] dispatch::web: correct /usr/local/bin/dispatch ensure

https://gerrit.wikimedia.org/r/959220

herron updated the task description. (Show Details)Sep 20 2023, 1:50 PM
herron updated the task description. (Show Details)
ops-monitoring-bot added a comment.Sep 25 2023, 2:43 PM

cookbooks.sre.hosts.decommission executed by herron@cumin1001 for hosts: dispatch-be2001.codfw.wmnet,dispatch-be1001.eqiad.wmnet

  • dispatch-be2001.codfw.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster codfw to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster codfw to Netbox
  • dispatch-be1001.eqiad.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster eqiad to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster eqiad to Netbox
herron updated the task description. (Show Details)Sep 25 2023, 2:44 PM
gerritbot added a comment.Sep 25 2023, 2:48 PM

Change 957799 merged by Herron:

[operations/dns@master] remove dispatch dns record

https://gerrit.wikimedia.org/r/957799

herron updated the task description. (Show Details)Sep 25 2023, 2:54 PM
gerritbot added a comment.Sep 25 2023, 2:54 PM

Change 957756 merged by Herron:

[operations/puppet@production] dispatch: remove puppetization

https://gerrit.wikimedia.org/r/957756

herron updated the task description. (Show Details)Sep 25 2023, 2:57 PM
Maintenance_bot removed a project: Patch-For-Review.Sep 25 2023, 3:11 PM
Stashbot added a comment.Sep 25 2023, 3:19 PM

Mentioned in SAL (#wikimedia-operations) [2023-09-25T15:19:00Z] <herron> alert[12]001 -- apt remove docker.io T344937

Stashbot added a comment.Sep 25 2023, 3:21 PM

Mentioned in SAL (#wikimedia-operations) [2023-09-25T15:21:28Z] <herron> alert[12]001 -- rm /etc/apache2/sites-available/50-dispatch-wikimedia-org.conf && apachectl graceful T344937

herron updated the task description. (Show Details)Sep 25 2023, 3:21 PM
herron added a comment.Sep 25 2023, 3:30 PM

GCP: Project "Dispatch" is now shut down and scheduled to be deleted after Oct 25, 2023.

herron updated the task description. (Show Details)Sep 25 2023, 3:30 PM
herron updated the task description. (Show Details)Sep 25 2023, 3:32 PM
MoritzMuehlenhoff updated the task description. (Show Details)Sep 25 2023, 3:37 PM
herron updated the task description. (Show Details)Sep 25 2023, 3:39 PM
herron updated the task description. (Show Details)
Aklapper set Due Date to Dec 30 2023, 11:00 PM.Sep 25 2023, 4:53 PM
herron closed this task as Resolved.Sep 25 2023, 5:11 PM
herron claimed this task.
build2001:~$ sudo -i docker-registryctl delete-tags docker-registry.discovery.wmnet/dispatch
We're about to delete the following tags for image docker-registry.discovery.wmnet/dispatch:
latest
v20220801-1-20220821
v20220801-1-20220828
v20220801-1-20220904
v20220801-1-20220911
v20220801-1-20220918
v20220801-1-20220925
v20220801-1-20221009
v20220801-1-20221016
v20220801-1-20221023
v20220801-1
v20220915-1-20221030
v20220915-1
v20220915-2-20221106
v20220915-2-20221113
v20220915-2
v20220915-3-20221120
v20220915-3-20221127
v20220915-3-20221204
v20220915-3-20221211
v20220915-3-20221218
v20220915-3-20221225
v20220915-3-20230101
v20220915-3-20230108
v20220915-3-20230115
v20220915-3-20230122
v20220915-3-20230129
v20220915-3-20230205
v20220915-3-20230212
v20220915-3-20230305
v20220915-3-20230312
v20220915-3-20230319
v20220915-3-20230326
v20220915-3-20230402
v20220915-3-20230409
v20220915-3-20230416
v20220915-3-20230423
v20220915-3-20230430
v20220915-3-20230507
v20220915-3-20230514
v20220915-3-20230521
v20220915-3-20230528
v20220915-3-20230604
v20220915-3-20230611
v20220915-3-20230612
v20220915-3-20230618
v20220915-3-20230625
v20220915-3-20230702
v20220915-3-20230716
v20220915-3-20230723
v20220915-3-20230730
v20220915-3-20230806
v20220915-3-20230813
v20220915-3-20230820
v20220915-3-20230827
v20220915-3-20230903
v20220915-3-20230910
v20220915-3-20230917
v20220915-3-20230924
v20220915-3
Ok to proceed? (y/n)y

docker-registry.discovery.wmnet/dispatch:latest                           [DONE]
docker-registry.discovery.wmnet/dispatch:v20220801-1-20220821             [DONE]
docker-registry.discovery.wmnet/dispatch:v20220801-1-20220828             [DONE]
docker-registry.discovery.wmnet/dispatch:v20220801-1-20220904             [DONE]
docker-registry.discovery.wmnet/dispatch:v20220801-1-20220911             [DONE]
docker-registry.discovery.wmnet/dispatch:v20220801-1-20220918             [DONE]
docker-registry.discovery.wmnet/dispatch:v20220801-1-20220925             [DONE]
docker-registry.discovery.wmnet/dispatch:v20220801-1-20221009             [DONE]
docker-registry.discovery.wmnet/dispatch:v20220801-1-20221016             [DONE]
docker-registry.discovery.wmnet/dispatch:v20220801-1-20221023             [DONE]
docker-registry.discovery.wmnet/dispatch:v20220801-1                      [DONE]
docker-registry.discovery.wmnet/dispatch:v20220915-1-20221030             [DONE]
docker-registry.discovery.wmnet/dispatch:v20220915-1                      [DONE]
docker-registry.discovery.wmnet/dispatch:v20220915-2-20221106             [DONE]
docker-registry.discovery.wmnet/dispatch:v20220915-2-20221113             [DONE]
docker-registry.discovery.wmnet/dispatch:v20220915-2                      [DONE]
docker-registry.discovery.wmnet/dispatch:v20220915-3-20221120             [DONE]
docker-registry.discovery.wmnet/dispatch:v20220915-3-20221127             [DONE]
docker-registry.discovery.wmnet/dispatch:v20220915-3-20221204             [DONE]
docker-registry.discovery.wmnet/dispatch:v20220915-3-20221211             [DONE]
docker-registry.discovery.wmnet/dispatch:v20220915-3-20221218             [DONE]
docker-registry.discovery.wmnet/dispatch:v20220915-3-20221225             [DONE]
docker-registry.discovery.wmnet/dispatch:v20220915-3-20230101             [DONE]
docker-registry.discovery.wmnet/dispatch:v20220915-3-20230108             [DONE]
docker-registry.discovery.wmnet/dispatch:v20220915-3-20230115             [DONE]
docker-registry.discovery.wmnet/dispatch:v20220915-3-20230122             [DONE]
docker-registry.discovery.wmnet/dispatch:v20220915-3-20230129             [DONE]
docker-registry.discovery.wmnet/dispatch:v20220915-3-20230205             [DONE]
docker-registry.discovery.wmnet/dispatch:v20220915-3-20230212             [DONE]
docker-registry.discovery.wmnet/dispatch:v20220915-3-20230305             [DONE]
docker-registry.discovery.wmnet/dispatch:v20220915-3-20230312             [DONE]
docker-registry.discovery.wmnet/dispatch:v20220915-3-20230319             [DONE]
docker-registry.discovery.wmnet/dispatch:v20220915-3-20230326             [DONE]
docker-registry.discovery.wmnet/dispatch:v20220915-3-20230402             [DONE]
docker-registry.discovery.wmnet/dispatch:v20220915-3-20230409             [DONE]
docker-registry.discovery.wmnet/dispatch:v20220915-3-20230416             [DONE]
docker-registry.discovery.wmnet/dispatch:v20220915-3-20230423             [DONE]
docker-registry.discovery.wmnet/dispatch:v20220915-3-20230430             [DONE]
docker-registry.discovery.wmnet/dispatch:v20220915-3-20230507             [DONE]
docker-registry.discovery.wmnet/dispatch:v20220915-3-20230514             [DONE]
docker-registry.discovery.wmnet/dispatch:v20220915-3-20230521             [DONE]
docker-registry.discovery.wmnet/dispatch:v20220915-3-20230528             [DONE]
docker-registry.discovery.wmnet/dispatch:v20220915-3-20230604             [DONE]
docker-registry.discovery.wmnet/dispatch:v20220915-3-20230611             [DONE]
docker-registry.discovery.wmnet/dispatch:v20220915-3-20230612             [DONE]
docker-registry.discovery.wmnet/dispatch:v20220915-3-20230618             [DONE]
docker-registry.discovery.wmnet/dispatch:v20220915-3-20230625             [DONE]
docker-registry.discovery.wmnet/dispatch:v20220915-3-20230702             [DONE]
docker-registry.discovery.wmnet/dispatch:v20220915-3-20230716             [DONE]
docker-registry.discovery.wmnet/dispatch:v20220915-3-20230723             [DONE]
docker-registry.discovery.wmnet/dispatch:v20220915-3-20230730             [DONE]
docker-registry.discovery.wmnet/dispatch:v20220915-3-20230806             [DONE]
docker-registry.discovery.wmnet/dispatch:v20220915-3-20230813             [DONE]
docker-registry.discovery.wmnet/dispatch:v20220915-3-20230820             [DONE]
docker-registry.discovery.wmnet/dispatch:v20220915-3-20230827             [DONE]
docker-registry.discovery.wmnet/dispatch:v20220915-3-20230903             [DONE]
docker-registry.discovery.wmnet/dispatch:v20220915-3-20230910             [DONE]
docker-registry.discovery.wmnet/dispatch:v20220915-3-20230917             [DONE]
docker-registry.discovery.wmnet/dispatch:v20220915-3-20230924             [GONE]
docker-registry.discovery.wmnet/dispatch:v20220915-3                      [DONE]

And with that I think we're done here!

herron updated the task description. (Show Details)Sep 25 2023, 5:11 PM
colewhite subscribed.Oct 2 2023, 8:02 PM

There's an outstanding Icinga alert that seems related: CRITICAL - degraded: The following units failed: dispatch-scheduler.service,docker-image-prune-old.service

herron added a comment.Oct 3 2023, 2:32 PM

That's been sorted: RECOVERY - Check systemd state on alert1001 is OK: OK - running: The system is fully operational

gerritbot added a comment.Oct 23 2023, 3:14 PM

Change 967952 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] modules: cleanup last dispatch renmants

https://gerrit.wikimedia.org/r/967952

gerritbot added a project: Patch-For-Review.Oct 23 2023, 3:14 PM
gerritbot added a comment.Nov 1 2023, 12:57 PM

Change 967952 merged by Filippo Giunchedi:

[operations/puppet@production] modules: cleanup last dispatch renmants

https://gerrit.wikimedia.org/r/967952

Maintenance_bot removed a project: Patch-For-Review.Nov 1 2023, 1:10 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits