Page MenuHomePhabricator
Create Task
Maniphest T329945

Migrate the URL downloaders to Bullseye
Closed, ResolvedPublic
Actions

Description

These currently use Buster and should be migrated to Bullseye.

Details

SubjectRepoBranchLines +/-
Remove old URL downloadersoperations/deployment-chartsmaster+0 -8
Remove urldownloader role from old buster serversoperations/puppetproduction+2 -2
Point eqiad URL downloaders to bullseye hostoperations/dnsmaster+1 -1
Point codfw URL downloader to new bullseye hostoperations/dnsmaster+1 -1
Point codfw URL downloader to new bullseye hostoperations/dnsmaster+1 -1
Also apply url downloader role to remaining Bullseye hostsoperations/puppetproduction+1 -10
Add new url downloaders to ACLsoperations/deployment-chartsmaster+8 -0
Apply url_downloader role to urldownloader2004operations/puppetproduction+9 -1
Add urldownloader100[34] to site.ppoperations/puppetproduction+1 -1
Add new VMsoperations/puppetproduction+4 -1
Customize query in gerrit

Related Objects

Event Timeline

MoritzMuehlenhoff created this task.Feb 17 2023, 3:35 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 17 2023, 3:35 PM
MoritzMuehlenhoff claimed this task.Feb 17 2023, 3:35 PM
MoritzMuehlenhoff triaged this task as Medium priority.
gerritbot added a comment.Feb 17 2023, 3:36 PM

Change 890003 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Add new VMs

https://gerrit.wikimedia.org/r/890003

gerritbot added a project: Patch-For-Review.Feb 17 2023, 3:36 PM
gerritbot added a comment.Feb 17 2023, 3:47 PM

Change 890003 merged by Muehlenhoff:

[operations/puppet@production] Add new VMs

https://gerrit.wikimedia.org/r/890003

Maintenance_bot removed a project: Patch-For-Review.Feb 17 2023, 4:10 PM
ops-monitoring-bot added a comment.Feb 24 2023, 10:13 AM

Cookbook cookbooks.sre.ganeti.reimage was started by jmm@cumin2002 for host urldownloader2003.wikimedia.org with OS bullseye

ops-monitoring-bot added a comment.Feb 24 2023, 10:58 AM

Cookbook cookbooks.sre.ganeti.reimage started by jmm@cumin2002 for host urldownloader2003.wikimedia.org with OS bullseye completed:

  • urldownloader2003 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Set boot to disk
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/ganeti/reimage/202302241013_jmm_848980_urldownloader2003.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
ops-monitoring-bot added a comment.Feb 24 2023, 11:13 AM

Cookbook cookbooks.sre.ganeti.reimage was started by jmm@cumin2002 for host urldownloader2004.wikimedia.org with OS bullseye

ops-monitoring-bot added a comment.Feb 24 2023, 11:58 AM

Cookbook cookbooks.sre.ganeti.reimage started by jmm@cumin2002 for host urldownloader2004.wikimedia.org with OS bullseye completed:

  • urldownloader2004 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Set boot to disk
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/ganeti/reimage/202302241113_jmm_861678_urldownloader2004.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
Stashbot added a comment.Feb 24 2023, 3:11 PM

Mentioned in SAL (#wikimedia-operations) [2023-02-24T15:11:56Z] <jmm@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "new url downloaders - jmm@cumin2002 - T329945"

Stashbot added a comment.Feb 24 2023, 3:21 PM

Mentioned in SAL (#wikimedia-operations) [2023-02-24T15:21:20Z] <jmm@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "new url downloaders - jmm@cumin2002 - T329945"

JMeybohm mentioned this in T275035: Refactor common_templates/0.2/default-network-policy-conf.yaml into a GlobalNetworkPolicy.Feb 27 2023, 5:01 PM
ops-monitoring-bot added a comment.Mar 8 2023, 11:25 AM

Cookbook cookbooks.sre.ganeti.reimage was started by jmm@cumin2002 for host urldownloader1003.wikimedia.org with OS bullseye

ops-monitoring-bot added a comment.Mar 8 2023, 12:09 PM

Cookbook cookbooks.sre.ganeti.reimage started by jmm@cumin2002 for host urldownloader1003.wikimedia.org with OS bullseye executed with errors:

  • urldownloader1003 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Set boot to disk
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Mar 8 2023, 12:14 PM

Cookbook cookbooks.sre.ganeti.reimage was started by jmm@cumin2002 for host urldownloader1003.wikimedia.org with OS bullseye

ops-monitoring-bot added a comment.Mar 8 2023, 1:13 PM

Cookbook cookbooks.sre.ganeti.reimage started by jmm@cumin2002 for host urldownloader1003.wikimedia.org with OS bullseye executed with errors:

  • urldownloader1003 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Set boot to disk
    • The reimage failed, see the cookbook logs for the details
gerritbot added a comment.Mar 9 2023, 1:49 PM

Change 896099 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Add urldownloader100[34] to site.pp

https://gerrit.wikimedia.org/r/896099

gerritbot added a project: Patch-For-Review.Mar 9 2023, 1:49 PM
gerritbot added a comment.Mar 9 2023, 1:51 PM

Change 896099 merged by Muehlenhoff:

[operations/puppet@production] Add urldownloader100[34] to site.pp

https://gerrit.wikimedia.org/r/896099

Maintenance_bot removed a project: Patch-For-Review.Mar 9 2023, 2:12 PM
ops-monitoring-bot added a comment.Mar 10 2023, 11:08 AM

Cookbook cookbooks.sre.ganeti.reimage was started by jmm@cumin2002 for host urldownloader1004.wikimedia.org with OS bullseye

ops-monitoring-bot added a comment.Mar 10 2023, 11:51 AM

Cookbook cookbooks.sre.ganeti.reimage started by jmm@cumin2002 for host urldownloader1004.wikimedia.org with OS bullseye completed:

  • urldownloader1004 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Set boot to disk
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/ganeti/reimage/202303101107_jmm_503880_urldownloader1004.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
gerritbot added a comment.Mar 10 2023, 11:57 AM

Change 896325 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Apply url_downloader role to urldownloader2004

https://gerrit.wikimedia.org/r/896325

gerritbot added a project: Patch-For-Review.Mar 10 2023, 11:57 AM
gerritbot added a comment.Mar 13 2023, 9:03 AM

Change 896325 merged by Muehlenhoff:

[operations/puppet@production] Apply url_downloader role to urldownloader2004

https://gerrit.wikimedia.org/r/896325

Maintenance_bot removed a project: Patch-For-Review.Mar 13 2023, 9:10 AM
gerritbot added a comment.May 5 2023, 3:05 PM

Change 916512 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/deployment-charts@master] Add new url downloaders to ACLs

https://gerrit.wikimedia.org/r/916512

gerritbot added a project: Patch-For-Review.May 5 2023, 3:05 PM
gerritbot added a comment.May 8 2023, 9:36 AM

Change 916512 merged by jenkins-bot:

[operations/deployment-charts@master] Add new url downloaders to ACLs

https://gerrit.wikimedia.org/r/916512

Maintenance_bot removed a project: Patch-For-Review.May 8 2023, 10:11 AM
gerritbot added a comment.Jun 2 2023, 8:26 AM

Change 926421 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/dns@master] Point codfw URL downloader to new bullseye host

https://gerrit.wikimedia.org/r/926421

gerritbot added a project: Patch-For-Review.Jun 2 2023, 8:26 AM
gerritbot added a comment.Jun 2 2023, 1:37 PM

Change 926496 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Also apply url downloader role to remaining Bullseye hosts

https://gerrit.wikimedia.org/r/926496

gerritbot added a comment.Jun 2 2023, 1:44 PM

Change 926496 merged by Muehlenhoff:

[operations/puppet@production] Also apply url downloader role to remaining Bullseye hosts

https://gerrit.wikimedia.org/r/926496

gerritbot added a comment.Jun 5 2023, 9:02 AM

Change 926421 merged by Muehlenhoff:

[operations/dns@master] Point codfw URL downloader to new bullseye host

https://gerrit.wikimedia.org/r/926421

Maintenance_bot removed a project: Patch-For-Review.Jun 5 2023, 9:11 AM
gerritbot added a comment.Jun 26 2023, 10:01 AM

Change 932830 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/dns@master] Point codfw URL downloader to new bullseye host

https://gerrit.wikimedia.org/r/932830

gerritbot added a project: Patch-For-Review.Jun 26 2023, 10:01 AM
gerritbot added a comment.Jun 26 2023, 10:04 AM

Change 932830 merged by Muehlenhoff:

[operations/dns@master] Point codfw URL downloader to new bullseye host

https://gerrit.wikimedia.org/r/932830

Maintenance_bot removed a project: Patch-For-Review.Jun 26 2023, 10:12 AM
gerritbot added a comment.Jun 27 2023, 12:26 PM

Change 933441 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/dns@master] Point eqiad URL downloaders to bullseye host

https://gerrit.wikimedia.org/r/933441

gerritbot added a project: Patch-For-Review.Jun 27 2023, 12:26 PM
gerritbot added a comment.Jun 28 2023, 11:49 AM

Change 933441 merged by Muehlenhoff:

[operations/dns@master] Point eqiad URL downloaders to bullseye host

https://gerrit.wikimedia.org/r/933441

Maintenance_bot removed a project: Patch-For-Review.Jun 28 2023, 12:10 PM
gerritbot added a comment.Jun 28 2023, 1:35 PM

Change 933904 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Remove urldownloader role from old buster servers

https://gerrit.wikimedia.org/r/933904

gerritbot added a project: Patch-For-Review.Jun 28 2023, 1:35 PM
gerritbot added a comment.Jun 29 2023, 12:11 PM

Change 933904 merged by Muehlenhoff:

[operations/puppet@production] Remove urldownloader role from old buster servers

https://gerrit.wikimedia.org/r/933904

Maintenance_bot removed a project: Patch-For-Review.Jun 29 2023, 12:30 PM
ops-monitoring-bot added a comment.Jun 30 2023, 6:41 AM

Icinga downtime and Alertmanager silence (ID=04703ca5-7468-4229-a4bf-5a47b58763e0) set by jmm@cumin2002 for 7 days, 0:00:00 on 2 host(s) and their services with reason: pending decom

urldownloader[2001-2002].wikimedia.org
gerritbot added a comment.Jul 5 2023, 11:03 AM

Change 935713 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/deployment-charts@master] Remove old URL downloaders

https://gerrit.wikimedia.org/r/935713

gerritbot added a project: Patch-For-Review.Jul 5 2023, 11:03 AM
ops-monitoring-bot added a comment.Jul 5 2023, 11:25 AM

cookbooks.sre.hosts.decommission executed by jmm@cumin2002 for hosts: urldownloader2001.wikimedia.org

  • urldownloader2001.wikimedia.org (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster codfw to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster codfw to Netbox
ops-monitoring-bot added a comment.Jul 5 2023, 11:36 AM

cookbooks.sre.hosts.decommission executed by jmm@cumin2002 for hosts: urldownloader2002.wikimedia.org

  • urldownloader2002.wikimedia.org (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster codfw to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster codfw to Netbox
ops-monitoring-bot added a comment.Jul 5 2023, 11:50 AM

cookbooks.sre.hosts.decommission executed by jmm@cumin2002 for hosts: urldownloader1002.wikimedia.org

  • urldownloader1002.wikimedia.org (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster eqiad to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster eqiad to Netbox
ops-monitoring-bot added a comment.Jul 5 2023, 11:59 AM

cookbooks.sre.hosts.decommission executed by jmm@cumin2002 for hosts: urldownloader1001.wikimedia.org

  • urldownloader1001.wikimedia.org (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster eqiad to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster eqiad to Netbox
gerritbot added a comment.Jul 5 2023, 2:10 PM

Change 935713 merged by jenkins-bot:

[operations/deployment-charts@master] Remove old URL downloaders

https://gerrit.wikimedia.org/r/935713

Maintenance_bot removed a project: Patch-For-Review.Jul 5 2023, 2:12 PM
MoritzMuehlenhoff closed this task as Resolved.Jul 5 2023, 2:35 PM

We're running new URL downloaders on Bullseye (urldownloader[12]00[34].wikimedia.org) an the one Buster systems have been decommissioned.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits