Page MenuHomePhabricator
Create Task
Maniphest T364016

Q4:magru VM tracking task
Closed, ResolvedPublic
Actions

Description

magru01 (ready, running on ganeti7001/7003):

  • netflow7001.magru.wmnet
  • doh7001.wikimedia.org
  • durum7001.magru.wmnet
  • install7001.wikimedia.org
  • ncredir7001.magru.wmnet

magru02 (ready, running on ganeti7002/7004)

  • doh7002.wikimedia.org
  • durum7002.magru.wmnet
  • ncredir7002.magru.wmnet
  • bast7001.wikimedia.org
  • prometheus7001.magru.wmnet

Details

Show related patches Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedNoneT364016 Q4:magru VM tracking task
 ResolvedcmooneyT364454 mgmt ssh access for prometheus hosts in magru

Event Timeline

ssingh created this task.May 2 2024, 3:24 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 2 2024, 3:24 PM
MoritzMuehlenhoff triaged this task as High priority.May 2 2024, 3:31 PM
MoritzMuehlenhoff updated the task description. (Show Details)
MoritzMuehlenhoff updated the task description. (Show Details)May 2 2024, 3:33 PM
MoritzMuehlenhoff updated the task description. (Show Details)May 2 2024, 3:37 PM
ops-monitoring-bot added a comment.May 2 2024, 3:44 PM

Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin1002 for host durum7001.magru.wmnet with OS bookworm

ayounsi updated the task description. (Show Details)May 2 2024, 3:53 PM
ops-monitoring-bot added a comment.May 2 2024, 4:40 PM

Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin1002 for host durum7001.magru.wmnet with OS bookworm completed:

  • durum7001 (WARN)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Set boot media to disk
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202405021616_sukhe_4120555_durum7001.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.May 2 2024, 6:41 PM

Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin1002 for host doh7001.wikimedia.org with OS bookworm

ops-monitoring-bot added a comment.May 2 2024, 7:36 PM

Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin1002 for host doh7001.wikimedia.org with OS bookworm completed:

  • doh7001 (WARN)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Set boot media to disk
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202405021911_sukhe_4141526_doh7001.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
ssingh updated the task description. (Show Details)May 2 2024, 8:43 PM
gerritbot added a comment.May 3 2024, 7:27 AM

Change #1026729 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Add install7001 to site.pp

https://gerrit.wikimedia.org/r/1026729

gerritbot added a project: Patch-For-Review.May 3 2024, 7:27 AM
gerritbot added a comment.May 3 2024, 7:34 AM

Change #1026729 merged by Muehlenhoff:

[operations/puppet@production] Add install7001 to site.pp

https://gerrit.wikimedia.org/r/1026729

gerritbot added a comment.May 3 2024, 7:35 AM

Change #1026786 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Add bast7001 to site.pp

https://gerrit.wikimedia.org/r/1026786

gerritbot added a comment.May 3 2024, 7:37 AM

Change #1026786 merged by Muehlenhoff:

[operations/puppet@production] Add bast7001 to site.pp

https://gerrit.wikimedia.org/r/1026786

gerritbot added a comment.May 3 2024, 7:39 AM

Change #1026787 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] preseed: Extend globbing for bast and prometheus to cover magru

https://gerrit.wikimedia.org/r/1026787

gerritbot added a comment.May 3 2024, 7:43 AM

Change #1026787 merged by Muehlenhoff:

[operations/puppet@production] preseed: Extend globbing for bast and prometheus to cover magru

https://gerrit.wikimedia.org/r/1026787

Maintenance_bot removed a project: Patch-For-Review.May 3 2024, 8:31 AM
Stashbot added a comment.May 3 2024, 10:32 AM

Mentioned in SAL (#wikimedia-operations) [2024-05-03T10:32:03Z] <jmm@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "add bast7001 - jmm@cumin2002 - T364016"

Stashbot added a comment.May 3 2024, 10:33 AM

Mentioned in SAL (#wikimedia-operations) [2024-05-03T10:33:43Z] <jmm@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "add bast7001 - jmm@cumin2002 - T364016"

gerritbot added a comment.May 3 2024, 10:35 AM

Change #1026824 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Make bast7001 a bastion

https://gerrit.wikimedia.org/r/1026824

gerritbot added a project: Patch-For-Review.May 3 2024, 10:35 AM
gerritbot added a comment.May 3 2024, 10:42 AM

Change #1026824 merged by Muehlenhoff:

[operations/puppet@production] Make bast7001 a bastion

https://gerrit.wikimedia.org/r/1026824

ops-monitoring-bot added a comment.May 3 2024, 10:58 AM

Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin1002 for host durum7002.magru.wmnet with OS bookworm

ops-monitoring-bot added a comment.May 3 2024, 11:11 AM

Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin1002 for host doh7002.wikimedia.org with OS bookworm

Maintenance_bot removed a project: Patch-For-Review.May 3 2024, 11:30 AM
MoritzMuehlenhoff updated the task description. (Show Details)May 3 2024, 11:39 AM
ops-monitoring-bot added a comment.May 3 2024, 11:45 AM

Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin1002 for host durum7002.magru.wmnet with OS bookworm completed:

  • durum7002 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Set boot media to disk
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202405031127_sukhe_72384_durum7002.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.May 3 2024, 12:02 PM

Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin1002 for host doh7002.wikimedia.org with OS bookworm completed:

  • doh7002 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Set boot media to disk
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202405031141_sukhe_77196_doh7002.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ssingh updated the task description. (Show Details)May 3 2024, 1:24 PM
ssingh updated the task description. (Show Details)May 3 2024, 5:42 PM
gerritbot added a comment.May 6 2024, 6:35 AM

Change #1028229 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Make install7001 an installserver

https://gerrit.wikimedia.org/r/1028229

gerritbot added a project: Patch-For-Review.May 6 2024, 6:35 AM
gerritbot added a comment.May 6 2024, 6:37 AM

Change #1028229 merged by Muehlenhoff:

[operations/puppet@production] Make install7001 an installserver

https://gerrit.wikimedia.org/r/1028229

gerritbot added a comment.May 6 2024, 6:51 AM

Change #1028236 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[labs/private@master] Add dummy keytab for install7001

https://gerrit.wikimedia.org/r/1028236

gerritbot added a comment.May 6 2024, 6:54 AM

Change #1028236 merged by Muehlenhoff:

[labs/private@master] Add dummy keytab for install7001

https://gerrit.wikimedia.org/r/1028236

Muehlenhoff mentioned this in rLPRI5522af170141: Add dummy keytab for install7001.May 6 2024, 6:55 AM
gerritbot added a comment.May 6 2024, 7:03 AM

Change #1028245 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/dns@master] Enable install7001 as webproxy in magru

https://gerrit.wikimedia.org/r/1028245

gerritbot added a comment.May 6 2024, 7:12 AM

Change #1028245 merged by Muehlenhoff:

[operations/dns@master] Enable install7001 as webproxy in magru

https://gerrit.wikimedia.org/r/1028245

Maintenance_bot removed a project: Patch-For-Review.May 6 2024, 7:30 AM
MoritzMuehlenhoff updated the task description. (Show Details)May 6 2024, 9:56 AM
gerritbot added a comment.May 6 2024, 12:20 PM

Change #1028456 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] site: add prometheus7001

https://gerrit.wikimedia.org/r/1028456

gerritbot added a project: Patch-For-Review.May 6 2024, 12:20 PM
gerritbot added a comment.May 6 2024, 12:25 PM

Change #1028456 merged by Filippo Giunchedi:

[operations/puppet@production] site: add prometheus7001

https://gerrit.wikimedia.org/r/1028456

Maintenance_bot removed a project: Patch-For-Review.May 6 2024, 12:31 PM
gerritbot added a comment.May 6 2024, 12:41 PM

Change #1028464 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/dns@master] wmnet: add prometheus.svc.magru

https://gerrit.wikimedia.org/r/1028464

gerritbot added a project: Patch-For-Review.May 6 2024, 12:41 PM
gerritbot added a comment.May 6 2024, 1:59 PM

Change #1028501 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] site: provision prometheus7001 with insetup

https://gerrit.wikimedia.org/r/1028501

gerritbot added a comment.May 6 2024, 1:59 PM

Change #1028502 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] prometheus: use datacenters for snmp_exporter

https://gerrit.wikimedia.org/r/1028502

gerritbot added a comment.May 6 2024, 1:59 PM

Change #1028503 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] grafana: add magru prometheus

https://gerrit.wikimedia.org/r/1028503

gerritbot added a comment.May 6 2024, 1:59 PM

Change #1028504 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] trafficserver: add prometheus-magru.w.o

https://gerrit.wikimedia.org/r/1028504

gerritbot added a comment.May 6 2024, 2:01 PM

Change #1028501 merged by Filippo Giunchedi:

[operations/puppet@production] site: provision prometheus7001 with insetup

https://gerrit.wikimedia.org/r/1028501

ops-monitoring-bot added a comment.May 6 2024, 2:11 PM

cookbooks.sre.hosts.decommission executed by filippo@cumin1002 for hosts: prometheus7001.magru.wmnet

  • prometheus7001.magru.wmnet (WARN)
    • Host not found on Icinga, unable to downtime it
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster magru02 to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster magru02 to Netbox
fgiunchedi added subscribers: Muehlenhoff, fgiunchedi.May 6 2024, 2:43 PM

I've tried installing prometheus7001 today with help from @Muehlenhoff although there's no console and some pxe/tftp interaction with install7001 is suspected. I'll hold off further steps for now until VMs can be installed

gerritbot added a comment.May 7 2024, 7:10 AM

Change #1028707 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Revert "hiera: update installserver for magru"

https://gerrit.wikimedia.org/r/1028707

gerritbot added a comment.May 7 2024, 7:25 AM

Change #1028707 merged by Muehlenhoff:

[operations/puppet@production] Revert "hiera: update installserver for magru"

https://gerrit.wikimedia.org/r/1028707

ops-monitoring-bot added a comment.May 7 2024, 8:00 AM

cookbooks.sre.hosts.decommission executed by jmm@cumin2002 for hosts: install7001.wikimedia.org

  • install7001.wikimedia.org (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster magru01 to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster magru01 to Netbox
gerritbot added a comment.May 7 2024, 8:14 AM

Change #1028502 merged by Filippo Giunchedi:

[operations/puppet@production] prometheus: use datacenters for snmp_exporter

https://gerrit.wikimedia.org/r/1028502

gerritbot added a comment.May 7 2024, 8:23 AM

Change #1028759 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] prometheus: assemble snmp.yml when updating modules

https://gerrit.wikimedia.org/r/1028759

gerritbot added a comment.May 7 2024, 8:27 AM

Change #1028759 merged by Filippo Giunchedi:

[operations/puppet@production] prometheus: assemble snmp.yml when updating modules

https://gerrit.wikimedia.org/r/1028759

ops-monitoring-bot added a comment.May 7 2024, 10:28 AM

cookbooks.sre.hosts.decommission executed by filippo@cumin1002 for hosts: prometheus7001.magru.wmnet

  • prometheus7001.magru.wmnet (WARN)
    • Host not found on Icinga, unable to downtime it
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster magru02 to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster magru02 to Netbox
ops-monitoring-bot added a comment.May 7 2024, 1:40 PM

cookbooks.sre.hosts.decommission executed by filippo@cumin1002 for hosts: prometheus7001.magru.wmnet

  • prometheus7001.magru.wmnet (WARN)
    • Host not found on Icinga, unable to downtime it
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster magru02 to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster magru02 to Netbox
gerritbot added a comment.May 7 2024, 2:35 PM

Change #1028848 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] Revert "site: provision prometheus7001 with insetup"

https://gerrit.wikimedia.org/r/1028848

gerritbot added a comment.May 7 2024, 2:36 PM

Change #1028848 merged by Filippo Giunchedi:

[operations/puppet@production] Revert "site: provision prometheus7001 with insetup"

https://gerrit.wikimedia.org/r/1028848

gerritbot added a comment.May 7 2024, 2:39 PM

Change #1028464 merged by Filippo Giunchedi:

[operations/dns@master] wmnet: add prometheus.svc.magru

https://gerrit.wikimedia.org/r/1028464

Stashbot added a comment.May 7 2024, 2:50 PM

Mentioned in SAL (#wikimedia-operations) [2024-05-07T14:50:19Z] <godog> silence site=magru alerts during prometheus7001 - T364016

gerritbot added a comment.May 7 2024, 2:51 PM

Change #1028503 merged by Filippo Giunchedi:

[operations/puppet@production] grafana: add magru prometheus

https://gerrit.wikimedia.org/r/1028503

gerritbot added a comment.May 7 2024, 2:54 PM

Change #1028504 merged by Filippo Giunchedi:

[operations/puppet@production] trafficserver: add prometheus-magru.w.o

https://gerrit.wikimedia.org/r/1028504

Stashbot added a comment.May 7 2024, 3:13 PM

Mentioned in SAL (#wikimedia-operations) [2024-05-07T15:13:17Z] <godog> remove accidentally set site!=magru silence, add site=magru silence instead - T364016

Maintenance_bot removed a project: Patch-For-Review.May 7 2024, 3:31 PM
cmooney closed subtask T364454: mgmt ssh access for prometheus hosts in magru as Resolved.May 8 2024, 4:05 PM
fgiunchedi updated the task description. (Show Details)May 10 2024, 9:22 AM

From my POV prometheus in magru is live and working, see also https://prometheus-magru.wikimedia.org/

MoritzMuehlenhoff closed this task as Resolved.May 10 2024, 10:12 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL