Page MenuHomePhabricator
Create Task
Maniphest T372418

Put the alert1002 and alert2002 hosts in production
Closed, ResolvedPublic
Actions

Description

Overview

This task tracks the hardware migration of the alert1002 and alert2002 hosts to replace the alert1001 and alert2001 hosts.

The four host use Debian Bookworm so it's expected for the Puppet role to work as is.

Proposed solution:

Stage 1: Prepare hosts

  1. Add the alert1002, and the alert2002 hosts to the Acme Chief list of authorized domains.
    1. Merge Gerrit patch #1064107 alert: Add the alertx002 hosts to acme chief
  2. Apply the alerting_host role for the alert1002, and alert2002 hosts.
    1. Merge Gerrit patch #1062444 - alert: Ensure the alert*002 hosts use the alerting_host role
    2. Run Puppet on the alert[12]002 hosts: sudo cumin 'alert*[02]*' 'run-puppet-agent'
  3. Arm the keyholder agent in the alert1002, and alert2002 hosts.
    1. Use the metamonitor passphrase pwstore/pw.git/metamonitor-key-passphrase
    2. Ensure the keyholder agent is armed sudo cumin 'alert*002wikimedia.org,' 'keyholder status'
  4. Verify the IP addresses are propagated across the infrastructure
  5. Add the new hosts to the Prometheus Blackbox exporter list
    1. Merge Gerrit patch #1064097 - alert: Add the alertx002 hosts to Prometheus blackbox exporter

Stage 2: Enable hosts as passive alertmanagers

  1. Allow connections from the alert[12]002 addresses.
    1. Merge Gerrit patch #1064818 - alert: Allow connections from the alertx002 addresses
    2. Merge Gerrit patch #1064821 - alert: Allow Apache2 connections for the alertx002 hosts
    3. Run Puppet on the alert hosts: sudo cumin 'alert*' 'run-puppet-agent'
  2. Add the alert1002, and alert2002 hosts as Icinga and Alertmanager partners to work as passive hosts.
    1. Merge Gerrit patch #1064820 - alert: Add the alertx002 hosts as Icinga and AM partners
    2. Run Puppet on the alert hosts: sudo cumin 'alert*' 'run-puppet-agent'
  3. Enable the alert[12]002 hosts as alertmanagers
    1. Merge Gerrit patch #1072318 - alert: Enable the alertx002 hosts as alertmanagers
  4. Verify the hosts are working as intended as standby hosts (e.g. no puppet or unit failures)

Stage 3: Make alert2002 the active alertmanager host

  1. Disable meta-monitoring for the alert hosts.
    1. SSH as root into wikitech-static.wikimedia.org with the metamonitor-key-passphrase.
    2. Comment the following crontab entries to stop meta-monitoring against both alert hosts:
      • */2 * * * * /usr/bin/systemd-cat -t "check_icinga" /usr/local/bin/check_icinga alert1001.wikimedia.org
      • */2 * * * * /usr/bin/systemd-cat -t "check_icinga" /usr/local/bin/check_icinga alert2001.wikimedia.org
  2. Stop services in the alert1001 host.
  3. Make alert2002 the active host.
    1. Merge Gerrit patch #1071700 - alert: Failover from alert1001 to alert2002
    2. Run Puppet on the alert hosts: sudo cumin 'alert*' 'run-puppet-agent'
    3. Merge Gerrit patch #1072326 - alert: Resolve alerts DNS queries to alert2002
    4. Update DNS records: $ sudo cumin 'dns1004.wikimedia.org' 'sudo -i authdns-update'
  4. Ensure services work as expected.
  5. Enable metamonitoring for the alert1001, and alert2002 hosts.
    1. SSH as root into wikitech-static.wikimedia.org with the metamonitor-key-passphrase.
    2. Uncomment the following crontab entries to enable meta-monitoring for the alert1001 host, and add meta-monitoring for the alert2002 host.
      • # */2 * * * * /usr/bin/systemd-cat -t "check_icinga" /usr/local/bin/check_icinga alert1001.wikimedia.org
      • # */2 * * * * /usr/bin/systemd-cat -t "check_icinga" /usr/local/bin/check_icinga alert2002.wikimedia.org

Stage 4: Make alert1002 the active alertmanager host

  1. Disable meta-monitoring for the alert hosts.
    1. SSH as root into wikitech-static.wikimedia.org with the metamonitor-key-passphrase.
    2. Comment the following crontab entries to stop meta-monitoring against both alert hosts:
      • */2 * * * * /usr/bin/systemd-cat -t "check_icinga" /usr/local/bin/check_icinga alert1001.wikimedia.org
      • */2 * * * * /usr/bin/systemd-cat -t "check_icinga" /usr/local/bin/check_icinga alert2002.wikimedia.org
  2. Stop services in the alert2002 host.
  3. Make alert1002 the active host.
    1. Merge Gerrit patch #1071701 - alert: Failover from alert1001 to alert2002
    2. Run Puppet on the alert hosts: sudo cumin 'alert*' 'run-puppet-agent'
    3. Merge Gerrit patch #1063078 - alert: Resolve alerts DNS queries to alert1002
    4. Update DNS records: $ sudo cumin 'dns1004.wikimedia.org' 'sudo -i authdns-update'
  4. Ensure services work as expected.
  5. Enable metamonitoring for the alert1002, and alert2002 hosts.
    1. SSH as root into wikitech-static.wikimedia.org with the metamonitor-key-passphrase.
    2. Uncomment the following crontab entries to enable meta-monitoring for the alert2002 host, and add meta-monitoring for the alert1002 host.
      • # */2 * * * * /usr/bin/systemd-cat -t "check_icinga" /usr/local/bin/check_icinga alert1002.wikimedia.org
      • # */2 * * * * /usr/bin/systemd-cat -t "check_icinga" /usr/local/bin/check_icinga alert2002.wikimedia.org

Step 5: Cleanup

  1. Update hostnames for alertmanager tests
    1. Merge Gerrit patch #1063235 - alert: Update alertmanager tests hostnames

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
alert: Failover from alert2002 to alert1002operations/puppetproduction+4 -4
alert: Resolve alerts DNS queries to alert1002operations/dnsmaster+5 -5
alert: Failover from alert2002 to alert1002operations/puppetproduction+5 -5
alert: Resolve alerts DNS queries to alert2002operations/dnsmaster+5 -5
alert: Failover from alert1001 to alert2002operations/puppetproduction+5 -5
alert: Enable the alert[12]002 hosts as alertmanagersoperations/puppetproduction+2 -0
alert: Enable the alert[12]002 hosts as alertmanagersoperations/puppetproduction+2 -0
alert: Failover from alert1001 to alert2002operations/puppetproduction+4 -4
alert: Resolve alerts DNS queries to alert2002operations/dnsmaster+5 -5
alert: Update alertmanager tests hostnamesoperations/puppetproduction+19 -19
alert: Add the alert[12]002 hosts as Icinga and AM partnersoperations/puppetproduction+4 -0
alert: Allow connections from the alert[12]002 addressesoperations/puppetproduction+4 -0
alert: Allow Apache2 connections for the alert[12]002 hostsoperations/puppetproduction+4 -0
alert: Ensure alert1002 is the active alert hostoperations/puppetproduction+5 -5
alert: Add the alert[12]002 hosts to puppet realmoperations/puppetproduction+23 -2
alert: Add the alert[12]002 hosts to Prometheus blackbox exporteroperations/puppetproduction+8 -0
alert: Ensure the alert[12]002 hosts use the alerting_host roleoperations/puppetproduction+1 -1
alert: Add the alert[12]002 hosts to acme chiefoperations/puppetproduction+3 -2
alert: Add alert2002 as active host and add new partnersoperations/puppetproduction+9 -5
alert: Add the alert[12]002 hosts as alertmanagersoperations/puppetproduction+14 -0
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
fgiunchedi added a comment.Aug 19 2024, 8:36 AM
In T372418#10065217, @andrea.denisse wrote:
In T372418#10063876, @fgiunchedi wrote:

Thank you @andrea.denisse for getting the ball rolling on this! The solution as-is works well for failing over hosts that are already running alerting_host role. In other words, we should be applying alerting_host role first, including adding the new hostnames and IP addresses wherever they appear in puppet.git. Then apply the procedure above to in turn put the new hosts in service, and finally decom the old hosts, what do you think ?

Thanks for your suggestions, the plan you propose sounds good to me, I'll update the task description.

Thank you, the procedure is definitely coming together! I've re-read the task description and I think we should be recognise/split the procedure in at least these parts:

  • We stage the new hosts with alerting_host role, add to puppet, arm keyholder, etc. The hosts are not active in icinga/alertmanager yet (i.e. alertmanagers and profile::icinga::active_host and profile::alertmanager::active_host variables)
  • Verify the ip addresses are propagated across the infrastructure
    • The ip addresses for alert hosts are deployed by homer to network devices in production
    • Ditto for fundraising firewalls, they do have the alert hosts IP addresses in them, though I'm not sure how to add/remove them? cc @ayounsi for this and the homer point above
  • Verify the hosts are working as intended as standby hosts (e.g. no puppet or unit failures)

This might take more than one day, and it is the "stage" phase. After this, we can proceed with making the hosts active. I think we should test making alert2002 active too, to avoid surprises later, therefore:

  • Make alert2002 active, verify functionality (as per your procedure, there will be more steps)
  • Make alert1002 active, verify functionality (ditto)

I'd also like to know more about the "migrate data" step, what data/commands/etc are involved ?

ayounsi added subscribers: Dwisehaupt, Jgreen.Aug 19 2024, 8:51 AM

Ditto for fundraising firewalls, they do have the alert hosts IP addresses in them, though I'm not sure how to add/remove them? cc @ayounsi for this and the homer point above

Homer is generated automatically from Netbox data using the capirca script: https://netbox.wikimedia.org/extras/scripts/1/jobs/ then a homer run is needed to actually update the network devices from using that data.
For the fundraising hosts/network you need to ping @Jgreen and @Dwisehaupt

Dwisehaupt added a comment.Aug 19 2024, 4:06 PM
In T372418#10072313, @ayounsi wrote:

Ditto for fundraising firewalls, they do have the alert hosts IP addresses in them, though I'm not sure how to add/remove them? cc @ayounsi for this and the homer point above

Homer is generated automatically from Netbox data using the capirca script: https://netbox.wikimedia.org/extras/scripts/1/jobs/ then a homer run is needed to actually update the network devices from using that data.
For the fundraising hosts/network you need to ping @Jgreen and @Dwisehaupt

I have added them to our PFW config and created T372520 for the deployment.

Dwisehaupt added a comment.Aug 19 2024, 4:20 PM

I have put in the iptables and nagios config changes for the new hosts into the fundraising puppet repos. The iptables changes will be rolling out today as we do updates. We can then pull alert1001 and alert2001 when you all are ready.

ayounsi added a comment.Aug 20 2024, 3:15 PM

I have added them to our PFW config and created T372520 for the deployment.

Deployed.

andrea.denisse updated the task description. (Show Details)Aug 20 2024, 4:03 PM
gerritbot added a comment.Aug 20 2024, 8:38 PM

Change #1064097 had a related patch set uploaded (by Andrea Denisse; author: Andrea Denisse):

[operations/puppet@production] alert: Add the alert[12]002 hosts to Prometheus blackbox exporter

https://gerrit.wikimedia.org/r/1064097

andrea.denisse updated the task description. (Show Details)Aug 20 2024, 8:46 PM
gerritbot added a comment.Aug 20 2024, 9:15 PM

Change #1064107 had a related patch set uploaded (by Andrea Denisse; author: Andrea Denisse):

[operations/puppet@production] alert: Add the alert[12]002 hosts to acme chief

https://gerrit.wikimedia.org/r/1064107

andrea.denisse updated the task description. (Show Details)Aug 20 2024, 9:18 PM
andrea.denisse set the point value for this task to 1.
ayounsi unsubscribed.Aug 21 2024, 6:57 AM
fgiunchedi added a subscriber: ayounsi.Aug 21 2024, 8:12 AM
In T372418#10074204, @Dwisehaupt wrote:

I have put in the iptables and nagios config changes for the new hosts into the fundraising puppet repos. The iptables changes will be rolling out today as we do updates. We can then pull alert1001 and alert2001 when you all are ready.

In T372418#10078011, @ayounsi wrote:

I have added them to our PFW config and created T372520 for the deployment.

Deployed.

Thank you so much for your quick action here! Appreciate it

ayounsi unsubscribed.Aug 21 2024, 8:50 AM
gerritbot added a comment.Aug 21 2024, 2:59 PM

Change #1064107 merged by Andrea Denisse:

[operations/puppet@production] alert: Add the alert[12]002 hosts to acme chief

https://gerrit.wikimedia.org/r/1064107

gerritbot added a comment.Aug 21 2024, 5:40 PM

Change #1062444 merged by Andrea Denisse:

[operations/puppet@production] alert: Ensure the alert[12]002 hosts use the alerting_host role

https://gerrit.wikimedia.org/r/1062444

gerritbot added a comment.Aug 21 2024, 6:02 PM

Change #1064097 merged by Andrea Denisse:

[operations/puppet@production] alert: Add the alert[12]002 hosts to Prometheus blackbox exporter

https://gerrit.wikimedia.org/r/1064097

gerritbot added a comment.Aug 22 2024, 4:57 PM

Change #1064806 had a related patch set uploaded (by Andrea Denisse; author: Andrea Denisse):

[operations/puppet@production] alert: Enable the alert[12]002 hosts as alertmanagers

https://gerrit.wikimedia.org/r/1064806

andrea.denisse updated the task description. (Show Details)Aug 22 2024, 4:59 PM
andrea.denisse updated the task description. (Show Details)
gerritbot added a comment.Aug 22 2024, 5:59 PM

Change #1064818 had a related patch set uploaded (by Andrea Denisse; author: Andrea Denisse):

[operations/puppet@production] alert: Allow connections from the alert[12]002 addresses

https://gerrit.wikimedia.org/r/1064818

andrea.denisse updated the task description. (Show Details)Aug 22 2024, 6:04 PM
andrea.denisse removed the point value 1 for this task.
gerritbot added a comment.Aug 22 2024, 6:10 PM

Change #1064820 had a related patch set uploaded (by Andrea Denisse; author: Andrea Denisse):

[operations/puppet@production] alert: Add the alert[12]002 hosts as Icinga and AM partners

https://gerrit.wikimedia.org/r/1064820

andrea.denisse updated the task description. (Show Details)Aug 22 2024, 6:12 PM
gerritbot added a comment.Aug 22 2024, 6:19 PM

Change #1064821 had a related patch set uploaded (by Andrea Denisse; author: Andrea Denisse):

[operations/puppet@production] alert: Allow Apache2 connections for the alert[12]002 hosts

https://gerrit.wikimedia.org/r/1064821

andrea.denisse updated the task description. (Show Details)Aug 22 2024, 6:21 PM
gerritbot added a comment.Aug 22 2024, 6:38 PM

Change #1064826 had a related patch set uploaded (by Andrea Denisse; author: Andrea Denisse):

[operations/puppet@production] alert: Failover from alert1001 to alert2002

https://gerrit.wikimedia.org/r/1064826

gerritbot added a comment.Aug 22 2024, 6:41 PM

Change #1063063 abandoned by Andrea Denisse:

[operations/puppet@production] alert: Add the alert[12]002 hosts to puppet realm

Reason:

Abandoning in favor of smaller patches.

https://gerrit.wikimedia.org/r/1063063

gerritbot added a comment.Aug 22 2024, 6:41 PM

Change #1063075 abandoned by Andrea Denisse:

[operations/puppet@production] alert: Ensure alert1002 is the active alert host

Reason:

Abandoning in favor of smaller patches.

https://gerrit.wikimedia.org/r/1063075

gerritbot added a comment.Aug 22 2024, 6:46 PM

Change #1064828 had a related patch set uploaded (by Andrea Denisse; author: Andrea Denisse):

[operations/puppet@production] alert: Failover from alert2002 to alert1002

https://gerrit.wikimedia.org/r/1064828

gerritbot added a comment.Aug 23 2024, 5:41 PM

Change #1065258 had a related patch set uploaded (by Andrea Denisse; author: Andrea Denisse):

[operations/dns@master] alert: Resolve alerts DNS queries to alert2002

https://gerrit.wikimedia.org/r/1065258

andrea.denisse updated the task description. (Show Details)Aug 23 2024, 5:50 PM
gerritbot added a comment.Aug 23 2024, 6:00 PM

Change #1064821 merged by Andrea Denisse:

[operations/puppet@production] alert: Allow Apache2 connections for the alert[12]002 hosts

https://gerrit.wikimedia.org/r/1064821

gerritbot added a comment.Aug 23 2024, 6:01 PM

Change #1064818 merged by Andrea Denisse:

[operations/puppet@production] alert: Allow connections from the alert[12]002 addresses

https://gerrit.wikimedia.org/r/1064818

fgiunchedi updated the task description. (Show Details)Aug 26 2024, 2:28 PM
gerritbot added a comment.Aug 26 2024, 6:26 PM

Change #1064820 merged by Andrea Denisse:

[operations/puppet@production] alert: Add the alert[12]002 hosts as Icinga and AM partners

https://gerrit.wikimedia.org/r/1064820

gerritbot added a comment.Aug 27 2024, 11:27 PM

Change #1063235 merged by Andrea Denisse:

[operations/puppet@production] alert: Update alertmanager tests hostnames

https://gerrit.wikimedia.org/r/1063235

andrea.denisse updated the task description. (Show Details)Aug 29 2024, 7:41 PM
gerritbot added a comment.Sep 3 2024, 1:49 PM

Change #1064806 merged by Andrea Denisse:

[operations/puppet@production] alert: Enable the alert[12]002 hosts as alertmanagers

https://gerrit.wikimedia.org/r/1064806

Stashbot added a comment.Sep 3 2024, 2:00 PM

Mentioned in SAL (#wikimedia-operations) [2024-09-03T14:00:50Z] <denisse> Disabling meta-monitoring for the alert hosts - T372418

Stashbot added a comment.Sep 3 2024, 2:03 PM

Mentioned in SAL (#wikimedia-operations) [2024-09-03T14:03:43Z] <denisse> Stopping services in the alert1001 host - T372418

gerritbot added a comment.Sep 3 2024, 2:06 PM

Change #1064826 merged by Andrea Denisse:

[operations/puppet@production] alert: Failover from alert1001 to alert2002

https://gerrit.wikimedia.org/r/1064826

Stashbot added a comment.Sep 3 2024, 2:06 PM

Mentioned in SAL (#wikimedia-operations) [2024-09-03T14:06:42Z] <denisse> Failing over to alert2002 - T372418

Stashbot added a comment.Sep 3 2024, 2:10 PM

Mentioned in SAL (#wikimedia-operations) [2024-09-03T14:10:35Z] <denisse> Resolve DNS queries to alert2002 - T372418

gerritbot added a comment.Sep 3 2024, 2:12 PM

Change #1065258 merged by Andrea Denisse:

[operations/dns@master] alert: Resolve alerts DNS queries to alert2002

https://gerrit.wikimedia.org/r/1065258

Stashbot added a comment.Sep 3 2024, 3:58 PM

Mentioned in SAL (#wikimedia-operations) [2024-09-03T15:58:47Z] <denisse> Reverting back to alert1001 - T372418

Stashbot added a comment.Sep 3 2024, 3:59 PM

Mentioned in SAL (#wikimedia-operations) [2024-09-03T15:59:06Z] <denisse> Enabling meta monitoring for alert[12]001 - T372418

BCornwall subscribed.Sep 3 2024, 4:13 PM
Jgreen added a comment.Sep 3 2024, 5:26 PM

Fundraising servers use nsca exclusively, so we've configured them to send to all four alert[12]00[12] hosts while things are in flux. I'll try to keep an eye on this task but please let us know if when we should remove extraneous hosts from the list.

fgiunchedi added a comment.Sep 4 2024, 9:02 AM
In T372418#10114221, @Jgreen wrote:

Fundraising servers use nsca exclusively, so we've configured them to send to all four alert[12]00[12] hosts while things are in flux. I'll try to keep an eye on this task but please let us know if when we should remove extraneous hosts from the list.

Thank you @Jgreen, we'll keep you posted once we're done with the old hosts and have fully switched to new hosts.

Dzahn closed subtask T373980: Hosts using nftables are not reachable via ssh from alert[12]002. Reboot needed. as Resolved.Sep 6 2024, 5:43 PM
fgiunchedi added a comment.Sep 9 2024, 8:15 AM

With https://gerrit.wikimedia.org/r/c/operations/puppet/+/1060516 merged now corto's profile::corto::active_host hiera setting needs to be changed on failover, @andrea.denisse FYI

fgiunchedi added a comment.Sep 9 2024, 12:13 PM

Also please note that we're blocked on T374340 before the failover can happen

andrea.denisse closed subtask T374340: No ntp query ACL for new alert hosts as Resolved.Sep 9 2024, 6:19 PM
andrea.denisse added a comment.Sep 9 2024, 6:30 PM

@fgiunchedi Thanks for the heads-up regarding corto, I'm sending a patch for that.

T374340 is resolved now, kudos to Sukhbir for his help with it!

gerritbot added a comment.Sep 9 2024, 8:13 PM

Change #1071700 had a related patch set uploaded (by Andrea Denisse; author: Andrea Denisse):

[operations/puppet@production] alert: Make alert2002 the active host for corto

https://gerrit.wikimedia.org/r/1071700

gerritbot added a comment.Sep 9 2024, 8:14 PM

Change #1071701 had a related patch set uploaded (by Andrea Denisse; author: Andrea Denisse):

[operations/puppet@production] alert: Make alert1002 the active host for corto

https://gerrit.wikimedia.org/r/1071701

andrea.denisse updated the task description. (Show Details)Sep 9 2024, 8:17 PM
gerritbot added a comment.Sep 11 2024, 11:56 PM

Change #1072318 had a related patch set uploaded (by Andrea Denisse; author: Andrea Denisse):

[operations/puppet@production] alert: Enable the alert[12]002 hosts as alertmanagers

https://gerrit.wikimedia.org/r/1072318

andrea.denisse updated the task description. (Show Details)Sep 12 2024, 12:38 AM
gerritbot added a comment.Sep 12 2024, 12:39 AM

Change #1072326 had a related patch set uploaded (by Andrea Denisse; author: Andrea Denisse):

[operations/dns@master] alert: Resolve alerts DNS queries to alert2002

https://gerrit.wikimedia.org/r/1072326

andrea.denisse updated the task description. (Show Details)Sep 12 2024, 12:40 AM
andrea.denisse updated the task description. (Show Details)
Stashbot added a comment.Sep 12 2024, 2:01 PM

Mentioned in SAL (#wikimedia-operations) [2024-09-12T14:01:45Z] <denisse> Enable the alert[12]002 hosts as alertmanagers - T372418

gerritbot added a comment.Sep 12 2024, 2:01 PM

Change #1072318 merged by Andrea Denisse:

[operations/puppet@production] alert: Enable the alert[12]002 hosts as alertmanagers

https://gerrit.wikimedia.org/r/1072318

Stashbot added a comment.Sep 12 2024, 2:02 PM

Mentioned in SAL (#wikimedia-operations) [2024-09-12T14:02:44Z] <denisse> Disable meta-monitoring for the alert hosts - T372418

Stashbot added a comment.Sep 12 2024, 2:04 PM

Mentioned in SAL (#wikimedia-operations) [2024-09-12T14:04:25Z] <denisse> Make alert2002 the active host - T372418

gerritbot added a comment.Sep 12 2024, 2:04 PM

Change #1071700 merged by Andrea Denisse:

[operations/puppet@production] alert: Failover from alert1001 to alert2002

https://gerrit.wikimedia.org/r/1071700

gerritbot added a comment.Sep 12 2024, 2:07 PM

Change #1072326 merged by Andrea Denisse:

[operations/dns@master] alert: Resolve alerts DNS queries to alert2002

https://gerrit.wikimedia.org/r/1072326

fgiunchedi mentioned this in T374711: keyholder-proxy doesn't restart on config change.Sep 13 2024, 12:33 PM
Stashbot added a comment.Sep 18 2024, 3:00 PM

Mentioned in SAL (#wikimedia-operations) [2024-09-18T15:00:50Z] <denisse> Disable meta-monitoring for the alert hosts - T372418

Stashbot added a comment.Sep 18 2024, 3:01 PM

Mentioned in SAL (#wikimedia-operations) [2024-09-18T15:01:44Z] <denisse> Make alert1002 the active host - T372418

gerritbot added a comment.Sep 18 2024, 3:02 PM

Change #1071701 merged by Andrea Denisse:

[operations/puppet@production] alert: Failover from alert2002 to alert1002

https://gerrit.wikimedia.org/r/1071701

Stashbot added a comment.Sep 18 2024, 3:08 PM

Mentioned in SAL (#wikimedia-operations) [2024-09-18T15:08:44Z] <denisse> Resolve alerts DNS queries to alert1002 - T372418

gerritbot added a comment.Sep 18 2024, 3:08 PM

Change #1063078 merged by Andrea Denisse:

[operations/dns@master] alert: Resolve alerts DNS queries to alert1002

https://gerrit.wikimedia.org/r/1063078

Stashbot added a comment.Sep 18 2024, 3:21 PM

Mentioned in SAL (#wikimedia-operations) [2024-09-18T15:21:08Z] <denisse> Enable metamonitoring for the alert1002, and alert2002 hosts - T372418

andrea.denisse closed this task as Resolved.Sep 18 2024, 3:24 PM
andrea.denisse mentioned this in T375138: The prometheus-alertmanager service does not start at boot.Sep 18 2024, 9:09 PM
andrea.denisse added a comment.Sep 18 2024, 9:23 PM

Hi @Jgreen and @Dwisehaupt , we have finished the fail over and plan to decommission the alert1001, and the alert2001 hosts next week, progress is being tracked in T372607 .

Once the decommission is done I'll ping you so the IP addresses of the old hosts can be removed from the PFW configuration, thank you!

andrea.denisse mentioned this in T375139: The corto service fails to start after the alert hosts failover.Sep 18 2024, 9:25 PM
andrea.denisse mentioned this in T375143: Sirenbot keeps joining and parting channels after failing over to alert1002.Sep 18 2024, 9:57 PM
andrea.denisse closed subtask T375138: The prometheus-alertmanager service does not start at boot as Resolved.Sep 19 2024, 4:25 PM
andrea.denisse closed subtask T375139: The corto service fails to start after the alert hosts failover as Resolved.Sep 23 2024, 8:32 PM
andrea.denisse added a comment.Oct 2 2024, 4:58 PM

Hi @Jgreen and @Dwisehaupt , we finished the alert1001, and alert2001` hosts decommission, the IP addresses of the old hosts can now be removed from the PFW configuration, thank you!

Jgreen added a comment.Oct 2 2024, 6:57 PM
In T372418#10197106, @andrea.denisse wrote:

Hi @Jgreen and @Dwisehaupt , we finished the alert1001, and alert2001` hosts decommission, the IP addresses of the old hosts can now be removed from the PFW configuration, thank you!

Done! Thank you.

VRiley-WMF closed subtask T372607: Decommission the alert1001 and alert2001 hosts as Resolved.Oct 8 2024, 7:25 PM
gerritbot added a comment.Oct 9 2024, 5:37 PM

Change #1064828 abandoned by Andrea Denisse:

[operations/puppet@production] alert: Failover from alert2002 to alert1002

https://gerrit.wikimedia.org/r/1064828

andrea.denisse changed the status of subtask T375143: Sirenbot keeps joining and parting channels after failing over to alert1002 from Open to In Progress.Oct 14 2024, 9:13 PM
andrea.denisse closed subtask T375143: Sirenbot keeps joining and parting channels after failing over to alert1002 as Resolved.Oct 23 2024, 3:23 PM
andrea.denisse reopened subtask T375143: Sirenbot keeps joining and parting channels after failing over to alert1002 as Open.Oct 24 2024, 5:10 PM
andrea.denisse changed the status of subtask T375143: Sirenbot keeps joining and parting channels after failing over to alert1002 from Open to In Progress.Oct 24 2024, 5:21 PM
andrea.denisse closed subtask T375143: Sirenbot keeps joining and parting channels after failing over to alert1002 as Resolved.Oct 24 2024, 11:14 PM
• lmata moved this task from Inbox to Done on the SRE Observability (FY2024/2025-Q1) board.Nov 5 2024, 5:02 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits