Page MenuHomePhabricator
Create Task
Maniphest T247966

Migrate role::alerting_host to Buster
Closed, ResolvedPublic
Actions

Description

As per title, all hosts running Icinga should be on Buster. Ideally by FQ2 FY20-21.

root@cumin1001:~# cumin 'P{O:alerting_host} and not P{F:lsbdistcodename = buster}'
2 hosts will be targeted:
icinga[1001,2001].wikimedia.org
DRY-RUN mode enabled, aborting

Action plan:

  • Apply the role to alert2001 (new icinga host, T252032)
  • Verify all things are working as expected, i.e. state matches icinga1001
  • alert latency is acceptable, cpu load is lower (hw is more powerful) etc
  • Make sure metamonitoring checks for alert[12]001 are passing
  • Switchover icinga1001 -> alert1001 (enable notifications for alert1001, switch DNS)
  • Enable meta monitoring notifications for alert1001 and alert2001 (via root crontab on wikitech-static)

Details

SubjectRepoBranchLines +/-
remove alerting_host role from icinga[12]001operations/puppetproduction+4 -34
icinga: switch active server from icinga1001 to alert1001operations/puppetproduction+2 -2
dns: point icinga CNAMEs to alert1001operations/dnsmaster+3 -3
icinga: make sure update-etcd-mw-config-lastindex is enabledoperations/puppetproduction+1 -1
profile: refactor logmsgbot to follow Icinga failoveroperations/puppetproduction+12 -5
hieradata: add alert[12]001 to monitoring hostsoperations/puppetproduction+8 -1
acme_cheif: add alert[12]001 SNI and permit to fetch icinga certoperations/puppetproduction+3 -0
alerting_host: assign alert[12]001 role::alerting_hostoperations/puppetproduction+8 -10
Customize query in gerrit

Related Objects
Search...

Event Timeline

fgiunchedi created this task.Mar 18 2020, 12:38 PM
fgiunchedi edited projects, added observability; removed SRE.
fgiunchedi removed a project: Epic.Mar 19 2020, 10:15 AM
herron subscribed.Mar 23 2020, 3:34 PM
fgiunchedi moved this task from Inbox to Backlog on the observability board.Apr 6 2020, 12:33 PM
fgiunchedi added a comment.Jul 10 2020, 9:03 AM

I tested the role on Buster in WMCS and it appears to be working as expected!

fgiunchedi updated the task description. (Show Details)Jul 10 2020, 9:08 AM
gerritbot added a comment.Aug 4 2020, 4:28 PM

Change 618345 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] alerting_host: assign alert[12]001 role::alerting_host

https://gerrit.wikimedia.org/r/618345

gerritbot added a project: Patch-For-Review.Aug 4 2020, 4:28 PM
herron moved this task from Backlog to In progress on the observability board.Aug 4 2020, 4:45 PM
gerritbot added a comment.Aug 5 2020, 3:46 PM

Change 618345 merged by Herron:
[operations/puppet@production] alerting_host: assign alert[12]001 role::alerting_host

https://gerrit.wikimedia.org/r/618345

gerritbot added a comment.Aug 5 2020, 4:04 PM

Change 618545 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] acme_cheif: permit alert[12]001 to fetch icinga cert

https://gerrit.wikimedia.org/r/618545

gerritbot added a comment.Aug 5 2020, 6:02 PM

Change 618545 merged by Herron:
[operations/puppet@production] acme_cheif: add alert[12]001 SNI and permit to fetch icinga cert

https://gerrit.wikimedia.org/r/618545

gerritbot added a comment.Aug 6 2020, 9:16 AM

Change 618719 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] hieradata: add alert[12]001 to monitoring hosts

https://gerrit.wikimedia.org/r/618719

gerritbot added a comment.Aug 6 2020, 10:23 AM

Change 618719 merged by Filippo Giunchedi:
[operations/puppet@production] hieradata: add alert[12]001 to monitoring hosts

https://gerrit.wikimedia.org/r/618719

gerritbot added a comment.Aug 17 2020, 1:26 PM

Change 620701 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] profile: refactor logmsgbot to follow Icinga failover

https://gerrit.wikimedia.org/r/620701

ayounsi closed subtask T260533: Add alert[12]001 to network ACLs as Resolved.Aug 18 2020, 9:01 AM
fgiunchedi reopened subtask T260533: Add alert[12]001 to network ACLs as Open.Aug 18 2020, 9:12 AM
fgiunchedi added a comment.Aug 18 2020, 9:45 AM

Something else I noticed while looking at icinga logs: check_nrpe for jessie hosts fails:

etcd1003;puppet last run;CRITICAL;SOFT;2;CHECK_NRPE: (ssl_err != 5) Error - Could not complete SSL handshake with 10.64.0.42: 1

Digging further this turns out to be too short dh params on the nrpe server side (from check_nrpe logs)

[1597742562] SSL Certificate File: None
[1597742562] SSL Private Key File: None
[1597742562] SSL CA Certificate File: None
[1597742562] SSL Cipher List: ALL:!MD5:@STRENGTH:@SECLEVEL=0
[1597742562] SSL Allow ADH: 1
[1597742562] SSL Log Options: 0xff
[1597742562] SSL Version: TLSv1_plus And Above
[1597742562] Connected to 10.64.0.42
[1597742562] Error: (ERR_get_error_line_data = 337260938), Could not complete SSL handshake with 10.64.0.42: dh key too small
fgiunchedi updated the task description. (Show Details)Aug 18 2020, 9:46 AM
gerritbot added a comment.Aug 18 2020, 12:56 PM

Change 620929 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] icinga: make sure update-etcd-mw-config-lastindex is enabled

https://gerrit.wikimedia.org/r/620929

gerritbot added a comment.Aug 19 2020, 9:12 AM

Change 620701 merged by Filippo Giunchedi:
[operations/puppet@production] profile: refactor logmsgbot to follow Icinga failover

https://gerrit.wikimedia.org/r/620701

gerritbot added a comment.Aug 19 2020, 9:21 AM

Change 620929 merged by Filippo Giunchedi:
[operations/puppet@production] icinga: make sure update-etcd-mw-config-lastindex is enabled

https://gerrit.wikimedia.org/r/620929

jcrespo mentioned this in T260686: check_mariadb_dump failing on alert[12]* hosts.Aug 24 2020, 8:44 AM
fgiunchedi closed subtask T261198: nagios-nrpe-server in jessie not compatibile with Buster version as Resolved.Aug 26 2020, 12:36 PM
herron closed subtask T260688: WMCS galera cluster checks failing from new alert[12]* hosts as Resolved.Aug 26 2020, 5:03 PM
herron updated the task description. (Show Details)Aug 26 2020, 6:02 PM
jcrespo closed subtask T260686: check_mariadb_dump failing on alert[12]* hosts as Resolved.Aug 28 2020, 9:26 AM
herron updated the task description. (Show Details)Sep 2 2020, 7:10 PM
herron closed subtask T261342: ensure alert[12]001 are prepared for meta monitoring as Resolved.Sep 3 2020, 3:30 PM
herron updated the task description. (Show Details)Sep 3 2020, 4:10 PM
herron updated the task description. (Show Details)
herron updated the task description. (Show Details)Sep 15 2020, 7:43 PM
gerritbot added a comment.Sep 23 2020, 2:32 PM

Change 629408 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] icinga: switch active server from icinga1001 to alert1001

https://gerrit.wikimedia.org/r/629408

gerritbot added a comment.Sep 23 2020, 3:13 PM

Change 629415 had a related patch set uploaded (by Herron; owner: Herron):
[operations/dns@master] dns: point icinga CNAMEs to alert1001

https://gerrit.wikimedia.org/r/629415

Stashbot added a comment.Sep 23 2020, 4:00 PM

Mentioned in SAL (#wikimedia-operations) [2020-09-23T16:00:44Z] <herron> switching icinga over from icinga1001 to alert1001 T247966

gerritbot added a comment.Sep 23 2020, 4:01 PM

Change 629415 merged by Herron:
[operations/dns@master] dns: point icinga CNAMEs to alert1001

https://gerrit.wikimedia.org/r/629415

gerritbot added a comment.Sep 23 2020, 4:05 PM

Change 629408 merged by Herron:
[operations/puppet@production] icinga: switch active server from icinga1001 to alert1001

https://gerrit.wikimedia.org/r/629408

herron updated the task description. (Show Details)Sep 23 2020, 4:21 PM
herron added a comment.EditedSep 23 2020, 4:25 PM

Alert1001 is now the active Icinga server. Meta monitoring for alert[12]001 has been enabled as well.

There may be a few alerts needing TLC due to the transition, either from service checks changing from host icinga1001 to alert1001 or due to other migration related issues. Please join me in keeping an eye out for these.

jcrespo closed subtask T263662: check-mariadb-backups pkg_resources.VersionConflict: 0.2 (/usr/lib/python3/dist-packages), Requirement.parse('wmfbackups==0.1') as Resolved.Sep 23 2020, 5:05 PM
fgiunchedi closed subtask T260521: icinga-exporter failing on alert hosts as Resolved.Sep 29 2020, 12:13 PM
herron closed subtask T260533: Add alert[12]001 to network ACLs as Resolved.Oct 26 2020, 1:32 PM
MoritzMuehlenhoff subscribed.Mar 12 2021, 9:10 AM

What's up with icinga1001/icinga2001, they are still up and running?

gerritbot added a comment.Apr 7 2021, 9:02 PM

Change 677656 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] remove alerting_host role from icinga[12]001

https://gerrit.wikimedia.org/r/677656

colewhite added subtasks: T279602: reclaim icinga2001.wikimedia.org, T279601: decommission icinga1001.wikimedia.org.Apr 7 2021, 10:02 PM
gerritbot added a comment.May 18 2021, 3:30 PM

Change 677656 abandoned by Cwhite:

[operations/puppet@production] remove alerting_host role from icinga[12]001

Reason:

superseded

https://gerrit.wikimedia.org/r/677656

Papaul closed subtask T279602: reclaim icinga2001.wikimedia.org as Resolved.May 24 2021, 8:20 PM
lmata closed this task as Resolved.Jun 14 2021, 3:47 PM
lmata claimed this task.
Stashbot added a comment.Jul 16 2021, 3:14 PM

Mentioned in SAL (#wikimedia-operations) [2021-07-16T15:14:41Z] <godog> set alert2001 as active in netbox (was staged) - T247966

• Cmjohnson closed subtask T279601: decommission icinga1001.wikimedia.org as Resolved.Sep 14 2021, 4:30 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL