Page MenuHomePhabricator
Create Task
Maniphest T202782

upgrade icinga server to stretch and replace einsteinium
Closed, ResolvedPublic
Actions

Description

The production Icinga server (alerting_host) is einsteinium. The hardware needs to be replaced and it is running jessie.

This task consists of:

  • provide replacement hardware, called icinga1001 [T201344]
  • ensure the current alerting_host puppet role works on stretch, fix it if it doesn't (cloud VPS)
  • make changes to the role so that it can be applied on the new host in eqiad without being the "active" host to avoid duplicate alerts and notification spam
  • optional but very nice to have (otherwise we have to override jenkins style checks): refactor the icinga role to a profile, move the Hiera calls to parameters
  • apply role on icinga1001, confirm puppet is fine, if not fix things
  • confirm web UI works, test login, user privileges, sending email out as notification
  • confirm all checks are green (or like in prod) , without having notifications enabled
  • schedule switchover from einsteinium to icinga1001
  • switch over and disable notifications permanently on einsteinium but leave it running for a grace period (?)

Details

SubjectRepoBranchLines +/-
naggen2: python3 and remove activerecord supportoperations/puppetproduction+34 -92
icinga: remove jessie supportoperations/puppetproduction+113 -230
icinga: stop having separate configs for jessie/stretchoperations/puppetproduction+181 -1 K
icinga::web: do not use PHP anymoreoperations/puppetproduction+0 -5
icinga/NSCA (passive checks): remove jessie support hacksoperations/puppetproduction+8 -24
icinga: remove einsteinium as an alerting_hostoperations/puppetproduction+10 -14
icinga: do not manage retention.dat in puppetoperations/puppetproduction+0 -8
icinga: fix path to puppet_hosts/services in default_icinga.shoperations/puppetproduction+1 -1
icinga: on stretch, use fping instead of ping for faster host checksoperations/puppetproduction+8 -2
smokeping: replace target einsteinium with authdns1001operations/puppetproduction+4 -5
icinga: enable daemon log, in addition to syslog, againoperations/puppetproduction+1 -1
icinga: fix path to puppet_hosts in icinga-downtimeoperations/puppetproduction+5 -3
icinga: transition to new active hostoperations/puppetproduction+2 -2
icinga: disable service notifications on einsteinium and enable on icinga1001operations/puppetproduction+1 -1
switch icinga-stretch to icinga and icinga to icinga-oldoperations/dnsmaster+2 -2
icinga: make icinga1001 active and einsteinium passiveoperations/puppetproduction+6 -5
icinga: add tmpfs options to stretchoperations/puppetproduction+8 -4
icinga: fix path to retention.dat file on stretchoperations/puppetproduction+5 -3
base/icinga: use monitoring_hosts constant as NRPE allowed_hostsoperations/puppetproduction+3 -1
mailman: whitelist icinga1001 IP alongside einsteinium and tegmenoperations/puppetproduction+4 -3
base/NRPE: make nrpe_allowed_hosts flexible on $realmoperations/puppetproduction+1 -1
icinga: fix ownership of command file dir on stretchoperations/puppetproduction+10 -5
nagios_common: ensure libtimedate-perl is installed for check_ssloperations/puppetproduction+1 -0
icinga: on stretch, use systemd::service, unit file by systemd-sysv-generatoroperations/puppetproduction+33 -6
icinga: on stretch, don't use custom logrotate configoperations/puppetproduction+5 -5
icinga: don't install PHP httpd module on stretchoperations/puppetproduction+7 -12
etcd::monitoring: replace Nagios::Plugin with Monitoring::Plugin on stretchoperations/puppetproduction+52 -2
icinga: logging optimizationsoperations/puppetproduction+12 -10
icinga/alerting_host: ensure mpm_prefork is selected for httpd and php7.0operations/puppetproduction+6 -0
icinga/nsca: fix command_file path on stretchoperations/puppetproduction+3 -1
icinga/nsca: fix nsca_chroot path on stretchoperations/puppetproduction+12 -7
icinga: allow configuring max_concurrent_checks via Hieraoperations/puppetproduction+7 -2
icinga: hot fix for puppet issue with rsync and use_ipv6operations/puppetproduction+2 -1
icinga: install nsca_frack.cfg in objects on stretchoperations/puppetproduction+16 -9
icinga/etcd: /var/run/icinga/ -> /var/run/nagios/operations/puppetproduction+2 -2
icinga: don't log service/host check retriesoperations/puppetproduction+2 -2
icinga/nsca: use systemd::service, unit file by systemd-sysv-generatoroperations/puppetproduction+39 -3
icinga: avoid hardcoded nsca_user,nsca_group for NSCA serviceoperations/puppetproduction+12 -3
nagios_common: set flag -2 on check_nrpe for nrpe on stretchoperations/puppetproduction+24 -20
icinga: make icinga user flexible in update-etcd-mw-config-lastindexoperations/puppetproduction+8 -3
icinga: remove check_lonqueries.ploperations/puppetproduction+0 -246
nagios_common: convert check_jnx_alarms to use Monitoring::Pluginoperations/puppetproduction+6 -6
nagios_common: remove check_sslxNNoperations/puppetproduction+0 -127
nagios_common: convert check_bgp to use Monitoring::Pluginoperations/puppetproduction+5 -5
nagios_common: switch check_ssl to use Monitoring::Pluginoperations/puppetproduction+6 -6
nagios_common: add stretch support to check_ssloperations/puppetproduction+43 -9
nagios_common: on jessie, also install libmonitoring-plugin-perloperations/puppetproduction+6 -7
icinga/check_ssl: add support for stretch, rename it to check_tlsoperations/puppetproduction+361 -3
base/nrpe: add icinga1001 to allowed_hostsoperations/puppetproduction+1 -1
icinga: enable icinga service on icinga1001operations/puppetproduction+0 -1
base/icinga/nrpe: move nrpe_allowed IPs to Hiera, add icinga1001operations/puppetproduction+3 -1
base: do not allow mailman server to NRPE to other hosts for no reasonoperations/puppetproduction+1 -1
icinga: move configuration directory for resource.cfgoperations/puppetproduction+8 -1
icinga: use correct user for tmpfs mountoperations/puppetproduction+1 -1
icinga, nagios_common: use commands.cfg on stretchoperations/puppetproduction+19 -6
icinga: add new icinga.cfg and move configurationoperations/puppetproduction+1 K -54
icinga: service management depends on debian versionoperations/puppetproduction+21 -11
icinga: move init script customizations to defaultoperations/puppetproduction+33 -10
network::constancts: add icinga1001 to monitoring_hostsoperations/puppetproduction+2 -0
naggen2: restrict generated defines to valid optionsoperations/puppetproduction+94 -4
icinga: set user/group for event_handlers::raidoperations/puppetproduction+12 -4
nagios_common: make user/group configurable from Hieraoperations/puppetproduction+16 -16
icinga::plugins: set user/group for nagios_common::commandsoperations/puppetproduction+3 -1
icinga: tell rsync server to use IPv6operations/puppetproduction+5 -3
fix IPv6 for icinga1001, is in eqiad row C, not row Aoperations/dnsmaster+2 -2
icinga: allow rsync from multiple hostsoperations/puppetproduction+27 -20
icinga: allow rsync from multiple hostsoperations/puppetproduction+30 -20
icinga: linter compliance and stretch useroperations/puppetproduction+37 -33
icinga: enforce mode on nagios filesoperations/puppetproduction+5 -0
icinga: pass user/group from profile, change to nagios on icinga1001operations/puppetproduction+6 -0
icinga: fix more places with hardcoded user nameoperations/puppetproduction+6 -6
icinga: make group configurable, make user configurable pt2operations/puppetproduction+24 -23
icinga: use $icinga::icinga_user in all classesoperations/puppetproduction+15 -18
icinga: make icinga user configurableoperations/puppetproduction+27 -23
icinga: (if on stretch) don't let puppet manage user/groupoperations/puppetproduction+23 -19
icinga: remove nsca::firewall class, use ferm::service in profileoperations/puppetproduction+9 -42
icinga: remove icinga::group classoperations/puppetproduction+7 -9
monitoring: set mode on host and service configsoperations/puppetproduction+6 -1
icinga: enable acme (letsencrypt) on icinga1001operations/puppetproduction+1 -1
icinga: make the apache virtual host name configurableoperations/puppetproduction+20 -10
add icinga-stretch subdomain cnamed for icinga1001operations/dnsmaster+1 -0
profile/cumin: update python scripts to detect command fileoperations/puppetproduction+16 -2
icinga: add stretch support to submit_check_result.shoperations/puppetproduction+7 -2
icinga-downtime: add stretch supportoperations/puppetproduction+6 -1
profile: use mariadb-client in stretch.operations/puppetproduction+8 -1
icinga::web: install libapache2-mod-php if on stretchoperations/puppetproduction+6 -3
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
gerritbot added a comment.Oct 23 2018, 10:08 PM

Change 469333 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] icinga: use fping instead of ping for faster host checks

https://gerrit.wikimedia.org/r/469333

gerritbot added a comment.Oct 24 2018, 12:37 AM

Change 469253 merged by Dzahn:
[operations/puppet@production] icinga: allow configuring max_concurrent_checks via Hiera

https://gerrit.wikimedia.org/r/469253

gerritbot added a comment.Oct 24 2018, 12:48 AM

Change 469317 merged by Dzahn:
[operations/puppet@production] icinga: don't log service/host check retries

https://gerrit.wikimedia.org/r/469317

gerritbot added a comment.Oct 25 2018, 8:39 PM

Change 467017 abandoned by Dzahn:
icinga/etcd: /var/run/icinga/ -> /var/run/nagios/

Reason:
done in https://gerrit.wikimedia.org/r/#/c/operations/puppet/ /468414/ instead

https://gerrit.wikimedia.org/r/467017

gerritbot added a comment.Oct 25 2018, 9:20 PM

Change 469780 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] icinga: install nsca_frack.cfg in objects on stretch

https://gerrit.wikimedia.org/r/469780

gerritbot added a comment.Oct 25 2018, 10:32 PM

Change 469780 merged by Dzahn:
[operations/puppet@production] icinga: install nsca_frack.cfg in objects on stretch

https://gerrit.wikimedia.org/r/469780

gerritbot added a comment.Oct 26 2018, 12:32 AM

Change 469808 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] icinga/nsca: allow configuring nsca chroot in Hiera, change on stretch

https://gerrit.wikimedia.org/r/469808

Dzahn mentioned this in T208066: Push check latency and check execution time to Prometheus.Oct 26 2018, 5:37 PM
gerritbot added a comment.Oct 26 2018, 6:45 PM

Change 469808 merged by Dzahn:
[operations/puppet@production] icinga/nsca: fix nsca_chroot path on stretch

https://gerrit.wikimedia.org/r/469808

gerritbot added a comment.Oct 26 2018, 7:08 PM

Change 469988 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] icinga/nsca: fix command_file path on stretch

https://gerrit.wikimedia.org/r/469988

gerritbot added a comment.Oct 26 2018, 7:16 PM

Change 469988 merged by Dzahn:
[operations/puppet@production] icinga/nsca: fix command_file path on stretch

https://gerrit.wikimedia.org/r/469988

ops-monitoring-bot subscribed.Oct 26 2018, 9:32 PM

Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts:

icinga1001.wikimedia.org

The log can be found in /var/log/wmf-auto-reimage/201810262131_dzahn_122415_icinga1001_wikimedia_org.log.

ops-monitoring-bot added a comment.Oct 26 2018, 9:32 PM

Completed auto-reimage of hosts:

['icinga1001.wikimedia.org']

Of which those FAILED:

['icinga1001.wikimedia.org']
ops-monitoring-bot added a comment.Oct 26 2018, 9:33 PM

Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts:

icinga1001.wikimedia.org

The log can be found in /var/log/wmf-auto-reimage/201810262133_dzahn_122885_icinga1001_wikimedia_org.log.

Dzahn mentioned this in T208100: cumin tries to downtime Icinga even with --no-downtime.Oct 26 2018, 10:07 PM
ops-monitoring-bot added a comment.Oct 27 2018, 12:00 AM

Completed auto-reimage of hosts:

['icinga1001.wikimedia.org']

Of which those FAILED:

['icinga1001.wikimedia.org']
Stashbot added a comment.Oct 27 2018, 12:00 AM

Mentioned in SAL (#wikimedia-operations) [2018-10-27T00:00:06Z] <mutante> icinga1001 - using wmf-auto-reimage to reinstall gets stuck at initial puppet run after reboot - Still waiting for Puppet after 105.0 minutes - aborting on cumin, loggin in directly and manually running puppet (T202782 T208100)

gerritbot added a comment.Oct 27 2018, 12:33 AM

Change 470106 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] icinga/alerting_host: ensure mpm_prefork is selected for httpd and php7.0

https://gerrit.wikimedia.org/r/470106

gerritbot added a comment.Oct 27 2018, 12:34 AM

Change 470106 merged by Dzahn:
[operations/puppet@production] icinga/alerting_host: ensure mpm_prefork is selected for httpd and php7.0

https://gerrit.wikimedia.org/r/470106

Dzahn added a subtask: T208108: httpd class and php7.0 - conflict with mpm_event module.Oct 27 2018, 12:38 AM
Volans added a comment.Oct 27 2018, 9:53 AM
In T202782#4699100, @Stashbot wrote:

Mentioned in SAL (#wikimedia-operations) [2018-10-27T00:00:06Z] <mutante> icinga1001 - using wmf-auto-reimage to reinstall gets stuck at initial puppet run after reboot - Still waiting for Puppet after 105.0 minutes - aborting on cumin, loggin in directly and manually running puppet (T202782 T208100)

What got stuck?
The first puppet run can easily take up to 2h of time based on the host role and server capabilities. You can check its progress tailing the cumin log whose path is listed at the start of the reimage process to stdout (the 2nd of the two logs listed).
To check further what's going on directly in the host you could also login either with the install_console script or your personal key (based where the host is in the puppetization process) or, worse case scenario you can't with both, via the mgmt console as root.

gerritbot added a comment.Oct 30 2018, 11:21 PM

Change 469320 merged by Dzahn:
[operations/puppet@production] icinga: logging optimizations

https://gerrit.wikimedia.org/r/469320

gerritbot added a comment.Oct 31 2018, 9:10 PM

Change 467015 merged by Dzahn:
[operations/puppet@production] etcd::monitoring: replace Nagios::Plugin with Monitoring::Plugin on stretch

https://gerrit.wikimedia.org/r/467015

gerritbot added a comment.Oct 31 2018, 9:25 PM

Change 470915 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] icinga: don't install PHP httpd module on stretch

https://gerrit.wikimedia.org/r/470915

gerritbot added a comment.Oct 31 2018, 9:41 PM

Change 470915 merged by Dzahn:
[operations/puppet@production] icinga: don't install PHP httpd module on stretch

https://gerrit.wikimedia.org/r/470915

gerritbot added a comment.Oct 31 2018, 11:59 PM

Change 470955 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] icinga: on stretch, don't use custom logrotate config

https://gerrit.wikimedia.org/r/470955

gerritbot added a comment.Nov 1 2018, 12:27 AM

Change 470955 merged by Dzahn:
[operations/puppet@production] icinga: on stretch, don't use custom logrotate config

https://gerrit.wikimedia.org/r/470955

Dzahn added a comment.EditedNov 1 2018, 12:32 AM
In T202782#4699593, @Volans wrote:

What got stuck?

It never detected a succesful run because it had this error: T208108

which i fixed with https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/470106/

I will reinstall one more time to double check there was nothing else.

Logging in wasn't the problem, it was already after creating all the users.

gerritbot added a comment.Nov 1 2018, 1:44 AM

Change 462600 merged by Dzahn:
[operations/puppet@production] icinga: on stretch, use systemd::service, unit file by systemd-sysv-generator

https://gerrit.wikimedia.org/r/462600

ops-monitoring-bot added a comment.Nov 3 2018, 2:58 AM

Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts:

icinga1001.wikimedia.org

The log can be found in /var/log/wmf-auto-reimage/201811030258_dzahn_163438_icinga1001_wikimedia_org.log.

ops-monitoring-bot added a comment.Nov 3 2018, 3:35 AM

Completed auto-reimage of hosts:

['icinga1001.wikimedia.org']

and were ALL successful.

gerritbot added a comment.Nov 5 2018, 10:16 PM

Change 471866 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] nagios_common: ensure libtimedate-perl is installed for check_ssl

https://gerrit.wikimedia.org/r/471866

gerritbot added a comment.Nov 5 2018, 10:27 PM

Change 471866 merged by Dzahn:
[operations/puppet@production] nagios_common: ensure libtimedate-perl is installed for check_ssl

https://gerrit.wikimedia.org/r/471866

gerritbot added a comment.Nov 5 2018, 11:13 PM

Change 471873 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] icinga: fix ownership of command file dir on stretch

https://gerrit.wikimedia.org/r/471873

gerritbot added a comment.Nov 6 2018, 1:03 AM

Change 471873 merged by Dzahn:
[operations/puppet@production] icinga: fix ownership of command file dir on stretch

https://gerrit.wikimedia.org/r/471873

ops-monitoring-bot added a comment.Nov 6 2018, 1:34 AM

Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts:

icinga1001.wikimedia.org

The log can be found in /var/log/wmf-auto-reimage/201811060134_dzahn_77047_icinga1001_wikimedia_org.log.

gerritbot added a comment.Nov 6 2018, 1:58 AM

Change 471900 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] mailman: whitelist icinga1001 IP alongside einsteinium and tegmen

https://gerrit.wikimedia.org/r/471900

ops-monitoring-bot added a comment.Nov 6 2018, 2:07 AM

Completed auto-reimage of hosts:

['icinga1001.wikimedia.org']

Of which those FAILED:

['icinga1001.wikimedia.org']
gerritbot added a comment.Nov 6 2018, 4:29 PM

Change 465519 merged by Dzahn:
[operations/puppet@production] base/icinga: use monitoring_hosts constant as NRPE allowed_hosts

https://gerrit.wikimedia.org/r/465519

gerritbot added a comment.Nov 6 2018, 5:07 PM

Change 472006 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] base/NRPE: make nrpe_allowed_hosts flexible on $realm

https://gerrit.wikimedia.org/r/472006

gerritbot added a comment.Nov 6 2018, 5:13 PM

Change 472006 merged by Dzahn:
[operations/puppet@production] base/NRPE: make nrpe_allowed_hosts flexible on $realm

https://gerrit.wikimedia.org/r/472006

gerritbot added a comment.Nov 6 2018, 8:19 PM

Change 471900 merged by Dzahn:
[operations/puppet@production] mailman: whitelist icinga1001 IP alongside einsteinium and tegmen

https://gerrit.wikimedia.org/r/471900

colewhite closed subtask T207775: newer version of nagios-nrpe-plugin nrpe (check_nrpe) with fixed logging issue on stretch icinga as Resolved.Nov 6 2018, 8:55 PM
Stashbot added a comment.Nov 6 2018, 9:50 PM

Mentioned in SAL (#wikimedia-operations) [2018-11-06T21:50:34Z] <mutante> icinga1001-"MediaWiki EtcdConfig up-to-date" checks were all UNKNOWN because systemd unit update-etcd-mw-config-lastindex was present but service not running. it was turned off in https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/427328/ on purpose. manually ran "systemctl start update-etcd-mw-config-lastindex" and the checks all work (T202782)

gerritbot added a comment.Nov 7 2018, 11:54 PM

Change 472352 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] icinga: fix path to retention.dat file on stretch

https://gerrit.wikimedia.org/r/472352

gerritbot added a comment.Nov 8 2018, 6:48 PM

Change 472352 merged by Dzahn:
[operations/puppet@production] icinga: fix path to retention.dat file on stretch

https://gerrit.wikimedia.org/r/472352

gerritbot added a comment.Nov 8 2018, 7:39 PM

Change 472519 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] icinga: add tmpfs options to stretch

https://gerrit.wikimedia.org/r/472519

gerritbot added a comment.Nov 8 2018, 8:36 PM

Change 472519 merged by Dzahn:
[operations/puppet@production] icinga: add tmpfs options to stretch

https://gerrit.wikimedia.org/r/472519

jijiki subscribed.Nov 12 2018, 4:35 PM
gerritbot added a comment.Nov 13 2018, 3:51 PM

Change 473235 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/dns@master] switch icinga-stretch to icinga and icinga to icinga-old

https://gerrit.wikimedia.org/r/473235

Dzahn updated the task description. (Show Details)Nov 13 2018, 3:54 PM
gerritbot added a comment.Nov 13 2018, 4:24 PM

Change 473244 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] icinga: make icinga1001 active and einsteinium passive

https://gerrit.wikimedia.org/r/473244

gerritbot added a comment.Nov 13 2018, 5:03 PM

Change 473244 merged by Dzahn:
[operations/puppet@production] icinga: make icinga1001 active and einsteinium passive

https://gerrit.wikimedia.org/r/473244

gerritbot added a comment.Nov 13 2018, 5:12 PM

Change 473235 merged by Dzahn:
[operations/dns@master] switch icinga-stretch to icinga and icinga to icinga-old

https://gerrit.wikimedia.org/r/473235

gerritbot added a comment.Nov 13 2018, 5:19 PM

Change 473250 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] icinga: disable service notifications on einsteinium and enable on icinga1001

https://gerrit.wikimedia.org/r/473250

gerritbot added a comment.Nov 13 2018, 5:22 PM

Change 473251 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] icinga: transition to new active host

https://gerrit.wikimedia.org/r/473251

gerritbot added a comment.Nov 13 2018, 5:22 PM

Change 473250 merged by Dzahn:
[operations/puppet@production] icinga: disable service notifications on einsteinium and enable on icinga1001

https://gerrit.wikimedia.org/r/473250

gerritbot added a comment.Nov 13 2018, 5:24 PM

Change 473251 merged by Cwhite:
[operations/puppet@production] icinga: transition to new active host

https://gerrit.wikimedia.org/r/473251

Stashbot added a comment.Nov 13 2018, 6:03 PM

Mentioned in SAL (#wikimedia-operations) [2018-11-13T18:03:04Z] <mutante> icinga migration has concluded, we are now on stretch and icinga1001, einsteinium is passive (T202782)

gerritbot added a comment.Nov 13 2018, 6:48 PM

Change 473261 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] icinga: fix path to puppet_hosts in icinga-downtime

https://gerrit.wikimedia.org/r/473261

gerritbot added a comment.Nov 13 2018, 6:52 PM

Change 473261 merged by Dzahn:
[operations/puppet@production] icinga: fix path to puppet_hosts in icinga-downtime

https://gerrit.wikimedia.org/r/473261

Dzahn updated the task description. (Show Details)Nov 13 2018, 7:40 PM

The switch has happened. icinga1001 is now the active server and einsteinium is passive. This ticket still open for a grace period until we decom einsteinium.

gerritbot added a comment.Nov 13 2018, 8:12 PM

Change 473275 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] icinga: enable daemon log, in addition to syslog, again

https://gerrit.wikimedia.org/r/473275

gerritbot added a comment.Nov 13 2018, 8:26 PM

Change 473276 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] icinga: remove jessie support

https://gerrit.wikimedia.org/r/473276

gerritbot added a comment.Nov 13 2018, 8:29 PM

Change 473275 merged by Dzahn:
[operations/puppet@production] icinga: enable daemon log, in addition to syslog, again

https://gerrit.wikimedia.org/r/473275

gerritbot added a comment.Nov 13 2018, 9:28 PM

Change 473282 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] smokeping: replace target einsteinium with dns1002

https://gerrit.wikimedia.org/r/473282

gerritbot added a comment.Nov 14 2018, 12:33 AM

Change 473282 merged by Dzahn:
[operations/puppet@production] smokeping: replace target einsteinium with authdns1001

https://gerrit.wikimedia.org/r/473282

Dzahn closed subtask T208824: rename tegmen to icinga2001 and reinstall it with stretch as Resolved.Nov 15 2018, 9:13 PM
Dzahn added a subtask: T209738: decom einsteinium.Nov 16 2018, 11:55 PM
Dzahn changed the status of subtask T209738: decom einsteinium from Open to Stalled.Nov 16 2018, 11:58 PM
Dzahn closed this task as Resolved.Nov 17 2018, 12:00 AM
Dzahn updated the task description. (Show Details)
Dzahn updated the task description. (Show Details)

this ticket is resolved, einsteinium has been replaced by icinga1001 on stretch.

the rest of the steps will be part of the decom ticket now created at T209738 with the official decom checkboxes (and i added the one to remove from AQL that was here previously)

gerritbot added a comment.Nov 17 2018, 12:19 AM

Change 473278 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] icinga: remove einsteinium as an alerting_host

https://gerrit.wikimedia.org/r/473278

gerritbot added a comment.Nov 17 2018, 1:06 AM

Change 469333 abandoned by Dzahn:
icinga: on stretch, use fping instead of ping for faster host checks

https://gerrit.wikimedia.org/r/469333

gerritbot added a comment.Nov 17 2018, 2:35 PM

Change 474463 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] icinga: fix path to puppet_hosts/services in default_icinga.sh

https://gerrit.wikimedia.org/r/474463

gerritbot added a comment.Nov 17 2018, 2:37 PM

Change 474463 merged by Dzahn:
[operations/puppet@production] icinga: fix path to puppet_hosts/services in default_icinga.sh

https://gerrit.wikimedia.org/r/474463

gerritbot added a comment.Nov 17 2018, 2:50 PM

Change 474464 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] icinga: do not manage retention.dat in puppet

https://gerrit.wikimedia.org/r/474464

Dzahn added a subtask: T209757: Notifications disablement via puppet not working on icinga.Nov 17 2018, 2:51 PM
gerritbot added a comment.Nov 17 2018, 2:54 PM

Change 474464 merged by Dzahn:
[operations/puppet@production] icinga: do not manage retention.dat in puppet

https://gerrit.wikimedia.org/r/474464

Dzahn closed subtask T209757: Notifications disablement via puppet not working on icinga as Resolved.Nov 17 2018, 3:41 PM
gerritbot added a comment.Nov 26 2018, 6:14 PM

Change 473278 merged by Dzahn:
[operations/puppet@production] icinga: remove einsteinium as an alerting_host

https://gerrit.wikimedia.org/r/473278

gerritbot added a comment.Nov 26 2018, 9:58 PM

Change 475881 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] icinga/NSCA (passive checks): remove jessie support hacks

https://gerrit.wikimedia.org/r/475881

gerritbot added a comment.Nov 26 2018, 10:25 PM

Change 475881 merged by Dzahn:
[operations/puppet@production] icinga/NSCA (passive checks): remove jessie support hacks

https://gerrit.wikimedia.org/r/475881

gerritbot added a comment.Nov 26 2018, 10:31 PM

Change 475901 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] icinga::web: do not use PHP anymore

https://gerrit.wikimedia.org/r/475901

gerritbot added a comment.Nov 27 2018, 12:21 AM

Change 475901 merged by Dzahn:
[operations/puppet@production] icinga::web: do not use PHP anymore

https://gerrit.wikimedia.org/r/475901

gerritbot added a comment.Nov 27 2018, 12:28 AM

Change 475927 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] icinga: stop having separate configs for jessie/stretch

https://gerrit.wikimedia.org/r/475927

gerritbot added a comment.Nov 27 2018, 12:32 AM

Change 475927 merged by Dzahn:
[operations/puppet@production] icinga: stop having separate configs for jessie/stretch

https://gerrit.wikimedia.org/r/475927

gerritbot added a comment.Nov 27 2018, 12:44 AM

Change 473276 merged by Dzahn:
[operations/puppet@production] icinga: remove jessie support

https://gerrit.wikimedia.org/r/473276

Dzahn changed the status of subtask T209738: decom einsteinium from Stalled to Open.Nov 27 2018, 12:59 AM
colewhite closed subtask T208066: Push check latency and check execution time to Prometheus as Resolved.Dec 11 2018, 6:39 PM
fgiunchedi mentioned this in T212669: icinga doesn't log ampersand in notes_url links.Dec 28 2018, 3:29 PM
gerritbot added a comment.Nov 6 2019, 11:05 PM

Change 463133 abandoned by Cwhite:
naggen2: python3 and remove activerecord support

Reason:
This CS is getting long in the tooth and is generally terrible. Abandoning in favor of I82d8a0d833952475c4f665b3a317bd0c3131ab1b

https://gerrit.wikimedia.org/r/463133

Papaul closed subtask T209738: decom einsteinium as Resolved.Nov 8 2019, 7:45 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL