Page MenuHomePhabricator
Create Task
Maniphest T202782

upgrade icinga server to stretch and replace einsteinium
Closed, ResolvedPublic
Actions

Description

The production Icinga server (alerting_host) is einsteinium. The hardware needs to be replaced and it is running jessie.

This task consists of:

  • provide replacement hardware, called icinga1001 [T201344]
  • ensure the current alerting_host puppet role works on stretch, fix it if it doesn't (cloud VPS)
  • make changes to the role so that it can be applied on the new host in eqiad without being the "active" host to avoid duplicate alerts and notification spam
  • optional but very nice to have (otherwise we have to override jenkins style checks): refactor the icinga role to a profile, move the Hiera calls to parameters
  • apply role on icinga1001, confirm puppet is fine, if not fix things
  • confirm web UI works, test login, user privileges, sending email out as notification
  • confirm all checks are green (or like in prod) , without having notifications enabled
  • schedule switchover from einsteinium to icinga1001
  • switch over and disable notifications permanently on einsteinium but leave it running for a grace period (?)

Details

ProjectBranchLines +/-Subject
operations/puppetproduction+34 -92naggen2: python3 and remove activerecord support
operations/puppetproduction+113 -230icinga: remove jessie support
operations/puppetproduction+181 -1 Kicinga: stop having separate configs for jessie/stretch
operations/puppetproduction+0 -5icinga::web: do not use PHP anymore
operations/puppetproduction+8 -24icinga/NSCA (passive checks): remove jessie support hacks
operations/puppetproduction+10 -14icinga: remove einsteinium as an alerting_host
operations/puppetproduction+0 -8icinga: do not manage retention.dat in puppet
operations/puppetproduction+1 -1icinga: fix path to puppet_hosts/services in default_icinga.sh
operations/puppetproduction+8 -2icinga: on stretch, use fping instead of ping for faster host checks
operations/puppetproduction+4 -5smokeping: replace target einsteinium with authdns1001
operations/puppetproduction+1 -1icinga: enable daemon log, in addition to syslog, again
operations/puppetproduction+5 -3icinga: fix path to puppet_hosts in icinga-downtime
operations/puppetproduction+2 -2icinga: transition to new active host
operations/puppetproduction+1 -1icinga: disable service notifications on einsteinium and enable on icinga1001
operations/dnsmaster+2 -2switch icinga-stretch to icinga and icinga to icinga-old
operations/puppetproduction+6 -5icinga: make icinga1001 active and einsteinium passive
operations/puppetproduction+8 -4icinga: add tmpfs options to stretch
operations/puppetproduction+5 -3icinga: fix path to retention.dat file on stretch
operations/puppetproduction+3 -1base/icinga: use monitoring_hosts constant as NRPE allowed_hosts
operations/puppetproduction+4 -3mailman: whitelist icinga1001 IP alongside einsteinium and tegmen
operations/puppetproduction+1 -1base/NRPE: make nrpe_allowed_hosts flexible on $realm
operations/puppetproduction+10 -5icinga: fix ownership of command file dir on stretch
operations/puppetproduction+1 -0nagios_common: ensure libtimedate-perl is installed for check_ssl
operations/puppetproduction+33 -6icinga: on stretch, use systemd::service, unit file by systemd-sysv-generator
operations/puppetproduction+5 -5icinga: on stretch, don't use custom logrotate config
operations/puppetproduction+7 -12icinga: don't install PHP httpd module on stretch
operations/puppetproduction+52 -2etcd::monitoring: replace Nagios::Plugin with Monitoring::Plugin on stretch
operations/puppetproduction+12 -10icinga: logging optimizations
operations/puppetproduction+6 -0icinga/alerting_host: ensure mpm_prefork is selected for httpd and php7.0
operations/puppetproduction+3 -1icinga/nsca: fix command_file path on stretch
operations/puppetproduction+12 -7icinga/nsca: fix nsca_chroot path on stretch
operations/puppetproduction+7 -2icinga: allow configuring max_concurrent_checks via Hiera
operations/puppetproduction+2 -1icinga: hot fix for puppet issue with rsync and use_ipv6
operations/puppetproduction+16 -9icinga: install nsca_frack.cfg in objects on stretch
operations/puppetproduction+2 -2icinga/etcd: /var/run/icinga/ -> /var/run/nagios/
operations/puppetproduction+2 -2icinga: don't log service/host check retries
operations/puppetproduction+39 -3icinga/nsca: use systemd::service, unit file by systemd-sysv-generator
operations/puppetproduction+12 -3icinga: avoid hardcoded nsca_user,nsca_group for NSCA service
operations/puppetproduction+24 -20nagios_common: set flag -2 on check_nrpe for nrpe on stretch
operations/puppetproduction+8 -3icinga: make icinga user flexible in update-etcd-mw-config-lastindex
operations/puppetproduction+0 -246icinga: remove check_lonqueries.pl
operations/puppetproduction+6 -6nagios_common: convert check_jnx_alarms to use Monitoring::Plugin
operations/puppetproduction+0 -127nagios_common: remove check_sslxNN
operations/puppetproduction+5 -5nagios_common: convert check_bgp to use Monitoring::Plugin
operations/puppetproduction+6 -6nagios_common: switch check_ssl to use Monitoring::Plugin
operations/puppetproduction+43 -9nagios_common: add stretch support to check_ssl
operations/puppetproduction+6 -7nagios_common: on jessie, also install libmonitoring-plugin-perl
operations/puppetproduction+361 -3icinga/check_ssl: add support for stretch, rename it to check_tls
operations/puppetproduction+1 -1base/nrpe: add icinga1001 to allowed_hosts
operations/puppetproduction+0 -1icinga: enable icinga service on icinga1001
operations/puppetproduction+3 -1base/icinga/nrpe: move nrpe_allowed IPs to Hiera, add icinga1001
operations/puppetproduction+1 -1base: do not allow mailman server to NRPE to other hosts for no reason
operations/puppetproduction+8 -1icinga: move configuration directory for resource.cfg
operations/puppetproduction+1 -1icinga: use correct user for tmpfs mount
operations/puppetproduction+19 -6icinga, nagios_common: use commands.cfg on stretch
operations/puppetproduction+1 K -54icinga: add new icinga.cfg and move configuration
operations/puppetproduction+21 -11icinga: service management depends on debian version
operations/puppetproduction+33 -10icinga: move init script customizations to default
operations/puppetproduction+2 -0network::constancts: add icinga1001 to monitoring_hosts
operations/puppetproduction+94 -4naggen2: restrict generated defines to valid options
operations/puppetproduction+12 -4icinga: set user/group for event_handlers::raid
operations/puppetproduction+16 -16nagios_common: make user/group configurable from Hiera
operations/puppetproduction+3 -1icinga::plugins: set user/group for nagios_common::commands
operations/puppetproduction+5 -3icinga: tell rsync server to use IPv6
operations/dnsmaster+2 -2fix IPv6 for icinga1001, is in eqiad row C, not row A
operations/puppetproduction+27 -20icinga: allow rsync from multiple hosts
operations/puppetproduction+30 -20icinga: allow rsync from multiple hosts
operations/puppetproduction+37 -33icinga: linter compliance and stretch user
operations/puppetproduction+5 -0icinga: enforce mode on nagios files
operations/puppetproduction+6 -0icinga: pass user/group from profile, change to nagios on icinga1001
operations/puppetproduction+6 -6icinga: fix more places with hardcoded user name
operations/puppetproduction+24 -23icinga: make group configurable, make user configurable pt2
operations/puppetproduction+15 -18icinga: use $icinga::icinga_user in all classes
operations/puppetproduction+27 -23icinga: make icinga user configurable
operations/puppetproduction+23 -19icinga: (if on stretch) don't let puppet manage user/group
operations/puppetproduction+9 -42icinga: remove nsca::firewall class, use ferm::service in profile
operations/puppetproduction+7 -9icinga: remove icinga::group class
operations/puppetproduction+6 -1monitoring: set mode on host and service configs
operations/puppetproduction+1 -1icinga: enable acme (letsencrypt) on icinga1001
operations/puppetproduction+20 -10icinga: make the apache virtual host name configurable
operations/dnsmaster+1 -0add icinga-stretch subdomain cnamed for icinga1001
operations/puppetproduction+16 -2profile/cumin: update python scripts to detect command file
operations/puppetproduction+7 -2icinga: add stretch support to submit_check_result.sh
operations/puppetproduction+6 -1icinga-downtime: add stretch support
operations/puppetproduction+8 -1profile: use mariadb-client in stretch.
operations/puppetproduction+6 -3icinga::web: install libapache2-mod-php if on stretch
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
gerritbot added a comment.Oct 23 2018, 10:08 PM

Change 469333 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] icinga: use fping instead of ping for faster host checks

https://gerrit.wikimedia.org/r/469333

gerritbot added a comment.Oct 24 2018, 12:37 AM

Change 469253 merged by Dzahn:
[operations/puppet@production] icinga: allow configuring max_concurrent_checks via Hiera

https://gerrit.wikimedia.org/r/469253

gerritbot added a comment.Oct 24 2018, 12:48 AM

Change 469317 merged by Dzahn:
[operations/puppet@production] icinga: don't log service/host check retries

https://gerrit.wikimedia.org/r/469317

gerritbot added a comment.Oct 25 2018, 8:39 PM

Change 467017 abandoned by Dzahn:
icinga/etcd: /var/run/icinga/ -> /var/run/nagios/

Reason:
done in https://gerrit.wikimedia.org/r/#/c/operations/puppet/ /468414/ instead

https://gerrit.wikimedia.org/r/467017

gerritbot added a comment.Oct 25 2018, 9:20 PM

Change 469780 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] icinga: install nsca_frack.cfg in objects on stretch

https://gerrit.wikimedia.org/r/469780

gerritbot added a comment.Oct 25 2018, 10:32 PM

Change 469780 merged by Dzahn:
[operations/puppet@production] icinga: install nsca_frack.cfg in objects on stretch

https://gerrit.wikimedia.org/r/469780

gerritbot added a comment.Oct 26 2018, 12:32 AM

Change 469808 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] icinga/nsca: allow configuring nsca chroot in Hiera, change on stretch

https://gerrit.wikimedia.org/r/469808

Dzahn mentioned this in T208066: Push check latency and check execution time to Prometheus.Oct 26 2018, 5:37 PM
gerritbot added a comment.Oct 26 2018, 6:45 PM

Change 469808 merged by Dzahn:
[operations/puppet@production] icinga/nsca: fix nsca_chroot path on stretch

https://gerrit.wikimedia.org/r/469808

gerritbot added a comment.Oct 26 2018, 7:08 PM

Change 469988 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] icinga/nsca: fix command_file path on stretch

https://gerrit.wikimedia.org/r/469988

gerritbot added a comment.Oct 26 2018, 7:16 PM

Change 469988 merged by Dzahn:
[operations/puppet@production] icinga/nsca: fix command_file path on stretch

https://gerrit.wikimedia.org/r/469988

ops-monitoring-bot added a subscriber: ops-monitoring-bot.Oct 26 2018, 9:32 PM

Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts:

icinga1001.wikimedia.org

The log can be found in /var/log/wmf-auto-reimage/201810262131_dzahn_122415_icinga1001_wikimedia_org.log.

ops-monitoring-bot added a comment.Oct 26 2018, 9:32 PM

Completed auto-reimage of hosts:

['icinga1001.wikimedia.org']

Of which those FAILED:

['icinga1001.wikimedia.org']
ops-monitoring-bot added a comment.Oct 26 2018, 9:33 PM

Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts:

icinga1001.wikimedia.org

The log can be found in /var/log/wmf-auto-reimage/201810262133_dzahn_122885_icinga1001_wikimedia_org.log.

Dzahn mentioned this in T208100: cumin tries to downtime Icinga even with --no-downtime.Oct 26 2018, 10:07 PM
ops-monitoring-bot added a comment.Oct 27 2018, 12:00 AM

Completed auto-reimage of hosts:

['icinga1001.wikimedia.org']

Of which those FAILED:

['icinga1001.wikimedia.org']
Stashbot added a comment.Oct 27 2018, 12:00 AM

Mentioned in SAL (#wikimedia-operations) [2018-10-27T00:00:06Z] <mutante> icinga1001 - using wmf-auto-reimage to reinstall gets stuck at initial puppet run after reboot - Still waiting for Puppet after 105.0 minutes - aborting on cumin, loggin in directly and manually running puppet (T202782 T208100)

gerritbot added a comment.Oct 27 2018, 12:33 AM

Change 470106 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] icinga/alerting_host: ensure mpm_prefork is selected for httpd and php7.0

https://gerrit.wikimedia.org/r/470106

gerritbot added a comment.Oct 27 2018, 12:34 AM

Change 470106 merged by Dzahn:
[operations/puppet@production] icinga/alerting_host: ensure mpm_prefork is selected for httpd and php7.0

https://gerrit.wikimedia.org/r/470106

Dzahn added a subtask: T208108: httpd class and php7.0 - conflict with mpm_event module.Oct 27 2018, 12:38 AM
Volans added a comment.Oct 27 2018, 9:53 AM
In T202782#4699100, @Stashbot wrote:

Mentioned in SAL (#wikimedia-operations) [2018-10-27T00:00:06Z] <mutante> icinga1001 - using wmf-auto-reimage to reinstall gets stuck at initial puppet run after reboot - Still waiting for Puppet after 105.0 minutes - aborting on cumin, loggin in directly and manually running puppet (T202782 T208100)

What got stuck?
The first puppet run can easily take up to 2h of time based on the host role and server capabilities. You can check its progress tailing the cumin log whose path is listed at the start of the reimage process to stdout (the 2nd of the two logs listed).
To check further what's going on directly in the host you could also login either with the install_console script or your personal key (based where the host is in the puppetization process) or, worse case scenario you can't with both, via the mgmt console as root.

gerritbot added a comment.Oct 30 2018, 11:21 PM

Change 469320 merged by Dzahn:
[operations/puppet@production] icinga: logging optimizations

https://gerrit.wikimedia.org/r/469320

gerritbot added a comment.Oct 31 2018, 9:10 PM

Change 467015 merged by Dzahn:
[operations/puppet@production] etcd::monitoring: replace Nagios::Plugin with Monitoring::Plugin on stretch

https://gerrit.wikimedia.org/r/467015

gerritbot added a comment.Oct 31 2018, 9:25 PM

Change 470915 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] icinga: don't install PHP httpd module on stretch

https://gerrit.wikimedia.org/r/470915

gerritbot added a comment.Oct 31 2018, 9:41 PM

Change 470915 merged by Dzahn:
[operations/puppet@production] icinga: don't install PHP httpd module on stretch

https://gerrit.wikimedia.org/r/470915

gerritbot added a comment.Oct 31 2018, 11:59 PM

Change 470955 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] icinga: on stretch, don't use custom logrotate config

https://gerrit.wikimedia.org/r/470955

gerritbot added a comment.Nov 1 2018, 12:27 AM

Change 470955 merged by Dzahn:
[operations/puppet@production] icinga: on stretch, don't use custom logrotate config

https://gerrit.wikimedia.org/r/470955

Dzahn added a comment.EditedNov 1 2018, 12:32 AM
In T202782#4699593, @Volans wrote:

What got stuck?

It never detected a succesful run because it had this error: T208108

which i fixed with https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/470106/

I will reinstall one more time to double check there was nothing else.

Logging in wasn't the problem, it was already after creating all the users.

gerritbot added a comment.Nov 1 2018, 1:44 AM

Change 462600 merged by Dzahn:
[operations/puppet@production] icinga: on stretch, use systemd::service, unit file by systemd-sysv-generator

https://gerrit.wikimedia.org/r/462600

ops-monitoring-bot added a comment.Nov 3 2018, 2:58 AM

Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts:

icinga1001.wikimedia.org

The log can be found in /var/log/wmf-auto-reimage/201811030258_dzahn_163438_icinga1001_wikimedia_org.log.

ops-monitoring-bot added a comment.Nov 3 2018, 3:35 AM

Completed auto-reimage of hosts:

['icinga1001.wikimedia.org']

and were ALL successful.

gerritbot added a comment.Nov 5 2018, 10:16 PM

Change 471866 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] nagios_common: ensure libtimedate-perl is installed for check_ssl

https://gerrit.wikimedia.org/r/471866

gerritbot added a comment.Nov 5 2018, 10:27 PM

Change 471866 merged by Dzahn:
[operations/puppet@production] nagios_common: ensure libtimedate-perl is installed for check_ssl

https://gerrit.wikimedia.org/r/471866

gerritbot added a comment.Nov 5 2018, 11:13 PM

Change 471873 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] icinga: fix ownership of command file dir on stretch

https://gerrit.wikimedia.org/r/471873

gerritbot added a comment.Nov 6 2018, 1:03 AM

Change 471873 merged by Dzahn:
[operations/puppet@production] icinga: fix ownership of command file dir on stretch

https://gerrit.wikimedia.org/r/471873

ops-monitoring-bot added a comment.Nov 6 2018, 1:34 AM

Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts:

icinga1001.wikimedia.org

The log can be found in /var/log/wmf-auto-reimage/201811060134_dzahn_77047_icinga1001_wikimedia_org.log.

gerritbot added a comment.Nov 6 2018, 1:58 AM

Change 471900 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] mailman: whitelist icinga1001 IP alongside einsteinium and tegmen

https://gerrit.wikimedia.org/r/471900

ops-monitoring-bot added a comment.Nov 6 2018, 2:07 AM

Completed auto-reimage of hosts:

['icinga1001.wikimedia.org']

Of which those FAILED:

['icinga1001.wikimedia.org']
gerritbot added a comment.Nov 6 2018, 4:29 PM

Change 465519 merged by Dzahn:
[operations/puppet@production] base/icinga: use monitoring_hosts constant as NRPE allowed_hosts

https://gerrit.wikimedia.org/r/465519

gerritbot added a comment.Nov 6 2018, 5:07 PM

Change 472006 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] base/NRPE: make nrpe_allowed_hosts flexible on $realm

https://gerrit.wikimedia.org/r/472006

gerritbot added a comment.Nov 6 2018, 5:13 PM

Change 472006 merged by Dzahn:
[operations/puppet@production] base/NRPE: make nrpe_allowed_hosts flexible on $realm

https://gerrit.wikimedia.org/r/472006

gerritbot added a comment.Nov 6 2018, 8:19 PM

Change 471900 merged by Dzahn:
[operations/puppet@production] mailman: whitelist icinga1001 IP alongside einsteinium and tegmen

https://gerrit.wikimedia.org/r/471900

colewhite closed subtask T207775: newer version of nagios-nrpe-plugin nrpe (check_nrpe) with fixed logging issue on stretch icinga as Resolved.Nov 6 2018, 8:55 PM
Stashbot added a comment.Nov 6 2018, 9:50 PM

Mentioned in SAL (#wikimedia-operations) [2018-11-06T21:50:34Z] <mutante> icinga1001-"MediaWiki EtcdConfig up-to-date" checks were all UNKNOWN because systemd unit update-etcd-mw-config-lastindex was present but service not running. it was turned off in https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/427328/ on purpose. manually ran "systemctl start update-etcd-mw-config-lastindex" and the checks all work (T202782)

gerritbot added a comment.Nov 7 2018, 11:54 PM

Change 472352 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] icinga: fix path to retention.dat file on stretch

https://gerrit.wikimedia.org/r/472352

gerritbot added a comment.Nov 8 2018, 6:48 PM

Change 472352 merged by Dzahn:
[operations/puppet@production] icinga: fix path to retention.dat file on stretch

https://gerrit.wikimedia.org/r/472352

gerritbot added a comment.Nov 8 2018, 7:39 PM

Change 472519 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] icinga: add tmpfs options to stretch

https://gerrit.wikimedia.org/r/472519

gerritbot added a comment.Nov 8 2018, 8:36 PM

Change 472519 merged by Dzahn:
[operations/puppet@production] icinga: add tmpfs options to stretch

https://gerrit.wikimedia.org/r/472519

jijiki added a subscriber: jijiki.Nov 12 2018, 4:35 PM
gerritbot added a comment.Nov 13 2018, 3:51 PM

Change 473235 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/dns@master] switch icinga-stretch to icinga and icinga to icinga-old

https://gerrit.wikimedia.org/r/473235

Dzahn updated the task description. (Show Details)Nov 13 2018, 3:54 PM
gerritbot added a comment.Nov 13 2018, 4:24 PM

Change 473244 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] icinga: make icinga1001 active and einsteinium passive

https://gerrit.wikimedia.org/r/473244

gerritbot added a comment.Nov 13 2018, 5:03 PM

Change 473244 merged by Dzahn:
[operations/puppet@production] icinga: make icinga1001 active and einsteinium passive

https://gerrit.wikimedia.org/r/473244

gerritbot added a comment.Nov 13 2018, 5:12 PM

Change 473235 merged by Dzahn:
[operations/dns@master] switch icinga-stretch to icinga and icinga to icinga-old

https://gerrit.wikimedia.org/r/473235

gerritbot added a comment.Nov 13 2018, 5:19 PM

Change 473250 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] icinga: disable service notifications on einsteinium and enable on icinga1001

https://gerrit.wikimedia.org/r/473250

gerritbot added a comment.Nov 13 2018, 5:22 PM

Change 473251 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] icinga: transition to new active host

https://gerrit.wikimedia.org/r/473251

gerritbot added a comment.Nov 13 2018, 5:22 PM

Change 473250 merged by Dzahn:
[operations/puppet@production] icinga: disable service notifications on einsteinium and enable on icinga1001

https://gerrit.wikimedia.org/r/473250

gerritbot added a comment.Nov 13 2018, 5:24 PM

Change 473251 merged by Cwhite:
[operations/puppet@production] icinga: transition to new active host

https://gerrit.wikimedia.org/r/473251

Stashbot added a comment.Nov 13 2018, 6:03 PM

Mentioned in SAL (#wikimedia-operations) [2018-11-13T18:03:04Z] <mutante> icinga migration has concluded, we are now on stretch and icinga1001, einsteinium is passive (T202782)

gerritbot added a comment.Nov 13 2018, 6:48 PM

Change 473261 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] icinga: fix path to puppet_hosts in icinga-downtime

https://gerrit.wikimedia.org/r/473261

gerritbot added a comment.Nov 13 2018, 6:52 PM

Change 473261 merged by Dzahn:
[operations/puppet@production] icinga: fix path to puppet_hosts in icinga-downtime

https://gerrit.wikimedia.org/r/473261

Dzahn updated the task description. (Show Details)Nov 13 2018, 7:40 PM

The switch has happened. icinga1001 is now the active server and einsteinium is passive. This ticket still open for a grace period until we decom einsteinium.

gerritbot added a comment.Nov 13 2018, 8:12 PM

Change 473275 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] icinga: enable daemon log, in addition to syslog, again

https://gerrit.wikimedia.org/r/473275

gerritbot added a comment.Nov 13 2018, 8:26 PM

Change 473276 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] icinga: remove jessie support

https://gerrit.wikimedia.org/r/473276

gerritbot added a comment.Nov 13 2018, 8:29 PM

Change 473275 merged by Dzahn:
[operations/puppet@production] icinga: enable daemon log, in addition to syslog, again

https://gerrit.wikimedia.org/r/473275

gerritbot added a comment.Nov 13 2018, 9:28 PM

Change 473282 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] smokeping: replace target einsteinium with dns1002

https://gerrit.wikimedia.org/r/473282

gerritbot added a comment.Nov 14 2018, 12:33 AM

Change 473282 merged by Dzahn:
[operations/puppet@production] smokeping: replace target einsteinium with authdns1001

https://gerrit.wikimedia.org/r/473282

Dzahn closed subtask T208824: rename tegmen to icinga2001 and reinstall it with stretch as Resolved.Nov 15 2018, 9:13 PM
Dzahn added a subtask: T209738: decom einsteinium.Nov 16 2018, 11:55 PM
Dzahn changed the status of subtask T209738: decom einsteinium from Open to Stalled.Nov 16 2018, 11:58 PM
Dzahn closed this task as Resolved.Nov 17 2018, 12:00 AM
Dzahn updated the task description. (Show Details)
Dzahn updated the task description. (Show Details)

this ticket is resolved, einsteinium has been replaced by icinga1001 on stretch.

the rest of the steps will be part of the decom ticket now created at T209738 with the official decom checkboxes (and i added the one to remove from AQL that was here previously)

gerritbot added a comment.Nov 17 2018, 12:19 AM

Change 473278 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] icinga: remove einsteinium as an alerting_host

https://gerrit.wikimedia.org/r/473278

gerritbot added a comment.Nov 17 2018, 1:06 AM

Change 469333 abandoned by Dzahn:
icinga: on stretch, use fping instead of ping for faster host checks

https://gerrit.wikimedia.org/r/469333

gerritbot added a comment.Nov 17 2018, 2:35 PM

Change 474463 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] icinga: fix path to puppet_hosts/services in default_icinga.sh

https://gerrit.wikimedia.org/r/474463

gerritbot added a comment.Nov 17 2018, 2:37 PM

Change 474463 merged by Dzahn:
[operations/puppet@production] icinga: fix path to puppet_hosts/services in default_icinga.sh

https://gerrit.wikimedia.org/r/474463

gerritbot added a comment.Nov 17 2018, 2:50 PM

Change 474464 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] icinga: do not manage retention.dat in puppet

https://gerrit.wikimedia.org/r/474464

Dzahn added a subtask: T209757: Notifications disablement via puppet not working on icinga.Nov 17 2018, 2:51 PM
gerritbot added a comment.Nov 17 2018, 2:54 PM

Change 474464 merged by Dzahn:
[operations/puppet@production] icinga: do not manage retention.dat in puppet

https://gerrit.wikimedia.org/r/474464

Dzahn closed subtask T209757: Notifications disablement via puppet not working on icinga as Resolved.Nov 17 2018, 3:41 PM
gerritbot added a comment.Nov 26 2018, 6:14 PM

Change 473278 merged by Dzahn:
[operations/puppet@production] icinga: remove einsteinium as an alerting_host

https://gerrit.wikimedia.org/r/473278

gerritbot added a comment.Nov 26 2018, 9:58 PM

Change 475881 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] icinga/NSCA (passive checks): remove jessie support hacks

https://gerrit.wikimedia.org/r/475881

gerritbot added a comment.Nov 26 2018, 10:25 PM

Change 475881 merged by Dzahn:
[operations/puppet@production] icinga/NSCA (passive checks): remove jessie support hacks

https://gerrit.wikimedia.org/r/475881

gerritbot added a comment.Nov 26 2018, 10:31 PM

Change 475901 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] icinga::web: do not use PHP anymore

https://gerrit.wikimedia.org/r/475901

gerritbot added a comment.Nov 27 2018, 12:21 AM

Change 475901 merged by Dzahn:
[operations/puppet@production] icinga::web: do not use PHP anymore

https://gerrit.wikimedia.org/r/475901

gerritbot added a comment.Nov 27 2018, 12:28 AM

Change 475927 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] icinga: stop having separate configs for jessie/stretch

https://gerrit.wikimedia.org/r/475927

gerritbot added a comment.Nov 27 2018, 12:32 AM

Change 475927 merged by Dzahn:
[operations/puppet@production] icinga: stop having separate configs for jessie/stretch

https://gerrit.wikimedia.org/r/475927

gerritbot added a comment.Nov 27 2018, 12:44 AM

Change 473276 merged by Dzahn:
[operations/puppet@production] icinga: remove jessie support

https://gerrit.wikimedia.org/r/473276

Dzahn changed the status of subtask T209738: decom einsteinium from Stalled to Open.Nov 27 2018, 12:59 AM
colewhite closed subtask T208066: Push check latency and check execution time to Prometheus as Resolved.Dec 11 2018, 6:39 PM
fgiunchedi mentioned this in T212669: icinga doesn't log ampersand in notes_url links.Dec 28 2018, 3:29 PM
gerritbot added a comment.Nov 6 2019, 11:05 PM

Change 463133 abandoned by Cwhite:
naggen2: python3 and remove activerecord support

Reason:
This CS is getting long in the tooth and is generally terrible. Abandoning in favor of I82d8a0d833952475c4f665b3a317bd0c3131ab1b

https://gerrit.wikimedia.org/r/463133

Papaul closed subtask T209738: decom einsteinium as Resolved.Nov 8 2019, 7:45 PM
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL