Page MenuHomePhabricator

ops-monitoring-bot (Operations Monitoring Bot)
UserBot

Projects

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Saturday

  • Clear sailing ahead.

User Details

User Since
Aug 12 2016, 1:45 PM (170 w, 6 d)
Roles
Bot
Availability
Available
LDAP User
Unknown
MediaWiki User
Unknown

Bot managed by Operations for automated interaction with Phabricator from monitoring tools.

Recent Activity

Today

ops-monitoring-bot added a comment to T227432: Replace Varnish backends with ATS on cache text nodes.

Completed auto-reimage of hosts:

['cp1079.eqiad.wmnet']
Thu, Nov 21, 3:18 PM · Patch-For-Review, Traffic, Operations
ops-monitoring-bot added a comment to T227432: Replace Varnish backends with ATS on cache text nodes.

Script wmf-auto-reimage was launched by ema on cumin1001.eqiad.wmnet for hosts:

['cp1079.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201911211453_ema_89543.log.

Thu, Nov 21, 2:53 PM · Patch-For-Review, Traffic, Operations
ops-monitoring-bot added a comment to T227432: Replace Varnish backends with ATS on cache text nodes.

Completed auto-reimage of hosts:

['cp1077.eqiad.wmnet']
Thu, Nov 21, 1:50 PM · Patch-For-Review, Traffic, Operations
ops-monitoring-bot added a comment to T227432: Replace Varnish backends with ATS on cache text nodes.

Script wmf-auto-reimage was launched by ema on cumin1001.eqiad.wmnet for hosts:

['cp1077.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201911211325_ema_72700.log.

Thu, Nov 21, 1:25 PM · Patch-For-Review, Traffic, Operations
ops-monitoring-bot added a comment to T238817: Request routing to active/passive services active in codfw only stopped working.

Completed auto-reimage of hosts:

['cp2023.codfw.wmnet']
Thu, Nov 21, 10:04 AM · Operations, Traffic
ops-monitoring-bot added a comment to T238817: Request routing to active/passive services active in codfw only stopped working.

Script wmf-auto-reimage was launched by ema on cumin2001.codfw.wmnet for hosts:

['cp2023.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201911210942_ema_6885.log.

Thu, Nov 21, 9:43 AM · Operations, Traffic

Yesterday

ops-monitoring-bot added a comment to T237360: cp5012 fails to boot after reimage: junk in compressed archive unpacking initramfs.

Completed auto-reimage of hosts:

['cp2023.codfw.wmnet']
Wed, Nov 20, 5:25 PM · Traffic, Operations
ops-monitoring-bot added a comment to T237360: cp5012 fails to boot after reimage: junk in compressed archive unpacking initramfs.

Script wmf-auto-reimage was launched by ema on cumin2001.codfw.wmnet for hosts:

['cp2023.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201911201650_ema_5874.log.

Wed, Nov 20, 4:50 PM · Traffic, Operations
ops-monitoring-bot added a comment to T227432: Replace Varnish backends with ATS on cache text nodes.

Script wmf-auto-reimage was launched by ema on cumin2001.codfw.wmnet for hosts:

['cp2023.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201911201625_ema_1090.log.

Wed, Nov 20, 4:25 PM · Patch-For-Review, Traffic, Operations
ops-monitoring-bot added a comment to T227432: Replace Varnish backends with ATS on cache text nodes.

Completed auto-reimage of hosts:

['cp2019.codfw.wmnet']
Wed, Nov 20, 3:54 PM · Patch-For-Review, Traffic, Operations
ops-monitoring-bot added a comment to T227432: Replace Varnish backends with ATS on cache text nodes.

Script wmf-auto-reimage was launched by ema on cumin2001.codfw.wmnet for hosts:

['cp2019.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201911201523_ema_21320.log.

Wed, Nov 20, 3:23 PM · Patch-For-Review, Traffic, Operations
ops-monitoring-bot added a comment to T227432: Replace Varnish backends with ATS on cache text nodes.

Completed auto-reimage of hosts:

['cp2016.codfw.wmnet']
Wed, Nov 20, 2:37 PM · Patch-For-Review, Traffic, Operations
ops-monitoring-bot added a comment to T227432: Replace Varnish backends with ATS on cache text nodes.

Script wmf-auto-reimage was launched by ema on cumin2001.codfw.wmnet for hosts:

['cp2016.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201911201409_ema_5537.log.

Wed, Nov 20, 2:09 PM · Patch-For-Review, Traffic, Operations
ops-monitoring-bot added a comment to T227432: Replace Varnish backends with ATS on cache text nodes.

Completed auto-reimage of hosts:

['cp2013.codfw.wmnet']
Wed, Nov 20, 12:38 PM · Patch-For-Review, Traffic, Operations
ops-monitoring-bot added a comment to T227432: Replace Varnish backends with ATS on cache text nodes.

Script wmf-auto-reimage was launched by ema on cumin2001.codfw.wmnet for hosts:

['cp2013.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201911201209_ema_14766.log.

Wed, Nov 20, 12:09 PM · Patch-For-Review, Traffic, Operations
ops-monitoring-bot added a comment to T227432: Replace Varnish backends with ATS on cache text nodes.

Completed auto-reimage of hosts:

['cp2012.codfw.wmnet']
Wed, Nov 20, 11:52 AM · Patch-For-Review, Traffic, Operations
ops-monitoring-bot added a comment to T227432: Replace Varnish backends with ATS on cache text nodes.

Script wmf-auto-reimage was launched by ema on cumin2001.codfw.wmnet for hosts:

['cp2012.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201911201124_ema_4809.log.

Wed, Nov 20, 11:24 AM · Patch-For-Review, Traffic, Operations
ops-monitoring-bot added a comment to T227432: Replace Varnish backends with ATS on cache text nodes.

Completed auto-reimage of hosts:

['cp2010.codfw.wmnet']
Wed, Nov 20, 11:04 AM · Patch-For-Review, Traffic, Operations
ops-monitoring-bot added a comment to T227432: Replace Varnish backends with ATS on cache text nodes.

Script wmf-auto-reimage was launched by ema on cumin2001.codfw.wmnet for hosts:

['cp2010.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201911201036_ema_25695.log.

Wed, Nov 20, 10:36 AM · Patch-For-Review, Traffic, Operations
ops-monitoring-bot added a comment to T238526: Decommission db2061.codfw.wmnet.

cookbooks.sre.hosts.decommission executed by marostegui@cumin1001 for hosts: db2061.codfw.wmnet

  • db2061.codfw.wmnet (PASS)
    • Downtimed host on Icinga
    • Downtimed management interface on Icinga
    • Wiped bootloaders
    • Powered off
    • Set Netbox status to Decommissioning
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
Wed, Nov 20, 8:02 AM · Operations, ops-codfw, decommission

Tue, Nov 19

ops-monitoring-bot added a comment to T227432: Replace Varnish backends with ATS on cache text nodes.

Completed auto-reimage of hosts:

['cp2007.codfw.wmnet']
Tue, Nov 19, 4:45 PM · Patch-For-Review, Traffic, Operations
ops-monitoring-bot added a comment to T227432: Replace Varnish backends with ATS on cache text nodes.

Script wmf-auto-reimage was launched by ema on cumin2001.codfw.wmnet for hosts:

['cp2007.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201911191612_ema_8527.log.

Tue, Nov 19, 4:12 PM · Patch-For-Review, Traffic, Operations
ops-monitoring-bot added a comment to T227432: Replace Varnish backends with ATS on cache text nodes.

Completed auto-reimage of hosts:

['cp2006.codfw.wmnet']
Tue, Nov 19, 4:06 PM · Patch-For-Review, Traffic, Operations
ops-monitoring-bot added a comment to T227432: Replace Varnish backends with ATS on cache text nodes.

Script wmf-auto-reimage was launched by ema on cumin2001.codfw.wmnet for hosts:

['cp2006.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201911191519_ema_30917.log.

Tue, Nov 19, 3:19 PM · Patch-For-Review, Traffic, Operations
ops-monitoring-bot added a comment to T227432: Replace Varnish backends with ATS on cache text nodes.

Completed auto-reimage of hosts:

['cp2004.codfw.wmnet']
Tue, Nov 19, 2:52 PM · Patch-For-Review, Traffic, Operations
ops-monitoring-bot added a comment to T227432: Replace Varnish backends with ATS on cache text nodes.

Script wmf-auto-reimage was launched by ema on cumin2001.codfw.wmnet for hosts:

['cp2004.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201911191420_ema_19278.log.

Tue, Nov 19, 2:21 PM · Patch-For-Review, Traffic, Operations
ops-monitoring-bot added a comment to T227432: Replace Varnish backends with ATS on cache text nodes.

Completed auto-reimage of hosts:

['cp2001.codfw.wmnet']
Tue, Nov 19, 1:52 PM · Patch-For-Review, Traffic, Operations
ops-monitoring-bot added a comment to T227432: Replace Varnish backends with ATS on cache text nodes.

Script wmf-auto-reimage was launched by ema on cumin2001.codfw.wmnet for hosts:

['cp2001.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201911191318_ema_6176.log.

Tue, Nov 19, 1:18 PM · Patch-For-Review, Traffic, Operations

Mon, Nov 18

ops-monitoring-bot added a comment to T190568: Reimage both phab1001 and phab2001 to stretch / buster.

Completed auto-reimage of hosts:

['phab2001.codfw.wmnet']
Mon, Nov 18, 11:20 PM · Patch-For-Review, Release-Engineering-Team (Development services), Release-Engineering-Team-TODO, serviceops, Phabricator, Operations
ops-monitoring-bot added a comment to T190568: Reimage both phab1001 and phab2001 to stretch / buster.

Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts:

phab2001.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/201911182229_dzahn_139445_phab2001_codfw_wmnet.log.

Mon, Nov 18, 10:31 PM · Patch-For-Review, Release-Engineering-Team (Development services), Release-Engineering-Team-TODO, serviceops, Phabricator, Operations
ops-monitoring-bot added a comment to T224564: Reimage wezen to Stretch or Buster (and rename to centrallog2001).

Completed auto-reimage of hosts:

['wezen.codfw.wmnet']
Mon, Nov 18, 12:02 PM · User-fgiunchedi, observability, Operations
ops-monitoring-bot added a comment to T224564: Reimage wezen to Stretch or Buster (and rename to centrallog2001).

Script wmf-auto-reimage was launched by filippo on cumin1001.eqiad.wmnet for hosts:

wezen.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/201911181201_filippo_12902_wezen_codfw_wmnet.log.

Mon, Nov 18, 12:02 PM · User-fgiunchedi, observability, Operations

Sat, Nov 16

ops-monitoring-bot added a comment to T224563: Migrate dumpsdata hosts to Stretch/Buster.

Completed auto-reimage of hosts:

['dumpsdata1002.eqiad.wmnet']
Sat, Nov 16, 8:31 PM · Patch-For-Review, Dumps-Generation, Operations
ops-monitoring-bot added a comment to T224563: Migrate dumpsdata hosts to Stretch/Buster.

Script wmf-auto-reimage was launched by ariel on cumin1001.eqiad.wmnet for hosts:

['dumpsdata1002.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201911161927_ariel_118105.log.

Sat, Nov 16, 7:28 PM · Patch-For-Review, Dumps-Generation, Operations

Fri, Nov 15

ops-monitoring-bot added a comment to T224188: rack/setup/install (3) new osd ceph nodes.

Completed auto-reimage of hosts:

['cloudcephosd1001.wikimedia.org']
Fri, Nov 15, 10:14 PM · Operations, cloud-services-team (Kanban), Cloud-Services
ops-monitoring-bot added a comment to T224188: rack/setup/install (3) new osd ceph nodes.

Script wmf-auto-reimage was launched by jeh on cumin1001.eqiad.wmnet for hosts:

['cloudcephosd1001.wikimedia.org']

The log can be found in /var/log/wmf-auto-reimage/201911152158_jeh_146929.log.

Fri, Nov 15, 9:58 PM · Operations, cloud-services-team (Kanban), Cloud-Services
ops-monitoring-bot added a comment to T224188: rack/setup/install (3) new osd ceph nodes.

Completed auto-reimage of hosts:

['cloudcephosd1003.wikimedia.org']
Fri, Nov 15, 9:58 PM · Operations, cloud-services-team (Kanban), Cloud-Services
ops-monitoring-bot added a comment to T224188: rack/setup/install (3) new osd ceph nodes.

Script wmf-auto-reimage was launched by jeh on cumin1001.eqiad.wmnet for hosts:

['cloudcephosd1003.wikimedia.org']

The log can be found in /var/log/wmf-auto-reimage/201911152139_jeh_143586.log.

Fri, Nov 15, 9:39 PM · Operations, cloud-services-team (Kanban), Cloud-Services
ops-monitoring-bot added a comment to T224188: rack/setup/install (3) new osd ceph nodes.

Completed auto-reimage of hosts:

['cloudcephosd1002.wikimedia.org']
Fri, Nov 15, 9:35 PM · Operations, cloud-services-team (Kanban), Cloud-Services
ops-monitoring-bot added a comment to T224188: rack/setup/install (3) new osd ceph nodes.

Script wmf-auto-reimage was launched by jeh on cumin1001.eqiad.wmnet for hosts:

['cloudcephosd1002.wikimedia.org']

The log can be found in /var/log/wmf-auto-reimage/201911152117_jeh_138328.log.

Fri, Nov 15, 9:17 PM · Operations, cloud-services-team (Kanban), Cloud-Services
ops-monitoring-bot added a comment to T224188: rack/setup/install (3) new osd ceph nodes.

Script wmf-auto-reimage was launched by jeh on cumin1001.eqiad.wmnet for hosts:

['cloudcephosd1002.wikimedia.org']

The log can be found in /var/log/wmf-auto-reimage/201911152041_jeh_132144.log.

Fri, Nov 15, 8:42 PM · Operations, cloud-services-team (Kanban), Cloud-Services
ops-monitoring-bot added a comment to T224188: rack/setup/install (3) new osd ceph nodes.

Script wmf-auto-reimage was launched by jeh on cumin1001.eqiad.wmnet for hosts:

['cloudcephosd1002.wikimedia.org']

The log can be found in /var/log/wmf-auto-reimage/201911151709_jeh_92596.log.

Fri, Nov 15, 5:10 PM · Operations, cloud-services-team (Kanban), Cloud-Services
ops-monitoring-bot added a comment to T227432: Replace Varnish backends with ATS on cache text nodes.

Completed auto-reimage of hosts:

['cp3064.esams.wmnet']
Fri, Nov 15, 2:58 PM · Patch-For-Review, Traffic, Operations
ops-monitoring-bot added a comment to T227432: Replace Varnish backends with ATS on cache text nodes.

Script wmf-auto-reimage was launched by ema on cumin1001.eqiad.wmnet for hosts:

['cp3064.esams.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201911151428_ema_140812.log.

Fri, Nov 15, 2:28 PM · Patch-For-Review, Traffic, Operations
ops-monitoring-bot added a comment to T227432: Replace Varnish backends with ATS on cache text nodes.

Completed auto-reimage of hosts:

['cp3062.esams.wmnet']
Fri, Nov 15, 10:29 AM · Patch-For-Review, Traffic, Operations
ops-monitoring-bot added a comment to T227432: Replace Varnish backends with ATS on cache text nodes.

Script wmf-auto-reimage was launched by ema on cumin1001.eqiad.wmnet for hosts:

['cp3062.esams.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201911150954_ema_187006.log.

Fri, Nov 15, 9:54 AM · Patch-For-Review, Traffic, Operations

Thu, Nov 14

ops-monitoring-bot added a comment to T224188: rack/setup/install (3) new osd ceph nodes.

Completed auto-reimage of hosts:

['cloudcephosd1002.wikimedia.org']
Thu, Nov 14, 4:21 PM · Operations, cloud-services-team (Kanban), Cloud-Services
ops-monitoring-bot added a comment to T224188: rack/setup/install (3) new osd ceph nodes.

Script wmf-auto-reimage was launched by jeh on cumin1001.eqiad.wmnet for hosts:

['cloudcephosd1002.wikimedia.org']

The log can be found in /var/log/wmf-auto-reimage/201911141607_jeh_94737.log.

Thu, Nov 14, 4:07 PM · Operations, cloud-services-team (Kanban), Cloud-Services
ops-monitoring-bot added a comment to T224188: rack/setup/install (3) new osd ceph nodes.

Completed auto-reimage of hosts:

['cloudcephosd1002.wikimedia.org']
Thu, Nov 14, 4:06 PM · Operations, cloud-services-team (Kanban), Cloud-Services
ops-monitoring-bot added a comment to T224188: rack/setup/install (3) new osd ceph nodes.

Script wmf-auto-reimage was launched by jeh on cumin1001.eqiad.wmnet for hosts:

['cloudcephosd1002.wikimedia.org']

The log can be found in /var/log/wmf-auto-reimage/201911141550_jeh_90677.log.

Thu, Nov 14, 3:51 PM · Operations, cloud-services-team (Kanban), Cloud-Services
ops-monitoring-bot added a comment to T227432: Replace Varnish backends with ATS on cache text nodes.

Completed auto-reimage of hosts:

['cp3060.esams.wmnet']
Thu, Nov 14, 2:39 PM · Patch-For-Review, Traffic, Operations
ops-monitoring-bot added a comment to T227432: Replace Varnish backends with ATS on cache text nodes.

Script wmf-auto-reimage was launched by ema on cumin1001.eqiad.wmnet for hosts:

['cp3060.esams.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201911141403_ema_69463.log.

Thu, Nov 14, 2:04 PM · Patch-For-Review, Traffic, Operations
ops-monitoring-bot added a comment to T227432: Replace Varnish backends with ATS on cache text nodes.

Completed auto-reimage of hosts:

['cp3058.esams.wmnet']
Thu, Nov 14, 10:36 AM · Patch-For-Review, Traffic, Operations
ops-monitoring-bot added a comment to T227432: Replace Varnish backends with ATS on cache text nodes.

Script wmf-auto-reimage was launched by ema on cumin1001.eqiad.wmnet for hosts:

['cp3058.esams.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201911141001_ema_240171.log.

Thu, Nov 14, 10:02 AM · Patch-For-Review, Traffic, Operations
ops-monitoring-bot added a comment to T227432: Replace Varnish backends with ATS on cache text nodes.

Script wmf-auto-reimage was launched by ema on cumin1001.eqiad.wmnet for hosts:

['cp3058.esams.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201911140935_ema_224372.log.

Thu, Nov 14, 9:36 AM · Patch-For-Review, Traffic, Operations
ops-monitoring-bot added a comment to T238183: Productionize db213[2-5}.

Completed auto-reimage of hosts:

['db2133.codfw.wmnet']
Thu, Nov 14, 9:13 AM · DBA
ops-monitoring-bot added a comment to T238183: Productionize db213[2-5}.

Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:

['db2133.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201911140849_marostegui_198980.log.

Thu, Nov 14, 8:49 AM · DBA
ops-monitoring-bot added a comment to T238183: Productionize db213[2-5}.

Completed auto-reimage of hosts:

['db2133.codfw.wmnet']
Thu, Nov 14, 8:46 AM · DBA
ops-monitoring-bot added a comment to T238183: Productionize db213[2-5}.

Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:

['db2133.codfw.wmnet', 'db2134.codfw.wmnet', 'db2135.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201911140823_marostegui_192468.log.

Thu, Nov 14, 8:23 AM · DBA

Wed, Nov 13

ops-monitoring-bot added a comment to T224188: rack/setup/install (3) new osd ceph nodes.

Completed auto-reimage of hosts:

['cloudcephosd1001.wikimedia.org']
Wed, Nov 13, 11:05 PM · Operations, cloud-services-team (Kanban), Cloud-Services
ops-monitoring-bot added a comment to T224188: rack/setup/install (3) new osd ceph nodes.

Script wmf-auto-reimage was launched by jeh on cumin1001.eqiad.wmnet for hosts:

['cloudcephosd1001.wikimedia.org']

The log can be found in /var/log/wmf-auto-reimage/201911132244_jeh_88199.log.

Wed, Nov 13, 10:45 PM · Operations, cloud-services-team (Kanban), Cloud-Services
ops-monitoring-bot added a comment to T224188: rack/setup/install (3) new osd ceph nodes.

Completed auto-reimage of hosts:

['cloudcephosd1001.wikimedia.org']
Wed, Nov 13, 9:44 PM · Operations, cloud-services-team (Kanban), Cloud-Services
ops-monitoring-bot added a comment to T224188: rack/setup/install (3) new osd ceph nodes.

Script wmf-auto-reimage was launched by jeh on cumin1001.eqiad.wmnet for hosts:

['cloudcephosd1001.wikimedia.org']

The log can be found in /var/log/wmf-auto-reimage/201911132125_jeh_73756.log.

Wed, Nov 13, 9:25 PM · Operations, cloud-services-team (Kanban), Cloud-Services
ops-monitoring-bot added a comment to T227432: Replace Varnish backends with ATS on cache text nodes.

Completed auto-reimage of hosts:

['cp3054.esams.wmnet']
Wed, Nov 13, 4:13 PM · Patch-For-Review, Traffic, Operations
ops-monitoring-bot added a comment to T227432: Replace Varnish backends with ATS on cache text nodes.

Script wmf-auto-reimage was launched by ema on cumin1001.eqiad.wmnet for hosts:

['cp3054.esams.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201911131538_ema_16683.log.

Wed, Nov 13, 3:39 PM · Patch-For-Review, Traffic, Operations
ops-monitoring-bot added a comment to T238183: Productionize db213[2-5}.

Completed auto-reimage of hosts:

['db2132.codfw.wmnet']
Wed, Nov 13, 8:25 AM · DBA
ops-monitoring-bot added a comment to T238183: Productionize db213[2-5}.

Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:

['db2132.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201911130800_marostegui_190803.log.

Wed, Nov 13, 8:00 AM · DBA
ops-monitoring-bot added projects to T238176: Degraded RAID on db2135: ops-codfw, Operations.
Wed, Nov 13, 1:19 AM · Operations, ops-codfw
ops-monitoring-bot added projects to T238175: Degraded RAID on db2134: ops-codfw, Operations.
Wed, Nov 13, 1:09 AM · Operations, ops-codfw

Tue, Nov 12

ops-monitoring-bot added a comment to T228102: rack/setup/install cloudcephmon100[123].

Completed auto-reimage of hosts:

['cloudcephmon1003.wikimedia.org']
Tue, Nov 12, 11:53 PM · cloud-services-team (Kanban), Operations, Cloud-Services
ops-monitoring-bot added a comment to T228102: rack/setup/install cloudcephmon100[123].

Script wmf-auto-reimage was launched by jeh on cumin1001.eqiad.wmnet for hosts:

['cloudcephmon1003.wikimedia.org']

The log can be found in /var/log/wmf-auto-reimage/201911122331_jeh_92744.log.

Tue, Nov 12, 11:31 PM · cloud-services-team (Kanban), Operations, Cloud-Services
ops-monitoring-bot added a comment to T228102: rack/setup/install cloudcephmon100[123].

Completed auto-reimage of hosts:

['cloudcephmon1002.wikimedia.org']
Tue, Nov 12, 11:27 PM · cloud-services-team (Kanban), Operations, Cloud-Services
ops-monitoring-bot added a comment to T228102: rack/setup/install cloudcephmon100[123].

Script wmf-auto-reimage was launched by jeh on cumin1001.eqiad.wmnet for hosts:

['cloudcephmon1002.wikimedia.org']

The log can be found in /var/log/wmf-auto-reimage/201911122306_jeh_87570.log.

Tue, Nov 12, 11:06 PM · cloud-services-team (Kanban), Operations, Cloud-Services
ops-monitoring-bot added a comment to T237133: analytics1062 lost one of its power supplies.

Icinga downtime for 2:00:00 set by otto@cumin1001 on 1 host(s) and their services with reason: analytics1062 lost one of its power supplies

analytics1062.eqiad.wmnet
Tue, Nov 12, 8:11 PM · Analytics, Operations, Analytics-Cluster, ops-eqiad
ops-monitoring-bot added a comment to T227432: Replace Varnish backends with ATS on cache text nodes.

Completed auto-reimage of hosts:

['cp3052.esams.wmnet']
Tue, Nov 12, 4:56 PM · Patch-For-Review, Traffic, Operations
ops-monitoring-bot added a comment to T227432: Replace Varnish backends with ATS on cache text nodes.

Script wmf-auto-reimage was launched by ema on cumin1001.eqiad.wmnet for hosts:

['cp3052.esams.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201911121619_ema_7588.log.

Tue, Nov 12, 4:19 PM · Patch-For-Review, Traffic, Operations
ops-monitoring-bot added a comment to T237913: Decommission db2048.codfw.wmnet.

cookbooks.sre.hosts.decommission executed by marostegui@cumin1001 for hosts: db2048.codfw.wmnet

  • db2048.codfw.wmnet (PASS)
    • Downtimed host on Icinga
    • Downtimed management interface on Icinga
    • Wiped bootloaders
    • Powered off
    • Set Netbox status to Decommissioning
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
Tue, Nov 12, 6:58 AM · ops-codfw, decommission, Operations

Mon, Nov 11

ops-monitoring-bot added a comment to T228102: rack/setup/install cloudcephmon100[123].

Completed auto-reimage of hosts:

['cloudcephmon1001.wikimedia.org']
Mon, Nov 11, 5:45 PM · cloud-services-team (Kanban), Operations, Cloud-Services
ops-monitoring-bot added a comment to T228102: rack/setup/install cloudcephmon100[123].

Script wmf-auto-reimage was launched by jeh on cumin1001.eqiad.wmnet for hosts:

['cloudcephmon1001.wikimedia.org']

The log can be found in /var/log/wmf-auto-reimage/201911111721_jeh_9851.log.

Mon, Nov 11, 5:22 PM · cloud-services-team (Kanban), Operations, Cloud-Services
ops-monitoring-bot added a comment to T227432: Replace Varnish backends with ATS on cache text nodes.

Completed auto-reimage of hosts:

['cp3050.esams.wmnet']
Mon, Nov 11, 2:02 PM · Patch-For-Review, Traffic, Operations
ops-monitoring-bot added a comment to T227432: Replace Varnish backends with ATS on cache text nodes.

Script wmf-auto-reimage was launched by ema on cumin1001.eqiad.wmnet for hosts:

['cp3050.esams.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201911111327_ema_226775.log.

Mon, Nov 11, 1:27 PM · Patch-For-Review, Traffic, Operations

Fri, Nov 8

ops-monitoring-bot added a comment to T228102: rack/setup/install cloudcephmon100[123].

Completed auto-reimage of hosts:

['cloudcephmon1001.wikimedia.org']
Fri, Nov 8, 4:55 PM · cloud-services-team (Kanban), Operations, Cloud-Services
ops-monitoring-bot added a comment to T228102: rack/setup/install cloudcephmon100[123].

Script wmf-auto-reimage was launched by jeh on cumin1001.eqiad.wmnet for hosts:

['cloudcephmon1001.wikimedia.org']

The log can be found in /var/log/wmf-auto-reimage/201911081636_jeh_236894.log.

Fri, Nov 8, 4:38 PM · cloud-services-team (Kanban), Operations, Cloud-Services
ops-monitoring-bot added a comment to T228102: rack/setup/install cloudcephmon100[123].

Completed auto-reimage of hosts:

['cloudcephmon1001.wikimedia.org']
Fri, Nov 8, 4:23 PM · cloud-services-team (Kanban), Operations, Cloud-Services
ops-monitoring-bot added a comment to T228102: rack/setup/install cloudcephmon100[123].

Script wmf-auto-reimage was launched by jeh on cumin1001.eqiad.wmnet for hosts:

['cloudcephmon1001.wikimedia.org']

The log can be found in /var/log/wmf-auto-reimage/201911081559_jeh_227246.log.

Fri, Nov 8, 4:00 PM · cloud-services-team (Kanban), Operations, Cloud-Services
ops-monitoring-bot added a comment to T229586: decommission cp1008, cp1071, cp1072, cp1073, cp1074, cp1099.

cookbooks.sre.hosts.decommission executed by volans@cumin1001 for hosts: cp1072.eqiad.wmnet

  • cp1072.eqiad.wmnet (PASS)
    • Downtimed host on Icinga
    • Downtimed management interface on Icinga
    • Wiped bootloaders
    • Powered off
    • Set Netbox status to Decommissioning
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
Fri, Nov 8, 2:56 PM · ops-eqiad, DC-Ops, Operations, decommission

Tue, Nov 5

ops-monitoring-bot added projects to T237457: Degraded RAID on cobalt: Operations, ops-eqiad.
Tue, Nov 5, 8:46 PM · ops-eqiad, Operations
ops-monitoring-bot added a comment to T236187: decom cobalt.

cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: cobalt.wikimedia.org

  • cobalt.wikimedia.org (PASS)
    • Downtimed host on Icinga
    • Downtimed management interface on Icinga
    • Wiped bootloaders
    • Powered off
    • Set Netbox status to Decommissioning
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
Tue, Nov 5, 8:44 PM · serviceops, Operations
ops-monitoring-bot added a comment to T237360: cp5012 fails to boot after reimage: junk in compressed archive unpacking initramfs.

Completed auto-reimage of hosts:

['cp5012.eqsin.wmnet']
Tue, Nov 5, 9:54 AM · Traffic, Operations
ops-monitoring-bot added a comment to T237360: cp5012 fails to boot after reimage: junk in compressed archive unpacking initramfs.

Script wmf-auto-reimage was launched by ema on cumin1001.eqiad.wmnet for hosts:

['cp5012.eqsin.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201911050905_ema_87780.log.

Tue, Nov 5, 9:05 AM · Traffic, Operations
ops-monitoring-bot added a comment to T237360: cp5012 fails to boot after reimage: junk in compressed archive unpacking initramfs.

Completed auto-reimage of hosts:

['cp5012.eqsin.wmnet']
Tue, Nov 5, 9:05 AM · Traffic, Operations
ops-monitoring-bot added a comment to T237360: cp5012 fails to boot after reimage: junk in compressed archive unpacking initramfs.

Script wmf-auto-reimage was launched by ema on cumin1001.eqiad.wmnet for hosts:

['cp5012.eqsin.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201911050904_ema_86783.log.

Tue, Nov 5, 9:04 AM · Traffic, Operations
ops-monitoring-bot added a comment to T237360: cp5012 fails to boot after reimage: junk in compressed archive unpacking initramfs.

Script wmf-auto-reimage was launched by ema on cumin1001.eqiad.wmnet for hosts:

['cp5012.eqsin.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201911050832_ema_80975.log.

Tue, Nov 5, 8:32 AM · Traffic, Operations

Mon, Nov 4

ops-monitoring-bot added a comment to T228606: Degraded RAID on elastic1046.

Completed auto-reimage of hosts:

['elastic1046.eqiad.wmnet']
Mon, Nov 4, 11:12 PM · Patch-For-Review, Discovery-Search (Current work), ops-eqiad, Operations
ops-monitoring-bot added a comment to T228606: Degraded RAID on elastic1046.

Script wmf-auto-reimage was launched by gehel on cumin1001.eqiad.wmnet for hosts:

['elastic1046.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201911042249_gehel_234323.log.

Mon, Nov 4, 10:49 PM · Patch-For-Review, Discovery-Search (Current work), ops-eqiad, Operations
ops-monitoring-bot added a comment to T227432: Replace Varnish backends with ATS on cache text nodes.

Completed auto-reimage of hosts:

['cp5012.eqsin.wmnet']
Mon, Nov 4, 3:50 PM · Patch-For-Review, Traffic, Operations
ops-monitoring-bot added a comment to T227432: Replace Varnish backends with ATS on cache text nodes.

Script wmf-auto-reimage was launched by ema on cumin1001.eqiad.wmnet for hosts:

['cp5012.eqsin.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201911041406_ema_127312.log.

Mon, Nov 4, 2:07 PM · Patch-For-Review, Traffic, Operations
ops-monitoring-bot added a comment to T227432: Replace Varnish backends with ATS on cache text nodes.

Script wmf-auto-reimage was launched by ema on cumin1001.eqiad.wmnet for hosts:

['cp5012.eqsin.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201911041319_ema_113165.log.

Mon, Nov 4, 1:19 PM · Patch-For-Review, Traffic, Operations

Fri, Nov 1

ops-monitoring-bot added a comment to T176774: Reimage gerrit1001 and gerrit2001 as buster.

Completed auto-reimage of hosts:

['gerrit2001.wikimedia.org']
Fri, Nov 1, 8:31 PM · Release-Engineering-Team (Development services), Release-Engineering-Team-TODO, Gerrit, Operations
ops-monitoring-bot added a comment to T176774: Reimage gerrit1001 and gerrit2001 as buster.

Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts:

gerrit2001.wikimedia.org

The log can be found in /var/log/wmf-auto-reimage/201911011935_dzahn_192096_gerrit2001_wikimedia_org.log.

Fri, Nov 1, 7:36 PM · Release-Engineering-Team (Development services), Release-Engineering-Team-TODO, Gerrit, Operations