Page MenuHomePhabricator

ops-monitoring-bot (Operations Monitoring Bot)
UserBot

Projects

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Wednesday

  • Clear sailing ahead.

User Details

User Since
Aug 12 2016, 1:45 PM (136 w, 3 d)
Roles
Bot
Availability
Available
LDAP User
Unknown
MediaWiki User
Unknown

Bot managed by Operations for automated interaction with Phabricator from monitoring tools.

Recent Activity

Wed, Mar 20

ops-monitoring-bot added projects to T218776: Degraded RAID on db2052: ops-codfw, Operations.
Wed, Mar 20, 1:12 PM · DBA, Operations, ops-codfw

Fri, Mar 15

ops-monitoring-bot added projects to T218405: Degraded RAID on labtestservices2002: ops-codfw, Operations.
Fri, Mar 15, 3:50 PM · cloud-services-team, Operations, ops-codfw
ops-monitoring-bot added projects to T218403: Degraded RAID on labtestcontrol2003: ops-codfw, Operations.
Fri, Mar 15, 3:43 PM · Operations, ops-codfw

Fri, Mar 8

ops-monitoring-bot added a comment to T209181: Decommission rdb1001, rdb1002, rdb1003, rdb1004, rdb1007, rdb1008.

wmf-decommission-host was executed by robh for rdb1008.eqiad.wmnet and performed the following actions:

  • Revoked Puppet certificate
  • Removed from PuppetDB
  • Downtimed host on Icinga
  • Downtimed mgmt interface on Icinga
  • Removed from DebMonitor
Fri, Mar 8, 6:04 PM · ops-eqiad, decommission, User-jijiki, Operations
ops-monitoring-bot added a comment to T209181: Decommission rdb1001, rdb1002, rdb1003, rdb1004, rdb1007, rdb1008.

wmf-decommission-host was executed by robh for rdb1007.eqiad.wmnet and performed the following actions:

  • Revoked Puppet certificate
  • Removed from PuppetDB
  • Downtimed host on Icinga
  • Downtimed mgmt interface on Icinga
  • Removed from DebMonitor
Fri, Mar 8, 6:04 PM · ops-eqiad, decommission, User-jijiki, Operations
ops-monitoring-bot added a comment to T209181: Decommission rdb1001, rdb1002, rdb1003, rdb1004, rdb1007, rdb1008.

wmf-decommission-host was executed by robh for rdb1004.eqiad.wmnet and performed the following actions:

  • Revoked Puppet certificate
  • Removed from PuppetDB
  • Downtimed host on Icinga
  • Downtimed mgmt interface on Icinga
  • Removed from DebMonitor
Fri, Mar 8, 6:04 PM · ops-eqiad, decommission, User-jijiki, Operations
ops-monitoring-bot added a comment to T209181: Decommission rdb1001, rdb1002, rdb1003, rdb1004, rdb1007, rdb1008.

wmf-decommission-host was executed by robh for rdb1003.eqiad.wmnet and performed the following actions:

  • Revoked Puppet certificate
  • Removed from PuppetDB
  • Downtimed host on Icinga
  • Downtimed mgmt interface on Icinga
  • Removed from DebMonitor
Fri, Mar 8, 6:04 PM · ops-eqiad, decommission, User-jijiki, Operations
ops-monitoring-bot added a comment to T209181: Decommission rdb1001, rdb1002, rdb1003, rdb1004, rdb1007, rdb1008.

wmf-decommission-host was executed by robh for rdb1002.eqiad.wmnet and performed the following actions:

  • Revoked Puppet certificate
  • Removed from PuppetDB
  • Downtimed host on Icinga
  • Downtimed mgmt interface on Icinga
  • Removed from DebMonitor
Fri, Mar 8, 6:04 PM · ops-eqiad, decommission, User-jijiki, Operations
ops-monitoring-bot added a comment to T209181: Decommission rdb1001, rdb1002, rdb1003, rdb1004, rdb1007, rdb1008.

wmf-decommission-host was executed by robh for rdb1001.eqiad.wmnet and performed the following actions:

  • Revoked Puppet certificate
  • Removed from PuppetDB
  • Downtimed host on Icinga
  • Downtimed mgmt interface on Icinga
  • Removed from DebMonitor
Fri, Mar 8, 6:03 PM · ops-eqiad, decommission, User-jijiki, Operations

Thu, Mar 7

ops-monitoring-bot added a comment to T217556: Decommission old eqiad logstash hardware hosts logstash100[456].

wmf-decommission-host was executed by robh for logstash1006.eqiad.wmnet and performed the following actions:

  • Revoked Puppet certificate
  • Removed from PuppetDB
  • Downtimed host on Icinga
  • Downtimed mgmt interface on Icinga
  • Removed from DebMonitor
Thu, Mar 7, 10:00 PM · decommission, DC-Ops, ops-eqiad, User-herron, Operations, Wikimedia-Logstash
ops-monitoring-bot added a comment to T217556: Decommission old eqiad logstash hardware hosts logstash100[456].

wmf-decommission-host was executed by robh for logstash1005.eqiad.wmnet and performed the following actions:

  • Revoked Puppet certificate
  • Removed from PuppetDB
  • Downtimed host on Icinga
  • Downtimed mgmt interface on Icinga
  • Removed from DebMonitor
Thu, Mar 7, 10:00 PM · decommission, DC-Ops, ops-eqiad, User-herron, Operations, Wikimedia-Logstash
ops-monitoring-bot added a comment to T217556: Decommission old eqiad logstash hardware hosts logstash100[456].

wmf-decommission-host was executed by robh for logstash1004.eqiad.wmnet and performed the following actions:

  • Revoked Puppet certificate
  • Removed from PuppetDB
  • Downtimed host on Icinga
  • Downtimed mgmt interface on Icinga
  • Removed from DebMonitor
Thu, Mar 7, 10:00 PM · decommission, DC-Ops, ops-eqiad, User-herron, Operations, Wikimedia-Logstash
ops-monitoring-bot added a comment to T209425: Reclaim rdb2001, rdb2002.

wmf-decommission-host was executed by robh for rdb2002.codfw.wmnet and performed the following actions:

  • Revoked Puppet certificate
  • Removed from PuppetDB
  • Downtimed host on Icinga
  • Downtimed mgmt interface on Icinga
  • Removed from DebMonitor
Thu, Mar 7, 7:32 PM · ops-codfw, Patch-For-Review, User-jijiki, decommission, Operations
ops-monitoring-bot added a comment to T209425: Reclaim rdb2001, rdb2002.

wmf-decommission-host was executed by robh for rdb2001.codfw.wmnet and performed the following actions:

  • Revoked Puppet certificate
  • Removed from PuppetDB
  • Downtimed host on Icinga
  • Downtimed mgmt interface on Icinga
  • Removed from DebMonitor
Thu, Mar 7, 7:31 PM · ops-codfw, Patch-For-Review, User-jijiki, decommission, Operations
ops-monitoring-bot added a comment to T208586: Decommission lvs1007-1012.

wmf-decommission-host was executed by robh for lvs1012.eqiad.wmnet and performed the following actions:

  • Revoked Puppet certificate
  • Removed from PuppetDB
  • Downtimed host on Icinga
  • Downtimed mgmt interface on Icinga
  • Removed from DebMonitor
Thu, Mar 7, 5:18 PM · ops-eqiad, decommission, Traffic, Operations
ops-monitoring-bot added a comment to T208586: Decommission lvs1007-1012.

wmf-decommission-host was executed by robh for lvs1011.eqiad.wmnet and performed the following actions:

  • Revoked Puppet certificate
  • Removed from PuppetDB
  • Downtimed host on Icinga
  • Downtimed mgmt interface on Icinga
  • Removed from DebMonitor
Thu, Mar 7, 5:17 PM · ops-eqiad, decommission, Traffic, Operations
ops-monitoring-bot added a comment to T208586: Decommission lvs1007-1012.

wmf-decommission-host was executed by robh for lvs1010.eqiad.wmnet and performed the following actions:

  • Revoked Puppet certificate
  • Removed from PuppetDB
  • Downtimed host on Icinga
  • Downtimed mgmt interface on Icinga
  • Removed from DebMonitor
Thu, Mar 7, 5:17 PM · ops-eqiad, decommission, Traffic, Operations
ops-monitoring-bot added a comment to T208586: Decommission lvs1007-1012.

wmf-decommission-host was executed by robh for lvs1009.eqiad.wmnet and performed the following actions:

  • Revoked Puppet certificate
  • Removed from PuppetDB
  • Skipped downtime host on Icinga (likely already removed)
  • Skipped downtime mgmt interface on Icinga (likely already removed)
  • Removed from DebMonitor
Thu, Mar 7, 5:15 PM · ops-eqiad, decommission, Traffic, Operations
ops-monitoring-bot added a comment to T208586: Decommission lvs1007-1012.

wmf-decommission-host was executed by robh for lvs1008.eqiad.wmnet and performed the following actions:

  • Revoked Puppet certificate
  • Removed from PuppetDB
  • Skipped downtime host on Icinga (likely already removed)
  • Skipped downtime mgmt interface on Icinga (likely already removed)
  • Removed from DebMonitor
Thu, Mar 7, 5:15 PM · ops-eqiad, decommission, Traffic, Operations
ops-monitoring-bot added a comment to T208586: Decommission lvs1007-1012.

wmf-decommission-host was executed by robh for lvs1007.eqiad.wmnet and performed the following actions:

  • Revoked Puppet certificate
  • Removed from PuppetDB
  • Skipped downtime host on Icinga (likely already removed)
  • Skipped downtime mgmt interface on Icinga (likely already removed)
  • Removed from DebMonitor
Thu, Mar 7, 5:15 PM · ops-eqiad, decommission, Traffic, Operations

Wed, Mar 6

ops-monitoring-bot added a comment to T205507: Decommission analytics100[1,2].

wmf-decommission-host was executed by robh for analytics1002.eqiad.wmnet and performed the following actions:

  • Revoked Puppet certificate
  • Removed from PuppetDB
  • Downtimed host on Icinga
  • Downtimed mgmt interface on Icinga
  • Removed from DebMonitor
Wed, Mar 6, 11:20 PM · Operations, ops-eqiad, decommission, User-Elukey, Analytics
ops-monitoring-bot added a comment to T205507: Decommission analytics100[1,2].

wmf-decommission-host was executed by robh for analytics1001.eqiad.wmnet and performed the following actions:

  • Revoked Puppet certificate
  • Removed from PuppetDB
  • Downtimed host on Icinga
  • Downtimed mgmt interface on Icinga
  • Removed from DebMonitor
Wed, Mar 6, 11:20 PM · Operations, ops-eqiad, decommission, User-Elukey, Analytics
ops-monitoring-bot added a comment to T211826: decommission oxygen.eqiad.wmnet.

wmf-decommission-host was executed by robh for oxygen.eqiad.wmnet and performed the following actions:

  • Revoked Puppet certificate
  • Removed from PuppetDB
  • Downtimed host on Icinga
  • Downtimed mgmt interface on Icinga
  • Removed from DebMonitor
Wed, Mar 6, 5:57 PM · Patch-For-Review, Operations, ops-eqiad, DC-Ops, decommission
ops-monitoring-bot added a comment to T216491: Decommission dbstore1002.

wmf-decommission-host was executed by robh for dbstore1002.eqiad.wmnet and performed the following actions:

  • Revoked Puppet certificate
  • Removed from PuppetDB
  • Downtimed host on Icinga
  • Downtimed mgmt interface on Icinga
  • Removed from DebMonitor
Wed, Mar 6, 5:40 PM · Patch-For-Review, decommission, ops-eqiad, Operations, Analytics
ops-monitoring-bot added projects to T217755: Degraded RAID on db2044: ops-codfw, Operations.
Wed, Mar 6, 11:52 AM · Operations, ops-codfw

Tue, Mar 5

ops-monitoring-bot added a comment to T208584: Decommission old eqiad caches.

wmf-decommission-host was executed by robh for cp1068.eqiad.wmnet and performed the following actions:

  • Revoked Puppet certificate
  • Removed from PuppetDB
  • Skipped downtime host on Icinga (likely already removed)
  • Skipped downtime mgmt interface on Icinga (likely already removed)
  • Removed from DebMonitor
Tue, Mar 5, 11:04 PM · Patch-For-Review, ops-eqiad, decommission, Operations, Traffic
ops-monitoring-bot added a comment to T208584: Decommission old eqiad caches.

wmf-decommission-host was executed by robh for cp1067.eqiad.wmnet and performed the following actions:

  • Revoked Puppet certificate
  • Removed from PuppetDB
  • Skipped downtime host on Icinga (likely already removed)
  • Skipped downtime mgmt interface on Icinga (likely already removed)
  • Removed from DebMonitor
Tue, Mar 5, 11:04 PM · Patch-For-Review, ops-eqiad, decommission, Operations, Traffic
ops-monitoring-bot added a comment to T208584: Decommission old eqiad caches.

wmf-decommission-host was executed by robh for cp1066.eqiad.wmnet and performed the following actions:

  • Revoked Puppet certificate
  • Removed from PuppetDB
  • Skipped downtime host on Icinga (likely already removed)
  • Skipped downtime mgmt interface on Icinga (likely already removed)
  • Removed from DebMonitor
Tue, Mar 5, 11:04 PM · Patch-For-Review, ops-eqiad, decommission, Operations, Traffic
ops-monitoring-bot added a comment to T208584: Decommission old eqiad caches.

wmf-decommission-host was executed by robh for cp1065.eqiad.wmnet and performed the following actions:

  • Revoked Puppet certificate
  • Removed from PuppetDB
  • Skipped downtime host on Icinga (likely already removed)
  • Skipped downtime mgmt interface on Icinga (likely already removed)
  • Removed from DebMonitor
Tue, Mar 5, 11:03 PM · Patch-For-Review, ops-eqiad, decommission, Operations, Traffic
ops-monitoring-bot added a comment to T208584: Decommission old eqiad caches.

wmf-decommission-host was executed by robh for cp1064.eqiad.wmnet and performed the following actions:

  • Revoked Puppet certificate
  • Removed from PuppetDB
  • Skipped downtime host on Icinga (likely already removed)
  • Skipped downtime mgmt interface on Icinga (likely already removed)
  • Removed from DebMonitor
Tue, Mar 5, 11:03 PM · Patch-For-Review, ops-eqiad, decommission, Operations, Traffic
ops-monitoring-bot added a comment to T208584: Decommission old eqiad caches.

wmf-decommission-host was executed by robh for cp1063.eqiad.wmnet and performed the following actions:

  • Revoked Puppet certificate
  • Removed from PuppetDB
  • Skipped downtime host on Icinga (likely already removed)
  • Skipped downtime mgmt interface on Icinga (likely already removed)
  • Removed from DebMonitor
Tue, Mar 5, 11:03 PM · Patch-For-Review, ops-eqiad, decommission, Operations, Traffic
ops-monitoring-bot added a comment to T208584: Decommission old eqiad caches.

wmf-decommission-host was executed by robh for cp1062.eqiad.wmnet and performed the following actions:

  • Revoked Puppet certificate
  • Removed from PuppetDB
  • Skipped downtime host on Icinga (likely already removed)
  • Skipped downtime mgmt interface on Icinga (likely already removed)
  • Removed from DebMonitor
Tue, Mar 5, 11:03 PM · Patch-For-Review, ops-eqiad, decommission, Operations, Traffic
ops-monitoring-bot added a comment to T208584: Decommission old eqiad caches.

wmf-decommission-host was executed by robh for cp1061.eqiad.wmnet and performed the following actions:

  • Revoked Puppet certificate
  • Removed from PuppetDB
  • Skipped downtime host on Icinga (likely already removed)
  • Skipped downtime mgmt interface on Icinga (likely already removed)
  • Removed from DebMonitor
Tue, Mar 5, 11:03 PM · Patch-For-Review, ops-eqiad, decommission, Operations, Traffic
ops-monitoring-bot added a comment to T208584: Decommission old eqiad caches.

wmf-decommission-host was executed by robh for cp1060.eqiad.wmnet and performed the following actions:

  • Revoked Puppet certificate
  • Removed from PuppetDB
  • Skipped downtime host on Icinga (likely already removed)
  • Skipped downtime mgmt interface on Icinga (likely already removed)
  • Removed from DebMonitor
Tue, Mar 5, 11:02 PM · Patch-For-Review, ops-eqiad, decommission, Operations, Traffic
ops-monitoring-bot added a comment to T208584: Decommission old eqiad caches.

wmf-decommission-host was executed by robh for cp1059.eqiad.wmnet and performed the following actions:

  • Revoked Puppet certificate
  • Removed from PuppetDB
  • Skipped downtime host on Icinga (likely already removed)
  • Skipped downtime mgmt interface on Icinga (likely already removed)
  • Removed from DebMonitor
Tue, Mar 5, 11:02 PM · Patch-For-Review, ops-eqiad, decommission, Operations, Traffic
ops-monitoring-bot added a comment to T208584: Decommission old eqiad caches.

wmf-decommission-host was executed by robh for cp1058.eqiad.wmnet and performed the following actions:

  • Revoked Puppet certificate
  • Removed from PuppetDB
  • Skipped downtime host on Icinga (likely already removed)
  • Skipped downtime mgmt interface on Icinga (likely already removed)
  • Removed from DebMonitor
Tue, Mar 5, 11:02 PM · Patch-For-Review, ops-eqiad, decommission, Operations, Traffic
ops-monitoring-bot added a comment to T209527: Set up scratch and maps NFS services on cloudstore1008/9.

Completed auto-reimage of hosts:

['cloudstore1009.wikimedia.org']
Tue, Mar 5, 3:26 PM · Patch-For-Review, cloud-services-team (Kanban)
ops-monitoring-bot added a comment to T209527: Set up scratch and maps NFS services on cloudstore1008/9.

Script wmf-auto-reimage was launched by gtirloni on cumin1001.eqiad.wmnet for hosts:

cloudstore1009.wikimedia.org

The log can be found in /var/log/wmf-auto-reimage/201903051456_gtirloni_230796_cloudstore1009_wikimedia_org.log.

Tue, Mar 5, 2:57 PM · Patch-For-Review, cloud-services-team (Kanban)
ops-monitoring-bot added a comment to T161296: Upgrade mysqld_exporter in production.

Completed auto-reimage of hosts:

['db1114.eqiad.wmnet']
Tue, Mar 5, 2:10 PM · Patch-For-Review, DBA, User-fgiunchedi, Operations, Prometheus-metrics-monitoring
ops-monitoring-bot added a comment to T161296: Upgrade mysqld_exporter in production.

Script wmf-auto-reimage was launched by jynus on cumin1001.eqiad.wmnet for hosts:

['db1114.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201903051304_jynus_206262.log.

Tue, Mar 5, 1:04 PM · Patch-For-Review, DBA, User-fgiunchedi, Operations, Prometheus-metrics-monitoring

Mon, Mar 4

ops-monitoring-bot added a comment to T209527: Set up scratch and maps NFS services on cloudstore1008/9.

Completed auto-reimage of hosts:

['cloudstore1009.wikimedia.org']
Mon, Mar 4, 8:19 AM · Patch-For-Review, cloud-services-team (Kanban)
ops-monitoring-bot added a comment to T209527: Set up scratch and maps NFS services on cloudstore1008/9.

Script wmf-auto-reimage was launched by gtirloni on cumin1001.eqiad.wmnet for hosts:

cloudstore1009.wikimedia.org

The log can be found in /var/log/wmf-auto-reimage/201903040800_gtirloni_8934_cloudstore1009_wikimedia_org.log.

Mon, Mar 4, 8:00 AM · Patch-For-Review, cloud-services-team (Kanban)
ops-monitoring-bot added a comment to T209527: Set up scratch and maps NFS services on cloudstore1008/9.

Completed auto-reimage of hosts:

['cloudstore1008.wikimedia.org']
Mon, Mar 4, 7:59 AM · Patch-For-Review, cloud-services-team (Kanban)
ops-monitoring-bot added a comment to T209527: Set up scratch and maps NFS services on cloudstore1008/9.

Script wmf-auto-reimage was launched by gtirloni on cumin1001.eqiad.wmnet for hosts:

cloudstore1008.wikimedia.org

The log can be found in /var/log/wmf-auto-reimage/201903040740_gtirloni_4949_cloudstore1008_wikimedia_org.log.

Mon, Mar 4, 7:40 AM · Patch-For-Review, cloud-services-team (Kanban)
ops-monitoring-bot added a comment to T209527: Set up scratch and maps NFS services on cloudstore1008/9.

Completed auto-reimage of hosts:

['cloudstore1008.wikimedia.org']
Mon, Mar 4, 7:39 AM · Patch-For-Review, cloud-services-team (Kanban)
ops-monitoring-bot added a comment to T209527: Set up scratch and maps NFS services on cloudstore1008/9.

Script wmf-auto-reimage was launched by gtirloni on cumin1001.eqiad.wmnet for hosts:

cloudstore1008.wikimedia.org

The log can be found in /var/log/wmf-auto-reimage/201903040729_gtirloni_2166_cloudstore1008_wikimedia_org.log.

Mon, Mar 4, 7:29 AM · Patch-For-Review, cloud-services-team (Kanban)
ops-monitoring-bot added a comment to T209527: Set up scratch and maps NFS services on cloudstore1008/9.

Completed auto-reimage of hosts:

['cloudstore1008.wikimedia.org']
Mon, Mar 4, 7:29 AM · Patch-For-Review, cloud-services-team (Kanban)
ops-monitoring-bot added a comment to T209527: Set up scratch and maps NFS services on cloudstore1008/9.

Script wmf-auto-reimage was launched by gtirloni on cumin1001.eqiad.wmnet for hosts:

cloudstore1008.wikimedia.org

The log can be found in /var/log/wmf-auto-reimage/201903040728_gtirloni_1962_cloudstore1008_wikimedia_org.log.

Mon, Mar 4, 7:29 AM · Patch-For-Review, cloud-services-team (Kanban)
ops-monitoring-bot added a comment to T209527: Set up scratch and maps NFS services on cloudstore1008/9.

Completed auto-reimage of hosts:

['cloudstore1008.wikimedia.org']
Mon, Mar 4, 7:28 AM · Patch-For-Review, cloud-services-team (Kanban)
ops-monitoring-bot added a comment to T209527: Set up scratch and maps NFS services on cloudstore1008/9.

Script wmf-auto-reimage was launched by gtirloni on cumin1001.eqiad.wmnet for hosts:

cloudstore1008.wikimedia.org

The log can be found in /var/log/wmf-auto-reimage/201903040650_gtirloni_255354_cloudstore1008_wikimedia_org.log.

Mon, Mar 4, 6:51 AM · Patch-For-Review, cloud-services-team (Kanban)

Fri, Mar 1

ops-monitoring-bot added a comment to T215231: rack/setup/install labsdb1012.eqiad.wmnet.

Completed auto-reimage of hosts:

['labsdb1012.eqiad.wmnet']
Fri, Mar 1, 8:31 AM · DBA, Patch-For-Review, ops-eqiad, Analytics, User-Elukey, Operations
ops-monitoring-bot added a comment to T215231: rack/setup/install labsdb1012.eqiad.wmnet.

Script wmf-auto-reimage was launched by elukey on cumin1001.eqiad.wmnet for hosts:

['labsdb1012.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201903010815_elukey_158975.log.

Fri, Mar 1, 8:15 AM · DBA, Patch-For-Review, ops-eqiad, Analytics, User-Elukey, Operations

Thu, Feb 28

ops-monitoring-bot added a comment to T208584: Decommission old eqiad caches.

wmf-decommission-host was executed by robh for cp1055.eqiad.wmnet and performed the following actions:

  • Revoked Puppet certificate
  • Removed from PuppetDB
  • Downtimed host on Icinga
  • Downtimed mgmt interface on Icinga
  • Removed from DebMonitor
Thu, Feb 28, 11:16 PM · Patch-For-Review, ops-eqiad, decommission, Operations, Traffic
ops-monitoring-bot added a comment to T208584: Decommission old eqiad caches.

wmf-decommission-host was executed by robh for cp1054.eqiad.wmnet and performed the following actions:

  • Revoked Puppet certificate
  • Removed from PuppetDB
  • Downtimed host on Icinga
  • Downtimed mgmt interface on Icinga
  • Removed from DebMonitor
Thu, Feb 28, 11:15 PM · Patch-For-Review, ops-eqiad, decommission, Operations, Traffic
ops-monitoring-bot added a comment to T208584: Decommission old eqiad caches.

wmf-decommission-host was executed by robh for cp1053.eqiad.wmnet and performed the following actions:

  • Revoked Puppet certificate
  • Removed from PuppetDB
  • Downtimed host on Icinga
  • Downtimed mgmt interface on Icinga
  • Removed from DebMonitor
Thu, Feb 28, 11:15 PM · Patch-For-Review, ops-eqiad, decommission, Operations, Traffic
ops-monitoring-bot added a comment to T208584: Decommission old eqiad caches.

wmf-decommission-host was executed by robh for cp1052.eqiad.wmnet and performed the following actions:

  • Revoked Puppet certificate
  • Removed from PuppetDB
  • Downtimed host on Icinga
  • Downtimed mgmt interface on Icinga
  • Removed from DebMonitor
Thu, Feb 28, 11:15 PM · Patch-For-Review, ops-eqiad, decommission, Operations, Traffic
ops-monitoring-bot added a comment to T208584: Decommission old eqiad caches.

wmf-decommission-host was executed by robh for cp1051.eqiad.wmnet and performed the following actions:

  • Revoked Puppet certificate
  • Removed from PuppetDB
  • Downtimed host on Icinga
  • Downtimed mgmt interface on Icinga
  • Removed from DebMonitor
Thu, Feb 28, 11:15 PM · Patch-For-Review, ops-eqiad, decommission, Operations, Traffic
ops-monitoring-bot added a comment to T208584: Decommission old eqiad caches.

wmf-decommission-host was executed by robh for cp1050.eqiad.wmnet and performed the following actions:

  • Revoked Puppet certificate
  • Removed from PuppetDB
  • Downtimed host on Icinga
  • Downtimed mgmt interface on Icinga
  • Removed from DebMonitor
Thu, Feb 28, 11:15 PM · Patch-For-Review, ops-eqiad, decommission, Operations, Traffic
ops-monitoring-bot added a comment to T208584: Decommission old eqiad caches.

wmf-decommission-host was executed by robh for cp1049.eqiad.wmnet and performed the following actions:

  • Revoked Puppet certificate
  • Removed from PuppetDB
  • Downtimed host on Icinga
  • Downtimed mgmt interface on Icinga
  • Removed from DebMonitor
Thu, Feb 28, 11:14 PM · Patch-For-Review, ops-eqiad, decommission, Operations, Traffic
ops-monitoring-bot added a comment to T208584: Decommission old eqiad caches.

wmf-decommission-host was executed by robh for cp1048.eqiad.wmnet and performed the following actions:

  • Revoked Puppet certificate
  • Removed from PuppetDB
  • Downtimed host on Icinga
  • Downtimed mgmt interface on Icinga
  • Removed from DebMonitor
Thu, Feb 28, 11:14 PM · Patch-For-Review, ops-eqiad, decommission, Operations, Traffic
ops-monitoring-bot added a comment to T208584: Decommission old eqiad caches.

wmf-decommission-host was executed by robh for cp1047.eqiad.wmnet and performed the following actions:

  • Revoked Puppet certificate
  • Removed from PuppetDB
  • Downtimed host on Icinga
  • Downtimed mgmt interface on Icinga
  • Removed from DebMonitor
Thu, Feb 28, 11:14 PM · Patch-For-Review, ops-eqiad, decommission, Operations, Traffic
ops-monitoring-bot added a comment to T208584: Decommission old eqiad caches.

wmf-decommission-host was executed by robh for cp1046.eqiad.wmnet and performed the following actions:

  • Revoked Puppet certificate
  • Removed from PuppetDB
  • Downtimed host on Icinga
  • Downtimed mgmt interface on Icinga
  • Removed from DebMonitor
Thu, Feb 28, 11:14 PM · Patch-For-Review, ops-eqiad, decommission, Operations, Traffic
ops-monitoring-bot added a comment to T208584: Decommission old eqiad caches.

wmf-decommission-host was executed by robh for cp1045.eqiad.wmnet and performed the following actions:

  • Revoked Puppet certificate
  • Removed from PuppetDB
  • Downtimed host on Icinga
  • Downtimed mgmt interface on Icinga
  • Removed from DebMonitor
Thu, Feb 28, 11:13 PM · Patch-For-Review, ops-eqiad, decommission, Operations, Traffic
ops-monitoring-bot added a comment to T217140: cloudvirt1015: update raid config and move to 10Gb.

Completed auto-reimage of hosts:

['cloudvirt1015.eqiad.wmnet']
Thu, Feb 28, 6:19 PM · ops-eqiad, DC-Ops, Operations
ops-monitoring-bot added projects to T217356: Degraded RAID on sodium: Operations, ops-eqiad.
Thu, Feb 28, 6:00 PM · ops-eqiad, Operations
ops-monitoring-bot added a comment to T217140: cloudvirt1015: update raid config and move to 10Gb.

Script wmf-auto-reimage was launched by gtirloni on cumin1001.eqiad.wmnet for hosts:

cloudvirt1015.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/201902281728_gtirloni_234720_cloudvirt1015_eqiad_wmnet.log.

Thu, Feb 28, 5:28 PM · ops-eqiad, DC-Ops, Operations

Wed, Feb 27

ops-monitoring-bot added projects to T217301: Degraded RAID on db2033: ops-codfw, Operations.
Wed, Feb 27, 11:49 PM · DBA, Operations, ops-codfw

Tue, Feb 26

ops-monitoring-bot added a comment to T214597: Thumbor upgrade to stretch plan.

Completed auto-reimage of hosts:

['thumbor2004.codfw.wmnet']
Tue, Feb 26, 9:20 AM · Patch-For-Review, serviceops, Operations, Thumbor, User-jijiki
ops-monitoring-bot added a comment to T214597: Thumbor upgrade to stretch plan.

Completed auto-reimage of hosts:

['thumbor1003.eqiad.wmnet']
Tue, Feb 26, 8:44 AM · Patch-For-Review, serviceops, Operations, Thumbor, User-jijiki
ops-monitoring-bot added projects to T217121: Degraded RAID on thumbor1003: Operations, ops-eqiad.
Tue, Feb 26, 8:36 AM · ops-eqiad, Operations
ops-monitoring-bot added a comment to T214597: Thumbor upgrade to stretch plan.

Script wmf-auto-reimage was launched by jiji on cumin2001.codfw.wmnet for hosts:

thumbor2004.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/201902260827_jiji_15458_thumbor2004_codfw_wmnet.log.

Tue, Feb 26, 8:27 AM · Patch-For-Review, serviceops, Operations, Thumbor, User-jijiki
ops-monitoring-bot added a comment to T214597: Thumbor upgrade to stretch plan.

Completed auto-reimage of hosts:

['thumbor2003.codfw.wmnet']
Tue, Feb 26, 8:03 AM · Patch-For-Review, serviceops, Operations, Thumbor, User-jijiki
ops-monitoring-bot added projects to T217118: Degraded RAID on thumbor2003: ops-codfw, Operations.
Tue, Feb 26, 7:57 AM · Operations, ops-codfw
ops-monitoring-bot added a comment to T214597: Thumbor upgrade to stretch plan.

Script wmf-auto-reimage was launched by jiji on cumin2001.codfw.wmnet for hosts:

thumbor2003.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/201902260708_jiji_25581_thumbor2003_codfw_wmnet.log.

Tue, Feb 26, 7:08 AM · Patch-For-Review, serviceops, Operations, Thumbor, User-jijiki
ops-monitoring-bot added a comment to T214597: Thumbor upgrade to stretch plan.

Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts:

thumbor1003.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/201902260705_jiji_200381_thumbor1003_eqiad_wmnet.log.

Tue, Feb 26, 7:05 AM · Patch-For-Review, serviceops, Operations, Thumbor, User-jijiki

Mon, Feb 25

ops-monitoring-bot added a comment to T214597: Thumbor upgrade to stretch plan.

Completed auto-reimage of hosts:

['thumbor1002.eqiad.wmnet']
Mon, Feb 25, 7:46 PM · Patch-For-Review, serviceops, Operations, Thumbor, User-jijiki
ops-monitoring-bot added a comment to T214597: Thumbor upgrade to stretch plan.

Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts:

thumbor1002.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/201902251825_jiji_93856_thumbor1002_eqiad_wmnet.log.

Mon, Feb 25, 6:25 PM · Patch-For-Review, serviceops, Operations, Thumbor, User-jijiki
ops-monitoring-bot added a comment to T214597: Thumbor upgrade to stretch plan.

Completed auto-reimage of hosts:

['thumbor2001.codfw.wmnet']
Mon, Feb 25, 6:10 PM · Patch-For-Review, serviceops, Operations, Thumbor, User-jijiki
ops-monitoring-bot added a comment to T207389: Rename the Certcentral project to Acme-chief.

cookbooks.sre.hosts.decommission executed by vgutierrez@cumin1001 for hosts: certcentral2001.codfw.wmnet,certcentral1001.eqiad.wmnet

  • certcentral2001.codfw.wmnet
    • Removed from Puppet master and PuppetDB
    • Skipped downtime host on Icinga (likely already removed)
    • No management interface found (likely a VM)
    • Removed from DebMonitor
  • certcentral1001.eqiad.wmnet
    • Removed from Puppet master and PuppetDB
    • Skipped downtime host on Icinga (likely already removed)
    • No management interface found (likely a VM)
    • Removed from DebMonitor
Mon, Feb 25, 5:58 PM · Patch-For-Review, Acme-chief
ops-monitoring-bot added a comment to T214597: Thumbor upgrade to stretch plan.

Script wmf-auto-reimage was launched by jiji on cumin2001.codfw.wmnet for hosts:

['thumbor2001.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201902251627_jiji_595.log.

Mon, Feb 25, 4:28 PM · Patch-For-Review, serviceops, Operations, Thumbor, User-jijiki
ops-monitoring-bot added a comment to T214597: Thumbor upgrade to stretch plan.

Completed auto-reimage of hosts:

['thumbor1001.eqiad.wmnet']
Mon, Feb 25, 2:06 PM · Patch-For-Review, serviceops, Operations, Thumbor, User-jijiki
ops-monitoring-bot added a comment to T214597: Thumbor upgrade to stretch plan.

Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts:

thumbor1001.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/201902251118_jiji_23066_thumbor1001_eqiad_wmnet.log.

Mon, Feb 25, 11:18 AM · Patch-For-Review, serviceops, Operations, Thumbor, User-jijiki

Feb 21 2019

ops-monitoring-bot added projects to T216670: Degraded RAID on db2050: ops-codfw, Operations.
Feb 21 2019, 3:18 AM · DBA, Operations, ops-codfw

Feb 20 2019

ops-monitoring-bot added projects to T216649: Degraded RAID on cloudvirt1020: Operations, ops-eqiad.
Feb 20 2019, 7:33 PM · ops-eqiad, Operations

Feb 19 2019

ops-monitoring-bot added a comment to T214597: Thumbor upgrade to stretch plan.

Completed auto-reimage of hosts:

['thumbor2002.codfw.wmnet']
Feb 19 2019, 7:30 PM · Patch-For-Review, serviceops, Operations, Thumbor, User-jijiki
ops-monitoring-bot added projects to T216526: Degraded RAID on cloudvirt1018: Operations, ops-eqiad.
Feb 19 2019, 5:24 PM · cloud-services-team, ops-eqiad, Operations
ops-monitoring-bot added a comment to T214597: Thumbor upgrade to stretch plan.

Script wmf-auto-reimage was launched by jiji on cumin2001.codfw.wmnet for hosts:

['thumbor2002.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201902191553_jiji_10924.log.

Feb 19 2019, 3:53 PM · Patch-For-Review, serviceops, Operations, Thumbor, User-jijiki

Feb 18 2019

ops-monitoring-bot added a comment to T214597: Thumbor upgrade to stretch plan.

Completed auto-reimage of hosts:

['thumbor1004.eqiad.wmnet']
Feb 18 2019, 9:24 PM · Patch-For-Review, serviceops, Operations, Thumbor, User-jijiki
ops-monitoring-bot added a comment to T214597: Thumbor upgrade to stretch plan.

Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts:

['thumbor1004.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201902181750_jiji_246664.log.

Feb 18 2019, 5:51 PM · Patch-For-Review, serviceops, Operations, Thumbor, User-jijiki
ops-monitoring-bot added a comment to T214597: Thumbor upgrade to stretch plan.

Completed auto-reimage of hosts:

['thumbor2002.codfw.wmnet']
Feb 18 2019, 1:15 PM · Patch-For-Review, serviceops, Operations, Thumbor, User-jijiki
ops-monitoring-bot added a comment to T214597: Thumbor upgrade to stretch plan.

Script wmf-auto-reimage was launched by jiji on cumin2001.codfw.wmnet for hosts:

thumbor2002.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/201902181215_jiji_16296_thumbor2002_codfw_wmnet.log.

Feb 18 2019, 12:15 PM · Patch-For-Review, serviceops, Operations, Thumbor, User-jijiki
ops-monitoring-bot added a comment to T214597: Thumbor upgrade to stretch plan.

Completed auto-reimage of hosts:

['thumbor2002.codfw.wmnet']
Feb 18 2019, 12:15 PM · Patch-For-Review, serviceops, Operations, Thumbor, User-jijiki
ops-monitoring-bot added a comment to T214597: Thumbor upgrade to stretch plan.

Script wmf-auto-reimage was launched by jiji on cumin2001.codfw.wmnet for hosts:

thumbor2002.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/201902181144_jiji_8607_thumbor2002_codfw_wmnet.log.

Feb 18 2019, 11:44 AM · Patch-For-Review, serviceops, Operations, Thumbor, User-jijiki
ops-monitoring-bot added a comment to T214597: Thumbor upgrade to stretch plan.

Completed auto-reimage of hosts:

['thumbor2002.codfw.wmnet']
Feb 18 2019, 11:41 AM · Patch-For-Review, serviceops, Operations, Thumbor, User-jijiki
ops-monitoring-bot added a comment to T214597: Thumbor upgrade to stretch plan.

Script wmf-auto-reimage was launched by jiji on cumin2001.codfw.wmnet for hosts:

thumbor2002.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/201902181141_jiji_6965_thumbor2002_codfw_wmnet.log.

Feb 18 2019, 11:41 AM · Patch-For-Review, serviceops, Operations, Thumbor, User-jijiki
ops-monitoring-bot added a comment to T214597: Thumbor upgrade to stretch plan.

Completed auto-reimage of hosts:

['thumbor2002.codfw.wmnet']
Feb 18 2019, 11:39 AM · Patch-For-Review, serviceops, Operations, Thumbor, User-jijiki
ops-monitoring-bot added a comment to T214597: Thumbor upgrade to stretch plan.

Script wmf-auto-reimage was launched by jiji on cumin2001.codfw.wmnet for hosts:

thumbor2002.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/201902181139_jiji_6639_thumbor2002_codfw_wmnet.log.

Feb 18 2019, 11:39 AM · Patch-For-Review, serviceops, Operations, Thumbor, User-jijiki
ops-monitoring-bot added a comment to T214597: Thumbor upgrade to stretch plan.

Completed auto-reimage of hosts:

['thumbor2002.codfw.wmnet']
Feb 18 2019, 11:38 AM · Patch-For-Review, serviceops, Operations, Thumbor, User-jijiki
ops-monitoring-bot added a comment to T214597: Thumbor upgrade to stretch plan.

Script wmf-auto-reimage was launched by jiji on cumin2001.codfw.wmnet for hosts:

thumbor2002.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/201902181130_jiji_3443_thumbor2002_codfw_wmnet.log.

Feb 18 2019, 11:30 AM · Patch-For-Review, serviceops, Operations, Thumbor, User-jijiki

Feb 15 2019

ops-monitoring-bot added a comment to T193264: Replace labsdb100[4567] with instances on cloudvirt1019 and cloudvirt1020.

Completed auto-reimage of hosts:

['cloudvirt1019.eqiad.wmnet']
Feb 15 2019, 8:11 PM · Scoring-platform-team, Wikilabels, cloud-services-team (Kanban), Patch-For-Review, Epic, Cloud-VPS