Page MenuHomePhabricator
People ops-monitoring-bot

ops-monitoring-bot (Operations Monitoring Bot)
UserBot

Projects

Calendar

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Monday

  • Clear sailing ahead.

User Details

User Since
Aug 12 2016, 1:45 PM (162 w, 1 d)
Roles
Bot
Availability
Available
LDAP User
Unknown
MediaWiki User
Unknown

Bot managed by Operations for automated interaction with Phabricator from monitoring tools.

Recent Activity
View All

Yesterday

ops-monitoring-bot added a comment to T220504: Decommission sarin.

cookbooks.sre.hosts.decommission executed by jmm@cumin1001 for hosts: sarin.codfw.wmnet

  • sarin.codfw.wmnet (PASS)
    • Downtimed host on Icinga
    • Downtimed management interface on Icinga
    • Wiped bootloaders
    • Powered off
    • Set Netbox status to Decommissioning
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
Fri, Sep 20, 7:47 AM · Patch-For-Review, Operations, decommission, ops-codfw
ops-monitoring-bot added a comment to T220503: Decommission neodymium.

cookbooks.sre.hosts.decommission executed by jmm@cumin1001 for hosts: neodymium.eqiad.wmnet

  • neodymium.eqiad.wmnet (PASS)
    • Downtimed host on Icinga
    • Downtimed management interface on Icinga
    • Wiped bootloaders
    • Powered off
    • Set Netbox status to Decommissioning
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
Fri, Sep 20, 7:46 AM · Patch-For-Review, decommission, Operations, ops-eqiad
ops-monitoring-bot added a comment to T230311: Decommission db2043.codfw.wmnet.

cookbooks.sre.hosts.decommission executed by marostegui@cumin1001 for hosts: db2043.codfw.wmnet

  • db2043.codfw.wmnet (PASS)
    • Downtimed host on Icinga
    • Downtimed management interface on Icinga
    • Wiped bootloaders
    • Powered off
    • Set Netbox status to Decommissioning
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
Fri, Sep 20, 7:29 AM · Patch-For-Review, ops-codfw, decommission, Operations

Thu, Sep 19

ops-monitoring-bot added a comment to T231046: setup/install gerrit1001.

Completed auto-reimage of hosts:

['gerrit1001.wikimedia.org']
Thu, Sep 19, 11:26 PM · Gerrit, Operations
ops-monitoring-bot added a comment to T231046: setup/install gerrit1001.

Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts:

gerrit1001.wikimedia.org

The log can be found in /var/log/wmf-auto-reimage/201909192308_dzahn_39474_gerrit1001_wikimedia_org.log.

Thu, Sep 19, 11:08 PM · Gerrit, Operations
ops-monitoring-bot added a comment to T220505: Decommission iron.

cookbooks.sre.hosts.decommission executed by jmm@cumin1001 for hosts: iron.wikimedia.org

  • iron.wikimedia.org (PASS)
    • Downtimed host on Icinga
    • Downtimed management interface on Icinga
    • Wiped bootloaders
    • Powered off
    • Set Netbox status to Decommissioning
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
Thu, Sep 19, 3:06 PM · Cloud-VPS, ops-eqiad, Operations, decommission
ops-monitoring-bot added a comment to T229557: decommission lithium.

cookbooks.sre.hosts.decommission executed by jmm@cumin1001 for hosts: lithium.eqiad.wmnet

  • lithium.eqiad.wmnet (PASS)
    • Downtimed host on Icinga
    • Downtimed management interface on Icinga
    • Wiped bootloaders
    • Powered off
    • Set Netbox status to Decommissioning
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
Thu, Sep 19, 2:19 PM · Operations, ops-eqiad, DC-Ops, decommission

Wed, Sep 18

ops-monitoring-bot added a comment to T231546: decom krypton.eqiad.wmnet.

cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: krypton.eqiad.wmnet

  • krypton.eqiad.wmnet
    • Removed from Puppet master and PuppetDB
    • Downtimed host on Icinga
    • No management interface found (likely a VM)
    • Removed from DebMonitor
Wed, Sep 18, 7:56 PM · decommission, serviceops, Operations
ops-monitoring-bot added a comment to T215332: reimage WMF6937/mw1298.

Completed auto-reimage of hosts:

['mw1298.eqiad.wmnet']
Wed, Sep 18, 7:25 PM · Patch-For-Review, decommission, Operations, hardware-requests
ops-monitoring-bot added a comment to T215332: reimage WMF6937/mw1298.

Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts:

mw1298.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/201909181848_dzahn_252267_mw1298_eqiad_wmnet.log.

Wed, Sep 18, 6:48 PM · Patch-For-Review, decommission, Operations, hardware-requests
ops-monitoring-bot added a comment to T228606: Degraded RAID on elastic1046.

Completed auto-reimage of hosts:

['elastic1046.eqiad.wmnet']
Wed, Sep 18, 10:43 AM · Discovery-Search (Current work), ops-eqiad, Operations
ops-monitoring-bot added a comment to T228606: Degraded RAID on elastic1046.

Script wmf-auto-reimage was launched by gehel on cumin1001.eqiad.wmnet for hosts:

['elastic1046.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201909180937_gehel_148677.log.

Wed, Sep 18, 9:37 AM · Discovery-Search (Current work), ops-eqiad, Operations
ops-monitoring-bot added a comment to T229209: Strengthen backup infrastructure and support.

Completed auto-reimage of hosts:

['backup1001.eqiad.wmnet']
Wed, Sep 18, 9:04 AM · Patch-For-Review, Goal, DBA, serviceops, Operations
ops-monitoring-bot added a comment to T229209: Strengthen backup infrastructure and support.

Script wmf-auto-reimage was launched by jynus on cumin1001.eqiad.wmnet for hosts:

['backup1001.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201909180842_jynus_136869.log.

Wed, Sep 18, 8:42 AM · Patch-For-Review, Goal, DBA, serviceops, Operations

Mon, Sep 16

ops-monitoring-bot added a comment to T229209: Strengthen backup infrastructure and support.

Script wmf-auto-reimage was launched by jynus on cumin2001.codfw.wmnet for hosts:

['backup2001.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201909161514_jynus_29861.log.

Mon, Sep 16, 3:14 PM · Patch-For-Review, Goal, DBA, serviceops, Operations
ops-monitoring-bot added a comment to T229209: Strengthen backup infrastructure and support.

Script wmf-auto-reimage was launched by jynus on cumin2001.codfw.wmnet for hosts:

['backup2001.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201909161507_jynus_27549.log.

Mon, Sep 16, 3:07 PM · Patch-For-Review, Goal, DBA, serviceops, Operations
ops-monitoring-bot added a comment to T202367: Productionize dbproxy101[2-7].eqiad.wmnet and dbproxy200[1-4].

Completed auto-reimage of hosts:

['dbproxy1021.eqiad.wmnet']
Mon, Sep 16, 8:16 AM · DBA
ops-monitoring-bot added a comment to T202367: Productionize dbproxy101[2-7].eqiad.wmnet and dbproxy200[1-4].

Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:

['dbproxy1021.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201909160758_marostegui_74242.log.

Mon, Sep 16, 7:58 AM · DBA
ops-monitoring-bot added a comment to T225128: Move cloudvirtan* hardware out of CloudVPS back into production Analytics VLAN..

Completed auto-reimage of hosts:

['an-presto1001.eqiad.wmnet']
Mon, Sep 16, 6:24 AM · Analytics-Kanban, ops-eqiad, Operations, netops, Analytics
ops-monitoring-bot added a comment to T225128: Move cloudvirtan* hardware out of CloudVPS back into production Analytics VLAN..

Script wmf-auto-reimage was launched by elukey on cumin1001.eqiad.wmnet for hosts:

cloudvirtan1001.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/201909160615_elukey_55100_cloudvirtan1001_eqiad_wmnet.log.

Mon, Sep 16, 6:15 AM · Analytics-Kanban, ops-eqiad, Operations, netops, Analytics
ops-monitoring-bot added a comment to T225128: Move cloudvirtan* hardware out of CloudVPS back into production Analytics VLAN..

Completed auto-reimage of hosts:

['cloudvirtan1001.eqiad.wmnet']
Mon, Sep 16, 6:15 AM · Analytics-Kanban, ops-eqiad, Operations, netops, Analytics
ops-monitoring-bot added a comment to T225128: Move cloudvirtan* hardware out of CloudVPS back into production Analytics VLAN..

Script wmf-auto-reimage was launched by elukey on cumin1001.eqiad.wmnet for hosts:

cloudvirtan1001.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/201909160615_elukey_55008_cloudvirtan1001_eqiad_wmnet.log.

Mon, Sep 16, 6:15 AM · Analytics-Kanban, ops-eqiad, Operations, netops, Analytics
ops-monitoring-bot added a comment to T225128: Move cloudvirtan* hardware out of CloudVPS back into production Analytics VLAN..

Completed auto-reimage of hosts:

['cloudvirtan1001.eqiad.wmnet']
Mon, Sep 16, 6:13 AM · Analytics-Kanban, ops-eqiad, Operations, netops, Analytics
ops-monitoring-bot added a comment to T225128: Move cloudvirtan* hardware out of CloudVPS back into production Analytics VLAN..

Script wmf-auto-reimage was launched by elukey on cumin1001.eqiad.wmnet for hosts:

cloudvirtan1001.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/201909160613_elukey_54732_cloudvirtan1001_eqiad_wmnet.log.

Mon, Sep 16, 6:13 AM · Analytics-Kanban, ops-eqiad, Operations, netops, Analytics

Fri, Sep 13

ops-monitoring-bot added a comment to T190568: Reimage both phab1001 and phab2001 to stretch / buster.

Completed auto-reimage of hosts:

['phab1001.eqiad.wmnet']
Fri, Sep 13, 11:59 PM · Patch-For-Review, Release-Engineering-Team (Development services), Release-Engineering-Team-TODO, serviceops, Phabricator, Operations
ops-monitoring-bot added a comment to T190568: Reimage both phab1001 and phab2001 to stretch / buster.

Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts:

phab1001.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/201909132312_dzahn_241683_phab1001_eqiad_wmnet.log.

Fri, Sep 13, 11:12 PM · Patch-For-Review, Release-Engineering-Team (Development services), Release-Engineering-Team-TODO, serviceops, Phabricator, Operations
ops-monitoring-bot added a comment to T228606: Degraded RAID on elastic1046.

Completed auto-reimage of hosts:

['elastic1046.eqiad.wmnet']
Fri, Sep 13, 8:47 AM · Discovery-Search (Current work), ops-eqiad, Operations
ops-monitoring-bot added a comment to T228606: Degraded RAID on elastic1046.

Script wmf-auto-reimage was launched by gehel on cumin1001.eqiad.wmnet for hosts:

['elastic1046.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201909130742_gehel_73868.log.

Fri, Sep 13, 7:42 AM · Discovery-Search (Current work), ops-eqiad, Operations

Wed, Sep 11

ops-monitoring-bot added projects to T232592: Degraded RAID on db1074: Operations, ops-eqiad.
Wed, Sep 11, 11:00 AM · ops-eqiad, Operations
ops-monitoring-bot added a comment to T224559: Migrate Failoid hosts to Stretch/Buster.

cookbooks.sre.hosts.decommission executed by jmm@cumin2001 for hosts: roentgenium.eqiad.wmnet

  • roentgenium.eqiad.wmnet
    • Removed from Puppet master and PuppetDB
    • Downtimed host on Icinga
    • No management interface found (likely a VM)
    • Removed from DebMonitor
Wed, Sep 11, 10:18 AM · Traffic, serviceops, Operations
ops-monitoring-bot added a comment to T224559: Migrate Failoid hosts to Stretch/Buster.

cookbooks.sre.hosts.decommission executed by jmm@cumin2001 for hosts: tureis.codfw.wmnet

  • tureis.codfw.wmnet
    • Removed from Puppet master and PuppetDB
    • Downtimed host on Icinga
    • No management interface found (likely a VM)
    • Removed from DebMonitor
Wed, Sep 11, 10:17 AM · Traffic, serviceops, Operations

Tue, Sep 10

ops-monitoring-bot added projects to T232505: Degraded RAID on db2060: ops-codfw, Operations.
Tue, Sep 10, 4:56 PM · Operations, ops-codfw
ops-monitoring-bot added a comment to T225128: Move cloudvirtan* hardware out of CloudVPS back into production Analytics VLAN..

Completed auto-reimage of hosts:

['an-presto1001.eqiad.wmnet']
Tue, Sep 10, 4:45 PM · Analytics-Kanban, ops-eqiad, Operations, netops, Analytics
ops-monitoring-bot added a comment to T225128: Move cloudvirtan* hardware out of CloudVPS back into production Analytics VLAN..

Completed auto-reimage of hosts:

['an-presto1002.eqiad.wmnet']
Tue, Sep 10, 4:40 PM · Analytics-Kanban, ops-eqiad, Operations, netops, Analytics
ops-monitoring-bot added a comment to T225128: Move cloudvirtan* hardware out of CloudVPS back into production Analytics VLAN..

Script wmf-auto-reimage was launched by otto on cumin1001.eqiad.wmnet for hosts:

cloudvirtan1001.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/201909101544_otto_157468_cloudvirtan1001_eqiad_wmnet.log.

Tue, Sep 10, 3:44 PM · Analytics-Kanban, ops-eqiad, Operations, netops, Analytics
ops-monitoring-bot added a comment to T225128: Move cloudvirtan* hardware out of CloudVPS back into production Analytics VLAN..

Script wmf-auto-reimage was launched by otto on cumin1001.eqiad.wmnet for hosts:

cloudvirtan1002.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/201909101535_otto_156624_cloudvirtan1002_eqiad_wmnet.log.

Tue, Sep 10, 3:36 PM · Analytics-Kanban, ops-eqiad, Operations, netops, Analytics
ops-monitoring-bot added a comment to T225128: Move cloudvirtan* hardware out of CloudVPS back into production Analytics VLAN..

Completed auto-reimage of hosts:

['cloudvirtan1002.eqiad.wmnet']
Tue, Sep 10, 3:34 PM · Analytics-Kanban, ops-eqiad, Operations, netops, Analytics
ops-monitoring-bot added a comment to T225128: Move cloudvirtan* hardware out of CloudVPS back into production Analytics VLAN..

Script wmf-auto-reimage was launched by otto on cumin1001.eqiad.wmnet for hosts:

cloudvirtan1002.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/201909101534_otto_155583_cloudvirtan1002_eqiad_wmnet.log.

Tue, Sep 10, 3:34 PM · Analytics-Kanban, ops-eqiad, Operations, netops, Analytics
ops-monitoring-bot added a comment to T225128: Move cloudvirtan* hardware out of CloudVPS back into production Analytics VLAN..

Script wmf-auto-reimage was launched by otto on cumin1001.eqiad.wmnet for hosts:

['an-presto1001.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201909101525_otto_153478.log.

Tue, Sep 10, 3:26 PM · Analytics-Kanban, ops-eqiad, Operations, netops, Analytics

Sun, Sep 8

ops-monitoring-bot added projects to T232267: Degraded RAID on db2053: ops-codfw, Operations.
Sun, Sep 8, 4:54 AM · Operations, ops-codfw

Wed, Sep 4

ops-monitoring-bot added a comment to T202367: Productionize dbproxy101[2-7].eqiad.wmnet and dbproxy200[1-4].

Completed auto-reimage of hosts:

['dbproxy1017.eqiad.wmnet']
Wed, Sep 4, 6:12 AM · DBA
ops-monitoring-bot added a comment to T202367: Productionize dbproxy101[2-7].eqiad.wmnet and dbproxy200[1-4].

Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:

dbproxy1017.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/201909040539_marostegui_173520_dbproxy1017_eqiad_wmnet.log.

Wed, Sep 4, 5:39 AM · DBA

Mon, Sep 2

ops-monitoring-bot added projects to T231785: Degraded RAID on db2046: ops-codfw, Operations.
Mon, Sep 2, 8:10 AM · Operations, ops-codfw

Fri, Aug 30

ops-monitoring-bot added projects to T231690: Degraded RAID on db2059: ops-codfw, Operations.
Fri, Aug 30, 10:03 PM · Operations, ops-codfw
ops-monitoring-bot added projects to T231639: Degraded RAID on db1074: Operations, ops-eqiad.
Fri, Aug 30, 10:52 AM · ops-eqiad, Operations

Wed, Aug 28

ops-monitoring-bot added a comment to T202367: Productionize dbproxy101[2-7].eqiad.wmnet and dbproxy200[1-4].

Completed auto-reimage of hosts:

['dbproxy1018.eqiad.wmnet']
Wed, Aug 28, 8:11 AM · DBA
ops-monitoring-bot added a comment to T202367: Productionize dbproxy101[2-7].eqiad.wmnet and dbproxy200[1-4].

Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:

dbproxy1018.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/201908280754_marostegui_100781_dbproxy1018_eqiad_wmnet.log.

Wed, Aug 28, 7:54 AM · DBA

Tue, Aug 27

ops-monitoring-bot added a comment to T228606: Degraded RAID on elastic1046.

Completed auto-reimage of hosts:

['elastic1046.eqiad.wmnet']
Tue, Aug 27, 12:53 PM · Discovery-Search (Current work), ops-eqiad, Operations
ops-monitoring-bot added a comment to T228606: Degraded RAID on elastic1046.

Script wmf-auto-reimage was launched by gehel on cumin1001.eqiad.wmnet for hosts:

['elastic1046.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201908271218_gehel_128010.log.

Tue, Aug 27, 12:19 PM · Discovery-Search (Current work), ops-eqiad, Operations
ops-monitoring-bot added a comment to T228629: ATS Backends: Test live cache_text traffic .

Completed auto-reimage of hosts:

['cp1075.eqiad.wmnet']
Tue, Aug 27, 8:57 AM · Patch-For-Review, Goal, Traffic, Operations
ops-monitoring-bot added a comment to T228629: ATS Backends: Test live cache_text traffic .

Script wmf-auto-reimage was launched by ema on cumin1001.eqiad.wmnet for hosts:

['cp1075.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201908270828_ema_81110.log.

Tue, Aug 27, 8:28 AM · Patch-For-Review, Goal, Traffic, Operations
ops-monitoring-bot added projects to T231267: Degraded RAID on db2051: ops-codfw, Operations.
Tue, Aug 27, 5:47 AM · Operations, ops-codfw

Mon, Aug 26

ops-monitoring-bot added projects to T231199: Degraded RAID on db1063: Operations, ops-eqiad.
Mon, Aug 26, 1:33 PM · DBA, ops-eqiad, Operations

Sun, Aug 25

ops-monitoring-bot added projects to T231176: Degraded RAID on db2035: ops-codfw, Operations.
Sun, Aug 25, 9:00 PM · Operations, ops-codfw

Fri, Aug 23

ops-monitoring-bot added a comment to T223216: Decommission db2034.

cookbooks.sre.hosts.decommission executed by robh@cumin1001 for hosts: db2034.codfw.wmnet

  • db2034.codfw.wmnet
    • Removed from Puppet master and PuppetDB
    • Downtimed host on Icinga
    • Downtimed management interface on Icinga
    • Removed from DebMonitor
Fri, Aug 23, 11:34 PM · Operations, decommission, ops-codfw
ops-monitoring-bot added a comment to T228281: decommission db2045.codfw.wmnet.

cookbooks.sre.hosts.decommission executed by robh@cumin1001 for hosts: db2045.codfw.wmnet

  • db2045.codfw.wmnet
    • Removed from Puppet master and PuppetDB
    • Downtimed host on Icinga
    • Downtimed management interface on Icinga
    • Removed from DebMonitor
Fri, Aug 23, 11:03 PM · ops-codfw, Operations, DC-Ops, decommission
ops-monitoring-bot added a comment to T200209: Decom graphite2001/WMF6160 .

cookbooks.sre.hosts.decommission executed by robh@cumin1001 for hosts: graphite2001.codfw.wmnet

  • graphite2001.codfw.wmnet
    • Removed from Puppet master and PuppetDB
    • Downtimed host on Icinga
    • Downtimed management interface on Icinga
    • Removed from DebMonitor
Fri, Aug 23, 7:11 PM · Patch-For-Review, decommission, ops-codfw, Operations, observability
ops-monitoring-bot added a comment to T200210: Decom graphite2002 & return server to spares pool.

cookbooks.sre.hosts.decommission executed by robh@cumin1001 for hosts: graphite2002.codfw.wmnet

  • graphite2002.codfw.wmnet
    • Removed from Puppet master and PuppetDB
    • Downtimed host on Icinga
    • Downtimed management interface on Icinga
    • Removed from DebMonitor
Fri, Aug 23, 5:40 PM · decommission, observability, Operations, ops-codfw

Thu, Aug 22

ops-monitoring-bot added projects to T231056: Degraded RAID on db2056: ops-codfw, Operations.
Thu, Aug 22, 11:23 PM · Operations, ops-codfw

Aug 22 2019

ops-monitoring-bot added a comment to T231021: Decommission analytics-tool1002 (old turnilo vm).

cookbooks.sre.hosts.decommission executed by elukey@cumin1001 for hosts: analytics-tool1002.eqiad.wmnet

  • analytics-tool1002.eqiad.wmnet
    • Removed from Puppet master and PuppetDB
    • Downtimed host on Icinga
    • No management interface found (likely a VM)
    • Removed from DebMonitor
Aug 22 2019, 5:12 PM · vm-requests, Operations, Analytics
ops-monitoring-bot added a comment to T227025: (Need By: August 31) rack/setup/install (3) new zookeeper nodes.

Script wmf-auto-reimage was launched by elukey on cumin1001.eqiad.wmnet for hosts:

['an-conf1002.eqiad.wmnet', 'an-conf1003.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201908221226_elukey_32888.log.

Aug 22 2019, 12:26 PM · User-Elukey, Operations, ops-eqiad
ops-monitoring-bot added a comment to T227025: (Need By: August 31) rack/setup/install (3) new zookeeper nodes.

Completed auto-reimage of hosts:

['an-conf1001.eqiad.wmnet']
Aug 22 2019, 12:12 PM · User-Elukey, Operations, ops-eqiad
ops-monitoring-bot added a comment to T227025: (Need By: August 31) rack/setup/install (3) new zookeeper nodes.

Script wmf-auto-reimage was launched by elukey on cumin1001.eqiad.wmnet for hosts:

['an-conf1001.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201908221201_elukey_27190.log.

Aug 22 2019, 12:01 PM · User-Elukey, Operations, ops-eqiad
ops-monitoring-bot added a comment to T202367: Productionize dbproxy101[2-7].eqiad.wmnet and dbproxy200[1-4].

Completed auto-reimage of hosts:

['dbproxy1019.eqiad.wmnet']
Aug 22 2019, 9:07 AM · DBA
ops-monitoring-bot added a comment to T202367: Productionize dbproxy101[2-7].eqiad.wmnet and dbproxy200[1-4].

Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:

dbproxy1019.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/201908220848_marostegui_249574_dbproxy1019_eqiad_wmnet.log.

Aug 22 2019, 8:48 AM · DBA

Aug 19 2019

ops-monitoring-bot added a comment to T230705: Replace db2044 (m2 codfw master) with db2067.

Completed auto-reimage of hosts:

['db2067.codfw.wmnet']
Aug 19 2019, 9:37 AM · DBA
ops-monitoring-bot added a comment to T230705: Replace db2044 (m2 codfw master) with db2067.

Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:

['db2067.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201908190907_marostegui_167018.log.

Aug 19 2019, 9:07 AM · DBA
ops-monitoring-bot added a comment to T230705: Replace db2044 (m2 codfw master) with db2067.

Completed auto-reimage of hosts:

['db2067.codfw.wmnet']
Aug 19 2019, 8:23 AM · DBA
ops-monitoring-bot added a comment to T230705: Replace db2044 (m2 codfw master) with db2067.

Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:

['db2067.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201908190807_marostegui_154648.log.

Aug 19 2019, 8:07 AM · DBA
ops-monitoring-bot added a comment to T230705: Replace db2044 (m2 codfw master) with db2067.

Completed auto-reimage of hosts:

['db2067.codfw.wmnet']
Aug 19 2019, 7:25 AM · DBA
ops-monitoring-bot added a comment to T230705: Replace db2044 (m2 codfw master) with db2067.

Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:

['db2067.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201908190711_marostegui_144027.log.

Aug 19 2019, 7:11 AM · DBA

Aug 18 2019

ops-monitoring-bot added projects to T230682: Degraded RAID on db1063: Operations, ops-eqiad.
Aug 18 2019, 11:39 AM · DBA, ops-eqiad, Operations

Aug 16 2019

ops-monitoring-bot added a comment to T229871: relocate/reimage cloudvirt1023 with 10G interfaces.

Completed auto-reimage of hosts:

['cloudvirt1023.eqiad.wmnet']
Aug 16 2019, 9:56 AM · ops-eqiad, DC-Ops, Operations, Epic, cloud-services-team (Kanban)
ops-monitoring-bot added a comment to T229871: relocate/reimage cloudvirt1023 with 10G interfaces.

Script wmf-auto-reimage was launched by aborrero on cumin1001.eqiad.wmnet for hosts:

cloudvirt1023.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/201908160945_aborrero_145339_cloudvirt1023_eqiad_wmnet.log.

Aug 16 2019, 9:46 AM · ops-eqiad, DC-Ops, Operations, Epic, cloud-services-team (Kanban)
ops-monitoring-bot added a comment to T229871: relocate/reimage cloudvirt1023 with 10G interfaces.

Completed auto-reimage of hosts:

['cloudvirt1023.eqiad.wmnet']
Aug 16 2019, 9:44 AM · ops-eqiad, DC-Ops, Operations, Epic, cloud-services-team (Kanban)
ops-monitoring-bot added a comment to T229871: relocate/reimage cloudvirt1023 with 10G interfaces.

Script wmf-auto-reimage was launched by aborrero on cumin1001.eqiad.wmnet for hosts:

cloudvirt1023.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/201908160928_aborrero_141854_cloudvirt1023_eqiad_wmnet.log.

Aug 16 2019, 9:28 AM · ops-eqiad, DC-Ops, Operations, Epic, cloud-services-team (Kanban)

Aug 15 2019

ops-monitoring-bot added projects to T230575: Degraded RAID on cloudvirt1018: Operations, ops-eqiad.
Aug 15 2019, 8:25 PM · ops-eqiad, Operations
ops-monitoring-bot added a comment to T229871: relocate/reimage cloudvirt1023 with 10G interfaces.

Completed auto-reimage of hosts:

['cloudvirt1023.mgmt.eqiad.wmnet']
Aug 15 2019, 4:09 PM · ops-eqiad, DC-Ops, Operations, Epic, cloud-services-team (Kanban)
ops-monitoring-bot added a comment to T229871: relocate/reimage cloudvirt1023 with 10G interfaces.

Script wmf-auto-reimage was launched by phamhi on cumin1001.eqiad.wmnet for hosts:

cloudvirt1023.mgmt.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/201908151508_phamhi_184898_cloudvirt1023_mgmt_eqiad_wmnet.log.

Aug 15 2019, 3:09 PM · ops-eqiad, DC-Ops, Operations, Epic, cloud-services-team (Kanban)

Aug 14 2019

ops-monitoring-bot added a comment to T229871: relocate/reimage cloudvirt1023 with 10G interfaces.

Completed auto-reimage of hosts:

['cloudvirt1023.mgmt.eqiad.wmnet']
Aug 14 2019, 6:08 PM · ops-eqiad, DC-Ops, Operations, Epic, cloud-services-team (Kanban)
ops-monitoring-bot added a comment to T229871: relocate/reimage cloudvirt1023 with 10G interfaces.

Script wmf-auto-reimage was launched by phamhi on cumin1001.eqiad.wmnet for hosts:

cloudvirt1023.mgmt.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/201908141707_phamhi_95392_cloudvirt1023_mgmt_eqiad_wmnet.log.

Aug 14 2019, 5:07 PM · ops-eqiad, DC-Ops, Operations, Epic, cloud-services-team (Kanban)
ops-monitoring-bot added a comment to T229871: relocate/reimage cloudvirt1023 with 10G interfaces.

Completed auto-reimage of hosts:

['cloudvirt1023.mgmt.eqiad.wmnet']
Aug 14 2019, 4:30 PM · ops-eqiad, DC-Ops, Operations, Epic, cloud-services-team (Kanban)
ops-monitoring-bot added a comment to T229871: relocate/reimage cloudvirt1023 with 10G interfaces.

Script wmf-auto-reimage was launched by phamhi on cumin1001.eqiad.wmnet for hosts:

cloudvirt1023.mgmt.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/201908141630_phamhi_10976_cloudvirt1023_mgmt_eqiad_wmnet.log.

Aug 14 2019, 4:30 PM · ops-eqiad, DC-Ops, Operations, Epic, cloud-services-team (Kanban)
ops-monitoring-bot added a comment to T230459: Replace db2044 with db2063.

Completed auto-reimage of hosts:

['db2063.codfw.wmnet']
Aug 14 2019, 8:18 AM · DBA
ops-monitoring-bot added a comment to T230459: Replace db2044 with db2063.

Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:

['db2063.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201908140717_marostegui_75740.log.

Aug 14 2019, 7:18 AM · DBA

Aug 13 2019

ops-monitoring-bot added projects to T230442: Degraded RAID on cloudvirt1024 -- Filesystem mounted read-only: Operations, ops-eqiad.
Aug 13 2019, 6:31 PM · ops-eqiad, Operations
ops-monitoring-bot added a comment to T224572: Migrate pool counters to Buster.

cookbooks.sre.hosts.decommission executed by volans@cumin2001 for hosts: poolcounter2001.codfw.wmnet

  • poolcounter2001.codfw.wmnet
    • Removed from Puppet master and PuppetDB
    • Downtimed host on Icinga
    • No management interface found (likely a VM)
    • Removed from DebMonitor
Aug 13 2019, 2:04 PM · serviceops, Operations

Aug 11 2019

ops-monitoring-bot added projects to T230289: Degraded RAID on cloudvirt1024 -- Filesystem mounted read-only: Operations, ops-eqiad.
Aug 11 2019, 12:10 AM · cloud-services-team, ops-eqiad, Operations

Aug 10 2019

ops-monitoring-bot added projects to T230275: Degraded RAID on ms-be2021: ops-codfw, Operations.
Aug 10 2019, 5:43 PM · media-storage, Operations, ops-codfw

Aug 7 2019

ops-monitoring-bot added a comment to T224572: Migrate pool counters to Buster.

cookbooks.sre.hosts.decommission executed by jmm@cumin2001 for hosts: poolcounter2002.codfw.wmnet

  • poolcounter2002.codfw.wmnet
    • Removed from Puppet master and PuppetDB
    • Downtimed host on Icinga
    • No management interface found (likely a VM)
    • Removed from DebMonitor
Aug 7 2019, 10:03 AM · serviceops, Operations
ops-monitoring-bot added a comment to T224572: Migrate pool counters to Buster.

cookbooks.sre.hosts.decommission executed by jmm@cumin2001 for hosts: poolcounter2001.codfw.wmnet

  • poolcounter2001.codfw.wmnet
    • Removed from Puppet master and PuppetDB
    • Downtimed host on Icinga
    • No management interface found (likely a VM)
    • Removed from DebMonitor
Aug 7 2019, 8:37 AM · serviceops, Operations
ops-monitoring-bot added a comment to T224572: Migrate pool counters to Buster.

cookbooks.sre.hosts.decommission executed by jmm@cumin2001 for hosts: poolcounter1003.eqiad.wmnet

  • poolcounter1003.eqiad.wmnet
    • Removed from Puppet master and PuppetDB
    • Downtimed host on Icinga
    • No management interface found (likely a VM)
    • Removed from DebMonitor
Aug 7 2019, 7:36 AM · serviceops, Operations
ops-monitoring-bot added a comment to T224572: Migrate pool counters to Buster.

cookbooks.sre.hosts.decommission executed by jmm@cumin2001 for hosts: poolcounter1001.eqiad.wmnet

  • poolcounter1001.eqiad.wmnet
    • Removed from Puppet master and PuppetDB
    • Downtimed host on Icinga
    • No management interface found (likely a VM)
    • Removed from DebMonitor
Aug 7 2019, 7:32 AM · serviceops, Operations

Aug 1 2019

ops-monitoring-bot added a comment to T229586: decommission cp1008, cp1071, cp1072, cp1073, cp1074, cp1099.

Completed auto-reimage of hosts:

['cp1073.eqiad.wmnet', 'cp1074.eqiad.wmnet', 'cp1072.eqiad.wmnet', 'cp1071.eqiad.wmnet', 'cp1099.eqiad.wmnet', 'cp1008.wikimedia.org']
Aug 1 2019, 5:09 PM · ops-eqiad, DC-Ops, Operations, decommission
ops-monitoring-bot added a comment to T229586: decommission cp1008, cp1071, cp1072, cp1073, cp1074, cp1099.

Script wmf-auto-reimage was launched by bblack on cumin1001.eqiad.wmnet for hosts:

['cp1008.wikimedia.org', 'cp1071.eqiad.wmnet', 'cp1072.eqiad.wmnet', 'cp1073.eqiad.wmnet', 'cp1074.eqiad.wmnet', 'cp1099.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201908011643_bblack_108239.log.

Aug 1 2019, 4:44 PM · ops-eqiad, DC-Ops, Operations, decommission
ops-monitoring-bot added a comment to T228976: Allow to avoid installing HHVM from the mediawiki puppet module and profile.

Completed auto-reimage of hosts:

['mw1348.eqiad.wmnet']
Aug 1 2019, 8:51 AM · CPT Initiatives (Parsoid REST API in PHP (CDP2)), Patch-For-Review, serviceops, Parsoid-PHP
ops-monitoring-bot added a comment to T228976: Allow to avoid installing HHVM from the mediawiki puppet module and profile.

Script wmf-auto-reimage was launched by oblivian on cumin1001.eqiad.wmnet for hosts:

mw1348.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/201908010736_oblivian_258027_mw1348_eqiad_wmnet.log.

Aug 1 2019, 7:37 AM · CPT Initiatives (Parsoid REST API in PHP (CDP2)), Patch-For-Review, serviceops, Parsoid-PHP
ops-monitoring-bot added a comment to T228976: Allow to avoid installing HHVM from the mediawiki puppet module and profile.

Completed auto-reimage of hosts:

['mw1348.eqiad.wmnet']
Aug 1 2019, 7:36 AM · CPT Initiatives (Parsoid REST API in PHP (CDP2)), Patch-For-Review, serviceops, Parsoid-PHP
ops-monitoring-bot added a comment to T228976: Allow to avoid installing HHVM from the mediawiki puppet module and profile.

Script wmf-auto-reimage was launched by oblivian on cumin1001.eqiad.wmnet for hosts:

mw1348.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/201908010734_oblivian_256659_mw1348_eqiad_wmnet.log.

Aug 1 2019, 7:34 AM · CPT Initiatives (Parsoid REST API in PHP (CDP2)), Patch-For-Review, serviceops, Parsoid-PHP

Jul 31 2019

ops-monitoring-bot added a comment to T228190: Roll out Anycast RecDNS to more servers.

Completed auto-reimage of hosts:

['cp1008.wikimedia.org']
Jul 31 2019, 5:54 PM · Operations, Traffic
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL