Page MenuHomePhabricator

ops-monitoring-bot (Operations Monitoring Bot)
UserBot

Projects

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Thursday

  • Clear sailing ahead.

User Details

User Since
Aug 12 2016, 1:45 PM (153 w, 3 d)
Roles
Bot
Availability
Available
LDAP User
Unknown
MediaWiki User
Unknown

Bot managed by Operations for automated interaction with Phabricator from monitoring tools.

Recent Activity

Sun, Jul 21

ops-monitoring-bot added projects to T228606: Degraded RAID on elastic1046: Operations, ops-eqiad.
Sun, Jul 21, 8:10 PM · ops-eqiad, Operations
ops-monitoring-bot added projects to T228577: Degraded RAID on analytics1029: Operations, ops-eqiad.
Sun, Jul 21, 6:20 AM · ops-eqiad, Operations

Thu, Jul 18

ops-monitoring-bot added projects to T228478: Degraded RAID on cobalt: Operations, ops-eqiad.
Thu, Jul 18, 9:17 PM · ops-eqiad, Operations
ops-monitoring-bot added a comment to T26948: FlaggedRevs should not reinvent the autopromote wheel.

Completed auto-reimage of hosts:

['mw2250.codfw.wmnet']
Thu, Jul 18, 1:40 AM · MediaWiki-extensions-FlaggedRevs

Wed, Jul 17

ops-monitoring-bot added a comment to T26948: FlaggedRevs should not reinvent the autopromote wheel.

Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts:

mw2250.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/201907172234_dzahn_204900_mw2250_codfw_wmnet.log.

Wed, Jul 17, 10:34 PM · MediaWiki-extensions-FlaggedRevs
ops-monitoring-bot added a comment to T225128: Move cloudvirtan* hardware out of CloudVPS back into production Analytics VLAN..

cookbooks.sre.hosts.decommission executed by otto@cumin1001 for hosts: cloudvirtan[1001-1005].eqiad.wmnet

  • cloudvirtan1002.eqiad.wmnet
    • Removed from Puppet master and PuppetDB
    • Downtimed host on Icinga
    • Downtimed management interface on Icinga
    • Removed from DebMonitor
  • cloudvirtan1001.eqiad.wmnet
    • Removed from Puppet master and PuppetDB
    • Downtimed host on Icinga
    • Downtimed management interface on Icinga
    • Removed from DebMonitor
  • cloudvirtan1003.eqiad.wmnet
    • Removed from Puppet master and PuppetDB
    • Downtimed host on Icinga
    • Downtimed management interface on Icinga
    • Removed from DebMonitor
  • cloudvirtan1005.eqiad.wmnet
    • Removed from Puppet master and PuppetDB
    • Downtimed host on Icinga
    • Downtimed management interface on Icinga
    • Removed from DebMonitor
  • cloudvirtan1004.eqiad.wmnet
    • Removed from Puppet master and PuppetDB
    • Downtimed host on Icinga
    • Downtimed management interface on Icinga
    • Removed from DebMonitor
Wed, Jul 17, 2:41 PM · Analytics-Kanban, ops-eqiad, Operations, netops, Analytics
ops-monitoring-bot added a comment to T201342: rack/setup/install puppetmaster1003.eqiad.wmnet.

Completed auto-reimage of hosts:

['puppetmaster1003.eqiad.wmnet']
Wed, Jul 17, 10:22 AM · Operations
ops-monitoring-bot added a comment to T201342: rack/setup/install puppetmaster1003.eqiad.wmnet.

Script wmf-auto-reimage was launched by jbond on cumin1001.eqiad.wmnet for hosts:

['puppetmaster1003.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201907170947_jbond_58968.log.

Wed, Jul 17, 9:47 AM · Operations
ops-monitoring-bot added projects to T228246: Degraded RAID on ms-be2019: ops-codfw, Operations.
Wed, Jul 17, 8:10 AM · Operations, ops-codfw
ops-monitoring-bot added projects to T228245: Degraded RAID on ms-be2019: ops-codfw, Operations.
Wed, Jul 17, 8:03 AM · Operations, ops-codfw

Mon, Jul 15

ops-monitoring-bot added a comment to T225090: Decommission db2042.

cookbooks.sre.hosts.decommission executed by robh@cumin1001 for hosts: db2042.codfw.wmnet

  • db2042.codfw.wmnet
    • Removed from Puppet master and PuppetDB
    • Downtimed host on Icinga
    • Downtimed management interface on Icinga
    • Removed from DebMonitor
Mon, Jul 15, 11:31 PM · Operations, ops-codfw, decommission

Sat, Jul 13

ops-monitoring-bot added projects to T227940: (OoW) Degraded RAID on analytics1032: Operations, ops-eqiad.
Sat, Jul 13, 8:56 AM · ops-eqiad, Operations

Fri, Jul 12

ops-monitoring-bot added a comment to T220355: decom netmon1003.

cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: netmon1003.wikimedia.org

  • netmon1003.wikimedia.org
    • Removed from Puppet master and PuppetDB
    • Downtimed host on Icinga
    • No management interface found (likely a VM)
    • Removed from DebMonitor
Fri, Jul 12, 11:31 PM · Operations

Thu, Jul 11

ops-monitoring-bot added a comment to T222960: Fix restbase1017's physical rack.

Completed auto-reimage of hosts:

['restbase1017.eqiad.wmnet']
Thu, Jul 11, 11:47 PM · Patch-For-Review, serviceops, Core Platform Team Workboards (Team 2), Operations, Services (doing), Core Platform Team (Security, stability, performance and scalability (TEC1)), User-Eevans, Cassandra
ops-monitoring-bot added projects to T227829: Degraded RAID on db2044: ops-codfw, Operations.
Thu, Jul 11, 9:40 PM · Operations, ops-codfw
ops-monitoring-bot added a comment to T222960: Fix restbase1017's physical rack.

Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts:

restbase1017.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/201907112124_dzahn_140714_restbase1017_eqiad_wmnet.log.

Thu, Jul 11, 9:24 PM · Patch-For-Review, serviceops, Core Platform Team Workboards (Team 2), Operations, Services (doing), Core Platform Team (Security, stability, performance and scalability (TEC1)), User-Eevans, Cassandra

Wed, Jul 10

ops-monitoring-bot added a comment to T222960: Fix restbase1017's physical rack.

Completed auto-reimage of hosts:

['restbase1017.eqiad.wmnet']
Wed, Jul 10, 11:16 PM · Patch-For-Review, serviceops, Core Platform Team Workboards (Team 2), Operations, Services (doing), Core Platform Team (Security, stability, performance and scalability (TEC1)), User-Eevans, Cassandra
ops-monitoring-bot added a comment to T222960: Fix restbase1017's physical rack.

Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts:

restbase1017.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/201907102053_dzahn_114135_restbase1017_eqiad_wmnet.log.

Wed, Jul 10, 8:54 PM · Patch-For-Review, serviceops, Core Platform Team Workboards (Team 2), Operations, Services (doing), Core Platform Team (Security, stability, performance and scalability (TEC1)), User-Eevans, Cassandra

Sun, Jul 7

ops-monitoring-bot added projects to T227394: Degraded RAID on restbase-dev1006: Operations, ops-eqiad.
Sun, Jul 7, 12:59 AM · ops-eqiad, Operations

Fri, Jul 5

ops-monitoring-bot added a comment to T226638: Replace Varnish backends with ATS on cache upload nodes in eqiad.

Completed auto-reimage of hosts:

['cp1090.eqiad.wmnet']
Fri, Jul 5, 1:01 PM · Operations, Traffic
ops-monitoring-bot added a comment to T226638: Replace Varnish backends with ATS on cache upload nodes in eqiad.

Script wmf-auto-reimage was launched by ema on cumin1001.eqiad.wmnet for hosts:

['cp1090.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201907051215_ema_108774.log.

Fri, Jul 5, 12:15 PM · Operations, Traffic
ops-monitoring-bot added a comment to T226638: Replace Varnish backends with ATS on cache upload nodes in eqiad.

Completed auto-reimage of hosts:

['cp1088.eqiad.wmnet']
Fri, Jul 5, 11:45 AM · Operations, Traffic
ops-monitoring-bot added a comment to T226638: Replace Varnish backends with ATS on cache upload nodes in eqiad.

Script wmf-auto-reimage was launched by ema on cumin1001.eqiad.wmnet for hosts:

['cp1088.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201907051103_ema_88729.log.

Fri, Jul 5, 11:04 AM · Operations, Traffic
ops-monitoring-bot added a comment to T226638: Replace Varnish backends with ATS on cache upload nodes in eqiad.

Completed auto-reimage of hosts:

['cp1086.eqiad.wmnet']
Fri, Jul 5, 10:52 AM · Operations, Traffic
ops-monitoring-bot added a comment to T226638: Replace Varnish backends with ATS on cache upload nodes in eqiad.

Script wmf-auto-reimage was launched by ema on cumin1001.eqiad.wmnet for hosts:

['cp1086.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201907050945_ema_69518.log.

Fri, Jul 5, 9:45 AM · Operations, Traffic

Thu, Jul 4

ops-monitoring-bot added a comment to T226638: Replace Varnish backends with ATS on cache upload nodes in eqiad.

Completed auto-reimage of hosts:

['cp1084.eqiad.wmnet']
Thu, Jul 4, 4:34 PM · Operations, Traffic
ops-monitoring-bot added a comment to T226638: Replace Varnish backends with ATS on cache upload nodes in eqiad.

Script wmf-auto-reimage was launched by ema on cumin1001.eqiad.wmnet for hosts:

['cp1084.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201907041536_ema_101251.log.

Thu, Jul 4, 3:37 PM · Operations, Traffic
ops-monitoring-bot added a comment to T226638: Replace Varnish backends with ATS on cache upload nodes in eqiad.

Completed auto-reimage of hosts:

['cp1082.eqiad.wmnet']
Thu, Jul 4, 3:17 PM · Operations, Traffic
ops-monitoring-bot added a comment to T226638: Replace Varnish backends with ATS on cache upload nodes in eqiad.

Script wmf-auto-reimage was launched by ema on cumin1001.eqiad.wmnet for hosts:

['cp1082.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201907041428_ema_86233.log.

Thu, Jul 4, 2:28 PM · Operations, Traffic
ops-monitoring-bot added a comment to T226638: Replace Varnish backends with ATS on cache upload nodes in eqiad.

Completed auto-reimage of hosts:

['cp1080.eqiad.wmnet']
Thu, Jul 4, 1:31 PM · Operations, Traffic
ops-monitoring-bot added a comment to T226638: Replace Varnish backends with ATS on cache upload nodes in eqiad.

Script wmf-auto-reimage was launched by ema on cumin1001.eqiad.wmnet for hosts:

['cp1080.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201907041244_ema_54006.log.

Thu, Jul 4, 12:44 PM · Operations, Traffic
ops-monitoring-bot added a comment to T226638: Replace Varnish backends with ATS on cache upload nodes in eqiad.

Completed auto-reimage of hosts:

['cp1078.eqiad.wmnet']
Thu, Jul 4, 8:00 AM · Operations, Traffic
ops-monitoring-bot added a comment to T226638: Replace Varnish backends with ATS on cache upload nodes in eqiad.

Script wmf-auto-reimage was launched by ema on cumin1001.eqiad.wmnet for hosts:

['cp1078.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201907040720_ema_114973.log.

Thu, Jul 4, 7:20 AM · Operations, Traffic

Wed, Jul 3

ops-monitoring-bot added projects to T227222: Degraded RAID on cloudelastic1003: Operations, ops-eqiad.
Wed, Jul 3, 7:03 PM · ops-eqiad, Operations
ops-monitoring-bot added a comment to T220590: Decom ms-be101[345].

cookbooks.sre.hosts.decommission executed by robh@cumin1001 for hosts: ms-be1015.eqiad.wmnet

  • ms-be1015.eqiad.wmnet
    • Removed from Puppet master and PuppetDB
    • Downtimed host on Icinga
    • Downtimed management interface on Icinga
    • Removed from DebMonitor
Wed, Jul 3, 4:18 PM · ops-eqiad, decommission, User-fgiunchedi, media-storage, Operations
ops-monitoring-bot added a comment to T220590: Decom ms-be101[345].

cookbooks.sre.hosts.decommission executed by robh@cumin1001 for hosts: ms-be1014.eqiad.wmnet

  • ms-be1014.eqiad.wmnet
    • Removed from Puppet master and PuppetDB
    • Downtimed host on Icinga
    • Downtimed management interface on Icinga
    • Removed from DebMonitor
Wed, Jul 3, 4:18 PM · ops-eqiad, decommission, User-fgiunchedi, media-storage, Operations
ops-monitoring-bot added a comment to T220590: Decom ms-be101[345].

cookbooks.sre.hosts.decommission executed by robh@cumin1001 for hosts: ms-be1013.eqiad.wmnet

  • ms-be1013.eqiad.wmnet
    • Removed from Puppet master and PuppetDB
    • Downtimed host on Icinga
    • Downtimed management interface on Icinga
    • Removed from DebMonitor
Wed, Jul 3, 4:18 PM · ops-eqiad, decommission, User-fgiunchedi, media-storage, Operations
ops-monitoring-bot added a comment to T226638: Replace Varnish backends with ATS on cache upload nodes in eqiad.

Completed auto-reimage of hosts:

['cp1076.eqiad.wmnet']
Wed, Jul 3, 1:49 PM · Operations, Traffic
ops-monitoring-bot added a comment to T226638: Replace Varnish backends with ATS on cache upload nodes in eqiad.

Script wmf-auto-reimage was launched by ema on cumin1001.eqiad.wmnet for hosts:

['cp1076.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201907031310_ema_155735.log.

Wed, Jul 3, 1:11 PM · Operations, Traffic
ops-monitoring-bot added a comment to T226637: Replace Varnish backends with ATS on cache upload nodes in codfw.

Completed auto-reimage of hosts:

['cp2026.codfw.wmnet']
Wed, Jul 3, 12:51 PM · Operations, Traffic
ops-monitoring-bot added a comment to T226637: Replace Varnish backends with ATS on cache upload nodes in codfw.

Script wmf-auto-reimage was launched by ema on cumin2001.codfw.wmnet for hosts:

['cp2026.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201907031214_ema_26404.log.

Wed, Jul 3, 12:14 PM · Operations, Traffic

Tue, Jul 2

ops-monitoring-bot added projects to T227107: Degraded RAID on db2049: ops-codfw, Operations.
Tue, Jul 2, 4:04 PM · DBA, Operations, ops-codfw
ops-monitoring-bot added a comment to T226637: Replace Varnish backends with ATS on cache upload nodes in codfw.

Completed auto-reimage of hosts:

['cp2025.codfw.wmnet']
Tue, Jul 2, 3:39 PM · Operations, Traffic
ops-monitoring-bot added a comment to T226637: Replace Varnish backends with ATS on cache upload nodes in codfw.

Script wmf-auto-reimage was launched by ema on cumin2001.codfw.wmnet for hosts:

['cp2025.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201907021456_ema_30365.log.

Tue, Jul 2, 2:56 PM · Operations, Traffic
ops-monitoring-bot added a comment to T226637: Replace Varnish backends with ATS on cache upload nodes in codfw.

Completed auto-reimage of hosts:

['cp2024.codfw.wmnet']
Tue, Jul 2, 2:14 PM · Operations, Traffic
ops-monitoring-bot added a comment to T226637: Replace Varnish backends with ATS on cache upload nodes in codfw.

Script wmf-auto-reimage was launched by ema on cumin2001.codfw.wmnet for hosts:

['cp2024.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201907021332_ema_14640.log.

Tue, Jul 2, 1:33 PM · Operations, Traffic
ops-monitoring-bot added a comment to T226637: Replace Varnish backends with ATS on cache upload nodes in codfw.

Completed auto-reimage of hosts:

['cp2022.codfw.wmnet']
Tue, Jul 2, 12:55 PM · Operations, Traffic
ops-monitoring-bot added a comment to T226637: Replace Varnish backends with ATS on cache upload nodes in codfw.

Script wmf-auto-reimage was launched by ema on cumin2001.codfw.wmnet for hosts:

['cp2022.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201907021159_ema_27751.log.

Tue, Jul 2, 11:59 AM · Operations, Traffic
ops-monitoring-bot added projects to T227076: Upgrade firmware on ms-be1021 (Was: Degraded RAID on ms-be1021): Operations, ops-eqiad.
Tue, Jul 2, 11:19 AM · serviceops, ops-eqiad, Operations
ops-monitoring-bot added a comment to T226637: Replace Varnish backends with ATS on cache upload nodes in codfw.

Completed auto-reimage of hosts:

['cp2020.codfw.wmnet']
Tue, Jul 2, 9:37 AM · Operations, Traffic
ops-monitoring-bot added a comment to T226637: Replace Varnish backends with ATS on cache upload nodes in codfw.

Script wmf-auto-reimage was launched by ema on cumin2001.codfw.wmnet for hosts:

['cp2020.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201907020859_ema_24277.log.

Tue, Jul 2, 8:59 AM · Operations, Traffic
ops-monitoring-bot added a comment to T226637: Replace Varnish backends with ATS on cache upload nodes in codfw.

Completed auto-reimage of hosts:

['cp2018.codfw.wmnet']
Tue, Jul 2, 8:40 AM · Operations, Traffic
ops-monitoring-bot added a comment to T226637: Replace Varnish backends with ATS on cache upload nodes in codfw.

Script wmf-auto-reimage was launched by ema on cumin2001.codfw.wmnet for hosts:

['cp2018.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201907020802_ema_12167.log.

Tue, Jul 2, 8:02 AM · Operations, Traffic

Mon, Jul 1

ops-monitoring-bot added a comment to T226637: Replace Varnish backends with ATS on cache upload nodes in codfw.

Completed auto-reimage of hosts:

['cp2017.codfw.wmnet']
Mon, Jul 1, 3:42 PM · Operations, Traffic
ops-monitoring-bot added a comment to T226637: Replace Varnish backends with ATS on cache upload nodes in codfw.

Script wmf-auto-reimage was launched by ema on cumin2001.codfw.wmnet for hosts:

['cp2017.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201907011430_ema_1568.log.

Mon, Jul 1, 2:30 PM · Operations, Traffic
ops-monitoring-bot added a comment to T226637: Replace Varnish backends with ATS on cache upload nodes in codfw.

Completed auto-reimage of hosts:

['cp2014.codfw.wmnet']
Mon, Jul 1, 6:53 AM · Operations, Traffic
ops-monitoring-bot added a comment to T226637: Replace Varnish backends with ATS on cache upload nodes in codfw.

Script wmf-auto-reimage was launched by ema on cumin1001.eqiad.wmnet for hosts:

['cp2014.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201907010623_ema_256518.log.

Mon, Jul 1, 6:23 AM · Operations, Traffic
ops-monitoring-bot added projects to T226948: Degraded RAID on mw2250: ops-codfw, Operations.
Mon, Jul 1, 3:17 AM · User-jijiki, serviceops, Operations, ops-codfw

Sun, Jun 30

ops-monitoring-bot added projects to T226936: Degraded RAID on helium: Operations, ops-eqiad.
Sun, Jun 30, 12:37 PM · ops-eqiad, Operations
ops-monitoring-bot added projects to T226933: Degraded RAID on helium: Operations, ops-eqiad.
Sun, Jun 30, 11:56 AM · ops-eqiad, Operations
ops-monitoring-bot added projects to T226924: Degraded RAID on helium: Operations, ops-eqiad.
Sun, Jun 30, 9:40 AM · ops-eqiad, Operations
ops-monitoring-bot added projects to T226923: Degraded RAID on helium: Operations, ops-eqiad.
Sun, Jun 30, 9:08 AM · ops-eqiad, Operations
ops-monitoring-bot added projects to T226921: Degraded RAID on helium: Operations, ops-eqiad.
Sun, Jun 30, 8:37 AM · ops-eqiad, Operations
ops-monitoring-bot added projects to T226919: Degraded RAID on helium: Operations, ops-eqiad.
Sun, Jun 30, 6:10 AM · ops-eqiad, Operations
ops-monitoring-bot added projects to T226917: Degraded RAID on helium: Operations, ops-eqiad.
Sun, Jun 30, 5:18 AM · ops-eqiad, Operations
ops-monitoring-bot added projects to T226916: Degraded RAID on helium: Operations, ops-eqiad.
Sun, Jun 30, 3:13 AM · ops-eqiad, Operations
ops-monitoring-bot added projects to T226915: Degraded RAID on helium: Operations, ops-eqiad.
Sun, Jun 30, 2:10 AM · ops-eqiad, Operations
ops-monitoring-bot added projects to T226913: Degraded RAID on helium: Operations, ops-eqiad.
Sun, Jun 30, 1:38 AM · ops-eqiad, Operations
ops-monitoring-bot added projects to T226912: Degraded RAID on helium: Operations, ops-eqiad.
Sun, Jun 30, 1:07 AM · ops-eqiad, Operations

Sat, Jun 29

ops-monitoring-bot added projects to T226909: Degraded RAID on helium: Operations, ops-eqiad.
Sat, Jun 29, 11:12 PM · ops-eqiad, Operations
ops-monitoring-bot added projects to T226906: Degraded RAID on helium: Operations, ops-eqiad.
Sat, Jun 29, 10:09 PM · ops-eqiad, Operations
ops-monitoring-bot added projects to T226905: Degraded RAID on helium: Operations, ops-eqiad.
Sat, Jun 29, 9:38 PM · ops-eqiad, Operations
ops-monitoring-bot added projects to T226895: Degraded RAID on helium: Operations, ops-eqiad.
Sat, Jun 29, 5:16 PM · ops-eqiad, Operations
ops-monitoring-bot added projects to T226894: Degraded RAID on helium: Operations, ops-eqiad.
Sat, Jun 29, 4:55 PM · ops-eqiad, Operations
ops-monitoring-bot added projects to T226891: Degraded RAID on helium: Operations, ops-eqiad.
Sat, Jun 29, 2:40 PM · ops-eqiad, Operations
ops-monitoring-bot added projects to T226890: Degraded RAID on helium: Operations, ops-eqiad.
Sat, Jun 29, 2:08 PM · ops-eqiad, Operations
ops-monitoring-bot added projects to T226889: Degraded RAID on helium: Operations, ops-eqiad.
Sat, Jun 29, 1:37 PM · ops-eqiad, Operations
ops-monitoring-bot added projects to T226887: Degraded RAID on helium: Operations, ops-eqiad.
Sat, Jun 29, 12:13 PM · ops-eqiad, Operations
ops-monitoring-bot added projects to T226880: Degraded RAID on helium: Operations, ops-eqiad.
Sat, Jun 29, 10:39 AM · ops-eqiad, Operations
ops-monitoring-bot added projects to T226878: Degraded RAID on helium: Operations, ops-eqiad.
Sat, Jun 29, 10:18 AM · ops-eqiad, Operations

Fri, Jun 28

ops-monitoring-bot added a comment to T226637: Replace Varnish backends with ATS on cache upload nodes in codfw.

Completed auto-reimage of hosts:

['cp2011.codfw.wmnet']
Fri, Jun 28, 2:48 PM · Operations, Traffic
ops-monitoring-bot added a comment to T226637: Replace Varnish backends with ATS on cache upload nodes in codfw.

Script wmf-auto-reimage was launched by ema on cumin1001.eqiad.wmnet for hosts:

['cp2011.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201906281410_ema_30592.log.

Fri, Jun 28, 2:10 PM · Operations, Traffic
ops-monitoring-bot added a comment to T224603: rack/setup/ codfw: ganeti2009 - ganeti201[0-8].

Completed auto-reimage of hosts:

['ganeti2012.codfw.wmnet', 'ganeti2011.codfw.wmnet', 'ganeti2009.codfw.wmnet']
Fri, Jun 28, 12:16 PM · ops-codfw, Operations
ops-monitoring-bot added a comment to T226637: Replace Varnish backends with ATS on cache upload nodes in codfw.

Completed auto-reimage of hosts:

['cp2008.codfw.wmnet']
Fri, Jun 28, 9:59 AM · Operations, Traffic
ops-monitoring-bot added a comment to T224603: rack/setup/ codfw: ganeti2009 - ganeti201[0-8].

Script wmf-auto-reimage was launched by akosiaris on cumin1001.eqiad.wmnet for hosts:

['ganeti2009.codfw.wmnet', 'ganeti2010.codfw.wmnet', 'ganeti2011.codfw.wmnet', 'ganeti2012.codfw.wmnet', 'ganeti2013.codfw.wmnet', 'ganeti2014.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201906280956_akosiaris_239649.log.

Fri, Jun 28, 9:56 AM · ops-codfw, Operations
ops-monitoring-bot added a comment to T226637: Replace Varnish backends with ATS on cache upload nodes in codfw.

Script wmf-auto-reimage was launched by ema on cumin1001.eqiad.wmnet for hosts:

['cp2008.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201906280921_ema_232704.log.

Fri, Jun 28, 9:21 AM · Operations, Traffic
ops-monitoring-bot added a comment to T224603: rack/setup/ codfw: ganeti2009 - ganeti201[0-8].

Completed auto-reimage of hosts:

['ganeti2014.codfw.wmnet', 'ganeti2009.codfw.wmnet', 'ganeti2013.codfw.wmnet', 'ganeti2012.codfw.wmnet', 'ganeti2011.codfw.wmnet', 'ganeti2016.codfw.wmnet', 'ganeti2015.codfw.wmnet', 'ganeti2018.codfw.wmnet', 'ganeti2017.codfw.wmnet']
Fri, Jun 28, 8:17 AM · ops-codfw, Operations
ops-monitoring-bot added a comment to T226637: Replace Varnish backends with ATS on cache upload nodes in codfw.

Completed auto-reimage of hosts:

['cp2005.codfw.wmnet']
Fri, Jun 28, 7:48 AM · Operations, Traffic
ops-monitoring-bot added a comment to T224603: rack/setup/ codfw: ganeti2009 - ganeti201[0-8].

Script wmf-auto-reimage was launched by akosiaris on cumin1001.eqiad.wmnet for hosts:

['ganeti2009.codfw.wmnet', 'ganeti2010.codfw.wmnet', 'ganeti2011.codfw.wmnet', 'ganeti2012.codfw.wmnet', 'ganeti2013.codfw.wmnet', 'ganeti2014.codfw.wmnet', 'ganeti2015.codfw.wmnet', 'ganeti2016.codfw.wmnet', 'ganeti2017.codfw.wmnet', 'ganeti2018.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201906280716_akosiaris_206735.log.

Fri, Jun 28, 7:17 AM · ops-codfw, Operations
ops-monitoring-bot added a comment to T226637: Replace Varnish backends with ATS on cache upload nodes in codfw.

Script wmf-auto-reimage was launched by ema on cumin1001.eqiad.wmnet for hosts:

['cp2005.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201906280709_ema_205316.log.

Fri, Jun 28, 7:09 AM · Operations, Traffic

Thu, Jun 27

ops-monitoring-bot added a comment to T225988: decommission db2039.

cookbooks.sre.hosts.decommission executed by robh@cumin1001 for hosts: db2039.codfw.wmnet

  • db2039.codfw.wmnet
    • Removed from Puppet master and PuppetDB
    • Downtimed host on Icinga
    • Downtimed management interface on Icinga
    • Removed from DebMonitor
Thu, Jun 27, 6:05 PM · Patch-For-Review, DC-Ops, ops-codfw, Operations, decommission
ops-monitoring-bot added a comment to T224603: rack/setup/ codfw: ganeti2009 - ganeti201[0-8].

Completed auto-reimage of hosts:

['ganeti2013.codfw.wmnet', 'ganeti2014.codfw.wmnet', 'ganeti2012.codfw.wmnet', 'ganeti2010.codfw.wmnet', 'ganeti2009.codfw.wmnet', 'ganeti2011.codfw.wmnet', 'ganeti2016.codfw.wmnet', 'ganeti2015.codfw.wmnet', 'ganeti2017.codfw.wmnet', 'ganeti2018.codfw.wmnet']
Thu, Jun 27, 5:37 PM · ops-codfw, Operations
ops-monitoring-bot added a comment to T224603: rack/setup/ codfw: ganeti2009 - ganeti201[0-8].

Script wmf-auto-reimage was launched by akosiaris on cumin1001.eqiad.wmnet for hosts:

['ganeti2009.codfw.wmnet', 'ganeti2010.codfw.wmnet', 'ganeti2011.codfw.wmnet', 'ganeti2012.codfw.wmnet', 'ganeti2013.codfw.wmnet', 'ganeti2014.codfw.wmnet', 'ganeti2015.codfw.wmnet', 'ganeti2016.codfw.wmnet', 'ganeti2017.codfw.wmnet', 'ganeti2018.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201906271636_akosiaris_35762.log.

Thu, Jun 27, 4:37 PM · ops-codfw, Operations
ops-monitoring-bot added a comment to T226637: Replace Varnish backends with ATS on cache upload nodes in codfw.

Completed auto-reimage of hosts:

['cp2002.codfw.wmnet']
Thu, Jun 27, 4:16 PM · Operations, Traffic
ops-monitoring-bot added a comment to T226637: Replace Varnish backends with ATS on cache upload nodes in codfw.

Script wmf-auto-reimage was launched by ema on cumin1001.eqiad.wmnet for hosts:

['cp2002.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201906271512_ema_17312.log.

Thu, Jun 27, 3:12 PM · Operations, Traffic
ops-monitoring-bot added a comment to T226637: Replace Varnish backends with ATS on cache upload nodes in codfw.

Script wmf-auto-reimage was launched by ema on cumin1001.eqiad.wmnet for hosts:

['cp2002.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201906271447_ema_11715.log.

Thu, Jun 27, 2:48 PM · Operations, Traffic
ops-monitoring-bot added a comment to T226637: Replace Varnish backends with ATS on cache upload nodes in codfw.

Completed auto-reimage of hosts:

['cp2002.codfw.wmnet']
Thu, Jun 27, 2:47 PM · Operations, Traffic
ops-monitoring-bot added a comment to T226637: Replace Varnish backends with ATS on cache upload nodes in codfw.

Script wmf-auto-reimage was launched by ema on cumin1001.eqiad.wmnet for hosts:

['cp2002.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201906271446_ema_11539.log.

Thu, Jun 27, 2:47 PM · Operations, Traffic
ops-monitoring-bot added a comment to T226637: Replace Varnish backends with ATS on cache upload nodes in codfw.

Script wmf-auto-reimage was launched by ema on cumin1001.eqiad.wmnet for hosts:

['cp2002.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201906271417_ema_6093.log.

Thu, Jun 27, 2:17 PM · Operations, Traffic

Wed, Jun 26

ops-monitoring-bot added a comment to T226477: Replace Varnish backends with ATS on cache upload nodes in eqsin.

Completed auto-reimage of hosts:

['cp5006.eqsin.wmnet']
Wed, Jun 26, 4:02 PM · Operations, Traffic