Maniphest T385472

PuppetFailure
Closed, DuplicatePublic
Actions

Assigned To

None

Authored By

	phaultfinder
	Feb 3 2025, 1:34 PM

Tags

Referenced Files

None

Subscribers

Description

Common information

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
runbook: https://puppetboard.wikimedia.org/nodes?status=failed

alertname: PuppetFailure
cluster: wmcs
job: node
prometheus: ops
severity: critical
source: prometheus
team: wmcs

Firing alerts

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on cloudbackup1002-dev:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on cloudbackup1002-dev:9100
alertname: PuppetFailure
cluster: wmcs
instance: cloudbackup1002-dev:9100
job: node
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on cloudbackup2003:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on cloudbackup2003:9100
alertname: PuppetFailure
cluster: wmcs
instance: cloudbackup2003:9100
job: node
prometheus: ops
severity: critical
site: codfw
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on cloudbackup2004:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on cloudbackup2004:9100
alertname: PuppetFailure
cluster: wmcs
instance: cloudbackup2004:9100
job: node
prometheus: ops
severity: critical
site: codfw
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on cloudcephmon1005:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on cloudcephmon1005:9100
alertname: PuppetFailure
cluster: wmcs
instance: cloudcephmon1005:9100
job: node
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on cloudcephmon1006:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on cloudcephmon1006:9100
alertname: PuppetFailure
cluster: wmcs
instance: cloudcephmon1006:9100
job: node
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on cloudcephmon2004-dev:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on cloudcephmon2004-dev:9100
alertname: PuppetFailure
cluster: wmcs
instance: cloudcephmon2004-dev:9100
job: node
prometheus: ops
severity: critical
site: codfw
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on cloudcephmon2005-dev:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on cloudcephmon2005-dev:9100
alertname: PuppetFailure
cluster: wmcs
instance: cloudcephmon2005-dev:9100
job: node
prometheus: ops
severity: critical
site: codfw
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on cloudcephosd1001:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on cloudcephosd1001:9100
alertname: PuppetFailure
cluster: wmcs
instance: cloudcephosd1001:9100
job: node
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on cloudcephosd1002:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on cloudcephosd1002:9100
alertname: PuppetFailure
cluster: wmcs
instance: cloudcephosd1002:9100
job: node
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on cloudcephosd1003:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on cloudcephosd1003:9100
alertname: PuppetFailure
cluster: wmcs
instance: cloudcephosd1003:9100
job: node
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on cloudcephosd1004:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on cloudcephosd1004:9100
alertname: PuppetFailure
cluster: wmcs
instance: cloudcephosd1004:9100
job: node
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on cloudcephosd1006:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on cloudcephosd1006:9100
alertname: PuppetFailure
cluster: wmcs
instance: cloudcephosd1006:9100
job: node
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on cloudcephosd1009:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on cloudcephosd1009:9100
alertname: PuppetFailure
cluster: wmcs
instance: cloudcephosd1009:9100
job: node
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on cloudcephosd1010:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on cloudcephosd1010:9100
alertname: PuppetFailure
cluster: wmcs
instance: cloudcephosd1010:9100
job: node
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on cloudcephosd1011:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on cloudcephosd1011:9100
alertname: PuppetFailure
cluster: wmcs
instance: cloudcephosd1011:9100
job: node
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on cloudcephosd1012:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on cloudcephosd1012:9100
alertname: PuppetFailure
cluster: wmcs
instance: cloudcephosd1012:9100
job: node
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on cloudcephosd1013:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on cloudcephosd1013:9100
alertname: PuppetFailure
cluster: wmcs
instance: cloudcephosd1013:9100
job: node
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on cloudcephosd1015:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on cloudcephosd1015:9100
alertname: PuppetFailure
cluster: wmcs
instance: cloudcephosd1015:9100
job: node
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on cloudcephosd1016:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on cloudcephosd1016:9100
alertname: PuppetFailure
cluster: wmcs
instance: cloudcephosd1016:9100
job: node
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on cloudcephosd1018:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on cloudcephosd1018:9100
alertname: PuppetFailure
cluster: wmcs
instance: cloudcephosd1018:9100
job: node
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on cloudcephosd1019:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on cloudcephosd1019:9100
alertname: PuppetFailure
cluster: wmcs
instance: cloudcephosd1019:9100
job: node
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on cloudcephosd1020:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on cloudcephosd1020:9100
alertname: PuppetFailure
cluster: wmcs
instance: cloudcephosd1020:9100
job: node
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on cloudcephosd1021:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on cloudcephosd1021:9100
alertname: PuppetFailure
cluster: wmcs
instance: cloudcephosd1021:9100
job: node
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on cloudcephosd1022:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on cloudcephosd1022:9100
alertname: PuppetFailure
cluster: wmcs
instance: cloudcephosd1022:9100
job: node
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on cloudcephosd1024:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on cloudcephosd1024:9100
alertname: PuppetFailure
cluster: wmcs
instance: cloudcephosd1024:9100
job: node
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on cloudcephosd1026:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on cloudcephosd1026:9100
alertname: PuppetFailure
cluster: wmcs
instance: cloudcephosd1026:9100
job: node
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on cloudcephosd1027:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on cloudcephosd1027:9100
alertname: PuppetFailure
cluster: wmcs
instance: cloudcephosd1027:9100
job: node
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on cloudcephosd1029:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on cloudcephosd1029:9100
alertname: PuppetFailure
cluster: wmcs
instance: cloudcephosd1029:9100
job: node
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on cloudcephosd1031:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on cloudcephosd1031:9100
alertname: PuppetFailure
cluster: wmcs
instance: cloudcephosd1031:9100
job: node
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on cloudcephosd1033:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on cloudcephosd1033:9100
alertname: PuppetFailure
cluster: wmcs
instance: cloudcephosd1033:9100
job: node
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on cloudcephosd1034:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on cloudcephosd1034:9100
alertname: PuppetFailure
cluster: wmcs
instance: cloudcephosd1034:9100
job: node
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on cloudcephosd1035:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on cloudcephosd1035:9100
alertname: PuppetFailure
cluster: wmcs
instance: cloudcephosd1035:9100
job: node
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on cloudcephosd1036:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on cloudcephosd1036:9100
alertname: PuppetFailure
cluster: wmcs
instance: cloudcephosd1036:9100
job: node
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on cloudcephosd1037:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on cloudcephosd1037:9100
alertname: PuppetFailure
cluster: wmcs
instance: cloudcephosd1037:9100
job: node
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on cloudcephosd1038:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on cloudcephosd1038:9100
alertname: PuppetFailure
cluster: wmcs
instance: cloudcephosd1038:9100
job: node
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on cloudcephosd1039:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on cloudcephosd1039:9100
alertname: PuppetFailure
cluster: wmcs
instance: cloudcephosd1039:9100
job: node
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on cloudcephosd1040:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on cloudcephosd1040:9100
alertname: PuppetFailure
cluster: wmcs
instance: cloudcephosd1040:9100
job: node
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on cloudcephosd1041:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on cloudcephosd1041:9100
alertname: PuppetFailure
cluster: wmcs
instance: cloudcephosd1041:9100
job: node
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on cloudcephosd2001-dev:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on cloudcephosd2001-dev:9100
alertname: PuppetFailure
cluster: wmcs
instance: cloudcephosd2001-dev:9100
job: node
prometheus: ops
severity: critical
site: codfw
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on cloudcephosd2002-dev:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on cloudcephosd2002-dev:9100
alertname: PuppetFailure
cluster: wmcs
instance: cloudcephosd2002-dev:9100
job: node
prometheus: ops
severity: critical
site: codfw
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on cloudcephosd2003-dev:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on cloudcephosd2003-dev:9100
alertname: PuppetFailure
cluster: wmcs
instance: cloudcephosd2003-dev:9100
job: node
prometheus: ops
severity: critical
site: codfw
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on cloudcontrol1005:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on cloudcontrol1005:9100
alertname: PuppetFailure
cluster: wmcs
instance: cloudcontrol1005:9100
job: node
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on cloudcontrol1007:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on cloudcontrol1007:9100
alertname: PuppetFailure
cluster: wmcs
instance: cloudcontrol1007:9100
job: node
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on cloudcontrol2004-dev:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on cloudcontrol2004-dev:9100
alertname: PuppetFailure
cluster: wmcs
instance: cloudcontrol2004-dev:9100
job: node
prometheus: ops
severity: critical
site: codfw
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on cloudcontrol2005-dev:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on cloudcontrol2005-dev:9100
alertname: PuppetFailure
cluster: wmcs
instance: cloudcontrol2005-dev:9100
job: node
prometheus: ops
severity: critical
site: codfw
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on cloudcontrol2006-dev:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on cloudcontrol2006-dev:9100
alertname: PuppetFailure
cluster: wmcs
instance: cloudcontrol2006-dev:9100
job: node
prometheus: ops
severity: critical
site: codfw
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on cloudgw1001:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on cloudgw1001:9100
alertname: PuppetFailure
cluster: wmcs
instance: cloudgw1001:9100
job: node
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on cloudgw1002:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on cloudgw1002:9100
alertname: PuppetFailure
cluster: wmcs
instance: cloudgw1002:9100
job: node
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on cloudlb1002:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on cloudlb1002:9100
alertname: PuppetFailure
cluster: wmcs
instance: cloudlb1002:9100
job: node
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on cloudlb2001-dev:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on cloudlb2001-dev:9100
alertname: PuppetFailure
cluster: wmcs
instance: cloudlb2001-dev:9100
job: node
prometheus: ops
severity: critical
site: codfw
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on cloudlb2002-dev:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on cloudlb2002-dev:9100
alertname: PuppetFailure
cluster: wmcs
instance: cloudlb2002-dev:9100
job: node
prometheus: ops
severity: critical
site: codfw
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on cloudlb2003-dev:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on cloudlb2003-dev:9100
alertname: PuppetFailure
cluster: wmcs
instance: cloudlb2003-dev:9100
job: node
prometheus: ops
severity: critical
site: codfw
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on cloudnet1006:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on cloudnet1006:9100
alertname: PuppetFailure
cluster: wmcs
instance: cloudnet1006:9100
job: node
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on cloudnet2006-dev:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on cloudnet2006-dev:9100
alertname: PuppetFailure
cluster: wmcs
instance: cloudnet2006-dev:9100
job: node
prometheus: ops
severity: critical
site: codfw
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on cloudrabbit1001:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on cloudrabbit1001:9100
alertname: PuppetFailure
cluster: wmcs
instance: cloudrabbit1001:9100
job: node
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on cloudrabbit1002:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on cloudrabbit1002:9100
alertname: PuppetFailure
cluster: wmcs
instance: cloudrabbit1002:9100
job: node
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on cloudrabbit1003:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on cloudrabbit1003:9100
alertname: PuppetFailure
cluster: wmcs
instance: cloudrabbit1003:9100
job: node
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on cloudservices1005:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on cloudservices1005:9100
alertname: PuppetFailure
cluster: wmcs
instance: cloudservices1005:9100
job: node
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on cloudservices2004-dev:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on cloudservices2004-dev:9100
alertname: PuppetFailure
cluster: wmcs
instance: cloudservices2004-dev:9100
job: node
prometheus: ops
severity: critical
site: codfw
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on cloudvirt1031:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on cloudvirt1031:9100
alertname: PuppetFailure
cluster: wmcs
instance: cloudvirt1031:9100
job: node
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on cloudvirt1032:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on cloudvirt1032:9100
alertname: PuppetFailure
cluster: wmcs
instance: cloudvirt1032:9100
job: node
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on cloudvirt1033:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on cloudvirt1033:9100
alertname: PuppetFailure
cluster: wmcs
instance: cloudvirt1033:9100
job: node
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on cloudvirt1034:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on cloudvirt1034:9100
alertname: PuppetFailure
cluster: wmcs
instance: cloudvirt1034:9100
job: node
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on cloudvirt1035:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on cloudvirt1035:9100
alertname: PuppetFailure
cluster: wmcs
instance: cloudvirt1035:9100
job: node
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on cloudvirt1036:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on cloudvirt1036:9100
alertname: PuppetFailure
cluster: wmcs
instance: cloudvirt1036:9100
job: node
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on cloudvirt1038:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on cloudvirt1038:9100
alertname: PuppetFailure
cluster: wmcs
instance: cloudvirt1038:9100
job: node
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on cloudvirt1039:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on cloudvirt1039:9100
alertname: PuppetFailure
cluster: wmcs
instance: cloudvirt1039:9100
job: node
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on cloudvirt1040:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on cloudvirt1040:9100
alertname: PuppetFailure
cluster: wmcs
instance: cloudvirt1040:9100
job: node
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on cloudvirt1042:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on cloudvirt1042:9100
alertname: PuppetFailure
cluster: wmcs
instance: cloudvirt1042:9100
job: node
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on cloudvirt1043:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on cloudvirt1043:9100
alertname: PuppetFailure
cluster: wmcs
instance: cloudvirt1043:9100
job: node
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on cloudvirt1044:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on cloudvirt1044:9100
alertname: PuppetFailure
cluster: wmcs
instance: cloudvirt1044:9100
job: node
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on cloudvirt1046:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on cloudvirt1046:9100
alertname: PuppetFailure
cluster: wmcs
instance: cloudvirt1046:9100
job: node
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on cloudvirt1047:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on cloudvirt1047:9100
alertname: PuppetFailure
cluster: wmcs
instance: cloudvirt1047:9100
job: node
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on cloudvirt1048:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on cloudvirt1048:9100
alertname: PuppetFailure
cluster: wmcs
instance: cloudvirt1048:9100
job: node
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on cloudvirt1049:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on cloudvirt1049:9100
alertname: PuppetFailure
cluster: wmcs
instance: cloudvirt1049:9100
job: node
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on cloudvirt1050:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on cloudvirt1050:9100
alertname: PuppetFailure
cluster: wmcs
instance: cloudvirt1050:9100
job: node
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on cloudvirt1051:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on cloudvirt1051:9100
alertname: PuppetFailure
cluster: wmcs
instance: cloudvirt1051:9100
job: node
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on cloudvirt1054:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on cloudvirt1054:9100
alertname: PuppetFailure
cluster: wmcs
instance: cloudvirt1054:9100
job: node
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on cloudvirt1055:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on cloudvirt1055:9100
alertname: PuppetFailure
cluster: wmcs
instance: cloudvirt1055:9100
job: node
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on cloudvirt1056:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on cloudvirt1056:9100
alertname: PuppetFailure
cluster: wmcs
instance: cloudvirt1056:9100
job: node
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on cloudvirt1058:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on cloudvirt1058:9100
alertname: PuppetFailure
cluster: wmcs
instance: cloudvirt1058:9100
job: node
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on cloudvirt1059:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on cloudvirt1059:9100
alertname: PuppetFailure
cluster: wmcs
instance: cloudvirt1059:9100
job: node
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on cloudvirt1060:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on cloudvirt1060:9100
alertname: PuppetFailure
cluster: wmcs
instance: cloudvirt1060:9100
job: node
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on cloudvirt1063:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on cloudvirt1063:9100
alertname: PuppetFailure
cluster: wmcs
instance: cloudvirt1063:9100
job: node
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on cloudvirt1065:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on cloudvirt1065:9100
alertname: PuppetFailure
cluster: wmcs
instance: cloudvirt1065:9100
job: node
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on cloudvirt1066:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on cloudvirt1066:9100
alertname: PuppetFailure
cluster: wmcs
instance: cloudvirt1066:9100
job: node
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on cloudvirt1067:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on cloudvirt1067:9100
alertname: PuppetFailure
cluster: wmcs
instance: cloudvirt1067:9100
job: node
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on cloudvirt2004-dev:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on cloudvirt2004-dev:9100
alertname: PuppetFailure
cluster: wmcs
instance: cloudvirt2004-dev:9100
job: node
prometheus: ops
severity: critical
site: codfw
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on cloudvirt2006-dev:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on cloudvirt2006-dev:9100
alertname: PuppetFailure
cluster: wmcs
instance: cloudvirt2006-dev:9100
job: node
prometheus: ops
severity: critical
site: codfw
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on cloudvirtlocal1002:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on cloudvirtlocal1002:9100
alertname: PuppetFailure
cluster: wmcs
instance: cloudvirtlocal1002:9100
job: node
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on cloudvirtlocal1003:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on cloudvirtlocal1003:9100
alertname: PuppetFailure
cluster: wmcs
instance: cloudvirtlocal1003:9100
job: node
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on cloudweb1003:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on cloudweb1003:9100
alertname: PuppetFailure
cluster: wmcs
instance: cloudweb1003:9100
job: node
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: wmcs
Source

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on cloudweb2002-dev:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on cloudweb2002-dev:9100
alertname: PuppetFailure
cluster: wmcs
instance: cloudweb2002-dev:9100
job: node
prometheus: ops
severity: critical
site: codfw
source: prometheus
team: wmcs
Source

Event Timeline

phaultfinder created this task.Feb 3 2025, 1:34 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 3 2025, 1:34 PM

• aborrero closed this task as a duplicate of T380960: kernel error detector: have a way to ignore certain messages.Feb 3 2025, 1:35 PM