Deployment-prep hosts with puppet errors (tracking)
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Krenair
	Apr 9 2016, 6:51 PM

Description

This is a Tracking-Neverending task for any puppet errors we might encounter on the beta cluster infrastructure. Puppet errors typically occurs when Puppet changes are made that are not necessarily taking in account beta (typically, class being renamed but not updated in wikitech, hiera parameters missing ...).

Related Objects
Search...

Status	Assigned	Task
Resolved	EddieGP	T132259 Deployment-prep hosts with puppet errors (tracking)
Resolved	ArielGlenn	T117028 deployment-fluorine puppet failure due to '/usr/sbin/usermod -u 10003 datasets' returned 4: usermod: UID '10003' already exists
Resolved	Krenair	T132260 deployment-cache-parsoid05 puppet failures due to removal of role::cache::parsoid
Resolved	None	T132262 deployment-memc0[2-4] puppet failure due to missing hiera data mediawiki::redis_servers::codfw
Resolved	None	T132263 deployment-redis0[12], deployment-sentry2 puppet failures due to missing hiera data redis::shards
Resolved	• mobrovac	T132265 deployment-(mathoid\|sca0[12]\|cxserver03) puppet failures "Error: Could not set home on user" due to user being in use by process
Resolved	• mmodell	T132269 deployment-puppetmaster puppet failures due to apache trying to start on same port as nginx
Resolved	• mmodell	T132689 /etc/puppet/puppet.conf keeps getting double content - first for labs-wide puppetmaster, then for the correct puppetmaster
Resolved	hashar	T133989 deployment-puppetmaster puppet fails due to "Could not render to Puppet::Network::Format[msgpack]: undefined method `to_msgpack' for #<Puppet::Util::TagSet:0x007f56e728de68>"
Declined	None	T134226 deployment-parsoid06 puppet fails due to having role::parsoid::beta (requiring upstart) on jessie
Invalid	• mmodell	T134277 puppet failure on deployment-phab due to missing packages php5-mailparse and python-phabricator
Resolved	Eevans	T134630 deployment-restbase01 puppet fails due to issues with cassandra package
Duplicate	None	T134631 /etc/puppet/puppet.conf sometimes gets the old puppetmaster FQDN and breaks puppet
Resolved	hashar	T134808 Puppet fails on labs instances due to Ganglia (ex: using apache::site puppet class)
Resolved	Krenair	T136077 deployment-cache-upload04/text04 Could not find data item cache::cluster in hiera no default supplied at /etc/puppet/modules/role/manifests/cache/base.pp:17
Duplicate	None	T136080 Hiera hierarchy hieradata/role/* is not applied on labs (eg deployment-prep)
Declined	None	T120165 Implement role based hiera lookups for labs
Resolved	bd808	T143065 deployment-sca0[12] puppet failure due to issues involving /srv/deployment directory
Resolved	thcipriani	T143371 puppet on deployment-changeprop taking forever because of systemctl start salt-minion
Resolved	None	T116206 Set up AQS in Beta
Resolved	None	T114999 Deploy AQS with scap3
Resolved	None	T120841 Create fake data for beta AQS deployment
Resolved	None	T126294 Separate AQS off of RESTBase
Resolved	Ottomata	T127720 Allow aqs-admins to deploy via scap using deploy-service ssh key
Resolved	Ladsgroup	T132267 deployment-((sca\|aqs)01\|ores-web) puppet failures due to scap3 errors
Resolved	Eevans	T137706 Puppet errors on deploment-aqs01 because E: Version '2.2.6' for 'cassandra' was not found
Resolved	• AlexMonk-WMF	T144703 Puppet failing on deployment-conf03 due to missing files
Resolved	akosiaris	T147210 deployment-apertium01 puppet failing due to missing packages on trusty
Resolved	hashar	T147818 puppet failure on deployment-phab01: Service[ssh-phab] refuses to start
Resolved	hashar	T151169 deployment-fluorine02 puppet broken
Resolved	Ottomata	T152842 deployment-eventlogging03 has puppet failure due to missing class
Resolved	Krenair	T173552 Puppet broken on deployment-pdf01
Resolved	Krenair	T173554 Puppet broken on deployment-sentry01
Declined	hashar	T180935 Various puppet issues in deployment-prep
Resolved	• mmodell	T184233 deployment-phab completely broken
Resolved	None	T184234 Puppet broken on deployment-cache-text04 due to varnishkafka issues
Resolved	Ottomata	T184235 Puppet broken on deployment-kafka03 due to full disk
Resolved	Krenair	T184236 Puppet broken on deployment-ms-be0[34] with evaluation error in swift module
Resolved	Ottomata	T184238 Puppet broken on deployment-eventlogging04 due to missing directory '/var/lib/superset'?
Resolved	fgiunchedi	T184239 Puppet broken on deployment-mediawiki07, deployment-imagescaler02, deployment-redis06, deployment-videoscaler01 due to prometheus exporter packages being missing in stretch
Resolved	Krenair	T184240 Puppet broken on deployment-kafka-jumbo-[12] due to version of a package being missing
Resolved	• mobrovac	T184241 Puppet broken on deployment-trending01 due to removal of role
Resolved	ayounsi	T184242 Puppet broken on deployment-netbox, looks like it thinks its a prod box
Duplicate	None	T184243 Puppet broken on deployment-redis0[12] due to systemd on trusty
Resolved	Krenair	T184244 Puppet broken on deployment-mx due to systemd on trusty
Resolved	MoritzMuehlenhoff	T153468 Ferm's upstream Net::DNS Perl library questionable handling of NOERROR responses without records causing puppet errors when we try to @resolve AAAA in labs
Resolved	fgiunchedi	T179371 Move deployment-prep redis instances to stretch
Resolved	• mobrovac	T184477 Puppet disabled for a month on deployment-restbase0[12] instances
Resolved	Ladsgroup	T184478 Puppet broken on deployment-ores01 due to missing hieradata
Resolved	EddieGP	T190632 Puppet errors on deployment-mediawiki07
Resolved	MarcoAurelio	T188921 Puppet broken on deployment-logstash2.deployment-prep.eqiad.wmflabs
Resolved	Krenair	T187736 Host deployment-puppetdb01 is DOWN: CRITICAL - Host Unreachable (10.68.23.76)
Duplicate	None	T190683 Puppet broken on deployment-ms-be03
Resolved	None	T191107 deployment-etcd-01 puppet errors
Resolved	elukey	T191109 deployment-eventlog05 puppet error about missing mysql heartbeat.heartbeat table
Resolved	None	T191110 Puppet broken on deployment-mira

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Krenair closed subtask T132267: deployment-((sca|aqs)01|ores-web) puppet failures due to scap3 errors as Resolved.Apr 26 2016, 10:25 PM

Krenair closed subtask T132260: deployment-cache-parsoid05 puppet failures due to removal of role::cache::parsoid as Resolved.Apr 27 2016, 12:16 AM

Krenair created subtask T133989: deployment-puppetmaster puppet fails due to "Could not render to Puppet::Network::Format[msgpack]: undefined method `to_msgpack' for #<Puppet::Util::TagSet:0x007f56e728de68>".Apr 29 2016, 3:36 PM

hashar closed subtask T133989: deployment-puppetmaster puppet fails due to "Could not render to Puppet::Network::Format[msgpack]: undefined method `to_msgpack' for #<Puppet::Util::TagSet:0x007f56e728de68>" as Resolved.Apr 29 2016, 8:57 PM

Krenair created subtask T134226: deployment-parsoid06 puppet fails due to having role::parsoid::beta (requiring upstart) on jessie.May 3 2016, 1:54 AM

Krenair created subtask T134277: puppet failure on deployment-phab due to missing packages php5-mailparse and python-phabricator.May 3 2016, 4:08 PM

• mmodell closed subtask T134277: puppet failure on deployment-phab due to missing packages php5-mailparse and python-phabricator as Invalid.May 6 2016, 9:14 PM

Krenair created subtask T134630: deployment-restbase01 puppet fails due to issues with cassandra package.May 7 2016, 3:51 AM

Krenair created subtask T134631: /etc/puppet/puppet.conf sometimes gets the old puppetmaster FQDN and breaks puppet.May 7 2016, 4:15 AM

• mobrovac closed subtask T134630: deployment-restbase01 puppet fails due to issues with cassandra package as Resolved.May 9 2016, 2:00 PM

Puppet failures related to Ganglia would be due to T134808 which have fixed cherry picked on beta puppet master but do not cover every cases.

hashar closed subtask T134808: Puppet fails on labs instances due to Ganglia (ex: using apache::site puppet class) as Resolved.May 10 2016, 7:50 PM

Krenair added a subtask: T136077: deployment-cache-upload04/text04 Could not find data item cache::cluster in hiera no default supplied at /etc/puppet/modules/role/manifests/cache/base.pp:17.May 27 2016, 5:47 PM

Krenair closed subtask T136077: deployment-cache-upload04/text04 Could not find data item cache::cluster in hiera no default supplied at /etc/puppet/modules/role/manifests/cache/base.pp:17 as Resolved.Jun 13 2016, 12:54 PM

Krenair created subtask T137706: Puppet errors on deploment-aqs01 because E: Version '2.2.6' for 'cassandra' was not found.Jun 13 2016, 1:06 PM

Krenair closed subtask T137706: Puppet errors on deploment-aqs01 because E: Version '2.2.6' for 'cassandra' was not found as Resolved.Jun 13 2016, 9:02 PM

hashar triaged this task as Medium priority.Jun 30 2016, 10:33 AM

ssastry closed subtask T134226: deployment-parsoid06 puppet fails due to having role::parsoid::beta (requiring upstart) on jessie as Declined.Jul 15 2016, 9:37 PM

greg moved this task from Next: Maintenance to Epics / Tracking on the Beta-Cluster-Infrastructure board.Aug 5 2016, 8:49 PM

• AlexMonk-WMF created subtask T143065: deployment-sca0[12] puppet failure due to issues involving /srv/deployment directory.Aug 15 2016, 11:00 PM

• AlexMonk-WMF created subtask T143371: puppet on deployment-changeprop taking forever because of systemctl start salt-minion.Aug 19 2016, 12:04 AM

thcipriani closed subtask T143371: puppet on deployment-changeprop taking forever because of systemctl start salt-minion as Resolved.Aug 19 2016, 6:52 PM

hashar closed subtask T132689: /etc/puppet/puppet.conf keeps getting double content - first for labs-wide puppetmaster, then for the correct puppetmaster as Resolved.Aug 25 2016, 8:12 AM

bd808 added a subtask: T116206: Set up AQS in Beta.Aug 30 2016, 7:15 PM

• AlexMonk-WMF closed subtask T143065: deployment-sca0[12] puppet failure due to issues involving /srv/deployment directory as Resolved.Aug 31 2016, 10:04 PM

• AlexMonk-WMF created subtask T144703: Puppet failing on deployment-conf03 due to missing files.Sep 4 2016, 9:27 PM

hashar updated the task description. (Show Details)Sep 12 2016, 8:48 AM

• AlexMonk-WMF closed subtask T144703: Puppet failing on deployment-conf03 due to missing files as Resolved.Sep 20 2016, 4:58 PM

• AlexMonk-WMF created subtask T147210: deployment-apertium01 puppet failing due to missing packages on trusty.Oct 3 2016, 5:21 PM

akosiaris closed subtask T147210: deployment-apertium01 puppet failing due to missing packages on trusty as Resolved.Oct 4 2016, 9:04 AM

• AlexMonk-WMF created subtask T147818: puppet failure on deployment-phab01: Service[ssh-phab] refuses to start.Oct 10 2016, 9:44 PM

hashar reopened subtask T147210: deployment-apertium01 puppet failing due to missing packages on trusty as Open.Nov 9 2016, 10:37 AM

akosiaris closed subtask T147210: deployment-apertium01 puppet failing due to missing packages on trusty as Resolved.Nov 9 2016, 11:12 AM

hashar reopened subtask T147210: deployment-apertium01 puppet failing due to missing packages on trusty as Open.Nov 17 2016, 11:23 AM

hashar closed subtask T147210: deployment-apertium01 puppet failing due to missing packages on trusty as Resolved.Nov 17 2016, 11:26 AM

Krenair added a subtask: T151169: deployment-fluorine02 puppet broken.Nov 20 2016, 10:04 PM

hashar closed subtask T151169: deployment-fluorine02 puppet broken as Resolved.Nov 22 2016, 2:33 PM

Krenair created subtask T152842: deployment-eventlogging03 has puppet failure due to missing class.Dec 10 2016, 4:41 PM

Krenair closed subtask T152842: deployment-eventlogging03 has puppet failure due to missing class as Resolved.Jan 19 2017, 3:42 AM

• mmodell closed subtask T147818: puppet failure on deployment-phab01: Service[ssh-phab] refuses to start as Resolved.May 8 2017, 11:11 PM

• Nuria closed subtask T116206: Set up AQS in Beta as Resolved.Jul 12 2017, 7:49 PM

Krenair created subtask T173552: Puppet broken on deployment-pdf01.Aug 18 2017, 1:07 AM

Krenair created subtask T173554: Puppet broken on deployment-sentry01.Aug 18 2017, 1:15 AM

Krenair closed subtask T173552: Puppet broken on deployment-pdf01 as Resolved.Aug 19 2017, 3:38 PM

hashar added a subtask: T180935: Various puppet issues in deployment-prep.Nov 20 2017, 9:27 AM

greg mentioned this in T180935: Various puppet issues in deployment-prep.Dec 18 2017, 5:56 PM

hashar closed subtask T180935: Various puppet issues in deployment-prep as Declined.Jan 3 2018, 7:46 PM

• mobrovac closed subtask T184241: Puppet broken on deployment-trending01 due to removal of role as Resolved.Jan 5 2018, 9:17 AM

• mmodell closed subtask T184233: deployment-phab completely broken as Resolved.Jan 5 2018, 8:43 PM

Krenair added a subtask: T153468: Ferm's upstream Net::DNS Perl library questionable handling of NOERROR responses without records causing puppet errors when we try to @resolve AAAA in labs.Jan 5 2018, 11:21 PM

Is this really best as a tracking task or should we add it to the deployment-prep workboard column? The task by its nature is always gonna be open (or reopened).

It's fine with me if you want to move them all to a particular workboard column instead of a tracking task

hashar added a subtask: T179371: Move deployment-prep redis instances to stretch.Jan 6 2018, 9:20 AM

hashar closed subtask T184239: Puppet broken on deployment-mediawiki07, deployment-imagescaler02, deployment-redis06, deployment-videoscaler01 due to prometheus exporter packages being missing in stretch as Resolved.Jan 8 2018, 11:12 AM

Krenair added a subtask: T184477: Puppet disabled for a month on deployment-restbase0[12] instances.Jan 8 2018, 9:16 PM

-snapshot01 is T184270 (package it wants is missing from stretch, moritz to fix when higher priority things are done)

• mobrovac closed subtask T184477: Puppet disabled for a month on deployment-restbase0[12] instances as Resolved.Jan 9 2018, 9:26 AM

Krenair closed subtask T184238: Puppet broken on deployment-eventlogging04 due to missing directory '/var/lib/superset'? as Resolved.Jan 9 2018, 9:56 PM

Ottomata closed subtask T184235: Puppet broken on deployment-kafka03 due to full disk as Resolved.Jan 18 2018, 4:08 PM

ayounsi closed subtask T184242: Puppet broken on deployment-netbox, looks like it thinks its a prod box as Resolved.Jan 19 2018, 9:24 PM

Krenair closed subtask T184240: Puppet broken on deployment-kafka-jumbo-[12] due to version of a package being missing as Resolved.Jan 19 2018, 9:36 PM

Krenair closed subtask T173554: Puppet broken on deployment-sentry01 as Resolved.Jan 19 2018, 9:38 PM

Ladsgroup closed subtask T184478: Puppet broken on deployment-ores01 due to missing hieradata as Resolved.Jan 20 2018, 1:07 PM

As of today there are 16 shinken alerts (most puppet but at least one disk warning) on this project, and three VMs that are shut down but not deleted. All of this is viewable here: http://shinken.wmflabs.org/problems?search=deployment

MarcoAurelio added a subtask: T190632: Puppet errors on deployment-mediawiki07.Mar 24 2018, 9:26 PM

MarcoAurelio added a subtask: T188921: Puppet broken on deployment-logstash2.deployment-prep.eqiad.wmflabs.

MarcoAurelio added a subtask: T187736: Host deployment-puppetdb01 is DOWN: CRITICAL - Host Unreachable (10.68.23.76).

deployment-videoscaler01 seems no longer exist?

$ ssh -a deployment-videoscaler01
channel 0: open failed: connect failed: No route to host / stdio forwarding failed
ssh_exchange_identification: Connection closed by remote host

@Andrew et al. Some docs on Wikitech on usual puppet errors and how to fix them would IMHO help. I feel some of us who has access to deployment-prep could help if we had some guidance. Also, IRC assistance would be great, if at all possible. Thanks.

hashar unsubscribed.Mar 24 2018, 10:57 PM

MarcoAurelio closed subtask T188921: Puppet broken on deployment-logstash2.deployment-prep.eqiad.wmflabs as Resolved.Mar 26 2018, 1:47 PM

In T132259#3879429, demon wrote:

Is this really best as a tracking task or should we add it to the deployment-prep workboard column? The task by its nature is always gonna be open (or reopened).