Puppet failure on deployment-memc08.deployment-prep.eqiad.wmflabs
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	hashar
	Nov 6 2020, 8:58 AM

Description

deployment-memc08 fails puppet. Apparently the instance got created a year ago as part of T213089: Upgrade memcached cluster to Debian Buster

The prompt shows:

The last Puppet run was at Fri Oct 30 18:00:44 UTC 2020 (9535 minutes ago).

/var/log/puppet.log

Using configured environment 'production'
Retrieving pluginfacts
Retrieving plugin
Retrieving locales
Loading facts
Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Resource Statement, Evaluation Error: Error while evaluating a Function Call, Class[Profile::Memcached::Instance]:
  parameter 'size' expects an Integer value, got String
  parameter 'growth_factor' expects a Float value, got String (file: /etc/puppet/modules/role/manifests/memcached.pp, line: 9, column: 5) on node deployment-memc08.deployment-prep.eqiad.wmflabs
Not using cache on failed catalog
Could not retrieve catalog; skipping run

Related Objects
Search...

Status	Assigned	Task
Stalled	None	T302086 Set scap minimum python version to 3.7
Open	None	T247045 Migrate all of production metal and VMs to Buster or later
Resolved	MoritzMuehlenhoff	T224549 Track remaining jessie systems in production
Resolved	None	T244852 Upgrade and improve our application object caching service (memcached)
Resolved	jijiki	T213089 Upgrade memcached cluster to Debian Buster
Resolved	None	T267006 Puppet failures on many hosts
Resolved	hashar	T267388 Puppet failure on deployment-memc08.deployment-prep.eqiad.wmflabs

Event Timeline

hashar created this task.Nov 6 2020, 8:58 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 6 2020, 8:58 AM

hashar added parent tasks: T267006: Puppet failures on many hosts, T213089: Upgrade memcached cluster to Debian Buster.Nov 6 2020, 8:58 AM

Hiera at https://horizon.wikimedia.org/project/instances/70f350ff-1c96-45c0-8174-b13fd66a35c3/ has:

profile::memcached::extended_options:
- modern
profile::memcached::growth_factor: '1.15'
profile::memcached::size: '3000'
profile::memcached::version: present
puppetmaster: deployment-puppetmaster04.deployment-prep.eqiad.wmflabs

That hasn't been changed recently https://gerrit.wikimedia.org/g/cloud/instance-puppet/+log/master/deployment-prep/deployment-memc08.deployment-prep.eqiad1.wikimedia.cloud.yaml . I guess the Puppet class now enforces type.

I have dropped the single quotes surrounding the values (diff), and surely Puppet is now working.

hashar mentioned this in T267006: Puppet failures on many hosts.Nov 6 2020, 9:02 AM

Puppet failure on deployment-memc08.deployment-prep.eqiad.wmflabs Closed, ResolvedPublicActions

Description

Related ObjectsSearch...

Event Timeline

Puppet failure on deployment-memc08.deployment-prep.eqiad.wmflabs
Closed, ResolvedPublic
Actions

Related Objects
Search...