Page MenuHomePhabricator

Puppet failures with "Attempt to assign to a reserved variable name: 'trusted'"
Closed, ResolvedPublic

Description

Observing the icinga alerts for puppet failures I noticed there were some sporadic failures not related to e.g. recent merges. It looks like the bulk of it is due to the same error

root@lithium:/srv/syslog# cat archive/syslog.log-20161214 syslog.log | fgrep 'SERVER: Attempt to assign to a reserved variable name'
Dec 13 09:21:02 ms-be1024 puppet-agent[4896]: Could not retrieve catalog from remote server: Error 400 on SERVER: Attempt to assign to a reserved variable name: 'trusted' on node ms-be1024.eqiad.wmnet
Dec 13 09:24:31 maps1002 puppet-agent[15700]: Could not retrieve catalog from remote server: Error 400 on SERVER: Attempt to assign to a reserved variable name: 'trusted' on node maps1002.eqiad.wmnet
Dec 13 09:37:40 db1036 puppet-agent[9559]: Could not retrieve catalog from remote server: Error 400 on SERVER: Attempt to assign to a reserved variable name: 'trusted' on node db1036.eqiad.wmnet
Dec 13 12:09:37 es1013 puppet-agent[26739]: Could not retrieve catalog from remote server: Error 400 on SERVER: Attempt to assign to a reserved variable name: 'trusted' on node es1013.eqiad.wmnet
Dec 13 13:00:52 mw1301 puppet-agent[42665]: Could not retrieve catalog from remote server: Error 400 on SERVER: Attempt to assign to a reserved variable name: 'trusted' on node mw1301.eqiad.wmnet
Dec 13 14:04:27 cp1045 puppet-agent[2384]: Could not retrieve catalog from remote server: Error 400 on SERVER: Attempt to assign to a reserved variable name: 'trusted' on node cp1045.eqiad.wmnet
Dec 13 14:34:02 mw1293 puppet-agent[26902]: Could not retrieve catalog from remote server: Error 400 on SERVER: Attempt to assign to a reserved variable name: 'trusted' on node mw1293.eqiad.wmnet
Dec 13 18:16:19 cp1008 puppet-agent[5032]: Could not retrieve catalog from remote server: Error 400 on SERVER: Attempt to assign to a reserved variable name: 'trusted' on node cp1008.wikimedia.org
Dec 13 18:26:03 snapshot1007 puppet-agent[37059]: Could not retrieve catalog from remote server: Error 400 on SERVER: Attempt to assign to a reserved variable name: 'trusted' on node snapshot1007.eqiad.wmnet
Dec 13 18:35:49 elastic1025 puppet-agent[23187]: Could not retrieve catalog from remote server: Error 400 on SERVER: Attempt to assign to a reserved variable name: 'trusted' on node elastic1025.eqiad.wmnet
Dec 13 19:58:07 bast4001 puppet-agent[38910]: Could not retrieve catalog from remote server: Error 400 on SERVER: Attempt to assign to a reserved variable name: 'trusted' on node bast4001.wikimedia.org
Dec 13 20:07:32 conf1003 puppet-agent[8428]: Could not retrieve catalog from remote server: Error 400 on SERVER: Attempt to assign to a reserved variable name: 'trusted' on node conf1003.eqiad.wmnet
Dec 13 20:12:03 labvirt1013 puppet-agent[47433]: Could not retrieve catalog from remote server: Error 400 on SERVER: Attempt to assign to a reserved variable name: 'trusted' on node labvirt1013.eqiad.wmnet
Dec 13 20:41:42 elastic1034 puppet-agent[19069]: Could not retrieve catalog from remote server: Error 400 on SERVER: Attempt to assign to a reserved variable name: 'trusted' on node elastic1034.eqiad.wmnet
Dec 13 23:11:03 labtestweb2001 puppet-agent[9885]: Could not retrieve catalog from remote server: Error 400 on SERVER: Attempt to assign to a reserved variable name: 'trusted' on node labtestweb2001.wikimedia.org
Dec 13 23:56:53 elastic1039 puppet-agent[2348]: Could not retrieve catalog from remote server: Error 400 on SERVER: Attempt to assign to a reserved variable name: 'trusted' on node elastic1039.eqiad.wmnet
Dec 14 00:34:31 restbase1013 puppet-agent[25102]: Could not retrieve catalog from remote server: Error 400 on SERVER: Attempt to assign to a reserved variable name: 'trusted' on node restbase1013.eqiad.wmnet
Dec 14 04:09:08 ms-be1016 puppet-agent[10188]: Could not retrieve catalog from remote server: Error 400 on SERVER: Attempt to assign to a reserved variable name: 'trusted' on node ms-be1016.eqiad.wmnet
Dec 14 05:21:46 relforge1001 puppet-agent[24708]: Could not retrieve catalog from remote server: Error 400 on SERVER: Attempt to assign to a reserved variable name: 'trusted' on node relforge1001.eqiad.wmnet
Dec 14 06:00:41 elastic1031 puppet-agent[36521]: Could not retrieve catalog from remote server: Error 400 on SERVER: Attempt to assign to a reserved variable name: 'trusted' on node elastic1031.eqiad.wmnet
Dec 14 06:29:57 poolcounter1002 puppet-agent[29235]: Could not retrieve catalog from remote server: Error 400 on SERVER: Attempt to assign to a reserved variable name: 'trusted' on node poolcounter1002.eqiad.wmnet
Dec 14 07:51:40 cp3045 puppet-agent[16116]: Could not retrieve catalog from remote server: Error 400 on SERVER: Attempt to assign to a reserved variable name: 'trusted' on node cp3045.esams.wmnet
Dec 14 11:21:40 netmon1001 puppet-agent[17129]: Could not retrieve catalog from remote server: Error 400 on SERVER: Attempt to assign to a reserved variable name: 'trusted' on node netmon1001.wikimedia.org
Dec 14 11:41:02 labtestweb2001 puppet-agent[10856]: Could not retrieve catalog from remote server: Error 400 on SERVER: Attempt to assign to a reserved variable name: 'trusted' on node labtestweb2001.wikimedia.org
Dec 14 13:36:41 restbase1015 puppet-agent[17948]: Could not retrieve catalog from remote server: Error 400 on SERVER: Attempt to assign to a reserved variable name: 'trusted' on node restbase1015.eqiad.wmnet
Dec 14 14:37:26 labstore1002 puppet-agent[11037]: Could not retrieve catalog from remote server: Error 400 on SERVER: Attempt to assign to a reserved variable name: 'trusted' on node labstore1002.eqiad.wmnet
Dec 14 19:17:35 wasat puppet-agent[3755]: Could not retrieve catalog from remote server: Error 400 on SERVER: Attempt to assign to a reserved variable name: 'trusted' on node wasat.codfw.wmnet
Dec 14 19:38:20 labstore1005 puppet-agent[145244]: Could not retrieve catalog from remote server: Error 400 on SERVER: Attempt to assign to a reserved variable name: 'trusted' on node labstore1005.eqiad.wmnet
Dec 14 20:29:03 mw1288 puppet-agent[54310]: Could not retrieve catalog from remote server: Error 400 on SERVER: Attempt to assign to a reserved variable name: 'trusted' on node mw1288.eqiad.wmnet
Dec 14 20:44:03 lvs1005 puppet-agent[14309]: Could not retrieve catalog from remote server: Error 400 on SERVER: Attempt to assign to a reserved variable name: 'trusted' on node lvs1005.wikimedia.org

And a quick search seems to link this to https://tickets.puppetlabs.com/browse/PDB-949

Event Timeline

fgiunchedi triaged this task as Medium priority.Dec 14 2016, 8:50 PM

this is now happening on iridium

@puppetmaster1001:/var/log# grep trusted /var/log/syslog
Feb 13 17:22:13 puppetmaster1001 puppet-master[170457]: Attempt to assign to a reserved variable name: 'trusted' on node iridium.eqiad.wmnet
Feb 13 17:22:13 puppetmaster1001 puppet-master[170457]: Attempt to assign to a reserved variable name: 'trusted' on node iridium.eqiad.wmnet
Feb 13 17:22:13 puppetmaster1001 puppet-master[170457]: Attempt to assign to a reserved variable name: 'trusted' on node iridium.eqiad.wmnet
jbond claimed this task.
jbond subscribed.

Im going to close this im pretty sure its fixed now but please re-open if not