Page MenuHomePhabricator

Salt minions randomly crashing when the deployment server grain gets changed
Closed, DeclinedPublic


When I switched over the deployment server, puppet ran

grain-ensure set trebuchet_master mira.codfw.wmnet

this worked fine on ~ 60% of the hosts, while on the others (independently of the OS version) this crashed the salt-minion.

This seems to be caused by some race condition; in the minion logs I find:

2016-01-25 10:10:58,656 [salt.log.setup   ][ERROR   ] An un-handled exception was caught by salt's global exception handler:
TypeError: string indices must be integers, not str
Traceback (most recent call last):
  File "/usr/bin/salt-minion", line 14, in <module>
  File "/usr/lib/python2.7/dist-packages/salt/", line 57, in salt_minion
  File "/usr/lib/python2.7/dist-packages/salt/", line 264, in start
  File "/usr/lib/python2.7/dist-packages/salt/", line 558, in tune_in
  File "/usr/lib/python2.7/dist-packages/salt/", line 1407, in pillar_refresh
  File "/usr/lib/python2.7/dist-packages/salt/pillar/", line 91, in compile_pillar
    ret_pillar = self.sreq.crypted_transfer_decode_dictentry(load, dictkey='pillar', tries=3, timeout=7200)
  File "/usr/lib/python2.7/dist-packages/salt/transport/", line 243, in crypted_transfer_decode_dictentry
    aes = key.private_decrypt(ret['key'], 4)
TypeError: string indices must be integers, not str

which honestly doesn't leave me any clue.

This seems serious enough to be investigated further though.

Event Timeline

Joe created this task.Jan 25 2016, 11:15 AM
Joe raised the priority of this task from to High.
Joe updated the task description. (Show Details)
Joe added projects: Operations, Salt.
Joe added a subscriber: Joe.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 25 2016, 11:15 AM
ArielGlenn set Security to None.

Tenatively this looks like an issue with the singleton cache of master aes keys at the minion end, a part of the code in transport that needs to be updated. Still investigating.

To keep minions from dying we should do this:

in transport/, in crypted_transfer_decode_dictentry()

instead of

aes = key.private_decrypt(ret['key'], 4)
pcrypt = salt.crypt.Crypticle(self.opts, aes)
return pcrypt.loads(ret[dictkey])

we should have

    aes = key.private_decrypt(ret['key'], 4)
except (TypeError, KeyError):
    return None
    pcrypt = salt.crypt.Crypticle(self.opts, aes)
    return pcrypt.loads(ret[dictkey])

Forgot to mention, this is actually an issue with the pillar refresh after the grain is set.

ArielGlenn moved this task from Backlog to active on the Salt board.Jan 26 2016, 10:17 AM

I've updated my docker salt testbed to work with latest docker api and latest wmf packages:

I'll be doing small scale testing to see if I can replicate this problem there; if not, I'll roll out the above change and log errors in hopes of catching the cause. The fix above will at least keep the minions from dying.

No joy so I'll add the above change to our salt packages with logging and update them all.

ArielGlenn moved this task from Blocked/Stalled to active on the Salt board.Mar 9 2016, 11:18 PM
ArielGlenn moved this task from active to testing needed on the Salt board.
MoritzMuehlenhoff closed this task as Declined.Sep 15 2017, 11:07 AM

Salt is being removed.