pybal error "exceptions.ValueError: Value of arguments is not a string or stringlist"
Closed, ResolvedPublic

Description

While restarting pybal on lvs1006, I an error about "exceptions.ValueError: Value of arguments is not a string or stringlist" in the logs (see below). Note that at that time I had not actually merged any change to pybal configuration, so the error might not be related to the restart at all.

Oct 05 13:50:04 lvs1006 pybal[19917]: [api_80] INFO: Merged enabled server mw1230.eqiad.wmnet, weight 20
Oct 05 13:50:04 lvs1006 pybal[19917]: [pybal] INFO: [api_80] Initialization complete
Oct 05 13:50:04 lvs1006 pybal[19917]: Unhandled error in Deferred:
Oct 05 13:50:04 lvs1006 pybal[19917]: Unhandled Error
Oct 05 13:50:04 lvs1006 pybal[19917]: Traceback (most recent call last):
Oct 05 13:50:04 lvs1006 pybal[19917]: File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 577, in _runCallbacks
Oct 05 13:50:04 lvs1006 pybal[19917]: current.result = callback(current.result, *args, **kw)
Oct 05 13:50:04 lvs1006 pybal[19917]: File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 839, in _cbDeferred
Oct 05 13:50:04 lvs1006 pybal[19917]: self.callback(self.resultList)
Oct 05 13:50:04 lvs1006 pybal[19917]: File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 382, in callback
Oct 05 13:50:04 lvs1006 pybal[19917]: self._startRunCallbacks(result)
Oct 05 13:50:04 lvs1006 pybal[19917]: File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 490, in _startRunCallbacks
Oct 05 13:50:04 lvs1006 pybal[19917]: self._runCallbacks()
Oct 05 13:50:04 lvs1006 pybal[19917]: --- <exception caught here> ---
Oct 05 13:50:04 lvs1006 pybal[19917]: File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 577, in _runCallbacks
Oct 05 13:50:04 lvs1006 pybal[19917]: current.result = callback(current.result, *args, **kw)
Oct 05 13:50:04 lvs1006 pybal[19917]: File "/usr/lib/python2.7/dist-packages/pybal/pybal.py", line 173, in _ready
Oct 05 13:50:04 lvs1006 pybal[19917]: self.createMonitoringInstances(coordinator)
Oct 05 13:50:04 lvs1006 pybal[19917]: File "/usr/lib/python2.7/dist-packages/pybal/pybal.py", line 213, in createMonitoringInstances
Oct 05 13:50:04 lvs1006 pybal[19917]: monitor = monitorclass(coordinator, self, lvsservice.configuration)
Oct 05 13:50:04 lvs1006 pybal[19917]: File "/usr/lib/python2.7/dist-packages/pybal/monitors/runcommand.py", line 104, in __init__
Oct 05 13:50:04 lvs1006 pybal[19917]: self.arguments = self._getConfigStringList('arguments', locals=locals)
Oct 05 13:50:04 lvs1006 pybal[19917]: File "/usr/lib/python2.7/dist-packages/pybal/monitor.py", line 107, in _getConfigStringList
Oct 05 13:50:04 lvs1006 pybal[19917]: optionname)
Oct 05 13:50:04 lvs1006 pybal[19917]: exceptions.ValueError: Value of arguments is not a string or stringlist
Oct 05 13:50:15 lvs1006 pybal[19917]: [apaches_80 ProxyFetch] WARN: mw1274.eqiad.wmnet (enabled/partially up/not pooled): Fetch failed, 5.002 s
Gehel created this task.Oct 5 2016, 2:06 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 5 2016, 2:06 PM
elukey triaged this task as "Normal" priority.Oct 19 2016, 12:38 PM
elukey added a subscriber: elukey.

@Gehel: did the the restart fail or just proceeded anyway?

Gehel added a comment.Oct 19 2016, 1:58 PM

The restart went just fine. There is no functional problem that I know related to this log. I was just surprised to see the error in the logs and it was suggested that I open a ticket so that someone with more knowledge of pybal could look into it.

Change 337400 had a related patch set uploaded (by Ema):
MonitoringProtocol: do not crash with ValueError on unicode strings

https://gerrit.wikimedia.org/r/337400

Change 337400 merged by Ema:
MonitoringProtocol: do not crash with ValueError on unicode strings

https://gerrit.wikimedia.org/r/337400

Mentioned in SAL (#wikimedia-operations) [2017-02-14T10:17:59Z] <ema> uploading pybal 1.13.5 to apt.w.o T147425

Mentioned in SAL (#wikimedia-operations) [2017-02-14T10:23:17Z] <ema> lvs200[456]: upgrade to jessie 8.7, pybal 1.13.5, reboot into kernel 4.4.2-3+wmf8 T155401 T147425

Mentioned in SAL (#wikimedia-operations) [2017-02-14T10:51:07Z] <ema> lvs200[123]: upgrade to jessie 8.7, pybal 1.13.5, reboot into kernel 4.4.2-3+wmf8 T155401 T147425

Mentioned in SAL (#wikimedia-operations) [2017-02-14T13:32:04Z] <ema> lvs400[34]: upgrade to jessie 8.7, pybal 1.13.5, reboot into kernel 4.4.2-3+wmf8 T155401 T147425

Mentioned in SAL (#wikimedia-operations) [2017-02-14T13:50:30Z] <ema> lvs400[12]: upgrade to jessie 8.7, pybal 1.13.5, reboot into kernel 4.4.2-3+wmf8 T155401 T147425

Mentioned in SAL (#wikimedia-operations) [2017-02-14T14:11:27Z] <ema> lvs300[34]: upgrade to jessie 8.7, pybal 1.13.5, reboot into kernel 4.4.2-3+wmf8 T155401 T147425

Mentioned in SAL (#wikimedia-operations) [2017-02-14T14:38:22Z] <ema> lvs300[12]: upgrade to jessie 8.7, pybal 1.13.5, reboot into kernel 4.4.2-3+wmf8 T155401 T147425

Mentioned in SAL (#wikimedia-operations) [2017-02-14T15:01:38Z] <ema> lvs10*: upgrade to pybal 1.13.5 T147425

ema closed this task as "Resolved".Feb 14 2017, 3:08 PM
ema claimed this task.
ema added a subscriber: ema.

I've upgraded all LVSs to pybal 1.13.5 which includes a fix for this: https://gerrit.wikimedia.org/r/#/c/337400/. Closing.