Page MenuHomePhabricator

proxylistener errors on tools-proxy-01
Closed, ResolvedPublic

Description

/var/log/syslog is full of:

Sep 30 09:58:08 tools-proxy-01 proxylistener[7658]: Traceback (most recent call last):
Sep 30 09:58:08 tools-proxy-01 proxylistener[7658]: File "/usr/lib/python2.7/logging/__init__.py", line 859, in emit
Sep 30 09:58:08 tools-proxy-01 proxylistener[7658]: msg = self.format(record)
Sep 30 09:58:08 tools-proxy-01 proxylistener[7658]: File "/usr/lib/python2.7/logging/__init__.py", line 732, in format
Sep 30 09:58:08 tools-proxy-01 proxylistener[7658]: return fmt.format(record)
Sep 30 09:58:08 tools-proxy-01 proxylistener[7658]: File "/usr/lib/python2.7/logging/__init__.py", line 471, in format
Sep 30 09:58:08 tools-proxy-01 proxylistener[7658]: record.message = record.getMessage()
Sep 30 09:58:08 tools-proxy-01 proxylistener[7658]: File "/usr/lib/python2.7/logging/__init__.py", line 335, in getMessage
Sep 30 09:58:08 tools-proxy-01 proxylistener[7658]: msg = msg % self.args
Sep 30 09:58:08 tools-proxy-01 proxylistener[7658]: TypeError: not all arguments converted during string formatting
Sep 30 09:58:08 tools-proxy-01 proxylistener[7658]: Logged from file proxylistener, line 111

Puppet is disabled:

scfc@tools-proxy-01:~$ sudo puppet agent -tv
Notice: Skipping run of Puppet configuration client; administratively disabled (Reason: 'reason not specified');
Use 'puppet agent --enable' to re-enable.
scfc@tools-proxy-01:~$

Event Timeline

scfc raised the priority of this task from to Unbreak Now!.
scfc updated the task description. (Show Details)
scfc added a project: Toolforge.
scfc moved this task to Ready to be worked on on the Toolforge board.
scfc subscribed.
Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Cloud-Services team (@yuvipanda, @coren, @Andrew): Do you know who is supposed to take a look at this?
This has been "Unbreak now" priority for a week without any comments.

This was fixed around Oct 1 06:51:32 UTC, apparently by @yuvipanda as he was working on the kubernetes proxy during that period.

@Joe was probably responsible :P But not sure why the proxylistener would've errored out, we didn't touch it.

I think the root cause was puppet not running; at some point it was restarted which seems to have fixed the issue.