Page MenuHomePhabricator

tools-k8s-master-01 (Kubernetes API server for toolforge) has failing puppet staleness cron
Closed, ResolvedPublic

Description

Since we use prometheus for monitoring puppet staleness, we are unable to tell whether puppet is stale on tools-k8s-master-01.tools.eqiad.wmflabs without logging into it, which is not really good. The cron that generates prometheus data fails for some reason with the following:

:~$ sudo /usr/local/bin/prometheus-puppet-agent-stats --outfile /var/lib/prometheus/node.d/puppet_agent.prom
Traceback (most recent call last):
  File "/usr/local/bin/prometheus-puppet-agent-stats", line 23, in <module>
    from prometheus_client import CollectorRegistry, Gauge, write_to_textfile
  File "/usr/lib/python3/dist-packages/prometheus_client/__init__.py", line 4, in <module>
    from . import exposition
  File "/usr/lib/python3/dist-packages/prometheus_client/exposition.py", line 10, in <module>
    from wsgiref.simple_server import make_server
  File "/usr/lib/python3.4/wsgiref/simple_server.py", line 13, in <module>
    from http.server import BaseHTTPRequestHandler, HTTPServer
  File "/usr/lib/python3.4/http/server.py", line 88, in <module>
    import http.client
  File "/usr/lib/python3.4/http/client.py", line 1014
    raise InvalidURL(f"URL can't contain control characters. {url!r} "
                                                                     ^
SyntaxError: invalid syntax

That looks like an f-string, so I think this is just a python version error (which we saw recently).

Event Timeline

Bstorm triaged this task as Low priority.Tue, Nov 5, 9:33 PM
Bstorm created this task.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptTue, Nov 5, 9:33 PM
Krenair added a subscriber: Krenair.Tue, Nov 5, 9:43 PM

f-strings are 3.6+, wonder what that's doing lurking under /usr/lib/python3.4

Bstorm added a comment.Tue, Nov 5, 9:49 PM

That happened a few months ago when someone at Debian backpatched something into 3.4.2-1+deb8u3, which we ended up with across the fleet due to unattended upgrades. It was corrected in 3.4.2-1+deb8u4. We are using the same version of python3 as the rest of the k8s cluster, so I am surprised to see this:

~$ apt-cache policy python3
python3:
  Installed: 3.4.2-2
  Candidate: 3.4.2-2
  Version table:
 *** 3.4.2-2 0
        500 http://mirrors.wikimedia.org/debian/ jessie/main amd64 Packages
        100 /var/lib/dpkg/status

At least that's what the OS thinks.

Bstorm added a comment.Tue, Nov 5, 9:51 PM

The weird thing is that it should crash a lot more if that's happening here...but other python3 things appear to be working?

Bstorm added a comment.EditedTue, Nov 5, 9:53 PM

For the record this is sudo -u prometheus /usr/local/bin/prometheus-puppet-agent-stats -d --outfile /var/lib/prometheus/node.d/puppet_agent.prom
That produces the error (-d should enable debug logging).

Bstorm added a comment.Tue, Nov 5, 9:54 PM

Happens after prometheus_client import

The weird thing is that it should crash a lot more if that's happening here...but other python3 things appear to be working?

Maybe not many things use the http module?

This comment was removed by Krenair.

Mentioned in SAL (#wikimedia-cloud) [2019-11-05T22:35:18Z] <bstorm_> upgraded libpython3.4 libpython3.4-dbg libpython3.4-minimal libpython3.4-stdlib python3.4 python3.4-dbg python3.4-minimal to fix an old broken patch T237468

Bstorm closed this task as Resolved.Tue, Nov 5, 10:38 PM
Bstorm claimed this task.

And that did it! Thanks @Krenair

For the record what I found (and then redacted pending upgrade) was this:

krenair@tools-k8s-master-01:~$ dpkg -S /usr/lib/python3.4/http/client.py
libpython3.4-stdlib:amd64: /usr/lib/python3.4/http/client.py
krenair@tools-k8s-master-01:~$ apt-cache policy libpython3.4-stdlib
libpython3.4-stdlib:
  Installed: 3.4.2-1+deb8u3
  Candidate: 3.4.2-1+deb8u7
  Version table:
     3.4.2-1+deb8u7 0
        500 http://security.debian.org/ jessie/updates/main amd64 Packages
 *** 3.4.2-1+deb8u3 0
        100 /var/lib/dpkg/status
     3.4.2-1 0
        500 http://mirrors.wikimedia.org/debian/ jessie/main amd64 Packages