[2020-10-05 16:54:21,198] ERROR in app: Error collecting the list of puppet classes
Traceback (most recent call last):
File "/data/project/openstack-browser/www/python/src/app.py", line 163, in all_puppetclasses
ctx.update({"puppetclasses": puppetclasses.all_classes()})
File "./keystone_browser/puppetclasses.py", line 69, in all_classes
return data["roles"]
TypeError: list indices must be integers or slices, not strDescription
Description
Related Objects
Related Objects
Event Timeline
Comment Actions
The backing service is returning a 500 response:
$ curl -kv http://puppetmaster.cloudinfra.wmflabs.org:8100/v1/roles * Trying 172.16.0.38... * TCP_NODELAY set * Connected to puppetmaster.cloudinfra.wmflabs.org (172.16.0.38) port 8100 (#0) > GET /v1/roles HTTP/1.1 > Host: puppetmaster.cloudinfra.wmflabs.org:8100 > User-Agent: curl/7.52.1 > Accept: */* > < HTTP/1.1 500 Internal Server Error < Server: nginx/1.14.2 < Date: Mon, 05 Oct 2020 17:31:49 GMT < Content-Type: text/plain < Transfer-Encoding: chunked < Connection: keep-alive < * Curl_http_done: called premature == 0 * Connection #0 to host puppetmaster.cloudinfra.wmflabs.org left intact
Comment Actions
Not sure if related, but just saw that puppet started failing in a bunch of nodes a few hours ago: https://tools-prometheus.wmflabs.org/tools/graph?g0.range_input=6h&g0.stacked=0&g0.expr=sum(puppet_agent_failed)&g0.tab=0&g1.range_input=6h&g1.expr=sum(puppet_agent_enabled)&g1.tab=0
Comment Actions
root@cloud-puppetmaster-03:~# journalctl -u uwsgi-labspuppetbackend.service ... Oct 05 14:43:08 cloud-puppetmaster-03 uwsgi-labspuppetbackend[435]: Traceback (most recent call last): Oct 05 14:43:08 cloud-puppetmaster-03 uwsgi-labspuppetbackend[435]: File "/usr/local/lib/python3.7/dist-packages/labspuppetbackend.py", line 2, in <module> Oct 05 14:43:08 cloud-puppetmaster-03 uwsgi-labspuppetbackend[435]: from statsd.defaults.env import statsd Oct 05 14:43:08 cloud-puppetmaster-03 uwsgi-labspuppetbackend[435]: File "/usr/lib/python3/dist-packages/statsd/defaults/env.py", line 17, in <module> Oct 05 14:43:08 cloud-puppetmaster-03 uwsgi-labspuppetbackend[435]: maxudpsize=maxudpsize, ipv6=ipv6) Oct 05 14:43:08 cloud-puppetmaster-03 uwsgi-labspuppetbackend[435]: File "/usr/lib/python3/dist-packages/statsd/client.py", line 139, in __init__ Oct 05 14:43:08 cloud-puppetmaster-03 uwsgi-labspuppetbackend[435]: host, port, fam, socket.SOCK_DGRAM)[0] Oct 05 14:43:08 cloud-puppetmaster-03 uwsgi-labspuppetbackend[435]: File "/usr/lib/python3.7/socket.py", line 748, in getaddrinfo Oct 05 14:43:08 cloud-puppetmaster-03 uwsgi-labspuppetbackend[435]: for res in _socket.getaddrinfo(host, port, family, type, proto, flags): Oct 05 14:43:08 cloud-puppetmaster-03 uwsgi-labspuppetbackend[435]: socket.gaierror: [Errno -3] Temporary failure in name resolution Oct 05 14:43:08 cloud-puppetmaster-03 uwsgi-labspuppetbackend[435]: unable to load app 0 (mountpoint='') (callable not found or import error) Oct 05 14:43:08 cloud-puppetmaster-03 uwsgi-labspuppetbackend[435]: *** no app loaded. going in full dynamic mode ***
Comment Actions
Mentioned in SAL (#wikimedia-cloud) [2020-10-05T17:40:17Z] <bd808> service uwsgi-labspuppetbackend restart on cloud-puppetmaster-03 (T264649)
Comment Actions
Mentioned in SAL (#wikimedia-cloud) [2020-10-05T17:44:29Z] <wm-bot> <bd808> Purged cache of puppet roles for T264649