labspuppetbackend service can fail to intialize when DNS blip happens at the wrong time
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	bd808
	Oct 5 2020, 7:23 PM

Description

Found while investigating T264649: openstack-browser not showing any puppet classes

root@cloud-puppetmaster-03:~# journalctl -u uwsgi-labspuppetbackend.service
...
Oct 05 14:43:08 cloud-puppetmaster-03 uwsgi-labspuppetbackend[435]: Traceback (most recent call last):
Oct 05 14:43:08 cloud-puppetmaster-03 uwsgi-labspuppetbackend[435]:   File "/usr/local/lib/python3.7/dist-packages/labspuppetbackend.py", line 2, in <module>
Oct 05 14:43:08 cloud-puppetmaster-03 uwsgi-labspuppetbackend[435]:     from statsd.defaults.env import statsd
Oct 05 14:43:08 cloud-puppetmaster-03 uwsgi-labspuppetbackend[435]:   File "/usr/lib/python3/dist-packages/statsd/defaults/env.py", line 17, in <module>
Oct 05 14:43:08 cloud-puppetmaster-03 uwsgi-labspuppetbackend[435]:     maxudpsize=maxudpsize, ipv6=ipv6)
Oct 05 14:43:08 cloud-puppetmaster-03 uwsgi-labspuppetbackend[435]:   File "/usr/lib/python3/dist-packages/statsd/client.py", line 139, in __init__
Oct 05 14:43:08 cloud-puppetmaster-03 uwsgi-labspuppetbackend[435]:     host, port, fam, socket.SOCK_DGRAM)[0]
Oct 05 14:43:08 cloud-puppetmaster-03 uwsgi-labspuppetbackend[435]:   File "/usr/lib/python3.7/socket.py", line 748, in getaddrinfo
Oct 05 14:43:08 cloud-puppetmaster-03 uwsgi-labspuppetbackend[435]:     for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
Oct 05 14:43:08 cloud-puppetmaster-03 uwsgi-labspuppetbackend[435]: socket.gaierror: [Errno -3] Temporary failure in name resolution
Oct 05 14:43:08 cloud-puppetmaster-03 uwsgi-labspuppetbackend[435]: unable to load app 0 (mountpoint='') (callable not found or import error)
Oct 05 14:43:08 cloud-puppetmaster-03 uwsgi-labspuppetbackend[435]: *** no app loaded. going in full dynamic mode ***

Details

	Subject	Repo	Branch	Lines +/-
	encapi (aka 'labspuppet backend'): add reload-on-exception to uwsgi config	operations/puppet	production	+7 -6

Customize query in gerrit

Related Objects

Mentioned Here: T264649: openstack-browser not showing any puppet classes

Event Timeline

bd808 created this task.Oct 5 2020, 7:23 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 5 2020, 7:23 PM

[19:00]  <andrewbogott> bd808: hm, uwsgi caught the error, would be nice if it retried
[19:01]  <    bd808> andrewbogott: yeah, I haven't looked at the code to see if there is an obvious fix.
[19:01]  <    bd808> I'll file a bug about it so we don't completely lose track of the potential problem
[19:02]  <andrewbogott> there's a 'reload-on-exception' flag for uwsgi
[19:02]  <andrewbogott> and 'reload-on-exception-type'
[19:02]  <andrewbogott> That might be what we want

Andrew claimed this task.Oct 5 2020, 7:24 PM

• nskaggs moved this task from Inbox to Doing on the cloud-services-team (Kanban) board.Oct 20 2020, 4:30 PM

Change 636045 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] encapi (aka 'labspuppet backend'): add reload-on-exception to uwsgi config

https://gerrit.wikimedia.org/r/636045

gerritbot added a project: Patch-For-Review.Oct 23 2020, 2:53 PM

Change 636045 merged by Andrew Bogott:
[operations/puppet@production] encapi (aka 'labspuppet backend'): add reload-on-exception to uwsgi config

https://gerrit.wikimedia.org/r/636045

Maintenance_bot removed a project: Patch-For-Review.Oct 23 2020, 3:10 PM

I'm not in a hurry to reproduce the exact cause of this failure, but the attached patch should prevent the puppet issue in the future.

labspuppetbackend service can fail to intialize when DNS blip happens at the wrong timeClosed, ResolvedPublicActions

Description

Details

Related Objects

Event Timeline

labspuppetbackend service can fail to intialize when DNS blip happens at the wrong time
Closed, ResolvedPublic
Actions