Passenger spews Exception NoMethodError in Rack application object
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	akosiaris
	Nov 20 2017, 12:32 PM

Description

While debugging the servermon puppet reporter I came across the following in puppetmaster2001 apache error.log. Not sure why this is happening, but it's happening quite often, even with very few agents hitting puppetmaster2001 so it's warranting taking a better look

App 190599 stderr: [ 2017-11-20 12:28:54.4168 190618/0x00000000d7a1b8(Worker 1) utils.rb:84 ]: *** Exception NoMethodError in Rack application object (undefined method `empty?' for nil:NilClass) (process 190618, thread 0x00000000d7a1b8(Worker 1)):
App 190599 stderr: 	from config.ru:49:in `call'
App 190599 stderr: 	from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:74:in `process_request'
App 190599 stderr: 	from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:141:in `accept_and_process_next_request'
App 190599 stderr: 	from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:109:in `main_loop'
App 190599 stderr: 	from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:455:in `block (3 levels) in start_threads'
[Mon Nov 20 12:28:54.417045 2017] [core:error] [pid 190587] [client 208.80.153.74:35570] End of script output before headers:

Details

	Subject	Repo	Branch	Lines +/-
	icinga: add support for puppet 4 in backend puppetmaster https checks	operations/puppet	production	+17 -5

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Resolved	aborrero	T178717 Upgrade wmcs instances and masters to puppet 4.8
Resolved	None	T177254 Upgrade to puppet 4 (4.8 or newer)
Resolved	fgiunchedi	T180944 Passenger spews Exception NoMethodError in Rack application object

Event Timeline

akosiaris created this task.Nov 20 2017, 12:32 PM

Restricted Application removed a project: Patch-For-Review. · View Herald TranscriptNov 20 2017, 12:32 PM

This is probably related. From icinga

HTTP CRITICAL - Invalid HTTP response received from host on port 8141: HTTP/1.1 500 Internal Server Error

Link is https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=puppetmaster2002&service=puppetmaster+backend+https

A quick execution of

/usr/lib/nagios/plugins/check_http -S -H puppetmaster2002.codfw.wmnet -p 8141 -e 400

on einsteinium does show that this is indeed the case. This looks like an upstream bug, I 'll upstream this to debian first as the version we run is patched

Alternatively updating the check to include -u /puppet/v3 works. My thought was to update our Icinga check once both codfw and eqiad are upgraded.

/usr/lib/nagios/plugins/check_http -S -H puppetmaster2002.codfw.wmnet -p 8141 -e 400 -u /puppet/v3
HTTP OK: Status line output matched "400" - 399 bytes in 0.210 second response time |time=0.210129s;;;0.000000 size=399B;;;0

Yeah, after a brief discussion I 've had, it turns out the culprit is probably the puppet 3 rack compatibility middleware (shipped in Debian only AFAIK) and for this to happen / needs to be polled. This should be fixed in Debian as well, but we can indeed workaround it for now by using the approach above.

Change 392423 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] icinga: add support for puppet 4 in backend puppetmaster https checks

https://gerrit.wikimedia.org/r/392423

gerritbot added a project: Patch-For-Review.Nov 20 2017, 2:35 PM

Change 392423 merged by Herron:
[operations/puppet@production] icinga: add support for puppet 4 in backend puppetmaster https checks

https://gerrit.wikimedia.org/r/392423

bd808 removed a project: cloud-services-team (FY2017-18).Nov 27 2017, 6:02 PM

ArielGlenn removed a project: Patch-For-Review.Nov 13 2018, 9:56 AM

• Phabricator_maintenance moved this task from Backlog to Acknowledged on the SRE board.Jan 26 2019, 9:36 PM