I found out a bunch of jobs that have been stuck for hours and hours. Seems the issue is mediawiki_selenium @0.4.0 not having any timeout when connecting to SauceLabs.
An [[ https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce/439/consoleFull | example console with stacktrace ]]. Extracted:
```
-- Ruby level backtrace information ------------
...
cucumber-1.3.17/lib/cucumber/core_ext/instance_exec.rb:48:in `instance_exec'
mediawiki_selenium-0.4.0/lib/mediawiki_selenium/support/hooks.rb:85:in `block in <top (required)>'
mediawiki_selenium-0.4.0/lib/mediawiki_selenium/support/env.rb:84:in `sauce_api'
^^^^^^^^^^^^^
rest-client-1.7.2/lib/restclient/request.rb:41:in `execute'
rest-client-1.7.2/lib/restclient/request.rb:176:in `execute'
rest-client-1.7.2/lib/restclient/request.rb:413:in `transmit'
/usr/lib/ruby/2.0.0/net/http.rb:851:in `start'
/usr/lib/ruby/2.0.0/net/http.rb:862:in `do_start'
/usr/lib/ruby/2.0.0/net/http.rb:918:in `connect'
/usr/lib/ruby/2.0.0/timeout.rb:52:in `timeout'
/usr/lib/ruby/2.0.0/net/http.rb:918:in `block in connect'
/usr/lib/ruby/2.0.0/net/http.rb:918:in `connect'
```
The relevant code is:
```
lang=ruby
def sauce_api(json, session_id)
RestClient::Request.execute(
:method => :put,
:url => "https://saucelabs.com/rest/v1/#{ENV['SAUCE_ONDEMAND_USERNAME']}/jobs/#{session_id}",
:user => ENV["SAUCE_ONDEMAND_USERNAME"],
:password => ENV["SAUCE_ONDEMAND_ACCESS_KEY"],
:headers => {:content_type => "application/json"},
:payload => json
)
end
```
And RestClient::Request.execute() does not have any timeout at all by default ( https://github.com/rest-client/rest-client/blob/v1.7.2/lib/restclient/request.rb#L26 ).
Looking at lib/mediawiki_selenium/remote_browser_factory.rb in our gem 1.0.0 , we do not pass a timeout either so seems it is affected as well.