convert wikitech.wikimedia.org from globalsign to letsencrypt certificate (deadline 2017-02-24)
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	RobH
	Jan 9 2017, 6:05 PM

Description

This task will track the conversion of wikitech from a GlobalSign paid certificate to a free LetsEncrypt certificate. This must be completed by the expiry date of the current certificate, which is 2017-02-24.

This particular certificate was listed on T133717 as a candidate to conversion to LE.

While this certificate initially appears to be a valid canidate to convert from GlobalSign to LetsEncrypt, it appears (after some IRC discussion with @Krenair) that labs isntances, which contact the wikitech.w.o domain, use ruby-httpclient. ruby-httpclient has its own list of trusted roots, this patch overrides it to use the system's.

Alex has also prepared a patch to fix the trust issue: https://gerrit.wikimedia.org/r/#/c/311048/ - this has been merged live by @akosiaris on 2017-01-11.

Alex also prepared the LE conversion patch - https://gerrit.wikimedia.org/r/#/c/331638/ - this cannot be merged until we plan a maint window with the labs team, since it has potential to require a labs puppetmaster restart, or other associated labs services that interact with wikitech via its certificate.

This task was escalated to the @Andrew in labs for review on 2017-01-11 by @RobH. Andrew was involved in past wikitech updates (if @RobH recalls correctly). If another labs person should be involved, please update and reassign!

Details

Subject	Repo	Branch	Lines +/-
wikitech: remove pre-LE sslcert class	operations/puppet	production	+0 -2
Use LE for wikitech	operations/puppet	production	+9 -92
ruby-httpclient callers: Use the operating system's certificate store	operations/puppet	production	+12 -0

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Resolved	None	T133717 Letsencrypt all the prod things we can - planning
Resolved	Andrew	T154913 convert wikitech.wikimedia.org from globalsign to letsencrypt certificate (deadline 2017-02-24)
		Unknown Object (Task)

Event Timeline

RobH created this task.Jan 9 2017, 6:05 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 9 2017, 6:05 PM

RobH updated the task description. (Show Details)Jan 9 2017, 6:08 PM

RobH added a subtask: Unknown Object (Task).

RobH added a subscriber: Krenair.

Paladox subscribed.Jan 9 2017, 6:09 PM

RobH updated the task description. (Show Details)Jan 9 2017, 7:07 PM

• ema moved this task from Backlog to TLS on the Traffic board.Jan 9 2017, 7:51 PM

Krenair claimed this task.Jan 9 2017, 8:33 PM

Change 311048 had a related patch set uploaded (by Alex Monk):
ruby-httpclient callers: Use the operating system's certificate store

https://gerrit.wikimedia.org/r/311048

gerritbot added a project: Patch-For-Review.Jan 9 2017, 8:33 PM

Dzahn added a parent task: T133717: Letsencrypt all the prod things we can - planning.Jan 9 2017, 8:36 PM

If the patch tested above works, it looks a good alternative to me. We 've had that issue in the past with nodejs as well and had to patch it to avoid using the built-in list of CAs.

labtestwikitech already uses LE so we can test it on that:

curl to show the system has no issue with the cert
Create HTTPClient
Try using that to get labtestwikitech, SSL failure
Try using that to get wikitech, it's fine (at the moment, because it has the GlobalSign cert still instead of LE)
Now run the two lines from the patch (to clear the cert store in memory and replace it with the system's)
Now try using it with labtestwikitech, it's fine
Now try using it with wikitech, it's still fine
Now try using it with self-signed.badssl.com and untrusted-root.badssl.com to make sure we haven't just accidentally trusted everything

krenair@deployment-puppetmaster02:~$ curl https://labtestwikitech.wikimedia.org
krenair@deployment-puppetmaster02:~$ irb
irb(main):001:0> require 'httpclient'
=> true
irb(main):002:0> @http = HTTPClient.new
=> #<HTTPClient:0x00000002d86e48 @proxy=nil, @no_proxy=nil, @no_proxy_regexps=[], @www_auth=#<HTTPClient::WWWAuth:0x00000002d86d80 @basic_auth=#<HTTPClient::BasicAuth:0x00000002d86d58 @cred=nil, @set=false, @auth={}, @challengeable={}, @scheme="Basic">, @digest_auth=#<HTTPClient::DigestAuth:0x00000002d86cb8 @auth={}, @challenge={}, @set=false, @nonce_count=0, @scheme="Digest">, @negotiate_auth=#<HTTPClient::NegotiateAuth:0x00000002d86c18 @auth={}, @auth_default=nil, @challenge={}, @scheme="Negotiate", @set=false, @ntlm_opt={:ntlmv2=>true}>, @ntlm_auth=#<HTTPClient::NegotiateAuth:0x00000002d86b28 @auth={}, @auth_default=nil, @challenge={}, @scheme="NTLM", @set=false, @ntlm_opt={:ntlmv2=>true}>, @sspi_negotiate_auth=#<HTTPClient::SSPINegotiateAuth:0x00000002d86a10 @challenge={}, @scheme="Negotiate">, @oauth=#<HTTPClient::OAuth:0x00000002d86998 @config=nil, @auth={}, @challengeable={}, @nonce_count=0, @signature_handler={"HMAC-SHA1"=>#<Method: HTTPClient::OAuth#sign_hmac_sha1>}, @scheme="OAuth">, @authenticator=[#<HTTPClient::OAuth:0x00000002d86998 @config=nil, @auth={}, @challengeable={}, @nonce_count=0, @signature_handler={"HMAC-SHA1"=>#<Method: HTTPClient::OAuth#sign_hmac_sha1>}, @scheme="OAuth">, #<HTTPClient::NegotiateAuth:0x00000002d86c18 @auth={}, @auth_default=nil, @challenge={}, @scheme="Negotiate", @set=false, @ntlm_opt={:ntlmv2=>true}>, #<HTTPClient::NegotiateAuth:0x00000002d86b28 @auth={}, @auth_default=nil, @challenge={}, @scheme="NTLM", @set=false, @ntlm_opt={:ntlmv2=>true}>, #<HTTPClient::SSPINegotiateAuth:0x00000002d86a10 @challenge={}, @scheme="Negotiate">, #<HTTPClient::DigestAuth:0x00000002d86cb8 @auth={}, @challenge={}, @set=false, @nonce_count=0, @scheme="Digest">, #<HTTPClient::BasicAuth:0x00000002d86d58 @cred=nil, @set=false, @auth={}, @challengeable={}, @scheme="Basic">]>, @proxy_auth=#<HTTPClient::ProxyAuth:0x00000002d86880 @basic_auth=#<HTTPClient::ProxyBasicAuth:0x00000002d86830 @cred=nil, @set=false, @auth={}, @challengeable={}, @scheme="Basic">, @negotiate_auth=#<HTTPClient::NegotiateAuth:0x00000002d86790 @auth={}, @auth_default=nil, @challenge={}, @scheme="Negotiate", @set=false, @ntlm_opt={:ntlmv2=>true}>, @ntlm_auth=#<HTTPClient::NegotiateAuth:0x00000002d866a0 @auth={}, @auth_default=nil, @challenge={}, @scheme="NTLM", @set=false, @ntlm_opt={:ntlmv2=>true}>, @sspi_negotiate_auth=#<HTTPClient::SSPINegotiateAuth:0x00000002d86600 @challenge={}, @scheme="Negotiate">, @digest_auth=#<HTTPClient::ProxyDigestAuth:0x00000002d86560 @auth={}, @challenge=nil, @set=false, @nonce_count=0, @scheme="Digest">, @authenticator=[#<HTTPClient::NegotiateAuth:0x00000002d86790 @auth={}, @auth_default=nil, @challenge={}, @scheme="Negotiate", @set=false, @ntlm_opt={:ntlmv2=>true}>, #<HTTPClient::NegotiateAuth:0x00000002d866a0 @auth={}, @auth_default=nil, @challenge={}, @scheme="NTLM", @set=false, @ntlm_opt={:ntlmv2=>true}>, #<HTTPClient::SSPINegotiateAuth:0x00000002d86600 @challenge={}, @scheme="Negotiate">, #<HTTPClient::ProxyDigestAuth:0x00000002d86560 @auth={}, @challenge=nil, @set=false, @nonce_count=0, @scheme="Digest">, #<HTTPClient::ProxyBasicAuth:0x00000002d86830 @cred=nil, @set=false, @auth={}, @challengeable={}, @scheme="Basic">]>, @request_filter=[#<HTTPClient::ProxyAuth:0x00000002d86880 @basic_auth=#<HTTPClient::ProxyBasicAuth:0x00000002d86830 @cred=nil, @set=false, @auth={}, @challengeable={}, @scheme="Basic">, @negotiate_auth=#<HTTPClient::NegotiateAuth:0x00000002d86790 @auth={}, @auth_default=nil, @challenge={}, @scheme="Negotiate", @set=false, @ntlm_opt={:ntlmv2=>true}>, @ntlm_auth=#<HTTPClient::NegotiateAuth:0x00000002d866a0 @auth={}, @auth_default=nil, @challenge={}, @scheme="NTLM", @set=false, @ntlm_opt={:ntlmv2=>true}>, @sspi_negotiate_auth=#<HTTPClient::SSPINegotiateAuth:0x00000002d86600 @challenge={}, @scheme="Negotiate">, @digest_auth=#<HTTPClient::ProxyDigestAuth:0x00000002d86560 @auth={}, @challenge=nil, @set=false, @nonce_count=0, @scheme="Digest">, @authenticator=[#<HTTPClient::NegotiateAuth:0x00000002d86790 @auth={}, @auth_default=nil, @challenge={}, @scheme="Negotiate", @set=false, @ntlm_opt={:ntlmv2=>true}>, #<HTTPClient::NegotiateAuth:0x00000002d866a0 @auth={}, @auth_default=nil, @challenge={}, @scheme="NTLM", @set=false, @ntlm_opt={:ntlmv2=>true}>, #<HTTPClient::SSPINegotiateAuth:0x00000002d86600 @challenge={}, @scheme="Negotiate">, #<HTTPClient::ProxyDigestAuth:0x00000002d86560 @auth={}, @challenge=nil, @set=false, @nonce_count=0, @scheme="Digest">, #<HTTPClient::ProxyBasicAuth:0x00000002d86830 @cred=nil, @set=false, @auth={}, @challengeable={}, @scheme="Basic">]>, #<HTTPClient::WWWAuth:0x00000002d86d80 @basic_auth=#<HTTPClient::BasicAuth:0x00000002d86d58 @cred=nil, @set=false, @auth={}, @challengeable={}, @scheme="Basic">, @digest_auth=#<HTTPClient::DigestAuth:0x00000002d86cb8 @auth={}, @challenge={}, @set=false, @nonce_count=0, @scheme="Digest">, @negotiate_auth=#<HTTPClient::NegotiateAuth:0x00000002d86c18 @auth={}, @auth_default=nil, @challenge={}, @scheme="Negotiate", @set=false, @ntlm_opt={:ntlmv2=>true}>, @ntlm_auth=#<HTTPClient::NegotiateAuth:0x00000002d86b28 @auth={}, @auth_default=nil, @challenge={}, @scheme="NTLM", @set=false, @ntlm_opt={:ntlmv2=>true}>, @sspi_negotiate_auth=#<HTTPClient::SSPINegotiateAuth:0x00000002d86a10 @challenge={}, @scheme="Negotiate">, @oauth=#<HTTPClient::OAuth:0x00000002d86998 @config=nil, @auth={}, @challengeable={}, @nonce_count=0, @signature_handler={"HMAC-SHA1"=>#<Method: HTTPClient::OAuth#sign_hmac_sha1>}, @scheme="OAuth">, @authenticator=[#<HTTPClient::OAuth:0x00000002d86998 @config=nil, @auth={}, @challengeable={}, @nonce_count=0, @signature_handler={"HMAC-SHA1"=>#<Method: HTTPClient::OAuth#sign_hmac_sha1>}, @scheme="OAuth">, #<HTTPClient::NegotiateAuth:0x00000002d86c18 @auth={}, @auth_default=nil, @challenge={}, @scheme="Negotiate", @set=false, @ntlm_opt={:ntlmv2=>true}>, #<HTTPClient::NegotiateAuth:0x00000002d86b28 @auth={}, @auth_default=nil, @challenge={}, @scheme="NTLM", @set=false, @ntlm_opt={:ntlmv2=>true}>, #<HTTPClient::SSPINegotiateAuth:0x00000002d86a10 @challenge={}, @scheme="Negotiate">, #<HTTPClient::DigestAuth:0x00000002d86cb8 @auth={}, @challenge={}, @set=false, @nonce_count=0, @scheme="Digest">, #<HTTPClient::BasicAuth:0x00000002d86d58 @cred=nil, @set=false, @auth={}, @challengeable={}, @scheme="Basic">]>], @debug_dev=nil, @redirect_uri_callback=#<Method: HTTPClient#default_redirect_uri_callback>, @test_loopback_response=[], @session_manager=#<HTTPClient::SessionManager:0x00000002d863d0 @client=#<HTTPClient:0x00000002d86e48 ...>, @proxy=nil, @agent_name="HTTPClient/1.0", @from=nil, @protocol_version=nil, @debug_dev=nil, @socket_sync=true, @chunk_size=16384, @connect_timeout=60, @connect_retry=1, @send_timeout=120, @receive_timeout=60, @keep_alive_timeout=15, @read_block_size=16384, @protocol_retry_count=5, @ssl_config=#<HTTPClient::SSLConfig:0x00000002d86128 @client=#<HTTPClient:0x00000002d86e48 ...>, @cert_store=#<OpenSSL::X509::Store:0x00000002d86100 @verify_callback=nil, @error=nil, @error_string=nil, @chain=nil, @time=nil>, @client_ca=nil, @client_key=nil, @client_cert=nil, @verify_mode=3, @verify_depth=nil, @verify_callback=nil, @dest=nil, @timeout=nil, @ssl_version=:auto, @options=-2097019905, @ciphers="ALL:!aNULL:!eNULL:!SSLv2", @cacerts_loaded=false>, @test_loopback_http_response=[], @transparent_gzip_decompression=false, @socket_local=#<HTTPClient::Site:0x16c3198 tcp://0.0.0.0:0>, @sess_pool={}, @sess_pool_mutex=#<Mutex:0x00000002d86268>, @sess_pool_last_checked=2017-01-10 16:53:51 +0000>, @ssl_config=#<HTTPClient::SSLConfig:0x00000002d86128 @client=#<HTTPClient:0x00000002d86e48 ...>, @cert_store=#<OpenSSL::X509::Store:0x00000002d86100 @verify_callback=nil, @error=nil, @error_string=nil, @chain=nil, @time=nil>, @client_ca=nil, @client_key=nil, @client_cert=nil, @verify_mode=3, @verify_depth=nil, @verify_callback=nil, @dest=nil, @timeout=nil, @ssl_version=:auto, @options=-2097019905, @ciphers="ALL:!aNULL:!eNULL:!SSLv2", @cacerts_loaded=false>, @cookie_manager=#<WebAgent::CookieManager:0x00000002d860b0 @cookies=[], @cookies_file=nil, @is_saved=true, @reject_domains=[], @accept_domains=[], @netscape_rule=false>, @follow_redirect_count=10>
irb(main):003:0> @http.get('https://labtestwikitech.wikimedia.org')
at depth 1 - 20: unable to get local issuer certificate
OpenSSL::SSL::SSLError: SSL_connect returned=1 errno=0 state=error: certificate verify failed
	from /usr/lib/ruby/vendor_ruby/httpclient/session.rb:303:in `connect'
	from /usr/lib/ruby/vendor_ruby/httpclient/session.rb:303:in `ssl_connect'
	from /usr/lib/ruby/vendor_ruby/httpclient/session.rb:760:in `block in connect'
	from /usr/lib/ruby/2.1.0/timeout.rb:91:in `block in timeout'
	from /usr/lib/ruby/2.1.0/timeout.rb:101:in `call'
	from /usr/lib/ruby/2.1.0/timeout.rb:101:in `timeout'
	from /usr/lib/ruby/2.1.0/timeout.rb:127:in `timeout'
	from /usr/lib/ruby/vendor_ruby/httpclient/session.rb:751:in `connect'
	from /usr/lib/ruby/vendor_ruby/httpclient/session.rb:609:in `query'
	from /usr/lib/ruby/vendor_ruby/httpclient/session.rb:164:in `query'
	from /usr/lib/ruby/vendor_ruby/httpclient.rb:1083:in `do_get_block'
	from /usr/lib/ruby/vendor_ruby/httpclient.rb:887:in `block in do_request'
	from /usr/lib/ruby/vendor_ruby/httpclient.rb:981:in `protect_keep_alive_disconnected'
	from /usr/lib/ruby/vendor_ruby/httpclient.rb:886:in `do_request'
	from /usr/lib/ruby/vendor_ruby/httpclient.rb:774:in `request'
	from /usr/lib/ruby/vendor_ruby/httpclient.rb:677:in `get'
	from (irb):3
	from /usr/bin/irb:11:in `<main>'irb(main):004:0> 
irb(main):005:0* @http.get('https://wikitech.wikimedia.org')
=> #<HTTP::Message:0x00000002c623a0 @http_header=#<HTTP::Message::Headers:0x00000002c62378 @http_version="1.1", @body_size=0, @chunked=false, @request_method="GET", @request_uri=#<URI::HTTPS:0x00000002c62850 URL:https://wikitech.wikimedia.org>, @request_query=nil, @request_absolute_uri=nil, @status_code=301, @reason_phrase="Moved Permanently", @body_type=nil, @body_charset=nil, @body_date=nil, @body_encoding=#<Encoding:UTF-8>, @is_request=false, @header_item=[["Date", "Tue, 10 Jan 2017 16:54:10 GMT"], ["Server", "Apache"], ["Strict-Transport-Security", "max-age=31536000; includeSubDomains; preload"], ["X-Powered-By", "PHP/5.5.9-1ubuntu4.20"], ["X-Content-Type-Options", "nosniff"], ["Cache-control", "private, must-revalidate, max-age=0"], ["Vary", "Accept-Encoding,X-Forwarded-Proto,Cookie,Authorization"], ["Expires", "Thu, 01 Jan 1970 00:00:00 GMT"], ["Last-Modified", "Tue, 10 Jan 2017 16:54:10 GMT"], ["Location", "https://wikitech.wikimedia.org/wiki/Main_Page"], ["Backend-Timing", "D=60642 t=1484067250654926"], ["Content-Length", "0"], ["Content-Type", "text/html; charset=utf-8"]], @dumped=false>, @peer_cert=#<OpenSSL::X509::Certificate subject=#<OpenSSL::X509::Name:0x00000002c47e88>, issuer=#<OpenSSL::X509::Name:0x00000002c47de8>, serial=#<OpenSSL::BN:0x00000002c47cf8>, not_before=2015-12-11 20:21:11 UTC, not_after=2017-02-24 04:44:13 UTC>, @http_body=#<HTTP::Message::Body:0x00000002c622d8 @body="", @size=0, @positions=nil, @chunk_size=nil>>
irb(main):006:0> @http.ssl_config.clear_cert_store
=> {}
irb(main):007:0> @http.ssl_config.set_default_paths
=> {}
irb(main):008:0> @http.get('https://labtestwikitech.wikimedia.org')
=> #<HTTP::Message:0x00000002bef738 @http_header=#<HTTP::Message::Headers:0x00000002bef710 @http_version="1.1", @body_size=0, @chunked=false, @request_method="GET", @request_uri=#<URI::HTTPS:0x00000002befbc0 URL:https://labtestwikitech.wikimedia.org>, @request_query=nil, @request_absolute_uri=nil, @status_code=301, @reason_phrase="Moved Permanently", @body_type=nil, @body_charset=nil, @body_date=nil, @body_encoding=#<Encoding:UTF-8>, @is_request=false, @header_item=[["Date", "Tue, 10 Jan 2017 16:54:33 GMT"], ["Server", "Apache"], ["Strict-Transport-Security", "max-age=31536000; includeSubDomains; preload"], ["X-Powered-By", "PHP/5.5.9-1ubuntu4.20"], ["X-Content-Type-Options", "nosniff"], ["Cache-control", "private, must-revalidate, max-age=0"], ["Vary", "Accept-Encoding,X-Forwarded-Proto,Cookie,Authorization"], ["Expires", "Thu, 01 Jan 1970 00:00:00 GMT"], ["Last-Modified", "Tue, 10 Jan 2017 16:54:33 GMT"], ["Location", "https://labtestwikitech.wikimedia.org/wiki/Main_Page"], ["Backend-Timing", "D=77571 t=1484067273711636"], ["Content-Length", "0"], ["Content-Type", "text/html; charset=utf-8"]], @dumped=false>, @peer_cert=#<OpenSSL::X509::Certificate subject=#<OpenSSL::X509::Name:0x00000002bd6ad0>, issuer=#<OpenSSL::X509::Name:0x00000002bd6a30>, serial=#<OpenSSL::BN:0x00000002bd69b8>, not_before=2017-01-02 17:12:00 UTC, not_after=2017-04-02 17:12:00 UTC>, @http_body=#<HTTP::Message::Body:0x00000002bef670 @body="", @size=0, @positions=nil, @chunk_size=nil>>
irb(main):009:0> @http.get('https://wikitech.wikimedia.org')
=> #<HTTP::Message:0x00000002bbfe70 @http_header=#<HTTP::Message::Headers:0x00000002bbfe48 @http_version="1.1", @body_size=0, @chunked=false, @request_method="GET", @request_uri=#<URI::HTTPS:0x00000002bc4330 URL:https://wikitech.wikimedia.org>, @request_query=nil, @request_absolute_uri=nil, @status_code=301, @reason_phrase="Moved Permanently", @body_type=nil, @body_charset=nil, @body_date=nil, @body_encoding=#<Encoding:UTF-8>, @is_request=false, @header_item=[["Date", "Tue, 10 Jan 2017 16:54:37 GMT"], ["Server", "Apache"], ["Strict-Transport-Security", "max-age=31536000; includeSubDomains; preload"], ["X-Powered-By", "PHP/5.5.9-1ubuntu4.20"], ["X-Content-Type-Options", "nosniff"], ["Cache-control", "private, must-revalidate, max-age=0"], ["Vary", "Accept-Encoding,X-Forwarded-Proto,Cookie,Authorization"], ["Expires", "Thu, 01 Jan 1970 00:00:00 GMT"], ["Last-Modified", "Tue, 10 Jan 2017 16:54:37 GMT"], ["Location", "https://wikitech.wikimedia.org/wiki/Main_Page"], ["Backend-Timing", "D=73299 t=1484067277395741"], ["Content-Length", "0"], ["Content-Type", "text/html; charset=utf-8"]], @dumped=false>, @peer_cert=#<OpenSSL::X509::Certificate subject=#<OpenSSL::X509::Name:0x00000002b88740>, issuer=#<OpenSSL::X509::Name:0x00000002b886c8>, serial=#<OpenSSL::BN:0x00000002b88650>, not_before=2015-12-11 20:21:11 UTC, not_after=2017-02-24 04:44:13 UTC>, @http_body=#<HTTP::Message::Body:0x00000002bbfdd0 @body="", @size=0, @positions=nil, @chunk_size=nil>>
irb(main):010:0> @http.get('https://self-signed.badssl.com')
at depth 0 - 18: self signed certificate
OpenSSL::SSL::SSLError: SSL_connect returned=1 errno=0 state=error: certificate verify failed
	from /usr/lib/ruby/vendor_ruby/httpclient/session.rb:303:in `connect'
	from /usr/lib/ruby/vendor_ruby/httpclient/session.rb:303:in `ssl_connect'
	from /usr/lib/ruby/vendor_ruby/httpclient/session.rb:760:in `block in connect'
	from /usr/lib/ruby/2.1.0/timeout.rb:91:in `block in timeout'
	from /usr/lib/ruby/2.1.0/timeout.rb:101:in `call'
	from /usr/lib/ruby/2.1.0/timeout.rb:101:in `timeout'
	from /usr/lib/ruby/2.1.0/timeout.rb:127:in `timeout'
	from /usr/lib/ruby/vendor_ruby/httpclient/session.rb:751:in `connect'
	from /usr/lib/ruby/vendor_ruby/httpclient/session.rb:609:in `query'
	from /usr/lib/ruby/vendor_ruby/httpclient/session.rb:164:in `query'
	from /usr/lib/ruby/vendor_ruby/httpclient.rb:1083:in `do_get_block'
	from /usr/lib/ruby/vendor_ruby/httpclient.rb:887:in `block in do_request'
	from /usr/lib/ruby/vendor_ruby/httpclient.rb:981:in `protect_keep_alive_disconnected'
	from /usr/lib/ruby/vendor_ruby/httpclient.rb:886:in `do_request'
	from /usr/lib/ruby/vendor_ruby/httpclient.rb:774:in `request'
	from /usr/lib/ruby/vendor_ruby/httpclient.rb:677:in `get'
	from (irb):10
	from /usr/bin/irb:11:in `<main>'irb(main):011:0> 
irb(main):012:0* @http.get('https://untrusted-root.badssl.com')
at depth 0 - 20: unable to get local issuer certificate
OpenSSL::SSL::SSLError: SSL_connect returned=1 errno=0 state=error: certificate verify failed
	from /usr/lib/ruby/vendor_ruby/httpclient/session.rb:303:in `connect'
	from /usr/lib/ruby/vendor_ruby/httpclient/session.rb:303:in `ssl_connect'
	from /usr/lib/ruby/vendor_ruby/httpclient/session.rb:760:in `block in connect'
	from /usr/lib/ruby/2.1.0/timeout.rb:91:in `block in timeout'
	from /usr/lib/ruby/2.1.0/timeout.rb:101:in `call'
	from /usr/lib/ruby/2.1.0/timeout.rb:101:in `timeout'
	from /usr/lib/ruby/2.1.0/timeout.rb:127:in `timeout'
	from /usr/lib/ruby/vendor_ruby/httpclient/session.rb:751:in `connect'
	from /usr/lib/ruby/vendor_ruby/httpclient/session.rb:609:in `query'
	from /usr/lib/ruby/vendor_ruby/httpclient/session.rb:164:in `query'
	from /usr/lib/ruby/vendor_ruby/httpclient.rb:1083:in `do_get_block'
	from /usr/lib/ruby/vendor_ruby/httpclient.rb:887:in `block in do_request'
	from /usr/lib/ruby/vendor_ruby/httpclient.rb:981:in `protect_keep_alive_disconnected'
	from /usr/lib/ruby/vendor_ruby/httpclient.rb:886:in `do_request'
	from /usr/lib/ruby/vendor_ruby/httpclient.rb:774:in `request'
	from /usr/lib/ruby/vendor_ruby/httpclient.rb:677:in `get'
	from (irb):12
	from /usr/bin/irb:11:in `<main>'irb(main):013:0>

Change 311048 merged by Alexandros Kosiaris:
ruby-httpclient callers: Use the operating system's certificate store

https://gerrit.wikimedia.org/r/311048

Great! thanks! Patch merged

Change 331638 had a related patch set uploaded (by Alex Monk):
Use LE for wikitech

https://gerrit.wikimedia.org/r/331638

Next steps:

We need to schedule a time with the labs team (likely @Andrew) for a maint window for wikitech to apply this patch. Associated labs services may be affected (hence escalation to labs for review.)

Both myself (@RobH) and @Dzahn have converted a few services each to LE. Please let either of us know, and we can assist with the merge and application of the LE certificate.