Page MenuHomePhabricator

Enable HTTPS for swift clients
Closed, ResolvedPublic

Description

MediaWiki

Swift is available over HTTPS internally now, mediawiki's FileBackend should be configured to use https instead of http. This could be a simple configuration change, though it was mentioned at the active/active datacenter meeting that Discovery had to do some mw change when activating https for search? (cc @Gehel @dcausse)

Done

SwiftRepl

Needs to contact both source and destination over https

Thumbor

Ditto, switch swift communication to https

Done

Details

Related Gerrit Patches:
operations/mediawiki-config : masterSwitch Swift URLs to HTTPs
mediawiki/core : masterHandle proxy-based TLS when placed in front of Swift

Related Objects

StatusAssignedTask
OpenNone
Resolvedaaron

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 16 2017, 9:15 AM
Gehel added a comment.Mar 16 2017, 9:23 AM

When activating HTTPS for mediawiki -> elasticsearch traffic, we had to enable HTTP connection pooling to mitigate the SSL handshake overhead. This was particularly important for cross DC traffic (increased latency), where the handshake overhead was on the same order of magnitude than the actual payload. This is particularly important for search where a lot of requests are answered in < 10ms.

dcausse added a comment.EditedMar 16 2017, 9:31 AM

Connection pooling is not available out of the box and only supported by HHVM.
The code needed to enable connection pooling is not available in a generic fashion in mediawiki core but here is what we use for Cirrus:
https://github.com/wikimedia/mediawiki-extensions-CirrusSearch/blob/master/includes/Elastica/PooledHttp.php

Note that currently connection pools are configured in puppet and cannot be changed on the fly, this is currently done in puppet in modules/mediawiki/manifests/hhvm.pp (see namedPools in this file)

So in short if this is a high volume service connection pooling might be required in situations where mediawiki and swift run in different datacenter.
Drawbacks:

  • no generic support in mediawiki-core, need to copy CirrusSearch implementation
  • only supported by HHVM
  • pools configuration is done in puppet and require HHVM restart (IIRC)

(Adding @EBernhardson to this ticket as he is the author of all of this)

Restricted Application added a project: Traffic. · View Herald TranscriptMar 16 2017, 9:52 AM
ema moved this task from Triage to Watching on the Traffic board.Mar 28 2017, 9:36 AM
aaron added a subscriber: aaron.Mar 29 2017, 7:39 PM

SwiftFileBackend will need to force an https URL when it gets the storage_url back in the JSON auth response.

Change 345430 had a related patch set uploaded (by Aaron Schulz):
[mediawiki/core@master] Handle proxy-based TLS when placed in front of Swift

https://gerrit.wikimedia.org/r/345430

Change 345430 merged by jenkins-bot:
[mediawiki/core@master] Handle proxy-based TLS when placed in front of Swift

https://gerrit.wikimedia.org/r/345430

Change 355174 had a related patch set uploaded (by Aaron Schulz; owner: Aaron Schulz):
[operations/mediawiki-config@master] Switch Swift URLs to HTTPs

https://gerrit.wikimedia.org/r/355174

Change 355174 merged by jenkins-bot:
[operations/mediawiki-config@master] Switch Swift URLs to HTTPs

https://gerrit.wikimedia.org/r/355174

Gilles added a subscriber: Gilles.May 30 2017, 1:55 PM

@aaron you mentioned in your weekly notes that this had a minor performance effect. Positive or negative? How much are we talking about?

fgiunchedi updated the task description. (Show Details)Jun 7 2017, 3:55 PM
aaron added a comment.Jun 7 2017, 4:35 PM

Deploy was 00:18 May 26 UTC, and

I can't discern an effect on api upload entry point runtime.

Gehel removed a subscriber: Gehel.Jun 20 2017, 1:35 PM
aaron added a comment.EditedJun 28 2017, 7:24 PM

Looks like SwiftRepl is the last element of this task, and part of https://www.mediawiki.org/wiki/Wikimedia_Technology/Goals/2017-18_Q1#Performance

@fgiunchedi , how difficult does look to add to the replication script? I know you said that script needs some rewritting.

@aaron yeah it will need some love, in the meantime I've patched in https support so swiftrepl will DTRT.

Looks like this completes the list

aaron added a comment.Jul 5 2017, 11:56 PM

@aaron yeah it will need some love, in the meantime I've patched in https support so swiftrepl will DTRT.
Looks like this completes the list

Thanks!

aaron closed this task as Resolved.Jul 5 2017, 11:56 PM
aaron claimed this task.