Migrate beta cluster Swift cluster from Trusty to Jessie
Closed, ResolvedPublic

Description

On beta, the Swift cluster is using Ubuntu Trusty and should be migrated to Debian Jessie. The three instances are:

InstanceIPFlavor
deployment-ms-fe0110.68.16.96m1.small
deployment-ms-be0110.68.16.24m1.xlarge
deployment-ms-be0210.68.21.135m1.xlarge

Note the backend instances use m1.xlarge flavor (8 CPUs, 16G RAM and 160G disk). Can probably switch to m1.large instead (4 CPUs, 8G RAM and 80G disk) which should fit.

hashar created this task.Apr 5 2017, 9:13 AM
hashar updated the task description. (Show Details)Apr 5 2017, 9:25 AM
Paladox added a subscriber: Paladox.Apr 5 2017, 9:36 AM

Indeed we'll have to do this also because production will no longer have trusty "soon" (cfr T162609). I'll start with provisioning a jessie ms-fe since that's the easiest and will allow us to test swift 2.10 too.

@hashar for ms-be the used ram seems in the order of ~12GB so m1.large would be tight. I'll go with m1.xlarge for now, we can revisit if resources get tighter

fgiunchedi moved this task from Backlog to Doing on the User-fgiunchedi board.Apr 26 2017, 9:23 AM

Change 350389 had a related patch set uploaded (by Filippo Giunchedi):
[operations/puppet@production] swift: create required LV in labs

https://gerrit.wikimedia.org/r/350389

fgiunchedi added a comment.EditedApr 26 2017, 3:21 PM

ms-be0[34] and ms-fe02 are up and running with swift 2.10, next steps:

  • add new backends to the deployment-prep ring in operations/software/swift-ring.git at e.g. weight 300 and wait for rebalance
  • increase new backend weights to 1000 and rebalance
  • decrease old backend weights to 300 / rebalance / to 0 / rebalance
  • point mediawiki to deployment-ms-fe02
  • deprovision all old trusty machines

Mentioned in SAL (#wikimedia-releng) [2017-04-26T15:24:40Z] <godog> add new deployment-ms-be0[34] backends to swift in deployment-prep - T162247

For reference, switching mw to talk to deployment-ms-fe02 the configuration is here: https://horizon.wikimedia.org/project/prefixpuppet/?tab=prefix_puppet__puppet-deployment-cache-upload for the varnish bits and here https://wikitech.wikimedia.org/wiki/Hiera:Deployment-prep for swift

Mentioned in SAL (#wikimedia-releng) [2017-05-02T09:41:37Z] <godog> flip deployment-cache-upload04 to deployment-ms-fe02 - T162247

Also private/PrivateSettings.php needed a change on deployment-tin to pick up deployment-ms02

Change 351285 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] swift: default to 127.0.0.1 for memcached

https://gerrit.wikimedia.org/r/351285

I'm going to pause (but not terminate) old deployment-ms trusty hosts since they should be completely decomissioned now. Will terminate say next week if no issues come up.

Change 351285 merged by Filippo Giunchedi:
[operations/puppet@production] swift: default to 127.0.0.1 for memcached

https://gerrit.wikimedia.org/r/351285

fgiunchedi closed this task as Resolved.May 8 2017, 1:47 PM

All done, old trusty instances are being deleted

Change 350389 merged by Filippo Giunchedi:
[operations/puppet@production] profile: introduce swift::storage::labs

https://gerrit.wikimedia.org/r/350389