Page MenuHomePhabricator

Switch AQS to new cluster
Closed, ResolvedPublic8 Estimated Story Points

Description

Switch AQS to new cluster

Event Timeline

Things to have in mind:

Aqs code is slightly different on new and old cluster as compression scheme is hardcoded as part of aqs setup.

Communicate switchover but downtime is not expected. We need to choose a date.

Cluster switch - we'll add the new nodes behind the AQS LVS load balancer leaving the current ones running, and then we'll deprecate old hosts if nothing major appears during the following days.

Remove old hosts from LVS and monitor.

Once it has baked for days, we remove loading job for old hosts.

Bring new cluster up to date with newer aqs code.

Nuria set the point value for this task to 8.Sep 8 2016, 4:09 PM
Milimetric triaged this task as Medium priority.

Change 310831 had a related patch set uploaded (by Elukey):
Add the new aqs nodes to conftool-data

https://gerrit.wikimedia.org/r/310831

Change 310831 merged by Elukey:
Add the new aqs nodes to conftool-data

https://gerrit.wikimedia.org/r/310831

Mentioned in SAL (#wikimedia-operations) [2016-09-19T14:21:01Z] <elukey> adding aqs1004 to live traffic - aqs.svc.eqiad.wmnet - T144497

Mentioned in SAL (#wikimedia-operations) [2016-09-20T16:58:23Z] <elukey> adding aqs1005 to live traffic - aqs.svc.eqiad.wmnet - T144497

Mentioned in SAL (#wikimedia-operations) [2016-09-20T17:01:53Z] <elukey> adding aqs1006 to live traffic - aqs.svc.eqiad.wmnet - T144497

Mentioned in SAL (#wikimedia-operations) [2016-09-21T06:21:06Z] <elukey> removing aqs100[123] from live traffic - aqs.svc.eqiad.wmnet - T144497

Change 314284 had a related patch set uploaded (by Nuria):
Bringing master up to date with aqs-new-cluster branch

https://gerrit.wikimedia.org/r/314284

Change 314284 abandoned by Nuria:
Bringing master up to date with aqs-new-cluster branch

https://gerrit.wikimedia.org/r/314284

Change 314722 had a related patch set uploaded (by Nuria):
Updating master with new-aqs-cluster branch

https://gerrit.wikimedia.org/r/314722

Change 314722 abandoned by Nuria:
Updating master with new-aqs-cluster branch

Reason:
not needed

https://gerrit.wikimedia.org/r/314722