Page MenuHomePhabricator

Expand SCB cluster
Closed, ResolvedPublic

Description

Currently, the average CPU usage on SCB nodes is about 60% which is too high for normal operation. If we loose one of the boxes we would be in trouble since a single box can't handle the load any more. Today it was forgotten to repool scb1001 for an extended period of time and Mobile-Content-Service was experiencing problems (worker deaths, monitoring flapping, occasional timeouts and 500 errors).

After transclusions processing was enabled in ChangeProp request rate to Mobile-Content-Service can grow significantly when some highly used template that changes html is edited, so the request rate could get as high as 100req/s. That obviously increases the load generated by MCS. As the amount or processing done by MSC grows, this issue is becoming more an ore pressing.

So, we either need to prioritise T96017 or get at least one more box for SCB

Details

Related Gerrit Patches:
mediawiki/services/mobileapps/deploy : masterScap: Add ex-SCA to the list of targets
mediawiki/services/mathoid/deploy : masterScap: Add ex-SCA to the list of targets
mediawiki/services/graphoid/deploy : masterScap: Add ex-SCA to the list of targets
mediawiki/services/cxserver/deploy : masterScap: Add ex-SCA to the list of targets
mediawiki/services/citoid/deploy : masterScap: Add ex-SCA to the list of targets
mediawiki/services/change-propagation/deploy : masterScap: Add ex-SCA to the list of targets

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 12 2016, 12:03 AM
mobrovac triaged this task as High priority.Oct 12 2016, 1:16 PM
mobrovac edited projects, added Operations, User-mobrovac; removed Blocked-on-Operations.
mobrovac added a subscriber: mobrovac.

So, we either need to prioritise T96017 or get at least one more box for SCB

After T147409: EQIAD|CODFW: (2) VM request for zotero is done, we'll be able to free the boxes and add them to SCB.

GWicke moved this task from Backlog to watching on the Services board.Oct 12 2016, 3:56 PM
GWicke edited projects, added Services (watching); removed Services.

Change 316793 had a related patch set uploaded (by Mobrovac):
Scap: Add ex-SCA to the list of targets

https://gerrit.wikimedia.org/r/316793

Change 316795 had a related patch set uploaded (by Mobrovac):
Scap: Add ex-SCA to the list of targets

https://gerrit.wikimedia.org/r/316795

Change 316796 had a related patch set uploaded (by Mobrovac):
Scap: Add ex-SCA to the list of targets

https://gerrit.wikimedia.org/r/316796

Change 316797 had a related patch set uploaded (by Mobrovac):
Scap: Add ex-SCA to the list of targets

https://gerrit.wikimedia.org/r/316797

Change 316798 had a related patch set uploaded (by Mobrovac):
Scap: Add ex-SCA to the list of targets

https://gerrit.wikimedia.org/r/316798

Change 316799 had a related patch set uploaded (by Mobrovac):
Scap: Add ex-SCA to the list of targets

https://gerrit.wikimedia.org/r/316799

Change 316793 merged by Mobrovac:
Scap: Add ex-SCA to the list of targets

https://gerrit.wikimedia.org/r/316793

Change 316795 merged by Mobrovac:
Scap: Add ex-SCA to the list of targets

https://gerrit.wikimedia.org/r/316795

Change 316796 merged by Mobrovac:
Scap: Add ex-SCA to the list of targets

https://gerrit.wikimedia.org/r/316796

Change 316797 merged by Mobrovac:
Scap: Add ex-SCA to the list of targets

https://gerrit.wikimedia.org/r/316797

Change 316798 merged by Mobrovac:
Scap: Add ex-SCA to the list of targets

https://gerrit.wikimedia.org/r/316798

Change 316799 merged by Mobrovac:
Scap: Add ex-SCA to the list of targets

https://gerrit.wikimedia.org/r/316799

mobrovac closed this task as Resolved.Oct 19 2016, 2:20 PM
mobrovac claimed this task.

This is done.