Page MenuHomePhabricator

Expand SCB cluster
Closed, ResolvedPublic

Description

Currently, the average CPU usage on SCB nodes is about 60% which is too high for normal operation. If we loose one of the boxes we would be in trouble since a single box can't handle the load any more. Today it was forgotten to repool scb1001 for an extended period of time and Mobile-Content-Service was experiencing problems (worker deaths, monitoring flapping, occasional timeouts and 500 errors).

After transclusions processing was enabled in ChangeProp request rate to Mobile-Content-Service can grow significantly when some highly used template that changes html is edited, so the request rate could get as high as 100req/s. That obviously increases the load generated by MCS. As the amount or processing done by MSC grows, this issue is becoming more an ore pressing.

So, we either need to prioritise T96017 or get at least one more box for SCB

Event Timeline

mobrovac edited projects, added SRE, User-mobrovac; removed Blocked-on-Operations.
mobrovac subscribed.

So, we either need to prioritise T96017 or get at least one more box for SCB

After T147409: EQIAD|CODFW: (2) VM request for zotero is done, we'll be able to free the boxes and add them to SCB.

Change 316793 had a related patch set uploaded (by Mobrovac):
Scap: Add ex-SCA to the list of targets

https://gerrit.wikimedia.org/r/316793

Change 316795 had a related patch set uploaded (by Mobrovac):
Scap: Add ex-SCA to the list of targets

https://gerrit.wikimedia.org/r/316795

Change 316796 had a related patch set uploaded (by Mobrovac):
Scap: Add ex-SCA to the list of targets

https://gerrit.wikimedia.org/r/316796

Change 316797 had a related patch set uploaded (by Mobrovac):
Scap: Add ex-SCA to the list of targets

https://gerrit.wikimedia.org/r/316797

Change 316798 had a related patch set uploaded (by Mobrovac):
Scap: Add ex-SCA to the list of targets

https://gerrit.wikimedia.org/r/316798

Change 316799 had a related patch set uploaded (by Mobrovac):
Scap: Add ex-SCA to the list of targets

https://gerrit.wikimedia.org/r/316799

Change 316793 merged by Mobrovac:
Scap: Add ex-SCA to the list of targets

https://gerrit.wikimedia.org/r/316793

Change 316795 merged by Mobrovac:
Scap: Add ex-SCA to the list of targets

https://gerrit.wikimedia.org/r/316795

Change 316796 merged by Mobrovac:
Scap: Add ex-SCA to the list of targets

https://gerrit.wikimedia.org/r/316796

Change 316797 merged by Mobrovac:
Scap: Add ex-SCA to the list of targets

https://gerrit.wikimedia.org/r/316797

Change 316798 merged by Mobrovac:
Scap: Add ex-SCA to the list of targets

https://gerrit.wikimedia.org/r/316798

Change 316799 merged by Mobrovac:
Scap: Add ex-SCA to the list of targets

https://gerrit.wikimedia.org/r/316799

mobrovac claimed this task.

This is done.