Page MenuHomePhabricator

Future of Thumbor's memcached backend
Closed, ResolvedPublic

Description

Background: https://wikitech.wikimedia.org/wiki/Thumbor#Throttling

Why?

Thumbor is using Memcached via Nutcracker. Right now, Thumbor's Memcached backend is a group of small Memcached instances running on each physical Thumbor host, accessed (and sharded) via Nutcracker. Nutcraker is a software we would like to completely retire from our infrastructure, while, with Thumbor's migration to K8s, the physical servers will be going away.

What?

While we are still working on the details as to how Mcrouter will exist in kubernetes, we can proceed in liberating the bare metal thumbor servers from the weight of having a memcached instance.

Possible solutions:

  • Create 2 VMs in each DC. One server to be the main memcached host for thumbor, and the second to be the "gutter pool".
  • Use our main memcached cluster
  • Use the wikifunctions cluster

We should evaluate the solutions for thumbor's needs, as well as if the first proposed solution (use mcrouter+memcached VMs), is like bringing a steamroller to cover a hole in the ground

How?

  • update thumbor's chart and helmfile to include mcrouter
  • temporarily, we will use mediawiki's memcached cluster for this (with complete understanding that this is an overkill)
  • start mcrouter on a different port, and ensure it works as it should
  • make the appropriate changes so thumbor will start using mcrouter
  • kill nutcracker with fire

Extras:

  • we could consider spinning up 2 small VMs to server thumbor's memached needs, but this is TBD

Event Timeline

jijiki renamed this task from Replace nutcracker with mcrouter to Improve Thumbor's memcached infrastructure.May 15 2023, 2:07 PM
jijiki updated the task description. (Show Details)
jijiki renamed this task from Improve Thumbor's memcached infrastructure to Improve Thumbor's memcached backend.May 15 2023, 2:46 PM
jijiki updated the task description. (Show Details)
jijiki updated the task description. (Show Details)
jijiki renamed this task from Improve Thumbor's memcached backend to Future of Thumbor's memcached backend.Jun 1 2023, 9:49 AM
jijiki updated the task description. (Show Details)
jijiki updated the task description. (Show Details)
jijiki removed a subscriber: Gilles.

fwiw I see no reason not to move to mcrouter

Change 937141 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/deployment-charts@master] thumbor: add mcrouter support

https://gerrit.wikimedia.org/r/937141

Change 937444 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/deployment-charts@master] thumbor: helmfile changes for mcrouter support

https://gerrit.wikimedia.org/r/937444

Change 937141 merged by jenkins-bot:

[operations/deployment-charts@master] thumbor: add mcrouter support

https://gerrit.wikimedia.org/r/937141

Change 937444 merged by jenkins-bot:

[operations/deployment-charts@master] thumbor: helmfile changes for mcrouter support

https://gerrit.wikimedia.org/r/937444

Change 937524 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/deployment-charts@master] thumbor: add failure_throttling_memcache variable

https://gerrit.wikimedia.org/r/937524

Change 937524 merged by jenkins-bot:

[operations/deployment-charts@master] thumbor: add failure_throttling_memcache variable

https://gerrit.wikimedia.org/r/937524

Change 937939 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/deployment-charts@master] thumbor: switch production to use mcrouter

https://gerrit.wikimedia.org/r/937939

Change 937939 merged by jenkins-bot:

[operations/deployment-charts@master] thumbor: switch production to use mcrouter

https://gerrit.wikimedia.org/r/937939

Change 938001 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/deployment-charts@master] thumbor: Bye bye nutcracker!

https://gerrit.wikimedia.org/r/938001

Change 938001 merged by jenkins-bot:

[operations/deployment-charts@master] thumbor: Bye bye nutcracker!

https://gerrit.wikimedia.org/r/938001

Mentioned in SAL (#wikimedia-operations) [2023-07-17T13:43:32Z] <akosiaris> deploy removal of nutcracker from thumbor. T318695

akosiaris subscribed.

@hnowlan @jijiki. nutcracker removal merged and deployed. I am gonna let you have the pleasure of resolving this task :-)

@hnowlan @jijiki. nutcracker removal merged and deployed. I am gonna let you have the pleasure of resolving this task :-)

So would someone like to resolve (assuming this task has reached its end of life)?

hnowlan assigned this task to jijiki.