Page MenuHomePhabricator

Upgrade memcached for Debian Stretch/Buster
Open, MediumPublic

Description

In T129963 we explored some newer versions of memcache to deploy for the MW object cache, but we have never decided to upgrade all the shards. On all the mcXXXX hosts we are running Jessie, and sooner or later we'll have to think about either Stretch or Buster :)

The major complication is the fact that Redis and the MW Session Storage is co-located on the same nodes, so upgrading the OS means upgrading both Memcached and Redis at the same time.

While we could wait for the new Session Storage Service to be alive (that should in theory get rid of Redis in favor of something else), I would like to choose a new version of memcached and try it on a couple of Production shards for a couple of months to study and tune settings, since from T129963 we know that a lot has changed. Some highlights:

  • the maximum number of slab classes for a "recent" 1.4 or 1.5 version of memcached is 64, meanwhile we are currently using a lot more (160+) on each shard due to the growth factor that we use. In T129963 we tested the increase of te growth factor to 1.15, it seemed working nicely.
  • the LRU logic has been completely changed, more info in https://github.com/memcached/memcached/blob/master/doc/new_lru.txt and https://memcached.org/blog/modern-lru
  • SLAB automover - freed memory can be reclaimed back into a global pool and reassigned to new slab classes (currently memory assigned to a slab class cannot be reclaimed, even if free, for another use).
  • new features are now ready to use and tested by a lot of people already.

One solution could be to decide a version to test/use (either Stretch's or Buster's), backport it to Jessie (that shouldn't be too difficult in theory) and start the testing in deployment-prep/prod as soon as possible, to be ready for a reimage to Stretch/Buster when the time comes.

Details

Related Gerrit Patches:
operations/puppet : productionrole::prometheus::beta: add memcached metrics
operations/puppet : productionAdd deployment-memc08 to the mcrouter config

Event Timeline

elukey triaged this task as Medium priority.Jan 7 2019, 3:50 PM
elukey created this task.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 7 2019, 3:50 PM
Paladox added a subscriber: Paladox.Jan 7 2019, 8:56 PM

When backporting this to jessie, we'll need to carefully review the systemd hardening options used in the memcached systemd unit, some of the feature are likely not yet supported in the systemd version in jessie.

Given that the Debian buster release isn't too far away and has 1.5.6 (and we could ask the maintainer if an update to 1.5.12 is in the works), I'd prefer to test with a backport of 1.5.6, by the time an update in production is ready, buster will be released and it feels like a bit of a waste to perform a massive migration effort to stretch for something we'd soon need to upgrade again.

elukey added a comment.Jan 9 2019, 3:55 PM

I'd prefer 1.5.6 too, we'd be really close to upstream (atm 1.5.12) and getting help from them would surely be easier if needed. I'd also love to be able to provide feedback to the memcached project about scalability and/or bottlenecks of running a recent version at scale (for example, LRU special use case, etc..).

The downside is of course that 1.5.6 is a lot different than our current version, so extensive testing will be needed :)

elukey added a subscriber: faidon.Jan 11 2019, 9:50 AM
jijiki added a subscriber: jijiki.Jan 18 2019, 11:40 AM
jijiki removed a subscriber: elukey.
jijiki moved this task from Backlog to Incoming on the serviceops board.Jan 18 2019, 12:03 PM
jijiki added a subscriber: elukey.Jan 18 2019, 1:55 PM
elukey updated the task description. (Show Details)Feb 9 2019, 2:37 PM
elukey added a comment.EditedFeb 9 2019, 2:39 PM

Found https://github.com/memcached/memcached/issues/446 today, in which upstream warns about stability issues with 1.5.6 that should have been resolved with 1.5.12. Let's keep it in mind when testing new versions :)

EDIT: after a chat with upstream it was suggested to me to follow up with Debian to avoid shipping 1.5.6 since it contains some bugs resolved in later versions. I'll try to follow up with Debian upstream asap!

EDIT: after a chat with upstream it was suggested to me to follow up with Debian to avoid shipping 1.5.6 since it contains some bugs resolved in later versions. I'll try to follow up with Debian upstream asap!

I got an answer from the Debian memcached maintainer and he was not able to upload 1.5.12 before the Buster freeze, so that version will be probably offered as Buster backport.

Leaving here also a reference of https://github.com/memcached/memcached/issues/359:

Regression in systemd-based sandboxing in 1.5.6

Leaving here also a reference of https://github.com/memcached/memcached/issues/359:

Regression in systemd-based sandboxing in 1.5.6

Can you file a bug in Debian, please? Given that testing has 1.5.6 this can still be cherrypicked despite buster being frozen for new upstream releases.

elukey moved this task from Backlog to Stalled on the User-Elukey board.Apr 15 2019, 12:55 PM
elukey moved this task from Stalled to Backlog on the User-Elukey board.Apr 16 2019, 11:02 AM
Joe moved this task from Incoming to Backlog on the serviceops board.Jun 21 2019, 8:45 AM
elukey moved this task from Backlog to Mcrouter/Memcached on the User-Elukey board.Jul 5 2019, 6:53 AM

Leaving here also a reference of https://github.com/memcached/memcached/issues/359:

Regression in systemd-based sandboxing in 1.5.6

Can you file a bug in Debian, please? Given that testing has 1.5.6 this can still be cherrypicked despite buster being frozen for new upstream releases.

Didn't follow up on this, but from the github issue it seems that Debian is not affected: https://github.com/memcached/memcached/issues/359#issuecomment-372938725

I have been discussing how to proceed with @jijiki recently, and T208934 will be prioritized otherwise each reimage of a mc host will cause impact to all the api/appservers mcrouters (since we don't have a separate cluster to failover to yet).

To test Buster and memcached: one idea is to install buster on the hosts that we'll buy for the failover cluster, and test settings etc.. in there first.

I have been discussing how to proceed with @jijiki recently, and T208934 will be prioritized otherwise each reimage of a mc host will cause impact to all the api/appservers mcrouters (since we don't have a separate cluster to failover to yet).
To test Buster and memcached: one idea is to install buster on the hosts that we'll buy for the failover cluster, and test settings etc.. in there first.

+1 !

Settings that we might want to use with memcached 1.5.6 (Buster's version):

  • -o modern: alias for the following settings on slab_reassign, slab_automove, lru_crawler, lru_maintainer, maxconns_fast, and hash_algorithm=murmur3
  • growth_factor set to 1.15 or greater. We currently use 1.05 to have more granularity in the distribution of the slab classes, but memcached changed a lot of internals and now only a maximum of 64 slab classes are allowed.

Mentioned in SAL (#wikimedia-releng) [2019-10-16T08:57:33Z] <elukey> created deployment-memc08 in deployment-prep as memcached test host for Buster - T213089

Mentioned in SAL (#wikimedia-operations) [2019-10-16T10:31:00Z] <elukey> upload prometheus-memcached-exporter 0.4.1+git20181010.2fa99eb-1+deb10u1 to buster-wikimedia - T213089

Change 543456 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Add deployment-memc08 to the mcrouter config

https://gerrit.wikimedia.org/r/543456

Change 543456 merged by Elukey:
[operations/puppet@production] Add deployment-memc08 to the mcrouter config

https://gerrit.wikimedia.org/r/543456

Mentioned in SAL (#wikimedia-releng) [2019-10-30T09:45:12Z] <elukey> mediawiki mcrouter's on deployment-prep are now using deployment-memc-08 (Buster node with memcached) - to report any issue T213089

Change 548264 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] role::prometheus::beta: add memcached metrics

https://gerrit.wikimedia.org/r/548264

Change 548264 merged by Elukey:
[operations/puppet@production] role::prometheus::beta: add memcached metrics

https://gerrit.wikimedia.org/r/548264

elukey added a comment.Nov 5 2019, 2:25 PM

Added metrics to http://beta-prometheus.wmflabs.org about memcached:

http://beta-prometheus.wmflabs.org/beta/graph?g0.range_input=12h&g0.expr=memcached_current_connections&g0.tab=0

One thing that I realize is that the prometheus-memcached-exporter probably doesn't display all the metrics available in memcached 1.5.x, so we should fix that :)

elukey added a comment.Nov 8 2019, 5:40 PM

stat items is something that changed a bit. For example, take slab 63:

  • Jessie
STAT items:63:number 1431
STAT items:63:age 1756986
STAT items:63:evicted 0
STAT items:63:evicted_nonzero 0
STAT items:63:evicted_time 0
STAT items:63:outofmemory 0
STAT items:63:tailrepairs 0
STAT items:63:reclaimed 190
STAT items:63:expired_unfetched 113
STAT items:63:evicted_unfetched 0
STAT items:63:crawler_reclaimed 0
STAT items:63:lrutail_reflocked 0
  • Buster
STAT items:63:number 2604
STAT items:63:number_hot 384
STAT items:63:number_warm 543
STAT items:63:number_cold 1677
STAT items:63:age_hot 25484
STAT items:63:age_warm 128596
STAT items:63:age 128596
STAT items:63:evicted 0
STAT items:63:evicted_nonzero 0
STAT items:63:evicted_time 0
STAT items:63:outofmemory 0
STAT items:63:tailrepairs 0
STAT items:63:reclaimed 439
STAT items:63:expired_unfetched 14904
STAT items:63:evicted_unfetched 0
STAT items:63:evicted_active 0
STAT items:63:crawler_reclaimed 25554
STAT items:63:crawler_items_checked 2085802
STAT items:63:lrutail_reflocked 0
STAT items:63:moves_to_cold 10859
STAT items:63:moves_to_warm 5007
STAT items:63:moves_within_lru 1107
STAT items:63:direct_reclaims 0
STAT items:63:hits_to_hot 542290
STAT items:63:hits_to_warm 175335
STAT items:63:hits_to_cold 4970
STAT items:63:hits_to_temp 0

That seem supported in the prometheus exporter from version 0.6, we have 0.4.1. Next step is then to upgrade the exporter!

Build and deployed 0.6.0 and deployed on deployment-prep-memc08. Also created the following:

https://grafana-labs.wikimedia.org/d/000000317/memcache-slabs
https://grafana-labs.wikimedia.org/d/000000316/memcache

Still need to add the new metrics!

Not all new slab metrics are rendered, opened an issue upstream: https://github.com/prometheus/memcached_exporter/issues/75

I'll send a pull request in case they will like the idea of including them.

Not all new slab metrics are rendered, opened an issue upstream: https://github.com/prometheus/memcached_exporter/issues/75
I'll send a pull request in case they will like the idea of including them.

https://github.com/prometheus/memcached_exporter/pull/76

elukey added a comment.Mon, Dec 9, 2:40 PM

Finally after a long back and forth with upstream I was able to have my pull request merged. Built the new exporter and uploaded to memc08 in deployment prep, new metrics plotted in the 1.5.x section of:

https://grafana-labs.wikimedia.org/d/000000317/memcache-slabs?orgId=1&var-datasource=Beta%20Prometheus&var-cluster=misc&var-instance=deployment-memc08&var-slab=All