Page MenuHomePhabricator
Create Task
Maniphest T352885

Enable extstore to a subset of memcached servers (experiment)
Open, Stalled, MediumPublic
Actions

Assigned To
jijiki
Authored By
jijiki
Dec 6 2023, 4:40 PM
Tags
Referenced Files
F54932370: image.png
Jun 4 2024, 11:22 AM
F54932229: image.png
Jun 4 2024, 11:22 AM
F54931929: image.png
Jun 4 2024, 11:22 AM
F54931668: image.png
Jun 4 2024, 11:22 AM
Subscribers
aaron
Aklapper
jijiki
Krinkle
tstarling

Description

What

Extstore is a memcached feature extending memcached's memory space. Instead of completely evicting a key (to make room for new ones), memcached leaves the hash table and keys in memory, but moves values to external storage (disk, flash, whatever)

Details can be found here: https://github.com/memcached/memcached/wiki/Extstore

Why?

We have slabs with higher eviction rate than others, and it is currently unknown how those evictions impact production (though we could investigate this in a separate tasks)

Enabling extstore in production so to determine if has positive impact or not, is a low effort project.

How?

  • puppet changes
  • hiera keys to enable/disable the feature
  • update prometheus-memcached-exporter to v0.14.2 T350807
  • update grafana dashboards to include extstore metrics memcached-Slabs and memcache

eqiad

  • mc1045.eqiad.wmnet
  • mc1046.eqiad.wmnet
  • mc1047.eqiad.wmnet
  • mc1048.eqiad.wmnet
  • mc1049.eqiad.wmnet
  • mc1050.eqiad.wmnet

codfw

  • mc2045.codfw.wmnet
  • mc2046.codfw.wmnet
  • mc2047.codfw.wmnet
  • mc2048.codfw.wmnet
  • mc2049.codfw.wmnet
  • mc2050.codfw.wmnet

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
memcached: enable extstore on codfwoperations/puppetproduction+9 -15
memcached: enable extstore to eqiad onlyoperations/puppetproduction+3 -17
memcached: enable extstore on mc1050/mc2050operations/puppetproduction+13 -0
memcached: test extstore on 10 serversoperations/puppetproduction+15 -20
hieradata: enable extstore on mc1049 and mc2049operations/puppetproduction+18 -0
memcached: add extstore optionoperations/puppetproduction+38 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

jijiki created this task.Dec 6 2023, 4:40 PM
jijiki triaged this task as Medium priority.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 6 2023, 4:40 PM
jijiki updated the task description. (Show Details)Dec 6 2023, 4:42 PM
jijiki added a subtask: T350807: Package latest version of prometheus-memcached-exporter (v0.14.2).
jijiki added a parent task: T352880: Memcache improvements and essential work (FY 23-24).
jijiki added a subtask: T273950: Modernise memcached systemd unit / sync, and make it presentable.Dec 6 2023, 4:56 PM
jijiki moved this task from Incoming 🐫 to this.quarter 🍕 on the serviceops board.Jan 9 2024, 9:52 AM
jijiki closed subtask T350807: Package latest version of prometheus-memcached-exporter (v0.14.2) as Resolved.Apr 11 2024, 11:29 AM
Andrew reopened subtask T350807: Package latest version of prometheus-memcached-exporter (v0.14.2) as Open.Apr 17 2024, 2:13 PM
jijiki closed subtask T350807: Package latest version of prometheus-memcached-exporter (v0.14.2) as Resolved.Apr 23 2024, 2:15 PM
gerritbot added a comment.May 24 2024, 7:31 AM

Change #1035633 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/puppet@production] memcached: add extstore option

https://gerrit.wikimedia.org/r/1035633

gerritbot added a project: Patch-For-Review.May 24 2024, 7:31 AM
jijiki changed the task status from Open to In Progress.May 24 2024, 11:49 AM
jijiki claimed this task.
jijiki changed the status of subtask T273950: Modernise memcached systemd unit / sync, and make it presentable from Open to In Progress.
jijiki updated the task description. (Show Details)
Krinkle added a project: MediaWiki-Platform-Team (Radar).May 24 2024, 11:06 PM
Krinkle subscribed.
gerritbot added a comment.May 27 2024, 2:45 PM

Change #1036281 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/puppet@production] hieradata: enable extstore on mc1049 and mc2049

https://gerrit.wikimedia.org/r/1036281

gerritbot added a comment.May 27 2024, 2:58 PM

Change #1035633 merged by Effie Mouzeli:

[operations/puppet@production] memcached: add extstore option

https://gerrit.wikimedia.org/r/1035633

gerritbot added a comment.May 27 2024, 3:25 PM

Change #1036281 merged by Effie Mouzeli:

[operations/puppet@production] hieradata: enable extstore on mc1049 and mc2049

https://gerrit.wikimedia.org/r/1036281

Maintenance_bot removed a project: Patch-For-Review.May 27 2024, 3:31 PM
gerritbot added a comment.May 29 2024, 11:31 AM

Change #1037053 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/puppet@production] memcached: test extstore on 10 servers

https://gerrit.wikimedia.org/r/1037053

gerritbot added a project: Patch-For-Review.May 29 2024, 11:31 AM
gerritbot added a comment.May 29 2024, 12:02 PM

Change #1037053 merged by Effie Mouzeli:

[operations/puppet@production] memcached: test extstore on 10 servers

https://gerrit.wikimedia.org/r/1037053

Maintenance_bot removed a project: Patch-For-Review.May 29 2024, 12:31 PM
gerritbot added a comment.Jun 3 2024, 11:16 AM

Change #1038262 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/puppet@production] memcached: enable extstore on mc1050/mc2050

https://gerrit.wikimedia.org/r/1038262

gerritbot added a project: Patch-For-Review.Jun 3 2024, 11:16 AM
jijiki updated the task description. (Show Details)Jun 3 2024, 11:18 AM
jijiki added a subscriber: aaron.
gerritbot added a comment.Jun 3 2024, 11:32 AM

Change #1038262 merged by Effie Mouzeli:

[operations/puppet@production] memcached: enable extstore on mc1050/mc2050

https://gerrit.wikimedia.org/r/1038262

Maintenance_bot removed a project: Patch-For-Review.Jun 3 2024, 12:31 PM
jijiki changed the status of subtask T273950: Modernise memcached systemd unit / sync, and make it presentable from In Progress to Open.Jun 4 2024, 8:42 AM
jijiki added a comment.EditedJun 4 2024, 11:22 AM

We have enabled extstore to 6/18 servers per memcached cluster. A few observations:

  • System: apart from the expected disc ops and increase in cached memory, nothing stands out
  • We do not have items being evicted before expiring, expected as they are moved to extstore.
  • hitrate is almost the same

I have noticed there are differences in the patterns of some slabs. Comparison was on mc1050 which
I switched to extstore a few days after it was running on bookworm.

  • 1d offset is before enabling extstore on mc1050
  • objects over 176B are eligible to be moved to extstore

Slab 4 (max 152B, yellow offset 1d)
current items and cas hits

image.png (490×1 px, 64 KB)

image.png (470×1 px, 157 KB)

Slab 10 (max 384B, yellow offset 1d)
memory requested

image.png (470×1 px, 68 KB)

Slab 30 (max 7k, yellow offset 1d)
memory requested

image.png (457×1 px, 76 KB)

I am not sure yet how this affects performance overall. cc @Krinkle or whoever would like to dig further with me. We could consider rolling this change on one for our 2 DCs, and dig through how other metrics are affected (or not?), eg mw-* latency, DB traffic etc.

jijiki changed the status of subtask T273950: Modernise memcached systemd unit / sync, and make it presentable from Open to In Progress.Jun 4 2024, 3:13 PM
Krinkle added a subscriber: tstarling.Jul 5 2024, 12:30 AM

We discussed this in the MwEng-SvcOps meeting (13 June 2024). Extstore was enabled on a few hosts in the DC. Some stats differered, but we weren't able to come up with a testing strategy to prove or disprove an observable benefit from the MW application in this state since keys are sharded across all hosts, and traffic generally involves many different keys.

We noticed that the given hosts, contrary to my own expectations, there is a continuous non-zero trickle of evictions of tiny keys (e.g. < 200 bytes). This is surprising, because during research for T278392 and T336004, we @tstarling and I found that these slabs were not under pressure, and emperical testing showed that these values reliably persisted for at least a minute in practice. This is worrying, because we've since migrated to mcrouter-primary-dc for short-lived auth/nonce tokens, Rdbms-ChronologyProtector positions, and rate limiter counters.

The good news, and the reason we noticed, is that with extstore enabled, much more space is given to the tiny value "slab 4", and there is a flat line of zero undue evictions since.

Aside from this little happy accident, we found no per-host stats that are reason for concern. I agreed with Effie that rolling it to fully to the primary DC with the secondary as A/B-esque control would be a fine next step. We expect the worst outcome to be "no effect". And hope that there is enough similarity between the DCs traffic, yet enough separation in our data collection to notice an improvement here (e.g. via MW statds->prometheus stats that we have per-DC, using WANCache as way to measure cache hit ratio on meaningful keys; as well as Apache-level latency numbers such as the Appserver RED dashboards).

gerritbot added a comment.Jul 8 2024, 1:22 PM

Change #1052739 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/puppet@production] memcached: enable extstore to eqiad only

https://gerrit.wikimedia.org/r/1052739

gerritbot added a project: Patch-For-Review.Jul 8 2024, 1:22 PM
Stashbot added a comment.Jul 10 2024, 2:15 PM

Mentioned in SAL (#wikimedia-operations) [2024-07-10T14:15:49Z] <effie> disable puppet on mw memcached hosts - T352885

gerritbot added a comment.Jul 11 2024, 8:41 AM

Change #1052739 merged by Effie Mouzeli:

[operations/puppet@production] memcached: enable extstore to eqiad only

https://gerrit.wikimedia.org/r/1052739

Maintenance_bot removed a project: Patch-For-Review.Jul 11 2024, 9:32 AM
jijiki added a comment.Jul 16 2024, 4:15 PM

@Krinkle Extstore is fully enabled on eqiad

Krinkle mentioned this in T370185: Analyze impact of memcached extstore experiment (MediaWiki Platform Team).Jul 16 2024, 5:24 PM
jijiki changed the task status from In Progress to Stalled.Aug 2 2024, 10:57 AM
jijiki closed subtask T370185: Analyze impact of memcached extstore experiment (MediaWiki Platform Team) as Resolved.

Based on findings in T370185, we are unsure if we saw any overall benefit here. I will mark this as stalled until I have some time to dig deeper.

gerritbot added a comment.Jun 23 2025, 1:25 PM

Change #1162904 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/puppet@production] memcached: enable extstore on codfw

https://gerrit.wikimedia.org/r/1162904

gerritbot added a project: Patch-For-Review.Jun 23 2025, 1:25 PM
gerritbot added a comment.Jun 24 2025, 10:03 AM

Change #1162904 merged by Effie Mouzeli:

[operations/puppet@production] memcached: enable extstore on codfw

https://gerrit.wikimedia.org/r/1162904

Maintenance_bot removed a project: Patch-For-Review.Jun 24 2025, 10:31 AM
jijiki mentioned this in T408916: Enable extstore to gutter pool cluster.Oct 31 2025, 12:41 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits