Page MenuHomePhabricator

Test different growth factors for memcached (prep step for upgrade to newer versions)
Closed, ResolvedPublic

Description

We are currently running memcached version 1.4.21, but from 1.4.23 the number of max slab classes reduced from 200 to 63, due to a major rewrite of the LRU system.

Recap: memcached uses slab classes to decide where/how to allocate an object. Each slab class is basically a collection of 1MB memory blocks called 'pages', each one divided in 'chunks' of a fixed size. When an item needs to be stored, memcached tries to find the most efficient class to use (i.e. the one with the smallest chunk size to fit the item's size).

The list of slab classes start from a minimum and increases following a growth factor (essentially the minimum size is multiplied by the growth factor n time up to the maximum slab class size, 1MB).

By default, memcached offers two parameters to tune the slab classes:

  • -n -> minimum space allocated for key+value+flags (default: 48)
  • -f -> chunk size growth factor (default: 1.25)

We have the following configured:

  • -n 5
  • -f 1.25

This translates into ~180 Slabs, as anybody can see in https://grafana.wikimedia.org/d/000000317/memcache-slabs. The rationale, I believe, for this change at the time was to avoid memory waste with small objects, offering a good granularity of slab classes to allocate objects.

The idea that I have is to do something like T129963, testing one/two shards with a different combination of the above parameters. At the time when I worked on T129963 I had less metrics to check and I had to admit that my knowledge of memcached internals was way less than now (so basically almost zero), and I didn't really analyzed the results as I'd do now. Since we need to reduce the slab classes as part of the upgrade, I'd do some real production tests to see how memcached behaves changing the slab classes distribution.

I created the following Python code to generate the list of slabs:

import argparse
import math

if __name__ == '__main__':
    parser = argparse.ArgumentParser(description='Generate slab list for memcached.')
    parser.add_argument('f', type=float, help='Chunk size growth factor / -f parameter')
    parser.add_argument('n', type=int, help='Minimum space allocated for key+value+flags')

    args = parser.parse_args()
    growth_factor = args.f
    # sizeof(item) + chunk_size
    chunk_size = 48 + args.n
    slab = 1
    item_size_max = 1024 * 1024

    while True:
        if chunk_size % 8 != 0:
            chunk_size += 8 - (chunk_size % 8);
        print("Slab: {} Chunk: {}".format(str(slab), str(math.floor(chunk_size))))
        chunk_size = math.ceil(chunk_size * growth_factor)
        slab += 1
        if chunk_size > (item_size_max / growth_factor):
            break

Values taken from:

  • slabs_init function in slabs.c (memcached 1.4.21)
  • memcached.h (the sizeof of the item struct, memcached 1.4.21)

Event Timeline

elukey renamed this task from Test different growth factors for memcached (prep step for upgrade to newer versions to Test different growth factors for memcached (prep step for upgrade to newer versions).Feb 25 2019, 12:36 PM
elukey triaged this task as Normal priority.
elukey created this task.
elukey updated the task description. (Show Details)Feb 26 2019, 3:14 PM
elukey added a comment.EditedFeb 27 2019, 6:44 PM

I had a very interesting chat with upstream on IRC (thanks dormando!) and it was suggested to me to:

  • test directly with 1.5.x since a lot has changed and things are difficult to compare
  • with all the performance/allocation improvements in 1.5.x it is less efficient/convenient to heavily tune the slabs, better to stick with something that allocates all the available ones.

So the idea is to test 1.5.x with -f 1.1.5 and default -n (48B). This would result in 64 slabs like the following:

Slab: 1 Chunk: 96
Slab: 2 Chunk: 112
Slab: 3 Chunk: 136
Slab: 4 Chunk: 160
Slab: 5 Chunk: 184
Slab: 6 Chunk: 216
Slab: 7 Chunk: 256
Slab: 8 Chunk: 296
Slab: 9 Chunk: 344
Slab: 10 Chunk: 400
Slab: 11 Chunk: 464
Slab: 12 Chunk: 536
Slab: 13 Chunk: 624
Slab: 14 Chunk: 720
Slab: 15 Chunk: 832
Slab: 16 Chunk: 960
Slab: 17 Chunk: 1104
Slab: 18 Chunk: 1272
Slab: 19 Chunk: 1464
Slab: 20 Chunk: 1688
Slab: 21 Chunk: 1944
Slab: 22 Chunk: 2240
Slab: 23 Chunk: 2576
Slab: 24 Chunk: 2968
Slab: 25 Chunk: 3416
Slab: 26 Chunk: 3936
Slab: 27 Chunk: 4528
Slab: 28 Chunk: 5208
Slab: 29 Chunk: 5992
Slab: 30 Chunk: 6896
Slab: 31 Chunk: 7936
Slab: 32 Chunk: 9128
Slab: 33 Chunk: 10504
Slab: 34 Chunk: 12080
Slab: 35 Chunk: 13896
Slab: 36 Chunk: 15984
Slab: 37 Chunk: 18384
Slab: 38 Chunk: 21144
Slab: 39 Chunk: 24320
Slab: 40 Chunk: 27968
Slab: 41 Chunk: 32168
Slab: 42 Chunk: 37000
Slab: 43 Chunk: 42552
Slab: 44 Chunk: 48936
Slab: 45 Chunk: 56280
Slab: 46 Chunk: 64728
Slab: 47 Chunk: 74440
Slab: 48 Chunk: 85608
Slab: 49 Chunk: 98456
Slab: 50 Chunk: 113232
Slab: 51 Chunk: 130224
Slab: 52 Chunk: 149760
Slab: 53 Chunk: 172224
Slab: 54 Chunk: 198064
Slab: 55 Chunk: 227776
Slab: 56 Chunk: 261944
Slab: 57 Chunk: 301240
Slab: 58 Chunk: 346432
Slab: 59 Chunk: 398400
Slab: 60 Chunk: 458160
Slab: 61 Chunk: 526888
Slab: 62 Chunk: 605928
Slab: 63 Chunk: 696824
Slab: 64 Chunk: 801352

Also, IIUC, the max slab size for 1.5.x is 512K, and bigger ones are made adding together different slab sizes. This allowed memcached to easily overcome the 1MB slab limit. For example, a slab of 2M in size would be created using 4x512K slabs. This is really interesting since we could think about raising up the maximum key size limit for tasks like T204742.

Joe added a comment.Feb 27 2019, 7:10 PM

FWIW I would like to decrease the max key size instead than increasing it.

elukey closed this task as Resolved.Mar 4 2019, 10:22 AM
elukey claimed this task.
elukey moved this task from Backlog to Done on the User-Elukey board.