Page MenuHomePhabricator

Memory consumption in Redis 3.2 vs Redis 2.8
Closed, DeclinedPublic

Description

We recently switched changeprop's redis servers from Redis 2.8 to Redis 3.2.

I noticed that there are differences in total memory consumption in between redis_jobqueue (Redis v2.8) and redis_misc (Redis v3.2):

eqiad redis_jobqueue ( Redis 2.8) was ~63 GB (3 redis masters):

https://grafana.wikimedia.org/dashboard/db/redis?orgId=1&from=1539711031726&to=1542388149112&var-datasource=eqiad%20prometheus%2Fops&var-job=redis_jobqueue

eqiad redis_misc (Redis 3.2) is ~9 GB (2 redis masters)

https://grafana.wikimedia.org/dashboard/db/redis?orgId=1&from=now-10d&to=now&var-datasource=eqiad%20prometheus%2Fops&var-job=redis_misc

codfw redis_jobqueue (Redis 2.8) ~55 GB

https://grafana.wikimedia.org/dashboard/db/redis?orgId=1&from=1539711031726&to=1542388149112&var-datasource=codfw%20prometheus%2Fops&var-job=redis_jobqueue

codfw redis_misc (Redis 3.2) ~75 GB

https://grafana.wikimedia.org/dashboard/db/redis?orgId=1&from=now-10d&to=now&var-datasource=codfw%20prometheus%2Fops&var-job=redis_misc

Things to consider:

  • redis_jobqueue had cross-dc replication, redis_misc doesn't
  • on codfw we have 6 scb servers while in eqiad we have 4
  • redis_misc servers have more maxmemory than redis_jobqueue servers did
  • this could probably be nothing:)

Related Objects

StatusSubtypeAssignedTask
ResolvedNone
DeclinedNone

Event Timeline

jijiki created this task.

There actually is something wrong here. The average TTL for redis_misc in codfw is 0, which, I believe, indicate there's no TTL set at all in codfw.

The commands rate graph indicates there's one 'expire' command per one 'setnx' command, which is perfectly aligned with the deduplication codepath in ChangeProp. Trying to set expiration for some key manually via redis-cli works in codfw. Change-Prop config is exactly the same in eqiad and codfw, so its still not clear why the graph is showing average expiration time as 0

Bluntly closing this, no updates/findings for a long time