@akosiaris , any thoughts here?

Thanks @fgiunchedi.

Is there any plans on syncing codfw/eqiad redis nodes? It would be needed if the codfw starts to get traffic.

I'd thought that maybe we could partition requests to limit our need for replication. E.g. even rev_ids go to eqiad and off rev_ids go to codfw. That way, we wouldn't need to replicate redis. In the case that one of the two goes down, we can send all requests to one datacenter and we'll suffer from a short period of half as many cache hits.

BUT that would be preventatively difficult for our multi-rev_id query pattern. E.g.
https://ores.wikimedia.org/v2/scores/enwiki/damaging?revids=1001|1000

In T139372#3051587, @Ladsgroup wrote:

Is there any plans on syncing codfw/eqiad redis nodes? It would be needed if the codfw starts to get traffic.

Yes, I'll be working on setting up oresdb nodes in codfw, specifically one VM on ganeti and one baremetal server.

@Halfak do you see any problem in redis-replicating everything eqiad -> codfw without splitting ? Replicating everything seems the simplest and we're already doing it e.g. for redis mediawiki.

@fgiunchedi, replication should work. How do you imagine that we'd configure the codfw workers to connect to redis? I imagine that they wouldn't be able to write to the replica.

@Halfak I jumped the gun on replication I think :) Taking a step back, I took a look at ores code and it seems redis is used for caching scores and celery workers, correct?

If that's the case I think the simplest working solution would be to have codfw and eqiad not talk to each other at all. Namely ores in each datacenter talks to its local redis for celery and caching.

In case of active/standby configuration we can warm standby caches before failover, in case of active/active configuration the caches will be split though I don't think that's a problem in practice.
For celery workers, each redis will host the datacenter-local queues and brokers so that shouldn't be a problem either.

@Ladsgroup @Halfak what do you think?

The oresrdb2001 VM is provisioned in codfw now so we can test there too.

Change 340485 had a related patch set uploaded (by Filippo Giunchedi):
[operations/puppet] hieradata: add oresrdb in codfw

https://gerrit.wikimedia.org/r/340485

In T139372#3063893, @fgiunchedi wrote:

in case of active/active configuration the caches will be split though I don't think that's a problem in practice.

I tend to disagree. ORES caches scores based on revision. So if someone makes a request and it gets cached in codfw, it will be useless when someone else makes request to eqiad. I highly encourage replication of redis caches (the broker part is okay without replication)

In T139372#3064206, @Ladsgroup wrote:

In T139372#3063893, @fgiunchedi wrote:

in case of active/active configuration the caches will be split though I don't think that's a problem in practice.

I tend to disagree. ORES caches scores based on revision. So if someone makes a request and it gets cached in codfw, it will be useless when someone else makes request to eqiad. I highly encourage replication of redis caches (the broker part is okay without replication)

My assumption here was the active/active case which can't be easily be dealt with by redis replication alone anyways, during April' switchover we'll be going active/standby instead.

To @Halfak question about configuration, I'd imagine the normal status would be codfw redis slaves are read-only. Upon failover we break the replication and turn the codfw redis read/write (and possibly replicate back to eqiad) like we do for mediawiki. This for caching redis (i.e. on port 6380), celery redis we've established is ok to keep in each datacenter without replication.

Also how much of an impact it is to ores to start from an empty redis cache?What's the "normal" hit ratio?

I'm asking because since ores is stateless and its API read-only, having to replicate its cache seems strange. Not replicating the redis cache simplify things considerably both in the active/standby (i.e. in April switchover) and the active/active case (long-term goal for all services, including MW) by not having codfw and eqiad talk to each other at all if not strictly required.

In T139372#3064206, @Ladsgroup wrote:

In T139372#3063893, @fgiunchedi wrote:

in case of active/active configuration the caches will be split though I don't think that's a problem in practice.

I tend to disagree. ORES caches scores based on revision. So if someone makes a request and it gets cached in codfw, it will be useless when someone else makes request to eqiad. I highly encourage replication of redis caches (the broker part is okay without replication)

Replicating a huge redis db cross-dc is not going to happen, period. It would create more issues than it solves -namely it will break down 3-4 times a day.

I think caches should be *local*, globally distributed caches don't work well and could cause all kinds of issues.

Also, we can't do cross-replication for redis. (Well, we could in theory with some shameful hacks, but we definitely won't)

Thanks for the explanation. If we want to go other ways the most feasible one to me is to duplicate precaching for ores. One in eqiad and one in codfw. @Pchelolo: Do you think it's possible?

Again regarding precaching (which is surely duplicable): do we *really* need it?

The cache hit ratio is so small that I'm not sure precaching is doing more good than harm.

I would propose that we try to see what happens when we don't have any pre-caching turned on; I am not sure the total cache hit ratio would significantly worsen.

Precaching got broken several times (when we were in labs) the result was a huge increase in average response time. I think response time is vitally important to the service, specially for outside users. Also cache hit ratio is not bad: https://grafana.wikimedia.org/dashboard/db/ores?panelId=12&fullscreen&from=now-7d&to=now (average is 40% which is okay-ish, I'm not also sure if it counts precaching requests and if it counts them the cache hit ratio for outside users is around 60%)

To get started with oresrdb2001 in codfw please see/review https://gerrit.wikimedia.org/r/#/c/34048

Hey! Just to confirm, cache_misses are recorded for precaching requests. We should probably exclude them.

T159502: Exclude precaching requests from cache_miss/cache_hit metrics

So I thought a bit about it and come up with the following alternative solution

1. the celery side of the redis for ores MUST NOT be replicated between datacenters
2. We can set up nutcracker to act as a proxy between ores and the cache redis instances in both datacenters, so that all ores nodes share all the caches, and
3. set up ipsec between where nutcracker is and the oresrdbXXXX that actually stores the data.

but honestly for now duplicating the precaching jobs to both DCs is ok.

Hey folks, I figured we should have a task specifically for identifying the option we'd like to pursue. I've created T159615: [spec] Active-active setup for ORES across datacenters (eqiad, codfw) so we can continue the conversation there.

Change 340485 merged by Alexandros Kosiaris:
[operations/puppet] hieradata: add oresrdb in codfw

https://gerrit.wikimedia.org/r/340485

I 've just merged the patch above and now oresrdb2001 is setup and oresrdb.svc.codfw.wmnet is working fine. I 'll leave the task open until T142190 is done so that we have a second oresrdb server to act as a slave for oresrdb2001.

Let's indeed discuss the actual implementation details about how those dbs will get populated in T159615.