The concerned raised by @Kormat is that the current set up causes lag spikes, […]

Could you elaborate on this? What is the cause-effect we see, and how is it caused or aggrevated by having bi-directional replication?

[…] with the question whether we actually need this or whether a locally kept cache would suffice.

My gut feeling is that we do not need this indeed. I think the main reason we have it is for switchovers and disaster recovery so that our secondary DC has an effectively "warm" standby parser cache.

However, when we operate read requests from both DCs (multi-DC) this probably isn't a concern, perhaps even comparable to how we keep other caches local to the DC as well (like Memcached). And unlike Memcached/WANObjectCache, the parser cache does not have a need for proactively sending purges (as far as I know).

The only thing that comes to mind as a minor caveat is that gargage collection is a maintenance cron, which if parser cache becomes dc-local means we presumably want to run that in all DCs, which would probably make that the first maintenance cron of its kind. The others are only enabled in the primary DC, I think.

This is a duplicate of T133523

In T277831#6928235, @jcrespo wrote:

This is a duplicate of T133523

That is quite a broad and open-ended ticket. But we can continue the discussion there, sure. I'll make it a child one for now, and close this when the replication/multi-dc part of it is resolved.

In T277831#8166216, @Krinkle wrote:

As part of today's incident review meeting for T315271, it was brought up that while ParserCache is ready for bi-directional replication, it is not actually enabled.

I have no preference at this time for whether it is bi-direction, but as requirement for this task I would say:

• pc hosts must be writeable in both DCs.

This is already ready

• pc hosts must replicate from eqiad-codfw to improve cache warmness (for now, let's keep status quo, especially as Codfw has little traffic now)

This is ready, but if it is not in use, I prefer not to enabled it until it is going to be used (so we can proceed with maintenance in an easier way)

• pc hosts may replicate in the other direction as well, this is optional and to be decided by DBAs on what seems easiest to manage.

So right now we do eqiad -> codfw (and the other way around when we have codfw as primary DC), not sure if you just meant also eqiad <-> codfw. If that is the case, see the above comment, it is fully ready, it just one command to enable it.

In T277831#8166216, @Krinkle wrote:

• pc hosts must replicate from eqiad-codfw to improve cache warmness (for now, let's keep status quo, especially as Codfw has little traffic now)

Given the x2 outage and the fact that pc doesn't have the modtoken protection (to my knowledge) which makes it even more fragile, I highly recommend simply breaking the replication and let them grow apart. It is a cache and this can benefit from cache locality (e.g. Eastern Asian languages being served from codfw) and scale it even better. I'm happy with this happening once codfw got fully active but it should happen eventually. It'd make it much more resilient and much easier to maintain (plus better cache locality as I said).

If the above model is fine with MW, I would also prefer that rather than having it to be multi-master.

In T277831#8166246, @Marostegui wrote:

In T277831#8166216, @Krinkle wrote:

• pc hosts may replicate in the other direction as well, this is optional […]

[…] not sure if you just meant also eqiad <-> codfw. If that is the case, see the above comment, it is fully ready, it just one command to enable it.

With "in the other direction as well", I indeed meant in addition to, not instead of. So either of eqiad>codfw or eqiad<>codfw will work to keep the status quo of Codfw being warm. I leave it up to your team.

My thinking was bi-di would be prefered as it will match x2-mainstash (thus less novel/unique different setups), and because bi-di is what MW expects/documents for SqlBagOStuff (which powers PC and MainStash). But, for PC specifically, we tolerate loss of writes and runtime validate the values. Consistency doesn't actually matter there. My main reason for preferring to replicating at least in one direction is that it will help keep it warm during roll-out and avoids making more changes at the same time right now.

This is ready, but if it is not in use, I prefer not to enabled it until it is going to be used (so we can proceed with maintenance in an easier way)

We have already begun rolling out traffic to www.mediawiki.org under T279664: Progressive Multi-DC roll out. By leaving this off, I think we are potentially not covering as much during this testing phase as we could. There are no longer any blockers left there. I would say "now" is basically the natural time to start enabling it, otherwise it will become a blocker very soon (TM).

Next step for this task: I'm waiting for someone in your team to close this task with a sign-off that ParserCache is ready to receive read-write traffic in Codfw and Eqiad.

In T277831#8166260, @Ladsgroup wrote:

[…] I highly recommend simply breaking the replication and let them grow apart. It is a cache and this can benefit from cache locality […]. I'm happy with this happening once codfw got fully active but it should happen eventually. […]

In T277831#8166377, @Marostegui wrote:

If the above model is fine with MW, I would also prefer that rather than having it to be multi-master.

This model has previously been what I was advocating as well. I think there is still long-term potential in turning off replication for PC. But, I recommend we keep the status quo during Multi-DC. Changing it afterward is fine. I do want to note however, there do exist (relatively new) reasons that may inform a future iteration of ParserCache in which we actually do need replication again:

• deprecation of RESTBase, which leads to increase in demand for ParserCache via rest.php. Current trajectory is to find a way to accomodate it with a TTL, but long-term it seems desirable to keep the "current" revision around.
• deployment of Parsoid, similarly needs to store its version of the HTML output.
• pageview latency improvements. Optimizations to JobQueue and RefreshLinks have been made to benefit PC effeciency, but these have come at the cost of making it more likely that logged-in users require a parse during pageviews. One direction I'd like to explore, with some metrics, is what it would cost to eliminate parsing during pageviews entirely, thus only parsing during POSTs and during jobqueue (and reducing parser options, as Parsoid desires). In that reality, the backend of PC would need to be replicated. (This isn't yet planned, but fits in long-term direction after T302623 and T140664.)

In T277831#8167494, @Krinkle wrote:

In T277831#8166246, @Marostegui wrote:

In T277831#8166216, @Krinkle wrote:

• pc hosts may replicate in the other direction as well, this is optional […]

[…] not sure if you just meant also eqiad <-> codfw. If that is the case, see the above comment, it is fully ready, it just one command to enable it.

With "in the other direction as well", I indeed meant in addition to, not instead of. So either of eqiad>codfw or eqiad<>codfw will work to keep the status quo of Codfw being warm. I leave it up to your team.

So replication from eqiad to codfw is there and has always been. We have no plans to remove that unless we agreed (below) that we want to keep both without knowing about each other.

My thinking was bi-di would be prefered as it will match x2-mainstash (thus less novel/unique different setups), and because bi-di is what MW expects/documents for SqlBagOStuff (which powers PC and MainStash). But, for PC specifically, we tolerate loss of writes and runtime validate the values. Consistency doesn't actually matter there. My main reason for preferring to replicating at least in one direction is that it will help keep it warm during roll-out and avoids making more changes at the same time right now.

I would prefer to keep it like it is now (like a normal section) where eqiad -> codfw (or codfw -> eqiad when codfw becomes our primary writable DC).

We have already begun rolling out traffic to www.mediawiki.org under T279664: Progressive Multi-DC roll out. By leaving this off, I think we are potentially not covering as much during this testing phase as we could. There are no longer any blockers left there. I would say "now" is basically the natural time to start enabling it, otherwise it will become a blocker very soon (TM).

Next step for this task: I'm waiting for someone in your team to close this task with a sign-off that ParserCache is ready to receive read-write traffic in Codfw and Eqiad.

If we can leave only eqiad -> codfw and having codfw -> eqiad isn't a blocker, we are ready. Just let us know when you want to start writing to codfw so we are aware.

Ideally, if we can have both topologies without any replication between them, I would prefer that. But as you noted, maybe that is something to do (it is just a single command from our side) after testing Multi-DC

In T277831#8167494, @Krinkle wrote:

This model has previously been what I was advocating as well. I think there is still long-term potential in turning off replication for PC. But, I recommend we keep the status quo during Multi-DC. Changing it afterward is fine.

Definitely, I'm not suggesting to do it right now. Let's talk about it after full deployment of multidc.

I do want to note however, there do exist (relatively new) reasons that may inform a future iteration of ParserCache in which we actually do need replication again:

• deprecation of RESTBase, which leads to increase in demand for ParserCache via rest.php. Current trajectory is to find a way to accomodate it with a TTL, but long-term it seems desirable to keep the "current" revision around.
• deployment of Parsoid, similarly needs to store its version of the HTML output.

These two are actually some of reasons I think we should switch to per-DC PC. These will increase storage of PC drastically and there is some risk that we run out of space for it again. By doing per-DC PC, we reduce the entries where it's not going to be accessed (e.g. entries in languages of east Asia will be stored mostly codfw only and languages like French or German will reside in eqiad only) which reduces PC storage by a reasonable amount.

@Krinkle I am not sure what else is needed here. Would this work for you?:

In T277831#8167992, @Marostegui wrote:

If we can leave only eqiad -> codfw and having codfw -> eqiad isn't a blocker, we are ready. Just let us know when you want to start writing to codfw so we are aware.

Ideally, if we can have both topologies without any replication between them, I would prefer that. But as you noted, maybe that is something to do (it is just a single command from our side) after testing Multi-DC