Using a redis or MySQL, some code stashes objects or data in places where both DCs should see it.
Examples include:
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | aaron | T88445 MediaWiki active/active datacenter investigation and work (tracking) | |||
| Resolved | aaron | T97620 Convert various core/extension cache users to ReplicatedBagOStuff |
Change 207718 had a related patch set uploaded (by Aaron Schulz):
Added ObjectStash factory class and $wgMainStash/$wgObjectStashes
Change 207718 merged by jenkins-bot:
Added ObjectCache::getMainStashInstance() and $wgMainStash
Change 212715 had a related patch set uploaded (by Aaron Schulz):
Conversion to using getMainStashInstance()
Change 212717 had a related patch set uploaded (by Aaron Schulz):
[WIP] Conversion to using getMainStashInstance()
Change 213762 had a related patch set uploaded (by Aaron Schulz):
[WIP] Conversion to using getMainStashInstance()
Just to clarify, this would be for things where it is important for all instances to see the data, i.e., sort of like a brief key-value store? (Just curious because OATHAuth might need to use this for storing token expiration data to prevent replay attacks.)
Yes, although note that get() can have lag. I was actually thinking about having a flag to avoid lag for the replicated cache class.
noonce tokens are interesting (I was also thinking about that a lot today). If the GET/POST distinction is complete and only the later really mutates anything (like edits/comments) then noonce cache could be dc-local for performance, since POSTs would all go to one DC and have full deduplication and GETS would only allow one extra replay in the other DC (if fast enough and if HTTP was being used) and would not do anything. Of course if one is paranoid they can use add() on the stash bagostuff :)
Change 223790 had a related patch set uploaded (by Aaron Schulz):
Conversion to using getMainStashInstance()
Change 221994 had a related patch set uploaded (by Aaron Schulz):
Conversion to using getMainStashInstance()
@aaron do we really want to rely on redis replication cross-datacenter? Or did I get that wrong? Redis replication is know not to be extraordiarily reliable even in a small-lag, same DC setup, did you do some experiments to see how reliable it would be?
If not, I'd love to help.
We used replication from tampa => eqiad during the switchover (though I don't think the consistent hashing was done correctly MW side). I was assuming we have replication likewise between eqiad => codfw.
@aaron at the moment we don't, as replicating redis would result in the codfw jobqueues processing again the same jobs as the eqiad ones, for instance.
Also, while I can think of that as a solution for a "definitive" switchover, I don't think it's a good idea long-term. But I'll look into options for making it as reliable as possible.
I'm just talking about the mc* redis instances (this bug is just about BagOStuff). We've done replication for that before.
I think the queues could be replicated in any case (though only duplicate jobs would be those who did not have the ACK replicated before switchover, which is tolerable), but that discussion could go on another task.
Change 234190 had a related patch set uploaded (by Aaron Schulz):
Conversion to using getMainStashInstance()
Change 234191 had a related patch set uploaded (by Aaron Schulz):
Conversion to using getMainStashInstance()