x4 is a new topology which will be a split from s4.
We need to setup a sanitarium master and a sanitarium instance for it.
s4 lives in db1155, so it makes sense to make it there too.
Description
Description
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T196020 Consider adding ContentTranslation (CX) tables to wiki replicas | |||
| Open | Feature | None | T395072 Add "wikishared" database to wiki replicas | ||
| Open | None | T428047 Upgrade clouddb* hosts to Debian Trixie | |||
| Unknown Object (Task) | |||||
| Resolved | Jclark-ctr | T409162 Q2:rack/setup/install clouddb1026-1033 | |||
| Stalled | None | T387419 Create wiki replicas views for globaljsonlinks tables | |||
| Stalled | None | T395881 Set up x1 replication to Wiki Replicas | |||
| Duplicate | Marostegui | T408692 Set up replication on new hosts clouddb102[2-5] | |||
| Open | None | T343131 Commons database is growing way too fast | |||
| Open | Ladsgroup | T398709 FY2025-26 WE 6.4.1: Move links tables of commons to a dedicated cluster | |||
| Open | Marostegui | T434048 Decommission clouddb1013-clouddb1020 | |||
| Resolved | Marostegui | T409557 Productionize new clouddb* hosts (clouddb1022-1033) | |||
| Open | Marostegui | T404715 Setup x4 section | |||
| Open | Marostegui | T431309 Setup x4 on sanitarium hosts |
Event Timeline
Comment Actions
@Ladsgroup @Zabe how inmediately available x4 has to be once we split from s4?.
Unless we drop stuff, it is going to be tight to copy x4 entirely from s4 given the space available on sanitarium:
root@db1155:/srv# du -sh sqldata.s4/ 2.3T sqldata.s4/ root@db1155:/srv# df -hT /srv Filesystem Type Size Used Avail Use% Mounted on /dev/mapper/tank-data xfs 8.7T 5.6T 3.1T 65% /srv
I guess if we run optimize we may be able to get some temporary space back.
Also, if we do an entire copy and we live with just 800G available for a few days, how long will it pass to do the _real_ split so only the final tables will be written so we can delete the other ones?
Comment Actions
I'd say let's set up sanitarium when we are closer to the split. I can also run an optimize table on s4 to save some space T410589: Optimize all core tables, late 2025. It'll take a while though.
Comment Actions
The biggest problem is that s4 is so big that we can't set it up before the split since we can't set up two copies of s4 at the same time (s4 is now twice as big as the second biggest database cluster).
! In T431309#12291647, @bd808 wrote:
the commons link tables will only be available from the x4 cluster. This is going to break a bunch of tools and they will stay broken until the x4 cluster is available in the Wiki Replicas.
We have a mitigation, it's not perfect but it worked for s8/x3 split. We are not going to drop the old links tables on s4 sanitarium master and we can keep it for a month or so. That means tools will just start to drift but won't break with errors contrary to when we drop a column or a table. So no fatals or errors or complete breakage, just outdated data which is also okay for most cases, if you want members of categories, most data still be useful.
Comment Actions
The announcement sent to the cloud-announce list, which I assume was done so that tool maintainers know things are changing, does not mention that there is no way to access x4 via the Wiki Replicas yet, when that is expected, or what will be possible when the tables are in a new db that cannot be linked across sections to other tables. These are all reasonable questions for folks to have beyond "where did the data go" which is what has been documented in that email.