Page MenuHomePhabricator
Create Task
Maniphest T408692

Set up replication on new hosts clouddb102[2-5]
Closed, DuplicatePublic
Actions

Description

In T401295: Decide how to use the new clouddb hosts (clouddb102[2-5]) we decided to distribute database sections in the following way:

clouddb1013 : s1           (s3 removed)
clouddb1014 : s2, s7       (unchanged)
clouddb1015 : s4, s6       (unchanged)
clouddb1016 : s8, s5       (x3 removed)
clouddb1017 : s1           (s3 removed)
clouddb1018 : s2, s7       (unchanged)
clouddb1019 : s4, s6       (unchanged)
clouddb1020 : s8, s5       (x3 removed)

### New hosts ###

clouddb1022 : s3, x3
clouddb1023 : s3, x3
clouddb1024 : x4, x1
clouddb1025 : x4, x1

This task is to track the work of setting up the replication on the new hosts clouddb10[22-25], and to remove s3 and x3 from the existing hosts clouddb10[13-20].

Related Objects
Search...

StatusSubtypeAssignedTask
 OpenNoneT196020 Consider adding ContentTranslation (CX) tables to wiki replicas
 OpenFeatureNoneT395072 Add "wikishared" database to wiki replicas
 StalledNoneT387419 Create wiki replicas views for globaljsonlinks tables
 StalledNoneT395881 Set up x1 replication to Wiki Replicas
 DuplicateMarosteguiT408692 Set up replication on new hosts clouddb102[2-5]
 ResolvedfnegriT401295 Decide how to use the new clouddb hosts (clouddb102[2-5])
 ResolvedMarosteguiT409557 Productionize new clouddb* hosts (clouddb1022-1033)
 ResolvedMarosteguiT404715 Setup x4 section
 ResolvedMarosteguiT406550 Productionize db126[0-3]
 ResolvedMarosteguiT406551 Productionize db224[5-8]
 OpenNoneT431984 dbctl instance edit crashes with "exceptions must derive from BaseException" instead of showing real validation errors
 ResolvedPRODUCTION ERRORZabeT431989 PHP Warning: Undefined array key "x4"
 ResolvedtaaviT437208 Update user-facing Wiki Replicas features and documentation for x4
 ResolvedtaaviT409560 Add support for x1 and x4 sections on wiki replicas on the load balancer layer
 ResolvedMarosteguiT431309 Setup x4 on sanitarium hosts
 ResolvedMarosteguiT437229 Check cookbooks/scripts to add x4
 ResolvedMarosteguiT437279 Setup x4 backups
 ResolvedMarosteguiT437678 Create x4 candidate masters
 InvalidNoneT437696 Switchover x4 master (db2248 -> db2245)
 InvalidNoneT437695 Switchover x4 master (db1263 -> db1261)
 ResolvedMarosteguiT437838 Add x4 to an-redacteddb1001
 ResolvedMarosteguiT437839 Add x4 to dbstore1007
 ResolvedMarosteguiT410365 SystemdUnitFailed: generate-mysqld-exporter-config.service on prometheus100[56]:9100
 ResolvedcmooneyT430132 cloudlb100X cannot talk to new host clouddb1026

Event Timeline

fnegri created this task.Oct 29 2025, 2:04 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 29 2025, 2:04 PM
fnegri added a project: cloud-services-team.Oct 29 2025, 2:05 PM
fnegri moved this task from Inbox to Watching on the cloud-services-team board.
fnegri mentioned this in T401295: Decide how to use the new clouddb hosts (clouddb102[2-5]).
fnegri added a subtask: T401295: Decide how to use the new clouddb hosts (clouddb102[2-5]).
taavi added a project: Data-Services.Oct 29 2025, 2:09 PM
Novem_Linguae subscribed.Oct 29 2025, 2:09 PM
taavi moved this task from Backlog to Wiki replicas on the Data-Services board.Oct 29 2025, 2:09 PM
fnegri added a parent task: T395881: Set up x1 replication to Wiki Replicas.Oct 29 2025, 3:15 PM
Marostegui subscribed.Oct 31 2025, 7:55 AM
Marostegui added a comment.Nov 6 2025, 3:27 PM

@fnegri do you want me to do this?

fnegri added a comment.Nov 6 2025, 6:38 PM

@Marostegui no rush at all, but when you have some time you can start. My understanding is that the initial setup of the dbs is entirely up to your team, and my team will then create the views and DNS records once the replication is up & running. Is that correct? Let me know if there's anything you need from us.

Marostegui claimed this task.Nov 7 2025, 6:23 AM
Marostegui triaged this task as Medium priority.
Marostegui edited projects, added: DBA; removed: Data-Persistence.
In T408692#11351084, @fnegri wrote:

@Marostegui no rush at all, but when you have some time you can start. My understanding is that the initial setup of the dbs is entirely up to your team, and my team will then create the views and DNS records once the replication is up & running. Is that correct? Let me know if there's anything you need from us.

Yep, that is correct. I will start next week and I will log the depools here (just one section at the time) in case users reach out for a slightly degraded service during that time.

Marostegui moved this task from Triage to In progress on the DBA board.Nov 7 2025, 6:23 AM
Marostegui added a comment.Nov 7 2025, 2:49 PM

Ha, I just realised the hosts aren't installed yet

Marostegui added a subtask: T409557: Productionize new clouddb* hosts (clouddb1022-1033).Nov 7 2025, 2:58 PM
Marostegui added a comment.Nov 7 2025, 3:15 PM

After talking to @fnegri the confusion was cleared up, these hosts are, the ones that aren't are the refreshes. I created a task with all the hosts so it is easier to track this down: T409557

taavi mentioned this in T409560: Add support for x1 and x4 sections on wiki replicas on the load balancer layer.Nov 7 2025, 3:20 PM
fnegri closed this task as a duplicate of T409557: Productionize new clouddb* hosts (clouddb1022-1033).Mar 20 2026, 3:41 PM
Marostegui closed subtask T409557: Productionize new clouddb* hosts (clouddb1022-1033) as Resolved.Aug 5 2026, 7:45 AM
Restricted Application added a project: tools-platform-team. · View Herald TranscriptAug 5 2026, 7:45 AM
dcaro moved this task from Backlog to Done on the tools-platform-team board.Aug 5 2026, 8:23 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits