Page MenuHomePhabricator
Create Task
Maniphest T409557

Productionize new clouddb* hosts (clouddb1022-1033)
Closed, ResolvedPublic
Actions

Description

The following hosts need to be productionized:

  • clouddb1022 expansion s3, x3 - cloned
  • clouddb1023 expansion s3, x3 - cloned
  • clouddb1024 expansion x4, x1 - (s4 cloned, will be renamed to x4 later, and x1 added when ready)
  • clouddb1025 expansion x4, x1 - (s4 cloned, will be renamed to x4 later, and x1 added when ready)
  • clouddb1026 refresh for clouddb1013 (s1)
  • clouddb1027 refresh for clouddb1014 (s2, s7)
  • clouddb1028 refresh for clouddb1015 (s4, s6)
  • clouddb1029 refresh for clouddb1016 (s8, s5)
  • clouddb1030 refresh for clouddb1017 (s1)
  • clouddb1031 refresh for clouddb1018 (s2, s7)
  • clouddb1032 refresh for clouddb1019 (s4, s6)
  • clouddb1033 refresh for clouddb1020 (s8, s5)
  • Check all hosts are in zarcillo

Depooling and repooling replicas (https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Runbooks/Depool_wikireplicas):

sudo confctl select name=clouddb1013.eqiad.wmnet get

sudo confctl select name=clouddb1013.eqiad.wmnet,service=s3 set/pooled=no


sudo confctl select name=clouddb1013.eqiad.wmnet,service=s3 set/pooled=yes

sudo confctl select name=clouddb1029.eqiad.wmnet,service=s8 set/weight=100

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
clouddb1025: Add x4Marosteguioperations/puppetproduction+1 -1
check_private_data_report: Add x4Marosteguioperations/puppetproduction+1 -1
clouddb1024: Add x4Marosteguioperations/puppetproduction+1 -1
clouddb: Revert temp changes to clouddb1025FNegrioperations/puppetproduction+1 -3
clouddb1033: Enable notificationsMarosteguioperations/puppetproduction+0 -1
eqiad.yaml: Fix hostnameMarosteguioperations/puppetproduction+1 -1
eqiad.yaml: Add clouddb1033Marosteguioperations/puppetproduction+1 -0
check_private_data_report: Add clouddb1033Marosteguioperations/puppetproduction+1 -1
installserver: Do not format clouddb1033Marosteguioperations/puppetproduction+1 -5
mariadb: Productionize clouddb1033Marosteguioperations/puppetproduction+5 -5
eqiad.yaml: Add clouddb1032Marosteguioperations/puppetproduction+1 -0
check_private_data_report: Add clouddb1032Marosteguioperations/puppetproduction+1 -1
installserver: Do not format clouddb1032Marosteguioperations/puppetproduction+2 -2
mariadb: Productionize clouddb1032Marosteguioperations/puppetproduction+6 -2
eqiad.yaml: Add clouddb1031Marosteguioperations/puppetproduction+1 -0
check_private_data_report: Add clouddb1031Marosteguioperations/puppetproduction+1 -1
clouddb1031: Enable notificationsMarosteguioperations/puppetproduction+0 -1
mariadb: Productionize clouddb1031Marosteguioperations/puppetproduction+6 -2
clouddb1030: Enable notificationsMarosteguioperations/puppetproduction+0 -1
eqiad.yaml: Add clouddb1030Marosteguioperations/puppetproduction+1 -0
check_private_data_report: Add clouddb1030Marosteguioperations/puppetproduction+14 -0
installserver: Do not format clouddb1030Marosteguioperations/puppetproduction+2 -2
mariadb: Productionize clouddb1030Marosteguioperations/puppetproduction+5 -2
clouddb1029: Enable notificationsMarosteguioperations/puppetproduction+0 -1
eqiad.yaml: Add clouddb1029Marosteguioperations/puppetproduction+1 -0
mariadb: Remove x3 from clouddb1029Marosteguioperations/puppetproduction+15 -2
aliases.yaml: Add clouddb1029 to db-clouddb-sanitizationMarosteguioperations/puppetproduction+2 -1
check_private_data_report: Add clouddb1029Marosteguioperations/puppetproduction+1 -1
mariadb: Productionize clouddb1029Marosteguioperations/puppetproduction+7 -2
eqiad.yaml: Add clouddb1028 to the LBMarosteguioperations/puppetproduction+1 -0
check_private_data_report: Add clouddb1028Marosteguioperations/puppetproduction+1 -1
clouddb1028: Enable notificationsMarosteguioperations/puppetproduction+0 -1
installserver: Do not format clouddb1028Marosteguioperations/puppetproduction+2 -2
mariadb: Productionize clouddb1028Marosteguioperations/puppetproduction+6 -2
clouddb1027: Enable notificationsMarosteguioperations/puppetproduction+0 -2
eqiad.yaml: Add clouddb1027Marosteguioperations/puppetproduction+1 -0
installserveR: Do not format /srv on clouddb102[2-7]Marosteguioperations/puppetproduction+1 -1
check_private_data_report: Add clouddb1027Marosteguioperations/puppetproduction+1 -1
mariadb: Productionize clouddb1027Marosteguioperations/puppetproduction+6 -2
installserver: Do not format clouddb102[6-7]Marosteguioperations/puppetproduction+2 -3
eqiad.yaml: Add clouddb1026Marosteguioperations/puppetproduction+1 -0
check_private_data_report: Add clouddb1026Marosteguioperations/puppetproduction+1 -1
mariadb: Productionize cloudb1026Marosteguioperations/puppetproduction+5 -2
eqiad.yaml: Add clouddb1025 to s6Marosteguioperations/puppetproduction+1 -1
clouddb1025: Disable notificationsMarosteguioperations/puppetproduction+1 -0
cloudb1025: Add s6Marosteguioperations/puppetproduction+2 -0
conftool-data: move s3, x3 to new hosts (part 2)FNegrioperations/puppetproduction+4 -4
eqiad.yaml: Add clouddb1025 to s4Marosteguioperations/puppetproduction+1 -1
site.pp: Move clouddb1024 to analyticsMarosteguioperations/puppetproduction+3 -2
mariadb: Productionize clouddb1032Marosteguioperations/puppetproduction+8 -2
conftool-data: move s3, x3 to new hosts (part 1)FNegrioperations/puppetproduction+2 -0
clouddb102[23]: Enable notificationsMarosteguioperations/puppetproduction+0 -2
conftool-data: Add clouddb1024/5 as x4Majavahoperations/puppetproduction+2 -0
clouddb1025: Enable notificationsMarosteguioperations/puppetproduction+0 -1
clouddb1024: Enable notificationsMarosteguioperations/puppetproduction+0 -1
check_private_data_report: Add clouddb102[45]Marosteguioperations/puppetproduction+14 -0
mariadb: Productionize clouddb1025Marosteguioperations/puppetproduction+8 -2
clouddb1024: Change s4 with x4Marosteguioperations/puppetproduction+1 -1
mariadb: Productionize clouddb1024Marosteguioperations/puppetproduction+8 -2
mariadb: Productionize clouddb1023Marosteguioperations/puppetproduction+6 -2
check_private_data_report: Add new hostsMarosteguioperations/puppetproduction+14 -0
clouddb1022: Initial puppet runMarosteguioperations/puppetproduction+6 -2
clouddb1026-clouddb1033: New hostsMarosteguioperations/puppetproduction+24 -0
Show related patches Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 OpenNoneT196020 Consider adding ContentTranslation (CX) tables to wiki replicas
 OpenFeatureNoneT395072 Add "wikishared" database to wiki replicas
 OpenfnegriT428047 Upgrade clouddb* hosts to Debian Trixie
Unknown Object (Task)
 ResolvedJclark-ctrT409162 Q2:rack/setup/install clouddb1026-1033
 StalledNoneT387419 Create wiki replicas views for globaljsonlinks tables
 StalledNoneT395881 Set up x1 replication to Wiki Replicas
 DuplicateMarosteguiT408692 Set up replication on new hosts clouddb102[2-5]
 OpenMarosteguiT434048 Decommission clouddb1013-clouddb1020
 ResolvedMarosteguiT409557 Productionize new clouddb* hosts (clouddb1022-1033)
 ResolvedMarosteguiT404715 Setup x4 section
 ResolvedMarosteguiT406550 Productionize db126[0-3]
 ResolvedMarosteguiT406551 Productionize db224[5-8]
 OpenNoneT431984 dbctl instance edit crashes with "exceptions must derive from BaseException" instead of showing real validation errors
 ResolvedPRODUCTION ERRORZabeT431989 PHP Warning: Undefined array key "x4"
 ResolvedtaaviT437208 Update user-facing Wiki Replicas features and documentation for x4
 ResolvedtaaviT409560 Add support for x1 and x4 sections on wiki replicas on the load balancer layer
 ResolvedMarosteguiT431309 Setup x4 on sanitarium hosts
 ResolvedMarosteguiT437229 Check cookbooks/scripts to add x4
 ResolvedMarosteguiT437279 Setup x4 backups
 ResolvedMarosteguiT437678 Create x4 candidate masters
 InvalidNoneT437696 Switchover x4 master (db2248 -> db2245)
 InvalidNoneT437695 Switchover x4 master (db1263 -> db1261)
 ResolvedMarosteguiT437838 Add x4 to an-redacteddb1001
 ResolvedMarosteguiT437839 Add x4 to dbstore1007
 ResolvedMarosteguiT410365 SystemdUnitFailed: generate-mysqld-exporter-config.service on prometheus100[56]:9100
 ResolvedcmooneyT430132 cloudlb100X cannot talk to new host clouddb1026

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
gerritbot added a comment.Jul 21 2026, 5:33 AM

Change #1312776 merged by Marostegui:

[operations/puppet@production] clouddb1029: Enable notifications

https://gerrit.wikimedia.org/r/1312776

Marostegui updated the task description. (Show Details)Jul 21 2026, 5:36 AM
gerritbot added a comment.Jul 21 2026, 5:40 AM

Change #1312788 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] mariadb: Productionize clouddb1030

https://gerrit.wikimedia.org/r/1312788

gerritbot added a comment.Jul 21 2026, 5:48 AM

Change #1312788 merged by Marostegui:

[operations/puppet@production] mariadb: Productionize clouddb1030

https://gerrit.wikimedia.org/r/1312788

Marostegui added a comment.Jul 21 2026, 8:32 AM

clouddb1030:3311 is now replicating.

gerritbot added a comment.Jul 21 2026, 9:49 AM

Change #1313087 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] installserver: Do not format clouddb1030

https://gerrit.wikimedia.org/r/1313087

gerritbot added a comment.Jul 21 2026, 9:55 AM

Change #1313087 merged by Marostegui:

[operations/puppet@production] installserver: Do not format clouddb1030

https://gerrit.wikimedia.org/r/1313087

gerritbot added a comment.Jul 22 2026, 7:56 AM

Change #1313811 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] check_private_data_report: Add clouddb1030

https://gerrit.wikimedia.org/r/1313811

gerritbot added a comment.Jul 22 2026, 7:58 AM

Change #1313811 merged by Marostegui:

[operations/puppet@production] check_private_data_report: Add clouddb1030

https://gerrit.wikimedia.org/r/1313811

gerritbot added a comment.Jul 22 2026, 8:01 AM

Change #1313818 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] eqiad.yaml: Add clouddb1030

https://gerrit.wikimedia.org/r/1313818

gerritbot added a comment.Jul 22 2026, 8:03 AM

Change #1313818 merged by Marostegui:

[operations/puppet@production] eqiad.yaml: Add clouddb1030

https://gerrit.wikimedia.org/r/1313818

gerritbot added a comment.Jul 22 2026, 8:35 AM

Change #1313839 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] clouddb1030: Enable notifications

https://gerrit.wikimedia.org/r/1313839

gerritbot added a comment.Jul 22 2026, 8:37 AM

Change #1313839 merged by Marostegui:

[operations/puppet@production] clouddb1030: Enable notifications

https://gerrit.wikimedia.org/r/1313839

Marostegui added a comment.Jul 22 2026, 8:39 AM

clouddb1030 has been pooled in s1

Marostegui updated the task description. (Show Details)Jul 22 2026, 8:39 AM
gerritbot added a comment.Jul 22 2026, 9:17 AM

Change #1313864 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] mariadb: Productionize clouddb1031

https://gerrit.wikimedia.org/r/1313864

gerritbot added a comment.Jul 22 2026, 9:18 AM

Change #1313864 merged by Marostegui:

[operations/puppet@production] mariadb: Productionize clouddb1031

https://gerrit.wikimedia.org/r/1313864

Marostegui added a comment.Jul 22 2026, 11:47 AM

clouddb1031 replicating on s2 and s7

gerritbot added a comment.Jul 23 2026, 7:00 AM

Change #1314611 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] clouddb1031: Enable notifications

https://gerrit.wikimedia.org/r/1314611

gerritbot added a comment.Jul 23 2026, 7:07 AM

Change #1314611 merged by Marostegui:

[operations/puppet@production] clouddb1031: Enable notifications

https://gerrit.wikimedia.org/r/1314611

gerritbot added a comment.Jul 23 2026, 7:09 AM

Change #1314612 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] check_private_data_report: Add clouddb1031

https://gerrit.wikimedia.org/r/1314612

gerritbot added a comment.Jul 23 2026, 7:14 AM

Change #1314612 merged by Marostegui:

[operations/puppet@production] check_private_data_report: Add clouddb1031

https://gerrit.wikimedia.org/r/1314612

gerritbot added a comment.Jul 23 2026, 7:18 AM

Change #1314622 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] eqiad.yaml: Add clouddb1031

https://gerrit.wikimedia.org/r/1314622

gerritbot added a comment.Jul 23 2026, 7:19 AM

Change #1314622 merged by Marostegui:

[operations/puppet@production] eqiad.yaml: Add clouddb1031

https://gerrit.wikimedia.org/r/1314622

Marostegui updated the task description. (Show Details)Jul 23 2026, 7:23 AM

clouddb1031 pooled in s2 and s7

Maintenance_bot removed a project: Patch-For-Review.Jul 23 2026, 7:30 AM
gerritbot added a comment.Jul 24 2026, 6:50 AM

Change #1315682 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] mariadb: Productionize clouddb1032

https://gerrit.wikimedia.org/r/1315682

gerritbot added a project: Patch-For-Review.Jul 24 2026, 6:50 AM
gerritbot added a comment.Jul 24 2026, 6:52 AM

Change #1315682 merged by Marostegui:

[operations/puppet@production] mariadb: Productionize clouddb1032

https://gerrit.wikimedia.org/r/1315682

Maintenance_bot removed a project: Patch-For-Review.Jul 24 2026, 7:31 AM
Marostegui added a comment.Jul 24 2026, 9:46 AM

clouddb1032 replicating on s4 and s6

gerritbot added a comment.Jul 24 2026, 9:47 AM

Change #1315781 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] installserver: Do not format clouddb1032

https://gerrit.wikimedia.org/r/1315781

gerritbot added a project: Patch-For-Review.Jul 24 2026, 9:47 AM
gerritbot added a comment.Jul 24 2026, 9:50 AM

Change #1315781 merged by Marostegui:

[operations/puppet@production] installserver: Do not format clouddb1032

https://gerrit.wikimedia.org/r/1315781

Maintenance_bot removed a project: Patch-For-Review.Jul 24 2026, 10:30 AM
gerritbot added a comment.Jul 27 2026, 6:03 AM

Change #1317825 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] check_private_data_report: Add clouddb1032

https://gerrit.wikimedia.org/r/1317825

gerritbot added a project: Patch-For-Review.Jul 27 2026, 6:03 AM
gerritbot added a comment.Jul 27 2026, 6:07 AM

Change #1317825 merged by Marostegui:

[operations/puppet@production] check_private_data_report: Add clouddb1032

https://gerrit.wikimedia.org/r/1317825

gerritbot added a comment.Jul 27 2026, 6:22 AM

Change #1317831 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] eqiad.yaml: Add clouddb1032

https://gerrit.wikimedia.org/r/1317831

gerritbot added a comment.Jul 27 2026, 6:23 AM

Change #1317831 merged by Marostegui:

[operations/puppet@production] eqiad.yaml: Add clouddb1032

https://gerrit.wikimedia.org/r/1317831

Marostegui added a comment.Jul 27 2026, 6:33 AM

clouddb1032 pooled: s4 and s6

Marostegui updated the task description. (Show Details)Jul 27 2026, 6:34 AM
gerritbot added a comment.Jul 27 2026, 6:44 AM

Change #1317834 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] mariadb: Productionize clouddb1033

https://gerrit.wikimedia.org/r/1317834

gerritbot added a comment.Jul 27 2026, 6:46 AM

Change #1317834 merged by Marostegui:

[operations/puppet@production] mariadb: Productionize clouddb1033

https://gerrit.wikimedia.org/r/1317834

Maintenance_bot removed a project: Patch-For-Review.Jul 27 2026, 7:31 AM
Marostegui updated the task description. (Show Details)Jul 27 2026, 8:38 AM
Marostegui added a comment.Jul 27 2026, 11:04 AM

clouddb1033 replicating s5 and s8

gerritbot added a comment.Jul 28 2026, 6:51 AM

Change #1318564 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] installserver: Do not format clouddb1033

https://gerrit.wikimedia.org/r/1318564

gerritbot added a project: Patch-For-Review.Jul 28 2026, 6:51 AM
gerritbot added a comment.Jul 28 2026, 6:58 AM

Change #1318564 merged by Marostegui:

[operations/puppet@production] installserver: Do not format clouddb1033

https://gerrit.wikimedia.org/r/1318564

Maintenance_bot removed a project: Patch-For-Review.Jul 28 2026, 7:31 AM
gerritbot added a comment.Jul 29 2026, 6:03 AM

Change #1318973 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] check_private_data_report: Add clouddb1033

https://gerrit.wikimedia.org/r/1318973

gerritbot added a project: Patch-For-Review.Jul 29 2026, 6:03 AM

Change #1318973 merged by Marostegui:

[operations/puppet@production] check_private_data_report: Add clouddb1033

https://gerrit.wikimedia.org/r/1318973

Maintenance_bot removed a project: Patch-For-Review.Jul 29 2026, 6:31 AM
gerritbot added a comment.Jul 29 2026, 7:48 AM

Change #1318989 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] eqiad.yaml: Add clouddb1033

https://gerrit.wikimedia.org/r/1318989

gerritbot added a project: Patch-For-Review.Jul 29 2026, 7:48 AM
gerritbot added a comment.Jul 29 2026, 8:34 AM

Change #1318989 merged by Marostegui:

[operations/puppet@production] eqiad.yaml: Add clouddb1033

https://gerrit.wikimedia.org/r/1318989

gerritbot added a comment.Jul 29 2026, 9:30 AM

Change #1319044 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] eqiad.yaml: Fix hostname

https://gerrit.wikimedia.org/r/1319044

gerritbot added a comment.Jul 29 2026, 9:32 AM

Change #1319044 merged by Marostegui:

[operations/puppet@production] eqiad.yaml: Fix hostname

https://gerrit.wikimedia.org/r/1319044

gerritbot added a comment.Jul 29 2026, 9:34 AM

Change #1319045 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] clouddb1033: Enable notifications

https://gerrit.wikimedia.org/r/1319045

gerritbot added a comment.Jul 29 2026, 9:35 AM

Change #1319045 merged by Marostegui:

[operations/puppet@production] clouddb1033: Enable notifications

https://gerrit.wikimedia.org/r/1319045

Maintenance_bot removed a project: Patch-For-Review.Jul 29 2026, 10:32 AM
Marostegui updated the task description. (Show Details)Jul 29 2026, 1:07 PM

clouddb1033 pooled in s5 and s8

BLiviero-WMF subscribed.Jul 29 2026, 5:29 PM

Hi! I got notice from DCOps that clouddb1030-1033 appear to be in "InSetup" Puppet mode. Just checking to see if they are configured as such intentionally, and will transition to in use soon?

BLiviero-WMF added a comment.Jul 29 2026, 5:40 PM

I should add that the InSetup message I received (regarding these hosts) was July 15 and so these systems may have transitioned out of that since! but i don't have a way of knowing

Marostegui added a comment.Jul 29 2026, 5:54 PM
In T409557#12168694, @BLiviero-WMF wrote:

I should add that the InSetup message I received (regarding these hosts) was July 15 and so these systems may have transitioned out of that since! but i don't have a way of knowing

They are not in setup anymore:
clouddb1030 was changed 21st July: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1312788/2/manifests/site.pp
clouddb1031 was changed 22nd July: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1313864/2/manifests/site.pp
clouddb1032 was changed 24th July: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1315682/1/manifests/site.pp
clouddb1033 was changed 27th July https://gerrit.wikimedia.org/r/c/operations/puppet/+/1317834/1/manifests/site.pp

Marostegui closed this task as Resolved.Aug 5 2026, 7:45 AM

All new hosts are now serving traffic. Closing this task and will open a new one to decommission the old ones if all goes fine, in a month.

Marostegui mentioned this in T434048: Decommission clouddb1013-clouddb1020.Aug 5 2026, 7:48 AM
Marostegui added a parent task: T434048: Decommission clouddb1013-clouddb1020.
Stashbot added a comment.Aug 5 2026, 7:53 AM

Mentioned in SAL (#wikimedia-operations) [2026-08-05T07:53:14Z] <marostegui> Depool clouddb1013:s1 T409557

dcaro moved this task from Backlog to Done on the tools-platform-team board.Aug 5 2026, 7:58 AM
Maintenance_bot moved this task from In progress to Done on the DBA board.Aug 5 2026, 8:29 AM
fnegri added a comment.Aug 18 2026, 12:53 PM

clouddb1032 was pooled on 2026-07-27: https://sal.toolforge.org/log/cFFDop8B8tZ8Ohr0dhaj

The HAProxy logs show the healthcheck was successful:

root@cloudlb1001:~# journalctl -u haproxy --since -30d -g 1032 --no-pager
-- Boot a8fbf9fa7b2d49cca4b66d01e1942c2c --
Jul 27 06:25:56 cloudlb1001 haproxy[2152137]: [WARNING]  (2152137) : Health check for server wikireplica-db-analytics-s6/clouddb1032.eqiad.wmnet succeeded, reason: Layer7 check passed, code: 0, info: "5.5.5-10.11.18-MariaDB", check duration: 1ms, status: 10/10 UP.
Jul 27 06:26:00 cloudlb1001 haproxy[2153697]: [WARNING]  (2153697) : Health check for server wikireplica-db-analytics-s4/clouddb1032.eqiad.wmnet succeeded, reason: Layer7 check passed, code: 0, info: "5.5.5-10.11.18-MariaDB", check duration: 1ms, status: 10/10 UP.
Jul 27 06:26:00 cloudlb1001 haproxy[2153697]: [WARNING]  (2153697) : Health check for server wikireplica-db-analytics-s6/clouddb1032.eqiad.wmnet succeeded, reason: Layer7 check passed, code: 0, info: "5.5.5-10.11.18-MariaDB", check duration: 0ms, status: 10/10 UP.
Jul 27 06:29:18 cloudlb1001 haproxy[2253999]: [WARNING]  (2253999) : Health check for server wikireplica-db-analytics-s4/clouddb1032.eqiad.wmnet succeeded, reason: Layer7 check passed, code: 0, info: "5.5.5-10.11.18-MariaDB", check duration: 1ms, status: 10/10 UP.
Jul 27 06:29:18 cloudlb1001 haproxy[2253999]: [WARNING]  (2253999) : Health check for server wikireplica-db-analytics-s6/clouddb1032.eqiad.wmnet succeeded, reason: Layer7 check passed, code: 0, info: "5.5.5-10.11.18-MariaDB", check duration: 1ms, status: 10/10 UP.
Jul 27 06:29:21 cloudlb1001 haproxy[2255973]: [WARNING]  (2255973) : Health check for server wikireplica-db-analytics-s4/clouddb1032.eqiad.wmnet succeeded, reason: Layer7 check passed, code: 0, info: "5.5.5-10.11.18-MariaDB", check duration: 1ms, status: 10/10 UP.
Jul 27 06:29:21 cloudlb1001 haproxy[2255973]: [WARNING]  (2255973) : Health check for server wikireplica-db-analytics-s6/clouddb1032.eqiad.wmnet succeeded, reason: Layer7 check passed, code: 0, info: "5.5.5-10.11.18-MariaDB", check duration: 1ms, status: 10/10 UP.

However, today confctl was returning this:

fnegri@cumin1003:~$ sudo confctl select name=clouddb1032.eqiad.wmnet get
{"clouddb1032.eqiad.wmnet": {"weight": 0, "pooled": "inactive"}, "tags": "dc=eqiad,cluster=wikireplica-db-analytics,service=s4"}
{"clouddb1032.eqiad.wmnet": {"weight": 0, "pooled": "inactive"}, "tags": "dc=eqiad,cluster=wikireplica-db-analytics,service=s6"}

I could not find an explanation either in SAL or in HAProxy logs. I repooled it with:

fnegri@cumin1003:~$ sudo confctl select name=clouddb1032.eqiad.wmnet set/pooled=yes
The selector you chose has selected the following objects:
{"/eqiad/wikireplica-db-analytics/s4": ["clouddb1032.eqiad.wmnet"], "/eqiad/wikireplica-db-analytics/s6": ["clouddb1032.eqiad.wmnet"]}
Ok to continue? [y/N]
confctl>y
eqiad/wikireplica-db-analytics/s4/clouddb1032.eqiad.wmnet: pooled changed inactive => yes
eqiad/wikireplica-db-analytics/s6/clouddb1032.eqiad.wmnet: pooled changed inactive => yes
WARNING:conftool.announce:conftool action : set/pooled=yes; selector: name=clouddb1032.eqiad.wmnet

fnegri@cumin1003:~$ sudo confctl select name=clouddb1032.eqiad.wmnet set/weight=100
The selector you chose has selected the following objects:
{"/eqiad/wikireplica-db-analytics/s6": ["clouddb1032.eqiad.wmnet"], "/eqiad/wikireplica-db-analytics/s4": ["clouddb1032.eqiad.wmnet"]}
Ok to continue? [y/N]
confctl>y
eqiad/wikireplica-db-analytics/s6/clouddb1032.eqiad.wmnet: weight changed 0 => 100
eqiad/wikireplica-db-analytics/s4/clouddb1032.eqiad.wmnet: weight changed 0 => 100
WARNING:conftool.announce:conftool action : set/weight=100; selector: name=clouddb1032.eqiad.wmnet

fnegri@cumin1003:~$ sudo confctl select name=clouddb1032.eqiad.wmnet get
{"clouddb1032.eqiad.wmnet": {"weight": 100, "pooled": "yes"}, "tags": "dc=eqiad,cluster=wikireplica-db-analytics,service=s6"}
{"clouddb1032.eqiad.wmnet": {"weight": 100, "pooled": "yes"}, "tags": "dc=eqiad,cluster=wikireplica-db-analytics,service=s4"}
gerritbot added a comment.Aug 18 2026, 1:11 PM

Change #1326830 had a related patch set uploaded (by FNegri; author: FNegri):

[operations/puppet@production] clouddb: Revert temp changes to clouddb1025

https://gerrit.wikimedia.org/r/1326830

gerritbot added a project: Patch-For-Review.Aug 18 2026, 1:11 PM
fnegri added a comment.Aug 18 2026, 1:19 PM

clouddb1024 and clouddb1025 should have the same config, they diverged when we needed a replacement for clouddb1019. I created https://gerrit.wikimedia.org/r/1326830 to put them back in sync.

gerritbot added a comment.Aug 20 2026, 1:32 PM

Change #1326830 merged by FNegri:

[operations/puppet@production] clouddb: Revert temp changes to clouddb1025

https://gerrit.wikimedia.org/r/1326830

fnegri added a comment.Aug 20 2026, 2:06 PM

I removed s6 from clouddb1025, with the following steps:

root@clouddb1025:~# systemctl disable wmf-pt-kill@s6.service
Removed "/etc/systemd/system/multi-user.target.wants/wmf-pt-kill@s6.service".
root@clouddb1025:~# systemctl disable wmf-pt-kill@s6.service
root@clouddb1025:~# systemctl disable wmf_auto_restart_prometheus-mysqld-exporter@s6.timer
Removed "/etc/systemd/system/multi-user.target.wants/wmf_auto_restart_prometheus-mysqld-exporter@s6.timer".
root@clouddb1025:~# systemctl disable wmf_auto_restart_prometheus-mysqld-exporter@s6.service
root@clouddb1025:~# systemctl reset-failed
root@clouddb1025:~# systemctl daemon-reload
root@clouddb1025:~# rm /usr/lib/systemd/system/wmf-pt-kill@s6.service
root@clouddb1025:~# rm /usr/lib/systemd/system/wmf_auto_restart_prometheus-mysqld-exporter@s6.timer
root@clouddb1025:~# rm /usr/lib/systemd/system/wmf_auto_restart_prometheus-mysqld-exporter@s6.service
  • cleaned up the data dirs:
root@clouddb1025:~# rm -rf /srv/sqldata.s6/
root@clouddb1025:~# rm -rf /srv/tmp.s6/
  • repooled clouddb1025 in confctl (note that the puppet patch above also updated the section name in confctl from s4 to x4, matching the config in clouddb1024):
fnegri@cumin1003:~$ sudo confctl select name=clouddb1025.eqiad.wmnet set/pooled=yes
fnegri@cumin1003:~$ sudo confctl select name=clouddb1025.eqiad.wmnet set/weight=100
fnegri@cumin1003:~$ sudo confctl select name=clouddb1025.eqiad.wmnet get
{"clouddb1025.eqiad.wmnet": {"weight": 100, "pooled": "yes"}, "tags": "dc=eqiad,cluster=wikireplica-db-analytics,service=x4"}
fnegri@cumin1003:~$ sudo confctl select name=clouddb1024.eqiad.wmnet get
{"clouddb1024.eqiad.wmnet": {"weight": 100, "pooled": "yes"}, "tags": "dc=eqiad,cluster=wikireplica-db-web,service=x4"}
Maintenance_bot removed a project: Patch-For-Review.Aug 20 2026, 2:31 PM
fnegri updated the task description. (Show Details)Aug 21 2026, 9:48 AM
fnegri added a comment.Aug 21 2026, 9:54 AM
root@clouddb1025:~# systemctl reset-failed
root@clouddb1025:~# systemctl daemon-reload
root@clouddb1025:~# rm /usr/lib/systemd/system/wmf-pt-kill@s6.service
root@clouddb1025:~# rm /usr/lib/systemd/system/wmf_auto_restart_prometheus-mysqld-exporter@s6.timer
root@clouddb1025:~# rm /usr/lib/systemd/system/wmf_auto_restart_prometheus-mysqld-exporter@s6.service

The order of the commands above was wrong, and systemd tried to restart wmf_auto_restart_prometheus-mysqld-exporter@s6.service a few hours later. systemctl daemon-reload must be run after removing the files from /usr/lib/systemd. I ran it again now, and the issue should be fixed.

Marostegui closed subtask T404715: Setup x4 section as Resolved.Sep 8 2026, 9:51 AM
gerritbot added a comment.Sep 10 2026, 10:06 AM

Change #1338768 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] clouddb1024: Add x4

https://gerrit.wikimedia.org/r/1338768

gerritbot added a project: Patch-For-Review.Sep 10 2026, 10:06 AM

Change #1338768 merged by Marostegui:

[operations/puppet@production] clouddb1024: Add x4

https://gerrit.wikimedia.org/r/1338768

Maintenance_bot removed a project: Patch-For-Review.Sep 10 2026, 10:30 AM
Marostegui added a comment.Sep 10 2026, 11:31 AM

clouddb1024 is now running x4 - it is not yet ready to hold views, I need to make sure everything in there is okay and filtered.

gerritbot added a comment.Sep 10 2026, 11:31 AM

Change #1338830 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] check_private_data_report: Add x4

https://gerrit.wikimedia.org/r/1338830

gerritbot added a project: Patch-For-Review.Sep 10 2026, 11:31 AM
gerritbot added a comment.Sep 10 2026, 11:48 AM

Change #1338830 merged by Marostegui:

[operations/puppet@production] check_private_data_report: Add x4

https://gerrit.wikimedia.org/r/1338830

gerritbot added a comment.Sep 10 2026, 12:05 PM

Change #1338868 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] clouddb1025: Add x4

https://gerrit.wikimedia.org/r/1338868

gerritbot added a comment.Sep 10 2026, 12:06 PM

Change #1338868 merged by Marostegui:

[operations/puppet@production] clouddb1025: Add x4

https://gerrit.wikimedia.org/r/1338868

Marostegui added a comment.Sep 11 2026, 5:36 AM

clouddb1025 is now running x4 - it is not yet ready to hold views, I need to make sure everything in there is okay and filtered.

Maintenance_bot removed a project: Patch-For-Review.Sep 11 2026, 6:31 AM
taavi added a comment.Sep 14 2026, 9:08 AM

created views and pooled clouddb1024/5:x4

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits