Page MenuHomePhabricator
Create Task
Maniphest T393989

Productionize new x3 hosts
Closed, ResolvedPublic
Actions

Description

For x3 expansion, we need to productionize:

  • db2244
  • db1258

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
db2244: Enable notificationsoperations/puppetproduction+0 -1
instances.yaml: Add db2244 to dbctloperations/puppetproduction+1 -0
mariadb: Productionize db2244operations/puppetproduction+4 -7
installserver: Remove db1258operations/puppetproduction+1 -1
instances.yaml: Add db1258 to dbctloperations/puppetproduction+1 -0
mariadb: Productionize db1258operations/puppetproduction+3 -6
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
Unknown Object (Task)
 ResolvedJhancock.wmT379757 Q2:rack/setup/install db224[12]
Unknown Object (Task)
Unknown Object (Task)
 ResolvedJhancock.wmT382425 Q2:rack/setup/install db2243
Unknown Object (Task)
 ResolvedNoneT351820 Move Wikidata term store to separate database cluster
 ResolvedJclark-ctrT384979 Q3:rack/setup/install db1257
 ResolvedMarosteguiT381475 Productionize x3 hosts
 ResolvedMarosteguiT390530 Create topology for x3 hosts
Unknown Object (Task)
 ResolvedJhancock.wmT393195 Q4:rack/setup/install db2244
Unknown Object (Task)
 ResolvedJclark-ctrT392493 Q4:rack/setup/install db1258
 ResolvedMarosteguiT393989 Productionize new x3 hosts

Event Timeline

Marostegui created this task.May 13 2025, 6:08 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 13 2025, 6:08 AM
Marostegui triaged this task as Medium priority.May 13 2025, 6:09 AM
Marostegui moved this task from Triage to Blocked on the DBA board.
Marostegui added a parent task: T393195: Q4:rack/setup/install db2244.
Marostegui added a parent task: T392493: Q4:rack/setup/install db1258.
Marostegui added a parent task: T390530: Create topology for x3 hosts.May 13 2025, 11:21 AM
Marostegui mentioned this in T390530: Create topology for x3 hosts.May 14 2025, 5:15 AM
gerritbot added a comment.May 14 2025, 5:18 AM

Change #1145428 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] mariadb: Productionize db1258

https://gerrit.wikimedia.org/r/1145428

gerritbot added a project: Patch-For-Review.May 14 2025, 5:18 AM
gerritbot added a comment.May 14 2025, 5:29 AM

Change #1145428 merged by Marostegui:

[operations/puppet@production] mariadb: Productionize db1258

https://gerrit.wikimedia.org/r/1145428

Maintenance_bot removed a project: Patch-For-Review.May 14 2025, 5:32 AM
ops-monitoring-bot added a comment.May 14 2025, 5:36 AM

Started cloning db1257.eqiad.wmnet to db1258.eqiad.wmnet - marostegui@cumin1002

ops-monitoring-bot added a comment.May 14 2025, 5:36 AM

Completed depool of db1257 - Depool db1257.eqiad.wmnet to then clone it to db1258.eqiad.wmnet - marostegui@cumin1002 - marostegui@cumin1002

gerritbot added a comment.May 14 2025, 7:48 AM

Change #1145799 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] instances.yaml: Add db1258 to dbctl

https://gerrit.wikimedia.org/r/1145799

gerritbot added a project: Patch-For-Review.May 14 2025, 7:48 AM
gerritbot added a comment.May 14 2025, 7:50 AM

Change #1145799 merged by Marostegui:

[operations/puppet@production] instances.yaml: Add db1258 to dbctl

https://gerrit.wikimedia.org/r/1145799

Stashbot added a comment.May 14 2025, 7:52 AM

Mentioned in SAL (#wikimedia-operations) [2025-05-14T07:52:54Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Add db1258 to dbctl T393989', diff saved to https://phabricator.wikimedia.org/P76101 and previous config saved to /var/cache/conftool/dbconfig/20250514-075254-marostegui.json

Marostegui added a comment.May 14 2025, 7:55 AM

db1258 has been recloned - only pending the repooling, which will happen after 24h.

Marostegui updated the task description. (Show Details)May 14 2025, 7:55 AM
ops-monitoring-bot added a comment.May 14 2025, 8:01 AM

Start pool of db1257 gradually with 4 steps - Pool db1257.eqiad.wmnet in after cloning - marostegui@cumin1002

Maintenance_bot removed a project: Patch-For-Review.May 14 2025, 8:31 AM
ops-monitoring-bot added a comment.May 14 2025, 8:31 AM

Completed pool of db1257 gradually with 4 steps - Pool db1257.eqiad.wmnet in after cloning - marostegui@cumin1002

ops-monitoring-bot added a comment.May 14 2025, 8:31 AM

Finished cloning db1257.eqiad.wmnet to db1258.eqiad.wmnet - marostegui@cumin1002

Marostegui moved this task from Blocked to In progress on the DBA board.May 14 2025, 8:35 AM
Marostegui added a comment.May 14 2025, 12:15 PM
In T393989#10820480, @Marostegui wrote:

db1258 has been recloned - only pending the repooling, which will happen after 24h.

I've decided to start pooling it already, it was working fine.

gerritbot added a comment.May 16 2025, 5:32 AM

Change #1146837 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] installserver: Remove db1258

https://gerrit.wikimedia.org/r/1146837

gerritbot added a project: Patch-For-Review.May 16 2025, 5:32 AM
gerritbot added a comment.May 16 2025, 5:35 AM

Change #1146837 merged by Marostegui:

[operations/puppet@production] installserver: Remove db1258

https://gerrit.wikimedia.org/r/1146837

Maintenance_bot removed a project: Patch-For-Review.May 16 2025, 6:30 AM
Marostegui mentioned this in T351820: Move Wikidata term store to separate database cluster.May 20 2025, 11:39 AM
Marostegui claimed this task.Jun 6 2025, 1:09 PM
Marostegui added a comment.Jun 9 2025, 4:51 AM
root@db2244:~# lvextend -L+1000G /dev/mapper/tank-data ; xfs_growfs /srv ; df -hT /srv ; pvs
  Size of logical volume tank/data changed from <7.56 TiB (1981022 extents) to 8.53 TiB (2237022 extents).
  Logical volume tank/data successfully resized.
meta-data=/dev/mapper/tank-data  isize=512    agcount=32, agsize=63392704 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=1, rmapbt=0
         =                       reflink=1    bigtime=1 inobtcount=1 nrext64=0
data     =                       bsize=4096   blocks=2028566528, imaxpct=5
         =                       sunit=64     swidth=256 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
log      =internal log           bsize=4096   blocks=521728, version=2
         =                       sectsz=512   sunit=64 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
data blocks changed from 2028566528 to 2290710528
Filesystem            Type  Size  Used Avail Use% Mounted on
/dev/mapper/tank-data xfs   8.6T   61G  8.5T   1% /srv
  PV         VG   Fmt  Attr PSize  PFree
  /dev/sda3  tank lvm2 a--  <8.69t 156.30g
gerritbot added a comment.Jun 9 2025, 4:53 AM

Change #1154540 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] mariadb: Productionize db2244

https://gerrit.wikimedia.org/r/1154540

gerritbot added a project: Patch-For-Review.Jun 9 2025, 4:53 AM

Change #1154540 merged by Marostegui:

[operations/puppet@production] mariadb: Productionize db2244

https://gerrit.wikimedia.org/r/1154540

ops-monitoring-bot added a comment.Jun 9 2025, 5:00 AM

Started cloning db2243.codfw.wmnet to db2244.codfw.wmnet - marostegui@cumin1002

ops-monitoring-bot added a comment.Jun 9 2025, 5:00 AM

Completed depool of db2243 - Depool db2243.codfw.wmnet to then clone it to db2244.codfw.wmnet - marostegui@cumin1002 - marostegui@cumin1002

gerritbot added a comment.Jun 9 2025, 5:01 AM

Change #1154541 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] instances.yaml: Add db2244 to dbctl

https://gerrit.wikimedia.org/r/1154541

gerritbot added a comment.Jun 9 2025, 5:02 AM

Change #1154541 merged by Marostegui:

[operations/puppet@production] instances.yaml: Add db2244 to dbctl

https://gerrit.wikimedia.org/r/1154541

Stashbot added a comment.Jun 9 2025, 5:24 AM

Mentioned in SAL (#wikimedia-operations) [2025-06-09T05:24:52Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Add db2244 to dbctl depooled T393989', diff saved to https://phabricator.wikimedia.org/P77205 and previous config saved to /var/cache/conftool/dbconfig/20250609-052451-marostegui.json

Maintenance_bot removed a project: Patch-For-Review.Jun 9 2025, 5:30 AM
ops-monitoring-bot added a comment.Jun 9 2025, 5:37 AM

Start pool of db2243 gradually with 4 steps - Pool db2243.codfw.wmnet in after cloning - marostegui@cumin1002

ops-monitoring-bot added a comment.Jun 9 2025, 6:22 AM

Completed pool of db2243 gradually with 4 steps - Pool db2243.codfw.wmnet in after cloning - marostegui@cumin1002

gerritbot added a comment.Jun 9 2025, 6:23 AM

Change #1154544 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db2244: Enable notifications

https://gerrit.wikimedia.org/r/1154544

gerritbot added a project: Patch-For-Review.Jun 9 2025, 6:23 AM
gerritbot added a comment.Jun 9 2025, 6:27 AM

Change #1154544 merged by Marostegui:

[operations/puppet@production] db2244: Enable notifications

https://gerrit.wikimedia.org/r/1154544

Maintenance_bot removed a project: Patch-For-Review.Jun 9 2025, 6:31 AM
Marostegui added a comment.Jun 9 2025, 6:32 AM

db2244 is going to be pooled automatically

ops-monitoring-bot added a comment.Jun 9 2025, 7:23 AM

Start pool of db2244 gradually with 4 steps - Pool db2244.codfw.wmnet in after cloning - marostegui@cumin1002

ops-monitoring-bot added a comment.Jun 9 2025, 8:08 AM

Completed pool of db2244 gradually with 4 steps - Pool db2244.codfw.wmnet in after cloning - marostegui@cumin1002

ops-monitoring-bot added a comment.Jun 9 2025, 8:08 AM

Finished cloning db2243.codfw.wmnet to db2244.codfw.wmnet - marostegui@cumin1002

Marostegui closed this task as Resolved.Jun 9 2025, 9:38 AM
Marostegui updated the task description. (Show Details)

All done

Maintenance_bot moved this task from In progress to Done on the DBA board.Jun 9 2025, 10:29 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits