Page MenuHomePhabricator
Create Task
Maniphest T395771

Productionize es2047, es2048, es1047, es1048
Closed, ResolvedPublic
Actions

Description

  • es1047 es6
  • es1048 es7
  • es2047 es6
  • es2048 es7

Details

Related Changes in Gerrit:
Show related patches Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
Unknown Object (Task)
 ResolvedJhancock.wmT393106 Q4:rack/setup/install es204[78]
Unknown Object (Task)
 ResolvedVRiley-WMFT393107 Q#:rack/setup/install es104[78]
 Resolved• MarosteguiT395771 Productionize es2047, es2048, es1047, es1048

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
• Marostegui moved this task from Triage to Ready on the DBA board.Jun 2 2025, 5:00 AM
• Marostegui updated the task description. (Show Details)

es2036 will clone es2047

Stashbot added a comment.Jun 2 2025, 5:01 AM

Mentioned in SAL (#wikimedia-operations) [2025-06-02T05:01:50Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Depool es2036 T395771', diff saved to https://phabricator.wikimedia.org/P76779 and previous config saved to /var/cache/conftool/dbconfig/20250602-050150-marostegui.json

gerritbot added a comment.Jun 2 2025, 5:06 AM

Change #1152451 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] mariadb: Productionize es2047 into es6

https://gerrit.wikimedia.org/r/1152451

gerritbot added a project: Patch-For-Review.Jun 2 2025, 5:06 AM
• Marostegui added a comment.Jun 2 2025, 5:08 AM
[05:06:43] marostegui@cumin1002:~$ sudo cumin 'es2047*,es2048*' 'lvextend -L+1000G /dev/mapper/tank-data ; xfs_growfs /srv ; df -hT /srv ; pvs'
2 hosts will be targeted:
es[2047-2048].codfw.wmnet
OK to proceed on 2 hosts? Enter the number of affected hosts to confirm or "q" to quit: 2
===== NODE GROUP =====
(2) es[2047-2048].codfw.wmnet
----- OUTPUT of 'lvextend -L+1000...f -hT /srv ; pvs' -----
  Size of logical volume tank/data changed from 9.09 TiB (2384188 extents) to 10.07 TiB (2640188 extents).
  Logical volume tank/data successfully resized.
meta-data=/dev/mapper/tank-data  isize=512    agcount=32, agsize=76294016 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=1, rmapbt=0
         =                       reflink=1    bigtime=1 inobtcount=1 nrext64=0
data     =                       bsize=4096   blocks=2441408512, imaxpct=5
         =                       sunit=64     swidth=256 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
log      =internal log           bsize=4096   blocks=521728, version=2
         =                       sectsz=512   sunit=64 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
data blocks changed from 2441408512 to 2703552512
Filesystem            Type  Size  Used Avail Use% Mounted on
/dev/mapper/tank-data xfs    11T   72G   10T   1% /srv
  PV         VG   Fmt  Attr PSize   PFree
  /dev/sda3  tank lvm2 a--  <21.79t 11.71t
================
PASS |██████████████████████████████████████████████████████████████████████████████████████████| 100% (2/2) [00:01<00:00,  1.50hosts/s]
FAIL |                                                                                                  |   0% (0/2) [00:01<?, ?hosts/s]
100.0% (2/2) success ratio (>= 100.0% threshold) for command: 'lvextend -L+1000...f -hT /srv ; pvs'.
100.0% (2/2) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
gerritbot added a comment.Jun 2 2025, 5:08 AM

Change #1152451 merged by Marostegui:

[operations/puppet@production] mariadb: Productionize es2047 into es6

https://gerrit.wikimedia.org/r/1152451

ops-monitoring-bot added a comment.Jun 2 2025, 5:15 AM

Started cloning es2036.codfw.wmnet to es2047.codfw.wmnet - marostegui@cumin1002

gerritbot added a comment.Jun 2 2025, 5:16 AM

Change #1152452 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] instances.yaml: Add es2047

https://gerrit.wikimedia.org/r/1152452

gerritbot added a comment.Jun 2 2025, 5:17 AM

Change #1152452 merged by Marostegui:

[operations/puppet@production] instances.yaml: Add es2047

https://gerrit.wikimedia.org/r/1152452

Stashbot added a comment.Jun 2 2025, 5:19 AM

Mentioned in SAL (#wikimedia-operations) [2025-06-02T05:19:59Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Add es2047 to dbctl depooled T395771', diff saved to https://phabricator.wikimedia.org/P76780 and previous config saved to /var/cache/conftool/dbconfig/20250602-051957-marostegui.json

• Marostegui moved this task from Ready to In progress on the DBA board.Jun 2 2025, 5:29 AM
Maintenance_bot removed a project: Patch-For-Review.Jun 2 2025, 5:30 AM
ops-monitoring-bot added a comment.Jun 2 2025, 10:02 AM

Start pool of es2036 gradually with 4 steps - Pool es2036.codfw.wmnet in after cloning - marostegui@cumin1002

• Marostegui added a comment.Jun 2 2025, 10:10 AM

es2047 has been cloned, once es2036 is pooled, it will get pooled too.

gerritbot added a comment.Jun 2 2025, 10:25 AM

Change #1152652 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] es2047: Enable notifications

https://gerrit.wikimedia.org/r/1152652

gerritbot added a project: Patch-For-Review.Jun 2 2025, 10:25 AM

Change #1152652 merged by Marostegui:

[operations/puppet@production] es2047: Enable notifications

https://gerrit.wikimedia.org/r/1152652

Maintenance_bot removed a project: Patch-For-Review.Jun 2 2025, 10:30 AM
• Marostegui updated the task description. (Show Details)Jun 2 2025, 10:32 AM
ops-monitoring-bot added a comment.Jun 2 2025, 10:48 AM

Completed pool of es2036 gradually with 4 steps - Pool es2036.codfw.wmnet in after cloning - marostegui@cumin1002

ops-monitoring-bot added a comment.Jun 2 2025, 11:48 AM

Start pool of es2047 gradually with 4 steps - Pool es2047.codfw.wmnet in after cloning - marostegui@cumin1002

ops-monitoring-bot added a comment.Jun 2 2025, 12:37 PM

Completed pool of es2047 gradually with 4 steps - Pool es2047.codfw.wmnet in after cloning - marostegui@cumin1002

ops-monitoring-bot added a comment.Jun 2 2025, 12:37 PM

Finished cloning es2036.codfw.wmnet to es2047.codfw.wmnet - marostegui@cumin1002

gerritbot added a comment.Jun 3 2025, 4:31 AM

Change #1152884 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] mariadb: Productionize es2048

https://gerrit.wikimedia.org/r/1152884

gerritbot added a project: Patch-For-Review.Jun 3 2025, 4:31 AM
Stashbot added a comment.Jun 3 2025, 4:31 AM

Mentioned in SAL (#wikimedia-operations) [2025-06-03T04:31:52Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Depool es2040 T395771', diff saved to https://phabricator.wikimedia.org/P76869 and previous config saved to /var/cache/conftool/dbconfig/20250603-043151-marostegui.json

gerritbot added a comment.Jun 3 2025, 4:32 AM

Change #1152884 merged by Marostegui:

[operations/puppet@production] mariadb: Productionize es2048

https://gerrit.wikimedia.org/r/1152884

ops-monitoring-bot added a comment.Jun 3 2025, 4:40 AM

Started cloning es2040.codfw.wmnet to es2048.codfw.wmnet - marostegui@cumin1002

gerritbot added a comment.Jun 3 2025, 4:42 AM

Change #1152886 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] instances.yaml: Add es2048 to dbctl

https://gerrit.wikimedia.org/r/1152886

gerritbot added a comment.Jun 3 2025, 4:43 AM

Change #1152886 merged by Marostegui:

[operations/puppet@production] instances.yaml: Add es2048 to dbctl

https://gerrit.wikimedia.org/r/1152886

Stashbot added a comment.Jun 3 2025, 4:45 AM

Mentioned in SAL (#wikimedia-operations) [2025-06-03T04:45:53Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Add es2048 to dbctl depooled T395771+', diff saved to https://phabricator.wikimedia.org/P76870 and previous config saved to /var/cache/conftool/dbconfig/20250603-044550-marostegui.json

Maintenance_bot removed a project: Patch-For-Review.Jun 3 2025, 5:30 AM
ops-monitoring-bot added a comment.Jun 3 2025, 9:18 AM

Start pool of es2040 gradually with 4 steps - Pool es2040.codfw.wmnet in after cloning - marostegui@cumin1002

ops-monitoring-bot added a comment.Jun 3 2025, 11:25 AM

Completed pool of es2040 gradually with 4 steps - Pool es2040.codfw.wmnet in after cloning - marostegui@cumin1002

gerritbot added a comment.Jun 4 2025, 6:22 AM

Change #1153434 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] es2048: Enable notifications

https://gerrit.wikimedia.org/r/1153434

gerritbot added a project: Patch-For-Review.Jun 4 2025, 6:22 AM

Change #1153434 merged by Marostegui:

[operations/puppet@production] es2048: Enable notifications

https://gerrit.wikimedia.org/r/1153434

• Marostegui updated the task description. (Show Details)Jun 4 2025, 6:24 AM
Maintenance_bot removed a project: Patch-For-Review.Jun 4 2025, 6:30 AM
gerritbot added a comment.Jul 9 2025, 7:38 AM

Change #1167520 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] mariadb: Productionize es1047

https://gerrit.wikimedia.org/r/1167520

gerritbot added a project: Patch-For-Review.Jul 9 2025, 7:38 AM
gerritbot added a comment.Jul 9 2025, 7:42 AM

Change #1167520 merged by Marostegui:

[operations/puppet@production] mariadb: Productionize es1047

https://gerrit.wikimedia.org/r/1167520

Maintenance_bot removed a project: Patch-For-Review.Jul 9 2025, 8:31 AM
• Marostegui added a comment.Jul 10 2025, 5:11 PM

es1047 is now replicating in es6 and catching up

gerritbot added a comment.Jul 11 2025, 6:25 AM

Change #1168036 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] mariadb: Productionize es1048

https://gerrit.wikimedia.org/r/1168036

gerritbot added a project: Patch-For-Review.Jul 11 2025, 6:25 AM

Change #1168036 merged by Marostegui:

[operations/puppet@production] mariadb: Productionize es1048

https://gerrit.wikimedia.org/r/1168036

Maintenance_bot removed a project: Patch-For-Review.Jul 11 2025, 6:31 AM
• Marostegui added a comment.Jul 11 2025, 6:47 AM

es1048 is replicating in es7

gerritbot added a comment.Jul 14 2025, 8:40 AM

Change #1169037 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] instances.yaml: Add es1047

https://gerrit.wikimedia.org/r/1169037

gerritbot added a project: Patch-For-Review.Jul 14 2025, 8:40 AM
gerritbot added a comment.Jul 14 2025, 8:43 AM

Change #1169037 merged by Marostegui:

[operations/puppet@production] instances.yaml: Add es1047

https://gerrit.wikimedia.org/r/1169037

Stashbot added a comment.Jul 14 2025, 8:45 AM

Mentioned in SAL (#wikimedia-operations) [2025-07-14T08:45:07Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Add es1047 to es6 depooled T395771', diff saved to https://phabricator.wikimedia.org/P78943 and previous config saved to /var/cache/conftool/dbconfig/20250714-084506-marostegui.json

gerritbot added a comment.Jul 14 2025, 8:50 AM

Change #1169039 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] es1047: Enable notifications

https://gerrit.wikimedia.org/r/1169039

gerritbot added a comment.Jul 14 2025, 8:52 AM

Change #1169039 merged by Marostegui:

[operations/puppet@production] es1047: Enable notifications

https://gerrit.wikimedia.org/r/1169039

Stashbot added a comment.Jul 14 2025, 8:54 AM

Mentioned in SAL (#wikimedia-operations) [2025-07-14T08:54:58Z] <marostegui@cumin1002> dbctl commit (dc=all): 'es1047 (re)pooling @ 1%: Pooling for the first time in es6 T395771', diff saved to https://phabricator.wikimedia.org/P78945 and previous config saved to /var/cache/conftool/dbconfig/20250714-085457-root.json

• Marostegui updated the task description. (Show Details)Jul 14 2025, 9:09 AM
Stashbot added a comment.Jul 14 2025, 9:10 AM

Mentioned in SAL (#wikimedia-operations) [2025-07-14T09:10:03Z] <marostegui@cumin1002> dbctl commit (dc=all): 'es1047 (re)pooling @ 5%: Pooling for the first time in es6 T395771', diff saved to https://phabricator.wikimedia.org/P78949 and previous config saved to /var/cache/conftool/dbconfig/20250714-091003-root.json

Stashbot added a comment.Jul 14 2025, 9:25 AM

Mentioned in SAL (#wikimedia-operations) [2025-07-14T09:25:09Z] <marostegui@cumin1002> dbctl commit (dc=all): 'es1047 (re)pooling @ 10%: Pooling for the first time in es6 T395771', diff saved to https://phabricator.wikimedia.org/P78953 and previous config saved to /var/cache/conftool/dbconfig/20250714-092508-root.json

Maintenance_bot removed a project: Patch-For-Review.Jul 14 2025, 9:32 AM
Stashbot added a comment.Jul 14 2025, 9:40 AM

Mentioned in SAL (#wikimedia-operations) [2025-07-14T09:40:15Z] <marostegui@cumin1002> dbctl commit (dc=all): 'es1047 (re)pooling @ 25%: Pooling for the first time in es6 T395771', diff saved to https://phabricator.wikimedia.org/P78957 and previous config saved to /var/cache/conftool/dbconfig/20250714-094014-root.json

Stashbot added a comment.Jul 14 2025, 9:55 AM

Mentioned in SAL (#wikimedia-operations) [2025-07-14T09:55:22Z] <marostegui@cumin1002> dbctl commit (dc=all): 'es1047 (re)pooling @ 40%: Pooling for the first time in es6 T395771', diff saved to https://phabricator.wikimedia.org/P78963 and previous config saved to /var/cache/conftool/dbconfig/20250714-095520-root.json

Stashbot added a comment.Jul 14 2025, 10:17 AM

Mentioned in SAL (#wikimedia-operations) [2025-07-14T10:17:46Z] <marostegui@cumin1002> dbctl commit (dc=all): 'es1047 (re)pooling @ 50%: Pooling for the first time in es6 T395771', diff saved to https://phabricator.wikimedia.org/P78968 and previous config saved to /var/cache/conftool/dbconfig/20250714-101745-root.json

Stashbot added a comment.Jul 14 2025, 10:32 AM

Mentioned in SAL (#wikimedia-operations) [2025-07-14T10:32:51Z] <marostegui@cumin1002> dbctl commit (dc=all): 'es1047 (re)pooling @ 60%: Pooling for the first time in es6 T395771', diff saved to https://phabricator.wikimedia.org/P78971 and previous config saved to /var/cache/conftool/dbconfig/20250714-103251-root.json

Stashbot added a comment.Jul 14 2025, 10:47 AM

Mentioned in SAL (#wikimedia-operations) [2025-07-14T10:47:57Z] <marostegui@cumin1002> dbctl commit (dc=all): 'es1047 (re)pooling @ 75%: Pooling for the first time in es6 T395771', diff saved to https://phabricator.wikimedia.org/P78976 and previous config saved to /var/cache/conftool/dbconfig/20250714-104756-root.json

Stashbot added a comment.Jul 14 2025, 11:03 AM

Mentioned in SAL (#wikimedia-operations) [2025-07-14T11:03:03Z] <marostegui@cumin1002> dbctl commit (dc=all): 'es1047 (re)pooling @ 90%: Pooling for the first time in es6 T395771', diff saved to https://phabricator.wikimedia.org/P78983 and previous config saved to /var/cache/conftool/dbconfig/20250714-110302-root.json

Stashbot added a comment.Jul 14 2025, 11:18 AM

Mentioned in SAL (#wikimedia-operations) [2025-07-14T11:18:11Z] <marostegui@cumin1002> dbctl commit (dc=all): 'es1047 (re)pooling @ 100%: Pooling for the first time in es6 T395771', diff saved to https://phabricator.wikimedia.org/P78988 and previous config saved to /var/cache/conftool/dbconfig/20250714-111808-root.json

Stashbot added a comment.Jul 15 2025, 12:51 PM

Mentioned in SAL (#wikimedia-operations) [2025-07-15T12:51:58Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Remove weight from the master T395771', diff saved to https://phabricator.wikimedia.org/P79109 and previous config saved to /var/cache/conftool/dbconfig/20250715-125157-marostegui.json

gerritbot added a comment.Jul 18 2025, 8:46 AM

Change #1170503 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] es1048: Enable notifications

https://gerrit.wikimedia.org/r/1170503

gerritbot added a project: Patch-For-Review.Jul 18 2025, 8:46 AM

Change #1170503 merged by Marostegui:

[operations/puppet@production] es1048: Enable notifications

https://gerrit.wikimedia.org/r/1170503

gerritbot added a comment.Jul 18 2025, 8:54 AM

Change #1170504 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] instances.yaml: Add es1048

https://gerrit.wikimedia.org/r/1170504

gerritbot added a comment.Jul 18 2025, 8:54 AM

Change #1170504 merged by Marostegui:

[operations/puppet@production] instances.yaml: Add es1048

https://gerrit.wikimedia.org/r/1170504

Stashbot added a comment.Jul 18 2025, 8:56 AM

Mentioned in SAL (#wikimedia-operations) [2025-07-18T08:56:53Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Add es1048 to es7 depooled T395771', diff saved to https://phabricator.wikimedia.org/P79363 and previous config saved to /var/cache/conftool/dbconfig/20250718-085652-marostegui.json

Stashbot added a comment.Jul 18 2025, 8:57 AM

Mentioned in SAL (#wikimedia-operations) [2025-07-18T08:57:56Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Pool es1048 with 1% weight on es7 T395771', diff saved to https://phabricator.wikimedia.org/P79365 and previous config saved to /var/cache/conftool/dbconfig/20250718-085755-marostegui.json

Maintenance_bot removed a project: Patch-For-Review.Jul 18 2025, 9:32 AM
• Marostegui closed this task as Resolved.Jul 18 2025, 11:40 AM
• Marostegui updated the task description. (Show Details)

es1048 is slowly automatically being repooled.

Maintenance_bot moved this task from In progress to Done on the DBA board.Jul 18 2025, 12:29 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits