Page MenuHomePhabricator

Productionize es2049-es2057
Closed, ResolvedPublic

Description

The hosts have been imaged in https://phabricator.wikimedia.org/T400195

We can use them to replace hosts deployed in 2020 as:

  • es2026 es2 --> es2049
  • es2027 es3 --> es2050
  • es2028 es1 --> es2051
  • es2029 es3 --> es2052
  • es2030 es1 --> es2053
  • es2031 es2 --> es2054
  • es2032 es1 --> es2055
  • es2033 es2 --> es2056
  • es2034 es3 --> es2057

The hosts are being provisioned using https://gitlab.wikimedia.org/repos/sre/wmfmariadbpy/-/blob/puppethelper/wmfdbtools/docs/runbooks/provisioning_es_hosts.adoc?ref_type=heads
with https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/1183646

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
operations/puppetproduction+4 -1
operations/dnsmaster+3 -3
operations/puppetproduction+2 -2
operations/puppetproduction+40 -30
operations/puppetproduction+3 -3
operations/puppetproduction+11 -2
operations/puppetproduction+2 -2
operations/puppetproduction+11 -2
operations/puppetproduction+2 -2
operations/puppetproduction+2 -2
operations/puppetproduction+4 -0
operations/puppetproduction+5 -0
operations/puppetproduction+1 -1
operations/puppetproduction+1 -1
operations/puppetproduction+5 -0
operations/puppetproduction+1 -1
operations/puppetproduction+1 -1
operations/puppetproduction+4 -0
operations/puppetproduction+1 -1
operations/puppetproduction+1 -0
operations/puppetproduction+1 -0
operations/puppetproduction+1 -2
operations/puppetproduction+9 -0
operations/puppetproduction+1 -1
operations/puppetproduction+1 -0
operations/puppetproduction+4 -0
operations/puppetproduction+5 -0
operations/puppetproduction+1 -1
operations/puppetproduction+82 -1
operations/puppetproduction+0 -1
operations/puppetproduction+0 -1
operations/puppetproduction+1 -0
operations/puppetproduction+9 -0
operations/puppetproduction+1 -1
operations/puppetproduction+1 -0
operations/puppetproduction+0 -1
operations/puppetproduction+9 -0
operations/puppetproduction+1 -1
Show related patches Customize query in gerrit

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Completed pool of es2052 gradually with 4 steps - Pooling in new host - fceratto@cumin1002

Change #1184091 abandoned by Federico Ceratto:

[operations/puppet@production] instances.yaml: Add es2049 to dbctl

Reason:

Superseded by commit 0c4e716757a

https://gerrit.wikimedia.org/r/1184091

Change #1194643 merged by Federico Ceratto:

[operations/puppet@production] preseed.yaml: Remove es2053 from preseeding

https://gerrit.wikimedia.org/r/1194643

Change #1194644 merged by Federico Ceratto:

[operations/puppet@production] es2053.yaml: Prepare es2053 for es1

https://gerrit.wikimedia.org/r/1194644

Icinga downtime and Alertmanager silence (ID=813482a5-2a36-45d5-8954-898560fcac05) set by fceratto@cumin1002 for 4 days, 0:00:00 on 1 host(s) and their services with reason: Setting up new ES host

es2053.codfw.wmnet

Change #1194979 had a related patch set uploaded (by Federico Ceratto; author: Federico Ceratto):

[operations/puppet@production] site.pp: Add es2052

https://gerrit.wikimedia.org/r/1194979

Change #1194979 merged by Federico Ceratto:

[operations/puppet@production] site.pp: Remove es2052 from insetup

https://gerrit.wikimedia.org/r/1194979

Icinga downtime and Alertmanager silence (ID=57a39c59-36bc-420a-aa61-35a7f4c5132e) set by fceratto@cumin1002 for 4 days, 0:00:00 on 1 host(s) and their services with reason: Setting up new ES host

es2053.codfw.wmnet

Change #1196020 had a related patch set uploaded (by Federico Ceratto; author: Federico Ceratto):

[operations/puppet@production] site.pp: Remove es2053 from insetup

https://gerrit.wikimedia.org/r/1196020

Change #1196021 had a related patch set uploaded (by Federico Ceratto; author: Federico Ceratto):

[operations/puppet@production] site.pp: Set role for es2053

https://gerrit.wikimedia.org/r/1196021

Change #1196020 merged by Federico Ceratto:

[operations/puppet@production] site.pp: Remove es2053 from insetup

https://gerrit.wikimedia.org/r/1196020

Change #1196030 had a related patch set uploaded (by Federico Ceratto; author: Federico Ceratto):

[operations/puppet@production] preseed.yaml: Remove es2054 from preseeding

https://gerrit.wikimedia.org/r/1196030

Change #1196031 had a related patch set uploaded (by Federico Ceratto; author: Federico Ceratto):

[operations/puppet@production] site.pp: Remove es2054 from insetup

https://gerrit.wikimedia.org/r/1196031

Change #1196032 had a related patch set uploaded (by Federico Ceratto; author: Federico Ceratto):

[operations/puppet@production] site.pp: Set role for es2054

https://gerrit.wikimedia.org/r/1196032

Change #1196033 had a related patch set uploaded (by Federico Ceratto; author: Federico Ceratto):

[operations/puppet@production] es2054.yaml: Prepare es2054 for es2

https://gerrit.wikimedia.org/r/1196033

Change #1196021 merged by Federico Ceratto:

[operations/puppet@production] site.pp: Set role for es2053

https://gerrit.wikimedia.org/r/1196021

Change #1196030 merged by Federico Ceratto:

[operations/puppet@production] preseed.yaml: Remove es2054 from preseeding

https://gerrit.wikimedia.org/r/1196030

Change #1196031 merged by Federico Ceratto:

[operations/puppet@production] site.pp: Remove es2054 from insetup

https://gerrit.wikimedia.org/r/1196031

Change #1196032 merged by Federico Ceratto:

[operations/puppet@production] site.pp: Set role for es2054

https://gerrit.wikimedia.org/r/1196032

Change #1196033 merged by Federico Ceratto:

[operations/puppet@production] es2054.yaml: Prepare es2054 for es2

https://gerrit.wikimedia.org/r/1196033

Started cloning es2032.codfw.wmnet to es2053.codfw.wmnet - fceratto@cumin1002

Completed depool of es2032 - Depool es2032.codfw.wmnet to then clone it to es2053.codfw.wmnet - fceratto@cumin1002 - fceratto@cumin1002

Start pool of es2032 gradually with 4 steps - Pool es2032.codfw.wmnet in after cloning - fceratto@cumin1002

Completed pool of es2032 gradually with 4 steps - Pool es2032.codfw.wmnet in after cloning - fceratto@cumin1002

Finished cloning es2032.codfw.wmnet to es2053.codfw.wmnet - fceratto@cumin1002

Change #1196397 had a related patch set uploaded (by Federico Ceratto; author: Federico Ceratto):

[operations/puppet@production] instances.yaml, es2053.yaml: Prepare es2053 for production

https://gerrit.wikimedia.org/r/1196397

Change #1196397 merged by Federico Ceratto:

[operations/puppet@production] instances.yaml, es2053.yaml: Prepare es2053 for production

https://gerrit.wikimedia.org/r/1196397

Mentioned in SAL (#wikimedia-operations) [2025-10-15T13:56:31Z] <fceratto@cumin1003> dbctl commit (dc=all): 'Add es2053 T402859', diff saved to https://phabricator.wikimedia.org/P83929 and previous config saved to /var/cache/conftool/dbconfig/20251015-135630-fceratto.json

Start pool of es2053 slowly with 10 steps - Pooling in new host - fceratto@cumin1003

Start pool of es2053 slowly with 10 steps - Pooling in new host - fceratto@cumin1003

Started cloning es2033.codfw.wmnet to es2054.codfw.wmnet - fceratto@cumin1003

Completed depool of es2033 - Depool es2033.codfw.wmnet to then clone it to es2054.codfw.wmnet - fceratto@cumin1003 - fceratto@cumin1003

Completed pool of es2053 slowly with 10 steps - Pooling in new host - fceratto@cumin1003

Start pool of es2033 gradually with 4 steps - Pool es2033.codfw.wmnet in after cloning - fceratto@cumin1003

Completed pool of es2033 gradually with 4 steps - Pool es2033.codfw.wmnet in after cloning - fceratto@cumin1003

Finished cloning es2033.codfw.wmnet to es2054.codfw.wmnet - fceratto@cumin1003

Change #1196632 had a related patch set uploaded (by Federico Ceratto; author: Federico Ceratto):

[operations/puppet@production] es2054.yaml, instances.yaml: enable notifications, add es2054 to dbctl

https://gerrit.wikimedia.org/r/1196632

Change #1196632 merged by Federico Ceratto:

[operations/puppet@production] es2054.yaml, instances.yaml: enable notifications, add es2054 to dbctl

https://gerrit.wikimedia.org/r/1196632

Mentioned in SAL (#wikimedia-operations) [2025-10-16T09:55:34Z] <fceratto@cumin1003> dbctl commit (dc=all): 'Add es2054 T402859', diff saved to https://phabricator.wikimedia.org/P84023 and previous config saved to /var/cache/conftool/dbconfig/20251016-095534-fceratto.json

Start pool of es2054 slowly with 10 steps - Pooling in new host - fceratto@cumin1003

Completed pool of es2054 slowly with 10 steps - Pooling in new host - fceratto@cumin1003

Change #1196696 had a related patch set uploaded (by Federico Ceratto; author: Federico Ceratto):

[operations/puppet@production] preseed.yaml, es2055.yaml, site.pp: Prepare es2055 for es1

https://gerrit.wikimedia.org/r/1196696

Change #1196696 merged by Federico Ceratto:

[operations/puppet@production] preseed.yaml, es2055.yaml, site.pp: Prepare es2055 for es1

https://gerrit.wikimedia.org/r/1196696

Icinga downtime and Alertmanager silence (ID=0e29204c-eb5d-4aa0-9f0c-07cc01ac4607) set by fceratto@cumin1003 for 4 days, 0:00:00 on 1 host(s) and their services with reason: Setting up new ES host

es2055.codfw.wmnet

Started cloning es2032.codfw.wmnet to es2055.codfw.wmnet - fceratto@cumin1003

Completed depool of es2032 - Depool es2032.codfw.wmnet to then clone it to es2055.codfw.wmnet - fceratto@cumin1003 - fceratto@cumin1003

Started cloning es2032.codfw.wmnet to es2055.codfw.wmnet - fceratto@cumin1003

Start pool of es2032 gradually with 4 steps - Pool es2032.codfw.wmnet in after cloning - fceratto@cumin1003

Change #1197216 had a related patch set uploaded (by Federico Ceratto; author: Federico Ceratto):

[operations/puppet@production] es2055.yaml: enable notifications

https://gerrit.wikimedia.org/r/1197216

Change #1197216 merged by Federico Ceratto:

[operations/puppet@production] es2055.yaml, instances.yaml: prepare es2055

https://gerrit.wikimedia.org/r/1197216

Mentioned in SAL (#wikimedia-operations) [2025-10-20T10:04:19Z] <fceratto@cumin1003> dbctl commit (dc=all): 'Add es2055 T402859', diff saved to https://phabricator.wikimedia.org/P84110 and previous config saved to /var/cache/conftool/dbconfig/20251020-100419-fceratto.json

Start pool of es2055 gradually with 4 steps - Pooling in new host - fceratto@cumin1003

Completed pool of es2055 gradually with 4 steps - Pooling in new host - fceratto@cumin1003

Change #1197238 had a related patch set uploaded (by Federico Ceratto; author: Federico Ceratto):

[operations/puppet@production] site.pp, es2056.yaml, preseed.yaml: Prepare es2056 for es2

https://gerrit.wikimedia.org/r/1197238

Change #1197238 merged by Federico Ceratto:

[operations/puppet@production] site.pp, es2056.yaml, preseed.yaml: Prepare es2056 for es2

https://gerrit.wikimedia.org/r/1197238

Icinga downtime and Alertmanager silence (ID=77ca089d-fd1b-4be1-a31f-08bb0e0f3f36) set by fceratto@cumin1003 for 4 days, 0:00:00 on 1 host(s) and their services with reason: Setting up new ES host

es2056.codfw.wmnet

Started cloning es2033.codfw.wmnet to es2056.codfw.wmnet - fceratto@cumin1003

Completed depool of es2033 - Depool es2033.codfw.wmnet to then clone it to es2056.codfw.wmnet - fceratto@cumin1003 - fceratto@cumin1003

Start pool of es2033 gradually with 4 steps - Pool es2033.codfw.wmnet in after cloning - fceratto@cumin1003

Completed pool of es2033 gradually with 4 steps - Pool es2033.codfw.wmnet in after cloning - fceratto@cumin1003

Finished cloning es2033.codfw.wmnet to es2056.codfw.wmnet - fceratto@cumin1003

Change #1197594 had a related patch set uploaded (by Federico Ceratto; author: Federico Ceratto):

[operations/puppet@production] instances.yaml, es2056.yaml: prepare es2056

https://gerrit.wikimedia.org/r/1197594

Change #1197594 merged by Federico Ceratto:

[operations/puppet@production] instances.yaml, es2056.yaml: prepare es2056

https://gerrit.wikimedia.org/r/1197594

Start pool of es2056 slowly with 10 steps - Pooling in new host - fceratto@cumin1003

Change #1197609 had a related patch set uploaded (by Federico Ceratto; author: Federico Ceratto):

[operations/puppet@production] preseed.yaml, site.pp, es2057.yaml: Prepare es2057 for es3

https://gerrit.wikimedia.org/r/1197609

Completed pool of es2056 slowly with 10 steps - Pooling in new host - fceratto@cumin1003

Change #1197609 merged by Federico Ceratto:

[operations/puppet@production] preseed.yaml, site.pp, es2057.yaml: Prepare es2057 for es3

https://gerrit.wikimedia.org/r/1197609

Icinga downtime and Alertmanager silence (ID=66d84406-ed86-43eb-90da-93fbec547332) set by fceratto@cumin1003 for 4 days, 0:00:00 on 1 host(s) and their services with reason: Setting up new ES host

es2057.codfw.wmnet

Started cloning es2034.codfw.wmnet to es2057.codfw.wmnet - fceratto@cumin1003

Completed depool of es2034 - Depool es2034.codfw.wmnet to then clone it to es2057.codfw.wmnet - fceratto@cumin1003 - fceratto@cumin1003

Start pool of es2034 gradually with 4 steps - Pooling in - fceratto@cumin1003

Change #1198290 had a related patch set uploaded (by Federico Ceratto; author: Federico Ceratto):

[operations/puppet@production] instances.yaml, es2057.yaml: Prepare for production

https://gerrit.wikimedia.org/r/1198290

Completed pool of es2034 gradually with 4 steps - Pooling in - fceratto@cumin1003

Change #1198290 merged by Federico Ceratto:

[operations/puppet@production] instances.yaml, es2057.yaml: Prepare for production

https://gerrit.wikimedia.org/r/1198290

Mentioned in SAL (#wikimedia-operations) [2025-10-23T15:40:58Z] <fceratto@cumin1003> dbctl commit (dc=all): 'Add es2057 T402859', diff saved to https://phabricator.wikimedia.org/P84277 and previous config saved to /var/cache/conftool/dbconfig/20251023-154056-fceratto.json

Start pool of es2057 slowly with 10 steps - Pooling in new host - fceratto@cumin1003

Completed pool of es2057 slowly with 10 steps - Pooling in new host - fceratto@cumin1003

FCeratto-WMF updated the task description. (Show Details)
FCeratto-WMF changed the status of subtask T406691: Decommission es20[26-34] from Open to In Progress.

Change #1201037 had a related patch set uploaded (by Federico Ceratto; author: Federico Ceratto):

[operations/dns@master] Flip es1, es2, es3 masters

https://gerrit.wikimedia.org/r/1201037

Change #1201037 merged by Federico Ceratto:

[operations/dns@master] Flip es1, es2, es3 masters

https://gerrit.wikimedia.org/r/1201037

Mentioned in SAL (#wikimedia-operations) [2025-11-03T13:00:12Z] <fceratto@cumin1003> dbctl commit (dc=all): 'Update masters for T402859', diff saved to https://phabricator.wikimedia.org/P84630 and previous config saved to /var/cache/conftool/dbconfig/20251103-130011-fceratto.json

Mentioned in SAL (#wikimedia-operations) [2025-11-03T13:33:43Z] <fceratto@cumin1003> dbctl commit (dc=all): 'Update masters for T402859', diff saved to https://phabricator.wikimedia.org/P84633 and previous config saved to /var/cache/conftool/dbconfig/20251103-133342-fceratto.json

Change #1184544 merged by Federico Ceratto:

[operations/puppet@production] mysqld_exporter.pp: make /var/log/prometheus 0775

https://gerrit.wikimedia.org/r/1184544