Page MenuHomePhabricator

Productionize es20[26-34] and es10[26-34]
Open, MediumPublic

Description

The following hosts will be used to replace existing es1, es2 and es3 hosts.

eqiad:

  • es1026
  • es1027
  • es1028
  • es1029
  • es1030
  • es1031
  • es1032
  • es1033
  • es1034

codfw:
(reminder: extend their lvm: lvextend -L+1600G /dev/mapper/tank-data && xfs_growfs /srv
(set them to active on netbox)

  • es2026 (A1) -> es2014 (es2)
  • es2027 (A6) -> es2017 (es3)
  • es2028 (B1) -> es2011 (es1)
  • es2029 (B1) -> es2018 (es3)
  • es2030 (C1) -> es2012 (es1)
  • es2031 (C1) -> es2015 (es2)
  • es2032 (D1) -> es2013 (es1)
  • es2033 (D1) -> es2016 (es2)
  • es2034 (A6) -> es2019 (es3)

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Mentioned in SAL (#wikimedia-operations) [2020-09-18T05:37:59Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool es2018, es2012 after cloning es2029 and es2030 T261717', diff saved to https://phabricator.wikimedia.org/P12643 and previous config saved to /var/cache/conftool/dbconfig/20200918-053758-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2020-09-18T06:01:03Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool es2018, es2012 after cloning es2029 and es2030 T261717', diff saved to https://phabricator.wikimedia.org/P12644 and previous config saved to /var/cache/conftool/dbconfig/20200918-060103-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2020-09-18T06:21:27Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Fully repool es2018, es2012 after cloning es2029 and es2030 T261717', diff saved to https://phabricator.wikimedia.org/P12647 and previous config saved to /var/cache/conftool/dbconfig/20200918-062127-marostegui.json

Change 628632 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] es2029,es2030: Enable notifications

https://gerrit.wikimedia.org/r/628632

Change 628632 merged by Marostegui:
[operations/puppet@production] es2029,es2030: Enable notifications

https://gerrit.wikimedia.org/r/628632

Mentioned in SAL (#wikimedia-operations) [2020-09-21T04:31:55Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Pool es2029 and es2030 for the first time with minimal weight T261717', diff saved to https://phabricator.wikimedia.org/P12670 and previous config saved to /var/cache/conftool/dbconfig/20200921-043154-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2020-09-21T04:59:20Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Pool es2029 and es2030 with more weight T261717', diff saved to https://phabricator.wikimedia.org/P12671 and previous config saved to /var/cache/conftool/dbconfig/20200921-045919-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2020-09-21T05:02:28Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Set es2015 as es2 codfw master T261717', diff saved to https://phabricator.wikimedia.org/P12672 and previous config saved to /var/cache/conftool/dbconfig/20200921-050228-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2020-09-21T05:03:05Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool es2013,es2016 and es2019 to clone new hosts T261717', diff saved to https://phabricator.wikimedia.org/P12673 and previous config saved to /var/cache/conftool/dbconfig/20200921-050305-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2020-09-21T05:06:32Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Pool es2029 and es2030 with more weight T261717', diff saved to https://phabricator.wikimedia.org/P12674 and previous config saved to /var/cache/conftool/dbconfig/20200921-050632-marostegui.json

Change 628635 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Productionize es2031,es2033,es2034

https://gerrit.wikimedia.org/r/628635

Change 628635 merged by Marostegui:
[operations/puppet@production] mariadb: Productionize es2031,es2033,es2034

https://gerrit.wikimedia.org/r/628635

Mentioned in SAL (#wikimedia-operations) [2020-09-21T05:18:13Z] <marostegui> Stop mysql on: es2013 es2016 es2019 to clone es2032 es2033 es2034 - T261717

On-going transfers:

es2013 -> es2032
es2016 -> es2033
es2019 -> es2034

Mentioned in SAL (#wikimedia-operations) [2020-09-21T05:27:05Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Pool es2029 and es2030 with more weight T261717', diff saved to https://phabricator.wikimedia.org/P12675 and previous config saved to /var/cache/conftool/dbconfig/20200921-052704-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2020-09-21T05:47:30Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Pool es2029 and es2030 with more weight T261717', diff saved to https://phabricator.wikimedia.org/P12676 and previous config saved to /var/cache/conftool/dbconfig/20200921-054730-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2020-09-21T06:00:53Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Fully pool es2029 and es2030 T261717', diff saved to https://phabricator.wikimedia.org/P12677 and previous config saved to /var/cache/conftool/dbconfig/20200921-060053-marostegui.json

es2029 fully pooled in es3
es2030 fully pooled in es1

Mentioned in SAL (#wikimedia-operations) [2020-09-22T04:59:20Z] <marostegui@cumin1001> dbctl commit (dc=all): 'es2013 (re)pooling @ 25%: Slowly repool after recloning es2032 T261717 ', diff saved to https://phabricator.wikimedia.org/P12697 and previous config saved to /var/cache/conftool/dbconfig/20200922-045919-root.json

Mentioned in SAL (#wikimedia-operations) [2020-09-22T04:59:28Z] <marostegui@cumin1001> dbctl commit (dc=all): 'es2016 (re)pooling @ 25%: Slowly repool after recloning es2032 T261717 ', diff saved to https://phabricator.wikimedia.org/P12698 and previous config saved to /var/cache/conftool/dbconfig/20200922-045928-root.json

Mentioned in SAL (#wikimedia-operations) [2020-09-22T04:59:45Z] <marostegui@cumin1001> dbctl commit (dc=all): 'es2019 (re)pooling @ 25%: Slowly repool after recloning es2034 T261717 ', diff saved to https://phabricator.wikimedia.org/P12699 and previous config saved to /var/cache/conftool/dbconfig/20200922-045944-root.json

Mentioned in SAL (#wikimedia-operations) [2020-09-22T05:00:04Z] <marostegui> Add es2032 es2033 and es2034 to tendril and zarcillo T261717

Mentioned in SAL (#wikimedia-operations) [2020-09-22T05:14:23Z] <marostegui@cumin1001> dbctl commit (dc=all): 'es2013 (re)pooling @ 50%: Slowly repool after recloning es2032 T261717 ', diff saved to https://phabricator.wikimedia.org/P12700 and previous config saved to /var/cache/conftool/dbconfig/20200922-051423-root.json

Mentioned in SAL (#wikimedia-operations) [2020-09-22T05:14:32Z] <marostegui@cumin1001> dbctl commit (dc=all): 'es2016 (re)pooling @ 50%: Slowly repool after recloning es2032 T261717 ', diff saved to https://phabricator.wikimedia.org/P12701 and previous config saved to /var/cache/conftool/dbconfig/20200922-051431-root.json

Mentioned in SAL (#wikimedia-operations) [2020-09-22T05:14:48Z] <marostegui@cumin1001> dbctl commit (dc=all): 'es2019 (re)pooling @ 50%: Slowly repool after recloning es2034 T261717 ', diff saved to https://phabricator.wikimedia.org/P12702 and previous config saved to /var/cache/conftool/dbconfig/20200922-051448-root.json

Change 629011 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] es2032,es2033,es2034: Enable notifications

https://gerrit.wikimedia.org/r/629011

Change 629011 merged by Marostegui:
[operations/puppet@production] es2032,es2033,es2034: Enable notifications

https://gerrit.wikimedia.org/r/629011

Change 629012 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] instances.yaml: Add es203{2,3,4} to dbctl

https://gerrit.wikimedia.org/r/629012

Change 629012 merged by Marostegui:
[operations/puppet@production] instances.yaml: Add es203{2,3,4} to dbctl

https://gerrit.wikimedia.org/r/629012

Mentioned in SAL (#wikimedia-operations) [2020-09-22T05:29:27Z] <marostegui@cumin1001> dbctl commit (dc=all): 'es2013 (re)pooling @ 75%: Slowly repool after recloning es2032 T261717 ', diff saved to https://phabricator.wikimedia.org/P12703 and previous config saved to /var/cache/conftool/dbconfig/20200922-052926-root.json

Mentioned in SAL (#wikimedia-operations) [2020-09-22T05:29:35Z] <marostegui@cumin1001> dbctl commit (dc=all): 'es2016 (re)pooling @ 75%: Slowly repool after recloning es2032 T261717 ', diff saved to https://phabricator.wikimedia.org/P12704 and previous config saved to /var/cache/conftool/dbconfig/20200922-052935-root.json

Mentioned in SAL (#wikimedia-operations) [2020-09-22T05:29:51Z] <marostegui@cumin1001> dbctl commit (dc=all): 'es2019 (re)pooling @ 75%: Slowly repool after recloning es2034 T261717 ', diff saved to https://phabricator.wikimedia.org/P12705 and previous config saved to /var/cache/conftool/dbconfig/20200922-052951-root.json

Mentioned in SAL (#wikimedia-operations) [2020-09-22T05:33:47Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Add es2032, es2033 and es2034 into dbctl T261717', diff saved to https://phabricator.wikimedia.org/P12706 and previous config saved to /var/cache/conftool/dbconfig/20200922-053346-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2020-09-22T05:44:30Z] <marostegui@cumin1001> dbctl commit (dc=all): 'es2013 (re)pooling @ 100%: Slowly repool after recloning es2032 T261717 ', diff saved to https://phabricator.wikimedia.org/P12707 and previous config saved to /var/cache/conftool/dbconfig/20200922-054430-root.json

Mentioned in SAL (#wikimedia-operations) [2020-09-22T05:44:38Z] <marostegui@cumin1001> dbctl commit (dc=all): 'es2016 (re)pooling @ 100%: Slowly repool after recloning es2032 T261717 ', diff saved to https://phabricator.wikimedia.org/P12708 and previous config saved to /var/cache/conftool/dbconfig/20200922-054438-root.json

Mentioned in SAL (#wikimedia-operations) [2020-09-22T05:44:55Z] <marostegui@cumin1001> dbctl commit (dc=all): 'es2019 (re)pooling @ 100%: Slowly repool after recloning es2034 T261717 ', diff saved to https://phabricator.wikimedia.org/P12709 and previous config saved to /var/cache/conftool/dbconfig/20200922-054455-root.json

Mentioned in SAL (#wikimedia-operations) [2020-09-22T08:11:54Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Pool es2032, es2033 and es2034 for the first time with minimal weight T261717', diff saved to https://phabricator.wikimedia.org/P12713 and previous config saved to /var/cache/conftool/dbconfig/20200922-081154-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2020-09-22T09:12:55Z] <marostegui@cumin1001> dbctl commit (dc=all): 'es2032 (re)pooling @ 10%: Slowly es2032 T261717 ', diff saved to https://phabricator.wikimedia.org/P12719 and previous config saved to /var/cache/conftool/dbconfig/20200922-091255-root.json

Mentioned in SAL (#wikimedia-operations) [2020-09-22T09:13:11Z] <marostegui@cumin1001> dbctl commit (dc=all): 'es2033 (re)pooling @ 10%: Slowly repool es2033 T261717 ', diff saved to https://phabricator.wikimedia.org/P12720 and previous config saved to /var/cache/conftool/dbconfig/20200922-091310-root.json

Mentioned in SAL (#wikimedia-operations) [2020-09-22T09:13:29Z] <marostegui@cumin1001> dbctl commit (dc=all): 'es2034 (re)pooling @ 10%: Slowly repool es2034 T261717 ', diff saved to https://phabricator.wikimedia.org/P12721 and previous config saved to /var/cache/conftool/dbconfig/20200922-091329-root.json

Mentioned in SAL (#wikimedia-operations) [2020-09-22T09:27:58Z] <marostegui@cumin1001> dbctl commit (dc=all): 'es2032 (re)pooling @ 25%: Slowly es2032 T261717 ', diff saved to https://phabricator.wikimedia.org/P12722 and previous config saved to /var/cache/conftool/dbconfig/20200922-092758-root.json

Mentioned in SAL (#wikimedia-operations) [2020-09-22T09:28:14Z] <marostegui@cumin1001> dbctl commit (dc=all): 'es2033 (re)pooling @ 25%: Slowly repool es2033 T261717 ', diff saved to https://phabricator.wikimedia.org/P12723 and previous config saved to /var/cache/conftool/dbconfig/20200922-092814-root.json

Mentioned in SAL (#wikimedia-operations) [2020-09-22T09:28:33Z] <marostegui@cumin1001> dbctl commit (dc=all): 'es2034 (re)pooling @ 25%: Slowly repool es2034 T261717 ', diff saved to https://phabricator.wikimedia.org/P12724 and previous config saved to /var/cache/conftool/dbconfig/20200922-092832-root.json

Mentioned in SAL (#wikimedia-operations) [2020-09-22T09:43:02Z] <marostegui@cumin1001> dbctl commit (dc=all): 'es2032 (re)pooling @ 50%: Slowly es2032 T261717 ', diff saved to https://phabricator.wikimedia.org/P12725 and previous config saved to /var/cache/conftool/dbconfig/20200922-094302-root.json

Mentioned in SAL (#wikimedia-operations) [2020-09-22T09:43:18Z] <marostegui@cumin1001> dbctl commit (dc=all): 'es2033 (re)pooling @ 50%: Slowly repool es2033 T261717 ', diff saved to https://phabricator.wikimedia.org/P12726 and previous config saved to /var/cache/conftool/dbconfig/20200922-094317-root.json

Mentioned in SAL (#wikimedia-operations) [2020-09-22T09:43:36Z] <marostegui@cumin1001> dbctl commit (dc=all): 'es2034 (re)pooling @ 50%: Slowly repool es2034 T261717 ', diff saved to https://phabricator.wikimedia.org/P12727 and previous config saved to /var/cache/conftool/dbconfig/20200922-094336-root.json

Mentioned in SAL (#wikimedia-operations) [2020-09-22T09:58:05Z] <marostegui@cumin1001> dbctl commit (dc=all): 'es2032 (re)pooling @ 75%: Slowly es2032 T261717 ', diff saved to https://phabricator.wikimedia.org/P12728 and previous config saved to /var/cache/conftool/dbconfig/20200922-095805-root.json

Mentioned in SAL (#wikimedia-operations) [2020-09-22T09:58:21Z] <marostegui@cumin1001> dbctl commit (dc=all): 'es2033 (re)pooling @ 75%: Slowly repool es2033 T261717 ', diff saved to https://phabricator.wikimedia.org/P12729 and previous config saved to /var/cache/conftool/dbconfig/20200922-095821-root.json

Mentioned in SAL (#wikimedia-operations) [2020-09-22T09:58:39Z] <marostegui@cumin1001> dbctl commit (dc=all): 'es2034 (re)pooling @ 75%: Slowly repool es2034 T261717 ', diff saved to https://phabricator.wikimedia.org/P12730 and previous config saved to /var/cache/conftool/dbconfig/20200922-095839-root.json

Mentioned in SAL (#wikimedia-operations) [2020-09-22T10:13:09Z] <marostegui@cumin1001> dbctl commit (dc=all): 'es2032 (re)pooling @ 100%: Slowly es2032 T261717 ', diff saved to https://phabricator.wikimedia.org/P12731 and previous config saved to /var/cache/conftool/dbconfig/20200922-101308-root.json

Mentioned in SAL (#wikimedia-operations) [2020-09-22T10:13:24Z] <marostegui@cumin1001> dbctl commit (dc=all): 'es2033 (re)pooling @ 100%: Slowly repool es2033 T261717 ', diff saved to https://phabricator.wikimedia.org/P12732 and previous config saved to /var/cache/conftool/dbconfig/20200922-101324-root.json

Mentioned in SAL (#wikimedia-operations) [2020-09-22T10:13:43Z] <marostegui@cumin1001> dbctl commit (dc=all): 'es2034 (re)pooling @ 100%: Slowly repool es2034 T261717 ', diff saved to https://phabricator.wikimedia.org/P12733 and previous config saved to /var/cache/conftool/dbconfig/20200922-101342-root.json

es2032 has been pooled into es1
es2033 has been pooled into es2
es2034 has been pooled into es3

Next steps will be to depool the old ones, make any of these master (this is a noop) and start decommissioning the old ones.

Change 629529 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/dns@master] dns: Remove es2014 from DNS

https://gerrit.wikimedia.org/r/629529

Change 629529 merged by Marostegui:
[operations/dns@master] dns: Remove es2014 from DNS

https://gerrit.wikimedia.org/r/629529