Page MenuHomePhabricator

Productionize es1020-es1025, es2020-es2025
Closed, ResolvedPublic

Description

Productionize the new external store servers that were purchased at {T235820} and {T235659}

codfw:

  • es2020 es4 master A3
  • es2021 es4 slave B3
  • es2022 es4 slave C6
  • es2023 es5 master D3
  • es2024 es5 slave A6
  • es2025 es5 slave B8

eqiad:

  • es1020 A3 es4 master
  • es1021 B3 es4 slave
  • es1022 C5 es4 slave
  • es1023 D6 es5 master
  • es1024 A5 es5 slave
  • es1025 B5 es5 slave

Details

Show related patches Customize query in gerrit

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Change 565510 merged by Marostegui:
[operations/puppet@production] es2020: Set role master for es4

https://gerrit.wikimedia.org/r/565510

Change 565816 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Move es2021 from spare to es4 slave

https://gerrit.wikimedia.org/r/565816

Change 565816 merged by Marostegui:
[operations/puppet@production] mariadb: Move es2021 from spare to es4 slave

https://gerrit.wikimedia.org/r/565816

Mentioned in SAL (#wikimedia-operations) [2020-01-20T07:10:09Z] <marostegui> Stop MySQL on es2020 to clone es2021 - T243052

Change 565830 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] install_server: Do not reimage es2020, es2021

https://gerrit.wikimedia.org/r/565830

Change 565830 merged by Marostegui:
[operations/puppet@production] install_server: Do not reimage es2020, es2021

https://gerrit.wikimedia.org/r/565830

Change 566129 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Move es2022 from spare to es4 slave

https://gerrit.wikimedia.org/r/566129

Mentioned in SAL (#wikimedia-operations) [2020-01-21T05:58:29Z] <marostegui> Stop MySQL on es2021 to clone es2022 - T243052

Change 566129 merged by Marostegui:
[operations/puppet@production] mariadb: Move es2022 from spare to es4 slave

https://gerrit.wikimedia.org/r/566129

Change 566131 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] install_server: Do not reimage es2022

https://gerrit.wikimedia.org/r/566131

Change 566131 merged by Marostegui:
[operations/puppet@production] install_server: Do not reimage es2022

https://gerrit.wikimedia.org/r/566131

Change 566465 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Productionize es2023 as es5 codfw master

https://gerrit.wikimedia.org/r/566465

Change 566465 merged by Marostegui:
[operations/puppet@production] mariadb: Productionize es2023 as es5 codfw master

https://gerrit.wikimedia.org/r/566465

Mentioned in SAL (#wikimedia-operations) [2020-01-22T08:27:56Z] <marostegui> Stop MySQL on es2021 to "clone" es2023 - T243052

Change 566489 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Productionize es2024 as es5 codfw slave

https://gerrit.wikimedia.org/r/566489

Change 566489 merged by Marostegui:
[operations/puppet@production] mariadb: Productionize es2024 as es5 codfw slave

https://gerrit.wikimedia.org/r/566489

Mentioned in SAL (#wikimedia-operations) [2020-01-22T10:09:11Z] <marostegui> Stop MySQL on es2023 to "clone" es2024 - T243052

Change 566528 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/software@master] check_depooled: Add es4 and es5

https://gerrit.wikimedia.org/r/566528

Change 566528 merged by jenkins-bot:
[operations/software@master] check_depooled: Add es4 and es5

https://gerrit.wikimedia.org/r/566528

Change 566688 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Productionize es2025 as es5 codfw slave

https://gerrit.wikimedia.org/r/566688

Change 566688 merged by Marostegui:
[operations/puppet@production] mariadb: Productionize es2025 as es5 codfw slave

https://gerrit.wikimedia.org/r/566688

Mentioned in SAL (#wikimedia-operations) [2020-01-23T08:45:56Z] <marostegui> Stop mysql on es2024 to "clone" es2025 - T243052

Grants are a bit of mess, I have cleaned them up a bit and left this on codfw hosts (and will do the same with eqiad ones):

root@es2020.codfw.wmnet[(none)]> select user,host from mysql.user where user like 'wiki%';
+-----------+-----------+
| user      | host      |
+-----------+-----------+
| wikiadmin | 10.%      |
| wikiuser  | 10.192.%  |
| wikiuser  | 10.64.%   |
| wikiadmin | localhost |
| wikiuser  | localhost |
+-----------+-----------+
5 rows in set (0.04 sec)

I think we should also remove 'wikiadmin'@'10.% and change it by 10.64.% and 10.192.%

And probably the same for root user, 10.64 and 10.192.%

Change 566930 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] install_server: Do not reimage es4/es5 codfw hosts

https://gerrit.wikimedia.org/r/566930

Change 566930 merged by Marostegui:
[operations/puppet@production] install_server: Do not reimage es4/es5 codfw hosts

https://gerrit.wikimedia.org/r/566930

Change 571460 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] es202[0-5]: Enable notifications

https://gerrit.wikimedia.org/r/571460

Change 571460 merged by Marostegui:
[operations/puppet@production] es202[0-5]: Enable notifications

https://gerrit.wikimedia.org/r/571460

Marostegui updated the task description. (Show Details)Feb 11 2020, 7:39 AM
Marostegui updated the task description. (Show Details)Feb 13 2020, 7:27 AM
Marostegui changed the task status from Open to Stalled.Feb 18 2020, 7:48 AM
Marostegui updated the task description. (Show Details)Feb 19 2020, 2:52 PM

Change 573303 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Productionize es1020

https://gerrit.wikimedia.org/r/573303

Change 573303 merged by Marostegui:
[operations/puppet@production] mariadb: Productionize es1020

https://gerrit.wikimedia.org/r/573303

Mentioned in SAL (#wikimedia-operations) [2020-02-19T14:59:53Z] <marostegui> Stop mysql on es2021 - T243052

Change 573308 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] install_server: Do not reimage es1020

https://gerrit.wikimedia.org/r/573308

Change 573308 merged by Marostegui:
[operations/puppet@production] install_server: Do not reimage es1020

https://gerrit.wikimedia.org/r/573308

Change 573441 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Productionize es1021

https://gerrit.wikimedia.org/r/573441

Change 573441 merged by Marostegui:
[operations/puppet@production] mariadb: Productionize es1021

https://gerrit.wikimedia.org/r/573441

Change 573443 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Productionize es1022

https://gerrit.wikimedia.org/r/573443

Change 573443 merged by Marostegui:
[operations/puppet@production] mariadb: Productionize es1022

https://gerrit.wikimedia.org/r/573443

Change 573447 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Productionize es1023

https://gerrit.wikimedia.org/r/573447

Change 573447 merged by Marostegui:
[operations/puppet@production] mariadb: Productionize es1023

https://gerrit.wikimedia.org/r/573447

Change 573449 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] es10123: Make it es5 master

https://gerrit.wikimedia.org/r/573449

Change 573449 merged by Marostegui:
[operations/puppet@production] es10123: Make it es5 master

https://gerrit.wikimedia.org/r/573449

Change 573461 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Productionize es1024

https://gerrit.wikimedia.org/r/573461

Change 573461 merged by Marostegui:
[operations/puppet@production] mariadb: Productionize es1024

https://gerrit.wikimedia.org/r/573461

Marostegui added a comment.EditedFeb 20 2020, 8:33 AM

Grants in es4:

root@cumin1001:/home/marostegui/git/tendril/bin# /home/marostegui/section es4 | while read host port; do echo "====$host====" ; mysql.py -h$host:$port -e "select user,host from mysql.user where user like 'wik%'";done
====es2022.codfw.wmnet====
user	host
wikiadmin	10.192.%
wikiuser	10.192.%
wikiadmin	10.64.%
wikiuser	10.64.%
wikiadmin	localhost
wikiuser	localhost
====es2021.codfw.wmnet====
user	host
wikiadmin	10.192.%
wikiuser	10.192.%
wikiadmin	10.64.%
wikiuser	10.64.%
wikiadmin	localhost
wikiuser	localhost
====es2020.codfw.wmnet====
user	host
wikiadmin	10.192.%
wikiuser	10.192.%
wikiadmin	10.64.%
wikiuser	10.64.%
wikiadmin	localhost
wikiuser	localhost
====es1022.eqiad.wmnet====
user	host
wikiadmin	10.192.%
wikiuser	10.192.%
wikiadmin	10.64.%
wikiuser	10.64.%
wikiadmin	localhost
wikiuser	localhost
====es1021.eqiad.wmnet====
user	host
wikiadmin	10.192.%
wikiuser	10.192.%
wikiadmin	10.64.%
wikiuser	10.64.%
wikiadmin	localhost
wikiuser	localhost
====es1020.eqiad.wmnet====
user	host
wikiadmin	10.192.%
wikiuser	10.192.%
wikiadmin	10.64.%
wikiuser	10.64.%
wikiadmin	localhost
wikiuser	localhost

They look fine T245720#5900563

Change 573860 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Productionize es1025

https://gerrit.wikimedia.org/r/573860

Mentioned in SAL (#wikimedia-operations) [2020-02-21T06:34:54Z] <marostegui> Stop mysql on es1024 to clone es1025 - T243052

Change 573860 merged by Marostegui:
[operations/puppet@production] mariadb: Productionize es1025

https://gerrit.wikimedia.org/r/573860

Change 574641 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] es102[0-5]: Enable notifications

https://gerrit.wikimedia.org/r/574641

Change 574641 merged by Marostegui:
[operations/puppet@production] es102[0-5]: Enable notifications

https://gerrit.wikimedia.org/r/574641

Marostegui closed this task as Resolved.Feb 25 2020, 5:56 AM
Marostegui updated the task description. (Show Details)

All hosts ready