Page MenuHomePhabricator
Create Task
Maniphest T243052

Productionize es1020-es1025, es2020-es2025
Closed, ResolvedPublic
Actions

Description

Productionize the new external store servers that were purchased at {T235820} and {T235659}

codfw:

  • es2020 es4 master A3
  • es2021 es4 slave B3
  • es2022 es4 slave C6
  • es2023 es5 master D3
  • es2024 es5 slave A6
  • es2025 es5 slave B8

eqiad:

  • es1020 A3 es4 master
  • es1021 B3 es4 slave
  • es1022 C5 es4 slave
  • es1023 D6 es5 master
  • es1024 A5 es5 slave
  • es1025 B5 es5 slave

Details

Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
gerritbot added a comment.Jan 17 2020, 8:31 AM

Change 565510 merged by Marostegui:
[operations/puppet@production] es2020: Set role master for es4

https://gerrit.wikimedia.org/r/565510

Maintenance_bot removed a project: Patch-For-Review.Jan 17 2020, 9:10 AM
• Marostegui claimed this task.Jan 17 2020, 1:48 PM
gerritbot added a comment.Jan 20 2020, 7:03 AM

Change 565816 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Move es2021 from spare to es4 slave

https://gerrit.wikimedia.org/r/565816

gerritbot added a project: Patch-For-Review.Jan 20 2020, 7:03 AM
gerritbot added a comment.Jan 20 2020, 7:08 AM

Change 565816 merged by Marostegui:
[operations/puppet@production] mariadb: Move es2021 from spare to es4 slave

https://gerrit.wikimedia.org/r/565816

Stashbot added a comment.Jan 20 2020, 7:10 AM

Mentioned in SAL (#wikimedia-operations) [2020-01-20T07:10:09Z] <marostegui> Stop MySQL on es2020 to clone es2021 - T243052

Maintenance_bot removed a project: Patch-For-Review.Jan 20 2020, 7:10 AM
gerritbot added a comment.Jan 20 2020, 7:24 AM

Change 565830 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] install_server: Do not reimage es2020, es2021

https://gerrit.wikimedia.org/r/565830

gerritbot added a project: Patch-For-Review.Jan 20 2020, 7:24 AM

Change 565830 merged by Marostegui:
[operations/puppet@production] install_server: Do not reimage es2020, es2021

https://gerrit.wikimedia.org/r/565830

Maintenance_bot removed a project: Patch-For-Review.Jan 20 2020, 8:10 AM
gerritbot added a comment.Jan 21 2020, 5:57 AM

Change 566129 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Move es2022 from spare to es4 slave

https://gerrit.wikimedia.org/r/566129

gerritbot added a project: Patch-For-Review.Jan 21 2020, 5:57 AM
Stashbot added a comment.Jan 21 2020, 5:58 AM

Mentioned in SAL (#wikimedia-operations) [2020-01-21T05:58:29Z] <marostegui> Stop MySQL on es2021 to clone es2022 - T243052

gerritbot added a comment.Jan 21 2020, 5:58 AM

Change 566129 merged by Marostegui:
[operations/puppet@production] mariadb: Move es2022 from spare to es4 slave

https://gerrit.wikimedia.org/r/566129

Maintenance_bot removed a project: Patch-For-Review.Jan 21 2020, 6:10 AM
gerritbot added a comment.Jan 21 2020, 6:21 AM

Change 566131 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] install_server: Do not reimage es2022

https://gerrit.wikimedia.org/r/566131

gerritbot added a project: Patch-For-Review.Jan 21 2020, 6:21 AM

Change 566131 merged by Marostegui:
[operations/puppet@production] install_server: Do not reimage es2022

https://gerrit.wikimedia.org/r/566131

Maintenance_bot removed a project: Patch-For-Review.Jan 21 2020, 7:10 AM
gerritbot added a comment.Jan 22 2020, 8:23 AM

Change 566465 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Productionize es2023 as es5 codfw master

https://gerrit.wikimedia.org/r/566465

gerritbot added a project: Patch-For-Review.Jan 22 2020, 8:23 AM

Change 566465 merged by Marostegui:
[operations/puppet@production] mariadb: Productionize es2023 as es5 codfw master

https://gerrit.wikimedia.org/r/566465

Stashbot added a comment.Jan 22 2020, 8:27 AM

Mentioned in SAL (#wikimedia-operations) [2020-01-22T08:27:56Z] <marostegui> Stop MySQL on es2021 to "clone" es2023 - T243052

Maintenance_bot removed a project: Patch-For-Review.Jan 22 2020, 9:10 AM
gerritbot added a comment.Jan 22 2020, 9:51 AM

Change 566489 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Productionize es2024 as es5 codfw slave

https://gerrit.wikimedia.org/r/566489

gerritbot added a project: Patch-For-Review.Jan 22 2020, 9:51 AM
gerritbot added a comment.Jan 22 2020, 10:08 AM

Change 566489 merged by Marostegui:
[operations/puppet@production] mariadb: Productionize es2024 as es5 codfw slave

https://gerrit.wikimedia.org/r/566489

Stashbot added a comment.Jan 22 2020, 10:09 AM

Mentioned in SAL (#wikimedia-operations) [2020-01-22T10:09:11Z] <marostegui> Stop MySQL on es2023 to "clone" es2024 - T243052

Maintenance_bot removed a project: Patch-For-Review.Jan 22 2020, 10:10 AM
gerritbot added a comment.Jan 22 2020, 2:14 PM

Change 566528 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/software@master] check_depooled: Add es4 and es5

https://gerrit.wikimedia.org/r/566528

gerritbot added a project: Patch-For-Review.Jan 22 2020, 2:14 PM

Change 566528 merged by jenkins-bot:
[operations/software@master] check_depooled: Add es4 and es5

https://gerrit.wikimedia.org/r/566528

Maintenance_bot removed a project: Patch-For-Review.Jan 22 2020, 3:10 PM
gerritbot added a comment.Jan 23 2020, 8:44 AM

Change 566688 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Productionize es2025 as es5 codfw slave

https://gerrit.wikimedia.org/r/566688

gerritbot added a project: Patch-For-Review.Jan 23 2020, 8:44 AM

Change 566688 merged by Marostegui:
[operations/puppet@production] mariadb: Productionize es2025 as es5 codfw slave

https://gerrit.wikimedia.org/r/566688

Stashbot added a comment.Jan 23 2020, 8:46 AM

Mentioned in SAL (#wikimedia-operations) [2020-01-23T08:45:56Z] <marostegui> Stop mysql on es2024 to "clone" es2025 - T243052

Maintenance_bot removed a project: Patch-For-Review.Jan 23 2020, 9:10 AM
• Marostegui added a comment.Jan 23 2020, 1:32 PM

Grants are a bit of mess, I have cleaned them up a bit and left this on codfw hosts (and will do the same with eqiad ones):

root@es2020.codfw.wmnet[(none)]> select user,host from mysql.user where user like 'wiki%';
+-----------+-----------+
| user      | host      |
+-----------+-----------+
| wikiadmin | 10.%      |
| wikiuser  | 10.192.%  |
| wikiuser  | 10.64.%   |
| wikiadmin | localhost |
| wikiuser  | localhost |
+-----------+-----------+
5 rows in set (0.04 sec)
• Marostegui added a comment.Jan 23 2020, 1:33 PM

I think we should also remove 'wikiadmin'@'10.% and change it by 10.64.% and 10.192.%

• Marostegui added a comment.Jan 23 2020, 1:34 PM

And probably the same for root user, 10.64 and 10.192.%

gerritbot added a comment.Jan 24 2020, 6:39 AM

Change 566930 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] install_server: Do not reimage es4/es5 codfw hosts

https://gerrit.wikimedia.org/r/566930

gerritbot added a project: Patch-For-Review.Jan 24 2020, 6:39 AM
gerritbot added a comment.Jan 24 2020, 6:43 AM

Change 566930 merged by Marostegui:
[operations/puppet@production] install_server: Do not reimage es4/es5 codfw hosts

https://gerrit.wikimedia.org/r/566930

Maintenance_bot removed a project: Patch-For-Review.Jan 24 2020, 7:10 AM
gerritbot added a comment.Feb 11 2020, 7:29 AM

Change 571460 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] es202[0-5]: Enable notifications

https://gerrit.wikimedia.org/r/571460

gerritbot added a project: Patch-For-Review.Feb 11 2020, 7:29 AM
gerritbot added a comment.Feb 11 2020, 7:38 AM

Change 571460 merged by Marostegui:
[operations/puppet@production] es202[0-5]: Enable notifications

https://gerrit.wikimedia.org/r/571460

• Marostegui updated the task description. (Show Details)Feb 11 2020, 7:39 AM
Maintenance_bot removed a project: Patch-For-Review.Feb 11 2020, 8:11 AM
• Marostegui updated the task description. (Show Details)Feb 13 2020, 7:27 AM
• Marostegui changed the task status from Open to Stalled.Feb 18 2020, 7:48 AM
• Marostegui closed subtask T243512: Clean up wikiadmin2 user from core hosts as Resolved.Feb 19 2020, 8:42 AM
• Marostegui changed the task status from Stalled to Open.Feb 19 2020, 2:41 PM
• Marostegui closed subtask T241359: (Needed by 31st January) eqiad: rack/setup/install es102[0-5].eqiad.wmnet as Resolved.
• Marostegui updated the task description. (Show Details)Feb 19 2020, 2:52 PM
gerritbot added a comment.Feb 19 2020, 2:56 PM

Change 573303 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Productionize es1020

https://gerrit.wikimedia.org/r/573303

gerritbot added a project: Patch-For-Review.Feb 19 2020, 2:56 PM

Change 573303 merged by Marostegui:
[operations/puppet@production] mariadb: Productionize es1020

https://gerrit.wikimedia.org/r/573303

Stashbot added a comment.Feb 19 2020, 2:59 PM

Mentioned in SAL (#wikimedia-operations) [2020-02-19T14:59:53Z] <marostegui> Stop mysql on es2021 - T243052

Maintenance_bot removed a project: Patch-For-Review.Feb 19 2020, 3:11 PM
gerritbot added a comment.Feb 19 2020, 3:22 PM

Change 573308 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] install_server: Do not reimage es1020

https://gerrit.wikimedia.org/r/573308

gerritbot added a project: Patch-For-Review.Feb 19 2020, 3:22 PM

Change 573308 merged by Marostegui:
[operations/puppet@production] install_server: Do not reimage es1020

https://gerrit.wikimedia.org/r/573308

Maintenance_bot removed a project: Patch-For-Review.Feb 19 2020, 4:11 PM
gerritbot added a comment.Feb 20 2020, 6:31 AM

Change 573441 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Productionize es1021

https://gerrit.wikimedia.org/r/573441

gerritbot added a project: Patch-For-Review.Feb 20 2020, 6:31 AM

Change 573441 merged by Marostegui:
[operations/puppet@production] mariadb: Productionize es1021

https://gerrit.wikimedia.org/r/573441

gerritbot added a comment.Feb 20 2020, 6:53 AM

Change 573443 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Productionize es1022

https://gerrit.wikimedia.org/r/573443

gerritbot added a comment.Feb 20 2020, 6:55 AM

Change 573443 merged by Marostegui:
[operations/puppet@production] mariadb: Productionize es1022

https://gerrit.wikimedia.org/r/573443

Maintenance_bot removed a project: Patch-For-Review.Feb 20 2020, 7:10 AM
gerritbot added a comment.Feb 20 2020, 7:13 AM

Change 573447 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Productionize es1023

https://gerrit.wikimedia.org/r/573447

gerritbot added a project: Patch-For-Review.Feb 20 2020, 7:13 AM

Change 573447 merged by Marostegui:
[operations/puppet@production] mariadb: Productionize es1023

https://gerrit.wikimedia.org/r/573447

gerritbot added a comment.Feb 20 2020, 7:23 AM

Change 573449 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] es10123: Make it es5 master

https://gerrit.wikimedia.org/r/573449

gerritbot added a comment.Feb 20 2020, 7:25 AM

Change 573449 merged by Marostegui:
[operations/puppet@production] es10123: Make it es5 master

https://gerrit.wikimedia.org/r/573449

Maintenance_bot removed a project: Patch-For-Review.Feb 20 2020, 8:10 AM
gerritbot added a comment.Feb 20 2020, 8:19 AM

Change 573461 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Productionize es1024

https://gerrit.wikimedia.org/r/573461

gerritbot added a project: Patch-For-Review.Feb 20 2020, 8:19 AM

Change 573461 merged by Marostegui:
[operations/puppet@production] mariadb: Productionize es1024

https://gerrit.wikimedia.org/r/573461

• Marostegui added a comment.EditedFeb 20 2020, 8:33 AM

Grants in es4:

root@cumin1001:/home/marostegui/git/tendril/bin# /home/marostegui/section es4 | while read host port; do echo "====$host====" ; mysql.py -h$host:$port -e "select user,host from mysql.user where user like 'wik%'";done
====es2022.codfw.wmnet====
user	host
wikiadmin	10.192.%
wikiuser	10.192.%
wikiadmin	10.64.%
wikiuser	10.64.%
wikiadmin	localhost
wikiuser	localhost
====es2021.codfw.wmnet====
user	host
wikiadmin	10.192.%
wikiuser	10.192.%
wikiadmin	10.64.%
wikiuser	10.64.%
wikiadmin	localhost
wikiuser	localhost
====es2020.codfw.wmnet====
user	host
wikiadmin	10.192.%
wikiuser	10.192.%
wikiadmin	10.64.%
wikiuser	10.64.%
wikiadmin	localhost
wikiuser	localhost
====es1022.eqiad.wmnet====
user	host
wikiadmin	10.192.%
wikiuser	10.192.%
wikiadmin	10.64.%
wikiuser	10.64.%
wikiadmin	localhost
wikiuser	localhost
====es1021.eqiad.wmnet====
user	host
wikiadmin	10.192.%
wikiuser	10.192.%
wikiadmin	10.64.%
wikiuser	10.64.%
wikiadmin	localhost
wikiuser	localhost
====es1020.eqiad.wmnet====
user	host
wikiadmin	10.192.%
wikiuser	10.192.%
wikiadmin	10.64.%
wikiuser	10.64.%
wikiadmin	localhost
wikiuser	localhost

They look fine T245720#5900563

Maintenance_bot removed a project: Patch-For-Review.Feb 20 2020, 9:10 AM
Papaul closed subtask T245714: es2022 power supply issues as Resolved.Feb 20 2020, 3:55 PM
gerritbot added a comment.Feb 21 2020, 6:32 AM

Change 573860 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Productionize es1025

https://gerrit.wikimedia.org/r/573860

gerritbot added a project: Patch-For-Review.Feb 21 2020, 6:32 AM
Stashbot added a comment.Feb 21 2020, 6:34 AM

Mentioned in SAL (#wikimedia-operations) [2020-02-21T06:34:54Z] <marostegui> Stop mysql on es1024 to clone es1025 - T243052

gerritbot added a comment.Feb 21 2020, 6:35 AM

Change 573860 merged by Marostegui:
[operations/puppet@production] mariadb: Productionize es1025

https://gerrit.wikimedia.org/r/573860

Maintenance_bot removed a project: Patch-For-Review.Feb 21 2020, 7:10 AM
• Marostegui closed subtask T245806: Create es4 and es5 sections in dbctl as Resolved.Feb 24 2020, 3:00 PM
gerritbot added a comment.Feb 25 2020, 5:54 AM

Change 574641 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] es102[0-5]: Enable notifications

https://gerrit.wikimedia.org/r/574641

gerritbot added a project: Patch-For-Review.Feb 25 2020, 5:54 AM

Change 574641 merged by Marostegui:
[operations/puppet@production] es102[0-5]: Enable notifications

https://gerrit.wikimedia.org/r/574641

• Marostegui closed this task as Resolved.Feb 25 2020, 5:56 AM
• Marostegui updated the task description. (Show Details)

All hosts ready

• Marostegui mentioned this in T246072: Enable es4 and es5 as writable new external store sections and set es2 and es3 as read only.Feb 25 2020, 8:50 AM
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL