Page MenuHomePhabricator
Create Task
Maniphest T243052

Productionize es1020-es1025, es2020-es2025
Closed, ResolvedPublic
Actions

Description

Productionize the new external store servers that were purchased at {T235820} and {T235659}

codfw:

  • es2020 es4 master A3
  • es2021 es4 slave B3
  • es2022 es4 slave C6
  • es2023 es5 master D3
  • es2024 es5 slave A6
  • es2025 es5 slave B8

eqiad:

  • es1020 A3 es4 master
  • es1021 B3 es4 slave
  • es1022 C5 es4 slave
  • es1023 D6 es5 master
  • es1024 A5 es5 slave
  • es1025 B5 es5 slave

Details

Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
gerritbot added a comment.Jan 17 2020, 8:31 AM

Change 565510 merged by Marostegui:
[operations/puppet@production] es2020: Set role master for es4

https://gerrit.wikimedia.org/r/565510

Maintenance_bot removed a project: Patch-For-Review.Jan 17 2020, 9:10 AM
• Marostegui claimed this task.Jan 17 2020, 1:48 PM
gerritbot added a comment.Jan 20 2020, 7:03 AM

Change 565816 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Move es2021 from spare to es4 slave

https://gerrit.wikimedia.org/r/565816

gerritbot added a project: Patch-For-Review.Jan 20 2020, 7:03 AM
gerritbot added a comment.Jan 20 2020, 7:08 AM

Change 565816 merged by Marostegui:
[operations/puppet@production] mariadb: Move es2021 from spare to es4 slave

https://gerrit.wikimedia.org/r/565816

Stashbot added a comment.Jan 20 2020, 7:10 AM

Mentioned in SAL (#wikimedia-operations) [2020-01-20T07:10:09Z] <marostegui> Stop MySQL on es2020 to clone es2021 - T243052

Maintenance_bot removed a project: Patch-For-Review.Jan 20 2020, 7:10 AM
gerritbot added a comment.Jan 20 2020, 7:24 AM

Change 565830 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] install_server: Do not reimage es2020, es2021

https://gerrit.wikimedia.org/r/565830

gerritbot added a project: Patch-For-Review.Jan 20 2020, 7:24 AM

Change 565830 merged by Marostegui:
[operations/puppet@production] install_server: Do not reimage es2020, es2021

https://gerrit.wikimedia.org/r/565830

Maintenance_bot removed a project: Patch-For-Review.Jan 20 2020, 8:10 AM
gerritbot added a comment.Jan 21 2020, 5:57 AM

Change 566129 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Move es2022 from spare to es4 slave

https://gerrit.wikimedia.org/r/566129

gerritbot added a project: Patch-For-Review.Jan 21 2020, 5:57 AM
Stashbot added a comment.Jan 21 2020, 5:58 AM

Mentioned in SAL (#wikimedia-operations) [2020-01-21T05:58:29Z] <marostegui> Stop MySQL on es2021 to clone es2022 - T243052

gerritbot added a comment.Jan 21 2020, 5:58 AM

Change 566129 merged by Marostegui:
[operations/puppet@production] mariadb: Move es2022 from spare to es4 slave

https://gerrit.wikimedia.org/r/566129

Maintenance_bot removed a project: Patch-For-Review.Jan 21 2020, 6:10 AM
gerritbot added a comment.Jan 21 2020, 6:21 AM

Change 566131 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] install_server: Do not reimage es2022

https://gerrit.wikimedia.org/r/566131

gerritbot added a project: Patch-For-Review.Jan 21 2020, 6:21 AM

Change 566131 merged by Marostegui:
[operations/puppet@production] install_server: Do not reimage es2022

https://gerrit.wikimedia.org/r/566131

Maintenance_bot removed a project: Patch-For-Review.Jan 21 2020, 7:10 AM
gerritbot added a comment.Jan 22 2020, 8:23 AM

Change 566465 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Productionize es2023 as es5 codfw master

https://gerrit.wikimedia.org/r/566465

gerritbot added a project: Patch-For-Review.Jan 22 2020, 8:23 AM

Change 566465 merged by Marostegui:
[operations/puppet@production] mariadb: Productionize es2023 as es5 codfw master

https://gerrit.wikimedia.org/r/566465

Stashbot added a comment.Jan 22 2020, 8:27 AM

Mentioned in SAL (#wikimedia-operations) [2020-01-22T08:27:56Z] <marostegui> Stop MySQL on es2021 to "clone" es2023 - T243052

Maintenance_bot removed a project: Patch-For-Review.Jan 22 2020, 9:10 AM
gerritbot added a comment.Jan 22 2020, 9:51 AM

Change 566489 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Productionize es2024 as es5 codfw slave

https://gerrit.wikimedia.org/r/566489

gerritbot added a project: Patch-For-Review.Jan 22 2020, 9:51 AM
gerritbot added a comment.Jan 22 2020, 10:08 AM

Change 566489 merged by Marostegui:
[operations/puppet@production] mariadb: Productionize es2024 as es5 codfw slave

https://gerrit.wikimedia.org/r/566489

Stashbot added a comment.Jan 22 2020, 10:09 AM

Mentioned in SAL (#wikimedia-operations) [2020-01-22T10:09:11Z] <marostegui> Stop MySQL on es2023 to "clone" es2024 - T243052

Maintenance_bot removed a project: Patch-For-Review.Jan 22 2020, 10:10 AM
gerritbot added a comment.Jan 22 2020, 2:14 PM

Change 566528 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/software@master] check_depooled: Add es4 and es5

https://gerrit.wikimedia.org/r/566528

gerritbot added a project: Patch-For-Review.Jan 22 2020, 2:14 PM

Change 566528 merged by jenkins-bot:
[operations/software@master] check_depooled: Add es4 and es5

https://gerrit.wikimedia.org/r/566528

Maintenance_bot removed a project: Patch-For-Review.Jan 22 2020, 3:10 PM
gerritbot added a comment.Jan 23 2020, 8:44 AM

Change 566688 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Productionize es2025 as es5 codfw slave

https://gerrit.wikimedia.org/r/566688

gerritbot added a project: Patch-For-Review.Jan 23 2020, 8:44 AM

Change 566688 merged by Marostegui:
[operations/puppet@production] mariadb: Productionize es2025 as es5 codfw slave

https://gerrit.wikimedia.org/r/566688

Stashbot added a comment.Jan 23 2020, 8:46 AM

Mentioned in SAL (#wikimedia-operations) [2020-01-23T08:45:56Z] <marostegui> Stop mysql on es2024 to "clone" es2025 - T243052

Maintenance_bot removed a project: Patch-For-Review.Jan 23 2020, 9:10 AM
• Marostegui added a comment.Jan 23 2020, 1:32 PM

Grants are a bit of mess, I have cleaned them up a bit and left this on codfw hosts (and will do the same with eqiad ones):

root@es2020.codfw.wmnet[(none)]> select user,host from mysql.user where user like 'wiki%';
+-----------+-----------+
| user      | host      |
+-----------+-----------+
| wikiadmin | 10.%      |
| wikiuser  | 10.192.%  |
| wikiuser  | 10.64.%   |
| wikiadmin | localhost |
| wikiuser  | localhost |
+-----------+-----------+
5 rows in set (0.04 sec)
• Marostegui added a comment.Jan 23 2020, 1:33 PM

I think we should also remove 'wikiadmin'@'10.% and change it by 10.64.% and 10.192.%

• Marostegui added a comment.Jan 23 2020, 1:34 PM

And probably the same for root user, 10.64 and 10.192.%

gerritbot added a comment.Jan 24 2020, 6:39 AM

Change 566930 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] install_server: Do not reimage es4/es5 codfw hosts

https://gerrit.wikimedia.org/r/566930

gerritbot added a project: Patch-For-Review.Jan 24 2020, 6:39 AM
gerritbot added a comment.Jan 24 2020, 6:43 AM

Change 566930 merged by Marostegui:
[operations/puppet@production] install_server: Do not reimage es4/es5 codfw hosts

https://gerrit.wikimedia.org/r/566930

Maintenance_bot removed a project: Patch-For-Review.Jan 24 2020, 7:10 AM
gerritbot added a comment.Feb 11 2020, 7:29 AM

Change 571460 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] es202[0-5]: Enable notifications

https://gerrit.wikimedia.org/r/571460

gerritbot added a project: Patch-For-Review.Feb 11 2020, 7:29 AM
gerritbot added a comment.Feb 11 2020, 7:38 AM

Change 571460 merged by Marostegui:
[operations/puppet@production] es202[0-5]: Enable notifications

https://gerrit.wikimedia.org/r/571460

• Marostegui updated the task description. (Show Details)Feb 11 2020, 7:39 AM
Maintenance_bot removed a project: Patch-For-Review.Feb 11 2020, 8:11 AM
• Marostegui updated the task description. (Show Details)Feb 13 2020, 7:27 AM
• Marostegui changed the task status from Open to Stalled.Feb 18 2020, 7:48 AM
• Marostegui closed subtask T243512: Clean up wikiadmin2 user from core hosts as Resolved.Feb 19 2020, 8:42 AM
• Marostegui changed the task status from Stalled to Open.Feb 19 2020, 2:41 PM
• Marostegui closed subtask T241359: (Needed by 31st January) eqiad: rack/setup/install es102[0-5].eqiad.wmnet as Resolved.
• Marostegui updated the task description. (Show Details)Feb 19 2020, 2:52 PM
gerritbot added a comment.Feb 19 2020, 2:56 PM

Change 573303 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Productionize es1020

https://gerrit.wikimedia.org/r/573303

gerritbot added a project: Patch-For-Review.Feb 19 2020, 2:56 PM

Change 573303 merged by Marostegui:
[operations/puppet@production] mariadb: Productionize es1020

https://gerrit.wikimedia.org/r/573303

Stashbot added a comment.Feb 19 2020, 2:59 PM

Mentioned in SAL (#wikimedia-operations) [2020-02-19T14:59:53Z] <marostegui> Stop mysql on es2021 - T243052

Maintenance_bot removed a project: Patch-For-Review.Feb 19 2020, 3:11 PM
gerritbot added a comment.Feb 19 2020, 3:22 PM

Change 573308 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] install_server: Do not reimage es1020

https://gerrit.wikimedia.org/r/573308

gerritbot added a project: Patch-For-Review.Feb 19 2020, 3:22 PM

Change 573308 merged by Marostegui:
[operations/puppet@production] install_server: Do not reimage es1020

https://gerrit.wikimedia.org/r/573308

Maintenance_bot removed a project: Patch-For-Review.Feb 19 2020, 4:11 PM
gerritbot added a comment.Feb 20 2020, 6:31 AM

Change 573441 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Productionize es1021

https://gerrit.wikimedia.org/r/573441

gerritbot added a project: Patch-For-Review.Feb 20 2020, 6:31 AM

Change 573441 merged by Marostegui:
[operations/puppet@production] mariadb: Productionize es1021

https://gerrit.wikimedia.org/r/573441

gerritbot added a comment.Feb 20 2020, 6:53 AM

Change 573443 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Productionize es1022

https://gerrit.wikimedia.org/r/573443

gerritbot added a comment.Feb 20 2020, 6:55 AM

Change 573443 merged by Marostegui:
[operations/puppet@production] mariadb: Productionize es1022

https://gerrit.wikimedia.org/r/573443

Maintenance_bot removed a project: Patch-For-Review.Feb 20 2020, 7:10 AM
gerritbot added a comment.Feb 20 2020, 7:13 AM

Change 573447 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Productionize es1023

https://gerrit.wikimedia.org/r/573447

gerritbot added a project: Patch-For-Review.Feb 20 2020, 7:13 AM

Change 573447 merged by Marostegui:
[operations/puppet@production] mariadb: Productionize es1023

https://gerrit.wikimedia.org/r/573447

gerritbot added a comment.Feb 20 2020, 7:23 AM

Change 573449 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] es10123: Make it es5 master

https://gerrit.wikimedia.org/r/573449

gerritbot added a comment.Feb 20 2020, 7:25 AM

Change 573449 merged by Marostegui:
[operations/puppet@production] es10123: Make it es5 master

https://gerrit.wikimedia.org/r/573449

Maintenance_bot removed a project: Patch-For-Review.Feb 20 2020, 8:10 AM
gerritbot added a comment.Feb 20 2020, 8:19 AM

Change 573461 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Productionize es1024

https://gerrit.wikimedia.org/r/573461

gerritbot added a project: Patch-For-Review.Feb 20 2020, 8:19 AM

Change 573461 merged by Marostegui:
[operations/puppet@production] mariadb: Productionize es1024

https://gerrit.wikimedia.org/r/573461

• Marostegui added a comment.EditedFeb 20 2020, 8:33 AM

Grants in es4:

root@cumin1001:/home/marostegui/git/tendril/bin# /home/marostegui/section es4 | while read host port; do echo "====$host====" ; mysql.py -h$host:$port -e "select user,host from mysql.user where user like 'wik%'";done
====es2022.codfw.wmnet====
user	host
wikiadmin	10.192.%
wikiuser	10.192.%
wikiadmin	10.64.%
wikiuser	10.64.%
wikiadmin	localhost
wikiuser	localhost
====es2021.codfw.wmnet====
user	host
wikiadmin	10.192.%
wikiuser	10.192.%
wikiadmin	10.64.%
wikiuser	10.64.%
wikiadmin	localhost
wikiuser	localhost
====es2020.codfw.wmnet====
user	host
wikiadmin	10.192.%
wikiuser	10.192.%
wikiadmin	10.64.%
wikiuser	10.64.%
wikiadmin	localhost
wikiuser	localhost
====es1022.eqiad.wmnet====
user	host
wikiadmin	10.192.%
wikiuser	10.192.%
wikiadmin	10.64.%
wikiuser	10.64.%
wikiadmin	localhost
wikiuser	localhost
====es1021.eqiad.wmnet====
user	host
wikiadmin	10.192.%
wikiuser	10.192.%
wikiadmin	10.64.%
wikiuser	10.64.%
wikiadmin	localhost
wikiuser	localhost
====es1020.eqiad.wmnet====
user	host
wikiadmin	10.192.%
wikiuser	10.192.%
wikiadmin	10.64.%
wikiuser	10.64.%
wikiadmin	localhost
wikiuser	localhost

They look fine T245720#5900563

Maintenance_bot removed a project: Patch-For-Review.Feb 20 2020, 9:10 AM
Papaul closed subtask T245714: es2022 power supply issues as Resolved.Feb 20 2020, 3:55 PM
gerritbot added a comment.Feb 21 2020, 6:32 AM

Change 573860 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Productionize es1025

https://gerrit.wikimedia.org/r/573860

gerritbot added a project: Patch-For-Review.Feb 21 2020, 6:32 AM
Stashbot added a comment.Feb 21 2020, 6:34 AM

Mentioned in SAL (#wikimedia-operations) [2020-02-21T06:34:54Z] <marostegui> Stop mysql on es1024 to clone es1025 - T243052

gerritbot added a comment.Feb 21 2020, 6:35 AM

Change 573860 merged by Marostegui:
[operations/puppet@production] mariadb: Productionize es1025

https://gerrit.wikimedia.org/r/573860

Maintenance_bot removed a project: Patch-For-Review.Feb 21 2020, 7:10 AM
• Marostegui closed subtask T245806: Create es4 and es5 sections in dbctl as Resolved.Feb 24 2020, 3:00 PM
gerritbot added a comment.Feb 25 2020, 5:54 AM

Change 574641 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] es102[0-5]: Enable notifications

https://gerrit.wikimedia.org/r/574641

gerritbot added a project: Patch-For-Review.Feb 25 2020, 5:54 AM

Change 574641 merged by Marostegui:
[operations/puppet@production] es102[0-5]: Enable notifications

https://gerrit.wikimedia.org/r/574641

• Marostegui closed this task as Resolved.Feb 25 2020, 5:56 AM
• Marostegui updated the task description. (Show Details)

All hosts ready

• Marostegui mentioned this in T246072: Enable es4 and es5 as writable new external store sections and set es2 and es3 as read only.Feb 25 2020, 8:50 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL