Page MenuHomePhabricator
Create Task
Maniphest T202367

Productionize dbproxy101[2-7].eqiad.wmnet and dbproxy200[1-4]
Closed, ResolvedPublic
Actions

Description

These new proxies are ready to be productionized

eqiad:

  • dbproxy1012 rack: A5 will replace dbproxy1001
  • dbproxy1013 rack: A6 will replace dbproxy1002 (non primary)
  • dbproxy1014 rack: B1 will replace dbproxy1006 (non primary)
  • dbproxy1015 rack: B8 will replace dbproxy1007
  • dbproxy1016 rack: D1 will replace dbproxy1003 (non primary)
  • dbproxy1017 rack: D3 will replace dbproxy1005 in m5 (not in use at the moment)
  • dbproxy1018 (cloud VLAN) rack: C5 (will replace dbproxy1010)
  • dbproxy1019 (cloud VLAN) rack: C5 (will replace dbproxy1011)
  • dbproxy1020 rack: C5 will replace dbproxy1008
  • dbproxy1021 rack: C8 will go to m5 to replace dbproxy1005 (currently only has one proxy - and not in use)

codfw:

  • dbproxy2001 - m1
  • dbproxy2002 - m2
  • dbproxy2003 - m3
  • dbproxy2004- spare

Details

SubjectRepoBranchLines +/-
report_users: Remove dbproxy1011 addressoperations/softwaremaster+1 -1
mariadb: new dbproxy role & profileoperations/puppetproduction+91 -2
wmnet: Failover m3 primary dbproxyoperations/dnsmaster+1 -1
install_server: Reimage dbproxy1016 with Busteroperations/puppetproduction+1 -2
report_users: Add dbproxy1020operations/softwaremaster+3 -3
dbproxy1020: Enable notificationsoperations/puppetproduction+0 -1
production-m3.sql: Add dbproxy1020 grantsoperations/puppetproduction+49 -0
mariadb: Productionize dbproxy1020operations/puppetproduction+7 -6
install_server: Allow reimage of dbproxy1020operations/puppetproduction+1 -1
site: let new dbproxy machines use role(insetup)operations/puppetproduction+2 -2
wmnet: Promote dbproxy1015 to m2-masteroperations/dnsmaster+1 -1
dbproxy1015: Enable notificationsoperations/puppetproduction+0 -1
wmnet: Promote dbproxy1013 as m2-masteroperations/dnsmaster+1 -1
site.pp: Productionize dbproxy1015operations/puppetproduction+112 -3
wmnet: Failover m1-master to dbproxy1012operations/dnsmaster+1 -1
wmnet: Failover dbproxy1001 to dbproxy1014operations/dnsmaster+1 -1
dbproxy1012: Enable notificationsoperations/puppetproduction+0 -1
mariadb: Productionize dbproxy1012operations/puppetproduction+51 -2
bacula: Make bacula db parameters configurable on hieraoperations/puppetproduction+11 -5
wmnet: Point m1-master to dbproxy1014operations/dnsmaster+1 -1
dbproxy1021: Enable notificationsoperations/puppetproduction+0 -1
production-m5.sql.erb: Add dbproxy1021 grantsoperations/puppetproduction+29 -0
mariadb: Productionize dbproxy1021operations/puppetproduction+10 -3
dbproxy1017: Clarify that it belongs to m5operations/puppetproduction+1 -0
dbproxy1017: Enable notificationsoperations/puppetproduction+0 -1
dbproxy1017: Allow reimageoperations/puppetproduction+1 -0
mariadb: Provision dbproxy1017 to replace dbproxy1005operations/puppetproduction+18 -12
dbproxy1016: Enable notificationsoperations/puppetproduction+0 -1
install_server: Do not reimage dbproxy1018,dbproxy1019operations/puppetproduction+0 -1
dbproxy1018: Enable notificationsoperations/puppetproduction+0 -1
dbproxy1018: Productionize dbproxy1018, will replace dbproxy1010operations/puppetproduction+14 -2
dbproxy1019: Enable notificationsoperations/puppetproduction+0 -1
mariadb: Productionize dbproxy1016operations/puppetproduction+49 -2
install_server: Allow re-image dbproxy1018,dbproxy1019operations/puppetproduction+1 -0
dbproxy1019: Provision dbproxy1019 to replace dbproxy1011operations/puppetproduction+18 -2
dbproxy2003: Enable notificationsoperations/puppetproduction+0 -1
wmnet: Point m3-master codfw to dbproxy2003operations/dnsmaster+2 -0
mariadb: Productionize dbproxy2003 into m3-codfwoperations/puppetproduction+51 -2
wmnet: Add m1-master for codfwoperations/dnsmaster+2 -0
wmnet: Point m2-master.codfw to dbproxy2002operations/dnsmaster+2 -2
mariadb: Provision dbproxy1013operations/puppetproduction+118 -2
dbproxy2001: Enable notificationsoperations/puppetproduction+1 -1
mariadb: Productionize dbproxy2002 into m2-codfwoperations/puppetproduction+123 -2
mariadb: Replace dbproxy1006 with dbproxy1014 in m1operations/puppetproduction+24 -16
mariadb: Provision dbproxy2001 into codfw m1operations/puppetproduction+72 -2
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
gerritbot added a comment.Sep 16 2019, 7:52 AM

Change 536812 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Productionize dbproxy1021

https://gerrit.wikimedia.org/r/536812

gerritbot added a project: Patch-For-Review.Sep 16 2019, 7:52 AM

Change 536812 merged by Marostegui:
[operations/puppet@production] mariadb: Productionize dbproxy1021

https://gerrit.wikimedia.org/r/536812

ops-monitoring-bot added a comment.Sep 16 2019, 7:58 AM

Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:

['dbproxy1021.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201909160758_marostegui_74242.log.

Maintenance_bot removed a project: Patch-For-Review.Sep 16 2019, 8:10 AM
ops-monitoring-bot added a comment.Sep 16 2019, 8:16 AM

Completed auto-reimage of hosts:

['dbproxy1021.eqiad.wmnet']

and were ALL successful.

gerritbot added a comment.Sep 16 2019, 8:22 AM

Change 536961 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] production-m5.sql.erb: Add dbproxy1021 grants

https://gerrit.wikimedia.org/r/536961

gerritbot added a project: Patch-For-Review.Sep 16 2019, 8:22 AM
Stashbot added a comment.Sep 16 2019, 8:50 AM

Mentioned in SAL (#wikimedia-operations) [2019-09-16T08:50:08Z] <marostegui> Apply grants for dbproxy1021 on db1133 (m5 master) with replication - T202367

gerritbot added a comment.Sep 16 2019, 8:54 AM

Change 536961 merged by Marostegui:
[operations/puppet@production] production-m5.sql.erb: Add dbproxy1021 grants

https://gerrit.wikimedia.org/r/536961

Maintenance_bot removed a project: Patch-For-Review.Sep 16 2019, 9:11 AM
gerritbot added a comment.Sep 16 2019, 11:54 AM

Change 537085 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] dbproxy1021: Enable notifications

https://gerrit.wikimedia.org/r/537085

gerritbot added a project: Patch-For-Review.Sep 16 2019, 11:54 AM
gerritbot added a comment.Sep 16 2019, 11:57 AM

Change 537085 merged by Marostegui:
[operations/puppet@production] dbproxy1021: Enable notifications

https://gerrit.wikimedia.org/r/537085

Marostegui updated the task description. (Show Details)Sep 16 2019, 11:57 AM

dbproxy1021 has been placed in m5:

root@cumin1001:~# mysql --skip-ssl -h dbproxy1017 -e "select @@hostname"
+------------+
| @@hostname |
+------------+
| db1133     |
+------------+
gerritbot added a comment.Sep 18 2019, 9:59 AM

Change 537617 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/dns@master] wmnet: Point m1-master to dbproxy1014

https://gerrit.wikimedia.org/r/537617

gerritbot added a comment.Sep 18 2019, 10:10 AM

Change 537617 merged by Marostegui:
[operations/dns@master] wmnet: Point m1-master to dbproxy1014

https://gerrit.wikimedia.org/r/537617

Marostegui added a comment.Sep 18 2019, 10:29 AM

dbproxy1014 has been tested and it is now m1-master. In a couple of days I will revert this change as dbproxy1014 is in a rack that requires maintenance to the PDU, so I will move back m1-master to dbproxy1001 until that has passed.
Given that dbproxy1014 works fine, dbproxy1006 can go away.

Maintenance_bot removed a project: Patch-For-Review.Sep 18 2019, 11:10 AM
Marostegui added a comment.Sep 19 2019, 6:48 AM
In T202367#5502575, @Marostegui wrote:

dbproxy1014 has been tested and it is now m1-master. In a couple of days I will revert this change as dbproxy1014 is in a rack that requires maintenance to the PDU, so I will move back m1-master to dbproxy1001 until that has passed.
Given that dbproxy1014 works fine, dbproxy1006 can go away.

Change reverted, and dbproxy1001 back as active proxy for m1 after testing that dbproxy1014 works fine.

jcrespo awarded a token.Sep 19 2019, 2:03 PM
gerritbot added a comment.Sep 19 2019, 3:05 PM

Change 538042 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] bacula: Make bacula db parameters configurable on hiera

https://gerrit.wikimedia.org/r/538042

gerritbot added a project: Patch-For-Review.Sep 19 2019, 3:05 PM
gerritbot added a comment.Sep 20 2019, 7:24 AM

Change 538042 merged by Jcrespo:
[operations/puppet@production] bacula: Make bacula db parameters configurable on hiera

https://gerrit.wikimedia.org/r/538042

Maintenance_bot removed a project: Patch-For-Review.Sep 20 2019, 8:10 AM
Marostegui moved this task from In progress to Pending comment on the DBA board.Nov 18 2019, 10:18 AM
RobH closed subtask T231967: Decommission dbproxy1005.eqiad.wmnet as Resolved.Dec 19 2019, 11:09 PM
gerritbot added a comment.Feb 6 2020, 7:47 AM

Change 570574 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Productionize dbproxy1012

https://gerrit.wikimedia.org/r/570574

gerritbot added a project: Patch-For-Review.Feb 6 2020, 7:47 AM
gerritbot added a comment.Feb 6 2020, 8:03 AM

Change 570574 merged by Marostegui:
[operations/puppet@production] mariadb: Productionize dbproxy1012

https://gerrit.wikimedia.org/r/570574

gerritbot added a comment.Feb 6 2020, 8:30 AM

Change 570580 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/dns@master] wmnet: Failover dbproxy1001 to dbproxy1014

https://gerrit.wikimedia.org/r/570580

gerritbot added a comment.Feb 6 2020, 8:32 AM

Change 570581 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] dbproxy1012: Enable notifications

https://gerrit.wikimedia.org/r/570581

gerritbot added a comment.Feb 6 2020, 8:33 AM

Change 570581 merged by Marostegui:
[operations/puppet@production] dbproxy1012: Enable notifications

https://gerrit.wikimedia.org/r/570581

Marostegui updated the task description. (Show Details)Feb 6 2020, 8:34 AM
gerritbot added a comment.Feb 6 2020, 9:07 AM

Change 570580 merged by Marostegui:
[operations/dns@master] wmnet: Failover dbproxy1001 to dbproxy1014

https://gerrit.wikimedia.org/r/570580

Maintenance_bot removed a project: Patch-For-Review.Feb 6 2020, 9:10 AM
gerritbot added a comment.Feb 11 2020, 6:14 AM

Change 571418 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/dns@master] wmnet: Failover m1-master to dbproxy1012

https://gerrit.wikimedia.org/r/571418

gerritbot added a project: Patch-For-Review.Feb 11 2020, 6:14 AM
gerritbot added a comment.Feb 11 2020, 6:17 AM

Change 571418 merged by Marostegui:
[operations/dns@master] wmnet: Failover m1-master to dbproxy1012

https://gerrit.wikimedia.org/r/571418

Stashbot added a comment.Feb 11 2020, 6:18 AM

Mentioned in SAL (#wikimedia-operations) [2020-02-11T06:18:10Z] <marostegui> Failover m1-master from dbproxy1014 to dbproxy1012 - T202367

Maintenance_bot removed a project: Patch-For-Review.Feb 11 2020, 7:10 AM
gerritbot added a comment.Feb 12 2020, 7:40 AM

Change 571659 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] site.pp: Productionize dbproxy1015

https://gerrit.wikimedia.org/r/571659

gerritbot added a project: Patch-For-Review.Feb 12 2020, 7:40 AM
gerritbot added a comment.Feb 12 2020, 8:58 AM

Change 571659 merged by Marostegui:
[operations/puppet@production] site.pp: Productionize dbproxy1015

https://gerrit.wikimedia.org/r/571659

Stashbot added a comment.Feb 12 2020, 9:11 AM

Mentioned in SAL (#wikimedia-operations) [2020-02-12T09:11:31Z] <marostegui> Upgrade and reboot dbproxy1013 before making it master - T202367

gerritbot added a comment.Feb 12 2020, 9:14 AM

Change 571679 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/dns@master] wmnet: Promote dbproxy1013 as m2-master

https://gerrit.wikimedia.org/r/571679

gerritbot added a comment.Feb 12 2020, 9:17 AM

Change 571679 merged by Marostegui:
[operations/dns@master] wmnet: Promote dbproxy1013 as m2-master

https://gerrit.wikimedia.org/r/571679

Stashbot added a comment.Feb 12 2020, 9:17 AM

Mentioned in SAL (#wikimedia-operations) [2020-02-12T09:17:57Z] <marostegui> Failover m2 master dbproxy from dbproxy1007 to dbproxy1013 - T202367

Maintenance_bot removed a project: Patch-For-Review.Feb 12 2020, 10:11 AM
gerritbot added a comment.Feb 13 2020, 8:07 AM

Change 571920 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] dbproxy1015: Enable notifications

https://gerrit.wikimedia.org/r/571920

gerritbot added a project: Patch-For-Review.Feb 13 2020, 8:07 AM

Change 571920 merged by Marostegui:
[operations/puppet@production] dbproxy1015: Enable notifications

https://gerrit.wikimedia.org/r/571920

Maintenance_bot removed a project: Patch-For-Review.Feb 13 2020, 8:10 AM
Marostegui moved this task from Pending comment to In progress on the DBA board.Feb 13 2020, 9:27 AM
gerritbot added a comment.Feb 17 2020, 6:20 AM

Change 572512 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/dns@master] wmnet: Promote dbproxy1015 to m2-master

https://gerrit.wikimedia.org/r/572512

gerritbot added a project: Patch-For-Review.Feb 17 2020, 6:20 AM

Change 572512 merged by Marostegui:
[operations/dns@master] wmnet: Promote dbproxy1015 to m2-master

https://gerrit.wikimedia.org/r/572512

Marostegui updated the task description. (Show Details)Feb 17 2020, 9:29 AM
Maintenance_bot removed a project: Patch-For-Review.Feb 17 2020, 10:11 AM
gerritbot added a comment.Mar 11 2020, 6:48 PM

Change 579024 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] site: let new dbproxy machines use role(insetup)

https://gerrit.wikimedia.org/r/579024

gerritbot added a project: Patch-For-Review.Mar 11 2020, 6:48 PM
gerritbot added a comment.Mar 11 2020, 6:52 PM

Change 579024 merged by Dzahn:
[operations/puppet@production] site: let new dbproxy machines use role(insetup)

https://gerrit.wikimedia.org/r/579024

Maintenance_bot removed a project: Patch-For-Review.Mar 11 2020, 7:11 PM
gerritbot added a comment.May 12 2020, 3:23 PM

Change 595960 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] install_server: Allow reimage of dbproxy1020

https://gerrit.wikimedia.org/r/595960

gerritbot added a project: Patch-For-Review.May 12 2020, 3:23 PM

Change 595960 merged by Marostegui:
[operations/puppet@production] install_server: Allow reimage of dbproxy1020

https://gerrit.wikimedia.org/r/595960

Maintenance_bot removed a project: Patch-For-Review.May 12 2020, 4:11 PM
gerritbot added a comment.Jun 15 2020, 6:01 AM

Change 605444 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Productionize dbproxy1020

https://gerrit.wikimedia.org/r/605444

gerritbot added a project: Patch-For-Review.Jun 15 2020, 6:01 AM
gerritbot added a comment.Jun 15 2020, 6:04 AM

Change 605444 merged by Marostegui:
[operations/puppet@production] mariadb: Productionize dbproxy1020

https://gerrit.wikimedia.org/r/605444

ops-monitoring-bot added a comment.Jun 15 2020, 6:06 AM

Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:

['dbproxy1020.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202006150606_marostegui_57510.log.

Maintenance_bot removed a project: Patch-For-Review.Jun 15 2020, 6:10 AM
ops-monitoring-bot added a comment.Jun 15 2020, 6:26 AM

Completed auto-reimage of hosts:

['dbproxy1020.eqiad.wmnet']

and were ALL successful.

gerritbot added a comment.Jun 15 2020, 6:50 AM

Change 605524 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] production-m3.sql: Add dbproxy1020 grants

https://gerrit.wikimedia.org/r/605524

gerritbot added a project: Patch-For-Review.Jun 15 2020, 6:50 AM
gerritbot added a comment.Jun 15 2020, 6:53 AM

Change 605524 merged by Marostegui:
[operations/puppet@production] production-m3.sql: Add dbproxy1020 grants

https://gerrit.wikimedia.org/r/605524

Maintenance_bot removed a project: Patch-For-Review.Jun 15 2020, 7:10 AM
gerritbot added a comment.Jun 15 2020, 7:15 AM

Change 605525 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] dbproxy1020: Enable notifications

https://gerrit.wikimedia.org/r/605525

gerritbot added a project: Patch-For-Review.Jun 15 2020, 7:15 AM
gerritbot added a comment.Jun 15 2020, 7:22 AM

Change 605526 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/software@master] report_users: Add dbproxy1020

https://gerrit.wikimedia.org/r/605526

gerritbot added a comment.Jun 15 2020, 7:22 AM

Change 605525 merged by Marostegui:
[operations/puppet@production] dbproxy1020: Enable notifications

https://gerrit.wikimedia.org/r/605525

gerritbot added a comment.Jun 15 2020, 7:23 AM

Change 605526 merged by Marostegui:
[operations/software@master] report_users: Add dbproxy1020

https://gerrit.wikimedia.org/r/605526

gerritbot added a comment.Jun 15 2020, 7:33 AM

Change 605528 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] install_server: Reimage dbproxy1016 with Buster

https://gerrit.wikimedia.org/r/605528

gerritbot added a comment.Jun 15 2020, 7:34 AM

Change 605528 merged by Marostegui:
[operations/puppet@production] install_server: Reimage dbproxy1016 with Buster

https://gerrit.wikimedia.org/r/605528

ops-monitoring-bot added a comment.Jun 15 2020, 7:35 AM

Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:

['dbproxy1016.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202006150735_marostegui_70581.log.

ops-monitoring-bot added a comment.Jun 15 2020, 7:56 AM

Completed auto-reimage of hosts:

['dbproxy1016.eqiad.wmnet']

and were ALL successful.

gerritbot added a comment.Jun 15 2020, 8:02 AM

Change 605531 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/dns@master] wmnet: Failover m3 primary dbproxy

https://gerrit.wikimedia.org/r/605531

gerritbot added a comment.Jun 15 2020, 8:21 AM

Change 605531 merged by Marostegui:
[operations/dns@master] wmnet: Failover m3 primary dbproxy

https://gerrit.wikimedia.org/r/605531

Stashbot added a comment.Jun 15 2020, 8:22 AM

Mentioned in SAL (#wikimedia-operations) [2020-06-15T08:22:06Z] <marostegui> Switchover m3-master from dbproxy1008 to dbproxy1016 - T202367

Marostegui closed this task as Resolved.Jun 15 2020, 8:43 AM
Marostegui updated the task description. (Show Details)

All hosts have been productionized.
The hosts that are still pending to be decommissioned (dbproxy1003, dbproxy1008) will be handled in separated tasks.

Marostegui closed subtask T231280: Remove grants for the old dbproxy hosts from the misc databases as Resolved.Jul 8 2020, 8:40 AM
gerritbot added a comment.Feb 14 2022, 7:33 AM

Change 473546 abandoned by Marostegui:

[operations/puppet@production] mariadb: new dbproxy role & profile

Reason:

Old patch

https://gerrit.wikimedia.org/r/473546

gerritbot added a comment.Nov 16 2023, 7:18 AM

Change 974727 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/software@master] report_users: Remove dbproxy1011 address

https://gerrit.wikimedia.org/r/974727

gerritbot added a comment.Nov 16 2023, 7:19 AM

Change 974727 merged by jenkins-bot:

[operations/software@master] report_users: Remove dbproxy1011 address

https://gerrit.wikimedia.org/r/974727

Maintenance_bot moved this task from In progress to Done on the DBA board.Nov 16 2023, 7:29 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits