Page MenuHomePhabricator

Productionize dbproxy101[2-7].eqiad.wmnet and dbproxy200[1-4]
Closed, ResolvedPublic

Description

These new proxies are ready to be productionized

eqiad:

  • dbproxy1012 rack: A5 will replace dbproxy1001
  • dbproxy1013 rack: A6 will replace dbproxy1002 (non primary)
  • dbproxy1014 rack: B1 will replace dbproxy1006 (non primary)
  • dbproxy1015 rack: B8 will replace dbproxy1007
  • dbproxy1016 rack: D1 will replace dbproxy1003 (non primary)
  • dbproxy1017 rack: D3 will replace dbproxy1005 in m5 (not in use at the moment)
  • dbproxy1018 (cloud VLAN) rack: C5 (will replace dbproxy1010)
  • dbproxy1019 (cloud VLAN) rack: C5 (will replace dbproxy1011)
  • dbproxy1020 rack: C5 will replace dbproxy1008
  • dbproxy1021 rack: C8 will go to m5 to replace dbproxy1005 (currently only has one proxy - and not in use)

codfw:

  • dbproxy2001 - m1
  • dbproxy2002 - m2
  • dbproxy2003 - m3
  • dbproxy2004- spare

Details

ProjectBranchLines +/-Subject
operations/dnsmaster+1 -1
operations/puppetproduction+1 -2
operations/softwaremaster+3 -3
operations/puppetproduction+0 -1
operations/puppetproduction+49 -0
operations/puppetproduction+7 -6
operations/puppetproduction+1 -1
operations/puppetproduction+2 -2
operations/dnsmaster+1 -1
operations/puppetproduction+0 -1
operations/dnsmaster+1 -1
operations/puppetproduction+112 -3
operations/dnsmaster+1 -1
operations/dnsmaster+1 -1
operations/puppetproduction+0 -1
operations/puppetproduction+51 -2
operations/puppetproduction+11 -5
operations/dnsmaster+1 -1
operations/puppetproduction+0 -1
operations/puppetproduction+29 -0
operations/puppetproduction+10 -3
operations/puppetproduction+1 -0
operations/puppetproduction+0 -1
operations/puppetproduction+1 -0
operations/puppetproduction+18 -12
operations/puppetproduction+0 -1
operations/puppetproduction+0 -1
operations/puppetproduction+0 -1
operations/puppetproduction+14 -2
operations/puppetproduction+0 -1
operations/puppetproduction+49 -2
operations/puppetproduction+1 -0
operations/puppetproduction+18 -2
operations/puppetproduction+0 -1
operations/dnsmaster+2 -0
operations/puppetproduction+51 -2
operations/dnsmaster+2 -0
operations/dnsmaster+2 -2
operations/puppetproduction+118 -2
operations/puppetproduction+1 -1
operations/puppetproduction+123 -2
operations/puppetproduction+24 -16
operations/puppetproduction+72 -2
operations/puppetproduction+91 -2
Show related patches Customize query in gerrit

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Change 534297 merged by Marostegui:
[operations/puppet@production] dbproxy1017: Clarify that it belongs to m5

https://gerrit.wikimedia.org/r/534297

dbproxy1017 is now ready to replace dbproxy1005 (even though they are not in use at the moment)

#  mysql --skip-ssl -hdbproxy1017 -e "select @@hostname"
+------------+
| @@hostname |
+------------+
| db1133     |
+------------+

Change 536812 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Productionize dbproxy1021

https://gerrit.wikimedia.org/r/536812

Change 536812 merged by Marostegui:
[operations/puppet@production] mariadb: Productionize dbproxy1021

https://gerrit.wikimedia.org/r/536812

Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:

['dbproxy1021.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201909160758_marostegui_74242.log.

Completed auto-reimage of hosts:

['dbproxy1021.eqiad.wmnet']

and were ALL successful.

Change 536961 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] production-m5.sql.erb: Add dbproxy1021 grants

https://gerrit.wikimedia.org/r/536961

Mentioned in SAL (#wikimedia-operations) [2019-09-16T08:50:08Z] <marostegui> Apply grants for dbproxy1021 on db1133 (m5 master) with replication - T202367

Change 536961 merged by Marostegui:
[operations/puppet@production] production-m5.sql.erb: Add dbproxy1021 grants

https://gerrit.wikimedia.org/r/536961

Change 537085 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] dbproxy1021: Enable notifications

https://gerrit.wikimedia.org/r/537085

Change 537085 merged by Marostegui:
[operations/puppet@production] dbproxy1021: Enable notifications

https://gerrit.wikimedia.org/r/537085

dbproxy1021 has been placed in m5:

root@cumin1001:~# mysql --skip-ssl -h dbproxy1017 -e "select @@hostname"
+------------+
| @@hostname |
+------------+
| db1133     |
+------------+

Change 537617 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/dns@master] wmnet: Point m1-master to dbproxy1014

https://gerrit.wikimedia.org/r/537617

Change 537617 merged by Marostegui:
[operations/dns@master] wmnet: Point m1-master to dbproxy1014

https://gerrit.wikimedia.org/r/537617

dbproxy1014 has been tested and it is now m1-master. In a couple of days I will revert this change as dbproxy1014 is in a rack that requires maintenance to the PDU, so I will move back m1-master to dbproxy1001 until that has passed.
Given that dbproxy1014 works fine, dbproxy1006 can go away.

dbproxy1014 has been tested and it is now m1-master. In a couple of days I will revert this change as dbproxy1014 is in a rack that requires maintenance to the PDU, so I will move back m1-master to dbproxy1001 until that has passed.
Given that dbproxy1014 works fine, dbproxy1006 can go away.

Change reverted, and dbproxy1001 back as active proxy for m1 after testing that dbproxy1014 works fine.

Change 538042 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] bacula: Make bacula db parameters configurable on hiera

https://gerrit.wikimedia.org/r/538042

Change 538042 merged by Jcrespo:
[operations/puppet@production] bacula: Make bacula db parameters configurable on hiera

https://gerrit.wikimedia.org/r/538042

Marostegui moved this task from In progress to Next on the DBA board.Nov 18 2019, 10:18 AM

Change 570574 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Productionize dbproxy1012

https://gerrit.wikimedia.org/r/570574

Change 570574 merged by Marostegui:
[operations/puppet@production] mariadb: Productionize dbproxy1012

https://gerrit.wikimedia.org/r/570574

Change 570580 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/dns@master] wmnet: Failover dbproxy1001 to dbproxy1014

https://gerrit.wikimedia.org/r/570580

Change 570581 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] dbproxy1012: Enable notifications

https://gerrit.wikimedia.org/r/570581

Change 570581 merged by Marostegui:
[operations/puppet@production] dbproxy1012: Enable notifications

https://gerrit.wikimedia.org/r/570581

Change 570580 merged by Marostegui:
[operations/dns@master] wmnet: Failover dbproxy1001 to dbproxy1014

https://gerrit.wikimedia.org/r/570580

Change 571418 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/dns@master] wmnet: Failover m1-master to dbproxy1012

https://gerrit.wikimedia.org/r/571418

Change 571418 merged by Marostegui:
[operations/dns@master] wmnet: Failover m1-master to dbproxy1012

https://gerrit.wikimedia.org/r/571418

Mentioned in SAL (#wikimedia-operations) [2020-02-11T06:18:10Z] <marostegui> Failover m1-master from dbproxy1014 to dbproxy1012 - T202367

Change 571659 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] site.pp: Productionize dbproxy1015

https://gerrit.wikimedia.org/r/571659

Change 571659 merged by Marostegui:
[operations/puppet@production] site.pp: Productionize dbproxy1015

https://gerrit.wikimedia.org/r/571659

Mentioned in SAL (#wikimedia-operations) [2020-02-12T09:11:31Z] <marostegui> Upgrade and reboot dbproxy1013 before making it master - T202367

Change 571679 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/dns@master] wmnet: Promote dbproxy1013 as m2-master

https://gerrit.wikimedia.org/r/571679

Change 571679 merged by Marostegui:
[operations/dns@master] wmnet: Promote dbproxy1013 as m2-master

https://gerrit.wikimedia.org/r/571679

Mentioned in SAL (#wikimedia-operations) [2020-02-12T09:17:57Z] <marostegui> Failover m2 master dbproxy from dbproxy1007 to dbproxy1013 - T202367

Change 571920 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] dbproxy1015: Enable notifications

https://gerrit.wikimedia.org/r/571920

Change 571920 merged by Marostegui:
[operations/puppet@production] dbproxy1015: Enable notifications

https://gerrit.wikimedia.org/r/571920

Marostegui moved this task from Next to In progress on the DBA board.Feb 13 2020, 9:27 AM

Change 572512 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/dns@master] wmnet: Promote dbproxy1015 to m2-master

https://gerrit.wikimedia.org/r/572512

Change 572512 merged by Marostegui:
[operations/dns@master] wmnet: Promote dbproxy1015 to m2-master

https://gerrit.wikimedia.org/r/572512

Change 579024 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] site: let new dbproxy machines use role(insetup)

https://gerrit.wikimedia.org/r/579024

Change 579024 merged by Dzahn:
[operations/puppet@production] site: let new dbproxy machines use role(insetup)

https://gerrit.wikimedia.org/r/579024

Change 595960 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] install_server: Allow reimage of dbproxy1020

https://gerrit.wikimedia.org/r/595960

Change 595960 merged by Marostegui:
[operations/puppet@production] install_server: Allow reimage of dbproxy1020

https://gerrit.wikimedia.org/r/595960

Change 605444 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Productionize dbproxy1020

https://gerrit.wikimedia.org/r/605444

Change 605444 merged by Marostegui:
[operations/puppet@production] mariadb: Productionize dbproxy1020

https://gerrit.wikimedia.org/r/605444

Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:

['dbproxy1020.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202006150606_marostegui_57510.log.

Completed auto-reimage of hosts:

['dbproxy1020.eqiad.wmnet']

and were ALL successful.

Change 605524 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] production-m3.sql: Add dbproxy1020 grants

https://gerrit.wikimedia.org/r/605524

Change 605524 merged by Marostegui:
[operations/puppet@production] production-m3.sql: Add dbproxy1020 grants

https://gerrit.wikimedia.org/r/605524

Change 605525 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] dbproxy1020: Enable notifications

https://gerrit.wikimedia.org/r/605525

Change 605526 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/software@master] report_users: Add dbproxy1020

https://gerrit.wikimedia.org/r/605526

Change 605525 merged by Marostegui:
[operations/puppet@production] dbproxy1020: Enable notifications

https://gerrit.wikimedia.org/r/605525

Change 605526 merged by Marostegui:
[operations/software@master] report_users: Add dbproxy1020

https://gerrit.wikimedia.org/r/605526

Change 605528 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] install_server: Reimage dbproxy1016 with Buster

https://gerrit.wikimedia.org/r/605528

Change 605528 merged by Marostegui:
[operations/puppet@production] install_server: Reimage dbproxy1016 with Buster

https://gerrit.wikimedia.org/r/605528

Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:

['dbproxy1016.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202006150735_marostegui_70581.log.

Completed auto-reimage of hosts:

['dbproxy1016.eqiad.wmnet']

and were ALL successful.

Change 605531 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/dns@master] wmnet: Failover m3 primary dbproxy

https://gerrit.wikimedia.org/r/605531

Change 605531 merged by Marostegui:
[operations/dns@master] wmnet: Failover m3 primary dbproxy

https://gerrit.wikimedia.org/r/605531

Mentioned in SAL (#wikimedia-operations) [2020-06-15T08:22:06Z] <marostegui> Switchover m3-master from dbproxy1008 to dbproxy1016 - T202367

Marostegui closed this task as Resolved.Mon, Jun 15, 8:43 AM
Marostegui updated the task description. (Show Details)

All hosts have been productionized.
The hosts that are still pending to be decommissioned (dbproxy1003, dbproxy1008) will be handled in separated tasks.