Page MenuHomePhabricator

Productionize dbproxy101[2-7].eqiad.wmnet and dbproxy200[1-4]
Open, MediumPublic

Description

These new proxies are ready to be productionized

eqiad:

  • dbproxy1012 rack: A5 will replace dbproxy1001
  • dbproxy1013 rack: A6 will replace dbproxy1002 (non primary)
  • dbproxy1014 rack: B1 will replace dbproxy1006 (non primary)
  • dbproxy1015 rack: B8 will replace dbproxy1007
  • dbproxy1016 rack: D1 will replace dbproxy1003 (non primary)
  • dbproxy1017 rack: D3 will replace dbproxy1005 in m5 (not in use at the moment)
  • dbproxy1018 (cloud VLAN) rack: C5 (will replace dbproxy1010)
  • dbproxy1019 (cloud VLAN) rack: C5 (will replace dbproxy1011)
  • dbproxy1020 rack: C5 will replace dbproxy1008
  • dbproxy1021 rack: C8 will go to m5 to replace dbproxy1005 (currently only has one proxy - and not in use)

codfw:

  • dbproxy2001 - m1
  • dbproxy2002 - m2
  • dbproxy2003 - m3
  • dbproxy2004- spare

Details

Related Gerrit Patches:
operations/dns : masterwmnet: Promote dbproxy1015 to m2-master
operations/puppet : productiondbproxy1015: Enable notifications
operations/dns : masterwmnet: Promote dbproxy1013 as m2-master
operations/puppet : productionsite.pp: Productionize dbproxy1015
operations/dns : masterwmnet: Failover m1-master to dbproxy1012
operations/dns : masterwmnet: Failover dbproxy1001 to dbproxy1014
operations/puppet : productiondbproxy1012: Enable notifications
operations/puppet : productionmariadb: Productionize dbproxy1012
operations/puppet : productionbacula: Make bacula db parameters configurable on hiera
operations/dns : masterwmnet: Point m1-master to dbproxy1014
operations/puppet : productiondbproxy1021: Enable notifications
operations/puppet : productionproduction-m5.sql.erb: Add dbproxy1021 grants
operations/puppet : productionmariadb: Productionize dbproxy1021
operations/puppet : productiondbproxy1017: Clarify that it belongs to m5
operations/puppet : productiondbproxy1017: Enable notifications
operations/puppet : productiondbproxy1017: Allow reimage
operations/puppet : productionmariadb: Provision dbproxy1017 to replace dbproxy1005
operations/puppet : productiondbproxy1016: Enable notifications
operations/puppet : productioninstall_server: Do not reimage dbproxy1018,dbproxy1019
operations/puppet : productiondbproxy1018: Enable notifications
operations/puppet : productiondbproxy1018: Productionize dbproxy1018, will replace dbproxy1010
operations/puppet : productiondbproxy1019: Enable notifications
operations/puppet : productionmariadb: Productionize dbproxy1016
operations/puppet : productioninstall_server: Allow re-image dbproxy1018,dbproxy1019
operations/puppet : productiondbproxy1019: Provision dbproxy1019 to replace dbproxy1011
operations/puppet : productiondbproxy2003: Enable notifications
operations/dns : masterwmnet: Point m3-master codfw to dbproxy2003
operations/puppet : productionmariadb: Productionize dbproxy2003 into m3-codfw
operations/dns : masterwmnet: Add m1-master for codfw
operations/dns : masterwmnet: Point m2-master.codfw to dbproxy2002
operations/puppet : productionmariadb: Provision dbproxy1013
operations/puppet : productiondbproxy2001: Enable notifications
operations/puppet : productionmariadb: Productionize dbproxy2002 into m2-codfw
operations/puppet : productionmariadb: Replace dbproxy1006 with dbproxy1014 in m1
operations/puppet : productionmariadb: Provision dbproxy2001 into codfw m1
operations/puppet : productionmariadb: new dbproxy role & profile

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Completed auto-reimage of hosts:

['dbproxy1018.eqiad.wmnet']

and were ALL successful.

Mentioned in SAL (#wikimedia-operations) [2019-08-28T08:17:35Z] <marostegui> Deploy grants on labsdb hosts for dbproxy1018 - T202367

dbproxy1018 is now monitoring the hosts:

root@dbproxy1018:~# echo "show stat" | socat /run/haproxy/haproxy.sock stdio
# pxname,svname,qcur,qmax,scur,smax,slim,stot,bin,bout,dreq,dresp,ereq,econ,eresp,wretr,wredis,status,weight,act,bck,chkfail,chkdown,lastchg,downtime,qlimit,pid,iid,sid,throttle,lbtot,tracked,type,rate,rate_lim,rate_max,check_status,check_code,check_duration,hrsp_1xx,hrsp_2xx,hrsp_3xx,hrsp_4xx,hrsp_5xx,hrsp_other,hanafail,req_rate,req_rate_max,req_tot,cli_abrt,srv_abrt,comp_in,comp_out,comp_byp,comp_rsp,lastsess,last_chk,last_agt,qtime,ctime,rtime,ttime,agent_status,agent_code,agent_duration,check_desc,agent_desc,check_rise,check_fall,check_health,agent_rise,agent_fall,agent_health,addr,cookie,mode,algo,conn_rate,conn_rate_max,conn_tot,intercepted,dcon,dses,
mariadb,FRONTEND,,,0,0,5000,0,0,0,0,0,0,,,,,OPEN,,,,,,,,,1,2,0,,,,0,0,0,0,,,,,,,,,,,0,0,0,,,0,0,0,0,,,,,,,,,,,,,,,,,,,,,tcp,,0,0,0,,0,0,
mariadb,labsdb1011,0,0,0,0,,0,0,0,,0,,0,0,0,0,UP,1,1,0,0,0,3,0,,1,2,1,,0,,2,0,,0,L7OK,0,5,,,,,,,,,,,0,0,,,,,-1,5.5.5-10.1.39-MariaDB,,0,0,0,0,,,,Layer7 check passed,,99999999,20,100000018,,,,,,tcp,,,,,,,,
mariadb,labsdb1010,0,0,0,0,,0,0,0,,0,,0,0,0,0,UP,1,0,1,0,0,3,0,,1,2,2,,0,,2,0,,0,L7OK,0,2,,,,,,,,,,,0,0,,,,,-1,5.5.5-10.1.39-MariaDB,,0,0,0,0,,,,Layer7 check passed,,2,3,4,,,,,,tcp,,,,,,,,
mariadb,BACKEND,0,0,0,0,500,0,0,0,0,0,,0,0,0,0,UP,1,1,1,,0,3,0,,1,2,0,,0,,1,0,,0,,,,,,,,,,,,,,0,0,0,0,0,0,-1,,,0,0,0,0,,,,,,,,,,,,,,tcp,,,,,,,,

Waiting for T231418: Review switches ACL to connect from tools-bastion to dbproxy1018 to be complete to consider this host ready to replace dbproxy1010

Marostegui updated the task description. (Show Details)Aug 28 2019, 12:13 PM

Change 533144 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] dbproxy1018: Enable notifications

https://gerrit.wikimedia.org/r/533144

Change 533144 merged by Marostegui:
[operations/puppet@production] dbproxy1018: Enable notifications

https://gerrit.wikimedia.org/r/533144

Marostegui updated the task description. (Show Details)Aug 29 2019, 5:34 AM

Change 533171 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] install_server: Do not reimage dbproxy1018,dbproxy1019

https://gerrit.wikimedia.org/r/533171

Change 533171 merged by Marostegui:
[operations/puppet@production] install_server: Do not reimage dbproxy1018,dbproxy1019

https://gerrit.wikimedia.org/r/533171

Change 533371 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] dbproxy1016: Enable notifications

https://gerrit.wikimedia.org/r/533371

Change 533371 merged by Marostegui:
[operations/puppet@production] dbproxy1016: Enable notifications

https://gerrit.wikimedia.org/r/533371

Change 534270 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Provision dbproxy1017 to replace dbproxy1005

https://gerrit.wikimedia.org/r/534270

Change 534270 merged by Marostegui:
[operations/puppet@production] mariadb: Provision dbproxy1017 to replace dbproxy1005

https://gerrit.wikimedia.org/r/534270

Change 534272 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] dbproxy1017: Allow reimage

https://gerrit.wikimedia.org/r/534272

Change 534272 merged by Marostegui:
[operations/puppet@production] dbproxy1017: Allow reimage

https://gerrit.wikimedia.org/r/534272

Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:

dbproxy1017.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/201909040539_marostegui_173520_dbproxy1017_eqiad_wmnet.log.

Completed auto-reimage of hosts:

['dbproxy1017.eqiad.wmnet']

and were ALL successful.

Change 534292 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] dbproxy1017: Enable notifications

https://gerrit.wikimedia.org/r/534292

Change 534292 merged by Marostegui:
[operations/puppet@production] dbproxy1017: Enable notifications

https://gerrit.wikimedia.org/r/534292

Change 534297 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] dbproxy1017: Clarify that it belongs to m5

https://gerrit.wikimedia.org/r/534297

Change 534297 merged by Marostegui:
[operations/puppet@production] dbproxy1017: Clarify that it belongs to m5

https://gerrit.wikimedia.org/r/534297

dbproxy1017 is now ready to replace dbproxy1005 (even though they are not in use at the moment)

#  mysql --skip-ssl -hdbproxy1017 -e "select @@hostname"
+------------+
| @@hostname |
+------------+
| db1133     |
+------------+
Marostegui updated the task description. (Show Details)Sep 5 2019, 6:13 AM
Marostegui updated the task description. (Show Details)Sep 16 2019, 7:51 AM

Change 536812 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Productionize dbproxy1021

https://gerrit.wikimedia.org/r/536812

Change 536812 merged by Marostegui:
[operations/puppet@production] mariadb: Productionize dbproxy1021

https://gerrit.wikimedia.org/r/536812

Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:

['dbproxy1021.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201909160758_marostegui_74242.log.

Completed auto-reimage of hosts:

['dbproxy1021.eqiad.wmnet']

and were ALL successful.

Change 536961 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] production-m5.sql.erb: Add dbproxy1021 grants

https://gerrit.wikimedia.org/r/536961

Mentioned in SAL (#wikimedia-operations) [2019-09-16T08:50:08Z] <marostegui> Apply grants for dbproxy1021 on db1133 (m5 master) with replication - T202367

Change 536961 merged by Marostegui:
[operations/puppet@production] production-m5.sql.erb: Add dbproxy1021 grants

https://gerrit.wikimedia.org/r/536961

Change 537085 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] dbproxy1021: Enable notifications

https://gerrit.wikimedia.org/r/537085

Change 537085 merged by Marostegui:
[operations/puppet@production] dbproxy1021: Enable notifications

https://gerrit.wikimedia.org/r/537085

Marostegui updated the task description. (Show Details)Sep 16 2019, 11:57 AM

dbproxy1021 has been placed in m5:

root@cumin1001:~# mysql --skip-ssl -h dbproxy1017 -e "select @@hostname"
+------------+
| @@hostname |
+------------+
| db1133     |
+------------+

Change 537617 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/dns@master] wmnet: Point m1-master to dbproxy1014

https://gerrit.wikimedia.org/r/537617

Change 537617 merged by Marostegui:
[operations/dns@master] wmnet: Point m1-master to dbproxy1014

https://gerrit.wikimedia.org/r/537617

dbproxy1014 has been tested and it is now m1-master. In a couple of days I will revert this change as dbproxy1014 is in a rack that requires maintenance to the PDU, so I will move back m1-master to dbproxy1001 until that has passed.
Given that dbproxy1014 works fine, dbproxy1006 can go away.

dbproxy1014 has been tested and it is now m1-master. In a couple of days I will revert this change as dbproxy1014 is in a rack that requires maintenance to the PDU, so I will move back m1-master to dbproxy1001 until that has passed.
Given that dbproxy1014 works fine, dbproxy1006 can go away.

Change reverted, and dbproxy1001 back as active proxy for m1 after testing that dbproxy1014 works fine.

Change 538042 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] bacula: Make bacula db parameters configurable on hiera

https://gerrit.wikimedia.org/r/538042

Change 538042 merged by Jcrespo:
[operations/puppet@production] bacula: Make bacula db parameters configurable on hiera

https://gerrit.wikimedia.org/r/538042

Marostegui moved this task from In progress to Next on the DBA board.Nov 18 2019, 10:18 AM

Change 570574 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Productionize dbproxy1012

https://gerrit.wikimedia.org/r/570574

Change 570574 merged by Marostegui:
[operations/puppet@production] mariadb: Productionize dbproxy1012

https://gerrit.wikimedia.org/r/570574

Change 570580 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/dns@master] wmnet: Failover dbproxy1001 to dbproxy1014

https://gerrit.wikimedia.org/r/570580

Change 570581 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] dbproxy1012: Enable notifications

https://gerrit.wikimedia.org/r/570581

Change 570581 merged by Marostegui:
[operations/puppet@production] dbproxy1012: Enable notifications

https://gerrit.wikimedia.org/r/570581

Marostegui updated the task description. (Show Details)Thu, Feb 6, 8:34 AM

Change 570580 merged by Marostegui:
[operations/dns@master] wmnet: Failover dbproxy1001 to dbproxy1014

https://gerrit.wikimedia.org/r/570580

Change 571418 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/dns@master] wmnet: Failover m1-master to dbproxy1012

https://gerrit.wikimedia.org/r/571418

Change 571418 merged by Marostegui:
[operations/dns@master] wmnet: Failover m1-master to dbproxy1012

https://gerrit.wikimedia.org/r/571418

Mentioned in SAL (#wikimedia-operations) [2020-02-11T06:18:10Z] <marostegui> Failover m1-master from dbproxy1014 to dbproxy1012 - T202367

Change 571659 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] site.pp: Productionize dbproxy1015

https://gerrit.wikimedia.org/r/571659

Change 571659 merged by Marostegui:
[operations/puppet@production] site.pp: Productionize dbproxy1015

https://gerrit.wikimedia.org/r/571659

Mentioned in SAL (#wikimedia-operations) [2020-02-12T09:11:31Z] <marostegui> Upgrade and reboot dbproxy1013 before making it master - T202367

Change 571679 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/dns@master] wmnet: Promote dbproxy1013 as m2-master

https://gerrit.wikimedia.org/r/571679

Change 571679 merged by Marostegui:
[operations/dns@master] wmnet: Promote dbproxy1013 as m2-master

https://gerrit.wikimedia.org/r/571679

Mentioned in SAL (#wikimedia-operations) [2020-02-12T09:17:57Z] <marostegui> Failover m2 master dbproxy from dbproxy1007 to dbproxy1013 - T202367

Change 571920 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] dbproxy1015: Enable notifications

https://gerrit.wikimedia.org/r/571920

Change 571920 merged by Marostegui:
[operations/puppet@production] dbproxy1015: Enable notifications

https://gerrit.wikimedia.org/r/571920

Marostegui moved this task from Next to In progress on the DBA board.Thu, Feb 13, 9:27 AM

Change 572512 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/dns@master] wmnet: Promote dbproxy1015 to m2-master

https://gerrit.wikimedia.org/r/572512

Change 572512 merged by Marostegui:
[operations/dns@master] wmnet: Promote dbproxy1015 to m2-master

https://gerrit.wikimedia.org/r/572512

Marostegui updated the task description. (Show Details)Mon, Feb 17, 9:29 AM