Page MenuHomePhabricator

Productionize dbproxy101[2-7].eqiad.wmnet and dbproxy200[1-4]
Open, NormalPublic

Description

These new proxies are ready to be productionized

eqiad:

  • dbproxy1012 rack: A5 will replace dbproxy1001
  • dbproxy1013 rack: A6 will replace dbproxy1002 (non primary)
  • dbproxy1014 rack: B1 will replace dbproxy1006 (non primary)
  • dbproxy1015 rack: B8 will replace dbproxy1007
  • dbproxy1016 rack: D1 will replace dbproxy1003 (non primary)
  • dbproxy1017 rack: D3 will replace dbproxy1005 in m5 (not in use at the moment)
  • dbproxy1018 (cloud VLAN) rack: C5 (will replace dbproxy1010)
  • dbproxy1019 (cloud VLAN) rack: C5 (will replace dbproxy1011)
  • dbproxy1020 rack: C5 will replace dbproxy1008
  • dbproxy1021 rack: C8 will go to m5 to replace dbproxy1005 (currently only has one proxy - and not in use)

codfw:

  • dbproxy2001 - m1
  • dbproxy2002 - m2
  • dbproxy2003 - m3
  • dbproxy2004- spare

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Change 529847 merged by Marostegui:
[operations/dns@master] wmnet: Point m3-master codfw to dbproxy2003

https://gerrit.wikimedia.org/r/529847

Marostegui updated the task description. (Show Details)Aug 14 2019, 6:02 AM

Change 530025 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] dbproxy2003: Enable notifications

https://gerrit.wikimedia.org/r/530025

Change 530025 merged by Marostegui:
[operations/puppet@production] dbproxy2003: Enable notifications

https://gerrit.wikimedia.org/r/530025

Change 531598 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] dbproxy1019: Provision dbproxy1019 to replace dbproxy1011

https://gerrit.wikimedia.org/r/531598

Mentioned in SAL (#wikimedia-operations) [2019-08-22T07:46:47Z] <marostegui> Deploy grants on labsdb1009-labsdb1012 to allow connections for haproxy from dbproxy1019 - T202367

Change 531598 merged by Marostegui:
[operations/puppet@production] dbproxy1019: Provision dbproxy1019 to replace dbproxy1011

https://gerrit.wikimedia.org/r/531598

Change 531660 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] install_server: Allow re-image dbproxy1018,dbproxy1019

https://gerrit.wikimedia.org/r/531660

Change 531660 merged by Marostegui:
[operations/puppet@production] install_server: Allow re-image dbproxy1018,dbproxy1019

https://gerrit.wikimedia.org/r/531660

Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:

dbproxy1019.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/201908220848_marostegui_249574_dbproxy1019_eqiad_wmnet.log.

Completed auto-reimage of hosts:

['dbproxy1019.eqiad.wmnet']

and were ALL successful.

dbproxy1019 is ready to take over dbproxy1011

root@dbproxy1019:~# echo "show stat" | socat /run/haproxy/haproxy.sock stdio
# pxname,svname,qcur,qmax,scur,smax,slim,stot,bin,bout,dreq,dresp,ereq,econ,eresp,wretr,wredis,status,weight,act,bck,chkfail,chkdown,lastchg,downtime,qlimit,pid,iid,sid,throttle,lbtot,tracked,type,rate,rate_lim,rate_max,check_status,check_code,check_duration,hrsp_1xx,hrsp_2xx,hrsp_3xx,hrsp_4xx,hrsp_5xx,hrsp_other,hanafail,req_rate,req_rate_max,req_tot,cli_abrt,srv_abrt,comp_in,comp_out,comp_byp,comp_rsp,lastsess,last_chk,last_agt,qtime,ctime,rtime,ttime,agent_status,agent_code,agent_duration,check_desc,agent_desc,check_rise,check_fall,check_health,agent_rise,agent_fall,agent_health,addr,cookie,mode,algo,conn_rate,conn_rate_max,conn_tot,intercepted,dcon,dses,
mariadb,FRONTEND,,,0,1,5000,1,189,171,0,0,0,,,,,OPEN,,,,,,,,,1,2,0,,,,0,0,0,1,,,,,,,,,,,0,0,0,,,0,0,0,0,,,,,,,,,,,,,,,,,,,,,tcp,,0,1,1,,0,0,
mariadb,labsdb1009,0,0,0,1,,1,189,171,,0,,0,0,0,0,UP,1,1,0,0,0,447,0,,1,2,1,,1,,2,0,,1,L7OK,0,0,,,,,,,,,,,0,0,,,,,381,5.5.5-10.1.39-MariaDB,,0,0,0,1,,,,Layer7 check passed,,99999999,20,100000018,,,,,,tcp,,,,,,,,
mariadb,labsdb1010,0,0,0,0,,0,0,0,,0,,0,0,0,0,UP,1,1,0,0,0,447,0,,1,2,2,,0,,2,0,,0,L7OK,0,0,,,,,,,,,,,0,0,,,,,-1,5.5.5-10.1.39-MariaDB,,0,0,0,0,,,,Layer7 check passed,,99999999,20,100000018,,,,,,tcp,,,,,,,,
mariadb,BACKEND,0,0,0,1,500,1,189,171,0,0,,0,0,0,0,UP,2,2,0,,0,447,0,,1,2,0,,1,,1,0,,1,,,,,,,,,,,,,,0,0,0,0,0,0,381,,,0,0,0,1,,,,,,,,,,,,,,tcp,,,,,,,,
root@cumin1001:~# telnet dbproxy1019.eqiad.wmnet 3306
Trying 10.64.37.28...
Connected to dbproxy1019.eqiad.wmnet.
Escape character is '^]'.
Y
5.5.5-10.1.39-MariaDBv��S#RU5IM'�??�yO_o/=$:7D0bmysql_native_passwordConnection closed by foreign host.

From tools bastion I cannot connect, so I am going to open a ticket with netops to check the ACLs.

marostegui@tools-sgebastion-07:~$ telnet dbproxy1019.eqiad.wmnet 3306
Trying 10.64.37.28...
^C
marostegui@tools-sgebastion-07:~$ telnet dbproxy1011.eqiad.wmnet 3306
Trying 10.64.37.15...
Connected to dbproxy1011.eqiad.wmnet.
Escape character is '^]'.
Y
5.5.5-10.1.39-MariaDB��zyyrT*t.#�??�dPDB_611@0nYmysql_native_passwordConnection closed by foreign host.

Change 532562 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Productionize dbproxy1016

https://gerrit.wikimedia.org/r/532562

Mentioned in SAL (#wikimedia-operations) [2019-08-27T09:39:02Z] <marostegui> Deploy grants for dbproxy1016 on m3 - T202367

Change 532562 merged by Marostegui:
[operations/puppet@production] mariadb: Productionize dbproxy1016

https://gerrit.wikimedia.org/r/532562

dbproxy1016 is ready to replace dbproxy1003 on m3

# mysql --skip-ssl -hdbproxy1016
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 192195170
Server version: 10.1.39-MariaDB MariaDB Server

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

root@dbproxy1016[(none)]> select @@hostname;
+------------+
| @@hostname |
+------------+
| db1128     |
+------------+
1 row in set (0.00 sec)

root@dbproxy1016[(none)]>
Marostegui updated the task description. (Show Details)Aug 27 2019, 10:02 AM

Change 532692 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] dbproxy1019: Enable notifications

https://gerrit.wikimedia.org/r/532692

Change 532692 merged by Marostegui:
[operations/puppet@production] dbproxy1019: Enable notifications

https://gerrit.wikimedia.org/r/532692

Marostegui updated the task description. (Show Details)Aug 27 2019, 11:20 AM

dbproxy1019 is ready to replace dbproxy1011

root@dbproxy1019:~# echo "show stat" | socat /run/haproxy/haproxy.sock stdio
# pxname,svname,qcur,qmax,scur,smax,slim,stot,bin,bout,dreq,dresp,ereq,econ,eresp,wretr,wredis,status,weight,act,bck,chkfail,chkdown,lastchg,downtime,qlimit,pid,iid,sid,throttle,lbtot,tracked,type,rate,rate_lim,rate_max,check_status,check_code,check_duration,hrsp_1xx,hrsp_2xx,hrsp_3xx,hrsp_4xx,hrsp_5xx,hrsp_other,hanafail,req_rate,req_rate_max,req_tot,cli_abrt,srv_abrt,comp_in,comp_out,comp_byp,comp_rsp,lastsess,last_chk,last_agt,qtime,ctime,rtime,ttime,agent_status,agent_code,agent_duration,check_desc,agent_desc,check_rise,check_fall,check_health,agent_rise,agent_fall,agent_health,addr,cookie,mode,algo,conn_rate,conn_rate_max,conn_tot,intercepted,dcon,dses,
mariadb,FRONTEND,,,0,1,5000,6,204,747,0,0,0,,,,,OPEN,,,,,,,,,1,2,0,,,,0,0,0,1,,,,,,,,,,,0,0,0,,,0,0,0,0,,,,,,,,,,,,,,,,,,,,,tcp,,0,1,6,,0,0,
mariadb,labsdb1009,0,0,0,1,,3,199,431,,0,,0,2,0,0,UP,1,1,0,0,0,439868,0,,1,2,1,,3,,2,0,,1,L7OK,0,0,,,,,,,,,,,0,2,,,,,21337,5.5.5-10.1.39-MariaDB,,0,0,0,7,,,,Layer7 check passed,,99999999,20,100000018,,,,,,tcp,,,,,,,,
mariadb,labsdb1010,0,0,0,1,,3,5,316,,0,,0,1,0,0,UP,1,1,0,0,0,439868,0,,1,2,2,,3,,2,0,,1,L7OK,0,0,,,,,,,,,,,0,1,,,,,20395,5.5.5-10.1.39-MariaDB,,0,0,0,23,,,,Layer7 check passed,,99999999,20,100000018,,,,,,tcp,,,,,,,,
mariadb,BACKEND,0,0,0,1,500,6,204,747,0,0,,0,3,0,0,UP,2,2,0,,0,439868,0,,1,2,0,,6,,1,0,,1,,,,,,,,,,,,,,0,3,0,0,0,0,20395,,,0,0,0,30,,,,,,,,,,,,,,tcp,,,,,,,,

I can connect fine from tools bastion using the haproxy user to dbproxy1019:

MariaDB [(none)]> select @@hostname;
+------------+
| @@hostname |
+------------+
| labsdb1010 |
+------------+
1 row in set (0.01 sec)

Change 532867 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] dbproxy1018: Productionize dbproxy1018, will replace dbproxy1010

https://gerrit.wikimedia.org/r/532867

Change 532867 merged by Marostegui:
[operations/puppet@production] dbproxy1018: Productionize dbproxy1018, will replace dbproxy1010

https://gerrit.wikimedia.org/r/532867

Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:

dbproxy1018.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/201908280754_marostegui_100781_dbproxy1018_eqiad_wmnet.log.

Completed auto-reimage of hosts:

['dbproxy1018.eqiad.wmnet']

and were ALL successful.

Mentioned in SAL (#wikimedia-operations) [2019-08-28T08:17:35Z] <marostegui> Deploy grants on labsdb hosts for dbproxy1018 - T202367

dbproxy1018 is now monitoring the hosts:

root@dbproxy1018:~# echo "show stat" | socat /run/haproxy/haproxy.sock stdio
# pxname,svname,qcur,qmax,scur,smax,slim,stot,bin,bout,dreq,dresp,ereq,econ,eresp,wretr,wredis,status,weight,act,bck,chkfail,chkdown,lastchg,downtime,qlimit,pid,iid,sid,throttle,lbtot,tracked,type,rate,rate_lim,rate_max,check_status,check_code,check_duration,hrsp_1xx,hrsp_2xx,hrsp_3xx,hrsp_4xx,hrsp_5xx,hrsp_other,hanafail,req_rate,req_rate_max,req_tot,cli_abrt,srv_abrt,comp_in,comp_out,comp_byp,comp_rsp,lastsess,last_chk,last_agt,qtime,ctime,rtime,ttime,agent_status,agent_code,agent_duration,check_desc,agent_desc,check_rise,check_fall,check_health,agent_rise,agent_fall,agent_health,addr,cookie,mode,algo,conn_rate,conn_rate_max,conn_tot,intercepted,dcon,dses,
mariadb,FRONTEND,,,0,0,5000,0,0,0,0,0,0,,,,,OPEN,,,,,,,,,1,2,0,,,,0,0,0,0,,,,,,,,,,,0,0,0,,,0,0,0,0,,,,,,,,,,,,,,,,,,,,,tcp,,0,0,0,,0,0,
mariadb,labsdb1011,0,0,0,0,,0,0,0,,0,,0,0,0,0,UP,1,1,0,0,0,3,0,,1,2,1,,0,,2,0,,0,L7OK,0,5,,,,,,,,,,,0,0,,,,,-1,5.5.5-10.1.39-MariaDB,,0,0,0,0,,,,Layer7 check passed,,99999999,20,100000018,,,,,,tcp,,,,,,,,
mariadb,labsdb1010,0,0,0,0,,0,0,0,,0,,0,0,0,0,UP,1,0,1,0,0,3,0,,1,2,2,,0,,2,0,,0,L7OK,0,2,,,,,,,,,,,0,0,,,,,-1,5.5.5-10.1.39-MariaDB,,0,0,0,0,,,,Layer7 check passed,,2,3,4,,,,,,tcp,,,,,,,,
mariadb,BACKEND,0,0,0,0,500,0,0,0,0,0,,0,0,0,0,UP,1,1,1,,0,3,0,,1,2,0,,0,,1,0,,0,,,,,,,,,,,,,,0,0,0,0,0,0,-1,,,0,0,0,0,,,,,,,,,,,,,,tcp,,,,,,,,

Waiting for T231418: Review switches ACL to connect from tools-bastion to dbproxy1018 to be complete to consider this host ready to replace dbproxy1010

Marostegui updated the task description. (Show Details)Aug 28 2019, 12:13 PM

Change 533144 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] dbproxy1018: Enable notifications

https://gerrit.wikimedia.org/r/533144

Change 533144 merged by Marostegui:
[operations/puppet@production] dbproxy1018: Enable notifications

https://gerrit.wikimedia.org/r/533144

Marostegui updated the task description. (Show Details)Aug 29 2019, 5:34 AM

Change 533171 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] install_server: Do not reimage dbproxy1018,dbproxy1019

https://gerrit.wikimedia.org/r/533171

Change 533171 merged by Marostegui:
[operations/puppet@production] install_server: Do not reimage dbproxy1018,dbproxy1019

https://gerrit.wikimedia.org/r/533171

Change 533371 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] dbproxy1016: Enable notifications

https://gerrit.wikimedia.org/r/533371

Change 533371 merged by Marostegui:
[operations/puppet@production] dbproxy1016: Enable notifications

https://gerrit.wikimedia.org/r/533371

Change 534270 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Provision dbproxy1017 to replace dbproxy1005

https://gerrit.wikimedia.org/r/534270

Change 534270 merged by Marostegui:
[operations/puppet@production] mariadb: Provision dbproxy1017 to replace dbproxy1005

https://gerrit.wikimedia.org/r/534270

Change 534272 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] dbproxy1017: Allow reimage

https://gerrit.wikimedia.org/r/534272

Change 534272 merged by Marostegui:
[operations/puppet@production] dbproxy1017: Allow reimage

https://gerrit.wikimedia.org/r/534272

Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:

dbproxy1017.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/201909040539_marostegui_173520_dbproxy1017_eqiad_wmnet.log.

Completed auto-reimage of hosts:

['dbproxy1017.eqiad.wmnet']

and were ALL successful.

Change 534292 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] dbproxy1017: Enable notifications

https://gerrit.wikimedia.org/r/534292

Change 534292 merged by Marostegui:
[operations/puppet@production] dbproxy1017: Enable notifications

https://gerrit.wikimedia.org/r/534292

Change 534297 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] dbproxy1017: Clarify that it belongs to m5

https://gerrit.wikimedia.org/r/534297

Change 534297 merged by Marostegui:
[operations/puppet@production] dbproxy1017: Clarify that it belongs to m5

https://gerrit.wikimedia.org/r/534297

dbproxy1017 is now ready to replace dbproxy1005 (even though they are not in use at the moment)

#  mysql --skip-ssl -hdbproxy1017 -e "select @@hostname"
+------------+
| @@hostname |
+------------+
| db1133     |
+------------+
Marostegui updated the task description. (Show Details)Sep 5 2019, 6:13 AM
Marostegui updated the task description. (Show Details)Sep 16 2019, 7:51 AM

Change 536812 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Productionize dbproxy1021

https://gerrit.wikimedia.org/r/536812

Change 536812 merged by Marostegui:
[operations/puppet@production] mariadb: Productionize dbproxy1021

https://gerrit.wikimedia.org/r/536812

Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:

['dbproxy1021.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201909160758_marostegui_74242.log.

Completed auto-reimage of hosts:

['dbproxy1021.eqiad.wmnet']

and were ALL successful.

Change 536961 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] production-m5.sql.erb: Add dbproxy1021 grants

https://gerrit.wikimedia.org/r/536961

Mentioned in SAL (#wikimedia-operations) [2019-09-16T08:50:08Z] <marostegui> Apply grants for dbproxy1021 on db1133 (m5 master) with replication - T202367

Change 536961 merged by Marostegui:
[operations/puppet@production] production-m5.sql.erb: Add dbproxy1021 grants

https://gerrit.wikimedia.org/r/536961

Change 537085 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] dbproxy1021: Enable notifications

https://gerrit.wikimedia.org/r/537085

Change 537085 merged by Marostegui:
[operations/puppet@production] dbproxy1021: Enable notifications

https://gerrit.wikimedia.org/r/537085

Marostegui updated the task description. (Show Details)Sep 16 2019, 11:57 AM

dbproxy1021 has been placed in m5:

root@cumin1001:~# mysql --skip-ssl -h dbproxy1017 -e "select @@hostname"
+------------+
| @@hostname |
+------------+
| db1133     |
+------------+

Change 537617 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/dns@master] wmnet: Point m1-master to dbproxy1014

https://gerrit.wikimedia.org/r/537617

Change 537617 merged by Marostegui:
[operations/dns@master] wmnet: Point m1-master to dbproxy1014

https://gerrit.wikimedia.org/r/537617

dbproxy1014 has been tested and it is now m1-master. In a couple of days I will revert this change as dbproxy1014 is in a rack that requires maintenance to the PDU, so I will move back m1-master to dbproxy1001 until that has passed.
Given that dbproxy1014 works fine, dbproxy1006 can go away.

dbproxy1014 has been tested and it is now m1-master. In a couple of days I will revert this change as dbproxy1014 is in a rack that requires maintenance to the PDU, so I will move back m1-master to dbproxy1001 until that has passed.
Given that dbproxy1014 works fine, dbproxy1006 can go away.

Change reverted, and dbproxy1001 back as active proxy for m1 after testing that dbproxy1014 works fine.

Change 538042 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] bacula: Make bacula db parameters configurable on hiera

https://gerrit.wikimedia.org/r/538042

Change 538042 merged by Jcrespo:
[operations/puppet@production] bacula: Make bacula db parameters configurable on hiera

https://gerrit.wikimedia.org/r/538042