We are setting up a new database in CloudVPS to replace labsdb1004/labsdb1005. In order to accomplish the replacement, we need to set up temporal MySQL replication between the old and new servers. In this case temporal means until we drop/decom the hardware.
Involved addresses are:
- labsdb1004.eqiad.wnet IP addr: 10.64.37.8
- labsdb1005.eqiad.wmnet IP addr: 10.64.37.9
- clouddb1001.eqiad.wmflabs IP addr: 172.16.7.153 (and also
185.15.56.54, but we would like to avoid using this addr, was only set up for testing stuff) - clouddb1004.eqiad.wmflabs IP addr: 172.16.7.154
We need to establish the following connections, from labsdb to the VMs using MYSQL TCP port:
VMs --> hardware (3306/tcp) hardware --> VMs (3306/tcp)
According to my tests, the first case VM --> hadrware (3306/tcp) is already working:
aborrero@clouddb1001:~$ telnet labsdb1005.eqiad.wmnet 3306 Trying 10.64.37.9... Connected to labsdb1005.eqiad.wmnet. Escape character is '^]'. Y 5.5.5-10.0.34-MariaDB�5An'v=m/A��?�98Oc$'9'UZAwmysql_native_password^[[A^CConnection closed by foreign host.
However, the case hardware --> VMs (3306/tcp) is not working:
aborrero@labsdb1005:~$ telnet 172.16.7.153 3306 Trying 172.16.7.153... telnet: Unable to connect to remote host: Connection timed out
I could further confirm this by running tcpdump in our neutron virtual router, and see packets flowing but being dropped elsewhere:
aborrero@cloudnet1004:~$ sudo tcpdump -i eno50 host 172.16.7.153 and tcp port 3306 12:06:53.595239 IP labsdb1005.eqiad.wmnet.43366 > 172.16.7.153.mysql: Flags [S], seq 3383912041, win 29200, options [mss 1460,sackOK,TS val 88585927 ecr 0,nop,wscale 9], length 0 12:06:53.600242 IP 172.16.7.153.mysql > labsdb1005.eqiad.wmnet.43366: Flags [S.], seq 1656666748, ack 3383912042, win 28960, options [mss 1460,sackOK,TS val 17160861 ecr 88585927,nop,wscale 9], length 0
I've read the config in cr2-eqiad (https://librenms.wikimedia.org/device/device=2/tab=showconfig/) and I think adding this to filter cloud-in4 would solve our problems in the hardware --> VMs (3306/tcp) case:
term labsdb_inverse {
from {
destination-address {
/* labsdb1004, labsdb1005 */
10.64.37.8/31;
/* labsdb1006 */
10.64.37.11/32;
/* labsdb1007 */
10.64.37.12/32;
/* dbproxy1010, dbproxy1011 */
10.64.37.14/31;
}
protocol tcp;
source-port [ 3306 ];
}
then accept;
}since we already have this for the other case VMs --> hardware (3306/tcp):
/*
** labsdb100[4567] are being decommed - T193264
*/
term labsdb {
from {
destination-address {
/* labsdb1004, labsdb1005 */
10.64.37.8/31;
/* labsdb1006 */
10.64.37.11/32;
/* labsdb1007 */
10.64.37.12/32;
/* dbproxy1010, dbproxy1011 */
10.64.37.14/31;
}
protocol tcp;
destination-port [ 873 3306 5432 22 ];
}
then accept;
}Not sure if the two can be merged into one single term, I'm not familiar with the firewall expression logic.
We may need to create additional VMs, but since their addr is not in the filters, we should be fine.