We are setting up a new database in CloudVPS to replace labsdb1004/labsdb1005. In order to accomplish the replacement, we need to set up temporal MySQL replication between the old and new servers. In this case temporal means until we drop/decom the hardware.
Involved addresses are:
* labsdb1004.eqiad.wnet IP addr: `10.64.37.8`
* labsdb1005.eqiad.wmnet IP addr: `10.64.37.9`
* clouddb1001.eqiad.wmflabs IP addr: `172.16.7.153` (and also ~~`185.15.56.54`~~, but we would like to avoid using this addr, was only set up for testing stuff)
* clouddb1004.eqiad.wmflabs IP addr: `172.16.7.154`
We need to establish the following connections, from labsdb to the VMs using MYSQL TCP port:
```
VMs --> hardware (3306/tcp)
hardware --> VMs (3306/tcp)
```
According to my tests, the first case `VM --> hadrware (3306/tcp)` is already working:
```
aborrero@clouddb1001:~$ telnet labsdb1005.eqiad.wmnet 3306
Trying 10.64.37.9...
Connected to labsdb1005.eqiad.wmnet.
Escape character is '^]'.
Y
5.5.5-10.0.34-MariaDB�5An'v=m/A��?�98Oc$'9'UZAwmysql_native_password^[[A^CConnection closed by foreign host.
```
However, the case `hardware --> VMs (3306/tcp)` is not working:
```
aborrero@labsdb1005:~$ telnet 172.16.7.153 3306
Trying 172.16.7.153...
telnet: Unable to connect to remote host: Connection timed out
```
I could further confirm this by running tcpdump in our neutron virtual router, and see packets flowing but being dropped elsewhere:
```
aborrero@cloudnet1004:~$ sudo tcpdump -i eno50 host 172.16.7.153 and tcp port 3306
12:06:53.595239 IP labsdb1005.eqiad.wmnet.43366 > 172.16.7.153.mysql: Flags [S], seq 3383912041, win 29200, options [mss 1460,sackOK,TS val 88585927 ecr 0,nop,wscale 9], length 0
12:06:53.600242 IP 172.16.7.153.mysql > labsdb1005.eqiad.wmnet.43366: Flags [S.], seq 1656666748, ack 3383912042, win 28960, options [mss 1460,sackOK,TS val 17160861 ecr 88585927,nop,wscale 9], length 0
```
I've read the config in `cr2-eqiad` (https://librenms.wikimedia.org/device/device=2/tab=showconfig/) and I think adding this to `filter cloud-in4` would solve our problems in the `hardware --> VMs (3306/tcp)` case:
```
term labsdb_inverse {
from {
destination-address {
/* labsdb1004, labsdb1005 */
10.64.37.8/31;
/* labsdb1006 */
10.64.37.11/32;
/* labsdb1007 */
10.64.37.12/32;
/* dbproxy1010, dbproxy1011 */
10.64.37.14/31;
}
protocol tcp;
source-port [ 3306 ];
}
then accept;
}
```
since we already have this for the other case `VMs --> hardware (3306/tcp)`:
```
/*
** labsdb100[4567] are being decommed - T193264
*/
term labsdb {
from {
destination-address {
/* labsdb1004, labsdb1005 */
10.64.37.8/31;
/* labsdb1006 */
10.64.37.11/32;
/* labsdb1007 */
10.64.37.12/32;
/* dbproxy1010, dbproxy1011 */
10.64.37.14/31;
}
protocol tcp;
destination-port [ 873 3306 5432 22 ];
}
then accept;
}
```
Not sure if the two can be merged into one single term, I'm not familiar with the firewall expression logic.
We may need to create additional VMs, but since their addr is not in the filters, we should be fine.
NOTE:
3 additional IPs were missed in this 10.64.37.18-20. However! While they were as critical as the rest of this, we have a way around it now, making this effectively a low priority ticket that we don't have to worry about for now.