clouddb1026 is a new host that was just put into service.
The cloud load balancers at cloudlb100X cannot talk to it:
root@cloudlb1001:~# nc clouddb1013.eqiad.wmnet 3311 Z 5.5.5-10.11.16-MariaDBn?NLokN"\��?��*[k,7oI]zm7emysql_native_password ^C root@cloudlb1001:~# nc clouddb1026.eqiad.wmnet 3311 [HANGS] ^C
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T196020 Consider adding ContentTranslation (CX) tables to wiki replicas | |||
| Open | Feature | None | T395072 Add "wikishared" database to wiki replicas | ||
| Open | None | T428047 Upgrade clouddb* hosts to Debian Trixie | |||
| Unknown Object (Task) | |||||
| Resolved | Jclark-ctr | T409162 Q2:rack/setup/install clouddb1026-1033 | |||
| Stalled | None | T387419 Create wiki replicas views for globaljsonlinks tables | |||
| Stalled | None | T395881 Set up x1 replication to Wiki Replicas | |||
| Duplicate | Marostegui | T408692 Set up replication on new hosts clouddb102[2-5] | |||
| Open | Marostegui | T434048 Decommission clouddb1013-clouddb1020 | |||
| Resolved | Marostegui | T409557 Productionize new clouddb* hosts (clouddb1022-1033) | |||
| Resolved | cmooney | T430132 cloudlb100X cannot talk to new host clouddb1026 |
For what is worth a host which is in the same subnet also works:
root@cloudlb1001:~# telnet clouddb1015.eqiad.wmnet 3316 Trying 10.64.16.141... Connected to clouddb1015.eqiad.wmnet. Escape character is '^]'. Z
root@cloudlb1001:~# host clouddb1026; host clouddb1015 ; host clouddb1013 clouddb1026.eqiad.wmnet has address 10.64.16.21 clouddb1015.eqiad.wmnet has address 10.64.16.141 clouddb1013.eqiad.wmnet has address 10.64.0.28
The host is accessible from cumin1003 with no problem
[11:22:07] marostegui@cumin1003:~$ telnet clouddb1026 3311 Trying 10.64.16.21... Connected to clouddb1026. Escape character is '^]'. Z
I've quickly checked firewall and dns and they seem fine (no IPv6 AAAA record on the DB, cloudlb1001 listed in iptables for port 3311):
0 0 ACCEPT tcp -- * * 10.64.151.2 0.0.0.0/0 tcp dpt:3311
So, the packets are getting out of cloudlb1001 but not reaching clouddb1026:
$ sudo tcpdump -ni any host 10.64.16.21 and tcp port 3311 tcpdump: WARNING: any: That device doesn't support promiscuous mode (Promiscuous mode not supported on the "any" device) tcpdump: verbose output suppressed, use -v[v]... for full protocol decode listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes 15:51:53.891029 ens2f0np0 Out IP 10.64.151.2.37218 > 10.64.16.21.3311: Flags [S], seq 2394858215, win 35840, options [mss 8960,sackOK,TS val 684589864 ecr 0,nop,wscale 9], length 0 15:51:54.911012 ens2f0np0 Out IP 10.64.151.2.37218 > 10.64.16.21.3311: Flags [S], seq 2394858215, win 35840, options [mss 8960,sackOK,TS val 684590884 ecr 0,nop,wscale 9], length 0 15:51:55.939003 ens2f0np0 Out IP 10.64.151.2.37218 > 10.64.16.21.3311: Flags [S], seq 2394858215, win 35840, options [mss 8960,sackOK,TS val 684591912 ecr 0,nop,wscale 9], length 0 15:51:56.959063 ens2f0np0 Out IP 10.64.151.2.37218 > 10.64.16.21.3311: Flags [S], seq 2394858215, win 35840, options [mss 8960,sackOK,TS val 684592932 ecr 0,nop,wscale 9], length 0 15:51:57.983015 ens2f0np0 Out IP 10.64.151.2.37218 > 10.64.16.21.3311: Flags [S], seq 2394858215, win 35840, options [mss 8960,sackOK,TS val 684593956 ecr 0,nop,wscale 9], length 0 ^C 5 packets captured 7 packets received by filter 0 packets dropped by kernel
and the same tcpdump inverted on clouddb1026 sudo tcpdump -ni any host 10.64.151.2 and tcp port 3311 is empty.
Checking the switch.
Folks,
The packets here were getting blocked on the CR router as traffic left the cloud realm going to the new clouddb host.
Problem there is the access list is built from the hosts in Netbox - and needed a manual trigger of the Capirca script to pick up the new clouddb IPs. I have run that and run Homer against the CRs in eqiad and codfw, the connectivity now works:
cmooney@cloudlb1001:~$ telnet clouddb1026 3311 Trying 10.64.16.21... Connected to clouddb1026. Escape character is '^]'. Z 5.5.5-10.11.18-MariaDB`�X6C/_sFB�YxpIYdlqrh3amysql_native_passwordConnection closed by foreign host.
This situation isn't ideal, but one to be aware of for traffic that goes cross-realm. When we add new hosts we need to do this - or please ping us (also see T361549).
@cmooney thanks a lot for the fix.
All the new hosts are from cloudb1026 to clouddb1033.
Thanks @cmooney I am closing this as fixed then. We will reopen if needed.
Thanks for the help!
Confirm I can see users connecting too:
| 212939 | s51187 | 10.64.151.2:37508 | NULL | Query | 0 | Sending data | SET STATEMENT max_statement_time = 60 FOR SELECT *, ( SELECT user_editcount FROM `enwiki_p