Page MenuHomePhabricator

possible routing issue between eqiad and Maxmind network
Closed, ResolvedPublic

Description

We're seeing severe packet loss between the frack payments servers and API servers on Maxmind's network. I tested also from bast1002 since that's a simpler route.

bast1002 (0.0.0.0)                                                                                Tue Sep 24 01:04:55 2019
Resolver: Received error response 2. (server failure)er of fields   quit
                                                                                  Packets               Pings
 Host                                                                           Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. ae3-1003.cr1-eqiad.wikimedia.org               0.0%   196    0.2   0.6   0.2  14.0   1.6
 2. ae0.cr2-eqiad.wikimedia.org                           0.0%   196    0.2   0.4   0.2   7.1   0.6
 3. ???
 4. 104.16.38.47                                                      96.9%   195    0.5   0.5   0.5   0.7   0.0

Note that routing is clean between codfw-Maxmind and also from my home network to Maxmind. I checked https://status.maxmind.com.and they report all systems operational. I also checked other IPs on their networks with the same results. Is this a peering issue or something else at our end?

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Jgreen triaged this task as Unbreak Now! priority.Sep 24 2019, 1:12 AM

Flipping this to "Unbreak Now!" since it's a timely issue, and service outage interfering with the donation pipeline. We do have some donation activity at the moment.

ayounsi updated the task description. (Show Details)
ayounsi subscribed.

All those IPs are behind Cloudflare. Opened a ticket with them.

Resolved by Cloudflare.