Page MenuHomePhabricator
Paste P21629

No v4 routing from Liberty Global (Ireland) to ulsfo
ActivePublic

Authored by cmooney on Mar 1 2022, 11:30 PM.
cathal@nbgw:~$ mtr -4 -z -b -w -c 5 bast4003.wikimedia.org
Start: 2022-03-01T23:26:55+0000
HOST: nbgw Loss% Snt Last Avg Best Wrst StDev
1. AS6830 176.61.34.1 0.0% 5 11.6 11.9 8.8 16.9 3.0
2. AS??? ??? 100.0 5 0.0 0.0 0.0 0.0 0.0
cathal@nbgw:~$ mtr -z -b -w -c 5 bast4003.wikimedia.org
Start: 2022-03-01T23:26:38+0000
HOST: nbgw Loss% Snt Last Avg Best Wrst StDev
1. AS6939 tunnel650354.tunnel.tserv5.lon1.ipv6.he.net (2001:470:1f08:32c::1) 0.0% 5 21.2 23.2 21.2 26.8 2.3
2. AS6939 e0-19.core2.lon2.he.net (2001:470:0:67::1) 0.0% 5 23.0 24.7 23.0 25.7 1.1
3. AS6939 100ge11-2.core1.lon2.he.net (2001:470:0:541::1) 0.0% 5 20.2 27.3 18.4 56.9 16.6
4. AS6939 100ge4-1.core1.nyc4.he.net (2001:470:0:2cf::2) 0.0% 5 86.8 102.0 86.8 149.4 26.6
5. AS6939 100ge8-1.core1.sjc2.he.net (2001:470:0:296::2) 0.0% 5 166.2 165.1 153.7 170.4 6.6
6. AS6939 100ge4-2.core3.fmt2.he.net (2001:470:0:eb::1) 0.0% 5 166.6 155.3 147.0 166.6 7.1
7. AS6939 100ge2-1.core2.fmt2.he.net (2001:470:0:1e7::2) 0.0% 5 189.6 178.3 150.0 208.8 25.5
8. AS??? as14907.sfmix.org (2001:504:30::ba01:4907:1) 0.0% 5 150.5 171.4 150.2 208.1 29.1
9. AS14907 bast4003.wikimedia.org (2620:0:863:1:198:35:26:13) 0.0% 5 149.5 148.2 145.6 151.1 2.2
cathal@nbgw:~$
cathal@nbgw:~$

Event Timeline

locally, Comcast, Scotts Valley (near San Jose) over to SF

 ...
  3. AS7922   po-301-1221-rur01.scotts.ca.sfba.comcast.net (2001:558:82:5805::1)   0.0%     5   11.3  12.5  11.2  13.7   1.2
  4. AS7922   be-221-rar01.santaclara.ca.sfba.comcast.net (2001:558:80:1b0::1)     0.0%     5   12.7  15.5  11.6  28.6   7.3
  5. AS7922   be-39911-cs01.sunnyvale.ca.ibone.comcast.net (2001:558:3:25c::1)    60.0%     5   17.3  14.8  12.3  17.3   3.5
  6. AS7922   be-1112-cr12.sunnyvale.ca.ibone.comcast.net (2001:558:3:381::2)     40.0%     5   12.4  12.8  12.4  13.6   0.7
  7. AS7922   be-304-cr12.9greatoaks.ca.ibone.comcast.net (2001:558:3:16d::2)     80.0%     5   15.6  15.6  15.6  15.6   0.0
  8. AS7922   be-1112-cs01.9greatoaks.ca.ibone.comcast.net (2001:558:3:421::1)    60.0%     5   15.1  14.8  14.4  15.1   0.5
  9. AS7922   be-2101-pe01.9greatoaks.ca.ibone.comcast.net (2001:558:3:136::2)    40.0%     5   14.3  14.8  14.3  15.2   0.5
 10. AS6939   10gigabitethernet13-17.core1.sjc2.he.net (2001:470:0:31f::1)        40.0%     5   18.9  18.5  14.3  22.3   4.0
 11. AS6939   100ge4-2.core3.fmt2.he.net (2001:470:0:eb::1)                        0.0%     5  161.1 127.3  45.9 164.9  48.9
 12. AS6939   100ge2-1.core2.fmt2.he.net (2001:470:0:1e7::2)                       0.0%     5   14.9  27.9  14.9  48.8  14.0
 13. AS???    as14907.sfmix.org (2001:504:30::ba01:4907:1)                         0.0%     5   16.0  16.2  16.0  16.5   0.2
 14. AS14907  bast4003.wikimedia.org (2620:0:863:1:198:35:26:13)                   0.0%     5   14.3  15.4  14.3  17.5   1.3


but with IPv4:
   ....
  2. AS7922   96.120.89.193                                                   0.0%     5   11.2  11.8  11.0  14.1   1.3
  3. AS7922   po-301-1222-rur02.scotts.ca.sfba.comcast.net (96.110.102.153)   0.0%     5   11.4  11.5  10.9  12.8   0.8
  4. AS7922   po-2-rur01.scotts.ca.sfba.comcast.net (68.85.154.137)           0.0%     5   11.5  11.4  10.7  12.8   0.8
  5. AS7922   be-221-rar01.santaclara.ca.sfba.comcast.net (69.139.199.205)    0.0%     5   31.6  16.7  11.9  31.6   8.4
  6. AS???    ???                                                            100.0     5    0.0   0.0   0.0   0.0   0.0

core1.den1.he.net> traceroute 198.35.26.192 source 216.218.252.157 numeric

Tracing the route to IP node (198.35.26.192) from 1 to 30 hops

	  1    <1 ms    1 ms    1 ms 184.105.80.150 
	  2    35 ms   33 ms   40 ms 184.105.64.69 
	  3    36 ms   34 ms   25 ms 184.104.192.213 
	  4    34 ms   25 ms   68 ms 72.52.92.245 
	  5   149 ms   57 ms   30 ms 72.52.92.30 
	  6    56 ms   26 ms   39 ms 198.35.26.192
  1. Entry cached for another 12 seconds.

core2.sfo1.he.net> traceroute 198.35.26.192 source 216.218.252.5

	  traceroute to 198.35.26.192 (198.35.26.192), 30 hops max, 60 byte packets
	
	 1  72.52.92.25 (72.52.92.25)  2.731 ms  2.794 ms *
	 2  * * *
	 3  * * *
	 4  * * *
	 5  * * *
	 6  * 198.35.26.192 (198.35.26.192)  3.263 ms  3.579 ms

core2.dub1.he.net> traceroute 198.35.26.192 source 216.218.252.47

	  traceroute to 198.35.26.192 (198.35.26.192), 30 hops max, 60 byte packets
	
	 1  184.105.65.245 (184.105.65.245)  61.048 ms  61.068 ms  61.460 ms
	 2  184.105.213.217 (184.105.213.217)  65.261 ms  65.344 ms  65.433 ms
	 3  184.105.81.218 (184.105.81.218)  125.236 ms  125.809 ms  125.934 ms
	 4  72.52.92.245 (72.52.92.245)  127.024 ms  127.117 ms  127.354 ms
	 5  72.52.92.30 (72.52.92.30)  147.040 ms  147.193 ms  147.333 ms
	 6  198.35.26.192 (198.35.26.192)  130.979 ms  126.166 ms  126.126 ms
mtr -4 -z -b -w -c 5 bast4003.wikimedia.org
        
Start: 2022-03-01T16:13:19-0800
HOST: muskrat                                                                Loss%   Snt   Last   Avg  Best  Wrst StDev
  1. AS7922   96.120.89.193                                                   0.0%     5   12.5  10.2   5.4  12.5   2.8
        _gateway (10.0.0.1)                                           
     AS???    _gateway (10.0.0.1)
  2. AS7922   96.120.89.193                                                   0.0%     5   12.6  11.6  10.4  13.0   1.1
  3. AS7922   po-301-1222-rur02.scotts.ca.sfba.comcast.net (96.110.102.153)   0.0%     5   12.9  12.0  10.7  13.6   1.2
  4. AS7922   po-2-rur01.scotts.ca.sfba.comcast.net (68.85.154.137)           0.0%     5   11.4  11.2  10.6  12.5   0.8
  5. AS7922   be-221-rar01.santaclara.ca.sfba.comcast.net (69.139.199.205)    0.0%     5   12.3  16.0  11.8  31.7   8.8
  6. AS???    ???                                                            100.0     5    0.0   0.0   0.0   0.0   0.0

Example of Lumen failing via IPv4 but working on IPv6:

cmooney@cr3-ulsfo> traceroute 2606:4700:4700::1111 source 2620:0:863:ffff::1 no-resolve wait 2 as-number-lookup bypass-routing next-hop 2001:1900:2100::a99                                                    
traceroute6 to 2606:4700:4700::1111 (2606:4700:4700::1111) from 2620:0:863:ffff::1, 64 hops max, 12 byte packets
 1  2001:1900:2100::a99 [AS  3356]  5.142 ms  4.831 ms  5.768 ms
 2  * * *
 3  2001:1900:2100::4d56 [AS  3356]  3.546 ms  5.413 ms  5.323 ms
 4  2400:cb00:465:3:: [AS  13335]  42.995 ms 2400:cb00:464:3:: [AS  13335]  5.353 ms 2400:cb00:4:3:: [AS  13335]  5.769 ms
 5  2400:cb00:464:1024::a29e:a53b [AS  13335]  5.026 ms 2400:cb00:4:1024::a29e:fd54 [AS  13335]  5.075 ms 2400:cb00:464:1024::a29e:a557 [AS  13335]  4.819 ms

cmooney@cr3-ulsfo> traceroute 1.1.1.1 source 198.35.26.192 no-resolve wait 1 as-number-lookup bypass-routing next-hop 4.15.72.113                  
traceroute to 1.1.1.1 (1.1.1.1) from 198.35.26.192, 30 hops max, 52 byte packets
 1  * * *
 2  * * *
 3  * * *
 4  * * *
 5  * * *
cmooney@cr3-ulsfo> traceroute 104.16.44.99 source 198.35.26.192 no-resolve wait 1 as-number-lookup bypass-routing next-hop 4.15.72.113 
traceroute to 104.16.44.99 (104.16.44.99) from 198.35.26.192, 30 hops max, 52 byte packets
 1  * * *
 2  * * *
 3  * * *
 4  * * *
 5  * * *

cmooney@cr3-ulsfo> traceroute 2606:4700::6810:2d63 source 2620:0:863:ffff::1 no-resolve wait 2 as-number-lookup bypass-routing next-hop 2001:1900:2100::a99   
traceroute6 to 2606:4700::6810:2d63 (2606:4700::6810:2d63) from 2620:0:863:ffff::1, 64 hops max, 12 byte packets
 1  2001:1900:2100::a99 [AS  3356]  9.837 ms  4.955 ms  5.240 ms
 2  * * *
 3  2001:1900:2100::4d56 [AS  3356]  12.440 ms  17.481 ms  7.118 ms
 4  2400:cb00:4:3:: [AS  13335]  10.021 ms  8.791 ms 2400:cb00:465:3:: [AS  13335]  7.017 ms
 5  2400:cb00:464:1024::a29e:a540 [AS  13335]  4.920 ms 2400:cb00:464:1024::a29e:a53a [AS  13335]  5.514 ms 2400:cb00:4:1024::a29e:fcba [AS  13335]  7.884 ms

Services have now been restored, IPv4 is working again via Level3:

cmooney@cr3-ulsfo> traceroute 104.16.44.99 source 198.35.26.192 no-resolve wait 1 as-number-lookup bypass-routing next-hop 4.15.72.113 
traceroute to 104.16.44.99 (104.16.44.99) from 198.35.26.192, 30 hops max, 52 byte packets
 1  4.15.72.113 [AS  3356]  4.363 ms  2.136 ms  1.963 ms
 2  4.69.211.221 [AS  3356]  2.143 ms  2.037 ms  2.056 ms
 3  4.7.18.106 [AS  3356]  3.194 ms  2.946 ms  3.092 ms
 4  172.68.188.20 [AS  13335]  59.220 ms  31.784 ms 162.158.164.2 [AS  13335]  3.040 ms
 5  104.16.44.99 [AS  13335]  3.865 ms  3.890 ms  2.682 ms
cmooney@cr3-ulsfo> traceroute 104.16.44.99 source 198.35.26.192 no-resolve wait 1 as-number-lookup bypass-routing next-hop 4.15.72.113 
traceroute to 104.16.44.99 (104.16.44.99) from 198.35.26.192, 30 hops max, 52 byte packets
 1  4.15.72.113 [AS  3356]  6.817 ms  2.958 ms  2.103 ms
 2  * * *
 3  4.7.18.106 [AS  3356]  3.747 ms  3.811 ms  3.972 ms
 4  172.68.188.20 [AS  13335]  8.749 ms 172.69.132.2 [AS  13335]  3.548 ms  2.467 ms
 5  104.16.44.99 [AS  13335]  2.720 ms  2.476 ms  2.396 ms