Anyone else seeing IPv6 issues going through Level3?
I monitor over 30 IPv6 hosts, and starting around 1:54 am (Central) I couldn't access about a third of them. Here's the list of sites that I currently can't access: ipv6_akamai_com_29 (2001:418:2401:9::8367:8929) ipv6_akamai_com_40 (2001:418:2401:9::8367:8940_ ipv6_cnn_com ipv6_google_com ipv6_sprint_com www-v6_cisco_com www_charter_com www_facebook_com_b (2620:0:1c18:0:face:b00c:0:2) www_ipv6_cisco_com www_sprint_net www_sprintv6_net Here's a sample of traceroutes: nagios:/home/fbulk# tcptraceroute6 www.sprintv6.net traceroute to www.sprint.net (2600::) from 2607:fe28:0:1000::5, port 80, from port 58614, 30 hops max, 60 bytes packets 1 2607:fe28:0:1003::2 (2607:fe28:0:1003::2) 1.080 ms 1.550 ms 1.356 ms 2 router-core.mtcnet.net (2607:fe28:0:1000::1) 1.856 ms 1.738 ms 1.661 ms 3 sxct.movl.mtcnet.net (2607:fe28:11:1002::194) 2.064 ms 1.966 ms 1.311 ms 4 v6-siouxcenter.movl.153.netins.net (2001:5f8:7f06::9) 4.128 ms 3.417 ms 3.048 ms 5 v6-ins-kb1-te-13-2-220.kmrr.netins.net (2001:5f8:2:2::1) 9.347 ms 8.930 ms 8.606 ms 6 v6-ins-kc2-te-8-3.kmrr.netins.net (2001:5f8::11:1) 9.258 ms 8.897 ms 8.549 ms 7 v6-ins-dc1-et-8-1.desm.netins.net (2001:5f8::8:1) 10.902 ms 10.494 ms 10.268 ms 8 te-3-3.car2.KansasCity1.Level3.net (2001:1900:2100::e2d) 195.453 ms 232.440 ms 230.230 ms 9 vl-11.car1.KansasCity1.Level3.net (2001:1900:4:1::37d) 17.244 ms 16.887 ms 16.524 ms 10 vl-4049.car1.Dallas1.Level3.net (2001:1900:4:1::379) 26.470 ms 26.076 ms 25.764 ms 11 vl-70.edge2.Dallas3.Level3.net (2001:1900:17:2::e) 26.664 ms 26.417 ms 26.058 ms 12 * * * 13 * * * 14 * * * nagios:/home/fbulk# tcptraceroute6 www.sprint.net traceroute to www.sprint.net (2600::) from 2607:fe28:0:1000::5, port 80, from port 55242, 30 hops max, 60 bytes packets 1 2607:fe28:0:1003::2 (2607:fe28:0:1003::2) 1.072 ms 1.533 ms 1.486 ms 2 router-core.mtcnet.net (2607:fe28:0:1000::1) 1.847 ms 1.862 ms 1.698 ms 3 sxct.movl.mtcnet.net (2607:fe28:11:1002::194) 2.187 ms 1.957 ms 1.220 ms 4 v6-siouxcenter.movl.153.netins.net (2001:5f8:7f06::9) 4.114 ms 3.318 ms 3.055 ms 5 v6-ins-kb1-te-13-2-220.kmrr.netins.net (2001:5f8:2:2::1) 9.291 ms 8.873 ms 8.612 ms 6 v6-ins-kc2-te-8-3.kmrr.netins.net (2001:5f8::11:1) 9.252 ms 8.907 ms 8.610 ms 7 v6-ins-dc1-et-8-1.desm.netins.net (2001:5f8::8:1) 10.926 ms 10.539 ms 10.221 ms 8 te-3-3.car2.KansasCity1.Level3.net (2001:1900:2100::e2d) 17.221 ms 16.776 ms 16.511 ms 9 vl-11.car1.KansasCity1.Level3.net (2001:1900:4:1::37d) 17.159 ms 16.885 ms 16.518 ms 10 vl-4049.car1.Dallas1.Level3.net (2001:1900:4:1::379) 26.486 ms 26.074 ms 25.743 ms 11 vl-70.edge2.Dallas3.Level3.net (2001:1900:17:2::e) 26.751 ms 26.432 ms 26.132 ms 12 * * * 13 * * * 14 * * * nagios:/home/fbulk# tcptraceroute6 www.ipv6.cisco.com traceroute to www.ipv6.cisco.com (2001:420:80:1::5) from 2607:fe28:0:1000::5, port 80, from port 53137, 30 hops max, 60 bytes packets 1 2607:fe28:0:1003::2 (2607:fe28:0:1003::2) 1.131 ms 1.190 ms 1.849 ms 2 router-core.mtcnet.net (2607:fe28:0:1000::1) 1.628 ms 2.158 ms 1.554 ms 3 sxct.sxcy.mtcnet.net (2607:fe28:11:1002::197) 2.618 ms 1.810 ms 1.190 ms 4 v6-siouxcenter.sxcy.137.netins.net (2001:5f8:7f06::5) 3.621 ms 2.928 ms 2.638 ms 5 v6-ins-db1-te-13-2-219.desm.netins.net (2001:5f8:1:1::1) 7.297 ms 6.957 ms 6.652 ms 6 v6-ins-dc1-et-8-2.desm.netins.net (2001:5f8::1:1) 9.145 ms 8.752 ms 8.447 ms 7 te-3-3.car2.KansasCity1.Level3.net (2001:1900:2100::e2d) 15.418 ms 15.064 ms 14.727 ms 8 vl-11.car1.KansasCity1.Level3.net (2001:1900:4:1::37d) 15.492 ms 15.020 ms 14.721 ms 9 vl-4049.car1.Dallas1.Level3.net (2001:1900:4:1::379) 24.696 ms 24.282 ms 24.144 ms 10 vl-70.edge2.Dallas3.Level3.net (2001:1900:17:2::e) 24.738 ms 24.489 ms 24.148 ms 11 * * * 12 * * * 13 * * * (facebook) nagios:/home/fbulk# tcptraceroute6 2620:0:1c18:0:face:b00c:0:2 traceroute to 2620:0:1c18:0:face:b00c:0:2 (2620:0:1c18:0:face:b00c:0:2) from 2607:fe28:0:1000::5, port 80, from port 47528, 30 hops max, 60 bytes packets 1 2607:fe28:0:1003::2 (2607:fe28:0:1003::2) 1.066 ms 1.243 ms 1.960 ms 2 router-core.mtcnet.net (2607:fe28:0:1000::1) 1.626 ms 2.268 ms 1.307 ms 3 sxct.movl.mtcnet.net (2607:fe28:11:1002::194) 2.592 ms 1.874 ms 1.238 ms 4 v6-siouxcenter.movl.153.netins.net (2001:5f8:7f06::9) 4.018 ms 3.341 ms 2.994 ms 5 v6-ins-kb1-te-13-2-220.kmrr.netins.net (2001:5f8:2:2::1) 9.367 ms 8.873 ms 8.656 ms 6 v6-ins-kc2-te-8-3.kmrr.netins.net (2001:5f8::11:1) 9.285 ms 8.980 ms 8.529 ms 7 v6-ins-dc1-et-8-1.desm.netins.net (2001:5f8::8:1) 10.944 ms 10.493 ms 10.231 ms 8 te-3-3.car2.KansasCity1.Level3.net (2001:1900:2100::e2d) 17.559 ms 16.798 ms 16.514 ms 9 vl-11.car1.KansasCity1.Level3.net (2001:1900:4:1::37d) 200.575 ms 233.140 ms 224.720 ms 10 vl-4049.car1.Dallas1.Level3.net (2001:1900:4:1::379) 26.312 ms 26.100 ms 25.774 ms 11 vl-4080.car2.Dallas1.Level3.net (2001:1900:4:1::82) 111.124 ms 131.324 ms 226.032 ms 12 vl-4061.car1.Atlanta2.Level3.net (2001:1900:4:1::96) 46.276 ms 45.963 ms 45.564 ms 13 vl-51.edge5.Atlanta2.Level3.net (2001:1900:1c:1::f) 45.928 ms 45.579 ms 45.292 ms 14 * * * 15 * * * 16 * * * nagios:/home/fbulk# tcptraceroute6 www.charter.com traceroute to www.charter.com (2607:f428:3:1:80:80:80:1) from 2607:fe28:0:1000::5, port 80, from port 44115, 30 hops max, 60 bytes packets 1 2607:fe28:0:1003::2 (2607:fe28:0:1003::2) 1.108 ms 1.280 ms 1.852 ms 2 router-core.mtcnet.net (2607:fe28:0:1000::1) 1.584 ms 2.204 ms 1.805 ms 3 sxct.sxcy.mtcnet.net (2607:fe28:11:1002::197) 2.515 ms 2.218 ms 1.197 ms 4 v6-siouxcenter.sxcy.137.netins.net (2001:5f8:7f06::5) 4.017 ms 2.944 ms 2.744 ms 5 v6-ins-db1-te-13-2-219.desm.netins.net (2001:5f8:1:1::1) 7.451 ms 7.004 ms 6.764 ms 6 v6-ins-dc1-et-8-2.desm.netins.net (2001:5f8::1:1) 9.132 ms 8.831 ms 8.481 ms 7 te-3-3.car2.KansasCity1.Level3.net (2001:1900:2100::e2d) 184.676 ms 202.524 ms 203.065 ms 8 vl-11.car1.KansasCity1.Level3.net (2001:1900:4:1::37d) 15.518 ms 15.134 ms 14.832 ms 9 vl-4049.car1.Dallas1.Level3.net (2001:1900:4:1::379) 65.261 ms 207.228 ms 41.498 ms 10 vl-80.edge2.Dallas3.Level3.net (2001:1900:17:3::e) 24.999 ms 56.325 ms 30.592 ms 11 CenturyTel-level3-10.dallas3.Level3.net (2001:1900:4:3::1c2) 25.072 ms 24.780 ms 24.480 ms 12 2001:428::205:171:202:195 (2001:428::205:171:202:195) 37.480 ms 37.165 ms 36.814 ms 13 * * * 14 * * * 15 * * * nagios:/home/fbulk# tcptraceroute6 ipv6.google.com traceroute to gx-in-x68.1e100.net (2001:4860:8002::68) from 2607:fe28:0:1000::5, port 80, from port 42606, 30 hops max, 60 bytes packets 1 2607:fe28:0:1003::2 (2607:fe28:0:1003::2) 1.041 ms 1.482 ms 1.418 ms 2 router-core.mtcnet.net (2607:fe28:0:1000::1) 1.870 ms 1.729 ms 1.665 ms 3 sxct.sxcy.mtcnet.net (2607:fe28:11:1002::197) 2.041 ms 1.994 ms 1.187 ms 4 v6-siouxcenter.sxcy.137.netins.net (2001:5f8:7f06::5) 3.778 ms 2.955 ms 2.669 ms 5 v6-ins-db1-te-13-2-219.desm.netins.net (2001:5f8:1:1::1) 7.361 ms 7.053 ms 6.874 ms 6 v6-ins-dc1-et-8-2.desm.netins.net (2001:5f8::1:1) 9.146 ms 8.804 ms 8.407 ms 7 te-3-3.car2.KansasCity1.Level3.net (2001:1900:2100::e2d) 69.291 ms 39.541 ms 15.380 ms 8 vl-11.car1.KansasCity1.Level3.net (2001:1900:4:1::37d) 15.399 ms 15.015 ms 14.720 ms 9 vl-4049.car1.Dallas1.Level3.net (2001:1900:4:1::379) 24.589 ms 24.209 ms 23.975 ms 10 vl-90.edge2.Dallas3.Level3.net (2001:1900:17:4::e) 24.698 ms 24.368 ms 24.078 ms 11 2001:1900:4:3::256 (2001:1900:4:3::256) 24.889 ms 24.411 ms 24.131 ms 12 2001:4860::1:0:57f (2001:4860::1:0:57f) 25.081 ms 24.648 ms 24.285 ms 13 2001:4860::8:0:2c9c (2001:4860::8:0:2c9c) 25.154 ms 24.816 ms 24.506 ms 14 * * * 15 * * *
There are other sites going through Level3 that do work (I picked from a list that blipped at 5:05 am). Here are just two of them: tctnagios:/home/fbulk# tcptraceroute6 www.globalcrossing.com traceroute to www.globalcrossing.com (2001:450:2032:1::f500) from 2607:fe28:0:1000::5, port 80, from port 53367, 30 hops max, 60 bytes packets 1 2607:fe28:0:1003::2 (2607:fe28:0:1003::2) 1.049 ms 1.480 ms 1.370 ms 2 router-core.mtcnet.net (2607:fe28:0:1000::1) 1.796 ms 1.756 ms 1.737 ms 3 sxct.movl.mtcnet.net (2607:fe28:11:1002::194) 2.065 ms 1.998 ms 1.251 ms 4 v6-siouxcenter.movl.153.netins.net (2001:5f8:7f06::9) 4.169 ms 3.333 ms 2.973 ms 5 v6-ins-kb1-te-13-2-220.kmrr.netins.net (2001:5f8:2:2::1) 9.331 ms 8.894 ms 8.543 ms 6 v6-ins-kc2-te-8-3.kmrr.netins.net (2001:5f8::11:1) 9.213 ms 8.841 ms 8.505 ms 7 v6-ins-dc1-et-8-1.desm.netins.net (2001:5f8::8:1) 10.951 ms 10.540 ms 10.247 ms 8 te-3-3.car2.KansasCity1.Level3.net (2001:1900:2100::e2d) 17.223 ms 16.980 ms 16.490 ms 9 vl-11.car1.KansasCity1.Level3.net (2001:1900:4:1::37d) 17.282 ms 16.814 ms 16.585 ms 10 vl-4049.car1.Dallas1.Level3.net (2001:1900:4:1::379) 26.694 ms 26.212 ms 25.835 ms 11 vl-60.edge4.Dallas3.Level3.net (2001:1900:17:1::f) 26.727 ms 26.269 ms 25.950 ms 12 gblx-level3-40g.Dallas3.Level3.net (2001:1900:4:3::27e) 32.144 ms 26.371 ms 26.321 ms 13 2001:450:2001:8007::2 (2001:450:2001:8007::2) 26.894 ms 26.980 ms 26.211 ms 14 2001:450:2001:1000:0:670:1708:26 (2001:450:2001:1000:0:670:1708:26) 93.342 ms 92.727 ms 92.295 ms 15 2001:450:2001:8010::ff (2001:450:2001:8010::ff) 92.756 ms 92.198 ms 66.187 ms 16 2001:450:2032:1::f500 (2001:450:2032:1::f500) 3067.196 ms [open] 66.140 ms 65.601 ms nagios:/home/fbulk# nagios:/home/fbulk# tcptraceroute6 www.frontier.com traceroute to www.frontier.com (2001:1960:21:1::92) from 2607:fe28:0:1000::5, port 80, from port 45938, 30 hops max, 60 bytes packets 1 2607:fe28:0:1003::2 (2607:fe28:0:1003::2) 1.074 ms 1.211 ms 1.901 ms 2 router-core.mtcnet.net (2607:fe28:0:1000::1) 1.594 ms 2.202 ms 1.316 ms 3 sxct.movl.mtcnet.net (2607:fe28:11:1002::194) 2.520 ms 1.565 ms 1.258 ms 4 v6-siouxcenter.movl.153.netins.net (2001:5f8:7f06::9) 3.733 ms 3.376 ms 2.976 ms 5 v6-ins-kb1-te-13-2-220.kmrr.netins.net (2001:5f8:2:2::1) 9.606 ms 9.156 ms 8.735 ms 6 v6-ins-kc2-te-8-3.kmrr.netins.net (2001:5f8::11:1) 9.316 ms 8.947 ms 8.521 ms 7 v6-ins-dc1-et-8-1.desm.netins.net (2001:5f8::8:1) 11.109 ms 10.703 ms 10.255 ms 8 te-3-3.car2.KansasCity1.Level3.net (2001:1900:2100::e2d) 203.609 ms 199.474 ms 54.992 ms 9 vl-11.car1.KansasCity1.Level3.net (2001:1900:4:1::37d) 17.261 ms 16.885 ms 16.566 ms 10 vl-4049.car1.Dallas1.Level3.net (2001:1900:4:1::379) 26.386 ms 26.026 ms 25.667 ms 11 vl-4083.car2.Dallas1.Level3.net (2001:1900:4:1::8e) 26.387 ms 26.056 ms 25.688 ms 12 vl-4061.car1.Atlanta2.Level3.net (2001:1900:4:1::96) 45.931 ms 45.631 ms 45.678 ms 13 vl-51.car1.Atlanta4.Level3.net (2001:1900:1c:1::b) 45.929 ms 45.600 ms 45.308 ms 14 CITIZENS-CO.car1.Atlanta4.Level3.net (2001:1900:2100::1a) 45.913 ms 65.633 ms 45.271 ms 15 ae2---0.cor02.atln.ga.frontiernet.net (2001:1960:c:1f::) 45.667 ms 45.283 ms 71.426 ms 16 ae1---0.cor02.asbn.va.frontiernet.net (2001:1960:c:3c::1) 59.559 ms 90.101 ms 58.887 ms 17 ae3---0.cor01.roch.ny.frontiernet.net (2001:1960:c:3d::) 72.579 ms 72.547 ms 71.792 ms 18 ge--0-1-0---0.car02.roch.ny.frontiernet.net (2001:1960:c:28::1) 70.913 ms 70.199 ms 69.699 ms 19 2001:1960:21:1::92 (2001:1960:21:1::92) 71.011 ms [open] 70.533 ms 70.162 ms nagios:/home/fbulk# -----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Frank Bulk Sent: Friday, March 16, 2012 7:12 AM To: outages@outages.org Subject: [outages] Anyone else seeing IPv6 issues going through Level3? I monitor over 30 IPv6 hosts, and starting around 1:54 am (Central) I couldn't access about a third of them. Here's the list of sites that I currently can't access: ipv6_akamai_com_29 (2001:418:2401:9::8367:8929) ipv6_akamai_com_40 (2001:418:2401:9::8367:8940_ ipv6_cnn_com ipv6_google_com ipv6_sprint_com www-v6_cisco_com www_charter_com www_facebook_com_b (2620:0:1c18:0:face:b00c:0:2) www_ipv6_cisco_com www_sprint_net www_sprintv6_net Here's a sample of traceroutes: nagios:/home/fbulk# tcptraceroute6 www.sprintv6.net traceroute to www.sprint.net (2600::) from 2607:fe28:0:1000::5, port 80, from port 58614, 30 hops max, 60 bytes packets 1 2607:fe28:0:1003::2 (2607:fe28:0:1003::2) 1.080 ms 1.550 ms 1.356 ms 2 router-core.mtcnet.net (2607:fe28:0:1000::1) 1.856 ms 1.738 ms 1.661 ms 3 sxct.movl.mtcnet.net (2607:fe28:11:1002::194) 2.064 ms 1.966 ms 1.311 ms 4 v6-siouxcenter.movl.153.netins.net (2001:5f8:7f06::9) 4.128 ms 3.417 ms 3.048 ms 5 v6-ins-kb1-te-13-2-220.kmrr.netins.net (2001:5f8:2:2::1) 9.347 ms 8.930 ms 8.606 ms 6 v6-ins-kc2-te-8-3.kmrr.netins.net (2001:5f8::11:1) 9.258 ms 8.897 ms 8.549 ms 7 v6-ins-dc1-et-8-1.desm.netins.net (2001:5f8::8:1) 10.902 ms 10.494 ms 10.268 ms 8 te-3-3.car2.KansasCity1.Level3.net (2001:1900:2100::e2d) 195.453 ms 232.440 ms 230.230 ms 9 vl-11.car1.KansasCity1.Level3.net (2001:1900:4:1::37d) 17.244 ms 16.887 ms 16.524 ms 10 vl-4049.car1.Dallas1.Level3.net (2001:1900:4:1::379) 26.470 ms 26.076 ms 25.764 ms 11 vl-70.edge2.Dallas3.Level3.net (2001:1900:17:2::e) 26.664 ms 26.417 ms 26.058 ms 12 * * * 13 * * * 14 * * * nagios:/home/fbulk# tcptraceroute6 www.sprint.net traceroute to www.sprint.net (2600::) from 2607:fe28:0:1000::5, port 80, from port 55242, 30 hops max, 60 bytes packets 1 2607:fe28:0:1003::2 (2607:fe28:0:1003::2) 1.072 ms 1.533 ms 1.486 ms 2 router-core.mtcnet.net (2607:fe28:0:1000::1) 1.847 ms 1.862 ms 1.698 ms 3 sxct.movl.mtcnet.net (2607:fe28:11:1002::194) 2.187 ms 1.957 ms 1.220 ms 4 v6-siouxcenter.movl.153.netins.net (2001:5f8:7f06::9) 4.114 ms 3.318 ms 3.055 ms 5 v6-ins-kb1-te-13-2-220.kmrr.netins.net (2001:5f8:2:2::1) 9.291 ms 8.873 ms 8.612 ms 6 v6-ins-kc2-te-8-3.kmrr.netins.net (2001:5f8::11:1) 9.252 ms 8.907 ms 8.610 ms 7 v6-ins-dc1-et-8-1.desm.netins.net (2001:5f8::8:1) 10.926 ms 10.539 ms 10.221 ms 8 te-3-3.car2.KansasCity1.Level3.net (2001:1900:2100::e2d) 17.221 ms 16.776 ms 16.511 ms 9 vl-11.car1.KansasCity1.Level3.net (2001:1900:4:1::37d) 17.159 ms 16.885 ms 16.518 ms 10 vl-4049.car1.Dallas1.Level3.net (2001:1900:4:1::379) 26.486 ms 26.074 ms 25.743 ms 11 vl-70.edge2.Dallas3.Level3.net (2001:1900:17:2::e) 26.751 ms 26.432 ms 26.132 ms 12 * * * 13 * * * 14 * * * nagios:/home/fbulk# tcptraceroute6 www.ipv6.cisco.com traceroute to www.ipv6.cisco.com (2001:420:80:1::5) from 2607:fe28:0:1000::5, port 80, from port 53137, 30 hops max, 60 bytes packets 1 2607:fe28:0:1003::2 (2607:fe28:0:1003::2) 1.131 ms 1.190 ms 1.849 ms 2 router-core.mtcnet.net (2607:fe28:0:1000::1) 1.628 ms 2.158 ms 1.554 ms 3 sxct.sxcy.mtcnet.net (2607:fe28:11:1002::197) 2.618 ms 1.810 ms 1.190 ms 4 v6-siouxcenter.sxcy.137.netins.net (2001:5f8:7f06::5) 3.621 ms 2.928 ms 2.638 ms 5 v6-ins-db1-te-13-2-219.desm.netins.net (2001:5f8:1:1::1) 7.297 ms 6.957 ms 6.652 ms 6 v6-ins-dc1-et-8-2.desm.netins.net (2001:5f8::1:1) 9.145 ms 8.752 ms 8.447 ms 7 te-3-3.car2.KansasCity1.Level3.net (2001:1900:2100::e2d) 15.418 ms 15.064 ms 14.727 ms 8 vl-11.car1.KansasCity1.Level3.net (2001:1900:4:1::37d) 15.492 ms 15.020 ms 14.721 ms 9 vl-4049.car1.Dallas1.Level3.net (2001:1900:4:1::379) 24.696 ms 24.282 ms 24.144 ms 10 vl-70.edge2.Dallas3.Level3.net (2001:1900:17:2::e) 24.738 ms 24.489 ms 24.148 ms 11 * * * 12 * * * 13 * * * (facebook) nagios:/home/fbulk# tcptraceroute6 2620:0:1c18:0:face:b00c:0:2 traceroute to 2620:0:1c18:0:face:b00c:0:2 (2620:0:1c18:0:face:b00c:0:2) from 2607:fe28:0:1000::5, port 80, from port 47528, 30 hops max, 60 bytes packets 1 2607:fe28:0:1003::2 (2607:fe28:0:1003::2) 1.066 ms 1.243 ms 1.960 ms 2 router-core.mtcnet.net (2607:fe28:0:1000::1) 1.626 ms 2.268 ms 1.307 ms 3 sxct.movl.mtcnet.net (2607:fe28:11:1002::194) 2.592 ms 1.874 ms 1.238 ms 4 v6-siouxcenter.movl.153.netins.net (2001:5f8:7f06::9) 4.018 ms 3.341 ms 2.994 ms 5 v6-ins-kb1-te-13-2-220.kmrr.netins.net (2001:5f8:2:2::1) 9.367 ms 8.873 ms 8.656 ms 6 v6-ins-kc2-te-8-3.kmrr.netins.net (2001:5f8::11:1) 9.285 ms 8.980 ms 8.529 ms 7 v6-ins-dc1-et-8-1.desm.netins.net (2001:5f8::8:1) 10.944 ms 10.493 ms 10.231 ms 8 te-3-3.car2.KansasCity1.Level3.net (2001:1900:2100::e2d) 17.559 ms 16.798 ms 16.514 ms 9 vl-11.car1.KansasCity1.Level3.net (2001:1900:4:1::37d) 200.575 ms 233.140 ms 224.720 ms 10 vl-4049.car1.Dallas1.Level3.net (2001:1900:4:1::379) 26.312 ms 26.100 ms 25.774 ms 11 vl-4080.car2.Dallas1.Level3.net (2001:1900:4:1::82) 111.124 ms 131.324 ms 226.032 ms 12 vl-4061.car1.Atlanta2.Level3.net (2001:1900:4:1::96) 46.276 ms 45.963 ms 45.564 ms 13 vl-51.edge5.Atlanta2.Level3.net (2001:1900:1c:1::f) 45.928 ms 45.579 ms 45.292 ms 14 * * * 15 * * * 16 * * * nagios:/home/fbulk# tcptraceroute6 www.charter.com traceroute to www.charter.com (2607:f428:3:1:80:80:80:1) from 2607:fe28:0:1000::5, port 80, from port 44115, 30 hops max, 60 bytes packets 1 2607:fe28:0:1003::2 (2607:fe28:0:1003::2) 1.108 ms 1.280 ms 1.852 ms 2 router-core.mtcnet.net (2607:fe28:0:1000::1) 1.584 ms 2.204 ms 1.805 ms 3 sxct.sxcy.mtcnet.net (2607:fe28:11:1002::197) 2.515 ms 2.218 ms 1.197 ms 4 v6-siouxcenter.sxcy.137.netins.net (2001:5f8:7f06::5) 4.017 ms 2.944 ms 2.744 ms 5 v6-ins-db1-te-13-2-219.desm.netins.net (2001:5f8:1:1::1) 7.451 ms 7.004 ms 6.764 ms 6 v6-ins-dc1-et-8-2.desm.netins.net (2001:5f8::1:1) 9.132 ms 8.831 ms 8.481 ms 7 te-3-3.car2.KansasCity1.Level3.net (2001:1900:2100::e2d) 184.676 ms 202.524 ms 203.065 ms 8 vl-11.car1.KansasCity1.Level3.net (2001:1900:4:1::37d) 15.518 ms 15.134 ms 14.832 ms 9 vl-4049.car1.Dallas1.Level3.net (2001:1900:4:1::379) 65.261 ms 207.228 ms 41.498 ms 10 vl-80.edge2.Dallas3.Level3.net (2001:1900:17:3::e) 24.999 ms 56.325 ms 30.592 ms 11 CenturyTel-level3-10.dallas3.Level3.net (2001:1900:4:3::1c2) 25.072 ms 24.780 ms 24.480 ms 12 2001:428::205:171:202:195 (2001:428::205:171:202:195) 37.480 ms 37.165 ms 36.814 ms 13 * * * 14 * * * 15 * * * nagios:/home/fbulk# tcptraceroute6 ipv6.google.com traceroute to gx-in-x68.1e100.net (2001:4860:8002::68) from 2607:fe28:0:1000::5, port 80, from port 42606, 30 hops max, 60 bytes packets 1 2607:fe28:0:1003::2 (2607:fe28:0:1003::2) 1.041 ms 1.482 ms 1.418 ms 2 router-core.mtcnet.net (2607:fe28:0:1000::1) 1.870 ms 1.729 ms 1.665 ms 3 sxct.sxcy.mtcnet.net (2607:fe28:11:1002::197) 2.041 ms 1.994 ms 1.187 ms 4 v6-siouxcenter.sxcy.137.netins.net (2001:5f8:7f06::5) 3.778 ms 2.955 ms 2.669 ms 5 v6-ins-db1-te-13-2-219.desm.netins.net (2001:5f8:1:1::1) 7.361 ms 7.053 ms 6.874 ms 6 v6-ins-dc1-et-8-2.desm.netins.net (2001:5f8::1:1) 9.146 ms 8.804 ms 8.407 ms 7 te-3-3.car2.KansasCity1.Level3.net (2001:1900:2100::e2d) 69.291 ms 39.541 ms 15.380 ms 8 vl-11.car1.KansasCity1.Level3.net (2001:1900:4:1::37d) 15.399 ms 15.015 ms 14.720 ms 9 vl-4049.car1.Dallas1.Level3.net (2001:1900:4:1::379) 24.589 ms 24.209 ms 23.975 ms 10 vl-90.edge2.Dallas3.Level3.net (2001:1900:17:4::e) 24.698 ms 24.368 ms 24.078 ms 11 2001:1900:4:3::256 (2001:1900:4:3::256) 24.889 ms 24.411 ms 24.131 ms 12 2001:4860::1:0:57f (2001:4860::1:0:57f) 25.081 ms 24.648 ms 24.285 ms 13 2001:4860::8:0:2c9c (2001:4860::8:0:2c9c) 25.154 ms 24.816 ms 24.506 ms 14 * * * 15 * * * _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
On Fri, 2012-03-16 at 07:12 -0500, Frank Bulk wrote:
I monitor over 30 IPv6 hosts, and starting around 1:54 am (Central) I couldn't access about a third of them. Here's the list of sites that I currently can't access:
http://ring.nlnog.net promotime :) Just ran a few tests from 88 host in 78 ASN's, 24 countries:
ipv6_akamai_com_29 (2001:418:2401:9::8367:8929)
$ ring-ping 2001:418:2401:9::8367:8929 2001:418:2401:9::8367:8929 - 87 servers: 119ms average 2001:418:2401:9::8367:8929 - unreachable via: 1 server
ipv6_akamai_com_40 (2001:418:2401:9::8367:8940_
$ ring-ping 2001:418:2401:9::8367:8940 2001:418:2401:9::8367:8940 - 87 servers: 120ms average 2001:418:2401:9::8367:8940 - unreachable via: 1 server
ipv6_cnn_com
bit@bit01:~$ ring-http ipv6.cnn.com unreachable via: 88 hosts
ipv6_google_com
$ ring-http ipv6.google.com unreachable via: 88 hosts
ipv6_sprint_com
$ ring-http ipv6.sprint.com 88 servers: OK A ring-trace from all these hosts to 2001:418:2401:9::8367:8940 https://ring.nlnog.net/paste/p/pv2jaodnosy8mget regards, Teun
Thanks, Teun. Someone else pointed to me offlist that this could be just our prefix, 2607:fe28::/32. I've already contacted our upstream provider who buys the transit from Level3. Frank -----Original Message----- From: Teun Vink [mailto:teun@teun.tv] Sent: Friday, March 16, 2012 7:39 AM To: Frank Bulk Cc: outages@outages.org Subject: Re: [outages] Anyone else seeing IPv6 issues going through Level3? On Fri, 2012-03-16 at 07:12 -0500, Frank Bulk wrote:
I monitor over 30 IPv6 hosts, and starting around 1:54 am (Central) I couldn't access about a third of them. Here's the list of sites that I currently can't access:
http://ring.nlnog.net promotime :) Just ran a few tests from 88 host in 78 ASN's, 24 countries:
ipv6_akamai_com_29 (2001:418:2401:9::8367:8929)
$ ring-ping 2001:418:2401:9::8367:8929 2001:418:2401:9::8367:8929 - 87 servers: 119ms average 2001:418:2401:9::8367:8929 - unreachable via: 1 server
ipv6_akamai_com_40 (2001:418:2401:9::8367:8940_
$ ring-ping 2001:418:2401:9::8367:8940 2001:418:2401:9::8367:8940 - 87 servers: 120ms average 2001:418:2401:9::8367:8940 - unreachable via: 1 server
ipv6_cnn_com
bit@bit01:~$ ring-http ipv6.cnn.com unreachable via: 88 hosts
ipv6_google_com
$ ring-http ipv6.google.com unreachable via: 88 hosts
ipv6_sprint_com
$ ring-http ipv6.sprint.com 88 servers: OK A ring-trace from all these hosts to 2001:418:2401:9::8367:8940 https://ring.nlnog.net/paste/p/pv2jaodnosy8mget regards, Teun
Our upstream provider opened up a ticket with Level3 late this afternoon. Hopefully it will be resolved, soon. Our upstream provider has three dual-stacked transit providers and they shut down the Level3 IPv6 peering, but then we lost access to another third or so of the sites. For example, ipv6.t-mobile.com is only visible to us from Level3, same with www.globalcrossing.com. ipv6.cablelabs.com is with HE and Level3 and we should be able to get HE via Sprint, but we lost visibility anyways. Frank -----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Frank Bulk Sent: Friday, March 16, 2012 8:05 AM To: 'Teun Vink' Cc: outages@outages.org Subject: Re: [outages] Anyone else seeing IPv6 issues going through Level3? Thanks, Teun. Someone else pointed to me offlist that this could be just our prefix, 2607:fe28::/32. I've already contacted our upstream provider who buys the transit from Level3. Frank -----Original Message----- From: Teun Vink [mailto:teun@teun.tv] Sent: Friday, March 16, 2012 7:39 AM To: Frank Bulk Cc: outages@outages.org Subject: Re: [outages] Anyone else seeing IPv6 issues going through Level3? On Fri, 2012-03-16 at 07:12 -0500, Frank Bulk wrote:
I monitor over 30 IPv6 hosts, and starting around 1:54 am (Central) I couldn't access about a third of them. Here's the list of sites that I currently can't access:
http://ring.nlnog.net promotime :) Just ran a few tests from 88 host in 78 ASN's, 24 countries:
ipv6_akamai_com_29 (2001:418:2401:9::8367:8929)
$ ring-ping 2001:418:2401:9::8367:8929 2001:418:2401:9::8367:8929 - 87 servers: 119ms average 2001:418:2401:9::8367:8929 - unreachable via: 1 server
ipv6_akamai_com_40 (2001:418:2401:9::8367:8940_
$ ring-ping 2001:418:2401:9::8367:8940 2001:418:2401:9::8367:8940 - 87 servers: 120ms average 2001:418:2401:9::8367:8940 - unreachable via: 1 server
ipv6_cnn_com
bit@bit01:~$ ring-http ipv6.cnn.com unreachable via: 88 hosts
ipv6_google_com
$ ring-http ipv6.google.com unreachable via: 88 hosts
ipv6_sprint_com
$ ring-http ipv6.sprint.com 88 servers: OK A ring-trace from all these hosts to 2001:418:2401:9::8367:8940 https://ring.nlnog.net/paste/p/pv2jaodnosy8mget regards, Teun _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Service was restored to me at 7:15 pm. I don't have an RFO -- I can only guess there was some routing table issue for our netblock in Level3's infrastructure. Frank -----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Frank Bulk Sent: Friday, March 16, 2012 6:50 PM To: outages@outages.org Subject: Re: [outages] Anyone else seeing IPv6 issues going through Level3? Our upstream provider opened up a ticket with Level3 late this afternoon. Hopefully it will be resolved, soon. Our upstream provider has three dual-stacked transit providers and they shut down the Level3 IPv6 peering, but then we lost access to another third or so of the sites. For example, ipv6.t-mobile.com is only visible to us from Level3, same with www.globalcrossing.com. ipv6.cablelabs.com is with HE and Level3 and we should be able to get HE via Sprint, but we lost visibility anyways. Frank -----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Frank Bulk Sent: Friday, March 16, 2012 8:05 AM To: 'Teun Vink' Cc: outages@outages.org Subject: Re: [outages] Anyone else seeing IPv6 issues going through Level3? Thanks, Teun. Someone else pointed to me offlist that this could be just our prefix, 2607:fe28::/32. I've already contacted our upstream provider who buys the transit from Level3. Frank -----Original Message----- From: Teun Vink [mailto:teun@teun.tv] Sent: Friday, March 16, 2012 7:39 AM To: Frank Bulk Cc: outages@outages.org Subject: Re: [outages] Anyone else seeing IPv6 issues going through Level3? On Fri, 2012-03-16 at 07:12 -0500, Frank Bulk wrote:
I monitor over 30 IPv6 hosts, and starting around 1:54 am (Central) I couldn't access about a third of them. Here's the list of sites that I currently can't access:
http://ring.nlnog.net promotime :) Just ran a few tests from 88 host in 78 ASN's, 24 countries:
ipv6_akamai_com_29 (2001:418:2401:9::8367:8929)
$ ring-ping 2001:418:2401:9::8367:8929 2001:418:2401:9::8367:8929 - 87 servers: 119ms average 2001:418:2401:9::8367:8929 - unreachable via: 1 server
ipv6_akamai_com_40 (2001:418:2401:9::8367:8940_
$ ring-ping 2001:418:2401:9::8367:8940 2001:418:2401:9::8367:8940 - 87 servers: 120ms average 2001:418:2401:9::8367:8940 - unreachable via: 1 server
ipv6_cnn_com
bit@bit01:~$ ring-http ipv6.cnn.com unreachable via: 88 hosts
ipv6_google_com
$ ring-http ipv6.google.com unreachable via: 88 hosts
ipv6_sprint_com
$ ring-http ipv6.sprint.com 88 servers: OK A ring-trace from all these hosts to 2001:418:2401:9::8367:8940 https://ring.nlnog.net/paste/p/pv2jaodnosy8mget regards, Teun _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
participants (2)
-
Frank Bulk -
Teun Vink