There are other sites going through Level3 that do work (I picked from a list that blipped at 5:05 am). Here are just two of them: tctnagios:/home/fbulk# tcptraceroute6 www.globalcrossing.com traceroute to www.globalcrossing.com (2001:450:2032:1::f500) from 2607:fe28:0:1000::5, port 80, from port 53367, 30 hops max, 60 bytes packets 1 2607:fe28:0:1003::2 (2607:fe28:0:1003::2) 1.049 ms 1.480 ms 1.370 ms 2 router-core.mtcnet.net (2607:fe28:0:1000::1) 1.796 ms 1.756 ms 1.737 ms 3 sxct.movl.mtcnet.net (2607:fe28:11:1002::194) 2.065 ms 1.998 ms 1.251 ms 4 v6-siouxcenter.movl.153.netins.net (2001:5f8:7f06::9) 4.169 ms 3.333 ms 2.973 ms 5 v6-ins-kb1-te-13-2-220.kmrr.netins.net (2001:5f8:2:2::1) 9.331 ms 8.894 ms 8.543 ms 6 v6-ins-kc2-te-8-3.kmrr.netins.net (2001:5f8::11:1) 9.213 ms 8.841 ms 8.505 ms 7 v6-ins-dc1-et-8-1.desm.netins.net (2001:5f8::8:1) 10.951 ms 10.540 ms 10.247 ms 8 te-3-3.car2.KansasCity1.Level3.net (2001:1900:2100::e2d) 17.223 ms 16.980 ms 16.490 ms 9 vl-11.car1.KansasCity1.Level3.net (2001:1900:4:1::37d) 17.282 ms 16.814 ms 16.585 ms 10 vl-4049.car1.Dallas1.Level3.net (2001:1900:4:1::379) 26.694 ms 26.212 ms 25.835 ms 11 vl-60.edge4.Dallas3.Level3.net (2001:1900:17:1::f) 26.727 ms 26.269 ms 25.950 ms 12 gblx-level3-40g.Dallas3.Level3.net (2001:1900:4:3::27e) 32.144 ms 26.371 ms 26.321 ms 13 2001:450:2001:8007::2 (2001:450:2001:8007::2) 26.894 ms 26.980 ms 26.211 ms 14 2001:450:2001:1000:0:670:1708:26 (2001:450:2001:1000:0:670:1708:26) 93.342 ms 92.727 ms 92.295 ms 15 2001:450:2001:8010::ff (2001:450:2001:8010::ff) 92.756 ms 92.198 ms 66.187 ms 16 2001:450:2032:1::f500 (2001:450:2032:1::f500) 3067.196 ms [open] 66.140 ms 65.601 ms nagios:/home/fbulk# nagios:/home/fbulk# tcptraceroute6 www.frontier.com traceroute to www.frontier.com (2001:1960:21:1::92) from 2607:fe28:0:1000::5, port 80, from port 45938, 30 hops max, 60 bytes packets 1 2607:fe28:0:1003::2 (2607:fe28:0:1003::2) 1.074 ms 1.211 ms 1.901 ms 2 router-core.mtcnet.net (2607:fe28:0:1000::1) 1.594 ms 2.202 ms 1.316 ms 3 sxct.movl.mtcnet.net (2607:fe28:11:1002::194) 2.520 ms 1.565 ms 1.258 ms 4 v6-siouxcenter.movl.153.netins.net (2001:5f8:7f06::9) 3.733 ms 3.376 ms 2.976 ms 5 v6-ins-kb1-te-13-2-220.kmrr.netins.net (2001:5f8:2:2::1) 9.606 ms 9.156 ms 8.735 ms 6 v6-ins-kc2-te-8-3.kmrr.netins.net (2001:5f8::11:1) 9.316 ms 8.947 ms 8.521 ms 7 v6-ins-dc1-et-8-1.desm.netins.net (2001:5f8::8:1) 11.109 ms 10.703 ms 10.255 ms 8 te-3-3.car2.KansasCity1.Level3.net (2001:1900:2100::e2d) 203.609 ms 199.474 ms 54.992 ms 9 vl-11.car1.KansasCity1.Level3.net (2001:1900:4:1::37d) 17.261 ms 16.885 ms 16.566 ms 10 vl-4049.car1.Dallas1.Level3.net (2001:1900:4:1::379) 26.386 ms 26.026 ms 25.667 ms 11 vl-4083.car2.Dallas1.Level3.net (2001:1900:4:1::8e) 26.387 ms 26.056 ms 25.688 ms 12 vl-4061.car1.Atlanta2.Level3.net (2001:1900:4:1::96) 45.931 ms 45.631 ms 45.678 ms 13 vl-51.car1.Atlanta4.Level3.net (2001:1900:1c:1::b) 45.929 ms 45.600 ms 45.308 ms 14 CITIZENS-CO.car1.Atlanta4.Level3.net (2001:1900:2100::1a) 45.913 ms 65.633 ms 45.271 ms 15 ae2---0.cor02.atln.ga.frontiernet.net (2001:1960:c:1f::) 45.667 ms 45.283 ms 71.426 ms 16 ae1---0.cor02.asbn.va.frontiernet.net (2001:1960:c:3c::1) 59.559 ms 90.101 ms 58.887 ms 17 ae3---0.cor01.roch.ny.frontiernet.net (2001:1960:c:3d::) 72.579 ms 72.547 ms 71.792 ms 18 ge--0-1-0---0.car02.roch.ny.frontiernet.net (2001:1960:c:28::1) 70.913 ms 70.199 ms 69.699 ms 19 2001:1960:21:1::92 (2001:1960:21:1::92) 71.011 ms [open] 70.533 ms 70.162 ms nagios:/home/fbulk# -----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Frank Bulk Sent: Friday, March 16, 2012 7:12 AM To: outages@outages.org Subject: [outages] Anyone else seeing IPv6 issues going through Level3? I monitor over 30 IPv6 hosts, and starting around 1:54 am (Central) I couldn't access about a third of them. Here's the list of sites that I currently can't access: ipv6_akamai_com_29 (2001:418:2401:9::8367:8929) ipv6_akamai_com_40 (2001:418:2401:9::8367:8940_ ipv6_cnn_com ipv6_google_com ipv6_sprint_com www-v6_cisco_com www_charter_com www_facebook_com_b (2620:0:1c18:0:face:b00c:0:2) www_ipv6_cisco_com www_sprint_net www_sprintv6_net Here's a sample of traceroutes: nagios:/home/fbulk# tcptraceroute6 www.sprintv6.net traceroute to www.sprint.net (2600::) from 2607:fe28:0:1000::5, port 80, from port 58614, 30 hops max, 60 bytes packets 1 2607:fe28:0:1003::2 (2607:fe28:0:1003::2) 1.080 ms 1.550 ms 1.356 ms 2 router-core.mtcnet.net (2607:fe28:0:1000::1) 1.856 ms 1.738 ms 1.661 ms 3 sxct.movl.mtcnet.net (2607:fe28:11:1002::194) 2.064 ms 1.966 ms 1.311 ms 4 v6-siouxcenter.movl.153.netins.net (2001:5f8:7f06::9) 4.128 ms 3.417 ms 3.048 ms 5 v6-ins-kb1-te-13-2-220.kmrr.netins.net (2001:5f8:2:2::1) 9.347 ms 8.930 ms 8.606 ms 6 v6-ins-kc2-te-8-3.kmrr.netins.net (2001:5f8::11:1) 9.258 ms 8.897 ms 8.549 ms 7 v6-ins-dc1-et-8-1.desm.netins.net (2001:5f8::8:1) 10.902 ms 10.494 ms 10.268 ms 8 te-3-3.car2.KansasCity1.Level3.net (2001:1900:2100::e2d) 195.453 ms 232.440 ms 230.230 ms 9 vl-11.car1.KansasCity1.Level3.net (2001:1900:4:1::37d) 17.244 ms 16.887 ms 16.524 ms 10 vl-4049.car1.Dallas1.Level3.net (2001:1900:4:1::379) 26.470 ms 26.076 ms 25.764 ms 11 vl-70.edge2.Dallas3.Level3.net (2001:1900:17:2::e) 26.664 ms 26.417 ms 26.058 ms 12 * * * 13 * * * 14 * * * nagios:/home/fbulk# tcptraceroute6 www.sprint.net traceroute to www.sprint.net (2600::) from 2607:fe28:0:1000::5, port 80, from port 55242, 30 hops max, 60 bytes packets 1 2607:fe28:0:1003::2 (2607:fe28:0:1003::2) 1.072 ms 1.533 ms 1.486 ms 2 router-core.mtcnet.net (2607:fe28:0:1000::1) 1.847 ms 1.862 ms 1.698 ms 3 sxct.movl.mtcnet.net (2607:fe28:11:1002::194) 2.187 ms 1.957 ms 1.220 ms 4 v6-siouxcenter.movl.153.netins.net (2001:5f8:7f06::9) 4.114 ms 3.318 ms 3.055 ms 5 v6-ins-kb1-te-13-2-220.kmrr.netins.net (2001:5f8:2:2::1) 9.291 ms 8.873 ms 8.612 ms 6 v6-ins-kc2-te-8-3.kmrr.netins.net (2001:5f8::11:1) 9.252 ms 8.907 ms 8.610 ms 7 v6-ins-dc1-et-8-1.desm.netins.net (2001:5f8::8:1) 10.926 ms 10.539 ms 10.221 ms 8 te-3-3.car2.KansasCity1.Level3.net (2001:1900:2100::e2d) 17.221 ms 16.776 ms 16.511 ms 9 vl-11.car1.KansasCity1.Level3.net (2001:1900:4:1::37d) 17.159 ms 16.885 ms 16.518 ms 10 vl-4049.car1.Dallas1.Level3.net (2001:1900:4:1::379) 26.486 ms 26.074 ms 25.743 ms 11 vl-70.edge2.Dallas3.Level3.net (2001:1900:17:2::e) 26.751 ms 26.432 ms 26.132 ms 12 * * * 13 * * * 14 * * * nagios:/home/fbulk# tcptraceroute6 www.ipv6.cisco.com traceroute to www.ipv6.cisco.com (2001:420:80:1::5) from 2607:fe28:0:1000::5, port 80, from port 53137, 30 hops max, 60 bytes packets 1 2607:fe28:0:1003::2 (2607:fe28:0:1003::2) 1.131 ms 1.190 ms 1.849 ms 2 router-core.mtcnet.net (2607:fe28:0:1000::1) 1.628 ms 2.158 ms 1.554 ms 3 sxct.sxcy.mtcnet.net (2607:fe28:11:1002::197) 2.618 ms 1.810 ms 1.190 ms 4 v6-siouxcenter.sxcy.137.netins.net (2001:5f8:7f06::5) 3.621 ms 2.928 ms 2.638 ms 5 v6-ins-db1-te-13-2-219.desm.netins.net (2001:5f8:1:1::1) 7.297 ms 6.957 ms 6.652 ms 6 v6-ins-dc1-et-8-2.desm.netins.net (2001:5f8::1:1) 9.145 ms 8.752 ms 8.447 ms 7 te-3-3.car2.KansasCity1.Level3.net (2001:1900:2100::e2d) 15.418 ms 15.064 ms 14.727 ms 8 vl-11.car1.KansasCity1.Level3.net (2001:1900:4:1::37d) 15.492 ms 15.020 ms 14.721 ms 9 vl-4049.car1.Dallas1.Level3.net (2001:1900:4:1::379) 24.696 ms 24.282 ms 24.144 ms 10 vl-70.edge2.Dallas3.Level3.net (2001:1900:17:2::e) 24.738 ms 24.489 ms 24.148 ms 11 * * * 12 * * * 13 * * * (facebook) nagios:/home/fbulk# tcptraceroute6 2620:0:1c18:0:face:b00c:0:2 traceroute to 2620:0:1c18:0:face:b00c:0:2 (2620:0:1c18:0:face:b00c:0:2) from 2607:fe28:0:1000::5, port 80, from port 47528, 30 hops max, 60 bytes packets 1 2607:fe28:0:1003::2 (2607:fe28:0:1003::2) 1.066 ms 1.243 ms 1.960 ms 2 router-core.mtcnet.net (2607:fe28:0:1000::1) 1.626 ms 2.268 ms 1.307 ms 3 sxct.movl.mtcnet.net (2607:fe28:11:1002::194) 2.592 ms 1.874 ms 1.238 ms 4 v6-siouxcenter.movl.153.netins.net (2001:5f8:7f06::9) 4.018 ms 3.341 ms 2.994 ms 5 v6-ins-kb1-te-13-2-220.kmrr.netins.net (2001:5f8:2:2::1) 9.367 ms 8.873 ms 8.656 ms 6 v6-ins-kc2-te-8-3.kmrr.netins.net (2001:5f8::11:1) 9.285 ms 8.980 ms 8.529 ms 7 v6-ins-dc1-et-8-1.desm.netins.net (2001:5f8::8:1) 10.944 ms 10.493 ms 10.231 ms 8 te-3-3.car2.KansasCity1.Level3.net (2001:1900:2100::e2d) 17.559 ms 16.798 ms 16.514 ms 9 vl-11.car1.KansasCity1.Level3.net (2001:1900:4:1::37d) 200.575 ms 233.140 ms 224.720 ms 10 vl-4049.car1.Dallas1.Level3.net (2001:1900:4:1::379) 26.312 ms 26.100 ms 25.774 ms 11 vl-4080.car2.Dallas1.Level3.net (2001:1900:4:1::82) 111.124 ms 131.324 ms 226.032 ms 12 vl-4061.car1.Atlanta2.Level3.net (2001:1900:4:1::96) 46.276 ms 45.963 ms 45.564 ms 13 vl-51.edge5.Atlanta2.Level3.net (2001:1900:1c:1::f) 45.928 ms 45.579 ms 45.292 ms 14 * * * 15 * * * 16 * * * nagios:/home/fbulk# tcptraceroute6 www.charter.com traceroute to www.charter.com (2607:f428:3:1:80:80:80:1) from 2607:fe28:0:1000::5, port 80, from port 44115, 30 hops max, 60 bytes packets 1 2607:fe28:0:1003::2 (2607:fe28:0:1003::2) 1.108 ms 1.280 ms 1.852 ms 2 router-core.mtcnet.net (2607:fe28:0:1000::1) 1.584 ms 2.204 ms 1.805 ms 3 sxct.sxcy.mtcnet.net (2607:fe28:11:1002::197) 2.515 ms 2.218 ms 1.197 ms 4 v6-siouxcenter.sxcy.137.netins.net (2001:5f8:7f06::5) 4.017 ms 2.944 ms 2.744 ms 5 v6-ins-db1-te-13-2-219.desm.netins.net (2001:5f8:1:1::1) 7.451 ms 7.004 ms 6.764 ms 6 v6-ins-dc1-et-8-2.desm.netins.net (2001:5f8::1:1) 9.132 ms 8.831 ms 8.481 ms 7 te-3-3.car2.KansasCity1.Level3.net (2001:1900:2100::e2d) 184.676 ms 202.524 ms 203.065 ms 8 vl-11.car1.KansasCity1.Level3.net (2001:1900:4:1::37d) 15.518 ms 15.134 ms 14.832 ms 9 vl-4049.car1.Dallas1.Level3.net (2001:1900:4:1::379) 65.261 ms 207.228 ms 41.498 ms 10 vl-80.edge2.Dallas3.Level3.net (2001:1900:17:3::e) 24.999 ms 56.325 ms 30.592 ms 11 CenturyTel-level3-10.dallas3.Level3.net (2001:1900:4:3::1c2) 25.072 ms 24.780 ms 24.480 ms 12 2001:428::205:171:202:195 (2001:428::205:171:202:195) 37.480 ms 37.165 ms 36.814 ms 13 * * * 14 * * * 15 * * * nagios:/home/fbulk# tcptraceroute6 ipv6.google.com traceroute to gx-in-x68.1e100.net (2001:4860:8002::68) from 2607:fe28:0:1000::5, port 80, from port 42606, 30 hops max, 60 bytes packets 1 2607:fe28:0:1003::2 (2607:fe28:0:1003::2) 1.041 ms 1.482 ms 1.418 ms 2 router-core.mtcnet.net (2607:fe28:0:1000::1) 1.870 ms 1.729 ms 1.665 ms 3 sxct.sxcy.mtcnet.net (2607:fe28:11:1002::197) 2.041 ms 1.994 ms 1.187 ms 4 v6-siouxcenter.sxcy.137.netins.net (2001:5f8:7f06::5) 3.778 ms 2.955 ms 2.669 ms 5 v6-ins-db1-te-13-2-219.desm.netins.net (2001:5f8:1:1::1) 7.361 ms 7.053 ms 6.874 ms 6 v6-ins-dc1-et-8-2.desm.netins.net (2001:5f8::1:1) 9.146 ms 8.804 ms 8.407 ms 7 te-3-3.car2.KansasCity1.Level3.net (2001:1900:2100::e2d) 69.291 ms 39.541 ms 15.380 ms 8 vl-11.car1.KansasCity1.Level3.net (2001:1900:4:1::37d) 15.399 ms 15.015 ms 14.720 ms 9 vl-4049.car1.Dallas1.Level3.net (2001:1900:4:1::379) 24.589 ms 24.209 ms 23.975 ms 10 vl-90.edge2.Dallas3.Level3.net (2001:1900:17:4::e) 24.698 ms 24.368 ms 24.078 ms 11 2001:1900:4:3::256 (2001:1900:4:3::256) 24.889 ms 24.411 ms 24.131 ms 12 2001:4860::1:0:57f (2001:4860::1:0:57f) 25.081 ms 24.648 ms 24.285 ms 13 2001:4860::8:0:2c9c (2001:4860::8:0:2c9c) 25.154 ms 24.816 ms 24.506 ms 14 * * * 15 * * * _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages