I noticed that IPv4 and IPv6 access to www.he.net went down at 5:30 pm Central. Traceroutes show different things, but basically it's getting stuck somewhere in HE's core. nagios:# tcptraceroute www.he.net Selected device eth0.3, address 96.31.0.5, port 36642 for outgoing packets Tracing the path to www.he.net (216.218.186.2) on TCP port 80 (www), 30 hops max 1 router-core-inside.mtcnet.net (96.31.0.254) 0.240 ms 0.201 ms 0.179 ms 2 sxct.sxcy.mtcnet.net (167.142.156.197) 0.208 ms 0.136 ms 0.127 ms 3 premier.sxcy-mlx.fbnt.netins.net (173.215.60.5) 1.939 ms 1.920 ms 1.904 ms 4 ins-db1-te-13-2-219.desm.netins.net (167.142.60.157) 7.886 ms 7.835 ms 7.957 ms 5 ins-dc2-et-8-3.desm.netins.net (167.142.67.25) 7.795 ms 7.759 ms 7.803 ms 6 ins-db3-te-0-7-0-0.desm.netins.net (167.142.67.182) 8.190 ms 7.977 ms 7.973 ms 7 ord1.ins.indatelservices.com (216.176.4.29) 23.014 ms 23.230 ms 22.932 ms 8 10gigabitethernet4-1.core1.chi1.he.NET (206.223.119.37) 22.819 ms 22.835 ms 22.776 ms 9 * * * 10 * * * 11 * * * 12 * * * nagios:# tcptraceroute6 www.he.net traceroute to he.net (2001:470:0:76::2) from 2607:fe28:0:1000::5, port 80, from port 60791, 30 hops max, 60 bytes packets 1 router-core.mtcnet.net (2607:fe28:0:1000::1) 0.240 ms 0.200 ms 0.181 ms 2 sxct.movl.mtcnet.net (2607:fe28:11:1002::194) 0.226 ms 0.856 ms 0.139 ms 3 v6-premier.movl-mlx.fbnt.netins.net (2001:5f8:7f0a:1::1) 1.962 ms 1.923 ms 1.888 ms 4 v6-ins-kb1-te-13-2-220.kmrr.netins.net (2001:5f8:2:2::1) 6.315 ms 6.354 ms 6.399 ms 5 v6-ins-kc1-te-9-2.kmrr.netins.net (2001:5f8::10:1) 6.360 ms 6.412 ms 6.306 ms 6 sl-crs2-chi-te1-2-0-1.v6.sprintlink.net (2600:4::6) 16.130 ms 15.853 ms 15.736 ms 7 sl-crs1-chi-po0-0-0-3.v6.sprintlink.net (2600:0:2:1239:144:232:26:8) 17.236 ms 16.344 ms 16.133 ms 8 sl-st30-chi-bu-20.v6.sprintlink.net (2600:0:2:1239:144:232:1:102) 17.649 ms 17.857 ms 17.544 ms 9 10gigabitethernet4-1.core1.chi1.he.net (2001:504:0:4::6939:1) 25.946 ms 16.488 ms 16.484 ms 10 10gigabitethernet5-7.core1.mci3.he.net (2001:470:0:270::2) 28.448 ms 38.399 ms 28.468 ms 11 10gigabitethernet5-1.core1.den1.he.net (2001:470:0:240::1) 40.525 ms 48.500 ms 48.141 ms 12 10gigabitethernet13-5.core1.sjc2.he.net (2001:470:0:1b4::1) 75.170 ms 74.256 ms 72.699 ms 13 * * * 14 * * * 15 * * * 16 * * * [fbulk@puck /]$ traceroute www.he.net traceroute to www.he.net (216.218.186.2), 30 hops max, 60 byte packets 1 ge-0-7-0-22.r05.chcgil09.us.bb.gin.ntt.net (204.42.253.5) 0.738 ms 0.650 ms 0.610 ms 2 ae-0.teliasonera.chcgil09.us.bb.gin.ntt.net (129.250.8.178) 0.219 ms 0.194 ms 0.172 ms 3 chi-bb1-link.telia.net (80.91.246.168) 0.290 ms chi-bb1-link.telia.net (80.91.246.116) 31.168 ms chi-bb1-link.telia.net (80.91.246.118) 31.363 ms 4 sjo-bb1-link.telia.net (213.248.80.25) 51.322 ms sjo-bb1-link.telia.net (213.155.132.180) 51.297 ms 51.275 ms 5 * * * 6 * * * 7 * * * 8 * * * 9 * * * [fbulk@puck /]$ traceroute6 www.he.net traceroute to www.he.net (2001:470:0:76::2), 30 hops max, 80 byte packets 1 ge-0-7-0-22.r05.chcgil09.us.bb.gin.ntt.net (2001:418:3f4::1) 0.725 ms 0.622 ms 0.666 ms 2 ae-6.r20.chcgil09.us.bb.gin.ntt.net (2001:418:0:2000::29) 80.228 ms 0.395 ms 0.446 ms 3 10gigabitethernet4-1.core1.chi1.he.net (2001:504:0:4::6939:1) 0.267 ms 0.402 ms 0.299 ms 4 10gigabitethernet5-7.core1.mci3.he.net (2001:470:0:270::2) 12.659 ms 10gigabitethernet4-1.core1.den1.he.net (2001:470:0:293::2) 24.342 ms 24.299 ms 5 10gigabitethernet13-5.core1.sjc2.he.net (2001:470:0:1b4::1) 59.785 ms 59.189 ms 59.450 ms 6 10gigabitethernet13-5.core1.sjc2.he.net (2001:470:0:1b4::1) 59.539 ms 58.630 ms * 7 * * * 8 * * * 9 * * * 10 * * * Incidentally, I noticed that ipv6.cablelabs.com went down at the same time, too. Frank