I'm being told by the NOC that someone is on the ground in FRA1, replacing a switch; and that it's a "scheduled maintenance", no ETA, and there was no announcement, since someone has to take a flight to Europe and its unpredictable. :-) 35 minutes of downtime, and counting... C. On 12 June 2013 15:19, Constantine A. Murenin <mureninc@gmail.com> wrote:
I smell a big outage:
% date; traceroute tserv1.fra1.he.net Wed Jun 12 15:17:04 PDT 2013 traceroute to tserv1.fra1.he.net (216.66.80.30), 30 hops max, 60 byte packets 1 192.168.105.3 (192.168.105.3) 0.673 ms 0.773 ms 0.923 ms 2 10gigabitethernet7-6.core3.fmt2.he.net (65.49.10.217) 1.795 ms 1.811 ms 1.795 ms 3 10gigabitethernet12-1.core1.lax1.he.net (184.105.213.26) 19.773 ms 10gigabitethernet10-1.core1.sjc2.he.net (184.105.222.14) 0.786 ms 0.756 ms 4 10gigabitethernet10-8.core1.nyc4.he.net (72.52.92.225) 71.714 ms 76.569 ms 10gigabitethernet14-2.core1.nyc4.he.net (184.105.213.198) 71.690 ms 5 * * * 6 * * * 7 * * * 8 * * * 9 * * * 10 * * * 11 * * * 12 * * * 13 * * * 14 * * * 15 * *^C
However, a "reverse" traceroute works fine:
% date; traceroute ns1.he.net Wed Jun 12 15:18:47 PDT 2013 traceroute to ns1.he.net (216.218.130.2), 64 hops max, 40 byte packets 1 static.33.203.4.46.clients.your-server.de (46.4.203.33) 0.682 ms 0.531 ms 0.481 ms 2 hos-tr4.juniper2.rz13.hetzner.de (213.239.224.97) 0.245 ms hos-tr3.juniper2.rz13.hetzner.de (213.239.224.65) 0.242 ms hos-tr2.juniper1.rz13.hetzner.de (213.239.224.33) 0.241 ms 3 hos-bb2.juniper4.ffm.hetzner.de (213.239.240.150) 5.951 ms hos-bb1.juniper1.ffm.hetzner.de (213.239.240.224) 4.803 ms 4.786 ms 4 30gigabitethernet4-3.core1.fra1.he.net (80.81.192.172) 6.354 ms 6.81 ms 5.451 ms 5 10gigabitethernet10-2.core1.par2.he.net (72.52.92.26) 22.803 ms 24.531 ms 24.787 ms 6 10gigabitethernet15-1.core1.ash1.he.net (184.105.213.93) 101.504 ms 99.563 ms 99.958 ms 7 10gigabitethernet11-1.core1.pao1.he.net (184.105.213.177) 163.687 ms 171.711 ms 175.34 ms 8 10gigabitethernet1-2.core1.fmt1.he.net (184.105.213.65) 163.940 ms 171.362 ms 167.67 ms 9 ns1.he.net (216.218.130.2) 163.52 ms 165.265 ms 164.143 ms
C.
On 12 June 2013 15:11, Constantine A. Murenin <mureninc@gmail.com> wrote:
he.net fra1 tserv1 is down since about 10 minutes ago (~14:57 PT), this time it seems like the whole thing is down, cannot even ping ordns.he.net over IPv6; a connection of my friend who's running a smokeping is also down, e.g. this is definitely widespread.
Not sure what's exactly down, but it seems to be bgp-related, perhaps:
1 2600:3c01::8678:acff:fe0d:79c1 (2600:3c01::8678:acff:fe0d:79c1) 0.699 ms 0.828 ms 0.970 ms 2 10gigabitethernet7-6.core3.fmt2.he.net (2001:470:1:3b8::1) 5.936 ms 5.877 ms 5.859 ms 3 10gigabitethernet5-4.core1.pao1.he.net (2001:470:0:263::2) 6.743 ms 6.877 ms 6.978 ms 4 * * * 5 * * * 6 * * * 7 * * * 8 * * * 9 * * * 10 * * * 11 * * * 12 * * * 13 * * * 14 * * * 15 * * * 16 * * * 17 * * * 18 * * *
C.