We have a Qwest MPLS VPN connection in Boise and saw the same drop from 04:05 to 04:14 MDT. BGP did not drop, but GRE tunnels on our router did. Will Melick Network Administrator Berkley Technology Services, LLC 405 Silverside Rd., Suite 205 Wilmington, DE 19809 (302)439-2005 From: Darrell Hyde <DHyde@hosting.com> To: "Jason J. W. Williams" <williamsjj@digitar.com> Cc: "outages@outages.org" <outages@outages.org> Date: 10/26/2011 07:42 AM Subject: Re: [outages] Qwest Re-Convergence Sent by: outages-bounces@outages.org Ditto. Our side of the /30 we share with Qwest was unreachable from approx 4:05 to 4:12 AM MT. Haven't looked yet to see if our BGP session dropped during that time. If not we (they) were likely blackholing traffic briefly. On Oct 26, 2011, at 6:40 AM, "Jason J. W. Williams" << mailto:williamsjj@digitar.com>williamsjj@digitar.com< mailto:williamsjj@digitar.com>> wrote: Has anyone noticed issues reaching Qwest destinations around 4:05a-4:07a Mountain Time today? We received alerts that two of our sites that happened to have Qwest connectivity in common were unavailable during that period. Both sites being monitored were in different facilities but both in Boise. -J Traceroute #1 1 <http://ip-67-205-112-66.static.privatedns.com> ip-67-205-112-66.static.privatedns.com< http://ip-67-205-112-66.static.privatedns.com> (67.205.112.66) 0.842 ms 0.908 ms 0.936 ms 2 te8-4.v0707.cl-core04.mtl.iweb.com (67.205.127.121) 1.047 ms 1.068 ms 1.089 ms 3 if-9-1.icore1.MTT-Montreal.as6453.net (206.82.135.29) 7.737 ms 7.813 ms 7.837 ms 4 if-10-0-0-19.mcore3.MTT-Montreal.as6453.net (216.6.114.13) 0.907 ms 0.935 ms 0.955 ms 5 if-9-0-0-0.tcore2.NYY-NewYork.as6453.net (216.6.99.9) 7.871 ms 7.904 ms 7.926 ms 6 if-11-2.tcore1.NYY-NewYork.as6453.net (216.6.99.2) 7.815 ms 8.013 ms 8.032 ms 7 <http://jfk-brdr-04.inet.qwest.net> jfk-brdr-04.inet.qwest.net< http://jfk-brdr-04.inet.qwest.net> (63.146.27.41) 7.942 ms 7.827 ms 7.790 ms Traceroute #2 1 207.97.207.194 (207.97.207.194) 0.549 ms 0.578 ms 0.620 ms 2 <http://core1-5.iad1.rackspace.com> core1-5.iad1.rackspace.com< http://core1-5.iad1.rackspace.com> (69.20.2.18) 0.303 ms 0.354 ms 0.400 ms 3 vlan901.edge3.iad1.rackspace.com (69.20.1.3) 196.221 ms 196.284 ms 196.328 ms 4 <http://dcx2-edge-01.inet.qwest.net> dcx2-edge-01.inet.qwest.net< http://dcx2-edge-01.inet.qwest.net> (67.133.246.157) 0.368 ms 0.355 ms 0.349 ms ------------------- Jason J. W. Williams COO/CTO, DigiTar <mailto:williamsjj@digitar.com>williamsjj@digitar.com< mailto:williamsjj@digitar.com> V: 208.343.8520 M: 208.863.0727 F: 208.322.8522 IM: <mailto:williamsjj@digitar.com> <mailto:williamsjj@digitar.com> williamsjj@digitar.com<mailto:williamsjj@digitar.com> _______________________________________________ Outages mailing list <mailto:Outages@outages.org>Outages@outages.org<mailto:Outages@outages.org

<https://puck.nether.net/mailman/listinfo/outages> https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages ************* CONFIDENTIALITY NOTICE: This e-mail and the transmitted documents contain private, privileged and confidential information belonging to the sender. The information therein is solely for the use of the addressee. If your receipt of this transmission has occurred as the result of an error, please immediately notify us so we can arrange for the return of the documents. In such circumstances, you are advised that you may not disclose, copy, distribute or take any other action in reliance on the information transmitted.

I should have noted that my times were Central, so the same time as the other two reports. Frank From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Frank Bulk Sent: Wednesday, October 26, 2011 8:25 AM To: 'Jason J. W. Williams'; outages@outages.org Subject: Re: [outages] Qwest Re-Convergence I also received an alert at 5:07 which cleared at 5:11 am about access to the v6 sites of www.qwest.com and www.informationweek.com # traceroute6 www.qwest.com traceroute to www.qwest.com (2001:428:b21:1::20), 30 hops max, 80 byte packets 1 2607:fe28:0:1003::2 (2607:fe28:0:1003::2) 3.937 ms 4.363 ms 5.375 ms 2 router-core.mtcnet.net (2607:fe28:0:1000::1) 6.884 ms 7.300 ms 8.069 ms 3 sxct.movl.mtcnet.net (2607:fe28:11:1002::194) 8.708 ms 8.953 ms 9.779 ms 4 v6-siouxcenter.movl.153.netins.net (2001:5f8:7f06::9) 12.212 ms 12.452 ms 12.825 ms 5 v6-ins-kb1-te-13-2-220.kmrr.netins.net (2001:5f8:2:2::1) 18.798 ms 19.071 ms 19.416 ms 6 v6-ins-kc2-te-8-3.kmrr.netins.net (2001:5f8::11:1) 19.755 ms 15.928 ms 16.124 ms 7 v6-ins-dc1-et-8-1.desm.netins.net (2001:5f8::8:1) 17.461 ms 16.944 ms 16.998 ms 8 2001:428:3801:210:0:1:0:1 (2001:428:3801:210:0:1:0:1) 29.131 ms 28.281 ms 28.456 ms 9 * * * 10 2001:428:b21:fc03::2 (2001:428:b21:fc03::2) 43.020 ms 43.045 ms 43.362 ms 11 * * * 12 2001:428:b21:1::20 (2001:428:b21:1::20) 44.318 ms 44.506 ms 44.730 ms # traceroute6 www.informationweek.com traceroute to www.informationweek.com (2001:428:1002:10:0:3:0:150), 30 hops max, 80 byte packets 1 2607:fe28:0:1003::2 (2607:fe28:0:1003::2) 3.949 ms 4.387 ms 5.835 ms 2 router-core.mtcnet.net (2607:fe28:0:1000::1) 6.162 ms 7.018 ms 7.658 ms 3 sxct.sxcy.mtcnet.net (2607:fe28:11:1002::197) 7.974 ms 8.798 ms 9.563 ms 4 v6-siouxcenter.sxcy.137.netins.net (2001:5f8:7f06::5) 11.640 ms 11.949 ms 12.261 ms 5 v6-ins-db1-te-13-2-219.desm.netins.net (2001:5f8:1:1::1) 16.679 ms 16.882 ms 17.242 ms 6 v6-ins-dc1-et-8-2.desm.netins.net (2001:5f8::1:1) 19.425 ms 15.611 ms 15.720 ms 7 2001:428:3801:210:0:1:0:1 (2001:428:3801:210:0:1:0:1) 27.257 ms 27.094 ms 26.704 ms 8 2001:428::205:171:203:118 (2001:428::205:171:203:118) 73.502 ms 73.745 ms 73.336 ms 9 2001:428:1002:10:0:3:0:150 (2001:428:1002:10:0:3:0:150) 74.478 ms 69.405 ms 71.713 ms nagios:/home/fbulk# Frank From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Jason J. W. Williams Sent: Wednesday, October 26, 2011 5:31 AM To: outages@outages.org Subject: [outages] Qwest Re-Convergence Has anyone noticed issues reaching Qwest destinations around 4:05a-4:07a Mountain Time today? We received alerts that two of our sites that happened to have Qwest connectivity in common were unavailable during that period. Both sites being monitored were in different facilities but both in Boise. -J Traceroute #1 1 ip-67-205-112-66.static.privatedns.com (67.205.112.66) 0.842 ms 0.908 ms 0.936 ms 2 te8-4.v0707.cl-core04.mtl.iweb.com (67.205.127.121) 1.047 ms 1.068 ms 1.089 ms 3 if-9-1.icore1.MTT-Montreal.as6453.net (206.82.135.29) 7.737 ms 7.813 ms 7.837 ms 4 if-10-0-0-19.mcore3.MTT-Montreal.as6453.net (216.6.114.13) 0.907 ms 0.935 ms 0.955 ms 5 if-9-0-0-0.tcore2.NYY-NewYork.as6453.net (216.6.99.9) 7.871 ms 7.904 ms 7.926 ms 6 if-11-2.tcore1.NYY-NewYork.as6453.net (216.6.99.2) 7.815 ms 8.013 ms 8.032 ms 7 jfk-brdr-04.inet.qwest.net (63.146.27.41) 7.942 ms 7.827 ms 7.790 ms Traceroute #2 1 207.97.207.194 (207.97.207.194) 0.549 ms 0.578 ms 0.620 ms 2 core1-5.iad1.rackspace.com (69.20.2.18) 0.303 ms 0.354 ms 0.400 ms 3 vlan901.edge3.iad1.rackspace.com (69.20.1.3) 196.221 ms 196.284 ms 196.328 ms 4 dcx2-edge-01.inet.qwest.net (67.133.246.157) 0.368 ms 0.355 ms 0.349 ms ------------------- Jason J. W. Williams COO/CTO, DigiTar williamsjj@digitar.com V: 208.343.8520 M: 208.863.0727 F: 208.322.8522 IM: williamsjj@digitar.com