Comcast - Seattle (maybe other areas)
Having serious issues getting outside of Comcast's network starting around 1250 Pacific time from Seattle area. -- "Genius might be described as a supreme capacity for getting its possessors into trouble of all kinds." -- Samuel Butler
Michael, Can you share some traces? Thanks, John On Mon, Oct 20, 2014 at 1:57 PM, Michael Loftis via Outages < outages@outages.org> wrote:
Having serious issues getting outside of Comcast's network starting around 1250 Pacific time from Seattle area.
--
"Genius might be described as a supreme capacity for getting its possessors into trouble of all kinds." -- Samuel Butler _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Its like alost like this netblock (24.16.0.0/13) I'm in isn't getting announced to transit or is getting filtered at transit - just paid peerings... loneos:~ mloftis$ traceroute www.mysql.com traceroute to mysql-adc.oracle.com (137.254.60.6), 64 hops max, 52 byte packets 1 v6fw (10.70.0.1) 3.065 ms 0.798 ms 1.448 ms 2 96.120.101.1 (96.120.101.1) 12.289 ms 9.274 ms 9.744 ms 3 te-0-1-0-0-sur02.bellevue.wa.seattle.comcast.net (68.86.113.145) 11.498 ms 8.708 ms 14.447 ms 4 be-40-ar01.burien.wa.seattle.comcast.net (69.139.164.2) 12.184 ms be-1-sur03.bellevue.wa.seattle.comcast.net (68.86.96.142) 9.187 ms be-40-ar01.burien.wa.seattle.comcast.net (69.139.164.2) 11.595 ms 5 he-0-15-0-1-ar01.seattle.wa.seattle.comcast.net (68.85.240.94) 12.756 ms be-40-ar01.seattle.wa.seattle.comcast.net (69.139.164.6) 12.613 ms 21.782 ms 6 he-1-3-0-0-10-cr01.seattle.wa.ibone.comcast.net (68.86.93.165) 12.035 ms 12.944 ms 11.011 ms 7 * * * loneos:~ mloftis$ traceroute www.he.net traceroute to he.net (216.218.186.2), 64 hops max, 52 byte packets 1 v6fw (10.70.0.1) 1.205 ms 1.218 ms 1.103 ms 2 96.120.101.1 (96.120.101.1) 8.987 ms 8.738 ms 9.187 ms 3 te-0-1-0-0-sur02.bellevue.wa.seattle.comcast.net (68.86.113.145) 9.956 ms 9.791 ms 9.429 ms 4 be-40-ar01.burien.wa.seattle.comcast.net (69.139.164.2) 13.325 ms be-1-sur03.bellevue.wa.seattle.comcast.net (68.86.96.142) 16.625 ms 16.423 ms 5 be-40-ar01.seattle.wa.seattle.comcast.net (69.139.164.6) 10.978 ms 11.470 ms he-0-15-0-1-ar01.seattle.wa.seattle.comcast.net (68.85.240.94) 11.714 ms 6 he-1-3-0-0-10-cr01.seattle.wa.ibone.comcast.net (68.86.93.165) 14.815 ms 12.986 ms 11.557 ms 7 * * * loneos:~ mloftis$ traceroute www.google.com traceroute: Warning: www.google.com has multiple addresses; using 74.125.20.99 traceroute to www.google.com (74.125.20.99), 64 hops max, 52 byte packets 1 v6fw (10.70.0.1) 1.390 ms 1.103 ms 1.116 ms 2 96.120.101.1 (96.120.101.1) 9.317 ms 8.923 ms 10.132 ms 3 te-0-2-0-0-sur03.bellevue.wa.seattle.comcast.net (68.86.113.153) 9.267 ms 11.765 ms 10.088 ms 4 be-40-ar01.seattle.wa.seattle.comcast.net (69.139.164.6) 10.993 ms 23.054 ms 11.369 ms 5 he-1-3-0-0-10-cr01.seattle.wa.ibone.comcast.net (68.86.93.165) 15.023 ms 13.000 ms 14.430 ms 6 be-12-pe03.56marietta.ga.ibone.comcast.net (68.86.82.234) 10.968 ms 11.789 ms 14.294 ms 7 * * * loneos:~ mloftis$ traceroute 8.8.8.8 traceroute to 8.8.8.8 (8.8.8.8), 64 hops max, 52 byte packets 1 v6fw (10.70.0.1) 1.382 ms 1.200 ms 1.118 ms 2 96.120.101.1 (96.120.101.1) 7.683 ms 7.869 ms 9.136 ms 3 te-0-1-0-1-sur02.bellevue.wa.seattle.comcast.net (68.86.113.141) 9.527 ms 9.973 ms 9.871 ms 4 be-40-ar01.burien.wa.seattle.comcast.net (69.139.164.2) 12.974 ms be-1-sur03.bellevue.wa.seattle.comcast.net (68.86.96.142) 12.171 ms 11.826 ms 5 be-40-ar01.seattle.wa.seattle.comcast.net (69.139.164.6) 12.322 ms 10.624 ms 26.744 ms 6 he-1-3-0-0-10-cr01.seattle.wa.ibone.comcast.net (68.86.93.165) 17.635 ms 13.682 ms 12.770 ms 7 he-0-12-0-1-pe04.seattle.wa.ibone.comcast.net (68.86.82.230) 11.546 ms 13.296 ms 12.413 ms 8 * * * loneos:~ mloftis$ traceroute netflix.com traceroute to netflix.com (69.53.236.17), 64 hops max, 52 byte packets 1 v6fw (10.70.0.1) 1.420 ms 0.927 ms 0.981 ms 2 96.120.101.1 (96.120.101.1) 11.685 ms 15.119 ms 8.369 ms 3 te-0-1-0-1-sur02.bellevue.wa.seattle.comcast.net (68.86.113.141) 24.195 ms 9.636 ms 9.928 ms 4 be-40-ar01.burien.wa.seattle.comcast.net (69.139.164.2) 13.081 ms be-1-sur03.bellevue.wa.seattle.comcast.net (68.86.96.142) 10.842 ms be-40-ar01.burien.wa.seattle.comcast.net (69.139.164.2) 13.250 ms 5 be-40-ar01.seattle.wa.seattle.comcast.net (69.139.164.6) 11.050 ms 11.298 ms 11.361 ms 6 he-1-3-0-0-10-cr01.seattle.wa.ibone.comcast.net (68.86.93.165) 14.352 ms 34.253 ms 13.392 ms 7 he-0-13-0-0-cr01.sanjose.ca.ibone.comcast.net (68.86.85.162) 30.279 ms 31.711 ms 35.863 ms 8 he-0-10-0-0-pe03.11greatoaks.ca.ibone.comcast.net (68.86.85.214) 29.173 ms 31.250 ms 29.302 ms 9 * * * On Mon, Oct 20, 2014 at 1:12 PM, John Neiberger <jneiberger@gmail.com> wrote:
Michael,
Can you share some traces?
Thanks, John
On Mon, Oct 20, 2014 at 1:57 PM, Michael Loftis via Outages <outages@outages.org> wrote:
Having serious issues getting outside of Comcast's network starting around 1250 Pacific time from Seattle area.
--
"Genius might be described as a supreme capacity for getting its possessors into trouble of all kinds." -- Samuel Butler _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- "Genius might be described as a supreme capacity for getting its possessors into trouble of all kinds." -- Samuel Butler
It seems to have now mysteriously gone away/cleared up. Not at all sure what was going on there. I'd written it off initially as an issue with my laptop (OSX since a while back seems to need a reboot every few weeks if you've Parallels installed and use it - something inside the network stack goes nuts) On Mon, Oct 20, 2014 at 1:12 PM, John Neiberger <jneiberger@gmail.com> wrote:
Michael,
Can you share some traces?
Thanks, John
On Mon, Oct 20, 2014 at 1:57 PM, Michael Loftis via Outages <outages@outages.org> wrote:
Having serious issues getting outside of Comcast's network starting around 1250 Pacific time from Seattle area.
--
"Genius might be described as a supreme capacity for getting its possessors into trouble of all kinds." -- Samuel Butler _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- "Genius might be described as a supreme capacity for getting its possessors into trouble of all kinds." -- Samuel Butler
Okay, good to hear! That's very odd. Let me know if it happens again and you need someone to check into it. Thanks, John On Mon, Oct 20, 2014 at 2:24 PM, Michael Loftis <mloftis@wgops.com> wrote:
It seems to have now mysteriously gone away/cleared up. Not at all sure what was going on there. I'd written it off initially as an issue with my laptop (OSX since a while back seems to need a reboot every few weeks if you've Parallels installed and use it - something inside the network stack goes nuts)
On Mon, Oct 20, 2014 at 1:12 PM, John Neiberger <jneiberger@gmail.com> wrote:
Michael,
Can you share some traces?
Thanks, John
On Mon, Oct 20, 2014 at 1:57 PM, Michael Loftis via Outages <outages@outages.org> wrote:
Having serious issues getting outside of Comcast's network starting around 1250 Pacific time from Seattle area.
--
"Genius might be described as a supreme capacity for getting its possessors into trouble of all kinds." -- Samuel Butler _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
--
"Genius might be described as a supreme capacity for getting its possessors into trouble of all kinds." -- Samuel Butler
Level3’s experiencing a massive back hoe outage. Reports started in Seattle, but now is being reported across the West Coast - updates from there still open ticket: Reverse cron 10/20/14 7:10:56 PM GMT The IP NOC continues to work with the Transport NOC to develop additional services routes so the impacted IP traffic can be migrated to relieve some overutilization. Simultaneously, the Transport NOC has advised that Field Services have arrived onsite and are now assessing the damage while working develop a repair strategy for this case. This is no ETTR for this case at this time. 10/20/14 6:29:31 PM GMT The IP NOC has specified that while working to add the necessary bandwidth to mitigate to overutilization causing the IP and Voice service impact, they are also working to re-route services, when possible, to expedite service restoral. The troubleshooting process for this case is ongoing and the IP NOC is unable to provide an ETTR for this case at this time. 10/20/14 6:07:50 PM GMT The IPNOC and Transport NOCs efforts to increase the bandwidth between the over utilized markets is being done to expedite service restoral for the impacted markets. The Transport NOC is also working to repair the damaged cable in El Centro, CA. At this time, Field services has isolated the root cause of the break to a construction project and outside plant engineers have been dispatched to the damage site with an ETA of 18:30 GMT. A splice crew has also been dispatched to the failure site with an ETA of 20:00 GMT. 10/20/14 5:56:31 PM GMT The IP NOC has advised that the fiber damage was impacting their connectivity to the impacted equipment in the various markets. The IP NOC has resolved the connectivity issue and continues to work with the Transport NOC to increase the bandwidth in the impacted markets. 10/20/14 4:56:40 PM GMT The IP NOC has advised that this issue is now impacting multiple markets across the Western United States the event case has been adjusted to reflect the change. The IP NOC has determined that a fiber cut in El Centro, CA has caused multiple route failures between various these western. Specifically, the fiber damage impacted multiple backbones and has caused overutilization between these markets. The IP NOC has engaged the Transport NOC help mitigate the Voice and IP service failure for this case. 10/20/14 4:04:09 PM GMT Multiple routing protocol failures in Seattle, WA are impacting IP and voice services. The IP NOC has engaged Senior Level Management to help troubleshoot this issue. The root cause investigation for this case is ongoing at this time. On Oct 20, 2014, at 4:12 PM, John Neiberger via Outages <outages@outages.org> wrote:
Michael,
Can you share some traces?
Thanks, John
On Mon, Oct 20, 2014 at 1:57 PM, Michael Loftis via Outages <outages@outages.org> wrote: Having serious issues getting outside of Comcast's network starting around 1250 Pacific time from Seattle area.
--
"Genius might be described as a supreme capacity for getting its possessors into trouble of all kinds." -- Samuel Butler _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
participants (3)
-
Andrew Losey -
John Neiberger -
Michael Loftis