Meant to send this to the whole list. Sorry!
I’m seeing a few reports via Twitter that Level3 is having issues. DownDetector shows a massive number of sites with high latency on their home page as well. Our Comcast link in Hartford, CT is going nowhere. Pings are off the charts and packet loss is quite high.
Jim
Sent from my iPhone 7 Plus
On 6 Nov 2017, at 13:25, Nate Schmoll via Outages <outages@outages.org> wrote:
anyone else is seeing lots of packet loss and 100ms+ latency mostly in east coast on comcast network?
Experiencing the same thing here in Everett, WA, US.
~Nate _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
It’s not just Comcast. Seeing Level3 affected sites/customers as well. On 11/6/17, 10:38 AM, "jstefanik Outages via Outages" <outages@outages.org> wrote: EXTERNAL MAIL: outages-bounces@outages.org Meant to send this to the whole list. Sorry! > I’m seeing a few reports via Twitter that Level3 is having issues. DownDetector shows a massive number of sites with high latency on their home page as well. Our Comcast link in Hartford, CT is going nowhere. Pings are off the charts and packet loss is quite high. > > Jim > > Sent from my iPhone 7 Plus > > On 6 Nov 2017, at 13:25, Nate Schmoll via Outages <outages@outages.org> wrote: > >>> anyone else is seeing lots of packet loss and 100ms+ latency mostly in east coast on comcast network? >> >> Experiencing the same thing here in Everett, WA, US. >> >> ~Nate >> _______________________________________________ >> Outages mailing list >> Outages@outages.org >> https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Here's what it looks like from the inside. Both v4 and v6 affected with some peers. $ sudo mtr -4webz facebook.com Start: 2017-11-06T10:52:14-0800 1. AS??? .... 0.0% 10 1.4 2.0 1.4 4.6 1.0 2. AS7922 96.120.14.185 0.0% 10 15.3 14.1 10.5 28.1 5.6 3. AS7922 ae-211-sur01.toyon.ca.ccal.comcast.net (68.85.104.205) 0.0% 10 11.8 12.5 10.4 15.6 1.5 4. AS7922 ae-33-ar01.fresno.ca.ccal.comcast.net (68.87.200.5) 0.0% 10 15.2 16.3 14.9 20.4 1.7 5. AS7922 be-33667-cr02.losangeles.ca.ibone.comcast.net (68.86.93.37) 0.0% 10 23.6 23.9 22.3 28.4 2.2 6. AS7922 be-11015-cr02.sunnyvale.ca.ibone.comcast.net (68.86.86.97) 0.0% 10 24.0 25.5 22.9 30.5 2.5 7. AS7922 be-11025-cr01.9greatoaks.ca.ibone.comcast.net (68.86.87.158) 0.0% 10 24.7 26.1 24.6 28.4 1.4 8. AS7922 be-12578-pe04.9greatoaks.ca.ibone.comcast.net (68.86.88.18) 0.0% 10 23.5 23.5 22.7 25.8 0.9 9. AS??? ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 10. AS??? ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 11. AS32934 po221.psw03.sjc2.tfbnw.net (74.119.79.93) 30.0% 10 215.9 221.9 213.5 244.9 10.7 12. AS32934 173.252.67.135 50.0% 10 219.0 217.9 207.8 228.1 7.3 13. AS32934 edge-star-mini-shv-01-sjc2.facebook.com (31.13.77.36) 70.0% 10 221.1 218.1 213.6 221.1 3.9 $ sudo mtr -6webz facebook.com Start: 2017-11-06T10:52:48-0800 1. AS7922 2601:205:...... 0.0% 10 96.2 11.0 1.3 96.2 29.9 2. AS7922 2001:558:4010:11::1 0.0% 10 12.3 12.0 11.1 13.3 0.7 3. AS7922 xe-11-1-0-sur01.toyon.ca.ccal.comcast.net (2001:558:1a2:9800::1) 0.0% 10 10.9 11.7 10.3 15.9 1.6 4. AS7922 ae-34-ar01.sacramento.ca.ccal.comcast.net (2001:558:210:f1::1) 0.0% 10 15.0 20.3 13.2 65.0 15.8 5. AS7922 be-33667-cr01.9greatoaks.ca.ibone.comcast.net (2001:558:0:f746::1) 0.0% 10 17.5 18.3 17.4 19.3 0.7 6. AS7922 be-12578-pe04.9greatoaks.ca.ibone.comcast.net (2001:558:0:f604::2) 0.0% 10 22.3 19.0 16.1 23.1 2.3 7. AS7922 2001:559::10a6 60.0% 10 221.4 214.0 204.7 221.4 7.0 8. AS??? ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 9. AS32934 po231.psw01.sjc2.tfbnw.net (2620:0:1cff:dead:beef::e5f) 40.0% 10 205.2 206.1 201.5 221.9 7.9 10. AS32934 po1.msw1al.01.sjc2.tfbnw.net (2a03:2880:f022:ffff::17) 70.0% 10 201.4 205.9 201.4 212.2 5.6 11. AS32934 edge-star-mini6-shv-01-sjc2.facebook.com (2a03:2880:f122:83:face:b00c::25de) 40.0% 10 202.5 204.6 197.3 211.8 4.7 ../C On Mon, Nov 6, 2017 at 10:39 AM, Donn Lasher via Outages < outages@outages.org> wrote:
It’s not just Comcast. Seeing Level3 affected sites/customers as well.
On 11/6/17, 10:38 AM, "jstefanik Outages via Outages" <outages@outages.org> wrote:
EXTERNAL MAIL: outages-bounces@outages.org
Meant to send this to the whole list. Sorry!
> I’m seeing a few reports via Twitter that Level3 is having issues. DownDetector shows a massive number of sites with high latency on their home page as well. Our Comcast link in Hartford, CT is going nowhere. Pings are off the charts and packet loss is quite high. > > Jim > > Sent from my iPhone 7 Plus > > On 6 Nov 2017, at 13:25, Nate Schmoll via Outages < outages@outages.org> wrote: > >>> anyone else is seeing lots of packet loss and 100ms+ latency mostly in east coast on comcast network? >> >> Experiencing the same thing here in Everett, WA, US. >> >> ~Nate >> _______________________________________________ >> Outages mailing list >> Outages@outages.org >> https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
If I had to make a guess, a bunch of more-specific routes are propagated into the Default-Free Zone that should not be there: BGP routing table entry for 174.54.0.0/15 2914 3356 7016 Nexthop 165.254.255.1 (via 165.254.255.1) from AS15562_scarlett_IPv4 (165.254.255.1) Origin IGP, metric 1016, localpref 100, weight 0, internal, valid, best Last update: 01:24:29 ago Communities: 2914:420 2914:1001 2914:2000 2914:3000 65504:3356 Originator Id: 165.254.255.1 Cluster ID List: 165.254.255.26 This is a more-specific for 174.48.0.0/12 (originated by AS 7922). These more-specifics win over less-specifics, so the world's traffic to comcast may now be flowing through level3, and maybe there is not sufficient bandwidth capacity available on those Level3<>Regional_Comcast interconnections. Kind regards, Job
we are also hearing that the problem is a level3 backbone outage in Chicago On Mon, Nov 6, 2017 at 12:32 PM, jstefanik Outages via Outages < outages@outages.org> wrote:
Meant to send this to the whole list. Sorry!
I’m seeing a few reports via Twitter that Level3 is having issues. DownDetector shows a massive number of sites with high latency on their home page as well. Our Comcast link in Hartford, CT is going nowhere. Pings are off the charts and packet loss is quite high.
Jim
Sent from my iPhone 7 Plus
On 6 Nov 2017, at 13:25, Nate Schmoll via Outages <outages@outages.org> wrote:
anyone else is seeing lots of packet loss and 100ms+ latency mostly in east coast on comcast network?
Experiencing the same thing here in Everett, WA, US.
~Nate _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
From here it looks like a problem around NTT/Level3 handoff. Relevant snippit from mtr going from our colo to our Comcast business line...
6. ae-11.r00.lsanca07.us.bb.gin.ntt.net 0.0% 128 71.8 71.8 71.7 72.3 0.1 7. ae-2.r23.lsanca07.us.bb.gin.ntt.net 0.0% 128 10.2 10.7 10.1 19.0 1.4 8. ae-6.r22.asbnva02.us.bb.gin.ntt.net 11.7% 128 75.2 75.4 75.0 81.0 1.0 9. ae-1.r05.asbnva02.us.bb.gin.ntt.net 0.0% 128 71.0 71.1 71.0 71.6 0.2 10. ae-0.a02.asbnva02.us.bb.gin.ntt.net 18.9% 128 70.3 71.7 69.8 102.5 5.2 11. lag-12.ear2.Washington12.Level3.net 99.2% 127 7085. 7085. 7085. 7085. 0.0 12. ae-1-3501.ear3.SanJose1.Level3.net 97.6% 127 19439 19547 18857 20346 750.4 13. ComcastRegional-level3-200G.SanJose1.Level3.net 51.6% 127 191.5 195.1 184.5 204.8 3.5 On Mon, Nov 6, 2017 at 10:32 AM, jstefanik Outages via Outages < outages@outages.org> wrote:
Meant to send this to the whole list. Sorry!
I’m seeing a few reports via Twitter that Level3 is having issues. DownDetector shows a massive number of sites with high latency on their home page as well. Our Comcast link in Hartford, CT is going nowhere. Pings are off the charts and packet loss is quite high.
Jim
Sent from my iPhone 7 Plus
On 6 Nov 2017, at 13:25, Nate Schmoll via Outages <outages@outages.org> wrote:
anyone else is seeing lots of packet loss and 100ms+ latency mostly in east coast on comcast network?
Experiencing the same thing here in Everett, WA, US.
~Nate _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Definitely seeing sporadic connection issues in Chicago. From IL office to FL colo. 7 18 ms 38 ms 28 ms be-33491-cr02.350ecermak.il.ibone.comcast.net [68.86.91.165] 8 37 ms 26 ms 28 ms be-10577-pe03.350ecermak.il.ibone.comcast.net [68.86.86.2] 9 * 170 ms * ae-26.a02.chcgil09.us.bb.gin.ntt.net [129.250.66.65] 10 * * 190 ms ae-3.r07.chcgil09.us.bb.gin.ntt.net [129.250.5.111] 11 * 170 ms 167 ms ae-5.r21.chcgil09.us.bb.gin.ntt.net [129.250.4.213] 12 163 ms 157 ms * ae-1.r23.asbnva02.us.bb.gin.ntt.net [129.250.2.138] 13 176 ms 182 ms * ae-1.r20.miamfl02.us.bb.gin.ntt.net [129.250.2.87] 14 * 178 ms 186 ms ae-1.r04.miamfl02.us.bb.gin.ntt.net [129.250.2.111] 15 189 ms 172 ms 173 ms ae-7.a00.miamfl02.us.bb.gin.ntt.net [129.250.2.203] 16 * 187 ms * xe-0-0-47-2.a00.miamfl02.bboi.net [157.238.179.114] 17 * 202 ms * boc-ten1-5-mia-ten3-1.bboi.net [66.216.1.246] 18 * 195 ms 198 ms host-boca-bboi.bboi.net [66.216.1.31] 19 * 185 ms 210 ms te3-4.cr01.900.bct.host.net [64.135.0.97] 20 * 315 ms * te2-1.cr01.hst.fll.host.net [64.135.1.199] 21 * * * Request timed out. 22 * * * Request timed out. 23 * * * Request timed out. 24 * * * Request timed out. 25 * * * Request timed out. 26 * * * Request timed out. 27 * * * Request timed out. 28 * * * Request timed out. 29 * * * Request timed out. 30 * * * Request timed out. ________________________________ From: Outages <outages-bounces@outages.org> on behalf of Alex Bame via Outages <outages@outages.org> Sent: Monday, November 6, 2017 12:41:36 PM Cc: outages@outages.org Subject: Re: [outages] Fwd: cocmast packetloss and high latency
From here it looks like a problem around NTT/Level3 handoff. Relevant snippit from mtr going from our colo to our Comcast business line...
6. ae-11.r00.lsanca07.us.bb.gin.ntt.net<http://ae-11.r00.lsanca07.us.bb.gin.ntt.net> 0.0% 128 71.8 71.8 71.7 72.3 0.1 7. ae-2.r23.lsanca07.us.bb.gin.ntt.net<http://ae-2.r23.lsanca07.us.bb.gin.ntt.net> 0.0% 128 10.2 10.7 10.1 19.0 1.4 8. ae-6.r22.asbnva02.us.bb.gin.ntt.net<http://ae-6.r22.asbnva02.us.bb.gin.ntt.net> 11.7% 128 75.2 75.4 75.0 81.0 1.0 9. ae-1.r05.asbnva02.us.bb.gin.ntt.net<http://ae-1.r05.asbnva02.us.bb.gin.ntt.net> 0.0% 128 71.0 71.1 71.0 71.6 0.2 10. ae-0.a02.asbnva02.us.bb.gin.ntt.net<http://ae-0.a02.asbnva02.us.bb.gin.ntt.net> 18.9% 128 70.3 71.7 69.8 102.5 5.2 11. lag-12.ear2.Washington12.Level3.net<http://lag-12.ear2.Washington12.Level3.net> 99.2% 127 7085. 7085. 7085. 7085. 0.0 12. ae-1-3501.ear3.SanJose1.Level3.net<http://ae-1-3501.ear3.SanJose1.Level3.net> 97.6% 127 19439 19547 18857 20346 750.4 13. ComcastRegional-level3-200G.SanJose1.Level3.net<http://ComcastRegional-level3-200G.SanJose1.Level3.net> 51.6% 127 191.5 195.1 184.5 204.8 3.5 On Mon, Nov 6, 2017 at 10:32 AM, jstefanik Outages via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: Meant to send this to the whole list. Sorry!
I’m seeing a few reports via Twitter that Level3 is having issues. DownDetector shows a massive number of sites with high latency on their home page as well. Our Comcast link in Hartford, CT is going nowhere. Pings are off the charts and packet loss is quite high.
Jim
Sent from my iPhone 7 Plus
On 6 Nov 2017, at 13:25, Nate Schmoll via Outages <outages@outages.org<mailto:outages@outages.org>> wrote:
anyone else is seeing lots of packet loss and 100ms+ latency mostly in east coast on comcast network?
Experiencing the same thing here in Everett, WA, US.
~Nate _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
Rumor has it this is fixed now
Downdetector isn’t really that accurate of a report. Its been lit red like that for over a year since the last times that I've checked the site. Id suspect some leaked routes along the wire somewhere that is effecting connectivity for a lot of carriers.
On Nov 6, 2017, at 12:32, jstefanik Outages via Outages <outages@outages.org> wrote:
Meant to send this to the whole list. Sorry!
I’m seeing a few reports via Twitter that Level3 is having issues. DownDetector shows a massive number of sites with high latency on their home page as well. Our Comcast link in Hartford, CT is going nowhere. Pings are off the charts and packet loss is quite high.
Jim
Sent from my iPhone 7 Plus
On 6 Nov 2017, at 13:25, Nate Schmoll via Outages <outages@outages.org> wrote:
anyone else is seeing lots of packet loss and 100ms+ latency mostly in east coast on comcast network?
Experiencing the same thing here in Everett, WA, US.
~Nate _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
participants (8)
-
Alex Bame -
Artur Tyksinski -
Curtis Doty -
Donn Lasher -
Gary Leato -
Jason Hellenthal -
Job Snijders -
jstefanik Outages