Intermittent Level 3 outage/connectivity issues on Dallas edges?
Seeing connectivity issues as traffic passes through Level 3 in Dallas - not seeing the same issues for where routes converge (above.net in DFW) over our TW Telecom link. Anybody else experiencing this as well? Level 3: Host Loss% Snt Last Avg Best Wrst StDev 1. vlan358.car1.Houston1.Level3.net 0.0% 167 105.9 80.5 31.7 276.4 59.8 2. ae-2-5.bar1.Houston1.Level3.net 0.0% 167 32.5 34.6 32.2 93.8 7.3 3. ae-13-13.ebr1.Dallas1.Level3.net 0.0% 167 39.5 37.4 36.7 41.2 0.9 4. ae-91-91.csw4.Dallas1.Level3.net 65.1% 167 44.5 43.5 39.7 53.8 4.0 5. ae-4-90.edge2.Dallas3.Level3.net 5.4% 166 37.1 37.9 36.1 75.6 5.1 6. ex-3-1-0.er1.dfw2.us.above.net 66.1% 166 39.5 43.5 38.9 104.8 12.5 7. xe-3-1-0.cr1.dfw2.us.above.net 0.0% 166 37.3 40.4 37.0 114.1 10.6 8. xe-0-0-0.er3.dfw2.us.above.net 0.0% 166 36.2 39.8 36.1 99.3 10.2 Host Loss% Snt Last Avg Best Wrst StDev 1. 50.58.18.1 0.0% 829 4.3 23.9 4.1 121.2 14.5 2. dal2-pr2-xe-0-3-0-0.us.twtelecom.net 0.1% 829 55.4 31.2 9.4 121.0 17.2 dal2-pr2-xe-2-3-0-0.us.twtelecom.net 3. xe-3-0-0.er2.dfw2.us.above.net 0.1% 829 44.8 31.6 9.5 143.7 17.9 4. xe-0-1-0.cr2.dfw2.us.above.net 0.2% 829 40.5 31.2 10.3 116.2 15.9 xe-3-0-0.cr2.dfw2.us.above.net 5. xe-0-2-0.er3.dfw2.us.above.net 0.2% 829 42.7 32.4 9.6 109.2 16.5 Regards, Chris Boulton Bigcommerce Email: chris@bigcommerce.com Web: http://www.bigcommerce.com/
Please see my below post on this matter, which I posted only 2 days ago: https://puck.nether.net/pipermail/outages/2013-January/005111.html What you've shown in your mtr does not show any issue of any kind with Level 3. The packet loss shown at hops 4/5/6 does not "trickle down" each subsequent hop (look at hop 7 onward for proof), nor is there any sign of latency increase either. On the other hand, the 2nd mtr you provided shows some degree of packet loss starting at hop 2 (TW Telecom) which *does* "trickle down" through all subsequent hops (egress). If you want to know how many packets were lost, look at the --order flag to mtr (I recommend --order=LSRNABW). Plain and simple: talk to TW Telecom about the loss in their network. -- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB | On Fri, Jan 25, 2013 at 05:46:27AM +1100, Chris Boulton wrote:
Seeing connectivity issues as traffic passes through Level 3 in Dallas - not seeing the same issues for where routes converge (above.net in DFW) over our TW Telecom link.
Anybody else experiencing this as well?
Level 3:
Host Loss% Snt Last Avg Best Wrst StDev 1. vlan358.car1.Houston1.Level3.net 0.0% 167 105.9 80.5 31.7 276.4 59.8 2. ae-2-5.bar1.Houston1.Level3.net 0.0% 167 32.5 34.6 32.2 93.8 7.3 3. ae-13-13.ebr1.Dallas1.Level3.net 0.0% 167 39.5 37.4 36.7 41.2 0.9 4. ae-91-91.csw4.Dallas1.Level3.net 65.1% 167 44.5 43.5 39.7 53.8 4.0 5. ae-4-90.edge2.Dallas3.Level3.net 5.4% 166 37.1 37.9 36.1 75.6 5.1 6. ex-3-1-0.er1.dfw2.us.above.net 66.1% 166 39.5 43.5 38.9 104.8 12.5 7. xe-3-1-0.cr1.dfw2.us.above.net 0.0% 166 37.3 40.4 37.0 114.1 10.6 8. xe-0-0-0.er3.dfw2.us.above.net 0.0% 166 36.2 39.8 36.1 99.3 10.2
Host Loss% Snt Last Avg Best Wrst StDev 1. 50.58.18.1 0.0% 829 4.3 23.9 4.1 121.2 14.5 2. dal2-pr2-xe-0-3-0-0.us.twtelecom.net 0.1% 829 55.4 31.2 9.4 121.0 17.2 dal2-pr2-xe-2-3-0-0.us.twtelecom.net 3. xe-3-0-0.er2.dfw2.us.above.net 0.1% 829 44.8 31.6 9.5 143.7 17.9 4. xe-0-1-0.cr2.dfw2.us.above.net 0.2% 829 40.5 31.2 10.3 116.2 15.9 xe-3-0-0.cr2.dfw2.us.above.net 5. xe-0-2-0.er3.dfw2.us.above.net 0.2% 829 42.7 32.4 9.6 109.2 16.5
Regards,
Chris Boulton Bigcommerce
Email: chris@bigcommerce.com
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Thanks, I am aware of what the traceroute shows and how a traceroute works. The intent of the traceroute was primarily to show where I see our paths between Level 3 and TW Telecom converge. Plain and simple - we are experiencing connectivity issues over our Level 3 connection, which we're not seeing over our TW Telecom connection for traffic destined to the same location. Regards, Chris Boulton Bigcommerce Email: chris@bigcommerce.com Web: http://www.bigcommerce.com/ On Fri, Jan 25, 2013 at 6:11 AM, Jeremy Chadwick <jdc@koitsu.org> wrote:
Please see my below post on this matter, which I posted only 2 days ago:
https://puck.nether.net/pipermail/outages/2013-January/005111.html
What you've shown in your mtr does not show any issue of any kind with Level 3. The packet loss shown at hops 4/5/6 does not "trickle down" each subsequent hop (look at hop 7 onward for proof), nor is there any sign of latency increase either.
On the other hand, the 2nd mtr you provided shows some degree of packet loss starting at hop 2 (TW Telecom) which *does* "trickle down" through all subsequent hops (egress). If you want to know how many packets were lost, look at the --order flag to mtr (I recommend --order=LSRNABW).
Plain and simple: talk to TW Telecom about the loss in their network.
-- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB |
On Fri, Jan 25, 2013 at 05:46:27AM +1100, Chris Boulton wrote:
Seeing connectivity issues as traffic passes through Level 3 in Dallas - not seeing the same issues for where routes converge (above.net in DFW) over our TW Telecom link.
Anybody else experiencing this as well?
Level 3:
Host Loss% Snt Last Avg Best Wrst StDev 1. vlan358.car1.Houston1.Level3.net 0.0% 167 105.9 80.5 31.7 276.4 59.8 2. ae-2-5.bar1.Houston1.Level3.net 0.0% 167 32.5 34.6 32.2 93.8 7.3 3. ae-13-13.ebr1.Dallas1.Level3.net 0.0% 167 39.5 37.4 36.7 41.2 0.9 4. ae-91-91.csw4.Dallas1.Level3.net 65.1% 167 44.5 43.5 39.7 53.8 4.0 5. ae-4-90.edge2.Dallas3.Level3.net 5.4% 166 37.1 37.9 36.1 75.6 5.1 6. ex-3-1-0.er1.dfw2.us.above.net 66.1% 166 39.5 43.5 38.9 104.8 12.5 7. xe-3-1-0.cr1.dfw2.us.above.net 0.0% 166 37.3 40.4 37.0 114.1 10.6 8. xe-0-0-0.er3.dfw2.us.above.net 0.0% 166 36.2 39.8 36.1 99.3 10.2
Host Loss% Snt Last Avg Best Wrst StDev 1. 50.58.18.1 0.0% 829 4.3 23.9 4.1 121.2 14.5 2. dal2-pr2-xe-0-3-0-0.us.twtelecom.net 0.1% 829 55.4 31.2 9.4 121.0 17.2 dal2-pr2-xe-2-3-0-0.us.twtelecom.net 3. xe-3-0-0.er2.dfw2.us.above.net 0.1% 829 44.8 31.6 9.5 143.7 17.9 4. xe-0-1-0.cr2.dfw2.us.above.net 0.2% 829 40.5 31.2 10.3 116.2 15.9 xe-3-0-0.cr2.dfw2.us.above.net 5. xe-0-2-0.er3.dfw2.us.above.net 0.2% 829 42.7 32.4 9.6 109.2 16.5
Regards,
Chris Boulton Bigcommerce
Email: chris@bigcommerce.com
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Chris, is there some reason you're not providing actual source/destination IPs for each of your mtrs? The reason I'm asking is because I was left to believe what you were providing was both ends of an asymmetric connection (i.e. mtr #1 was for x.x.x.x to y.y.y.y, mtr #2 was for y.y.y.y to x.x.x.x). However, your 2nd paragraph below ("we're seeing issues over one connections but not another") seems to indicate what you provided are two mtrs across completely different circuits. So which is it? -- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB | On Fri, Jan 25, 2013 at 06:14:05AM +1100, Chris Boulton wrote:
Thanks, I am aware of what the traceroute shows and how a traceroute works. The intent of the traceroute was primarily to show where I see our paths between Level 3 and TW Telecom converge.
Plain and simple - we are experiencing connectivity issues over our Level 3 connection, which we're not seeing over our TW Telecom connection for traffic destined to the same location.
Regards,
Chris Boulton Bigcommerce
Email: chris@bigcommerce.com
Web: http://www.bigcommerce.com/
On Fri, Jan 25, 2013 at 6:11 AM, Jeremy Chadwick <jdc@koitsu.org> wrote:
Please see my below post on this matter, which I posted only 2 days ago:
https://puck.nether.net/pipermail/outages/2013-January/005111.html
What you've shown in your mtr does not show any issue of any kind with Level 3. The packet loss shown at hops 4/5/6 does not "trickle down" each subsequent hop (look at hop 7 onward for proof), nor is there any sign of latency increase either.
On the other hand, the 2nd mtr you provided shows some degree of packet loss starting at hop 2 (TW Telecom) which *does* "trickle down" through all subsequent hops (egress). If you want to know how many packets were lost, look at the --order flag to mtr (I recommend --order=LSRNABW).
Plain and simple: talk to TW Telecom about the loss in their network.
-- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB |
On Fri, Jan 25, 2013 at 05:46:27AM +1100, Chris Boulton wrote:
Seeing connectivity issues as traffic passes through Level 3 in Dallas - not seeing the same issues for where routes converge (above.net in DFW) over our TW Telecom link.
Anybody else experiencing this as well?
Level 3:
Host Loss% Snt Last Avg Best Wrst StDev 1. vlan358.car1.Houston1.Level3.net 0.0% 167 105.9 80.5 31.7 276.4 59.8 2. ae-2-5.bar1.Houston1.Level3.net 0.0% 167 32.5 34.6 32.2 93.8 7.3 3. ae-13-13.ebr1.Dallas1.Level3.net 0.0% 167 39.5 37.4 36.7 41.2 0.9 4. ae-91-91.csw4.Dallas1.Level3.net 65.1% 167 44.5 43.5 39.7 53.8 4.0 5. ae-4-90.edge2.Dallas3.Level3.net 5.4% 166 37.1 37.9 36.1 75.6 5.1 6. ex-3-1-0.er1.dfw2.us.above.net 66.1% 166 39.5 43.5 38.9 104.8 12.5 7. xe-3-1-0.cr1.dfw2.us.above.net 0.0% 166 37.3 40.4 37.0 114.1 10.6 8. xe-0-0-0.er3.dfw2.us.above.net 0.0% 166 36.2 39.8 36.1 99.3 10.2
Host Loss% Snt Last Avg Best Wrst StDev 1. 50.58.18.1 0.0% 829 4.3 23.9 4.1 121.2 14.5 2. dal2-pr2-xe-0-3-0-0.us.twtelecom.net 0.1% 829 55.4 31.2 9.4 121.0 17.2 dal2-pr2-xe-2-3-0-0.us.twtelecom.net 3. xe-3-0-0.er2.dfw2.us.above.net 0.1% 829 44.8 31.6 9.5 143.7 17.9 4. xe-0-1-0.cr2.dfw2.us.above.net 0.2% 829 40.5 31.2 10.3 116.2 15.9 xe-3-0-0.cr2.dfw2.us.above.net 5. xe-0-2-0.er3.dfw2.us.above.net 0.2% 829 42.7 32.4 9.6 109.2 16.5
Regards,
Chris Boulton Bigcommerce
Email: chris@bigcommerce.com
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Ah, my apologies on the confusion. They're two distinct paths out, not traceroutes from each end. I chose not to include the destination IP because the issues we're seeing a more wide spread than to a single IP. Level 3 managed to get back to us - they were originally reporting back to us that they are experiencing connectivity issues for some of their clients in Austin, which they're now denying and still continuing to investigate. Regards, Chris Boulton Bigcommerce Email: chris@bigcommerce.com Web: http://www.bigcommerce.com/ On Fri, Jan 25, 2013 at 6:39 AM, Jeremy Chadwick <jdc@koitsu.org> wrote:
Chris, is there some reason you're not providing actual source/destination IPs for each of your mtrs?
The reason I'm asking is because I was left to believe what you were providing was both ends of an asymmetric connection (i.e. mtr #1 was for x.x.x.x to y.y.y.y, mtr #2 was for y.y.y.y to x.x.x.x).
However, your 2nd paragraph below ("we're seeing issues over one connections but not another") seems to indicate what you provided are two mtrs across completely different circuits.
So which is it?
-- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB |
On Fri, Jan 25, 2013 at 06:14:05AM +1100, Chris Boulton wrote:
Thanks, I am aware of what the traceroute shows and how a traceroute works. The intent of the traceroute was primarily to show where I see our paths between Level 3 and TW Telecom converge.
Plain and simple - we are experiencing connectivity issues over our Level 3 connection, which we're not seeing over our TW Telecom connection for traffic destined to the same location.
Regards,
Chris Boulton Bigcommerce
Email: chris@bigcommerce.com
Web: http://www.bigcommerce.com/
On Fri, Jan 25, 2013 at 6:11 AM, Jeremy Chadwick <jdc@koitsu.org> wrote:
Please see my below post on this matter, which I posted only 2 days ago:
https://puck.nether.net/pipermail/outages/2013-January/005111.html
What you've shown in your mtr does not show any issue of any kind with Level 3. The packet loss shown at hops 4/5/6 does not "trickle down" each subsequent hop (look at hop 7 onward for proof), nor is there any sign of latency increase either.
On the other hand, the 2nd mtr you provided shows some degree of packet loss starting at hop 2 (TW Telecom) which *does* "trickle down" through all subsequent hops (egress). If you want to know how many packets were lost, look at the --order flag to mtr (I recommend --order=LSRNABW).
Plain and simple: talk to TW Telecom about the loss in their network.
-- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB |
On Fri, Jan 25, 2013 at 05:46:27AM +1100, Chris Boulton wrote:
Seeing connectivity issues as traffic passes through Level 3 in Dallas - not seeing the same issues for where routes converge (above.net in DFW) over our TW Telecom link.
Anybody else experiencing this as well?
Level 3:
Host Loss% Snt Last Avg Best Wrst StDev 1. vlan358.car1.Houston1.Level3.net 0.0% 167 105.9 80.5 31.7 276.4 59.8 2. ae-2-5.bar1.Houston1.Level3.net 0.0% 167 32.5 34.6 32.2 93.8 7.3 3. ae-13-13.ebr1.Dallas1.Level3.net 0.0% 167 39.5 37.4 36.7 41.2 0.9 4. ae-91-91.csw4.Dallas1.Level3.net 65.1% 167 44.5 43.5 39.7 53.8 4.0 5. ae-4-90.edge2.Dallas3.Level3.net 5.4% 166 37.1 37.9 36.1 75.6 5.1 6. ex-3-1-0.er1.dfw2.us.above.net 66.1% 166 39.5 43.5 38.9 104.8 12.5 7. xe-3-1-0.cr1.dfw2.us.above.net 0.0% 166 37.3 40.4 37.0 114.1 10.6 8. xe-0-0-0.er3.dfw2.us.above.net 0.0% 166 36.2 39.8 36.1 99.3 10.2
Host Loss% Snt Last Avg Best Wrst StDev 1. 50.58.18.1 0.0% 829 4.3 23.9 4.1 121.2 14.5 2. dal2-pr2-xe-0-3-0-0.us.twtelecom.net 0.1% 829 55.4 31.2 9.4 121.0 17.2 dal2-pr2-xe-2-3-0-0.us.twtelecom.net 3. xe-3-0-0.er2.dfw2.us.above.net 0.1% 829 44.8 31.6 9.5 143.7 17.9 4. xe-0-1-0.cr2.dfw2.us.above.net 0.2% 829 40.5 31.2 10.3 116.2 15.9 xe-3-0-0.cr2.dfw2.us.above.net 5. xe-0-2-0.er3.dfw2.us.above.net 0.2% 829 42.7 32.4 9.6 109.2 16.5
Regards,
Chris Boulton Bigcommerce
Email: chris@bigcommerce.com
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
It looks like the situation has resolved itself in the last minute - I'm still waiting for an update from Level 3, but from 2 off-list replies it looks like there were a few people who've experienced routing issues with Level 3 in Houston today. Regards, Chris Boulton Bigcommerce Email: chris@bigcommerce.com Web: http://www.bigcommerce.com/ On Fri, Jan 25, 2013 at 6:45 AM, Chris Boulton <chris@bigcommerce.com>wrote:
Ah, my apologies on the confusion. They're two distinct paths out, not traceroutes from each end.
I chose not to include the destination IP because the issues we're seeing a more wide spread than to a single IP.
Level 3 managed to get back to us - they were originally reporting back to us that they are experiencing connectivity issues for some of their clients in Austin, which they're now denying and still continuing to investigate.
Regards,
Chris Boulton Bigcommerce
Email: chris@bigcommerce.com
Web: http://www.bigcommerce.com/
On Fri, Jan 25, 2013 at 6:39 AM, Jeremy Chadwick <jdc@koitsu.org> wrote:
Chris, is there some reason you're not providing actual source/destination IPs for each of your mtrs?
The reason I'm asking is because I was left to believe what you were providing was both ends of an asymmetric connection (i.e. mtr #1 was for x.x.x.x to y.y.y.y, mtr #2 was for y.y.y.y to x.x.x.x).
However, your 2nd paragraph below ("we're seeing issues over one connections but not another") seems to indicate what you provided are two mtrs across completely different circuits.
So which is it?
-- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB |
On Fri, Jan 25, 2013 at 06:14:05AM +1100, Chris Boulton wrote:
Thanks, I am aware of what the traceroute shows and how a traceroute works. The intent of the traceroute was primarily to show where I see our paths between Level 3 and TW Telecom converge.
Plain and simple - we are experiencing connectivity issues over our Level 3 connection, which we're not seeing over our TW Telecom connection for traffic destined to the same location.
Regards,
Chris Boulton Bigcommerce
Email: chris@bigcommerce.com
Web: http://www.bigcommerce.com/
On Fri, Jan 25, 2013 at 6:11 AM, Jeremy Chadwick <jdc@koitsu.org> wrote:
Please see my below post on this matter, which I posted only 2 days ago:
https://puck.nether.net/pipermail/outages/2013-January/005111.html
What you've shown in your mtr does not show any issue of any kind with Level 3. The packet loss shown at hops 4/5/6 does not "trickle down" each subsequent hop (look at hop 7 onward for proof), nor is there any sign of latency increase either.
On the other hand, the 2nd mtr you provided shows some degree of packet loss starting at hop 2 (TW Telecom) which *does* "trickle down" through all subsequent hops (egress). If you want to know how many packets were lost, look at the --order flag to mtr (I recommend --order=LSRNABW).
Plain and simple: talk to TW Telecom about the loss in their network.
-- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB |
On Fri, Jan 25, 2013 at 05:46:27AM +1100, Chris Boulton wrote:
Seeing connectivity issues as traffic passes through Level 3 in Dallas - not seeing the same issues for where routes converge (above.net in DFW) over our TW Telecom link.
Anybody else experiencing this as well?
Level 3:
Host Loss% Snt Last Avg Best Wrst StDev 1. vlan358.car1.Houston1.Level3.net 0.0% 167 105.9 80.5 31.7 276.4 59.8 2. ae-2-5.bar1.Houston1.Level3.net 0.0% 167 32.5 34.6 32.2 93.8 7.3 3. ae-13-13.ebr1.Dallas1.Level3.net 0.0% 167 39.5 37.4 36.7 41.2 0.9 4. ae-91-91.csw4.Dallas1.Level3.net 65.1% 167 44.5 43.5 39.7 53.8 4.0 5. ae-4-90.edge2.Dallas3.Level3.net 5.4% 166 37.1 37.9 36.1 75.6 5.1 6. ex-3-1-0.er1.dfw2.us.above.net 66.1% 166 39.5 43.5 38.9 104.8 12.5 7. xe-3-1-0.cr1.dfw2.us.above.net 0.0% 166 37.3 40.4 37.0 114.1 10.6 8. xe-0-0-0.er3.dfw2.us.above.net 0.0% 166 36.2 39.8 36.1 99.3 10.2
Host Loss% Snt Last Avg Best Wrst StDev 1. 50.58.18.1 0.0% 829 4.3 23.9 4.1 121.2 14.5 2. dal2-pr2-xe-0-3-0-0.us.twtelecom.net 0.1% 829 55.4 31.2 9.4 121.0 17.2 dal2-pr2-xe-2-3-0-0.us.twtelecom.net 3. xe-3-0-0.er2.dfw2.us.above.net 0.1% 829 44.8 31.6 9.5 143.7 17.9 4. xe-0-1-0.cr2.dfw2.us.above.net 0.2% 829 40.5 31.2 10.3 116.2 15.9 xe-3-0-0.cr2.dfw2.us.above.net 5. xe-0-2-0.er3.dfw2.us.above.net 0.2% 829 42.7 32.4 9.6 109.2 16.5
Regards,
Chris Boulton Bigcommerce
Email: chris@bigcommerce.com
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Gotcha, now we're on the same page! :-) So can you provide return-path mtr or traceroutes that are associated with both of your below links? Level 3 will probably want this as well, as most routing today is asymmetrical. Rephrased verbosely: we need to see a total of 4 mtrs/traceroutes: 1) From x.x.x.x to y.y.y.y (your Level 3 link) -- you provided this 2) From y.y.y.y to x.x.x.x -- need this 3) From a.a.a.a to b.b.b.b (your TW Telecom link) -- you provided this 4) From b.b.b.b to a.a.a.a -- need this What you showed in Item #1 (your L3 link) shows no actual/real issue. What you showed in Item #3 (your TW Telecom link) **does** show packet loss, but I cannot tell (without Item #4) if it's on the return-path or the forward-path. I understand "your TW Telecom link is working fine" but the 0.1-0.2% packet loss indicates it may not be as good as you think. That may be a separate issue however, or may be by design (QoS-like rules applied on router, etc.). Make sense? :-) -- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB | On Fri, Jan 25, 2013 at 06:45:54AM +1100, Chris Boulton wrote:
Ah, my apologies on the confusion. They're two distinct paths out, not traceroutes from each end.
I chose not to include the destination IP because the issues we're seeing a more wide spread than to a single IP.
Level 3 managed to get back to us - they were originally reporting back to us that they are experiencing connectivity issues for some of their clients in Austin, which they're now denying and still continuing to investigate.
Regards,
Chris Boulton Bigcommerce
Email: chris@bigcommerce.com
Web: http://www.bigcommerce.com/
On Fri, Jan 25, 2013 at 6:39 AM, Jeremy Chadwick <jdc@koitsu.org> wrote:
Chris, is there some reason you're not providing actual source/destination IPs for each of your mtrs?
The reason I'm asking is because I was left to believe what you were providing was both ends of an asymmetric connection (i.e. mtr #1 was for x.x.x.x to y.y.y.y, mtr #2 was for y.y.y.y to x.x.x.x).
However, your 2nd paragraph below ("we're seeing issues over one connections but not another") seems to indicate what you provided are two mtrs across completely different circuits.
So which is it?
-- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB |
On Fri, Jan 25, 2013 at 06:14:05AM +1100, Chris Boulton wrote:
Thanks, I am aware of what the traceroute shows and how a traceroute works. The intent of the traceroute was primarily to show where I see our paths between Level 3 and TW Telecom converge.
Plain and simple - we are experiencing connectivity issues over our Level 3 connection, which we're not seeing over our TW Telecom connection for traffic destined to the same location.
Regards,
Chris Boulton Bigcommerce
Email: chris@bigcommerce.com
Web: http://www.bigcommerce.com/
On Fri, Jan 25, 2013 at 6:11 AM, Jeremy Chadwick <jdc@koitsu.org> wrote:
Please see my below post on this matter, which I posted only 2 days ago:
https://puck.nether.net/pipermail/outages/2013-January/005111.html
What you've shown in your mtr does not show any issue of any kind with Level 3. The packet loss shown at hops 4/5/6 does not "trickle down" each subsequent hop (look at hop 7 onward for proof), nor is there any sign of latency increase either.
On the other hand, the 2nd mtr you provided shows some degree of packet loss starting at hop 2 (TW Telecom) which *does* "trickle down" through all subsequent hops (egress). If you want to know how many packets were lost, look at the --order flag to mtr (I recommend --order=LSRNABW).
Plain and simple: talk to TW Telecom about the loss in their network.
-- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB |
On Fri, Jan 25, 2013 at 05:46:27AM +1100, Chris Boulton wrote:
Seeing connectivity issues as traffic passes through Level 3 in Dallas - not seeing the same issues for where routes converge (above.net in DFW) over our TW Telecom link.
Anybody else experiencing this as well?
Level 3:
Host Loss% Snt Last Avg Best Wrst StDev 1. vlan358.car1.Houston1.Level3.net 0.0% 167 105.9 80.5 31.7 276.4 59.8 2. ae-2-5.bar1.Houston1.Level3.net 0.0% 167 32.5 34.6 32.2 93.8 7.3 3. ae-13-13.ebr1.Dallas1.Level3.net 0.0% 167 39.5 37.4 36.7 41.2 0.9 4. ae-91-91.csw4.Dallas1.Level3.net 65.1% 167 44.5 43.5 39.7 53.8 4.0 5. ae-4-90.edge2.Dallas3.Level3.net 5.4% 166 37.1 37.9 36.1 75.6 5.1 6. ex-3-1-0.er1.dfw2.us.above.net 66.1% 166 39.5 43.5 38.9 104.8 12.5 7. xe-3-1-0.cr1.dfw2.us.above.net 0.0% 166 37.3 40.4 37.0 114.1 10.6 8. xe-0-0-0.er3.dfw2.us.above.net 0.0% 166 36.2 39.8 36.1 99.3 10.2
Host Loss% Snt Last Avg Best Wrst StDev 1. 50.58.18.1 0.0% 829 4.3 23.9 4.1 121.2 14.5 2. dal2-pr2-xe-0-3-0-0.us.twtelecom.net 0.1% 829 55.4 31.2 9.4 121.0 17.2 dal2-pr2-xe-2-3-0-0.us.twtelecom.net 3. xe-3-0-0.er2.dfw2.us.above.net 0.1% 829 44.8 31.6 9.5 143.7 17.9 4. xe-0-1-0.cr2.dfw2.us.above.net 0.2% 829 40.5 31.2 10.3 116.2 15.9 xe-3-0-0.cr2.dfw2.us.above.net 5. xe-0-2-0.er3.dfw2.us.above.net 0.2% 829 42.7 32.4 9.6 109.2 16.5
Regards,
Chris Boulton Bigcommerce
Email: chris@bigcommerce.com
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
participants (2)
-
Chris Boulton -
Jeremy Chadwick