Starting at approx 06:00 UTC we had a strange issue with TCP/UDP traffic dying on Level3 in Dallas, although ICMP looked clean. For now we've stopped sending traffic to Level3 in Dallas. Anyone else seeing similar issues? -mark
FYI, Level3 has acknowledged the issue but no ETR yet. "The event, 9470903, reports a card failing in Dallas, TX impacting IP services. The silent card failure was detected when the Technical Service Center began receiving calls from customers. " On Thu, Jul 16, 2015 at 3:03 AM, Mark Price <mprice@tqhosting.com> wrote:
Starting at approx 06:00 UTC we had a strange issue with TCP/UDP traffic dying on Level3 in Dallas, although ICMP looked clean. For now we've stopped sending traffic to Level3 in Dallas.
Anyone else seeing similar issues?
-mark
7/16/15 8:39:56 PM GMTThe IP NOC is working with the equipment vendor to isolate the root cause of the errors.7/16/15 7:54:43 PM GMTNo new information is currently available. Updates will be provided as they are received.7/16/15 6:56:49 PM GMTInvestigations are ongoing to isolate the issue.7/16/15 6:06:45 PM GMTThe IP NOC reports a card failing in Dallas, TX impacting IP services. The silent card failure was detected when the Technical Service Center began receiving calls from customers. The equipment vendor is being engaged for assistance. Please be advised that updates for this event will be relayed at a minimum of hourly unless otherwise noted. On Thu, Jul 16, 2015 at 1:51 PM, Mark Price via Outages <outages@outages.org
wrote:
FYI, Level3 has acknowledged the issue but no ETR yet.
"The event, 9470903, reports a card failing in Dallas, TX impacting IP services. The silent card failure was detected when the Technical Service Center began receiving calls from customers. "
On Thu, Jul 16, 2015 at 3:03 AM, Mark Price <mprice@tqhosting.com> wrote:
Starting at approx 06:00 UTC we had a strange issue with TCP/UDP traffic dying on Level3 in Dallas, although ICMP looked clean. For now we've stopped sending traffic to Level3 in Dallas.
Anyone else seeing similar issues?
-mark
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Sadly, this is a repeat offender linecard/platform/customer-edge-device/something-or-other. This is probably the 3rd time this has taken place for this port for us in the past 14months? As a side note - We reported this at 7:45 AM GMT per our ticket timestamp when it originally took place. Not sure why it takes so long for the IP NOC to put it together. If there are any Level3 lurkers - may be an indication of other training or tool problems? In our ticket notes - they said there were 4 other ports impacted on the linecard with down BGP sessions, so I guess I know who else is on the linecard :) /wave -- jason On Thu, Jul 16, 2015 at 3:57 PM, Luke Rockwell via Outages < outages@outages.org> wrote:
7/16/15 8:39:56 PM GMTThe IP NOC is working with the equipment vendor to isolate the root cause of the errors.7/16/15 7:54:43 PM GMTNo new information is currently available. Updates will be provided as they are received.7/16/15 6:56:49 PM GMTInvestigations are ongoing to isolate the issue.7/16/15 6:06:45 PM GMTThe IP NOC reports a card failing in Dallas, TX impacting IP services. The silent card failure was detected when the Technical Service Center began receiving calls from customers. The equipment vendor is being engaged for assistance. Please be advised that updates for this event will be relayed at a minimum of hourly unless otherwise noted.
On Thu, Jul 16, 2015 at 1:51 PM, Mark Price via Outages < outages@outages.org> wrote:
FYI, Level3 has acknowledged the issue but no ETR yet.
"The event, 9470903, reports a card failing in Dallas, TX impacting IP services. The silent card failure was detected when the Technical Service Center began receiving calls from customers. "
On Thu, Jul 16, 2015 at 3:03 AM, Mark Price <mprice@tqhosting.com> wrote:
Starting at approx 06:00 UTC we had a strange issue with TCP/UDP traffic dying on Level3 in Dallas, although ICMP looked clean. For now we've stopped sending traffic to Level3 in Dallas.
Anyone else seeing similar issues?
-mark
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
participants (3)
-
Jason Ackley -
Luke Rockwell -
Mark Price