AT&T/SBCGlobal issues
We have a handful of users who have reported issues to us that point to a problem between AT&T/SBCGlobal in Knoxville, TN and Jacksonville, FL and our CDN provider -- Edgecast/Verizon/VDMS. It appears that Downdetector has a bit more of an uptick for AT&T than they usually do, so it could very well affect more than just the route to VDMS. Some traceroutes: Tracing route to cs493.wpc.edgecastcdn.net [192.229.210.64]
over a maximum of 30 hops: 1 10 ms 11 ms 19 ms homeportal [192.168.1.254] 2 140 ms 71 ms 30 ms 99-51-96-1.lightspeed.jcvlfl.sbcglobal.net 3 84 ms 23 ms 21 ms 99.166.204.185 4 59 ms 110 ms 29 ms 99.132.13.66 5 51 ms 88 ms 137 ms 12.83.101.37 6 * * * Request timed out. 7 * * * Request timed out.
Tracing route to cs493.wpc.edgecastcdn.net [192.229.210.64]
over a maximum of 30 hops: 1 2 ms 4 ms 3 ms homeportal [192.168.1.254] 2 29 ms 21 ms 25 ms 99-17-80-1.lightspeed.knvltn.sbcglobal.net 3 22 ms 21 ms 45 ms 99.17.140.5 4 * * * Request timed out. 5 * * * Request timed out. 6 * * * Request timed out.
-- Andy Walker System Administrator FBS - creators of flexmls 3415 39th St S Fargo, ND 58104 701-235-7300
We're seeing inbound calling issues with East TN numbers that probably backhaul/peer with AT&T via Bandwidth.com/Level 3. Started around 45 min ago Brian Ladd Customer Care Manager Voiceopia Communications On Fri, Jan 25, 2019 at 11:29 AM Andy W via Outages <outages@outages.org> wrote:
We have a handful of users who have reported issues to us that point to a problem between AT&T/SBCGlobal in Knoxville, TN and Jacksonville, FL and our CDN provider -- Edgecast/Verizon/VDMS. It appears that Downdetector has a bit more of an uptick for AT&T than they usually do, so it could very well affect more than just the route to VDMS.
Some traceroutes:
Tracing route to cs493.wpc.edgecastcdn.net [192.229.210.64]
over a maximum of 30 hops: 1 10 ms 11 ms 19 ms homeportal [192.168.1.254] 2 140 ms 71 ms 30 ms 99-51-96-1.lightspeed.jcvlfl.sbcglobal.net 3 84 ms 23 ms 21 ms 99.166.204.185 4 59 ms 110 ms 29 ms 99.132.13.66 5 51 ms 88 ms 137 ms 12.83.101.37 6 * * * Request timed out. 7 * * * Request timed out.
Tracing route to cs493.wpc.edgecastcdn.net [192.229.210.64]
over a maximum of 30 hops: 1 2 ms 4 ms 3 ms homeportal [192.168.1.254] 2 29 ms 21 ms 25 ms 99-17-80-1.lightspeed.knvltn.sbcglobal.net 3 22 ms 21 ms 45 ms 99.17.140.5 4 * * * Request timed out. 5 * * * Request timed out. 6 * * * Request timed out.
-- Andy Walker System Administrator FBS - creators of flexmls 3415 39th St S Fargo, ND 58104 701-235-7300 _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Seeing increased latency and packet loss across AT&T Uverse just outside of Oklahoma City, currently unable to communicate with most if not all of our remote clients at this time. AT&T has no information for us and won’t even acknowledge they are having issues or that an outage is occurring. Sent from ProtonMail Mobile On Fri, Jan 25, 2019 at 10:40, Brian Ladd via Outages <outages@outages.org> wrote:
We're seeing inbound calling issues with East TN numbers that probably backhaul/peer with AT&T via Bandwidth.com/Level 3. Started around 45 min ago
Brian Ladd Customer Care Manager Voiceopia Communications
On Fri, Jan 25, 2019 at 11:29 AM Andy W via Outages <outages@outages.org> wrote:
We have a handful of users who have reported issues to us that point to a problem between AT&T/SBCGlobal in Knoxville, TN and Jacksonville, FL and our CDN provider -- Edgecast/Verizon/VDMS. It appears that Downdetector has a bit more of an uptick for AT&T than they usually do, so it could very well affect more than just the route to VDMS.
Some traceroutes:
Tracing route to cs493.wpc.edgecastcdn.net [192.229.210.64] over a maximum of 30 hops: 1 10 ms 11 ms 19 ms homeportal [192.168.1.254]2 140 ms 71 ms 30 ms 99-51-96-1.lightspeed.jcvlfl.sbcglobal.net 3 84 ms 23 ms 21 ms 99.166.204.1854 59 ms 110 ms 29 ms 99.132.13.665 51 ms 88 ms 137 ms 12.83.101.376 * * * Request timed out.7 * * * Request timed out.
Tracing route to cs493.wpc.edgecastcdn.net [192.229.210.64] over a maximum of 30 hops: 1 2 ms 4 ms 3 ms homeportal [192.168.1.254] 2 29 ms 21 ms 25 ms 99-17-80-1.lightspeed.knvltn.sbcglobal.net 3 22 ms 21 ms 45 ms 99.17.140.5 4 * * * Request timed out. 5 * * * Request timed out. 6 * * * Request timed out.
-- Andy Walker System Administrator FBS - creators of flexmls 3415 39th St S Fargo, ND 58104 701-235-7300 _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
I can get there from Frontier: traceroute 192.229.210.64 traceroute to 192.229.210.64 (192.229.210.64), 64 hops max, 52 byte packets 1 47.155.227.1 (47.155.227.1) 9.475 ms 4.429 ms 4.657 ms 2 172.102.107.164 (172.102.107.164) 10.263 ms 9.723 ms 172.102.102.240 (172.102.102.240) 7.703 ms 3 ae8---0.scr01.lsan.ca.frontiernet.net (74.40.3.37) 7.241 ms ae8---0.scr02.lsan.ca.frontiernet.net (74.40.3.49) 7.696 ms 7.243 ms 4 ae0---0.cbr01.lsan.ca.frontiernet.net (74.40.3.198) 5.970 ms ae1---0.cbr01.lsan.ca.frontiernet.net (74.40.3.214) 6.743 ms ae0---0.cbr01.lsan.ca.frontiernet.net (74.40.3.198) 7.843 ms 5 74.43.94.17 (74.43.94.17) 6.711 ms 9.975 ms 10.083 ms 6 152.195.77.129 (152.195.77.129) 9.682 ms 9.829 ms 9.679 ms 7 192.229.210.64 (192.229.210.64) 7.438 ms 9.798 ms 7.496 ms As a precaution, you should check to make sure you’re not black-holed by AT&T. We had an incident over Christmas where AT&T automated security had black-holed our branch office connectivity to our AT&T hub site! We only proved it and got the block pulled by showing the destination was reachable from many other locations. -mel On Fri, 25 Jan 2019 18:03:30 +0000 Geeknik Labs via Outages <outages@outages.org> wrote:
Seeing increased latency and packet loss across AT&T Uverse just outside of Oklahoma City, currently unable to communicate with most if not all of our remote clients at this time. AT&T has no information for us and won’t even acknowledge they are having issues or that an outage is occurring.
Sent from ProtonMail Mobile
On Fri, Jan 25, 2019 at 10:40, Brian Ladd via Outages <outages@outages.org> wrote:
We're seeing inbound calling issues with East TN numbers that probably backhaul/peer with AT&T via Bandwidth.com/Level 3. Started around 45 min ago
Brian Ladd Customer Care Manager Voiceopia Communications
On Fri, Jan 25, 2019 at 11:29 AM Andy W via Outages <outages@outages.org> wrote:
We have a handful of users who have reported issues to us that point to a problem between AT&T/SBCGlobal in Knoxville, TN and Jacksonville, FL and our CDN provider -- Edgecast/Verizon/VDMS. It appears that Downdetector has a bit more of an uptick for AT&T than they usually do, so it could very well affect more than just the route to VDMS.
Some traceroutes:
Tracing route to cs493.wpc.edgecastcdn.net [192.229.210.64] over a maximum of 30 hops: 1 10 ms 11 ms 19 ms homeportal [192.168.1.254]2 140 ms 71 ms 30 ms 99-51-96-1.lightspeed.jcvlfl.sbcglobal.net 3 84 ms 23 ms 21 ms 99.166.204.1854 59 ms 110 ms 29 ms 99.132.13.665 51 ms 88 ms 137 ms 12.83.101.376 * * * Request timed out.7 * * * Request timed out.
Tracing route to cs493.wpc.edgecastcdn.net [192.229.210.64] over a maximum of 30 hops: 1 2 ms 4 ms 3 ms homeportal [192.168.1.254] 2 29 ms 21 ms 25 ms 99-17-80-1.lightspeed.knvltn.sbcglobal.net 3 22 ms 21 ms 45 ms 99.17.140.5 4 * * * Request timed out. 5 * * * Request timed out. 6 * * * Request timed out.
-- Andy Walker System Administrator FBS - creators of flexmls 3415 39th St S Fargo, ND 58104 701-235-7300 _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
FYI, VDMS acknowledged this, investigated, and just now considered the issue resolved: https://status.verizondigitalmedia.com/pages/incident/5736344c90417cda1a000f... Not sure if this holds true for any other entities impacted by the AT&T issues, but that should at least clear up the Verizon side of things. -- Andy Walker System Administrator FBS - creators of flexmls 3415 39th St S Fargo, ND 58104 701-235-7300 On Fri, Jan 25, 2019 at 12:37 PM Mel Beckman <mel@becknet.com> wrote:
I can get there from Frontier:
traceroute 192.229.210.64 traceroute to 192.229.210.64 (192.229.210.64), 64 hops max, 52 byte packets 1 47.155.227.1 (47.155.227.1) 9.475 ms 4.429 ms 4.657 ms 2 172.102.107.164 (172.102.107.164) 10.263 ms 9.723 ms 172.102.102.240 (172.102.102.240) 7.703 ms 3 ae8---0.scr01.lsan.ca.frontiernet.net (74.40.3.37) 7.241 ms ae8---0.scr02.lsan.ca.frontiernet.net (74.40.3.49) 7.696 ms 7.243 ms 4 ae0---0.cbr01.lsan.ca.frontiernet.net (74.40.3.198) 5.970 ms ae1---0.cbr01.lsan.ca.frontiernet.net (74.40.3.214) 6.743 ms ae0---0.cbr01.lsan.ca.frontiernet.net (74.40.3.198) 7.843 ms 5 74.43.94.17 (74.43.94.17) 6.711 ms 9.975 ms 10.083 ms 6 152.195.77.129 (152.195.77.129) 9.682 ms 9.829 ms 9.679 ms 7 192.229.210.64 (192.229.210.64) 7.438 ms 9.798 ms 7.496 ms
As a precaution, you should check to make sure you’re not black-holed by AT&T. We had an incident over Christmas where AT&T automated security had black-holed our branch office connectivity to our AT&T hub site! We only proved it and got the block pulled by showing the destination was reachable from many other locations.
-mel
On Fri, 25 Jan 2019 18:03:30 +0000 Geeknik Labs via Outages <outages@outages.org> wrote:
Seeing increased latency and packet loss across AT&T Uverse just outside of Oklahoma City, currently unable to communicate with most if not all of our remote clients at this time. AT&T has no information for us and won’t even acknowledge they are having issues or that an outage is occurring.
Sent from ProtonMail Mobile
On Fri, Jan 25, 2019 at 10:40, Brian Ladd via Outages <outages@outages.org> wrote:
We're seeing inbound calling issues with East TN numbers that probably backhaul/peer with AT&T via Bandwidth.com/Level 3. Started around 45 min ago
Brian Ladd Customer Care Manager Voiceopia Communications
On Fri, Jan 25, 2019 at 11:29 AM Andy W via Outages <outages@outages.org> wrote:
We have a handful of users who have reported issues to us that point to a problem between AT&T/SBCGlobal in Knoxville, TN and Jacksonville, FL and our CDN provider -- Edgecast/Verizon/VDMS. It appears that Downdetector has a bit more of an uptick for AT&T than they usually do, so it could very well affect more than just the route to VDMS.
Some traceroutes:
Tracing route to cs493.wpc.edgecastcdn.net [192.229.210.64] over a maximum of 30 hops: 1 10 ms 11 ms 19 ms homeportal [192.168.1.254]2 140 ms 71 ms 30 ms 99-51-96-1.lightspeed.jcvlfl.sbcglobal.net 3 84 ms 23 ms 21 ms 99.166.204.1854 59 ms 110 ms 29 ms 99.132.13.665 51 ms 88 ms 137 ms 12.83.101.376 * * * Request timed out.7 * * * Request timed out.
Tracing route to cs493.wpc.edgecastcdn.net [192.229.210.64] over a maximum of 30 hops: 1 2 ms 4 ms 3 ms homeportal [192.168.1.254] 2 29 ms 21 ms 25 ms 99-17-80-1.lightspeed.knvltn.sbcglobal.net 3 22 ms 21 ms 45 ms 99.17.140.5 4 * * * Request timed out. 5 * * * Request timed out. 6 * * * Request timed out.
-- Andy Walker System Administrator FBS - creators of flexmls 3415 39th St S Fargo, ND 58104 701-235-7300 _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
participants (4)
-
Andy W -
Brian Ladd -
Geeknik Labs -
Mel Beckman