Anyone seeing any strangeness with Bell Canada - AS577? A traceroute to www.facebook.com dies in Montreal. I shut down our BGP session with them, and can reach the site fine through Level3.
I am currently in Montreal, on a Videotron DSL connection. I can access facebook with no problems, however the trace dies very early (traces to other hosts work fine): traceroute to www.facebook.com (69.63.186.12), 64 hops max, 40 byte packets 1 10.0.1.1 (10.0.1.1) 1.910 ms 1.111 ms 1.011 ms 2 10.0.1.1 (10.0.1.1) 7.822 ms 7.705 ms 5.977 ms 3 24.200.225.153 (24.200.225.153) 6.376 ms 7.830 ms 5.766 ms 4 ia-bebn-bb04-ge11-0-0.vtl.net (216.113.122.41) 5.908 ms 5.875 ms 8.639 ms 5 ia-bebn-bb04-ge14-0-0-cpe098.vtl.net (216.113.122.98) 22.461 ms 19.738 ms 19.689 ms 6 * * * 7 * * * 8 * * * Dan On Thu, Jun 18, 2009 at 4:21 PM, Clayton Zekelman <clayton@mnsi.net> wrote:
Anyone seeing any strangeness with Bell Canada - AS577?
A traceroute to www.facebook.com dies in Montreal.
I shut down our BGP session with them, and can reach the site fine through Level3. _______________________________________________ outages mailing list outages@outages.org https://puck.nether.net/mailman/listinfo/outages
We drained AS32934<->AS577 peering at ~9:30 PM PDT last night. This resolved all of the reported Facebook issues at the time. As of 8:30 this morning, the problematic AS577 node appears to be normalized and traffic is back on it with no problems. Thanks, Robin. On Thu, Jun 18, 2009 at 10:22 PM, Dan Parsons <dparsons@nyip.net> wrote:
I am currently in Montreal, on a Videotron DSL connection. I can access facebook with no problems, however the trace dies very early (traces to other hosts work fine): traceroute to www.facebook.com (69.63.186.12), 64 hops max, 40 byte packets 1 10.0.1.1 (10.0.1.1) 1.910 ms 1.111 ms 1.011 ms 2 10.0.1.1 (10.0.1.1) 7.822 ms 7.705 ms 5.977 ms 3 24.200.225.153 (24.200.225.153) 6.376 ms 7.830 ms 5.766 ms 4 ia-bebn-bb04-ge11-0-0.vtl.net (216.113.122.41) 5.908 ms 5.875 ms 8.639 ms 5 ia-bebn-bb04-ge14-0-0-cpe098.vtl.net (216.113.122.98) 22.461 ms 19.738 ms 19.689 ms 6 * * * 7 * * * 8 * * *
Dan
On Thu, Jun 18, 2009 at 4:21 PM, Clayton Zekelman <clayton@mnsi.net>wrote:
Anyone seeing any strangeness with Bell Canada - AS577?
A traceroute to www.facebook.com dies in Montreal.
I shut down our BGP session with them, and can reach the site fine through Level3. _______________________________________________ outages mailing list outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ outages mailing list outages@outages.org https://puck.nether.net/mailman/listinfo/outages
We're seeing several T1s down in Oxnard, Camarillo, Thousand Oaks CA as well as circuits busy on POTS calls to the area. -- Jay Hennigan - CCIE #7880 - Network Engineering - jay@impulse.net Impulse Internet Service - http://www.impulse.net/ Your local telephone and internet company - 805 884-6323 - WB6RDV
We lost our ATT circuit at 10 am. still down. They claim 'Transport Issues' Clients are claiming that their ATT and Sprint T1's are down. Our Sprint circuit is doing fine. -bill PixelGate Networks (Thousand Oaks) Jay Hennigan wrote:
We're seeing several T1s down in Oxnard, Camarillo, Thousand Oaks CA as well as circuits busy on POTS calls to the area.
-- Jay Hennigan - CCIE #7880 - Network Engineering - jay@impulse.net Impulse Internet Service - http://www.impulse.net/ Your local telephone and internet company - 805 884-6323 - WB6RDV _______________________________________________ outages mailing list outages@outages.org https://puck.nether.net/mailman/listinfo/outages
3.5 hours and no restoration? If $BIGTELCOS are going to run everything through one physical path, then they had better be able to roll techs to a break point really fast. Get a police escort, ride a bicycle, something. So they rerouted PSAP traffic, but not internet service.... fascinating. Now I want to know how they have things setup. I used to live in Oxnard, and Verizon is the last mile dial tone provider out there. Cell reception was very bad, so I got a land line so the NOC could get a hold of me. We had a Verizon fiber feed to our office. Wasn't aware that ATT and Sprint had a presence out there. Or is it reselling Verizon loops? Something something redundant paths, something something we don't care about anything except maximizing profit margins. Bah. William Kern wrote:
We lost our ATT circuit at 10 am. still down.
They claim 'Transport Issues'
Clients are claiming that their ATT and Sprint T1's are down.
Our Sprint circuit is doing fine.
-bill PixelGate Networks (Thousand Oaks)
Jay Hennigan wrote:
We're seeing several T1s down in Oxnard, Camarillo, Thousand Oaks CA as well as circuits busy on POTS calls to the area.
-- Jay Hennigan - CCIE #7880 - Network Engineering - jay@impulse.net Impulse Internet Service - http://www.impulse.net/ Your local telephone and internet company - 805 884-6323 - WB6RDV _______________________________________________ outages mailing list outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ outages mailing list outages@outages.org https://puck.nether.net/mailman/listinfo/outages
any updates? Charles Wyble wrote:
3.5 hours and no restoration?
If $BIGTELCOS are going to run everything through one physical path, then they had better be able to roll techs to a break point really fast.
Get a police escort, ride a bicycle, something.
So they rerouted PSAP traffic, but not internet service.... fascinating. Now I want to know how they have things setup.
I used to live in Oxnard, and Verizon is the last mile dial tone provider out there. Cell reception was very bad, so I got a land line so the NOC could get a hold of me. We had a Verizon fiber feed to our office.
Wasn't aware that ATT and Sprint had a presence out there. Or is it reselling Verizon loops?
Something something redundant paths, something something we don't care about anything except maximizing profit margins.
Bah.
William Kern wrote:
We lost our ATT circuit at 10 am. still down.
They claim 'Transport Issues'
Clients are claiming that their ATT and Sprint T1's are down.
Our Sprint circuit is doing fine.
-bill PixelGate Networks (Thousand Oaks)
Jay Hennigan wrote:
We're seeing several T1s down in Oxnard, Camarillo, Thousand Oaks CA as well as circuits busy on POTS calls to the area.
-- Jay Hennigan - CCIE #7880 - Network Engineering - jay@impulse.net Impulse Internet Service - http://www.impulse.net/ Your local telephone and internet company - 805 884-6323 - WB6RDV _______________________________________________ outages mailing list outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ outages mailing list outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ outages mailing list outages@outages.org https://puck.nether.net/mailman/listinfo/outages
On Mon, Jul 6, 2009 at 6:09 PM, John Martinez<jmartinez@zero11.com> wrote:
any updates?
<N.B.> If the reader thinks of this as a breach of netiquette, please hit delete _now_. VERIZON CONFIDENTIAL FLASH SUMMARY </N.B.> ===== VERIZON CONFIDENTIAL FLASH SUMMARY FLASH NUMBER 2009070600119 ORGANIZATION NSMC SEVERITY LEVEL CATA CRITERIA HSI - HSI outage impacting 2500 or more customers IMPACT LOSS OF DATA AND/OR CONNECTIVITY NETWORK HSI SUBNETWORK ROUTER SPECIAL SERVICES STATE CA CITY THOUSAND OAKS COUNTRY USA OUTAGE START DATE 07-06-2009 OUTAGE START TIME 17:10:00 GMT OUTAGE END DATE 07-06-2009 OUTAGE END TIME 22:26:00 GMT DURATION 5:16:00 EQUIPMENT N/A CAUSE OF OUTAGE FIBER/CABLE OUTAGE - OUTSIDE CORRECTIVE ACTION Fiber spliced. TICKET SOURCE WHITEBOARD TICKET NUMBER 92475 LEC/OCC TICKET PVC/CKT Affected WEBSITE oasis.vzbi.com COMMENTS 07-06-2009 18:34 GMT Thousand Oaks, CA - HSI - Informational Bridge: (517) 477-3482 pc 865-5190. NSMC reports 3500 HSI customers are down. There is a fiber cut located 1.3 miles outside of the Camarillo Central Office on Dawson Drive. Approximately 1000 feet of fiber has been damaged. There is no estimated time of restoral. Number of Trouble Tickets = 107 ----------- 07-06-2009 19:45 GMT NSMC reports that there are 252 attached tickets. Construction is currently prepping the fiber with an ETA of new fiber onsite at 1:23 PM PT. 07-06-2009 21:24 GMT NSMC reports that there are 345 attached tickets. All fiber is onsite and pulled through duct, splicing commencing in approximately 5 minutes. 07-06-2009 22:49 GMT NSMC reports construction spliced the fiber. HSI service restored. 406 Attached tickets.
Any updates? Charles Wyble wrote:
3.5 hours and no restoration?
If $BIGTELCOS are going to run everything through one physical path, then they had better be able to roll techs to a break point really fast.
Get a police escort, ride a bicycle, something.
So they rerouted PSAP traffic, but not internet service.... fascinating. Now I want to know how they have things setup.
I used to live in Oxnard, and Verizon is the last mile dial tone provider out there. Cell reception was very bad, so I got a land line so the NOC could get a hold of me. We had a Verizon fiber feed to our office.
Wasn't aware that ATT and Sprint had a presence out there. Or is it reselling Verizon loops?
Something something redundant paths, something something we don't care about anything except maximizing profit margins.
Bah.
William Kern wrote:
We lost our ATT circuit at 10 am. still down.
They claim 'Transport Issues'
Clients are claiming that their ATT and Sprint T1's are down.
Our Sprint circuit is doing fine.
-bill PixelGate Networks (Thousand Oaks)
Jay Hennigan wrote:
We're seeing several T1s down in Oxnard, Camarillo, Thousand Oaks CA as well as circuits busy on POTS calls to the area.
-- Jay Hennigan - CCIE #7880 - Network Engineering - jay@impulse.net Impulse Internet Service - http://www.impulse.net/ Your local telephone and internet company - 805 884-6323 - WB6RDV _______________________________________________ outages mailing list outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ outages mailing list outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ outages mailing list outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Our ATT circuits are now up. -bill Chaim Rieger wrote:
John Martinez wrote:
Any updates?
According to VZW, most if not all customers should be back in business,
will continue repairs till 7 pm (local time) _______________________________________________ outages mailing list outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Jay Hennigan wrote:
We're seeing several T1s down in Oxnard, Camarillo, Thousand Oaks CA as well as circuits busy on POTS calls to the area.
Fiber cut on Dawson drive in Camarillo is the suspected cause of a major outage phone, network, & 911 in ventura county california
Evidently 911 has been restored. http://twitter.com/socalincidents TSP for the win. Chaim Rieger wrote:
Jay Hennigan wrote:
We're seeing several T1s down in Oxnard, Camarillo, Thousand Oaks CA as well as circuits busy on POTS calls to the area.
Fiber cut on Dawson drive in Camarillo is the suspected cause of a major outage phone, network, & 911 in ventura county california _______________________________________________ outages mailing list outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Looks fine over here. Traceroute completes with various hops through Toronto, Vancouver, Seattle ect... -----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Clayton Zekelman Sent: Thursday, June 18, 2009 7:21 PM To: outages@outages.org Subject: [outages] Bell AS577 Anyone seeing any strangeness with Bell Canada - AS577? A traceroute to www.facebook.com dies in Montreal. I shut down our BGP session with them, and can reach the site fine through Level3. _______________________________________________ outages mailing list outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Anyone having any data issues with Sprint? I'm seeing issues with Email and IM services. Anyone else experiencing this? Thanks, Derek McDaniel Systems Support MAXIMUM Communications, Inc. dmcdaniel@maximumcomm.com 800-589-6299 ext 111 -----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org]On Behalf Of Jeff Cartier Sent: Friday, June 19, 2009 8:16 AM To: clayton@mnsi.net; outages@outages.org Subject: Re: [outages] Bell AS577 Looks fine over here. Traceroute completes with various hops through Toronto, Vancouver, Seattle ect... -----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Clayton Zekelman Sent: Thursday, June 18, 2009 7:21 PM To: outages@outages.org Subject: [outages] Bell AS577 Anyone seeing any strangeness with Bell Canada - AS577? A traceroute to www.facebook.com dies in Montreal. I shut down our BGP session with them, and can reach the site fine through Level3. _______________________________________________ outages mailing list outages@outages.org https://puck.nether.net/mailman/listinfo/outages _______________________________________________ outages mailing list outages@outages.org https://puck.nether.net/mailman/listinfo/outages __________ NOD32 4170 (20090619) Information __________ This message was checked by NOD32 antivirus system. http://www.eset.com
Derek McDaniel wrote:
Anyone having any data issues with Sprint? I'm seeing issues with Email and IM services. Anyone else experiencing this?
Sprint the cell phone? Sprintlink? What IM services? What email? Where? No problems with my Sprintlink circuits. ~Seth
Sprint Cell phone, sorry about that. Derek McDaniel Systems Support MAXIMUM Communications, Inc. dmcdaniel@maximumcomm.com 800-589-6299 ext 111 -----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org]On Behalf Of Seth Mattinen Sent: Friday, June 19, 2009 11:22 AM To: outages@outages.org Subject: Re: [outages] Sprint Derek McDaniel wrote:
Anyone having any data issues with Sprint? I'm seeing issues with Email and IM services. Anyone else experiencing this?
Sprint the cell phone? Sprintlink? What IM services? What email? Where? No problems with my Sprintlink circuits. ~Seth _______________________________________________ outages mailing list outages@outages.org https://puck.nether.net/mailman/listinfo/outages __________ NOD32 4171 (20090619) Information __________ This message was checked by NOD32 antivirus system. http://www.eset.com
Derek McDaniel wrote:
Sprint Cell phone, sorry about that.
Derek McDaniel Systems Support MAXIMUM Communications, Inc. dmcdaniel@maximumcomm.com 800-589-6299 ext 111
-----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org]On Behalf Of Seth Mattinen Sent: Friday, June 19, 2009 11:22 AM To: outages@outages.org Subject: Re: [outages] Sprint
Derek McDaniel wrote:
Anyone having any data issues with Sprint? I'm seeing issues with Email
and
IM services. Anyone else experiencing this?
Sprint the cell phone? Sprintlink? What IM services? What email? Where?
No problems with my Sprintlink circuits.
~Seth _______________________________________________ outages mailing list outages@outages.org https://puck.nether.net/mailman/listinfo/outages
__________ NOD32 4171 (20090619) Information __________
This message was checked by NOD32 antivirus system. http://www.eset.com
_______________________________________________ outages mailing list outages@outages.org https://puck.nether.net/mailman/listinfo/outages ------------------------------------------------------------------------
No virus found in this incoming message. Checked by AVG - www.avg.com Version: 8.5.339 / Virus Database: 270.12.80/2187 - Release Date: 06/19/09 06:53:00
We have seen TXT messages being delayed 20 min at times this last week...which we rely on for up/down messages.....Sprint goes through periods of this st!t all the time and we never get a response as to why... --chris
Thanks Chris, Everything *seems* to be back to normal now. Everything is being pushed to my phone again like normal. Thanks, Derek McDaniel Systems Support MAXIMUM Communications, Inc. dmcdaniel@maximumcomm.com 800-589-6299 ext 111 -----Original Message----- From: ChrisSerafin [mailto:chris@chrisserafin.com] Sent: Friday, June 19, 2009 1:32 PM To: Derek McDaniel Cc: Seth Mattinen; outages@outages.org Subject: Re: [outages] Sprint Derek McDaniel wrote:
Sprint Cell phone, sorry about that.
Derek McDaniel Systems Support MAXIMUM Communications, Inc. dmcdaniel@maximumcomm.com 800-589-6299 ext 111
-----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org]On Behalf Of Seth Mattinen Sent: Friday, June 19, 2009 11:22 AM To: outages@outages.org Subject: Re: [outages] Sprint
Derek McDaniel wrote:
Anyone having any data issues with Sprint? I'm seeing issues with Email
and
IM services. Anyone else experiencing this?
Sprint the cell phone? Sprintlink? What IM services? What email? Where?
No problems with my Sprintlink circuits.
~Seth _______________________________________________ outages mailing list outages@outages.org https://puck.nether.net/mailman/listinfo/outages
__________ NOD32 4171 (20090619) Information __________
This message was checked by NOD32 antivirus system. http://www.eset.com
_______________________________________________ outages mailing list outages@outages.org https://puck.nether.net/mailman/listinfo/outages ------------------------------------------------------------------------
No virus found in this incoming message. Checked by AVG - www.avg.com Version: 8.5.339 / Virus Database: 270.12.80/2187 - Release Date: 06/19/09 06:53:00
We have seen TXT messages being delayed 20 min at times this last week...which we rely on for up/down messages.....Sprint goes through periods of this st!t all the time and we never get a response as to why... --chris __________ NOD32 4172 (20090619) Information __________ This message was checked by NOD32 antivirus system. http://www.eset.com
participants (13)
-
Chaim Rieger -
Charles Wyble -
ChrisSerafin -
Clayton Zekelman -
Dan Parsons -
Derek McDaniel -
Jay Hennigan -
Jeff Cartier -
John Martinez -
Larry Vaden -
Robin Pimentel -
Seth Mattinen -
William Kern