Level 3 router in CHI just barfed?
Anyone else having a problem with Level 3 in Chicago right now? We just saw a bunch of routes out of Chicago go bonkers. -- Brooks R. Bridges Telecommunications Manager Ifbyphone, Inc. Phone: (847) 983-3000 Fax: (847) 676-6553 bbridges@ifbyphone.com http://www.ifbyphone.com
Yeah, definitely wasn't a phantom. If anyone ends up with a master ticket that would be nice to know about. -----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Brooks Bridges Sent: Friday, February 11, 2011 7:18 AM To: outages@outages.org Subject: [outages] Level 3 router in CHI just barfed? Anyone else having a problem with Level 3 in Chicago right now? We just saw a bunch of routes out of Chicago go bonkers. -- Brooks R. Bridges Telecommunications Manager Ifbyphone, Inc. Phone: (847) 983-3000 Fax: (847) 676-6553 bbridges@ifbyphone.com http://www.ifbyphone.com _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Nothing on the portal, but the detail to 4622732 - Fiber Damage due to inclement weather/ice is an interesting read. Glad I am not a Level 3 field tech, I hate cold weather. Robert D. Scott Robert@ufl.edu Senior Network Engineer 352-273-0113 Phone CNS - Network Services 352-392-2061 CNS Phone Tree University of Florida 352-273-0743 FAX Florida Lambda Rail 352-294-3571 FLR NOC Gainesville, FL 32611 321-663-0421 Cell 3216630421@messaging.sprintpcs.com -----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Dan Pinkard Sent: Friday, February 11, 2011 10:54 AM To: Brooks Bridges; outages@outages.org Subject: Re: [outages] Level 3 router in CHI just barfed? Yeah, definitely wasn't a phantom. If anyone ends up with a master ticket that would be nice to know about. -----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Brooks Bridges Sent: Friday, February 11, 2011 7:18 AM To: outages@outages.org Subject: [outages] Level 3 router in CHI just barfed? Anyone else having a problem with Level 3 in Chicago right now? We just saw a bunch of routes out of Chicago go bonkers. -- Brooks R. Bridges Telecommunications Manager Ifbyphone, Inc. Phone: (847) 983-3000 Fax: (847) 676-6553 bbridges@ifbyphone.com http://www.ifbyphone.com _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Could you share some of those details? On 2/11/2011 8:13 AM, Scott, Robert D. wrote:
Nothing on the portal, but the detail to 4622732 - Fiber Damage due to inclement weather/ice is an interesting read. Glad I am not a Level 3 field tech, I hate cold weather.
Robert D. Scott Robert@ufl.edu Senior Network Engineer 352-273-0113 Phone CNS - Network Services 352-392-2061 CNS Phone Tree University of Florida 352-273-0743 FAX Florida Lambda Rail 352-294-3571 FLR NOC Gainesville, FL 32611 321-663-0421 Cell 3216630421@messaging.sprintpcs.com
-----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Dan Pinkard Sent: Friday, February 11, 2011 10:54 AM To: Brooks Bridges; outages@outages.org Subject: Re: [outages] Level 3 router in CHI just barfed?
Yeah, definitely wasn't a phantom. If anyone ends up with a master ticket that would be nice to know about.
-----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Brooks Bridges Sent: Friday, February 11, 2011 7:18 AM To: outages@outages.org Subject: [outages] Level 3 router in CHI just barfed?
Anyone else having a problem with Level 3 in Chicago right now? We just saw a bunch of routes out of Chicago go bonkers.
-- Brooks R. Bridges Telecommunications Manager Ifbyphone, Inc. Phone: (847) 983-3000 Fax: (847) 676-6553 bbridges@ifbyphone.com http://www.ifbyphone.com
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
I am not sure that the release of specific Portal information from a vendor is appropriate, or not a violation of the AUP. The public information is that a fiber conduit that travels near or under water between Tulsa and Barnsdale OK is being crushed by ice, and degrading the optical characteristics of the fiber. Besides affecting the vendor's circuits, several major circuits that make up part of the Research and Education network backbones are also affected. The internal notes indicate that there are other customers affected. Time of day and temperature are affecting the magnitude of disruptions. The problem goes back to the 8th of Feb. I can only imagine the work being done to try and thaw out and dry out the path, given the weather conditions. It is currently 35, with an expected low tonight around 18. Robert D. Scott Robert@ufl.edu Senior Network Engineer 352-273-0113 Phone CNS - Network Services 352-392-2061 CNS Phone Tree University of Florida 352-273-0743 FAX Florida Lambda Rail 352-294-3571 FLR NOC Gainesville, FL 32611 321-663-0421 Cell 3216630421@messaging.sprintpcs.com -----Original Message----- From: Janet Sullivan [mailto:netgeek@bgp4.net] Sent: Friday, February 11, 2011 12:03 PM To: Scott, Robert D. Cc: Dan Pinkard; Brooks Bridges; outages@outages.org Subject: Re: [outages] Level 3 router in CHI just barfed? Could you share some of those details? On 2/11/2011 8:13 AM, Scott, Robert D. wrote:
Nothing on the portal, but the detail to 4622732 - Fiber Damage due to inclement weather/ice is an interesting read. Glad I am not a Level 3 field tech, I hate cold weather.
Robert D. Scott Robert@ufl.edu Senior Network Engineer 352-273-0113 Phone CNS - Network Services 352-392-2061 CNS Phone Tree University of Florida 352-273-0743 FAX Florida Lambda Rail 352-294-3571 FLR NOC Gainesville, FL 32611 321-663-0421 Cell 3216630421@messaging.sprintpcs.com
-----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Dan Pinkard Sent: Friday, February 11, 2011 10:54 AM To: Brooks Bridges; outages@outages.org Subject: Re: [outages] Level 3 router in CHI just barfed?
Yeah, definitely wasn't a phantom. If anyone ends up with a master ticket that would be nice to know about.
-----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Brooks Bridges Sent: Friday, February 11, 2011 7:18 AM To: outages@outages.org Subject: [outages] Level 3 router in CHI just barfed?
Anyone else having a problem with Level 3 in Chicago right now? We just saw a bunch of routes out of Chicago go bonkers.
-- Brooks R. Bridges Telecommunications Manager Ifbyphone, Inc. Phone: (847) 983-3000 Fax: (847) 676-6553 bbridges@ifbyphone.com http://www.ifbyphone.com
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
aaaaand there it goes again... Brooks R. Bridges Telecommunications Manager Ifbyphone, Inc. Phone: (847) 983-3000 Fax: (847) 676-6553 bbridges@ifbyphone.com http://www.ifbyphone.com Scott, Robert D. wrote:
I am not sure that the release of specific Portal information from a vendor is appropriate, or not a violation of the AUP.
The public information is that a fiber conduit that travels near or under water between Tulsa and Barnsdale OK is being crushed by ice, and degrading the optical characteristics of the fiber. Besides affecting the vendor's circuits, several major circuits that make up part of the Research and Education network backbones are also affected. The internal notes indicate that there are other customers affected. Time of day and temperature are affecting the magnitude of disruptions.
The problem goes back to the 8th of Feb. I can only imagine the work being done to try and thaw out and dry out the path, given the weather conditions. It is currently 35, with an expected low tonight around 18.
Robert D. Scott Robert@ufl.edu Senior Network Engineer 352-273-0113 Phone CNS - Network Services 352-392-2061 CNS Phone Tree University of Florida 352-273-0743 FAX Florida Lambda Rail 352-294-3571 FLR NOC Gainesville, FL 32611 321-663-0421 Cell 3216630421@messaging.sprintpcs.com
-----Original Message----- From: Janet Sullivan [mailto:netgeek@bgp4.net] Sent: Friday, February 11, 2011 12:03 PM To: Scott, Robert D. Cc: Dan Pinkard; Brooks Bridges; outages@outages.org Subject: Re: [outages] Level 3 router in CHI just barfed?
Could you share some of those details?
On 2/11/2011 8:13 AM, Scott, Robert D. wrote:
Nothing on the portal, but the detail to 4622732 - Fiber Damage due to inclement weather/ice is an interesting read. Glad I am not a Level 3 field tech, I hate cold weather.
Robert D. Scott Robert@ufl.edu Senior Network Engineer 352-273-0113 Phone CNS - Network Services 352-392-2061 CNS Phone Tree University of Florida 352-273-0743 FAX Florida Lambda Rail 352-294-3571 FLR NOC Gainesville, FL 32611 321-663-0421 Cell 3216630421@messaging.sprintpcs.com
-----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Dan Pinkard Sent: Friday, February 11, 2011 10:54 AM To: Brooks Bridges; outages@outages.org Subject: Re: [outages] Level 3 router in CHI just barfed?
Yeah, definitely wasn't a phantom. If anyone ends up with a master ticket that would be nice to know about.
-----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Brooks Bridges Sent: Friday, February 11, 2011 7:18 AM To: outages@outages.org Subject: [outages] Level 3 router in CHI just barfed?
Anyone else having a problem with Level 3 in Chicago right now? We just saw a bunch of routes out of Chicago go bonkers.
-- Brooks R. Bridges Telecommunications Manager Ifbyphone, Inc. Phone: (847) 983-3000 Fax: (847) 676-6553 bbridges@ifbyphone.com http://www.ifbyphone.com
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Received information about a code upgrade on a router that was needed in Denver, CO, because of a bug. Travis Marlow | desk: 913.322.9822 | mobile: 913.522.9185 | NetOps Manager ________________________________ From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Brooks Bridges Sent: Friday, February 11, 2011 11:46 AM To: outages@outages.org Subject: Re: [outages] Level 3 router in CHI just barfed? aaaaand there it goes again... Brooks R. Bridges Telecommunications Manager Ifbyphone, Inc. Phone: (847) 983-3000 Fax: (847) 676-6553 bbridges@ifbyphone.com http://www.ifbyphone.com Scott, Robert D. wrote: I am not sure that the release of specific Portal information from a vendor is appropriate, or not a violation of the AUP. The public information is that a fiber conduit that travels near or under water between Tulsa and Barnsdale OK is being crushed by ice, and degrading the optical characteristics of the fiber. Besides affecting the vendor's circuits, several major circuits that make up part of the Research and Education network backbones are also affected. The internal notes indicate that there are other customers affected. Time of day and temperature are affecting the magnitude of disruptions. The problem goes back to the 8th of Feb. I can only imagine the work being done to try and thaw out and dry out the path, given the weather conditions. It is currently 35, with an expected low tonight around 18. Robert D. Scott Robert@ufl.edu Senior Network Engineer 352-273-0113 Phone CNS - Network Services 352-392-2061 CNS Phone Tree University of Florida 352-273-0743 FAX Florida Lambda Rail 352-294-3571 FLR NOC Gainesville, FL 32611 321-663-0421 Cell 3216630421@messaging.sprintpcs.com -----Original Message----- From: Janet Sullivan [mailto:netgeek@bgp4.net] Sent: Friday, February 11, 2011 12:03 PM To: Scott, Robert D. Cc: Dan Pinkard; Brooks Bridges; outages@outages.org Subject: Re: [outages] Level 3 router in CHI just barfed? Could you share some of those details? On 2/11/2011 8:13 AM, Scott, Robert D. wrote: Nothing on the portal, but the detail to 4622732 - Fiber Damage due to inclement weather/ice is an interesting read. Glad I am not a Level 3 field tech, I hate cold weather. Robert D. Scott Robert@ufl.edu Senior Network Engineer 352-273-0113 Phone CNS - Network Services 352-392-2061 CNS Phone Tree University of Florida 352-273-0743 FAX Florida Lambda Rail 352-294-3571 FLR NOC Gainesville, FL 32611 321-663-0421 Cell 3216630421@messaging.sprintpcs.com -----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Dan Pinkard Sent: Friday, February 11, 2011 10:54 AM To: Brooks Bridges; outages@outages.org Subject: Re: [outages] Level 3 router in CHI just barfed? Yeah, definitely wasn't a phantom. If anyone ends up with a master ticket that would be nice to know about. -----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Brooks Bridges Sent: Friday, February 11, 2011 7:18 AM To: outages@outages.org Subject: [outages] Level 3 router in CHI just barfed? Anyone else having a problem with Level 3 in Chicago right now? We just saw a bunch of routes out of Chicago go bonkers. -- Brooks R. Bridges Telecommunications Manager Ifbyphone, Inc. Phone: (847) 983-3000 Fax: (847) 676-6553 bbridges@ifbyphone.com http://www.ifbyphone.com _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
That would explain the blip I saw about an hour ago. -----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Travis Marlow Sent: Friday, February 11, 2011 12:32 PM To: outages@outages.org Subject: Re: [outages] Level 3 router in CHI just barfed? Received information about a code upgrade on a router that was needed in Denver, CO, because of a bug. Travis Marlow | desk: 913.322.9822 | mobile: 913.522.9185 | NetOps Manager ________________________________ From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Brooks Bridges Sent: Friday, February 11, 2011 11:46 AM To: outages@outages.org Subject: Re: [outages] Level 3 router in CHI just barfed? aaaaand there it goes again... Brooks R. Bridges Telecommunications Manager Ifbyphone, Inc. Phone: (847) 983-3000 Fax: (847) 676-6553 bbridges@ifbyphone.com http://www.ifbyphone.com Scott, Robert D. wrote: I am not sure that the release of specific Portal information from a vendor is appropriate, or not a violation of the AUP. The public information is that a fiber conduit that travels near or under water between Tulsa and Barnsdale OK is being crushed by ice, and degrading the optical characteristics of the fiber. Besides affecting the vendor's circuits, several major circuits that make up part of the Research and Education network backbones are also affected. The internal notes indicate that there are other customers affected. Time of day and temperature are affecting the magnitude of disruptions. The problem goes back to the 8th of Feb. I can only imagine the work being done to try and thaw out and dry out the path, given the weather conditions. It is currently 35, with an expected low tonight around 18. Robert D. Scott Robert@ufl.edu Senior Network Engineer 352-273-0113 Phone CNS - Network Services 352-392-2061 CNS Phone Tree University of Florida 352-273-0743 FAX Florida Lambda Rail 352-294-3571 FLR NOC Gainesville, FL 32611 321-663-0421 Cell 3216630421@messaging.sprintpcs.com -----Original Message----- From: Janet Sullivan [mailto:netgeek@bgp4.net] Sent: Friday, February 11, 2011 12:03 PM To: Scott, Robert D. Cc: Dan Pinkard; Brooks Bridges; outages@outages.org Subject: Re: [outages] Level 3 router in CHI just barfed? Could you share some of those details? On 2/11/2011 8:13 AM, Scott, Robert D. wrote: Nothing on the portal, but the detail to 4622732 - Fiber Damage due to inclement weather/ice is an interesting read. Glad I am not a Level 3 field tech, I hate cold weather. Robert D. Scott Robert@ufl.edu Senior Network Engineer 352-273-0113 Phone CNS - Network Services 352-392-2061 CNS Phone Tree University of Florida 352-273-0743 FAX Florida Lambda Rail 352-294-3571 FLR NOC Gainesville, FL 32611 321-663-0421 Cell 3216630421@messaging.sprintpcs.com -----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Dan Pinkard Sent: Friday, February 11, 2011 10:54 AM To: Brooks Bridges; outages@outages.org Subject: Re: [outages] Level 3 router in CHI just barfed? Yeah, definitely wasn't a phantom. If anyone ends up with a master ticket that would be nice to know about. -----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Brooks Bridges Sent: Friday, February 11, 2011 7:18 AM To: outages@outages.org Subject: [outages] Level 3 router in CHI just barfed? Anyone else having a problem with Level 3 in Chicago right now? We just saw a bunch of routes out of Chicago go bonkers. -- Brooks R. Bridges Telecommunications Manager Ifbyphone, Inc. Phone: (847) 983-3000 Fax: (847) 676-6553 bbridges@ifbyphone.com http://www.ifbyphone.com _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
We saw it out of Detroit. At 10:18 AM 2/11/2011, Brooks Bridges wrote:
Anyone else having a problem with Level 3 in Chicago right now? We just saw a bunch of routes out of Chicago go bonkers.
-- Brooks R. Bridges Telecommunications Manager Ifbyphone, Inc. Phone: (847) 983-3000 Fax: (847) 676-6553 bbridges@ifbyphone.com http://www.ifbyphone.com
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
--- Clayton Zekelman Managed Network Systems Inc. (MNSi) 3363 Tecumseh Rd. E Windsor, Ontario N8W 1H4 tel. 519-985-8410 fax. 519-985-8409
participants (7)
-
Blake Pfankuch -
Brooks Bridges -
Clayton Zekelman -
Dan Pinkard -
Janet Sullivan -
Scott, Robert D. -
Travis Marlow