216.115.160.0/20 being blackholed
FYI - got a call from a customer last night about a Xerox site unreachable through us. Found the prefix coming from Cymru. Instructed our NOC to work with them to resolve. Cymru first claimed they were not sending it because it was not a bogon. Once we proved it was coming from them, they woke somebody up at our request and found it listed as a 'bad route', whatever that means. Their workaround was to shut down the peering to us. Anybody else use Cymru and have this issue? Our 2 peering addresses are 38.229.66.20 and 193.231.140.82 Thanks. -ben
They were experiencing some issues and sent out an update today you may wish to reach out to them if you haven't heard anything. The details did mention something about incorrect prefixes getting announced Paul From: Ben Bartsch <uwcableguy@gmail.com> Date: Tuesday, 16 July, 2013 7:29 AM To: <outages@outages.org> Subject: [outages] 216.115.160.0/20 being blackholed
FYI - got a call from a customer last night about a Xerox site unreachable through us. Found the prefix coming from Cymru. Instructed our NOC to work with them to resolve. Cymru first claimed they were not sending it because it was not a bogon. Once we proved it was coming from them, they woke somebody up at our request and found it listed as a 'bad route', whatever that means. Their workaround was to shut down the peering to us.
Anybody else use Cymru and have this issue? Our 2 peering addresses are 38.229.66.20 and 193.231.140.82
Thanks.
-ben _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Team CYMRU is having issues with their BGP Bogon Service. Below is an email I received from them a little bit ago. To all, We have identified a hardware problem associated with the fullbogons peering sessions that were taken offline roughly 12 hours ago. The service has been disabled and at this time we do not have an ETA on when service will be restored. We will issue updates at approximately 8am, 12pm, and 5pm EST until this outage is resolved. For the finer details please read below. 8pm EST 7/15/13, Team Cymru was notified of a couple problems with our Bogon service. At this time we did not know the severity and scope of the larger problem. 11pm EST 7/15/13 the issue was escalated from Operations to Engineering. 1am EST 7/16/13, we confirmed the propagation of incorrect prefixes. Workarounds failed to fully mitigate the issue, so all sessions were disabled. This ceased the propagation of incorrect prefixes. ~2am EST 7/16/13, we sent out notices to everyone we have current contact details on regarding the outage. 12:15pm EST 7/16/13, we are working on replacing / reconfiguring hardware to resolve this outage. Regards, Dave Team Cymru From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Paul Stewart Sent: Tuesday, July 16, 2013 1:06 PM To: Ben Bartsch; outages@outages.org Subject: Re: [outages] 216.115.160.0/20 being blackholed They were experiencing some issues and sent out an update today - you may wish to reach out to them if you haven't heard anything. The details did mention something about incorrect prefixes getting announced... Paul From: Ben Bartsch <uwcableguy@gmail.com<mailto:uwcableguy@gmail.com>> Date: Tuesday, 16 July, 2013 7:29 AM To: <outages@outages.org<mailto:outages@outages.org>> Subject: [outages] 216.115.160.0/20 being blackholed FYI - got a call from a customer last night about a Xerox site unreachable through us. Found the prefix coming from Cymru. Instructed our NOC to work with them to resolve. Cymru first claimed they were not sending it because it was not a bogon. Once we proved it was coming from them, they woke somebody up at our request and found it listed as a 'bad route', whatever that means. Their workaround was to shut down the peering to us. Anybody else use Cymru and have this issue? Our 2 peering addresses are 38.229.66.20 and 193.231.140.82 Thanks. -ben _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 7/16/13 7:29 AM, Ben Bartsch wrote:
FYI - got a call from a customer last night about a Xerox site unreachable through us. Found the prefix coming from Cymru. Instructed our NOC to work with them to resolve. Cymru first claimed they were not sending it because it was not a bogon. Once we proved it was coming from them, they woke somebody up at our request and found it listed as a 'bad route', whatever that means. Their workaround was to shut down the peering to us.
Anybody else use Cymru and have this issue? Our 2 peering addresses are 38.229.66.20 and 193.231.140.82
Thanks.
-ben
Hello Outages list! We have identified a hardware problem associated with the full bogons peering sessions that were taken offline roughly 12 hours ago. The service has been disabled and at this time we do not have an ETA on when service will be restored. We will issue updates at approximately 8am, 12pm, and 5pm EST until this outage is resolved. For the finer details please read below. 8pm EST 7/15/13, Team Cymru was notified of a couple problems with our Bogon service. At this time we did not know the severity and scope of the larger problem. 11pm EST 7/15/13 the issue was escalated from Operations to Engineering. 1am EST 7/16/13, we confirmed the propagation of incorrect prefixes. Workarounds failed to fully mitigate the issue, so all sessions were disabled. This ceased the propagation of incorrect prefixes. ~2am EST 7/16/13, we sent out notices to everyone we have current contact details on regarding the outage. 12:15pm EST 7/16/13, we are working on replacing / reconfiguring hardware to resolve this outage. On behalf of Team Cymru, I would like to apologize for this outage. we are actively working on getting the Bogons back online for distribution. In the interim, we are having listing updates periodically, it is located here: https://www.team-cymru.org/Services/Bogons/ For those who are using our service, thank you! Be assured we are working towards resolution. Sincerely, Don Napier Team Cymru Research NFP -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.18 (Darwin) Comment: GPGTools - http://gpgtools.org iQEcBAEBAgAGBQJR5YMIAAoJEKgi/9FGk+IZTlgIALIinXjZhjkqHZsnSlEo7syt 73gq1e3BzRfPOBJoliIMCFr4n3iN7DMwgXG8GvcpnrOprAtZvX7MCvz9HGTkIqbP UjJl+HiB2OujCPWlW+QZ9x89/f+y2vJz/VTwCrp8UAWaFc1QM4smI/t9K7fSivW4 NTknofDKhtYORIzHEcEHl4Z9Ecx5cBBSqBPGOE6YZMTmvdZXXpBJpmG8HdQmaf7O IFx2mZH4lsCpmu1rw32NZ8o/Qn5wVnAwYlFRhhkX8XzC82gLFL5Se0E3rsuPfZLP YW9OGGqqgowQtPZPLGHIB3DTQY1JBIJa6abiPhCclcbxdhCFtstsPRIlOYVh7/w= =MVV7 -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 7/16/13 1:29 PM, Don Napier wrote:
On 7/16/13 7:29 AM, Ben Bartsch wrote:
FYI - got a call from a customer last night about a Xerox site unreachable through us. Found the prefix coming from Cymru. Instructed our NOC to work with them to resolve. Cymru first claimed they were not sending it because it was not a bogon. Once we proved it was coming from them, they woke somebody up at our request and found it listed as a 'bad route', whatever that means. Their workaround was to shut down the peering to us.
Anybody else use Cymru and have this issue? Our 2 peering addresses are 38.229.66.20 and 193.231.140.82
Thanks.
-ben
Hello Outages list!
We have identified a hardware problem associated with the full bogons peering sessions that were taken offline roughly 12 hours ago. The service has been disabled and at this time we do not have an ETA on when service will be restored.
We will issue updates at approximately 8am, 12pm, and 5pm EST until this outage is resolved.
For the finer details please read below.
8pm EST 7/15/13, Team Cymru was notified of a couple problems with our Bogon service. At this time we did not know the severity and scope of the larger problem.
11pm EST 7/15/13 the issue was escalated from Operations to Engineering.
1am EST 7/16/13, we confirmed the propagation of incorrect prefixes. Workarounds failed to fully mitigate the issue, so all sessions were disabled. This ceased the propagation of incorrect prefixes.
~2am EST 7/16/13, we sent out notices to everyone we have current contact details on regarding the outage.
12:15pm EST 7/16/13, we are working on replacing / reconfiguring hardware to resolve this outage.
On behalf of Team Cymru, I would like to apologize for this outage. we are actively working on getting the Bogons back online for distribution.
In the interim, we are having listing updates periodically, it is located here:
https://www.team-cymru.org/Services/Bogons/
For those who are using our service, thank you! Be assured we are working towards resolution.
Sincerely,
Don Napier Team Cymru Research NFP
Hello Outages, Service was restored tonight at about EST 530PM 7/16/13. We replaced the faulty hardware (a router in this case) at our data center. The Bogon feeds should be flowing properly now. In the future, status of the Bogon Reference Project will be listed at: https://www.team-cymru.org/Services/Bogons/index.html Thanks you for your patience, Don Napier Team Cymru Research NFP -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.18 (Darwin) Comment: GPGTools - http://gpgtools.org iQEcBAEBAgAGBQJR5cZWAAoJEKgi/9FGk+IZhQ4IAJmdfHYkGyFq/pCLRpd29fFu knSeZ13/5TWPMpE1kiXFJv+rtkIFVcI0855nW8XpE6f5c7VvI+6NqMw1HXcY5twD 6gnD0rUAwesFTpTMk36iU0+REsKWCsXuItCjMtKr7S3kLNjSQ8ymwRSeh+Fbgqp6 TYqyhjOwHsuFboII4iy18cg3Y4Gi1hP+sWOhTTSkvEEBxAQTM6xs0ENKXbPAsHkY 6tf/R31fREpg0HUcpUewYBv5jYrfUV2/e0Brr9ACkExhb9JxA3vNqBW8y+RXW7yu 66Om3LkD/4f3Wud04GohQRvr+vFe8cwK5ePTlSDX/S+kE677ItWaHLhwFyix/QI= =7QMB -----END PGP SIGNATURE-----
Thanks Team Cymru. Thanks LONI - ASN 32440 for your work to discover the problem. -bb On Tue, Jul 16, 2013 at 5:16 PM, Don Napier <dnapier@cymru.com> wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 7/16/13 1:29 PM, Don Napier wrote:
On 7/16/13 7:29 AM, Ben Bartsch wrote:
FYI - got a call from a customer last night about a Xerox site unreachable through us. Found the prefix coming from Cymru. Instructed our NOC to work with them to resolve. Cymru first claimed they were not sending it because it was not a bogon. Once we proved it was coming from them, they woke somebody up at our request and found it listed as a 'bad route', whatever that means. Their workaround was to shut down the peering to us.
Anybody else use Cymru and have this issue? Our 2 peering addresses are 38.229.66.20 and 193.231.140.82
Thanks.
-ben
Hello Outages list!
We have identified a hardware problem associated with the full bogons peering sessions that were taken offline roughly 12 hours ago. The service has been disabled and at this time we do not have an ETA on when service will be restored.
We will issue updates at approximately 8am, 12pm, and 5pm EST until this outage is resolved.
For the finer details please read below.
8pm EST 7/15/13, Team Cymru was notified of a couple problems with our Bogon service. At this time we did not know the severity and scope of the larger problem.
11pm EST 7/15/13 the issue was escalated from Operations to Engineering.
1am EST 7/16/13, we confirmed the propagation of incorrect prefixes. Workarounds failed to fully mitigate the issue, so all sessions were disabled. This ceased the propagation of incorrect prefixes.
~2am EST 7/16/13, we sent out notices to everyone we have current contact details on regarding the outage.
12:15pm EST 7/16/13, we are working on replacing / reconfiguring hardware to resolve this outage.
On behalf of Team Cymru, I would like to apologize for this outage. we are actively working on getting the Bogons back online for distribution.
In the interim, we are having listing updates periodically, it is located here:
https://www.team-cymru.org/Services/Bogons/
For those who are using our service, thank you! Be assured we are working towards resolution.
Sincerely,
Don Napier Team Cymru Research NFP
Hello Outages,
Service was restored tonight at about EST 530PM 7/16/13. We replaced the faulty hardware (a router in this case) at our data center. The Bogon feeds should be flowing properly now.
In the future, status of the Bogon Reference Project will be listed at:
https://www.team-cymru.org/Services/Bogons/index.html
Thanks you for your patience,
Don Napier Team Cymru Research NFP -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.18 (Darwin) Comment: GPGTools - http://gpgtools.org
iQEcBAEBAgAGBQJR5cZWAAoJEKgi/9FGk+IZhQ4IAJmdfHYkGyFq/pCLRpd29fFu knSeZ13/5TWPMpE1kiXFJv+rtkIFVcI0855nW8XpE6f5c7VvI+6NqMw1HXcY5twD 6gnD0rUAwesFTpTMk36iU0+REsKWCsXuItCjMtKr7S3kLNjSQ8ymwRSeh+Fbgqp6 TYqyhjOwHsuFboII4iy18cg3Y4Gi1hP+sWOhTTSkvEEBxAQTM6xs0ENKXbPAsHkY 6tf/R31fREpg0HUcpUewYBv5jYrfUV2/e0Brr9ACkExhb9JxA3vNqBW8y+RXW7yu 66Om3LkD/4f3Wud04GohQRvr+vFe8cwK5ePTlSDX/S+kE677ItWaHLhwFyix/QI= =7QMB -----END PGP SIGNATURE-----
participants (4)
-
Ben Bartsch -
Don Napier -
Paul Stewart -
Tim Glen