…apparently Slack still don’t know why they’re down. Probably because they try to use Slack to discuss it. https://status.slack.com/2021-09/06c1e17de93e7dc2 -Bill
Perhaps they're also not aware of the Letsencrypt intermediate certificate expiring as well and they have some internal stuff encrypted with LE? https://community.letsencrypt.org/t/r3-intermediate-certificate-has-expired-... The timings seem to be consistent On Thu, 30 Sept 2021 at 12:48, Bill Woodcock via Outages < outages@outages.org> wrote:
…apparently Slack still don’t know why they’re down.
Probably because they try to use Slack to discuss it.
https://status.slack.com/2021-09/06c1e17de93e7dc2
-Bill
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
I have a handful of clients that use LE on their FreePBX phone servers...and it's starting to look like the Digium D-series phones don't trust the new root certificate yet. No SIP-TLS, no SRTP, no "phone apps". Trying to debug those phones via syslog without being able to SSH into them and run commands and look at files is horrible. "State phone.idle, process event: error.sip_transport.SIP_TLS_CERT_INVALID.0x00000004" Maybe it's a coincidence, but the http://digium.com/ website is down too. I know they merged with Sangoma recently, but I couldn't imagine they'd just dump the domain. -A On Thu, Sep 30, 2021 at 12:39 PM Ken Gilmour via Outages < outages@outages.org> wrote:
Perhaps they're also not aware of the Letsencrypt intermediate certificate expiring as well and they have some internal stuff encrypted with LE? https://community.letsencrypt.org/t/r3-intermediate-certificate-has-expired-... The timings seem to be consistent
On Thu, 30 Sept 2021 at 12:48, Bill Woodcock via Outages < outages@outages.org> wrote:
…apparently Slack still don’t know why they’re down.
Probably because they try to use Slack to discuss it.
https://status.slack.com/2021-09/06c1e17de93e7dc2
-Bill
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
There is an update to FreePBX you'll want to install for the "certman" module. Install it from the edge repo, and it should fix the local LE intermediate cert issue on the PBX. Unsure of D-series phone operation/interaction though. "digium.com" (un)availability is due to some changes in site hosting the web heads are working on. Should be clear now! --Tim (CloudNetEng @sangoma) On Thu, Sep 30, 2021 at 3:48 PM Aaron C. de Bruyn via Outages <outages@outages.org> wrote:
I have a handful of clients that use LE on their FreePBX phone servers...and it's starting to look like the Digium D-series phones don't trust the new root certificate yet. No SIP-TLS, no SRTP, no "phone apps". Trying to debug those phones via syslog without being able to SSH into them and run commands and look at files is horrible.
"State phone.idle, process event: error.sip_transport.SIP_TLS_CERT_INVALID.0x00000004"
Maybe it's a coincidence, but the http://digium.com/ website is down too. I know they merged with Sangoma recently, but I couldn't imagine they'd just dump the domain.
-A
On Thu, Sep 30, 2021 at 12:39 PM Ken Gilmour via Outages <outages@outages.org> wrote:
Perhaps they're also not aware of the Letsencrypt intermediate certificate expiring as well and they have some internal stuff encrypted with LE? https://community.letsencrypt.org/t/r3-intermediate-certificate-has-expired-... The timings seem to be consistent
On Thu, 30 Sept 2021 at 12:48, Bill Woodcock via Outages <outages@outages.org> wrote:
…apparently Slack still don’t know why they’re down.
Probably because they try to use Slack to discuss it.
https://status.slack.com/2021-09/06c1e17de93e7dc2
-Bill
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Back in the day, I was responsible for an SMSC in one of the early GSM carriers. It meant that I had to carry a pager AND my phone because I needed to get alerts of the phone's messaging system didn't work. On September 30, 2021 11:28:19 AM PDT, Bill Woodcock via Outages <outages@outages.org> wrote:
…apparently Slack still don’t know why they’re down.
Probably because they try to use Slack to discuss it.
https://status.slack.com/2021-09/06c1e17de93e7dc2
-Bill
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Seems the facebook.com domain no longer has any DNS records? Getting nxdomain from most clients, same when digging @ 1.1.1.1 8.8.8.8 etc https://toolbox.googleapps.com/apps/dig/#A/facebook.com -- Jon Sands MFI Labs https://fohdeesha.com/
A carrier networking engineer friend of mine observed that the network that those DNS servers live in is no longer in the global routing table. Sounds like BGP issues rather than DNS ones, I guess. (I won’t admit to understanding BGP.) -Rich Rich Lafferty rich@lafferty.ca / 647-377-1165 Toronto, Canada http://www.lafferty.ca On Oct 4, 2021, 1:14 PM -0300, Jon Sands via Outages <outages@outages.org>, wrote:
Seems the facebook.com domain no longer has any DNS records? Getting nxdomain from most clients, same when digging @ 1.1.1.1 8.8.8.8 etc
https://toolbox.googleapps.com/apps/dig/#A/facebook.com
-- Jon Sands MFI Labs https://fohdeesha.com/
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
I’ve seen this BGP withdrawal cited in a number of places. However we peer with FB, and all four sessions are up and have been live for days/months (FB had maintenance on one peering router a couple of weeks ago) and all 30 prefixes are consistent and announced. The NS IP’s are in the announcements, but you can’t ICMP to them, or query them as NS for the auth records either.. Peter Knapp From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Rich Lafferty via Outages Sent: 04 October 2021 17:27 To: Outages mailing list <outages@outages.org> Subject: Re: [outages] Facebook.com no records? A carrier networking engineer friend of mine observed that the network that those DNS servers live in is no longer in the global routing table. Sounds like BGP issues rather than DNS ones, I guess. (I won’t admit to understanding BGP.) -Rich Rich Lafferty rich@lafferty.ca<mailto:rich@lafferty.ca> / 647-377-1165 Toronto, Canada http://www.lafferty.ca [https://s3.amazonaws.com/htmlsig-assets/round/twitter.png]<https://twitter.com/mendel> <https://twitter.com/mendel> [https://s3.amazonaws.com/htmlsig-assets/round/facebook.png] <https://facebook.com/richlafferty> <https://facebook.com/richlafferty> [https://s3.amazonaws.com/htmlsig-assets/round/linkedin.png] <https://linkedin.com/in/richlafferty> <https://linkedin.com/in/richlafferty> [https://s3.amazonaws.com/htmlsig-assets/round/instagram.png] <https://instagram.com/wlonkly> <https://instagram.com/wlonkly> [https://s3.amazonaws.com/htmlsig-assets/round/flickr.png] <https://www.flickr.com/photos/mendel> <https://www.flickr.com/photos/mendel> [https://s3.amazonaws.com/htmlsig-assets/round/github.png] <https://github.com/wlonkly> <https://github.com/wlonkly> [https://s3.amazonaws.com/htmlsig-assets/round/tumblr.png] <https://mendel.tumblr.com> <https://mendel.tumblr.com> On Oct 4, 2021, 1:14 PM -0300, Jon Sands via Outages <outages@outages.org<mailto:outages@outages.org>>, wrote: Seems the facebook.com domain no longer has any DNS records? Getting nxdomain from most clients, same when digging @ 1.1.1.1 8.8.8.8 etc https://toolbox.googleapps.com/apps/dig/#A/facebook.com -- Jon Sands MFI Labs https://fohdeesha.com/ _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
Hrm. There were definitely no routes for 129.134.31.0/24 from multiple route servers at around 14:00 UTC today, but I see that there are now routes for a containing /17 in AS 32934 now (20:25 UTC). You can see the /24 disappear: https://stat.ripe.net/widget/bgplay#w.resource=129.134.30.0/24&w.ignoreReann... -Rich Rich Lafferty rich@lafferty.ca / 647-377-1165 Toronto, Canada http://www.lafferty.ca On Oct 4, 2021, 5:01 PM -0300, Peter Knapp <Peter.Knapp@ccsleeds.co.uk>, wrote:
I’ve seen this BGP withdrawal cited in a number of places.
However we peer with FB, and all four sessions are up and have been live for days/months (FB had maintenance on one peering router a couple of weeks ago) and all 30 prefixes are consistent and announced.
The NS IP’s are in the announcements, but you can’t ICMP to them, or query them as NS for the auth records either..
Peter Knapp
From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Rich Lafferty via Outages Sent: 04 October 2021 17:27 To: Outages mailing list <outages@outages.org> Subject: Re: [outages] Facebook.com no records?
A carrier networking engineer friend of mine observed that the network that those DNS servers live in is no longer in the global routing table. Sounds like BGP issues rather than DNS ones, I guess. (I won’t admit to understanding BGP.)
-Rich
Rich Lafferty rich@lafferty.ca / 647-377-1165 Toronto, Canada http://www.lafferty.ca
On Oct 4, 2021, 1:14 PM -0300, Jon Sands via Outages <outages@outages.org>, wrote:
quote_type Seems the facebook.com domain no longer has any DNS records? Getting nxdomain from most clients, same when digging @ 1.1.1.1 8.8.8.8 etc
https://toolbox.googleapps.com/apps/dig/#A/facebook.com
-- Jon Sands MFI Labs https://fohdeesha.com/
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Yes, that's the underlying issue. A faulty BGP change caused a bunch of routes to be withdrawn, including the Facebook DNS servers, which they self host. On Mon, Oct 4, 2021 at 5:37 PM Rich Lafferty via Outages < outages@outages.org> wrote:
A carrier networking engineer friend of mine observed that the network that those DNS servers live in is no longer in the global routing table. Sounds like BGP issues rather than DNS ones, I guess. (I won’t admit to understanding BGP.)
-Rich
*Rich Lafferty* rich@lafferty.ca / 647-377-1165 Toronto, Canada http://www.lafferty.ca
<https://twitter.com/mendel> <https://twitter.com/mendel> <https://facebook.com/richlafferty> <https://facebook.com/richlafferty> <https://linkedin.com/in/richlafferty> <https://linkedin.com/in/richlafferty> <https://instagram.com/wlonkly> <https://instagram.com/wlonkly> <https://www.flickr.com/photos/mendel> <https://www.flickr.com/photos/mendel> <https://github.com/wlonkly> <https://github.com/wlonkly> <https://mendel.tumblr.com> <https://mendel.tumblr.com> On Oct 4, 2021, 1:14 PM -0300, Jon Sands via Outages <outages@outages.org>, wrote:
Seems the facebook.com domain no longer has any DNS records? Getting nxdomain from most clients, same when digging @ 1.1.1.1 8.8.8.8 etc
https://toolbox.googleapps.com/apps/dig/#A/facebook.com
-- Jon Sands MFI Labs https://fohdeesha.com/
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Cloudflare summed it up nicely: https://blog.cloudflare.com/october-2021-facebook-outage/ Rob++ ----- Original Message -----
From: "Jason Kuehl via Outages" <outages@outages.org> To: "Jason Kuehl via Outages" <outages@outages.org> Sent: Monday, October 4, 2021 11:26:52 AM Subject: Re: [outages] Facebook.com no records?
A carrier networking engineer friend of mine observed that the network that those DNS servers live in is no longer in the global routing table. Sounds like BGP issues rather than DNS ones, I guess. (I won’t admit to understanding BGP.)
-Rich
Rich Lafferty
-- Internet: windsor@warthog.com Life: Rob@Van_Alstyne.Texas.USA.Earth "They couldn't hit an elephant at this distance." -- Major General John Sedgwick
I wonder if someone was fired and deleted something.
host facebook.com Host facebook.com not found: 2(SERVFAIL) dig facebook.com
; <<>> DiG 9.6.1-P1 <<>> facebook.com ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 27869 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0 ;; QUESTION SECTION: ;facebook.com. IN A ;; Query time: 0 msec ;; SERVER: 10.64.4.2#53(10.64.4.2) ;; WHEN: Mon Oct 4 11:00:17 2021 ;; MSG SIZE rcvd: 30
John Quinby Vision Net | Network Operations Center Lead p: (406) 216-4620 m: (406) 836-0223 w: https://vision.net -----Original Message----- From: Outages <outages-bounces@outages.org> On Behalf Of Jon Sands via Outages Sent: Monday, October 4, 2021 9:57 AM To: outages@outages.org Subject: [outages] Facebook.com no records? Seems the facebook.com domain no longer has any DNS records? Getting nxdomain from most clients, same when digging @ 1.1.1.1 8.8.8.8 etc https://toolbox.googleapps.com/apps/dig/#A/facebook.com -- Jon Sands MFI Labs https://fohdeesha.com/ _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
I’d be surprised if a single person had that much authority. Good news is there might be some engineering jobs open up at FB ________________________________ From: Outages <outages-bounces@outages.org> on behalf of John Quinby via Outages <outages@outages.org> Sent: Monday, October 4, 2021 1:03:11 PM To: outages@outages.org <outages@outages.org> Subject: [EXTERNAL] Re: [outages] Facebook.com no records? CAUTION: The e-mail below is from an external source. Please exercise caution before opening attachments, clicking links, or following guidance. I wonder if someone was fired and deleted something.
host facebook.com Host facebook.com not found: 2(SERVFAIL) dig facebook.com
; <<>> DiG 9.6.1-P1 <<>> facebook.com ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 27869 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0 ;; QUESTION SECTION: ;facebook.com. IN A ;; Query time: 0 msec ;; SERVER: 10.64.4.2#53(10.64.4.2) ;; WHEN: Mon Oct 4 11:00:17 2021 ;; MSG SIZE rcvd: 30
John Quinby Vision Net | Network Operations Center Lead p: (406) 216-4620 m: (406) 836-0223 w: https://vision.net -----Original Message----- From: Outages <outages-bounces@outages.org> On Behalf Of Jon Sands via Outages Sent: Monday, October 4, 2021 9:57 AM To: outages@outages.org Subject: [outages] Facebook.com no records? Seems the facebook.com domain no longer has any DNS records? Getting nxdomain from most clients, same when digging @ 1.1.1.1 8.8.8.8 etc https://toolbox.googleapps.com/apps/dig/#A/facebook.com -- Jon Sands MFI Labs https://fohdeesha.com/ _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages E-MAIL CONFIDENTIALITY NOTICE: The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.
According to r/sysadmin, this is a peering issue. Update 1440 UTC: As many of you know, DNS for FB services has been affected and this is likely a symptom of the actual issue, and that's that BGP peering with Facebook peering routers has gone down, very likely due to a configuration change that went into effect shortly before the outages happened (started roughly 1540 UTC). There are people now trying to gain access to the peering routers to implement fixes, but the people with physical access is separate from the people with knowledge of how to actually authenticate to the systems and people who know what to actually do, so there is now a logistical challenge with getting all that knowledge unified. Part of this is also due to lower staffing in data centers due to pandemic measures. See: https://www.reddit.com/r/sysadmin/comments/q181fv/looks_like_facebook_is_dow... On Oct 4, 2021, at 11:56 AM, Jon Sands via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: Seems the facebook.com<http://facebook.com> domain no longer has any DNS records? Getting nxdomain from most clients, same when digging @ 1.1.1.1 8.8.8.8 etc https://toolbox.googleapps.com/apps/dig/#A/facebook.com -- Jon Sands MFI Labs https://fohdeesha.com/ _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
See here (with a grain of salt): https://www.reddit.com/r/sysadmin/comments/q181fv/looks_like_facebook_is_dow... "DNS for FB services has been affected and this is likely a symptom of the actual issue, and that's that BGP peering with Facebook peering routers has gone down, very likely due to a configuration change that went into effect shortly before the outages happened (started roughly 1540 UTC)." Hagen Miller Senior Systems Engineer Computer Services, Inc. -----Original Message----- From: Outages <outages-bounces@outages.org> On Behalf Of Jon Sands via Outages Sent: Monday, October 4, 2021 9:57 AM To: outages@outages.org Subject: [outages] Facebook.com no records? WARNING: This email is from an external source. Do not click links or attachments unless you recognize the sender and know the content is safe. Seems the facebook.com domain no longer has any DNS records? Getting nxdomain from most clients, same when digging @ 1.1.1.1 8.8.8.8 etc https://toolbox.googleapps.com/apps/dig/#A/facebook.com -- Jon Sands MFI Labs https://fohdeesha.com/ _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages ________________________________ This e-mail and any files transmitted with it are the property of Computer Services, Inc. and/or its divisions, are confidential, and are intended solely for the use of the individual or entity to whom this e-mail is addressed. The sender does not accept any responsibility for any loss, disruption or damage to your data or computer systems that may occur from content of any sort contained in, transmitted with, or added to this e-mail. If you are not a named recipient or otherwise have reason to believe that you have received this message in error, please notify the sender and delete this message immediately from your computer. Any other use, retention, dissemination, forwarding, printing, or copying of this e-mail is strictly prohibited.
This reddit thread has comments from someone on the FB recovery team about BGP peering routers being down and requiring on site resolution. https://www.reddit.com/r/sysadmin/comments/q181fv/looks_like_facebook_is_dow... Nick On Mon, Oct 4, 2021 at 12:08 PM Jon Sands via Outages <outages@outages.org> wrote:
Seems the facebook.com domain no longer has any DNS records? Getting nxdomain from most clients, same when digging @ 1.1.1.1 8.8.8.8 etc
https://toolbox.googleapps.com/apps/dig/#A/facebook.com
-- Jon Sands MFI Labs https://fohdeesha.com/
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
participants (15)
-
Aaron C. de Bruyn -
Bill Woodcock -
Hagen Miller -
John Quinby -
Jon Sands -
Josh Ritz -
Ken Gilmour -
Michael Wheeler -
Nick Rhyne -
Peter Knapp -
Rich Lafferty -
Rob Windsor -
Stephens, Jamie A -
Tim Nelson -
Trey Valenta