Email to Microsoft Office 365 users
Our last queue check reported DNS failures looking up <domain name>.mail.protection.outlook.com for 11 such domains. Just picking one, staples.com: root@nagios:/tmp# dig MX staples.com +short 10 staples-com.mail.protection.outlook.com. root@nagios:/tmp# dig staples-com.mail.protection.outlook.com ; <<>> DiG 9.7.3 <<>> staples-com.mail.protection.outlook.com ;; global options: +cmd ;; connection timed out; no servers could be reached root@nagios:/tmp# root@nagios:/tmp# host MX staples.com ;; connection timed out; no servers could be reached root@nagios:/tmp# Anyone else seeing this? A colleague says that O365 is having issues and will be forwarding me some details. Regards, Frank Bulk
I was seeing what looked like "general slowness" when it came to getting A, MX, or ANY back for staples-com.mail.protection.outlook.com. Much of the delay (using +trace on dig) showed high latency in a couple spots, but I never saw any actual timeouts: ;; Received 763 bytes from 192.5.5.241#53(f.root-servers.net) in 885 ms ;; Received 871 bytes from 192.48.79.30#53(j.gtld-servers.net) in 368 ms ;; Received 203 bytes from 65.55.37.62#53(ns1.msft.net) in 204 ms ;; Received 141 bytes from 134.170.140.11#53(ns1-gtm.glbdns.o365filtering.com) in 168 ms Total response time was roughly 4-5 full seconds. This was around 15:07:40 Pacific Time or so, give or take 3-4 minutes. Presently I'm not able to reproduce the behaviour; response times seem to be decent now (1 second tops), with the majority of the remaining delays being here: ;; Received 203 bytes from 208.84.2.53#53(ns2.msft.net) in 305 ms ;; Received 141 bytes from 157.56.116.52#53(ns1-gtm.glbdns.o365filtering.com) in 409 ms Nothing looks bad via mtr, except in the case of packets from me to 157.56.116.52, which appears to be going to Europe (I'm under the impression "ams" refers to Amsterdam): 15. xe-10-2-2-0.bl2-96c-1b.ntwk.msn.net (204.152.140.87) 0.0% 47 46 83.5 83.5 81.3 95.1 16. ae11-0.nyc-96cbe-1b.ntwk.msn.net (207.46.45.85) 37.8% 46 28 84.8 85.2 82.3 97.8 17. xe-0-3-1-0.lts-96cbe-1b.ntwk.msn.net (207.46.43.47) 0.0% 46 46 156.7 157.1 154.5 165.7 18. xe-2-2-2-0.ams-96c-1a.ntwk.msn.net (207.46.44.59) 4.3% 46 44 167.1 164.8 163.4 172.0 19. ??? The latencies here are likely normal, packet loss may be ICMP prioritisation-related (can't tell because destination or preceding routers don't respond to ICMP), but it does bring into question if Microsoft has a form of geo-located GSLB (if not, ignore. If so, why are queries from the US west coast going to Europe). -- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Making life hard for others since 1977. PGP 4BD6C0CB | On Tue, Oct 21, 2014 at 05:01:22PM -0500, Frank Bulk via Outages wrote:
Our last queue check reported DNS failures looking up <domain name>.mail.protection.outlook.com for 11 such domains.
Just picking one, staples.com:
root@nagios:/tmp# dig MX staples.com +short 10 staples-com.mail.protection.outlook.com. root@nagios:/tmp# dig staples-com.mail.protection.outlook.com
; <<>> DiG 9.7.3 <<>> staples-com.mail.protection.outlook.com ;; global options: +cmd ;; connection timed out; no servers could be reached root@nagios:/tmp# root@nagios:/tmp# host MX staples.com ;; connection timed out; no servers could be reached root@nagios:/tmp#
Anyone else seeing this? A colleague says that O365 is having issues and will be forwarding me some details.
Regards,
Frank Bulk
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
I'm not seeing it on my servers or a couple others... C:\Users\jluthman>dig MX staples.com +short 10 staples-com.mail.protection.outlook.com. C:\Users\jluthman>dig MX staples.com +short @4.2.2.2 10 staples-com.mail.protection.outlook.com. C:\Users\jluthman>dig MX staples.com +short @208.67.222.222 10 staples-com.mail.protection.outlook.com. C:\Users\jluthman>dig MX staples.com +short @pdns131.ultradns.com 10 staples-com.mail.protection.outlook.com. Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373 On Tue, Oct 21, 2014 at 6:01 PM, Frank Bulk via Outages <outages@outages.org
wrote:
Our last queue check reported DNS failures looking up <domain name>.mail.protection.outlook.com for 11 such domains.
Just picking one, staples.com:
root@nagios:/tmp# dig MX staples.com +short 10 staples-com.mail.protection.outlook.com. root@nagios:/tmp# dig staples-com.mail.protection.outlook.com
; <<>> DiG 9.7.3 <<>> staples-com.mail.protection.outlook.com ;; global options: +cmd ;; connection timed out; no servers could be reached root@nagios:/tmp# root@nagios:/tmp# host MX staples.com ;; connection timed out; no servers could be reached root@nagios:/tmp#
Anyone else seeing this? A colleague says that O365 is having issues and will be forwarding me some details.
Regards,
Frank Bulk
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
I received two confirmations off list of the issue – thanks. The one observed it from 4:27 pm and 5:11 pm, Central. Josh, you were checking just the MX records, but before any email server can actually connect to the email server it has to resolve staples-com.mail.protection.outlook.com, which was not working at the time of my original post. Frank From: Josh Luthman [mailto:josh@imaginenetworksllc.com] Sent: Tuesday, October 21, 2014 5:20 PM To: Frank Bulk Cc: outages@outages.org Subject: Re: [outages] Email to Microsoft Office 365 users I'm not seeing it on my servers or a couple others... C:\Users\jluthman>dig MX staples.com <http://staples.com> +short 10 staples-com.mail.protection.outlook.com <http://staples-com.mail.protection.outlook.com> . C:\Users\jluthman>dig MX staples.com <http://staples.com> +short @4.2.2.2 <http://4.2.2.2> 10 staples-com.mail.protection.outlook.com <http://staples-com.mail.protection.outlook.com> . C:\Users\jluthman>dig MX staples.com <http://staples.com> +short @208.67.222.222 <http://208.67.222.222> 10 staples-com.mail.protection.outlook.com <http://staples-com.mail.protection.outlook.com> . C:\Users\jluthman>dig MX staples.com <http://staples.com> +short @pdns131.ultradns.com <http://pdns131.ultradns.com> 10 staples-com.mail.protection.outlook.com <http://staples-com.mail.protection.outlook.com> . Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373 On Tue, Oct 21, 2014 at 6:01 PM, Frank Bulk via Outages <outages@outages.org <mailto:outages@outages.org> > wrote: Our last queue check reported DNS failures looking up <domain name>.mail.protection.outlook.com <http://mail.protection.outlook.com> for 11 such domains. Just picking one, staples.com <http://staples.com> : root@nagios:/tmp# dig MX staples.com <http://staples.com> +short 10 staples-com.mail.protection.outlook.com <http://staples-com.mail.protection.outlook.com> . root@nagios:/tmp# dig staples-com.mail.protection.outlook.com <http://staples-com.mail.protection.outlook.com> ; <<>> DiG 9.7.3 <<>> staples-com.mail.protection.outlook.com <http://staples-com.mail.protection.outlook.com> ;; global options: +cmd ;; connection timed out; no servers could be reached root@nagios:/tmp# root@nagios:/tmp# host MX staples.com <http://staples.com> ;; connection timed out; no servers could be reached root@nagios:/tmp# Anyone else seeing this? A colleague says that O365 is having issues and will be forwarding me some details. Regards, Frank Bulk _______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
I just pored through our own email logs – the issue first cropped up at 4:17 pm Central and the last failure I see is 5:01 pm, Central. But looking at delivery, the last delivery was at 4:22 pm and the first delivery was at 4:58 pm. So different domains may have started resolving at different times. Frank From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Frank Bulk via Outages Sent: Tuesday, October 21, 2014 6:26 PM To: 'Josh Luthman' Cc: outages@outages.org Subject: Re: [outages] Email to Microsoft Office 365 users I received two confirmations off list of the issue – thanks. The one observed it from 4:27 pm and 5:11 pm, Central. Josh, you were checking just the MX records, but before any email server can actually connect to the email server it has to resolve staples-com.mail.protection.outlook.com, which was not working at the time of my original post. Frank From: Josh Luthman [mailto:josh@imaginenetworksllc.com] Sent: Tuesday, October 21, 2014 5:20 PM To: Frank Bulk Cc: outages@outages.org <mailto:outages@outages.org> Subject: Re: [outages] Email to Microsoft Office 365 users I'm not seeing it on my servers or a couple others... C:\Users\jluthman>dig MX staples.com <http://staples.com> +short 10 staples-com.mail.protection.outlook.com <http://staples-com.mail.protection.outlook.com> . C:\Users\jluthman>dig MX staples.com <http://staples.com> +short @4.2.2.2 <http://4.2.2.2> 10 staples-com.mail.protection.outlook.com <http://staples-com.mail.protection.outlook.com> . C:\Users\jluthman>dig MX staples.com <http://staples.com> +short @208.67.222.222 <http://208.67.222.222> 10 staples-com.mail.protection.outlook.com <http://staples-com.mail.protection.outlook.com> . C:\Users\jluthman>dig MX staples.com <http://staples.com> +short @pdns131.ultradns.com <http://pdns131.ultradns.com> 10 staples-com.mail.protection.outlook.com <http://staples-com.mail.protection.outlook.com> . Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373 On Tue, Oct 21, 2014 at 6:01 PM, Frank Bulk via Outages <outages@outages.org <mailto:outages@outages.org> > wrote: Our last queue check reported DNS failures looking up <domain name>.mail.protection.outlook.com <http://mail.protection.outlook.com> for 11 such domains. Just picking one, staples.com <http://staples.com> : root@nagios:/tmp# dig MX staples.com <http://staples.com> +short 10 staples-com.mail.protection.outlook.com <http://staples-com.mail.protection.outlook.com> . root@nagios:/tmp# dig staples-com.mail.protection.outlook.com <http://staples-com.mail.protection.outlook.com> ; <<>> DiG 9.7.3 <<>> staples-com.mail.protection.outlook.com <http://staples-com.mail.protection.outlook.com> ;; global options: +cmd ;; connection timed out; no servers could be reached root@nagios:/tmp# root@nagios:/tmp# host MX staples.com <http://staples.com> ;; connection timed out; no servers could be reached root@nagios:/tmp# Anyone else seeing this? A colleague says that O365 is having issues and will be forwarding me some details. Regards, Frank Bulk _______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
participants (3)
-
Frank Bulk -
Jeremy Chadwick -
Josh Luthman