Hello Outages List Members, First and foremost, I apologize for my mis-use of the intention of this list. I realize this is not an outage from that perspective. I purposefully chose to post to the main list as to try to enhance my chances of my plea making it to someone who can attempt to act. I have received several private replies, which I appreciate very much, and that was really the intention of my post which I should have clearly stated up front. Because there is no known process that I can find for an AS that does not have a direct relationship with AT&T to reach out when there are problems like this is frustrating (we would really like to know if there is one). I hope you can understand, and forgive me, for my misuse of the intention of the list. No further replies from me will be sent to this thread. If replying, please do it privately moving forward. Thanks! ~ Anthony -- Anthony Hoppe Sr. Network Engineer ahoppe@backblaze.com Backblaze.com On Thu, Nov 13, 2025 at 12:20 AM Steve Haering <steve@thc-lab.net> wrote:

Evening, friends,

Please feel free to reach out if outages.org decides it is time to have a more robust application/stack, instead of a distribution list. I can make a quick, cheap, haphazard webapp that will facilitate handling things like this in...a different way where everyone on the distro list doesn't get the email chiding the initial user about their usage. I can't promise it will be a good webapp - but it might be better than effectively tagging @everyone for what amounts to a triage issue.

This is a cool list, appreciate all you do. Except for this particular email. 🙂

Much love, THC

________________________________ From: J. Hellenthal via Outages <outages@outages.org> Sent: Wednesday, November 12, 2025 7:16 PM To: Outages Reporting <outages@outages.org> Cc: Anthony Hoppe <ahoppe@backblaze.com>; Allan Paschall <APaschall@bankofsw.com>; noc <noc@nitelusa.com>; Outages Reporting <outages@outages.org>; J. Hellenthal <jhellenthal@dataix.net> Subject: [Outages] Re: Issue with AT&T customers reaching at least one of our IPs

This isn't an outage!

Take it to -discussion -- J. Hellenthal

The fact that there's a highway to Hell but only a stairway to Heaven says a lot about anticipated traffic volume.

...

On Nov 12, 2025, at 17:52, noc via Outages <outages@outages.org> wrote:

Hello,

Thank you for contacting the Nitel NOC. In order to properly open a case, we need to identify the affected service in our system. Please provide us with the following information to locate the service:

* NIT/HC Circuit ID: * Location Address:

Additionally, in order to start the troubleshooting process, please confirm the following information:

* Local Contact Name/Phone Number: * Site Access Hours:

Our team is committed to resolving your issue as efficiently as possible.

If you require immediate assistance, don't hesitate to contact us directly at (866) 892-0915.

Thank you,

Jafeth Quesada

NOC Analyst ll

866.892.0915

Comcast Business

Philadelphia, PA

[A blue text on a black background Description automatically generated]

Nitel is now part of Comcast Business.

________________________________ From: Allan Paschall via Outages <outages@outages.org> Sent: Wednesday, November 12, 2025 1:36 PM To: Outages (Planned & Unplanned) Reporting <outages@outages.org> Cc: Anthony Hoppe <ahoppe@backblaze.com>; Allan Paschall <APaschall@bankofsw.com> Subject: [Outages] Re: Issue with AT&T customers reaching at least one of our IPs

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.

Anthony,

I see that your IP address is on this blacklist: https://matrix.spfbl.net/

Below is the message reason given:

A domain is considered non-compliant when the WHOIS search result for that domain does not contain the email address of the domain owner. Update the registration data and remove privacy protection for this domain in WHOIS and wait one hour for the cached result of this WHOIS query to expire.

This IP was flagged due to misconfiguration of the e-mail service or the suspicion that there is no MTA at it.

The link below should take you to the message and the instructions to have the blacklist removed. I hope this helps!

https://matrix.spfbl.net/en/149.137.136.16

Allan Paschall

From: Anthony Hoppe via Outages <outages@outages.org> Date: Wednesday, November 12, 2025 at 12:21 PM To: outages@outages.org <outages@outages.org> Cc: Anthony Hoppe <ahoppe@backblaze.com> Subject: [Outages] Issue with AT&T customers reaching at least one of our IPs

[This message originated from outside Bank of the Southwest]

Hello List,

I need help getting in touch with someone helpful at AT&T (NetEng and/or trust & safety/abuse/compliance/security or preferably all the above). Starting on November 6th, customers behind AT&T have been unable to reach our IP address 149.137.136.16 which is the API VIP we use to serve downloads from Backblaze B2 buckets.

If you go one up (.17) or one down (.15) no issues.

I've tried calling the NOC contact per PeeringDB and they were ultimately unhelpful. Because we do not peer directly with AT&T they will not talk to us.

I can confirm the reachability issue via AT&T's own public route server (https://urldefense.proofpoint.com/v2/url?u=http-3A__route-2Dserver.ip.att.ne...) where I can ping 149.137.136.15 and .17 successfully, but cannot ping 149.137.136.16. MTRs from customers seem to show the path dies well before us. Here are a couple of examples that show 149.137.136.15 good and 149.137.136.16 bad:

This one is from Michigan:

$ mtr -w 149.137.136.15 Password: Start: 2025-11-12T13:39:10-0500 HOST: ip-192-168-0-231.ec2.internal Loss% Snt Last Avg Best Wrst StDev 1.|-- ip-192-168-0-1.ec2.internal 0.0% 10 4.0 2.9 2.5 4.0 0.4 2.|-- ip-192-168-1-254.ec2.internal 0.0% 10 3.3 3.3 2.6 3.5 0.2 3.|-- https://urldefense.proofpoint.com/v2/url?u=http-3A__108-2D202-2D236-2D1.ligh... 0.0% 10 5.4 5.0 4.2 6.5 0.8 4.|-- 71.151.199.86 0.0% 10 6.5 5.8 4.8 6.9 0.6 5.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 6.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 7.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 8.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 9.|-- https://urldefense.proofpoint.com/v2/url?u=http-3A__ae3.cr12-2Dchi1.ip4.gtt.... 0.0% 10 14.7 14.8 13.7 15.9 0.6 10.|-- https://urldefense.proofpoint.com/v2/url?u=http-3A__ae4.cr3-2Dwas1.ip4.gtt.n... 0.0% 10 34.7 34.9 34.0 35.4 0.4 11.|-- https://urldefense.proofpoint.com/v2/url?u=http-3A__ip4.gtt.net&d=DwICAg&c=e... 0.0% 10 34.7 34.6 33.7 35.7 0.5 12.|-- https://urldefense.proofpoint.com/v2/url?u=http-3A__api005.backblazeb2.com&d... 0.0% 10 34.8 35.1 34.1 37.5 0.9

$ mtr -w 149.137.136.16 Start: 2025-11-12T13:39:54-0500 HOST: ip-192-168-0-231.ec2.internal Loss% Snt Last Avg Best Wrst StDev 1.|-- ip-192-168-0-1.ec2.internal 0.0% 10 2.5 2.8 2.5 3.0 0.1 2.|-- ip-192-168-1-254.ec2.internal 0.0% 10 3.4 3.5 3.0 3.8 0.2 3.|-- https://urldefense.proofpoint.com/v2/url?u=http-3A__108-2D202-2D236-2D1.ligh... 0.0% 10 4.0 5.3 3.9 8.9 1.6 4.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0

This one is from East Bay, California:

$ mtr -w 149.137.136.15 Start: 2025-11-11T14:54:50-0800 HOST: maureen Loss% Snt Last Avg Best Wrst StDev 1.|-- router.lan 0.0% 10 0.6 0.6 0.4 0.8 0.1 2.|-- 192.168.1.254 0.0% 10 1.6 1.5 1.4 1.7 0.1 3.|-- https://urldefense.proofpoint.com/v2/url?u=http-3A__99-2D189-2D116-2D1.light... 0.0% 10 19.7 19.9 19.6 20.6 0.3 4.|-- 71.148.180.125 0.0% 10 20.7 20.2 20.0 20.7 0.3 5.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 6.|-- 32.130.91.83 40.0% 10 23.1 34.2 22.3 91.9 28.3 7.|-- 192.205.32.98 0.0% 10 29.4 24.9 23.1 30.2 2.6 8.|-- https://urldefense.proofpoint.com/v2/url?u=http-3A__ae4.cr3-2Dwas1.ip4.gtt.n... 0.0% 10 81.9 81.9 81.6 82.5 0.3 9.|-- https://urldefense.proofpoint.com/v2/url?u=http-3A__ip4.gtt.net&d=DwICAg&c=e... 0.0% 10 82.0 82.0 81.7 82.5 0.2 10.|-- https://urldefense.proofpoint.com/v2/url?u=http-3A__api005.backblazeb2.com&d... 0.0% 10 82.4 82.1 81.3 82.9 0.4

$ mtr -w 149.137.136.16 Start: 2025-11-11T13:49:14-0800 HOST: maureen Loss% Snt Last Avg Best Wrst StDev 1.|-- router.lan 0.0% 10 0.7 0.7 0.5 0.7 0.1 2.|-- 192.168.1.254 0.0% 10 1.5 1.5 1.4 1.5 0.1 3.|-- https://urldefense.proofpoint.com/v2/url?u=http-3A__99-2D189-2D116-2D1.light... 0.0% 10 20.3 20.1 19.6 20.8 0.3 4.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0

If there is anybody on this list that can throw me a bone, it would be very much appreciated.

--

Anthony Hoppe Sr. Network Engineer ahoppe@backblaze.com https://urldefense.proofpoint.com/v2/url?u=http-3A__Backblaze.com&d=DwICAg&c...

-- This email, including its contents and any attachment(s), may contain confidential and/or proprietary information and is solely for the review and use of the intended recipient(s). If you have received this email in error, please notify the sender and permanently delete this email, its content, and any attachment(s). Any disclosure, copying, or taking of any action in reliance on an email received in error is strictly prohibited. ______________________________________________ Outages mailing list outages@outages.org Sign up for an account https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.outages.org_accou... To subscribe send an email to outages-join@outages.org To unsubscribe send an email to outages-leave@outages.org To contact the list owners outages-owner@outages.org Archives https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.outages.org_archi...

Thank you for using Outages Lists! For post outage discussion, please redirect to outages-discussion@outages.org NOTICE: The information contained in this message is confidential and for the sole use of the intended recipient/s. If you are not the intended recipient, please (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. ______________________________________________ Outages mailing list outages@outages.org Sign up for an account https://lists.outages.org/accounts/signup/ To subscribe send an email to outages-join@outages.org To unsubscribe send an email to outages-leave@outages.org To contact the list owners outages-owner@outages.org Archives https://lists.outages.org/archives/list/outages@outages.org/

Thank you for using Outages Lists! For post outage discussion, please redirect to outages-discussion@outages.org

________________________________

CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you. ______________________________________________ Outages mailing list outages@outages.org Sign up for an account https://lists.outages.org/accounts/signup/ To subscribe send an email to outages-join@outages.org To unsubscribe send an email to outages-leave@outages.org To contact the list owners outages-owner@outages.org Archives https://lists.outages.org/archives/list/outages@outages.org/

Thank you for using Outages Lists! For post outage discussion, please redirect to outages-discussion@outages.org

---

Outages mailing list outages@outages.org Sign up for an account https://lists.outages.org/accounts/signup/ To subscribe send an email to outages-join@outages.org To unsubscribe send an email to outages-leave@outages.org To contact the list owners outages-owner@outages.org Archives https://lists.outages.org/archives/list/outages@outages.org/

Thank you for using Outages Lists! For post outage discussion, please redirect to outages-discussion@outages.org

-- This email, including its contents and any attachment(s), may contain confidential and/or proprietary information and is solely for the review and use of the intended recipient(s). If you have received this email in error, please notify the sender and permanently delete this email, its content, and any attachment(s). Any disclosure, copying, or taking of any action in reliance on an email received in error is strictly prohibited.