Problem with credit card machine processing? "Datawire"
Our folks have spent the better part of a day chasing an issue with a customer that had issues processing cards from their physical in-store terminal. That turned into 2, 3, and a handful more. We finally got info that all of these impacted terminals connect to a company "Datawire" who went down last night at 1800 and came back up at 0800 this morning (unknown TZ). They continued to point to us as the issue until just a short while ago when some person at this Datawire admitted a large portion of the country may still be down. Anyone else seeing anything like this? Perhaps it could save you chasing your tail as well. Perhaps better for a discussions-list conversation, but... seriously... a credit card processing firm that has an outage like this? Hmm... Jayson Peak Internet
We had a customer call saying they needed is to clear dns cache because they couldn’t process CCs. One of my guys read about the large outage so when it came in we knew it wasn’t anything to do with us. Sent from my iPhone On Jul 13, 2018, at 9:04 PM, Jayson Baker via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: Our folks have spent the better part of a day chasing an issue with a customer that had issues processing cards from their physical in-store terminal. That turned into 2, 3, and a handful more. We finally got info that all of these impacted terminals connect to a company "Datawire" who went down last night at 1800 and came back up at 0800 this morning (unknown TZ). They continued to point to us as the issue until just a short while ago when some person at this Datawire admitted a large portion of the country may still be down. Anyone else seeing anything like this? Perhaps it could save you chasing your tail as well. Perhaps better for a discussions-list conversation, but... seriously... a credit card processing firm that has an outage like this? Hmm... Jayson Peak Internet Luke Guillory Vice President – Technology and Innovation [cid:image92a120.JPG@d3b29774.43970625] <http://www.rtconline.com> Tel: 985.536.1212 Fax: 985.536.0300 Email: lguillory@reservetele.com Web: www.rtconline.com Reserve Telecommunications 100 RTC Dr Reserve, LA 70084 Disclaimer: The information transmitted, including attachments, is intended only for the person(s) or entity to which it is addressed and may contain confidential and/or privileged material which should not disseminate, distribute or be copied. Please notify Luke Guillory immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. E-mail transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. Luke Guillory therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
Yes, we learned of issues late Wednesday morning after receiving reports from two and then three business customers. Indications suggest the issue started Tuesday evening. One local Dairy Queen and another 20 minutes away couldn’t accept credit cards on Wednesday. The request to preform DNS flushes of vxn.datawire.net came to us Thursday afternoon from two of three customers who (eventually) called their credit card partners/processors. So we flushed our (ISP) caches and then encouraged those customers to power cycle their router and then their credit card machines, but that wasn’t 100% successful for them, either. At that point we directed them back to their credit card partners/processors. It was interesting to see DNS resolution for vxn.datawire.net pointing to a mixture of 216.220.36.75 (vxn.datawire.net) and 45.227.252.17 (hosting-by.net4web.org). Maybe it's normal that they have multiple, but on Wednesday it was just 216.220.36.75. The TTL for 45.227.252.17 was much longer (over 430,000) than 216.220.36.75 (about 300 seconds) and had a bad SSL certificate for https://vxn.datawire.net. I suspect they moved some operations to another data center, but made a mistake with TTL. All told we probably heard from six or seven different businesses. More here: https://twitter.com/ExecPro/status/1016860164983611392 https://status.cayan.com/issues/5b45477e8dc35afae9000fe6 https://status.cayan.com/issues/5b4546508dc35a5975000fdc https://status.cayan.com/issues/5b479ad48dc35ad03a0030e7 https://status.cayan.com/issues/5b478b918dc35aff310030c9 https://twitter.com/TriphenTech/status/1016852856408690693 https://twitter.com/C_Forrest/status/1017819893704593410 https://twitter.com/Vicinity_7/status/1017800989347401728 https://twitter.com/pokehbar/status/1017796090052128769 https://twitter.com/glyngh/status/1017790958610493440 https://twitter.com/tallbaby21/status/1017121159526133760 https://twitter.com/devin_ledude/status/1017451556000522241 https://status.cayan.com/issues/5b478ba38dc35a3da80030d9 Frank # whob 216.220.36.75 IP: 216.220.36.75 Origin-AS: 12188 Prefix: 216.220.32.0/20 AS-Path: 18106 6939 12188 AS-Org-Name: Q9 Networks Inc. Org-Name: Q9 Networks Inc. Net-Name: Q9-NET1 Cache-Date: 1531374425 Latitude: 43.508330 Longitude: -79.883333 City: Milton Region: Ontario Country: Canada Country-Code: CA # whob 45.227.252.17 IP: 45.227.252.17 Origin-AS: 58271 Prefix: 45.227.252.0/24 AS-Path: 34224 12389 44125 201765 48882 58271 AS-Org-Name: VSERVER-AS Org-Name: This network range is not fully allocated to APNIC. Net-Name: IANA-NETBLOCK-45 Cache-Date: 1531374425 Latitude: 0.000000 Longitude: 0.000000 City: NULL Region: NULL Country: NULL Country-Code: NULL From: Outages <outages-bounces@outages.org> On Behalf Of Luke Guillory via Outages Sent: Friday, July 13, 2018 9:18 PM To: jayson@peakinter.net; outages@outages.org Subject: Re: [outages] Problem with credit card machine processing? "Datawire" We had a customer call saying they needed is to clear dns cache because they couldn’t process CCs. One of my guys read about the large outage so when it came in we knew it wasn’t anything to do with us. Sent from my iPhone On Jul 13, 2018, at 9:04 PM, Jayson Baker via Outages <outages@outages.org <mailto:outages@outages.org> > wrote: Our folks have spent the better part of a day chasing an issue with a customer that had issues processing cards from their physical in-store terminal. That turned into 2, 3, and a handful more. We finally got info that all of these impacted terminals connect to a company "Datawire" who went down last night at 1800 and came back up at 0800 this morning (unknown TZ). They continued to point to us as the issue until just a short while ago when some person at this Datawire admitted a large portion of the country may still be down. Anyone else seeing anything like this? Perhaps it could save you chasing your tail as well. Perhaps better for a discussions-list conversation, but... seriously... a credit card processing firm that has an outage like this? Hmm... Jayson Peak Internet Luke Guillory Vice President – Technology and Innovation <http://www.rtconline.com> Tel: 985.536.1212 Fax: 985.536.0300 Email: lguillory@reservetele.com <mailto:lguillory@reservetele.com> Web: www.rtconline.com <http://www.rtconline.com> Reserve Telecommunications 100 RTC Dr Reserve, LA 70084 Disclaimer: The information transmitted, including attachments, is intended only for the person(s) or entity to which it is addressed and may contain confidential and/or privileged material which should not disseminate, distribute or be copied. Please notify Luke Guillory immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. E-mail transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. Luke Guillory therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. _______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
Just received this afternoon: ================== Support Team, You have several business customers being affected by an ongoing issue. In order to resolve this, First Data is requesting that you clear the cache on all DNS servers being used to support them. We propagated a correction over 16 hours ago and know that Google DNS and others are translating correctly. Would you please help us assist your customers? The correct resolutions are: vxn.datawire.net <http://vxn.datawire.net/> 216.220.36.75 vxn1.datawire.net <http://vxn1.datawire.net/> 205.167.140.10 vxn2.datawire.net <http://vxn2.datawire.net/> 64.243.142.36 vxn3.datawire.net <http://vxn3.datawire.net/> 206.112.91.167 vxn4.datawire.net <http://vxn4.datawire.net/> 63.240.199.76 If you are resolving it as anything starting with 45.x.x.x, it is incorrect. Please feel free to compare to the Google DNS resolution for confirmation. Please either reply all or call First Data's Network Operations at 888-377-8726 Option 3. <snip> First Data, 240 North Roosevelt Av Chandler, Arizona 85226 ================== That kind of confirms that the TTL for the 45.x.x.x record(s) were a bit too long – if they had been short, like they are now at 300 seconds, the issue would mostly have cleared up. From: Outages <outages-bounces@outages.org> On Behalf Of frnkblk--- via Outages Sent: Friday, July 13, 2018 9:56 PM To: 'Luke Guillory' <lguillory@reservetele.com>; jayson@peakinter.net; outages@outages.org Subject: Re: [outages] Problem with credit card machine processing? "Datawire" Yes, we learned of issues late Wednesday morning after receiving reports from two and then three business customers. Indications suggest the issue started Tuesday evening. One local Dairy Queen and another 20 minutes away couldn’t accept credit cards on Wednesday. The request to preform DNS flushes of vxn.datawire.net came to us Thursday afternoon from two of three customers who (eventually) called their credit card partners/processors. So we flushed our (ISP) caches and then encouraged those customers to power cycle their router and then their credit card machines, but that wasn’t 100% successful for them, either. At that point we directed them back to their credit card partners/processors. It was interesting to see DNS resolution for vxn.datawire.net pointing to a mixture of 216.220.36.75 (vxn.datawire.net) and 45.227.252.17 (hosting-by.net4web.org). Maybe it's normal that they have multiple, but on Wednesday it was just 216.220.36.75. The TTL for 45.227.252.17 was much longer (over 430,000) than 216.220.36.75 (about 300 seconds) and had a bad SSL certificate for https://vxn.datawire.net. I suspect they moved some operations to another data center, but made a mistake with TTL. All told we probably heard from six or seven different businesses. More here: https://twitter.com/ExecPro/status/1016860164983611392 https://status.cayan.com/issues/5b45477e8dc35afae9000fe6 https://status.cayan.com/issues/5b4546508dc35a5975000fdc https://status.cayan.com/issues/5b479ad48dc35ad03a0030e7 https://status.cayan.com/issues/5b478b918dc35aff310030c9 https://twitter.com/TriphenTech/status/1016852856408690693 https://twitter.com/C_Forrest/status/1017819893704593410 https://twitter.com/Vicinity_7/status/1017800989347401728 https://twitter.com/pokehbar/status/1017796090052128769 https://twitter.com/glyngh/status/1017790958610493440 https://twitter.com/tallbaby21/status/1017121159526133760 https://twitter.com/devin_ledude/status/1017451556000522241 https://status.cayan.com/issues/5b478ba38dc35a3da80030d9 Frank # whob 216.220.36.75 IP: 216.220.36.75 Origin-AS: 12188 Prefix: 216.220.32.0/20 AS-Path: 18106 6939 12188 AS-Org-Name: Q9 Networks Inc. Org-Name: Q9 Networks Inc. Net-Name: Q9-NET1 Cache-Date: 1531374425 Latitude: 43.508330 Longitude: -79.883333 City: Milton Region: Ontario Country: Canada Country-Code: CA # whob 45.227.252.17 IP: 45.227.252.17 Origin-AS: 58271 Prefix: 45.227.252.0/24 AS-Path: 34224 12389 44125 201765 48882 58271 AS-Org-Name: VSERVER-AS Org-Name: This network range is not fully allocated to APNIC. Net-Name: IANA-NETBLOCK-45 Cache-Date: 1531374425 Latitude: 0.000000 Longitude: 0.000000 City: NULL Region: NULL Country: NULL Country-Code: NULL From: Outages <outages-bounces@outages.org <mailto:outages-bounces@outages.org> > On Behalf Of Luke Guillory via Outages Sent: Friday, July 13, 2018 9:18 PM To: jayson@peakinter.net <mailto:jayson@peakinter.net> ; outages@outages.org <mailto:outages@outages.org> Subject: Re: [outages] Problem with credit card machine processing? "Datawire" We had a customer call saying they needed is to clear dns cache because they couldn’t process CCs. One of my guys read about the large outage so when it came in we knew it wasn’t anything to do with us. Sent from my iPhone On Jul 13, 2018, at 9:04 PM, Jayson Baker via Outages <outages@outages.org <mailto:outages@outages.org> > wrote: Our folks have spent the better part of a day chasing an issue with a customer that had issues processing cards from their physical in-store terminal. That turned into 2, 3, and a handful more. We finally got info that all of these impacted terminals connect to a company "Datawire" who went down last night at 1800 and came back up at 0800 this morning (unknown TZ). They continued to point to us as the issue until just a short while ago when some person at this Datawire admitted a large portion of the country may still be down. Anyone else seeing anything like this? Perhaps it could save you chasing your tail as well. Perhaps better for a discussions-list conversation, but... seriously... a credit card processing firm that has an outage like this? Hmm... Jayson Peak Internet Luke Guillory Vice President – Technology and Innovation <http://www.rtconline.com> Tel: 985.536.1212 Fax: 985.536.0300 Email: lguillory@reservetele.com <mailto:lguillory@reservetele.com> Web: www.rtconline.com <http://www.rtconline.com> Reserve Telecommunications 100 RTC Dr Reserve, LA 70084 Disclaimer: The information transmitted, including attachments, is intended only for the person(s) or entity to which it is addressed and may contain confidential and/or privileged material which should not disseminate, distribute or be copied. Please notify Luke Guillory immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. E-mail transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. Luke Guillory therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. _______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
same here. several reports. they were told to inform their service provider (us) to clear dns caches. -nick — Nick Peelman Network Engineer | Enhanced Telecommunications Corp. 812-222-0169<tel:812-222-0169> | npeelman@etc1.net<mailto:npeelman@etc1.net> | www.etczone.com<http://www.etczone.com/> Sent from my iPhone On Jul 13, 2018, at 22:19, Luke Guillory via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: We had a customer call saying they needed is to clear dns cache because they couldn’t process CCs. One of my guys read about the large outage so when it came in we knew it wasn’t anything to do with us. Sent from my iPhone On Jul 13, 2018, at 9:04 PM, Jayson Baker via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: Our folks have spent the better part of a day chasing an issue with a customer that had issues processing cards from their physical in-store terminal. That turned into 2, 3, and a handful more. We finally got info that all of these impacted terminals connect to a company "Datawire" who went down last night at 1800 and came back up at 0800 this morning (unknown TZ). They continued to point to us as the issue until just a short while ago when some person at this Datawire admitted a large portion of the country may still be down. Anyone else seeing anything like this? Perhaps it could save you chasing your tail as well. Perhaps better for a discussions-list conversation, but... seriously... a credit card processing firm that has an outage like this? Hmm... Jayson Peak Internet Luke Guillory Vice President – Technology and Innovation <image92a120.JPG><http://www.rtconline.com> Tel: 985.536.1212 Fax: 985.536.0300 Email: lguillory@reservetele.com<mailto:lguillory@reservetele.com> Web: www.rtconline.com<http://www.rtconline.com> Reserve Telecommunications 100 RTC Dr Reserve, LA 70084 Disclaimer: The information transmitted, including attachments, is intended only for the person(s) or entity to which it is addressed and may contain confidential and/or privileged material which should not disseminate, distribute or be copied. Please notify Luke Guillory immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. E-mail transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. Luke Guillory therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
participants (5)
-
Frank Bulk -
frnkblk＠iname.com
-
Jayson Baker -
Luke Guillory -
Nick Peelman