We have to query and compare against NIST time servers for FINRA compliance. This morning I noticed our systems are unable to DNS query the NIST time servers. Neither our local resolvers or google (8.8.8.8) work. [root@bacall log]# dig @8.8.8.8 time-a-g.nist.gov ; <<>> DiG 9.11.4-P2-RedHat-9.11.4-26.P2.el7_9.5 <<>> @8.8.8.8 time-a-g.nist.gov ; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 36018 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 512 ;; QUESTION SECTION: ;time-a-g.nist.gov. IN A ;; Query time: 6 msec ;; SERVER: 8.8.8.8#53(8.8.8.8) ;; WHEN: Mon Jun 14 06:27:45 EDT 2021 ;; MSG SIZE rcvd: 46 [root@bacall log]# dig @8.8.8.8 nist.gov in soa ; <<>> DiG 9.11.4-P2-RedHat-9.11.4-26.P2.el7_9.5 <<>> @8.8.8.8 nist.gov in soa ; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 17779 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 512 ;; QUESTION SECTION: ;nist.gov. IN SOA ;; Query time: 5 msec ;; SERVER: 8.8.8.8#53(8.8.8.8) ;; WHEN: Mon Jun 14 06:31:59 EDT 2021 ;; MSG SIZE rcvd: 37 The time servers are documented here: https://tf.nist.gov/tf-cgi/servers.cgi Using the IP addresses work, it look like the nist.gov domain is offline. Matthew Huff | Director of Technical Operations | OTA Management LLC Office: 914-460-4039 mhuff@ox.com | www.ox.com ...........................................................................................................................................
are your local resolvers forwarding to 8.8.8.8? I tried a small sample of public resolvers and only the google once failed. Maybe DNSSEC? (looks like the NIST signature rotated yesterday) 1.1.1.1 gm.nist.gov. netops.nist.gov. 2889174 10800 1080 2419200 300 8.8.8.8 failed 8.8.4.4 failed 75.75.75.75 gm.nist.gov. netops.nist.gov. 2889174 10800 1080 2419200 300 9.9.9.9 gm.nist.gov. netops.nist.gov. 2889174 10800 1080 2419200 300 On 6/14/21 6:35 AM, Matthew Huff via Outages wrote:
External email warning - This email originated outside the company. Please do not click links or open attachments unless you were expecting this communication. - SANS Security Team -
We have to query and compare against NIST time servers for FINRA compliance. This morning I noticed our systems are unable to DNS query the NIST time servers. Neither our local resolvers or google (8.8.8.8) work.
[root@bacall log]# dig @8.8.8.8 time-a-g.nist.gov
; <<>> DiG 9.11.4-P2-RedHat-9.11.4-26.P2.el7_9.5 <<>> @8.8.8.8 time-a-g.nist.gov ; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 36018 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 512 ;; QUESTION SECTION: ;time-a-g.nist.gov. IN A
;; Query time: 6 msec ;; SERVER: 8.8.8.8#53(8.8.8.8) ;; WHEN: Mon Jun 14 06:27:45 EDT 2021 ;; MSG SIZE rcvd: 46
[root@bacall log]# dig @8.8.8.8 nist.gov in soa
; <<>> DiG 9.11.4-P2-RedHat-9.11.4-26.P2.el7_9.5 <<>> @8.8.8.8 nist.gov in soa ; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 17779 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 512 ;; QUESTION SECTION: ;nist.gov. IN SOA
;; Query time: 5 msec ;; SERVER: 8.8.8.8#53(8.8.8.8) ;; WHEN: Mon Jun 14 06:31:59 EDT 2021 ;; MSG SIZE rcvd: 37
The time servers are documented here: https://urldefense.com/v3/__https://tf.nist.gov/tf-cgi/servers.cgi__;!!MlQdS...
Using the IP addresses work, it look like the nist.gov domain is offline.
Matthew Huff | Director of Technical Operations | OTA Management LLC
Office: 914-460-4039 mhuff@ox.com | https://urldefense.com/v3/__http://www.ox.com__;!!MlQdS1fu!DZRm9lRTouO4RyYps... ...........................................................................................................................................
_______________________________________________ Outages mailing list Outages@outages.org https://urldefense.com/v3/__https://puck.nether.net/mailman/listinfo/outages...
There's been some chatter on NANOG - they're under DDOS attack apparently. ________________________________ From: Outages <outages-bounces@outages.org> on behalf of Matthew Huff via Outages <outages@outages.org> Sent: 14 June 2021 11:35 To: outages@outages.org <outages@outages.org> Subject: [outages] DNS SERVFAIL for nist.gov We have to query and compare against NIST time servers for FINRA compliance. This morning I noticed our systems are unable to DNS query the NIST time servers. Neither our local resolvers or google (8.8.8.8) work. [root@bacall log]# dig @8.8.8.8 time-a-g.nist.gov ; <<>> DiG 9.11.4-P2-RedHat-9.11.4-26.P2.el7_9.5 <<>> @8.8.8.8 time-a-g.nist.gov ; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 36018 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 512 ;; QUESTION SECTION: ;time-a-g.nist.gov. IN A ;; Query time: 6 msec ;; SERVER: 8.8.8.8#53(8.8.8.8) ;; WHEN: Mon Jun 14 06:27:45 EDT 2021 ;; MSG SIZE rcvd: 46 [root@bacall log]# dig @8.8.8.8 nist.gov in soa ; <<>> DiG 9.11.4-P2-RedHat-9.11.4-26.P2.el7_9.5 <<>> @8.8.8.8 nist.gov in soa ; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 17779 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 512 ;; QUESTION SECTION: ;nist.gov. IN SOA ;; Query time: 5 msec ;; SERVER: 8.8.8.8#53(8.8.8.8) ;; WHEN: Mon Jun 14 06:31:59 EDT 2021 ;; MSG SIZE rcvd: 37 The time servers are documented here: https://tf.nist.gov/tf-cgi/servers.cgi Using the IP addresses work, it look like the nist.gov domain is offline. Matthew Huff | Director of Technical Operations | OTA Management LLC Office: 914-460-4039 mhuff@ox.com | www.ox.com ........................................................................................................................................... _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
On Mon, Jun 14, 2021 at 10:52:43AM +0000, Geoff Snowdon via Outages <outages@outages.org> wrote a message of 200 lines which said:
There's been some chatter on NANOG - they're under DDOS attack apparently.
Indeed, most of the authoritative name servers timeout. % check-soa -i nist.gov bea.nist.gov. 132.163.4.10: ERROR: read udp 193.70.85.11:60102->132.163.4.10:53: i/o timeout 2610:20:6b01:4::10: ERROR: read udp [2001:41d0:302:2200::180]:35149->[2610:20:6b01:4::10]:53: i/o timeout bea2.nist.gov. 132.163.4.11: ERROR: read udp 193.70.85.11:34369->132.163.4.11:53: i/o timeout 2610:20:6b01:4::11: ERROR: read udp [2001:41d0:302:2200::180]:47271->[2610:20:6b01:4::11]:53: i/o timeout gea.nist.gov. 129.6.13.3: ERROR: read udp 193.70.85.11:36517->129.6.13.3:53: i/o timeout 2610:20:6005:13::3: ERROR: read udp [2001:41d0:302:2200::180]:60519->[2610:20:6005:13::3]:53: i/o timeout gea2.nist.gov. 2610:20:6005:13::4: ERROR: read udp [2001:41d0:302:2200::180]:57731->[2610:20:6005:13::4]:53: i/o timeout 129.6.13.4: ERROR: read udp 193.70.85.11:38669->129.6.13.4:53: i/o timeout gea3.nist.gov. 129.6.14.193: OK: 2889174 (84 ms) 2610:20:6005:2192::193: ERROR: read udp [2001:41d0:302:2200::180]:39170->[2610:20:6005:2192::193]:53: i/o timeout The RIPE Atlas probes show some SERVFAILs: % blaeu-resolve -r 100 -c US --type A nist.gov [129.6.13.49] : 62 occurrences [ERROR: SERVFAIL] : 15 occurrences [198.105.244.131 198.105.254.131] : 1 occurrences Test #30851192 done at 2021-06-14T11:22:04Z
You should consider eliminating dependence on Internet-delivered NIST time and switch to GPS-based time servers. The GPS network has its own airborne atomic clocks that use a well-disciplined protocol to synchronize to NIST reference atomic time without transiting the Internet. According to NIST’s documentation: “Currently, the GPS system provides time to the general public with uncertainties measured in nanoseconds. With a well-designed receiver system the user can obtain the time to better than 100 ns in a few minutes, and to about +/- 10 ns with a 24 hour average (and a good local clock).” All sources of error in GPS time propagation total less than one millisecond, well within your 50ms tolerance. https://www.nist.gov/pml/time-and-frequency-division/time-services/one-way-g... NIST maintains publicly-accessible logs of al clock differences to provide documented compliance under FINRA clock synchronization rules. [cid:54276CD7-0F1E-4855-BA0B-BFE406A63AB5-L0-001] The log has a one-hour resolution, satisfying the FINRA requirement to verify synchronization “throughout the day”. IP-based GPS clocks are widely available with low-drift oven-controlled crystal oscillators (OXCO), or even internal cesium-based atomic clocks, for as little as a few thousand dollars. This lets you ride out time signal outages of days or even weeks. The US DHS recommends discontinuation of unauthenticated Internet-based reference clocks, owing to their vulnerability to IP address spoofing: https://www.dhs.gov/sites/default/files/publications/GPS-PNT-Best-Practices-... -mel via cell On Jun 14, 2021, at 3:51 AM, Matthew Huff via Outages <outages@outages.org> wrote: We have to query and compare against NIST time servers for FINRA compliance This morning I noticed our systems are unable to DNS query the NIST time servers. Neither our local resolvers or google (8.8.8.8) work. [root@bacall log]# dig @8.8.8.8 time-a-g.nist.gov ; <<>> DiG 9.11.4-P2-RedHat-9.11.4-26.P2.el7_9.5 <<>> @8.8.8.8 time-a-g.nist.gov ; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 36018 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 512 ;; QUESTION SECTION: ;time-a-g.nist.gov. IN A ;; Query time: 6 msec ;; SERVER: 8.8.8.8#53(8.8.8.8) ;; WHEN: Mon Jun 14 06:27:45 EDT 2021 ;; MSG SIZE rcvd: 46 [root@bacall log]# dig @8.8.8.8 nist.gov in soa ; <<>> DiG 9.11.4-P2-RedHat-9.11.4-26.P2.el7_9.5 <<>> @8.8.8.8 nist.gov in soa ; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 17779 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 512 ;; QUESTION SECTION: ;nist.gov. IN SOA ;; Query time: 5 msec ;; SERVER: 8.8.8.8#53(8.8.8.8) ;; WHEN: Mon Jun 14 06:31:59 EDT 2021 ;; MSG SIZE rcvd: 37 The time servers are documented here: https://tf.nist.gov/tf-cgi/servers.cgi Using the IP addresses work, it look like the nist.gov domain is offline. Matthew Huff | Director of Technical Operations | OTA Management LLC Office: 914-460-4039 mhuff@ox.com | www.ox.com ......................................................................................................................................... _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Of course. Like I stated in the original email, we don’t use NIST for time sync. We actually have a GPS and a PTP feed. WE MUST, HOWEVER, VALIDATE OUR SYSTEM TIMES VERSUS THE INTERNET NIST SERVERS VIA FINRA REGULATIONS Yes, it is stupid Yes, it isn’t a good idea But if we don’t want to get fined and prevented from trading, we have to follow FINRA regulations. Matthew Huff | Director of Technical Operations | OTA Management LLC Office: 914-460-4039 mhuff@ox.com<mailto:mhuff@ox.com> | www.ox.com<http://www.ox.com> ........................................................................................................................................... From: Mel Beckman <mel@beckman.org> Sent: Monday, June 14, 2021 9:40 AM To: Matthew Huff <mhuff@ox.com> Cc: outages@outages.org Subject: Re: [outages] DNS SERVFAIL for nist.gov You should consider eliminating dependence on Internet-delivered NIST time and switch to GPS-based time servers. The GPS network has its own airborne atomic clocks that use a well-disciplined protocol to synchronize to NIST reference atomic time without transiting the Internet. According to NIST’s documentation: “Currently, the GPS system provides time to the general public with uncertainties measured in nanoseconds. With a well-designed receiver system the user can obtain the time to better than 100 ns in a few minutes, and to about +/- 10 ns with a 24 hour average (and a good local clock).” All sources of error in GPS time propagation total less than one millisecond, well within your 50ms tolerance. https://www.nist.gov/pml/time-and-frequency-division/time-services/one-way-g... NIST maintains publicly-accessible logs of al clock differences to provide documented compliance under FINRA clock synchronization rules. [cid:image001.jpg@01D76104.DD1B3430] The log has a one-hour resolution, satisfying the FINRA requirement to verify synchronization “throughout the day”. IP-based GPS clocks are widely available with low-drift oven-controlled crystal oscillators (OXCO), or even internal cesium-based atomic clocks, for as little as a few thousand dollars. This lets you ride out time signal outages of days or even weeks. The US DHS recommends discontinuation of unauthenticated Internet-based reference clocks, owing to their vulnerability to IP address spoofing: https://www.dhs.gov/sites/default/files/publications/GPS-PNT-Best-Practices-... -mel via cell On Jun 14, 2021, at 3:51 AM, Matthew Huff via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: We have to query and compare against NIST time servers for FINRA compliance This morning I noticed our systems are unable to DNS query the NIST time servers. Neither our local resolvers or google (8.8.8.8) work. [root@bacall log]# dig @8.8.8.8 time-a-g.nist.gov ; <<>> DiG 9.11.4-P2-RedHat-9.11.4-26.P2.el7_9.5 <<>> @8.8.8.8 time-a-g.nist.gov ; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 36018 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 512 ;; QUESTION SECTION: ;time-a-g.nist.gov. IN A ;; Query time: 6 msec ;; SERVER: 8.8.8.8#53(8.8.8.8) ;; WHEN: Mon Jun 14 06:27:45 EDT 2021 ;; MSG SIZE rcvd: 46 [root@bacall log]# dig @8.8.8.8 nist.gov in soa ; <<>> DiG 9.11.4-P2-RedHat-9.11.4-26.P2.el7_9.5 <<>> @8.8.8.8 nist.gov in soa ; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 17779 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 512 ;; QUESTION SECTION: ;nist.gov. IN SOA ;; Query time: 5 msec ;; SERVER: 8.8.8.8#53(8.8.8.8) ;; WHEN: Mon Jun 14 06:31:59 EDT 2021 ;; MSG SIZE rcvd: 37 The time servers are documented here: https://tf.nist.gov/tf-cgi/servers.cgi Using the IP addresses work, it look like the nist.gov domain is offline. Matthew Huff | Director of Technical Operations | OTA Management LLC Office: 914-460-4039 mhuff@ox.com<mailto:mhuff@ox.com> | www.ox.com<http://www.ox.com> ......................................................................................................................................... _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
Matt, I support many financial networks, and comply with the same FINRA rule 6820 you do. This rule doesn’t state that the time must be synchronized over the Internet using public IP addresses, but only “to within a fifty (50) millisecond tolerance of the time maintained by the atomic clock” at NIST. Because GPS clock times are legally traceable to NIST, and deviation logs are available in real-time, there is no reason to depend on IP-based NTP over the Internet, and good reason not to as today’s event demonstrates. The 1ms accuracy is well within the 50ms limit. -mel On Jun 14, 2021, at 7:06 AM, Matthew Huff <mhuff@ox.com> wrote: Of course. Like I stated in the original email, we don’t use NIST for time sync. We actually have a GPS and a PTP feed. WE MUST, HOWEVER, VALIDATE OUR SYSTEM TIMES VERSUS THE INTERNET NIST SERVERS VIA FINRA REGULATIONS Yes, it is stupid Yes, it isn’t a good idea But if we don’t want to get fined and prevented from trading, we have to follow FINRA regulations. Matthew Huff | Director of Technical Operations | OTA Management LLC Office: 914-460-4039 mhuff@ox.com<mailto:mhuff@ox.com> | www.ox.com<http://www.ox.com> ........................................................................................................................................... From: Mel Beckman <mel@beckman.org> Sent: Monday, June 14, 2021 9:40 AM To: Matthew Huff <mhuff@ox.com> Cc: outages@outages.org Subject: Re: [outages] DNS SERVFAIL for nist.gov You should consider eliminating dependence on Internet-delivered NIST time and switch to GPS-based time servers. The GPS network has its own airborne atomic clocks that use a well-disciplined protocol to synchronize to NIST reference atomic time without transiting the Internet. According to NIST’s documentation: “Currently, the GPS system provides time to the general public with uncertainties measured in nanoseconds. With a well-designed receiver system the user can obtain the time to better than 100 ns in a few minutes, and to about +/- 10 ns with a 24 hour average (and a good local clock).” All sources of error in GPS time propagation total less than one millisecond, well within your 50ms tolerance. https://www.nist.gov/pml/time-and-frequency-division/time-services/one-way-g... NIST maintains publicly-accessible logs of al clock differences to provide documented compliance under FINRA clock synchronization rules. <image001.jpg> The log has a one-hour resolution, satisfying the FINRA requirement to verify synchronization “throughout the day”. IP-based GPS clocks are widely available with low-drift oven-controlled crystal oscillators (OXCO), or even internal cesium-based atomic clocks, for as little as a few thousand dollars. This lets you ride out time signal outages of days or even weeks. The US DHS recommends discontinuation of unauthenticated Internet-based reference clocks, owing to their vulnerability to IP address spoofing: https://www.dhs.gov/sites/default/files/publications/GPS-PNT-Best-Practices-... -mel via cell On Jun 14, 2021, at 3:51 AM, Matthew Huff via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: We have to query and compare against NIST time servers for FINRA compliance This morning I noticed our systems are unable to DNS query the NIST time servers. Neither our local resolvers or google (8.8.8.8) work. [root@bacall log]# dig @8.8.8.8 time-a-g.nist.gov ; <<>> DiG 9.11.4-P2-RedHat-9.11.4-26.P2.el7_9.5 <<>> @8.8.8.8 time-a-g.nist.gov ; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 36018 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 512 ;; QUESTION SECTION: ;time-a-g.nist.gov. IN A ;; Query time: 6 msec ;; SERVER: 8.8.8.8#53(8.8.8.8) ;; WHEN: Mon Jun 14 06:27:45 EDT 2021 ;; MSG SIZE rcvd: 46 [root@bacall log]# dig @8.8.8.8 nist.gov in soa ; <<>> DiG 9.11.4-P2-RedHat-9.11.4-26.P2.el7_9.5 <<>> @8.8.8.8 nist.gov in soa ; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 17779 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 512 ;; QUESTION SECTION: ;nist.gov. IN SOA ;; Query time: 5 msec ;; SERVER: 8.8.8.8#53(8.8.8.8) ;; WHEN: Mon Jun 14 06:31:59 EDT 2021 ;; MSG SIZE rcvd: 37 The time servers are documented here: https://tf.nist.gov/tf-cgi/servers.cgi Using the IP addresses work, it look like the nist.gov domain is offline. Matthew Huff | Director of Technical Operations | OTA Management LLC Office: 914-460-4039 mhuff@ox.com<mailto:mhuff@ox.com> | www.ox.com<http://www.ox.com> ......................................................................................................................................... _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
----- Original Message -----
From: "Mel Beckman via Outages" <outages@outages.org> Subject: Re: [outages] DNS SERVFAIL for nist.gov
I support many financial networks, and comply with the same FINRA rule 6820 you do. This rule doesn’t state that the time must be synchronized over the Internet using public IP addresses, but only “to within a fifty (50) millisecond tolerance of the time maintained by the atomic clock” at NIST . Because GPS clock times are legally traceable to NIST, and deviation logs are
They are? *Legally* traceable? I've just read the first couple pages of Judah's 2941 PDF, and the things it says to me suggest that you cannot make that assertion *about the GPS system*, only about specific measurements it produces, and that distinction seems material in this context.
available in real-time, there is no reason to depend on IP-based NTP over the Internet, and good reason not to as today’s event demonstrates. The 1ms accuracy is well within the 50ms limit.
Are you missing it, or are you purposely ignoring what he said? He is *not* "depending on [...] NTP". He merely has to have it and log it to comply with FINRA, or so his counsel tells him. I'm declaring this part of the thread out-of-bounds for outages anyway; take it to -discuss please, if you want to continue ignoring Matt. :-) Cheers, -- jr '<admin/>' a -- Jay R. Ashworth Baylink jra@baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://www.bcp38.info 2000 Land Rover DII St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 1274
Of course. Like I stated in the original email, we don’t use NIST for time sync. We actually have a GPS and a PTP feed. WE MUST, HOWEVER, VALIDATE OUR SYSTEM TIMES VERSUS THE INTERNET NIST SERVERS VIA FINRA REGULATIONS Yes, it is stupid No, it isn’t a good idea FINRA specifically state that we MUST run a comparison and log any differences between our current time and Internet NIST servers. I have challenged them on this a number of times, but with no luck. if we don’t want to get fined and prevented from trading, we have to follow FINRA regulations.. Matthew Huff | Director of Technical Operations | OTA Management LLC Office: 914-460-4039 mhuff@ox.com<mailto:mhuff@ox.com> | www.ox.com<http://www.ox.com>
----- Original Message -----
From: "Matthew Huff via Outages" <outages@outages.org>
Of course.
Like I stated in the original email, we don’t use NIST for time sync. We actually have a GPS and a PTP feed.
WE MUST, HOWEVER, VALIDATE OUR SYSTEM TIMES VERSUS THE INTERNET NIST SERVERS VIA FINRA REGULATIONS
Yes, it is stupid
No, it isn’t a good idea
FINRA specifically state that we MUST run a comparison and log any differences between our current time and Internet NIST servers. I have challenged them on this a number of times, but with no luck.
if we don’t want to get fined and prevented from trading, we have to follow FINRA regulations..
Sure. Your system for this is correctly logging the failures *at the other end outside your administrative span of control*, correct? That's Force Majeure, as far as I can see; you can't compare to a server inaccessible through no fault of your own. Find the NANOG messages, print them out, sign and date, put in your logbook. If you're really worried, consult your inside counsel to see if there's anything else you can do. And surely FINRA itself has counsel for this, too? Oh, and let's take this to -discuss. :-) Cheers, -- jr '<admin/>' a -- Jay R. Ashworth Baylink jra@baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://www.bcp38.info 2000 Land Rover DII St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 1274
On 6/14/21 8:58 AM, Jay R. Ashworth via Outages wrote:
----- Original Message -----
From: "Matthew Huff via Outages" <outages@outages.org> Of course.
Like I stated in the original email, we don’t use NIST for time sync. We actually have a GPS and a PTP feed.
WE MUST, HOWEVER, VALIDATE OUR SYSTEM TIMES VERSUS THE INTERNET NIST SERVERS VIA FINRA REGULATIONS
Yes, it is stupid
No, it isn’t a good idea
In prior job, I did work for a datacenter. Datacenter had both GPS and NTP. We monitored drift. We saw drift. At first, I was dumbfounded that our NTP pool could be drifting. Then I traced it back. Found out that there was a misconfiguration on the GPS Receiver side instead. Lesson learned: run both, track both, use each to confirm the other. Set one as priority over the other.
FINRA specifically state that we MUST run a comparison and log any differences between our current time and Internet NIST servers. I have challenged them on this a number of times, but with no luck.
if we don’t want to get fined and prevented from trading, we have to follow FINRA regulations..
participants (7)
-
Geoff Snowdon -
Jay R. Ashworth -
Matthew Huff -
Mel Beckman -
Raymond Burkholder -
Stephane Bortzmeyer -
Ullrich, Johannes