Re: [outages] NTP Issues Today
crossreplying to outages list. Is anyone ELSE seeing GPS issues? This could well have been an unrelated issue on that particular PBX. If this was real, then the mother of all infrastructure attacks might be underway... One glitch on tick and tock and one malfunctioning PBX is not sufficient evidence of pattern - much less hostile activity - to induce panic, but it would perhaps be a wise time to check time-related logs? -george On Mon, Nov 19, 2012 at 6:08 PM, Wallace Keith <kwallace@pcconnection.com> wrote:
Just got paged with a pbx alarm that had 1970 as the year. By the time I logged in , it was showing 2012. Using GPS for time and date.
-----Original Message----- From: Mark Andrews [mailto:marka@isc.org] Sent: Monday, November 19, 2012 8:42 PM To: Van Wolfe Cc: nanog@nanog.org Subject: Re: NTP Issues Today
In message <CAMeggd4cDQwhxQE_JbvpNR-PKKe9LXqA+KzJ97anHFonjwZhdQ@mail.gmail.com> , Van Wolfe writes:
Hello,
Did anyone else experience issues with NTP today? We had our server times update to the year 2000 at around 3:30 MT, then revert back to 2012.
Thanks, Van
NTP should be immune from this sort of behaviour unless you did a ntpdate at the wrong moment. The clocks should have been marked as insane.
Mark -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
-- -george william herbert george.herbert@gmail.com
We had multiple servers synchronized with Windows/MS time change their clock to the year 2000 today. It broke many things, including AD authentication. These servers had been properly synchronized for years. They were synchronized with Microsoft and NIST NTP servers. This may not be isolated. Sid Rao | CTI Group | +1 (317) 262-4677 On Nov 19, 2012, at 10:29 PM, "George Herbert" <george.herbert@gmail.com> wrote:
crossreplying to outages list.
Is anyone ELSE seeing GPS issues? This could well have been an unrelated issue on that particular PBX.
If this was real, then the mother of all infrastructure attacks might be underway...
One glitch on tick and tock and one malfunctioning PBX is not sufficient evidence of pattern - much less hostile activity - to induce panic, but it would perhaps be a wise time to check time-related logs?
-george
On Mon, Nov 19, 2012 at 6:08 PM, Wallace Keith <kwallace@pcconnection.com> wrote:
Just got paged with a pbx alarm that had 1970 as the year. By the time I logged in , it was showing 2012. Using GPS for time and date.
-----Original Message----- From: Mark Andrews [mailto:marka@isc.org] Sent: Monday, November 19, 2012 8:42 PM To: Van Wolfe Cc: nanog@nanog.org Subject: Re: NTP Issues Today
In message <CAMeggd4cDQwhxQE_JbvpNR-PKKe9LXqA+KzJ97anHFonjwZhdQ@mail.gmail.com> , Van Wolfe writes:
Hello,
Did anyone else experience issues with NTP today? We had our server times update to the year 2000 at around 3:30 MT, then revert back to 2012.
Thanks, Van
NTP should be immune from this sort of behaviour unless you did a ntpdate at the wrong moment. The clocks should have been marked as insane.
Mark -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
-- -george william herbert george.herbert@gmail.com
Anyone check out the NIST GPS Archive? http://www.nist.gov/pml/div688/grp40/gpsarchive.cfm -Mike On Mon, Nov 19, 2012 at 7:58 PM, Sid Rao <srao@ctigroup.com> wrote:
We had multiple servers synchronized with Windows/MS time change their clock to the year 2000 today. It broke many things, including AD authentication.
These servers had been properly synchronized for years.
They were synchronized with Microsoft and NIST NTP servers.
This may not be isolated.
Sid Rao | CTI Group | +1 (317) 262-4677
On Nov 19, 2012, at 10:29 PM, "George Herbert" <george.herbert@gmail.com> wrote:
crossreplying to outages list.
Is anyone ELSE seeing GPS issues? This could well have been an unrelated issue on that particular PBX.
If this was real, then the mother of all infrastructure attacks might be underway...
One glitch on tick and tock and one malfunctioning PBX is not sufficient evidence of pattern - much less hostile activity - to induce panic, but it would perhaps be a wise time to check time-related logs?
-george
On Mon, Nov 19, 2012 at 6:08 PM, Wallace Keith <kwallace@pcconnection.com> wrote:
Just got paged with a pbx alarm that had 1970 as the year. By the time I logged in , it was showing 2012. Using GPS for time and date.
-----Original Message----- From: Mark Andrews [mailto:marka@isc.org] Sent: Monday, November 19, 2012 8:42 PM To: Van Wolfe Cc: nanog@nanog.org Subject: Re: NTP Issues Today
In message < CAMeggd4cDQwhxQE_JbvpNR-PKKe9LXqA+KzJ97anHFonjwZhdQ@mail.gmail.com> , Van Wolfe writes:
Hello,
Did anyone else experience issues with NTP today? We had our server times update to the year 2000 at around 3:30 MT, then revert back to
Thanks, Van
NTP should be immune from this sort of behaviour unless you did a ntpdate at the wrong moment. The clocks should have been marked as insane.
Mark -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
-- -george william herbert george.herbert@gmail.com
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- Mike Lyon 408-621-4826 mike.lyon@gmail.com http://www.linkedin.com/in/mlyon
Same thing happened to us yesterday. ended up having to reboot everything after we got time fixed. Major outage. Scott On Mon, Nov 19, 2012 at 7:58 PM, Sid Rao <srao@ctigroup.com> wrote:
We had multiple servers synchronized with Windows/MS time change their clock to the year 2000 today. It broke many things, including AD authentication.
These servers had been properly synchronized for years.
They were synchronized with Microsoft and NIST NTP servers.
This may not be isolated.
Sid Rao | CTI Group | +1 (317) 262-4677
On Nov 19, 2012, at 10:29 PM, "George Herbert" <george.herbert@gmail.com> wrote:
crossreplying to outages list.
Is anyone ELSE seeing GPS issues? This could well have been an unrelated issue on that particular PBX.
If this was real, then the mother of all infrastructure attacks might be underway...
One glitch on tick and tock and one malfunctioning PBX is not sufficient evidence of pattern - much less hostile activity - to induce panic, but it would perhaps be a wise time to check time-related logs?
-george
On Mon, Nov 19, 2012 at 6:08 PM, Wallace Keith <kwallace@pcconnection.com> wrote:
Just got paged with a pbx alarm that had 1970 as the year. By the time I logged in , it was showing 2012. Using GPS for time and date.
-----Original Message----- From: Mark Andrews [mailto:marka@isc.org] Sent: Monday, November 19, 2012 8:42 PM To: Van Wolfe Cc: nanog@nanog.org Subject: Re: NTP Issues Today
In message < CAMeggd4cDQwhxQE_JbvpNR-PKKe9LXqA+KzJ97anHFonjwZhdQ@mail.gmail.com> , Van Wolfe writes:
Hello,
Did anyone else experience issues with NTP today? We had our server times update to the year 2000 at around 3:30 MT, then revert back to
Thanks, Van
NTP should be immune from this sort of behaviour unless you did a ntpdate at the wrong moment. The clocks should have been marked as insane.
Mark -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
-- -george william herbert george.herbert@gmail.com
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
I'm still waiting for someone who was affected by this to provide coherent logs from ntpd showing exactly when the time change happened. Getting these, at least on an *IX system, is far from difficult folks. Please don't omit anything from the logs either; for example if you know *exactly* what NTP servers were in use (not "ones you had configured" but which one was primarily chosen by ntpd ('*' mark) and which were secondary comparisons/fallbacks ('+' mark)), that would also be greatly helpful. This would be output from "ntpq -c peers" when run on your NTP server *at or around the time* the incident happened and recovered. What's been provided so far is that "something happened", with reports of clocks going back to year 2000, and other reports of clocks going back to (presumably) epoch time; those reporting it were using either usno.navy.mil, NIST, or Microsoft NTP servers. usno.navy.mil uses dedicated IRIG/AFNOR TCRs boxes, while NIST uses GPS. No idea what Microsoft uses. I asked on a public *IX forum if anyone saw anything NTP-wise that was out of the ordinary and not a single admin saw anything. I also saw nothing anomalous on either of my FreeBSD machines (9.1-PRERELEASE, running base system ntpd 4.2.4p8), but I sync with very specific stratum 1 and stratum 2 servers across the United States. As Mark Andrews from the ISC stated below (read slowly/carefully), ntpd will not allow large clock jumps -- the largest it'll allow out of the box is 1000s (and on some systems like Solaris ntpd, 500s) -- unless you're running with the -g flag (and shame on if you're you doing that). So I'm very surprised by this problem altogether. Can't deny what happened did, but figuring out *why* is important. Also, for Mike Lyon -- I looked at NIST's GPS graphs. Did you notice they have no data for 11/18, 11/19, or 11/20? I find that unnerving, do you not? -- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB | On Tue, Nov 20, 2012 at 07:18:45AM -0800, Scott Voll wrote:
Same thing happened to us yesterday. ended up having to reboot everything after we got time fixed. Major outage.
Scott
On Mon, Nov 19, 2012 at 7:58 PM, Sid Rao <srao@ctigroup.com> wrote:
We had multiple servers synchronized with Windows/MS time change their clock to the year 2000 today. It broke many things, including AD authentication.
These servers had been properly synchronized for years.
They were synchronized with Microsoft and NIST NTP servers.
This may not be isolated.
Sid Rao | CTI Group | +1 (317) 262-4677
On Nov 19, 2012, at 10:29 PM, "George Herbert" <george.herbert@gmail.com> wrote:
crossreplying to outages list.
Is anyone ELSE seeing GPS issues? This could well have been an unrelated issue on that particular PBX.
If this was real, then the mother of all infrastructure attacks might be underway...
One glitch on tick and tock and one malfunctioning PBX is not sufficient evidence of pattern - much less hostile activity - to induce panic, but it would perhaps be a wise time to check time-related logs?
-george
On Mon, Nov 19, 2012 at 6:08 PM, Wallace Keith <kwallace@pcconnection.com> wrote:
Just got paged with a pbx alarm that had 1970 as the year. By the time I logged in , it was showing 2012. Using GPS for time and date.
-----Original Message----- From: Mark Andrews [mailto:marka@isc.org] Sent: Monday, November 19, 2012 8:42 PM To: Van Wolfe Cc: nanog@nanog.org Subject: Re: NTP Issues Today
In message < CAMeggd4cDQwhxQE_JbvpNR-PKKe9LXqA+KzJ97anHFonjwZhdQ@mail.gmail.com> , Van Wolfe writes:
Hello,
Did anyone else experience issues with NTP today? We had our server times update to the year 2000 at around 3:30 MT, then revert back to
Thanks, Van
NTP should be immune from this sort of behaviour unless you did a ntpdate at the wrong moment. The clocks should have been marked as insane.
Mark -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
-- -george william herbert george.herbert@gmail.com
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Nov 19 16:04:38 marty ntpd[2214]: synchronized to LOCAL(0), stratum 10 Nov 19 16:18:23 marty named[2160]: zone my.slave.internal.zone/IN/internal: refresh: non-authoritative answer from master 127.0.0.1#53 (source 0.0.0.0#0) Nov 19 16:21:51 marty ntpd[2214]: synchronized to 192.5.41.41, stratum 1 #this is where it crashed Nov 19 16:38:41 marty ntpd[2214]: time correction of -378691201 seconds exceeds sanity limit (1000); set clock manually to the correct UTC time. #this is where i restarted ntpd by hand Nov 19 16:45:53 marty ntpd[5043]: ntpd 4.2.2p1@1.1570-o Fri Nov 18 13:21:16 UTC 2011 (1) Nov 19 16:45:54 marty ntpd[5044]: precision = 1.000 usec Nov 19 16:45:54 marty ntpd[5044]: Listening on interface wildcard, 0.0.0.0#123 Disabled Nov 19 16:45:54 marty ntpd[5044]: Listening on interface wildcard, ::#123 Disabled Nov 19 16:45:54 marty ntpd[5044]: Listening on interface lo, ::1#123 Enabled Nov 19 16:45:54 marty ntpd[5044]: Listening on interface eth0, fe80::20c:29ff:fe07:8dc7#123 Enabled Nov 19 16:45:54 marty ntpd[5044]: Listening on interface lo, 127.0.0.1#123 Enabled Nov 19 16:45:54 marty ntpd[5044]: Listening on interface eth0, 96.11.78.2#123 Enabled Nov 19 16:45:54 marty ntpd[5044]: Listening on interface pntun1, 10.255.255.254#123 Enabled Nov 19 16:45:54 marty ntpd[5044]: kernel time sync status 0040 Nov 19 16:45:54 marty ntpd[5044]: frequency initialized 1.864 PPM from /var/lib/ntp/drift Nov 19 16:49:10 marty ntpd[5044]: synchronized to LOCAL(0), stratum 10 Nov 19 16:49:10 marty ntpd[5044]: kernel time sync disabled 0001 Nov 19 17:38:20 marty ntpd[5044]: synchronized to 192.5.41.41, stratum 1 Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373 On Tue, Nov 20, 2012 at 10:38 AM, Jeremy Chadwick <jdc@koitsu.org> wrote:
I'm still waiting for someone who was affected by this to provide coherent logs from ntpd showing exactly when the time change happened. Getting these, at least on an *IX system, is far from difficult folks.
Please don't omit anything from the logs either; for example if you know *exactly* what NTP servers were in use (not "ones you had configured" but which one was primarily chosen by ntpd ('*' mark) and which were secondary comparisons/fallbacks ('+' mark)), that would also be greatly helpful. This would be output from "ntpq -c peers" when run on your NTP server *at or around the time* the incident happened and recovered.
What's been provided so far is that "something happened", with reports of clocks going back to year 2000, and other reports of clocks going back to (presumably) epoch time; those reporting it were using either usno.navy.mil, NIST, or Microsoft NTP servers. usno.navy.mil uses dedicated IRIG/AFNOR TCRs boxes, while NIST uses GPS. No idea what Microsoft uses.
I asked on a public *IX forum if anyone saw anything NTP-wise that was out of the ordinary and not a single admin saw anything. I also saw nothing anomalous on either of my FreeBSD machines (9.1-PRERELEASE, running base system ntpd 4.2.4p8), but I sync with very specific stratum 1 and stratum 2 servers across the United States.
As Mark Andrews from the ISC stated below (read slowly/carefully), ntpd will not allow large clock jumps -- the largest it'll allow out of the box is 1000s (and on some systems like Solaris ntpd, 500s) -- unless you're running with the -g flag (and shame on if you're you doing that). So I'm very surprised by this problem altogether. Can't deny what happened did, but figuring out *why* is important.
Also, for Mike Lyon -- I looked at NIST's GPS graphs. Did you notice they have no data for 11/18, 11/19, or 11/20? I find that unnerving, do you not?
-- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB |
On Tue, Nov 20, 2012 at 07:18:45AM -0800, Scott Voll wrote:
Same thing happened to us yesterday. ended up having to reboot everything after we got time fixed. Major outage.
Scott
On Mon, Nov 19, 2012 at 7:58 PM, Sid Rao <srao@ctigroup.com> wrote:
We had multiple servers synchronized with Windows/MS time change their clock to the year 2000 today. It broke many things, including AD authentication.
These servers had been properly synchronized for years.
They were synchronized with Microsoft and NIST NTP servers.
This may not be isolated.
Sid Rao | CTI Group | +1 (317) 262-4677
On Nov 19, 2012, at 10:29 PM, "George Herbert" < george.herbert@gmail.com> wrote:
crossreplying to outages list.
Is anyone ELSE seeing GPS issues? This could well have been an unrelated issue on that particular PBX.
If this was real, then the mother of all infrastructure attacks might be underway...
One glitch on tick and tock and one malfunctioning PBX is not sufficient evidence of pattern - much less hostile activity - to induce panic, but it would perhaps be a wise time to check time-related logs?
-george
On Mon, Nov 19, 2012 at 6:08 PM, Wallace Keith <kwallace@pcconnection.com> wrote:
Just got paged with a pbx alarm that had 1970 as the year. By the time I logged in , it was showing 2012. Using GPS for time and date.
-----Original Message----- From: Mark Andrews [mailto:marka@isc.org] Sent: Monday, November 19, 2012 8:42 PM To: Van Wolfe Cc: nanog@nanog.org Subject: Re: NTP Issues Today
In message < CAMeggd4cDQwhxQE_JbvpNR-PKKe9LXqA+KzJ97anHFonjwZhdQ@mail.gmail.com> , Van Wolfe writes:
Hello,
Did anyone else experience issues with NTP today? We had our server times update to the year 2000 at around 3:30 MT, then revert back to
Thanks, Van
NTP should be immune from this sort of behaviour unless you did a ntpdate at the wrong moment. The clocks should have been marked as insane.
Mark -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
-- -george william herbert george.herbert@gmail.com
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
From my core:
Nov 19 14:22:16.062 MST: %SYS-6-CLOCKUPDATE: System clock has been updated from 14:22:16 MST Mon Nov 19 2012 to 14:22:16 MST Sun Nov 19 2000, configured from NTP by 192.5.41.40. Regards, Ernesto Oddone This message and any attachments are intended only for the addressee(s) and may contain privileged or confidential information. Any unauthorized disclosure is strictly prohibited. If you have received this message in error, please notify us immediately so that we may correct our internal records. Please then permanently delete the original message and any attachments and destroy any copies. Thank you. -----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Jeremy Chadwick Sent: Tuesday, November 20, 2012 10:38 AM To: Scott Voll Cc: Sid Rao; outages; nanog@nanog.org Subject: Re: [outages] NTP Issues Today I'm still waiting for someone who was affected by this to provide coherent logs from ntpd showing exactly when the time change happened. Getting these, at least on an *IX system, is far from difficult folks. Please don't omit anything from the logs either; for example if you know *exactly* what NTP servers were in use (not "ones you had configured" but which one was primarily chosen by ntpd ('*' mark) and which were secondary comparisons/fallbacks ('+' mark)), that would also be greatly helpful. This would be output from "ntpq -c peers" when run on your NTP server *at or around the time* the incident happened and recovered. What's been provided so far is that "something happened", with reports of clocks going back to year 2000, and other reports of clocks going back to (presumably) epoch time; those reporting it were using either usno.navy.mil, NIST, or Microsoft NTP servers. usno.navy.mil uses dedicated IRIG/AFNOR TCRs boxes, while NIST uses GPS. No idea what Microsoft uses. I asked on a public *IX forum if anyone saw anything NTP-wise that was out of the ordinary and not a single admin saw anything. I also saw nothing anomalous on either of my FreeBSD machines (9.1-PRERELEASE, running base system ntpd 4.2.4p8), but I sync with very specific stratum 1 and stratum 2 servers across the United States. As Mark Andrews from the ISC stated below (read slowly/carefully), ntpd will not allow large clock jumps -- the largest it'll allow out of the box is 1000s (and on some systems like Solaris ntpd, 500s) -- unless you're running with the -g flag (and shame on if you're you doing that). So I'm very surprised by this problem altogether. Can't deny what happened did, but figuring out *why* is important. Also, for Mike Lyon -- I looked at NIST's GPS graphs. Did you notice they have no data for 11/18, 11/19, or 11/20? I find that unnerving, do you not? -- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB | On Tue, Nov 20, 2012 at 07:18:45AM -0800, Scott Voll wrote:
Same thing happened to us yesterday. ended up having to reboot everything after we got time fixed. Major outage.
Scott
On Mon, Nov 19, 2012 at 7:58 PM, Sid Rao <srao@ctigroup.com> wrote:
We had multiple servers synchronized with Windows/MS time change their clock to the year 2000 today. It broke many things, including AD authentication.
These servers had been properly synchronized for years.
They were synchronized with Microsoft and NIST NTP servers.
This may not be isolated.
Sid Rao | CTI Group | +1 (317) 262-4677
On Nov 19, 2012, at 10:29 PM, "George Herbert" <george.herbert@gmail.com> wrote:
crossreplying to outages list.
Is anyone ELSE seeing GPS issues? This could well have been an unrelated issue on that particular PBX.
If this was real, then the mother of all infrastructure attacks might be underway...
One glitch on tick and tock and one malfunctioning PBX is not sufficient evidence of pattern - much less hostile activity - to induce panic, but it would perhaps be a wise time to check time-related logs?
-george
On Mon, Nov 19, 2012 at 6:08 PM, Wallace Keith <kwallace@pcconnection.com> wrote:
Just got paged with a pbx alarm that had 1970 as the year. By the time I logged in , it was showing 2012. Using GPS for time and date.
-----Original Message----- From: Mark Andrews [mailto:marka@isc.org] Sent: Monday, November 19, 2012 8:42 PM To: Van Wolfe Cc: nanog@nanog.org Subject: Re: NTP Issues Today
In message < CAMeggd4cDQwhxQE_JbvpNR-PKKe9LXqA+KzJ97anHFonjwZhdQ@mail.gmail.com> , Van Wolfe writes:
Hello,
Did anyone else experience issues with NTP today? We had our server times update to the year 2000 at around 3:30 MT, then revert back to
Thanks, Van
NTP should be immune from this sort of behaviour unless you did a ntpdate at the wrong moment. The clocks should have been marked as insane.
Mark -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
-- -george william herbert george.herbert@gmail.com
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
On 20 Nov 2012, at 15:38, Jeremy Chadwick <jdc@koitsu.org> wrote:
I'm still waiting for someone who was affected by this to provide coherent logs from ntpd showing exactly when the time change happened. Getting these, at least on an *IX system, is far from difficult folks.
from firewall ntp logs Nov 19 09:58:06 [192.168.0.1.128.176] 2012:11:19-09:58:06 ntpd[21385]: ntpd exiting on signal 15 Nov 19 09:58:19 [192.168.0.1.128.176] 2012:11:19-09:58:19 selfmonng[3503]: W check Failed increment ntpd_running counter 3 - 3 Nov 19 09:58:22 [192.168.0.1.128.176] 2012:11:19-09:58:22 selfmonng[3503]: W NOTIFYEVENT Name=ntpd_running Level=INFO Id=147 sent Nov 19 09:58:22 [192.168.0.1.128.176] 2012:11:19-09:58:22 selfmonng[3503]: W triggerAction: 'cmd' Nov 19 09:58:22 [192.168.0.1.128.176] 2012:11:19-09:58:22 selfmonng[3503]: W actionCmd(+): '/var/mdw/scripts/ntp restart' Nov 19 09:58:25 [192.168.0.1.128.176] 2012:11:19-09:58:25 ntpd[24120]: ntpd 4.2.4p8@1.1612-o Tue Feb 2 21:46:54 UTC 2010 (1) Nov 19 09:58:25 [192.168.0.1.128.176] 2012:11:19-09:58:25 selfmonng[3503]: W child returned status: exit='0' signal='0' Nov 19 09:58:35 [192.168.0.1.128.176] 2012:11:19-09:58:35 ntpd[24121]: kernel time sync status change 0001 was sync'd to 84.25.175.98, stratum 2 at the time I believe Colin
Colin, Signal 15 = SIGTERM, so something intentionally shut ntpd down on your side. The logs I'd be interested in would be prior to what you've provided, i.e. what lead to the SIGTERM. Also, no timezone is mentioned anywhere in your timestamps, so please provide that (UTC offset would be best). -- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB | On Tue, Nov 20, 2012 at 05:02:06PM +0000, Colin Johnston wrote:
On 20 Nov 2012, at 15:38, Jeremy Chadwick <jdc@koitsu.org> wrote:
I'm still waiting for someone who was affected by this to provide coherent logs from ntpd showing exactly when the time change happened. Getting these, at least on an *IX system, is far from difficult folks.
from firewall ntp logs Nov 19 09:58:06 [192.168.0.1.128.176] 2012:11:19-09:58:06 ntpd[21385]: ntpd exiting on signal 15 Nov 19 09:58:19 [192.168.0.1.128.176] 2012:11:19-09:58:19 selfmonng[3503]: W check Failed increment ntpd_running counter 3 - 3 Nov 19 09:58:22 [192.168.0.1.128.176] 2012:11:19-09:58:22 selfmonng[3503]: W NOTIFYEVENT Name=ntpd_running Level=INFO Id=147 sent Nov 19 09:58:22 [192.168.0.1.128.176] 2012:11:19-09:58:22 selfmonng[3503]: W triggerAction: 'cmd' Nov 19 09:58:22 [192.168.0.1.128.176] 2012:11:19-09:58:22 selfmonng[3503]: W actionCmd(+): '/var/mdw/scripts/ntp restart' Nov 19 09:58:25 [192.168.0.1.128.176] 2012:11:19-09:58:25 ntpd[24120]: ntpd 4.2.4p8@1.1612-o Tue Feb 2 21:46:54 UTC 2010 (1) Nov 19 09:58:25 [192.168.0.1.128.176] 2012:11:19-09:58:25 selfmonng[3503]: W child returned status: exit='0' signal='0' Nov 19 09:58:35 [192.168.0.1.128.176] 2012:11:19-09:58:35 ntpd[24121]: kernel time sync status change 0001
was sync'd to 84.25.175.98, stratum 2 at the time I believe
Colin
no idea, re sigterm cause checked firewall system logs and could not see cause from that either times are GMT Colin On 20 Nov 2012, at 17:05, Jeremy Chadwick <jdc@koitsu.org> wrote:
Colin,
Signal 15 = SIGTERM, so something intentionally shut ntpd down on your side. The logs I'd be interested in would be prior to what you've provided, i.e. what lead to the SIGTERM.
Also, no timezone is mentioned anywhere in your timestamps, so please provide that (UTC offset would be best).
-- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB |
On Tue, Nov 20, 2012 at 05:02:06PM +0000, Colin Johnston wrote:
On 20 Nov 2012, at 15:38, Jeremy Chadwick <jdc@koitsu.org> wrote:
I'm still waiting for someone who was affected by this to provide coherent logs from ntpd showing exactly when the time change happened. Getting these, at least on an *IX system, is far from difficult folks.
from firewall ntp logs Nov 19 09:58:06 [192.168.0.1.128.176] 2012:11:19-09:58:06 ntpd[21385]: ntpd exiting on signal 15 Nov 19 09:58:19 [192.168.0.1.128.176] 2012:11:19-09:58:19 selfmonng[3503]: W check Failed increment ntpd_running counter 3 - 3 Nov 19 09:58:22 [192.168.0.1.128.176] 2012:11:19-09:58:22 selfmonng[3503]: W NOTIFYEVENT Name=ntpd_running Level=INFO Id=147 sent Nov 19 09:58:22 [192.168.0.1.128.176] 2012:11:19-09:58:22 selfmonng[3503]: W triggerAction: 'cmd' Nov 19 09:58:22 [192.168.0.1.128.176] 2012:11:19-09:58:22 selfmonng[3503]: W actionCmd(+): '/var/mdw/scripts/ntp restart' Nov 19 09:58:25 [192.168.0.1.128.176] 2012:11:19-09:58:25 ntpd[24120]: ntpd 4.2.4p8@1.1612-o Tue Feb 2 21:46:54 UTC 2010 (1) Nov 19 09:58:25 [192.168.0.1.128.176] 2012:11:19-09:58:25 selfmonng[3503]: W child returned status: exit='0' signal='0' Nov 19 09:58:35 [192.168.0.1.128.176] 2012:11:19-09:58:35 ntpd[24121]: kernel time sync status change 0001
was sync'd to 84.25.175.98, stratum 2 at the time I believe
Colin
Logs from a Juniper router in a customer network - we had hundreds of these affected. They all synchronize to internal hosts (172.20.167.251 and .252) which are configured to get time from NIST and USNO CORP-NTP-01#sh ntp as address ref clock st when poll reach delay offset disp *~192.5.41.41 .IRIG. 1 354 512 377 34.2 0.36 1.4 +~132.163.4.101 .ACTS. 1 336 512 377 35.0 -2.54 18.7 ~127.127.7.1 127.127.7.1 10 59 64 377 0.0 0.00 0.0 * master (synced), # master (unsynced), + selected, - candidate, ~ configured CORP-NTP-02#sh ntp as address ref clock st when poll reach delay offset disp *~192.5.41.41 .IRIG. 1 65 512 377 36.5 0.91 0.6 +~132.163.4.101 .ACTS. 1 95 512 377 34.3 -1.31 22.8 ~127.127.7.1 127.127.7.1 10 44 64 377 0.0 0.00 0.0 * master (synced), # master (unsynced), + selected, - candidate, ~ configured Here are the logs from one of the Junipers: Nov 19 14:24:48 XXXX xntpd[912]: kernel time sync enabled 2001 Nov 19 15:50:11 XXXX xntpd[912]: synchronized to 172.20.167.252, stratum=2 Nov 19 16:41:23 XXXX xntpd[912]: no servers reachable Nov 19 16:44:24 XXXX xntpd[912]: synchronized to 172.20.167.251, stratum=2 Nov 19 16:44:24 XXXX xntpd[912]: time correction of -378691200 seconds exceeds sanity limit (1000); set clock manually to the correct UTC time. Nov 19 16:44:24 XXXX init: ntp (PID 912) exited with status=255 Nov 19 16:44:24 XXXX init: ntp (PID 70200) started Nov 19 16:44:24 XXXX xntpd[70200]: ntpd 4.2.0-a Sat Apr 10 00:32:46 UTC 2010 (1) Nov 19 16:44:24 XXXX xntpd[70200]: mlockall(): Resource temporarily unavailable Nov 19 16:44:24 XXXX xntpd[70200]: precision = 0.582 usec Nov 19 16:44:24 XXXX xntpd[70200]: Listening on interface ggsn_vpn, 128.0.0.1#123 Nov 19 16:44:24 XXXX xntpd[70200]: kernel time sync status 2040 Nov 19 16:44:24 XXXX xntpd[70200]: frequency initialized -64.931 PPM from /var/db/ntp.drift Nov 19 16:44:24 XXXX xntpd[70200]: Configuring iburst flag for server Nov 19 16:44:24 XXXX xntpd[70200]: Configuring iburst flag for server Nov 19 16:44:33 XXXX xntpd[70200]: synchronized to 172.20.167.251, stratum=2 Nov 19 16:44:32 XXXX xntpd[70200]: time reset -378691200.411331 s Nov 19 16:44:32 XXXX xntpd[70200]: kernel time sync disabled 2041 Nov 19 16:45:44 XXXX xntpd[70200]: synchronized to 172.20.167.251, stratum=2 Nov 19 16:45:51 XXXX xntpd[70200]: kernel time sync enabled 2001 Nov 19 16:45:56 XXXX xntpd[70200]: NTP Server Unreachable Nov 19 16:53:25 XXXX xntpd[70200]: no servers reachable Nov 19 17:03:09 XXXX xntpd[70200]: NTP Server Unreachable Nov 19 17:13:00 XXXX xntpd[70200]: NTP Server Unreachable Nov 19 17:20:27 XXXX xntpd[70200]: synchronized to 172.20.167.252, stratum=2 Nov 19 17:20:27 XXXX xntpd[70200]: time correction of 378691200 seconds exceeds sanity limit (1000); set clock manually to the correct UTC time. Nov 19 17:20:27 XXXX init: ntp (PID 70200) exited with status=255 Nov 19 17:20:27 XXXX init: ntp (PID 70766) started Nov 19 17:20:27 XXXX xntpd[70766]: ntpd 4.2.0-a Sat Apr 10 00:32:46 UTC 2010 (1) Nov 19 17:20:27 XXXX xntpd[70766]: mlockall(): Resource temporarily unavailable Nov 19 17:20:27 XXXX xntpd[70766]: precision = 0.570 usec Nov 19 17:20:27 XXXX xntpd[70766]: Listening on interface ggsn_vpn, 128.0.0.1#123 Nov 19 17:20:27 XXXX xntpd[70766]: kernel time sync status 2040 Nov 19 17:20:27 XXXX xntpd[70766]: frequency initialized -64.931 PPM from /var/db/ntp.drift Nov 19 17:20:27 XXXX xntpd[70766]: Configuring iburst flag for server Nov 19 17:20:27 XXXX xntpd[70766]: Configuring iburst flag for server Nov 19 17:20:35 XXXX xntpd[70766]: synchronized to 172.20.167.252, stratum=2 Nov 19 17:20:36 XXXX xntpd[70766]: time reset +378691200.387434 s Nov 19 17:20:36 XXXX xntpd[70766]: kernel time sync disabled 6041 Nov 19 17:21:48 XXXX xntpd[70766]: synchronized to 172.20.167.252, stratum=2 Nov 19 17:21:48 XXXX xntpd[70766]: kernel time sync disabled 2041 Nov 19 17:21:52 XXXX xntpd[70766]: kernel time sync enabled 2001 Nov 20 00:02:29 XXXX xntpd[70766]: synchronized to 172.20.167.251, stratum=2 Nov 20 01:44:56 XXXX xntpd[70766]: kernel time sync enabled 6001 Nov 20 02:19:03 XXXX xntpd[70766]: kernel time sync enabled 2001 Nov 20 02:53:12 XXXX xntpd[70766]: kernel time sync enabled 6001 Nov 20 03:44:26 XXXX xntpd[70766]: kernel time sync enabled 2001 Nov 20 05:26:58 XXXX xntpd[70766]: kernel time sync enabled 6001 Nov 20 05:44:02 XXXX xntpd[70766]: kernel time sync enabled 2001 Nov 20 07:43:35 XXXX xntpd[70766]: kernel time sync enabled 6001 Nov 20 08:00:39 XXXX xntpd[70766]: kernel time sync enabled 2001 Nov 20 08:34:48 XXXX xntpd[70766]: kernel time sync enabled 6001 Nov 20 08:51:54 XXXX xntpd[70766]: kernel time sync enabled 2001 Nov 20 10:34:22 XXXX xntpd[70766]: synchronized to 172.20.167.252, stratum=2 Nov 20 11:25:16 XXXX xntpd[70766]: synchronized to 172.20.167.251, stratum=2 Nov 20 12:33:56 XXXX xntpd[70766]: synchronized to 172.20.167.252, stratum=2 Nov 20 14:16:05 XXXX xntpd[70766]: kernel time sync enabled 6001 Nov 20 14:33:10 XXXX xntpd[70766]: kernel time sync enabled 2001 Nov 20 15:07:19 XXXX xntpd[70766]: synchronized to 172.20.167.251, stratum=2 -----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Jeremy Chadwick Sent: Tuesday, November 20, 2012 10:38 AM To: Scott Voll Cc: Sid Rao; outages; nanog@nanog.org Subject: Re: [outages] NTP Issues Today I'm still waiting for someone who was affected by this to provide coherent logs from ntpd showing exactly when the time change happened. Getting these, at least on an *IX system, is far from difficult folks. Please don't omit anything from the logs either; for example if you know *exactly* what NTP servers were in use (not "ones you had configured" but which one was primarily chosen by ntpd ('*' mark) and which were secondary comparisons/fallbacks ('+' mark)), that would also be greatly helpful. This would be output from "ntpq -c peers" when run on your NTP server *at or around the time* the incident happened and recovered. What's been provided so far is that "something happened", with reports of clocks going back to year 2000, and other reports of clocks going back to (presumably) epoch time; those reporting it were using either usno.navy.mil, NIST, or Microsoft NTP servers. usno.navy.mil uses dedicated IRIG/AFNOR TCRs boxes, while NIST uses GPS. No idea what Microsoft uses. I asked on a public *IX forum if anyone saw anything NTP-wise that was out of the ordinary and not a single admin saw anything. I also saw nothing anomalous on either of my FreeBSD machines (9.1-PRERELEASE, running base system ntpd 4.2.4p8), but I sync with very specific stratum 1 and stratum 2 servers across the United States. As Mark Andrews from the ISC stated below (read slowly/carefully), ntpd will not allow large clock jumps -- the largest it'll allow out of the box is 1000s (and on some systems like Solaris ntpd, 500s) -- unless you're running with the -g flag (and shame on if you're you doing that). So I'm very surprised by this problem altogether. Can't deny what happened did, but figuring out *why* is important. Also, for Mike Lyon -- I looked at NIST's GPS graphs. Did you notice they have no data for 11/18, 11/19, or 11/20? I find that unnerving, do you not? -- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB | On Tue, Nov 20, 2012 at 07:18:45AM -0800, Scott Voll wrote:
Same thing happened to us yesterday. ended up having to reboot everything after we got time fixed. Major outage.
Scott
On Mon, Nov 19, 2012 at 7:58 PM, Sid Rao <srao@ctigroup.com> wrote:
We had multiple servers synchronized with Windows/MS time change their clock to the year 2000 today. It broke many things, including AD authentication.
These servers had been properly synchronized for years.
They were synchronized with Microsoft and NIST NTP servers.
This may not be isolated.
Sid Rao | CTI Group | +1 (317) 262-4677
On Nov 19, 2012, at 10:29 PM, "George Herbert" <george.herbert@gmail.com> wrote:
crossreplying to outages list.
Is anyone ELSE seeing GPS issues? This could well have been an unrelated issue on that particular PBX.
If this was real, then the mother of all infrastructure attacks might be underway...
One glitch on tick and tock and one malfunctioning PBX is not sufficient evidence of pattern - much less hostile activity - to induce panic, but it would perhaps be a wise time to check time-related logs?
-george
On Mon, Nov 19, 2012 at 6:08 PM, Wallace Keith <kwallace@pcconnection.com> wrote:
Just got paged with a pbx alarm that had 1970 as the year. By the time I logged in , it was showing 2012. Using GPS for time and date.
-----Original Message----- From: Mark Andrews [mailto:marka@isc.org] Sent: Monday, November 19, 2012 8:42 PM To: Van Wolfe Cc: nanog@nanog.org Subject: Re: NTP Issues Today
In message < CAMeggd4cDQwhxQE_JbvpNR-PKKe9LXqA+KzJ97anHFonjwZhdQ@mail.gmail.com> , Van Wolfe writes:
Hello,
Did anyone else experience issues with NTP today? We had our server times update to the year 2000 at around 3:30 MT, then revert back to
Thanks, Van
NTP should be immune from this sort of behaviour unless you did a ntpdate at the wrong moment. The clocks should have been marked as insane.
Mark -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
-- -george william herbert george.herbert@gmail.com
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
participants (9)
-
Colin Johnston -
George Herbert -
Jeremy Chadwick -
Josh Luthman -
Mike Lyon -
Oddone, Ernesto -
R. Benjamin Kessler -
Scott Voll -
Sid Rao