The impact and resolution of the GPS week number rollover of April 2019 on autonomous geophysical instrument platforms

Coyle, Shane; Clauer, C. Robert; Hartinger, Michael D.; Xu, Zhonghua; Peng, Yuxiang

doi:https://doi.org/10.5194/gi-10-161-2021

Articles | Volume 10, issue 2

https://doi.org/10.5194/gi-10-161-2021

© Author(s) 2021. This work is distributed under
the Creative Commons Attribution 4.0 License.

https://doi.org/10.5194/gi-10-161-2021

© Author(s) 2021. This work is distributed under
the Creative Commons Attribution 4.0 License.

Articles | Volume 10, issue 2

Review article

|

28 Jul 2021

Review article |

| 28 Jul 2021

The impact and resolution of the GPS week number rollover of April 2019 on autonomous geophysical instrument platforms

Shane Coyle, C. Robert Clauer, Michael D. Hartinger, Zhonghua Xu, and Yuxiang Peng

Download

Final revised paper (published on 28 Jul 2021)
Preprint (discussion started on 18 Jan 2021)

Interactive discussion

Status: closed

RC1:
'Comment on gi-2020-47', Anonymous Referee #1, 05 Apr 2021

General Comments: This manuscript provides a thorough description of a failure mode of the remotely operated instruments related to the GPS week rollover that occurred in 2019. A description of how the instruments failed is given as well as how the operators were able to bring the instruments back online. At the end of the manuscript, the authors provide some general advice for remote instrument operation that is a direct result of their lessons learned from their whole experience. In general, the manuscript is well organized and provides clear descriptions of what happened and how they were able to fix their problems. I feel that this is useful information to be out and available to the scientific community, especially with regards to how to avoid such instrument failures in the future. I feel that this manuscript is acceptable in its present form.

Citation: https://doi.org/10.5194/gi-2020-47-RC1
- AC1: 'Reply on RC1', Shane Coyle, 02 May 2021
  
  The authors thank the reviewer for their comments.
  
  Citation: https://doi.org/10.5194/gi-2020-47-AC1
RC2:
'Comment on gi-2020-47', Anonymous Referee #2, 14 Apr 2021

This manuscript contains a detailed account of the impact of GPS rollover on autonomous instrument platforms in Antarctica.

It provides a useful example of how modern data acquisition and communication systems can be rendered inoperative, and the

difficulties involved in restoring functionality. Altogether, this is a useful contribution and should be published.

I would suggest several very minor edits as indicated below. Following these, I have some general notes and questions which

might help clarify some aspects of the paper, but would accept the existing manuscript without requiring any substantive

changes (other than the minor edits).

==== Minor edits ===

>>line 35 "therefore the GPS week number rollovers every 1024 weeks"

...number rolls over every...

>>line 41 "Although the 2019 GPS rollover was warned of by Department of Homeland Security, the International Civil Aviation Organization and other relevant organizations beforehand,"

Despite advance warnings about the 2019 GPS rollover from the Department of Homeland Security, the International Civil Aviation Organization, and other relevant organizations ..."

>>line 42 "numerous" & "numerous"

roughly how many?

>>line 58 "one for the equipment cargo and another for the field team to put in."

...another for the $N$ member field team...

>>line 60 maybe give a table with # of flights per site per year?

>>line 69 "In the polar region, the available sunlight is limited throughout the year"

...limited during Southern summer as well?

>>line 81 "Thus, we use this fault as an example to describe best practices for designing future systems against similar

faults, or to at least enable graceful fault recovery and restorative action if a fault occurs."

...designing against similar faults... ==>> ...protection from similar faults...

>>line 85 "the remote instrument platform operated by the authors"

...the AAL-PIP...

>>line 89 inconsistent past/present tense?

>>line 114 "Up to this point 0 of the 5 RUDICS enabled stations"

...none of the five...

>>line 152 "land at PG0"

Station coordinates for PG2 provided in Figure 1, so should probably give PG0 lat/lon here.

>>line 163

Is PG0 the only one able to use Iridium? Clarify.

>>line 188 "all but one of the systems have come out of hibernation and each system has undergone a software update. The AAL-PIP array

has thus far been restored to full operational capacity."

N-1 of N is full capacity?

>>line 267 "John Bowman, and undergraduate student"

...an undergraduate student...

>> Figure 2 What does the dotted line indicate?

Couldn't this be reduced to a single figure just by adding 2nd line pointing down from Iridium to System clock.

=== General comments ===

Although there are some general lessons to be learned from this episode, the specific problems arose from a

particular combination of hardware and software. It might be helpful to include some technical details ie.

>>line 164 "GPS module" make/model/interface?

and possibly software versions eg. nptd

The complicated sequence of events is presented over several pages, so it might be useful

to provide some summary of what happens when. Perhaps something simple like this:

--------------------------------------

Typical state transition

0.0 system power on

0.0 SBC: start boot Linux

0.0 Iridium: boot local antenna, connect to SV

0.0 GPS: start boot local antenna

1.0 Linux: NTPD startup, initialize with SBC hardware clock time

2.0 GPS: acquire SVs, initial location & time estimate

3.0 GPS: RS232 available

4.0 Linux: initialize (software?) watchdog with SBC hardware clock time(?)

5.0 Linux: connect to GPS RS232, receive NMEA time, add source to NTPD

6.0 Linux: update NTPD with GPS time **possibly wrapped**

6.1 Linux: write NTPD time to SBC hardware clock

6.2 Linux: check software watchdog?

7.0 Iridium: read time from satellite

7.1 Iridium: RS232 available

7.2 Linux: connect to Iridium network

...and so on...

------------------------------------------

And centralize just a bit more detail about each of the elements in Figure 2 with their critical interactions eg.

SBC hardware - read by Linux software; written to on shutdown?

Linux software - NTPD(?) version

GPS hardware (make/model/interface=NMEA?) - default read by ntpd

Iridium satellite network - read?

Iridium local terminal - no read; no write?

then use these labels (SBC, NTPD, GPS etc.) to refer to different time sources consistently throughout the manuscript.

For example, "system time" and "system clock time" and "reported time"; are these all NTPD time?

Again- the manuscript is certainly acceptable as is, but could possibly benefit from some relatively minor modifications.

Citation: https://doi.org/10.5194/gi-2020-47-RC2
- AC2: 'Reply on RC2', Shane Coyle, 09 May 2021
  
  The authors thank the reviewer for their comments, and have corrected the minor typos as suggested. We hope to provide here some clarification in response to the reviewer's more detailed comments.
  RE: Line 42, "numerous" flights being grounded/delayed,
  
  - At least 30 flights were impacted, but official numbers were never released.
  RE: Line 60, the number of flights per site per year,
  
  - The sites do not recieve yearly visits by design, and so caches are only laid on demand and as determined by availability.
  RE: Line 69, the availability of sunlight,
  
  - There are ocassionally noted dropouts in sunlight likely due to weather, but on average at such high latitudes we have near constant sunlight during the southern summer.
  RE: Line 163, the Iridium network and PGs,
  
  - All of the sytems utilize the Iridium network for communications. PG1 uses a P2P setup, but the other stations, including PG0, utilize the RUDICS network.
  RE: Line 188, full capacity,
  
  - At the time of writing, it would not have been expected that the remaining system would have come out of hibernation. Based on such expectation and the success of the SW patch, the array was operating at it's expected operational capacity.
  RE: Figure 2 (previously), the subsystem diagram,
  
  - The difference in the figures is more than just an aditional line from Iridium to the system clock. The hardware clock now feeds the watchdog in stead of the system clock as well.
  RE: Aditional details and technical information,
  
  - We appreciate the desire for more detailed technical information regarding the AAL-PIP design and operation, specifically as it relates to the fault condition. Much of this information is available in previous work referenced within the manuscript. To answer the question of which GPS module was at fault, the AAL-PIPs described use the Garmin GPS 15xH. However, it appears that the rollover issue has been addressed in later firmware revisions of this module. In fact, one system was unaffected by the rollover because it wasn't built until several years after the initial deployment and thus benefited from the newer firmware.
  The systems as designed are unable to make use of NTPD, as the GPS module is only toggled for 5 minutes once every hour. Similarly, Iridium is only toggled for 5 minutes at a time every half hour. The method by which clocks are adjusted on AAL-PIP is by using the Linux command "adjtimex".
  Though we could provide great technical detail for this fault, it remains the intent of the authors to provide a high level summary in order to focus more on general design guidelines. However, because the sequence of timing is critical to the occurence of the fault, we have included a new figure sumarizing the process of clock synchronization on AAL-PIP (new Figure 2, attached).
  CAPTION: Diagram showing the process of updating the multiple clocks on an AAL-PIP. After the Linux OS finishes initialization, the AAL-PIP software sets the watchdog (WD) timer and spawns separate processes for communication and GPS synchronization. In this diagram, SYS(0) refers to the value of the system clock after boot, whereas SYS(1) is the system clock value after GPS sync. The fault occurs in block three, where the system clock SYS(1) is set to the erroneously offset GPS time.
  
  Citation: https://doi.org/10.5194/gi-2020-47-AC2

Peer review completion

AR: Author's response | RR: Referee report | ED: Editor decision | EF: Editorial file upload

AR by Shane Coyle on behalf of the Authors (09 May 2021) Author's response Author's tracked changes Manuscript

ED: Publish subject to minor revisions (review by editor) (21 May 2021) by Takehiko Satoh

AR by Shane Coyle on behalf of the Authors (01 Jun 2021) Author's response Manuscript

ED: Publish subject to minor revisions (review by editor) (04 Jun 2021) by Takehiko Satoh

AR by Shane Coyle on behalf of the Authors (07 Jun 2021) Author's response Manuscript

EF by Sarah Buchmann (08 Jun 2021) Author's tracked changes

ED: Publish as is (09 Jun 2021) by Takehiko Satoh

AR by Shane Coyle on behalf of the Authors (17 Jun 2021)

Short summary

Global satellite navigation systems are commonly used for timing and synchronization of instrument platforms. These system clocks periodically roll over from limitations in discrete counter math. Due to the rarity of these events (19.6 years for GPS), special consideration must be given to designing instruments intended for use in hard-to-reach locations like the Antarctic Plateau. A few best practices are presented to prevent total system failure from unexpected subsystem faults.