Telstra outage: The evening a community determined the yr was 2006
The reverse is definitely the case. If we do not need a typical understanding of what “now” is, a whole lot of issues we take with no consideration will cease working.
This summer time, Australia realized this the exhausting approach. Let me take this chance to present an instance of why time issues to a contemporary society, what occurred on this explicit case, and what our key takeaways are.
On 8 July 2026, a big a part of the cellular community run by Australia’s largest mobile phone operator Telstra stopped working with voice calls not getting by means of or textual content messages that did not arrive. There had been even calls to Australia’s emergency quantity that didn’t get by means of. But the outage affected methods even additional aside, corresponding to trains, cost terminals, ticketing methods and EV chargers being disrupted.
No networks had been attacked. Nobody by chance lower the fiber. All methods had electrical energy. The perpetrator, it’s possible you’ll ask? A single GPS receiver in a single chassis in Melbourne getting back from scheduled upkeep believing the yr was 2006, and the remainder of the community was persuaded to imagine it.
Telstra commissioned an independent review from the company Technology Audit Partners (TAP) and the report is a really attention-grabbing learn, as a result of the identical sort of failure may seem in lots of vital providers, together with some we rely on to maintain folks alive.
In order for a lot of of those methods to perform, time being appropriate, or not less than the identical all over the place, is essential. And as everybody within the corporate affairs is aware of, “appropriate” is a relative time period. There is not any actual time, solely time held inside a sure margin of a reference. How vast that margin could also be relies upon fully on what you’re doing or through which corporate affairs you use in.
Running a cellular community, like Telstra, is about as time-dependent as a corporate affairs will get.
Modern cellular communication explains why time issues
Modern mobile phone protocols is not going to work with out precision time. Mobile networks separate uplink knowledge from downlink by both FDD (Frequency Division Duplex) or TDD (Time Division Duplex).
FDD offers every course its personal slice of spectrum, so each can run constantly with out colliding.
TDD as an alternative makes use of a complete single block of spectrum for each instructions, alternating between transmitting and receiving in very quick intervals.
Since much more knowledge normally flows down than up, FDD’s fastened ratios go away a lot of the uplink spectrum idle, whereas TDD can shift the ratio to match the precise visitors. That is why most trendy 5G spectrum, together with Sweden’s principal 5G band at 3.5 GHz, is TDD.
It can also be why TDD is determined by correct time: each cell on the identical frequency has to change course in line with each different. A cell that lets its clock drift will transmit knowledge into its neighbour’s obtain window, with the consequence that the community will begin jamming itself.
Rather than allocating spectrum on maintaining the 2 instructions aside, the business selected to depend on time accuracy, and accepted a tough dependency on each cell agreeing about when “now” is. Thus, being depending on time is a design alternative.
However, given how a lot all of us rely upon the methods with the ability to agree on “now”, it’s considerably puzzling that point isn’t given as a lot consideration because it deserves. And that may be a lesson that may be very clear from the printed report.
So, what actually occurred?
Architecture of time distribution.
To start with, it’s vital to know the structure of time distribution.
Time distribution protocols all construct hierarchies; Network Time Protocol (NTP), which Telstra has deployed based on the report, expresses its hierarchy in strata.
- Stratum 0 is the reference itself, as an illustration a GPS receiver, or Netnod’s atomic clocks.
- Stratum 1 is a machine synchronised on to a stratum 0 reference, for instance the NTP servers that Netnod offers.
- Stratum 2 synchronises from a stratum 1 server, stratum 3 from a stratum 2, and so forth.
In Telstra’s case, that hierarchy had a selected form, not less than to start with. This design from 2010 had on the prime stratum 1 sources at Australia’s National Measurement Institute (NMI), which maintains the nation’s nationwide time scale, a lot because the Research Institute of Sweden does in Sweden. Telstra drew time from these exterior references into two stratum 2 servers of its personal, in Sydney and Melbourne, which in flip fed three stratum 3 servers, in Sydney, Melbourne and Perth.
Below them sat the shoppers. In this context that doesn’t imply laptops or telephones, however all the cellular community utilities, as an illustration nodes dealing with handovers between cell websites. There had been hundreds of nodes throughout an enormous geography and each one in all them wanted to have the identical thought of what “now” is, to inside just a few millionths of a second.
The TAP report describes this setup as “match for objective” and that it gave Telstra “a extremely dependable and authoritative reference time supply from NMI”.
Stratum in itself doesn’t say if the time is correct, solely the variety of steps from a server to its reference. A stratum 1 server with a nasty time reference remains to be a stratum 1 server.
Protection in opposition to unhealthy time sources
NTP will due to this fact want defence in opposition to unhealthy time sources. In reality, it has two completely different ones, they usually do various things, each of which assume they’re unbiased from one another.
- Among in any other case comparable candidates, the decrease stratum carries extra weight. This is the mechanism that determines which supply a consumer settles on.
- NTP compares a number of sources and discards those who disagree with the remaining. A single supply claiming an implausible time is outvoted and dropped, no matter how authoritative it claims to be.
Neither defence is particular to any explicit disruption; collectively they shield in opposition to a damaged receiver, a misconfigured server, or an exterior assault. But these protecting measures solely work if the time sources that the shoppers take heed to are genuinely unbiased of one another.
Two methods to deploy NTP
NTP might be deployed in two methods. In consumer/server mode the connection is said and directional: a node takes time from these servers, and nothing else.
The 2010 Telstra setup was in actuality such a consumer/server mannequin. Peering was allowed, however solely on the similar stratum stage and the TAP report, as famous at first, described this setup as “match for objective”.
The different approach is a symmetric (peering) mode, the place nodes change time mutually and decide on whichever supply the algorithms at the moment favour.
Peering is versatile and survives the lack of a supply gracefully. But it additionally means the topology in manufacturing is emergent quite than designed. What you documented is a setup that might quietly rearrange itself right into a form nobody ever authorised.
Telstra’s 2020 improve
In 2020 the cellular core timing system was upgraded, and new {hardware} was put in, together with a brand new NTP timing chassis. That set up launched just a few modifications.
The first one was compelled. The new chassis couldn’t let a stratum 2 server feed a stratum 3 server inside the identical field, so the 2 needed to be wired throughout one another: Sydney’s stratum 3 took its time from Melbourne’s stratum 2, and Melbourne’s stratum 3 from Sydney’s.
In actuality, as an alternative of getting two stratum 2 sources, every website was left with just one. The TAP report notes that this degradation in redundancy was identified and accepted. A second change was leaving the consumer/server-model in favour of the peering mannequin. The report isn’t clear in regards to the motivation, however it’s cheap to recommend that one wished compensation for this lack of redundancy. With every website having only one supply as an alternative of two, letting the servers discover their very own replacements may give the impression of higher resilience.
The TAP report clearly states that the lack of resilience was identified. However, it fails to search out any proof that the ensuing danger of so-called “timing loops” was recognized.
What is a timing loop?
A timing loop is the community equal of believing a hearsay to be true by asking three individuals who all heard it from one another. Each one agrees, so it have to be true. NTP works mainly the identical approach: it compares a number of sources and discards whichever disagrees with the remaining.
As it’s possible you’ll recall from above, NTP has two defenses in opposition to unhealthy time sources. The second one protects in opposition to timing loops, however provided that the sources are unbiased of one another. In such a loop, sources that seem unbiased are in truth taking their time from one another, both straight or not directly by tracing again by means of a shared reference.
Once a unsuitable worth is circulating, the sources will begin agreeing with one another and the vote shall be in favour of the bulk’s opinion, although the worth is unsuitable.
The protocol labored. The structure didn’t.
Both of NTP’s sorts of defenses got here to be disabled in Melbourne, however 5 years aside. Not intentionally, however by selections, every of them defensible on their very own phrases: a {hardware} limitation needed to be labored round, and later, a recurring fault needed to be stopped. Each resolution solved the issue in entrance of it. Nobody was requested to take a look at the sum of all actions.
The second defence was the primary one to be disabled. The introduction of peering in 2020 made timing loops attainable, and 5 years later, such a loop confirmed up. In Melbourne a server began taking time from a node beneath itself. That ought to have set off alarm bells. However, since correct time was nonetheless reaching the community by different paths, no actual hurt was accomplished. The underlying drawback, the round dependency, was there, however nobody issued a ticket about it.
The precise criticism was fairly apparent. Melbourne saved shedding contact with its solely stratum 2 supply in Sydney. With no fallback configuration, the server used peering to discover a substitute, generally a node beneath it within the hierarchy.
In October 2025, engineers activated the GPS receiver that had been sitting unused within the Melbourne chassis since 2020 and related it to the stratum 3 server, as a substitute for the unreliable Sydney supply.
By each seen measure it appeared to have labored. Melbourne now had a dependable supply of its personal and the alarms stopped. But the repair solely addressed the symptom, not the foundation trigger. Nobody established why Melbourne saved shedding its Sydney supply within the first place. The underlying drawback was nonetheless current within the community by July 2026.
To make issues even worse, no one appears to have understood what activating the GPS card did to the structure. By including the GPS card, the Melbourne server went from a stratum 3 server to stratum 1. The engineers didn’t add a supply subsequent to the opposite ones. By selling a server to the identical rank because the nationwide reference, a brand new supply was created on the very prime. As far as NTP is anxious, they carry the identical weight.
Suddenly this GPS card in a chassis in Melbourne, put in as a workaround and reviewed by nobody, turned probably the most authoritative server within the hierarchy for the most important cellular community in Australia.
Needless to say, nearly nothing of the 2010 design remained.
By July 2026 the community had a single supply that was each probably the most authoritative candidate accessible and unopposed, as a result of the sources that might have contradicted it had been downstream of it.
This behaviour was very tough to identify. The community served correct time daily for years. Architectures like this don’t normally degrade steadily. They work, they usually hold working, proper up till they cease.
GPS week quantity rollover
The second ingredient is a widely known property of GPS.
GPS broadcasts time as every week quantity plus seconds-into-week, counted from an epoch that started in early January 1980. In the principle civil GPS sign, the week quantity area is 10 bits, i.e. a most of 1,023 weeks. Every 1,024 weeks, or 19.6 years, the counter begins over. This has occurred twice: in August 1999 and in April 2019.
Working out which variety of epoch it’s and including the suitable a number of of 1,024 weeks, is the job of the receiver. And the knowledge must be in its firmware.
The drawback that occurred in Australia was not a late consequence of any of the GPS rollovers. The card in Melbourne had handed by means of the second rollover in 2019 with out hassle, as a result of a receiver that retains operating additionally retains counting. Each new week is solely added to the one earlier than, and the query of which epoch it belongs to isn’t raised.
However, when you flip it off, that information is gone. When it’s turned on once more, the receiver has to work out the epoch from scratch, and all it has to go on is what its firmware assumes. The firmware on the Melbourne card had not been up to date. Upon start-up it fell again on the sooner epoch and positioned the date 1,024 weeks previously.
What occurred subsequent is finest understood as the 2 defences being disabled once they had been wanted probably the most.
The first defence, the decrease stratum carrying extra weight, ranked the Melbourne server highest, as a result of the hooked up GPS card promoted it to a stratum 1 server. This was based on NTP protocol and thus steered shoppers in direction of the one supply which was 1,024 weeks unsuitable.
The second defence, outliers being voted down, was by no means engaged, as a result of nothing was left to establish Melbourne as an outlier. NTP doesn’t ask whether or not a date is believable; it asks whether or not a supply disagrees with the others. The 2010 setup had two stratum 2 servers. If one in all them had began asserting the yr 2006, the opposite one would have stayed with 2026 and no consensus would have been reached. That wouldn’t have been ideally suited, however not less than the unsuitable date wouldn’t have unfold.
But Melbourne’s stratum 2 counterpart had been switched off by the exact same chassis substitute, and the remaining sources had been downstream of Melbourne. As the unsuitable date unfold, they started reporting it again. Agreement grew, and settlement is what the algorithm is in search of.
So the shoppers did what they had been constructed to do. Once a majority of a consumer’s sources agreed on November 2006, the consumer accepted the date, and the additional the date travelled, the extra convincing it turned.
Neither defence malfunctioned. Both had merely been disadvantaged of what they rely upon: one wanted a supply value rating highest, the opposite wanted sources able to disagreeing. Two selections, 5 years aside, had eliminated every in flip.
Key takeaways from the incident
Prioritise and classify time and frequency distribution as vital utilities. Manage it accordingly
Document all capabilities that may take the entire community with them, and put timing on that listing. Classification isn’t paperwork; it’s what determines change danger class, overview depth, staffing ranges, monitoring protection and funds precedence. Telstra’s report is, at backside, the story of 1 lacking entry on that listing and all the pieces that adopted from it.
Document the entire utilities, and each change to it
There was no central repository of NTP configuration, no golden configuration, and no documented file of the servers aside from the units themselves. Without data you can’t carry out significant pre-checks, you can’t assess influence, and through an incident you can’t inform what “appropriate” appears to be like like.
Build redundancy in competence
Two engineers carried out the change, and each had been on obligatory stand-down earlier than the implications of the GPS card reboot had been understood.
Depth of experience is a resilience property precisely like a redundant energy feed. A single specialist, or a pair, means no second opinion, and nobody to ask in the midst of the evening when upkeep is normally accomplished.
Run a safety evaluation of the time and frequency utilities
Treat timing as an assault floor like another and analyse it accordingly.
Start with the place time enters the organisation. A GNSS sign arriving from house is weak and unauthenticated, and might be jammed or spoofed by low cost tools. If that sign is your solely reference, somebody exterior your constructing can determine what time you assume it’s.
Then take a look at the way it travels. Time distributed over a shared community might be intercepted and manipulated on its option to the consumer.
Then take a look at who’s allowed to talk. Which servers might your shoppers settle for time from, and who determined that? A supply that no one authorised is a supply no one is checking.
And don’t cease at deliberate assault. A timing loop produces a lot the identical impact as a profitable spoofing assault: a supply the community trusts, delivering a worth no one can contradict.
Use point-to-point connections
It is straightforward to see the attraction of peering. It seems like resilience with sources that again one another up: a community that heals itself when a node disappears. But redundancy that arranges itself isn’t redundancy you may depend on.
There are safer methods to realize the same stage of robustness. Netnod runs devoted point-to-point connections: each relationship is understood and documented. Every supply is understood, and the topology stays the way in which we designed it. Redundancy comes from a number of unbiased sources intentionally configured, not from nodes negotiating amongst themselves.
Build an efficient alarm organisation
Alarms from the timing platform weren’t in the usual monitoring instruments, and had been reviewed solely throughout corporate affairs hours by a handful of individuals. Client-side alarms carried neither the severity nor the element to drive instant motion. Getting this proper is organisational as a lot as technical: alarms attain 24×7 monitoring, severities replicate actual consequence, every alarm carries an instruction for what to do about it, and somebody owns the response. An alarm nobody is on name for is documentation at finest, not detection.
Use golden installations
For each class of timing system, hold a known-good reference construct and configuration below model management, and examine frequently and routinely that what’s deployed nonetheless matches it. The level is to show a query like “is that this chassis accurately configured and patched?” from one thing solely an skilled can reply, and solely slowly, right into a comparability anybody can run in seconds.
Telstra had nothing of the kind. The TAP report discovered no such configuration and no file of what the servers ought to appear like aside from the servers themselves. The lacking firmware replace on the Melbourne GPS card had been there for six years, in plain sight. There was merely no automated course of that will have flagged it to anybody.
Upgrade and consider software program constantly
The firmware repair for the rollover behaviour existed and the seller had printed bulletins about it. Vendor notifications want an outlined proprietor and a tracked path to motion, and updates have to be utilized on a schedule quite than when one thing forces the difficulty.
Evaluate earlier than deploying, in a lab, in opposition to the behaviour you really rely upon. Do not overlook to confirm afterwards. The Telstra modifications had been accomplished with out anybody checking that the chassis served the right date.
Redundancy
Redundancy in timing means, not solely a number of sources, however unbiased ones that can’t converge on a typical error. Two servers fed by the identical GNSS receiver remains to be one supply, however counted twice.
Netnod’s personal service is constructed on the precept of a number of autonomous nodes, every with unbiased atomic clocks and redundant servers which are traceable to Swedish National Time realization, UTC(SP).
Replace tools constantly
Timing utilities normally ages quietly. It retains working, it not often complains, and it’s due to this fact a pure candidate when the funds will get trimmed. There will all the time be different parts the place the implications of failure are extra seen and due to this fact will get prioritised.
Instead, plan substitute on a rolling cycle and design the goal structure first quite than accepting what new {hardware} installations impose on you. Keeping present utilities wholesome must be funded alongside new initiatives, not be paid for with the left overs.
Concluding remarks
The approach Telstra dealt with the aftermath deserves reward. Commissioning and publishing the unbiased overview is commendable. All suppliers of vital providers, together with Netnod, are higher off due to this.
The most unsettling half is probably that the outage occurred although NTP labored identical to it was meant to do.
The drawback was all the pieces round it. Architectural selections, funds cuts, low staffing stage, lack of correct monitoring, possession or documentation; all of it occurred as a result of nobody actually appreciated simply how very important time providers might be.
Let’s attempt to change that, we could?
