Μake us preferred on Google The Federal Aviation Administration first noticed glitches in communications at its Indianapolis center that oversees high-altitude flights over a swath of middle America.
Quickly they spilled into communications between pilots and air-traffic control. There was static and interruptions. Pilots had questions.
Initially FAA employees tasked with troubleshooting the problem— one that wound up grounding much of U.S. travel —started their hunt in Indianapolis, according to people familiar with the situation.
But then, they zeroed in on two locations: an FAA radio site in El Dorado, Ark.; and a radar site in Amarillo, Texas, the people said.
Network gear at the Arkansas site, and possibly the one in Texas, overloaded the communications system that thousands of air-traffic controllers use to guide pilots through the nation’s airspace.
“We saw two places that were actually pinging our network at a very high rate, a thousand connections over every four to five seconds,” Rebecca Guy, the FAA’s technology chief, said at a Saturday press conference.
These pings traversed a tangle of copper wires and fiber-optic cables that feed into the FAA Telecommunications Infrastructure, or FTI. The decades-old system is outdated and undergoing a multibillion-dollar upgrade to move into the fiber-optic age.
Defense contractor L3Harris Technologies oversees the system, which includes key connections for radar, communications and flight data. “L3Harris takes the nationwide ground stop and its impact on travelers, airlines and air traffic operations seriously,” the company said in a statement.
The system has suffered repeated failures in recent years as officials struggle to replace it with a modernized network and newer routing hardware. But efforts to maintain and replace the mission-critical network have taken years and become more expensive over time.
The risks that FTI failures posed to the FAA’s air-traffic system are well-known. A 2023 outside review warned that “system risk could quickly reach unacceptable levels” due to increased outages and inadequate funding for overhauls.
FAA Administrator Bryan Bedford said the agency spends hundreds of millions of dollars annually maintaining its antiquated telecom systems.
Generally changes to the system—like switching out copper wires for fiber—in busy areas are carried out overnight. Making updates in more remote locations during the day shouldn’t affect many flights, or so officials thought. Transportation Secretary Sean Duffy said officials would re-evaluate when they do such work.
Not long after L3Harris installed a new router made by HPE’s Juniper Networks at the Arkansas location, the machine began sending out a high volume of data Friday afternoon.
Data from that router and another at the Texas location overwhelmed the system, government and industry officials said. It led to 17 out of the FAA’s 22 centers that oversee high-altitude flying to experience disruptions in communications, Bedford said, resulting in the FAA ordering the ground stop.
There are about 60 Juniper routers active and operating normally in the FAA’s air-traffic network, said a government official familiar with the system. It wasn’t clear if the problems Friday resulted from a bad router or if L3Harris didn’t configure it properly.
“Was it a faulty installation? Was it a faulty configuration? Or was it just a bad piece of router technology that we put into this system that propagated these errors?” Bedford said Saturday, noting the FAA is still investigating the situation.
An HPE spokesman said the Juniper devices appeared to have been working as intended. “They did not fail,” he said.
FAA officials identified a feature on the router that appeared to be resulting in the data overload, according to government and industry officials. A government official described it as a broadcasting feature that sent data more broadly than necessary, and said the agency has since disabled it on all the Juniper routers.
As passengers rushed to rebook flights, book rental cars and reserve hotel rooms, the FAA was trying to diagnose the problem.
When the FAA cut off the routers in Arkansas and Texas from the broader network, the communication problems stopped. “We disabled both of those right away and were able to stabilize our network,” said Guy, the FAA’s technology chief.
What the FAA still doesn’t know is why the router went rogue. Duffy said they have found no evidence to suggest nefarious causes and instead are zeroing in on a mechanical problem with the router itself or “incompetence” by L3Harris.
An L3Harris spokeswoman said the contractor is supporting the agency’s probe into what went wrong.
Duffy said that until all the copper wires are replaced at the end of 2028, more disruptions could come.
“This was an FAA system that didn’t meet the moment. Our contractor didn’t meet the moment. Or the equipment didn’t meet the moment,” he said.
Write to Andrew Tangel at andrew.tangel@wsj.com and Drew FitzGerald at andrew.fitzgerald@wsj.com
