TEL AVIV—Well before Hamas launched its Oct. 7, 2023, attacks, Israel’s military had plenty of evidence that something was brewing.

Israel had been in possession of a secret Hamas plan for a mass invasion for more than a year. Soldiers on the border of Gaza had observed Hamas practicing raids on Israeli military bases and civilian communities for weeks. And the country’s security chiefs had been warning that months of contentious internal debate and protests over political issues had left Israel vulnerable.

The night before the attack, the Shin Bet, Israel’s domestic intelligence agency, learned that dozens to hundreds of Hamas members had activated mobile phone service on Israel’s networks, a strong signal they planned to be in Israel soon. As the night progressed, Israel’s spies picked up signals that some of Hamas’s leadership had gone underground and that groups of Hamas commandos had begun gathering in spots around the Gaza Strip.

Israel’s vaunted intelligence services debated what it all meant until deep in the night and decided to reassess in the morning. Around dawn, Hamas attacked essentially as outlined in the plan captured a year earlier, leaving roughly 1,200 dead and 250 taken as hostages to Gaza.

Israel detailed its failings in a report on the attack released by its military this week. Along with the discrete mistakes was a big-picture intelligence blunder of a type that has repeated itself regularly throughout history. They are so common that scholars have a name for them—“strategic intelligence failures”—an inability to see the forest for the trees.

They are easy to spot in hindsight but stubbornly hard to guard against as they are unfolding. They stem from the fact that intelligence is as much art as science. Operatives take pieces of information and form theories that can be hard to shake. Layer on intentional misdirection and ambiguous developments—is a big military mobilization a negotiating tactic or a prelude to war?—and what might later have seemed obvious can seem very complicated in the moment.

In that complex matrix, evidence that contradicts the prevailing assessment among intelligence officials and decision makers often gets dismissed.

“Part of the theory of strategic surprise is that even good intelligence is hard to use for decision makers,” said John Ferris, a history professor at the University of Calgary who studies intelligence and its failings. “If good intelligence tells you what you think won’t happen, you just won’t use it.”

In the case of the Oct. 7 attacks, Israel was under the impression that Hamas was angling for economic concessions by stirring up tension on the border. The group frequently used protests and threats of attack to agitate for greater access to work permits for Gazans—or so Israel thought. In fact, Hamas was preparing for an attack in plain sight. Yet no senior official thought the U.S.-designated terrorist group, significantly weaker than Israel’s military, actually wanted a full-scale war.

Almost exactly 50 years earlier, Israel had made a similar catastrophic blunder.

​Egypt and Syria were mobilizing forces on its border. The Arab states declared their intention to win back territory Israel took from them six years earlier in a decisive victory. Israel had received numerous warnings that war was imminent. This includes information from a top Egyptian official who warned Israel’s spy agency, the Mossad, on Oct. 5, 1973, that Syria and Egypt would invade the next day.

Yet, right up until just before war broke out, as warned, on Oct. 6​, 1973, Israel thought its enemies were bluffing and was caught flat-footed. The warnings of the Egyptian spy were dismissed as a double agent trying to spread misinformation. No way, thought Israeli intelligence at the time, did its neighbors want war again so soon after a crushing defeat in 1967.

Three decades earlier, that error in judgment had been preceded by an even bigger one involving the U.S. Before the Japanese dealt a harsh blow to the U.S. Pacific Fleet in Pearl Harbor on Dec. 7, 1941, Washington had clear signs of an impending attack.

The U.S. and Japan had been engaged in peace talks. The day before the attack, U.S. intelligence decrypted Japanese messages that indicated they were through with the talks. Weeks earlier, the Japanese fleet had disappeared into the Pacific Ocean. The U.S. Army and Navy sounded warnings of possible hostilities.

U.N. troops guard tents where Israeli and Egyptian chiefs of staff are meeting to discuss the separation of their respective forces, at KM 101 on the Suez-Cairo highway, Jan. 20, 1974. The chiefs of staff are Lt. Gen. David Elazar of Israel, and Lt. Gen. Mohammed Gamasy of Egypt. (AP Photo/Max Nash)

Yet, said Ferris, the U.S. top brass didn’t imagine Japan was capable of or interested in war with America and failed to react effectively. Japan was focused on battles closer to its borders, American leaders believed. Even a final opportunity to prepare for the attack—when American radar systems picked up 183 aircraft 137 miles off the Hawaiian coast about an hour before they struck—was mishandled.

After the war, Gen. Sherman Miles, the assistant chief of staff, summed up the intelligence failure. “We had a yardstick. We had no reason to doubt our yardstick’s approximate accuracy. Yet it was wholly false,” he said.

About six months earlier, the Soviet Union had made its own blunder. Soviet intelligence had given Joseph Stalin clear evidence that Nazi Germany, purportedly allied with Moscow, was preparing a massive invasion. But historians say Stalin failed to mobilize his army to defend against the onslaught.

Ferris said Oct. 7 was among the most egregious strategic intelligence failures because of how absolutely Israel was taken by surprise and how utterly it had failed to take any precautions for a mass attack from Gaza.

Israeli military officials said the failure to prepare for such a worst-case scenario stemmed from their mistaken belief that their intelligence apparatus was so good that they knew what would happen.

“We were addicted to the precise intel information. The addiction is thinking you know everything,” said one of the officers who presented the findings of the Oct. 7 investigation to reporters. “Sometimes you think you know everything, and by being addicted to that, we thought we would have sufficient understanding of what the enemy will do, what and when.”

Israeli intelligence officials said they also fostered a culture in which opinions contrary to the mainstream assessment were easily dismissed—despite the fact that half a century earlier, they had sought to solve this exact problem.

Following the 1973 intelligence debacle, teams were created inside Israeli military intelligence and later in the Shin Bet who were supposed to challenge mainstream assessments. The “devil’s advocate” unit was small but made of rising stars inside military intelligence’s research division, the organization chiefly responsible for early warnings of war, said Yossi Kuperwasser, who once ran the division.

Over time, as the sting of the 1973 failure subsided, the team stopped drawing from the elite, and the rank of its leader was lowered from a full colonel to a lieutenant colonel, Kuperwasser said.

A bigger problem was built in, said Uri Bar-Joseph, a professor at Israel’s Haifa University and a scholar of intelligence failures. Over time, people got in the habit of dismissing the opinions of the devil’s advocates precisely because they always took the opposing view by design. Created as a fail-safe, it became just another part of the bureaucracy.

“In the beginning, you take it seriously,” Bar-Joseph said. “But no one takes it seriously after a few years.”

Israel’s intelligence services have redeemed themselves in the eyes of the country’s leaders over the past year after precisely targeted attacks on Iran and the rapid dismantling of the Lebanese militia Hezbollah .

But the country is no longer content to rely on early warning. Israel has carved out a deep buffer along its border with Gaza; it is holding high ground in southern Lebanon after razing Hezbollah infrastructure there; it plans to remain indefinitely in a buffer zone inside Syria across from the Golan Heights; and it is insisting that a swath of southern Syria remain disarmed.

“It is not viable to ‘conflict manage’ against an enemy whose goal is your destruction,” the military said in its report on the Oct. 7 failures.

Write to Dov Lieber at dov.lieber@wsj.com