ISLAMABAD, Pakistan—Pakistan carried out airstrikes early Thursday inside Iranian territory, in retaliation for an Iranian airstrike in Pakistan on Tuesday that had targeted Iranian insurgents, Pakistani officials said.

The two nations were careful to say that they had only targeted their own nationals in the tit-for-tat strikes, an indication that neither country wants the situation to spiral, experts said. But risks of a miscalculation remain amid heightened tensions in the Middle East in the wake of the Israel-Hamas war.

Pakistan, which has had an uneasy but not hostile relationship with Iran in recent decades, is getting hit by the wider shock waves in the region.

Tehran is trying to show strength as it is in an indirect confrontation with the U.S. and Israel in the wake of Israel’s war in Gaza, experts said. Pakistan got sucked into that demonstration of Iran’s resolve, which also saw Tehran hit targets in Syria and Iraq this week as it flexed its regional muscles, they added.

Pakistan’s Ministry of Foreign Affairs described the bombardment as “precision military strikes against terrorist hideouts” in Iran’s bordering province of Sistan-Baluchistan. It said that the action was taken in light of credible intelligence of impending large-scale terrorist activity.

A Pakistani security official said 20 “terrorists” were killed, according to initial estimates. Both aircraft and missiles were used for the bombardment, no more than 31 miles inside Iran, which hit separatists based there from Pakistan’s west, the official said.

“This action is a manifestation of Pakistan’s unflinching resolve to protect and defend its national security against all threats,” said a statement from Pakistan’s foreign ministry.

The move was a response to Iran’s surprise airstrike inside Pakistani territory Tuesday, Pakistani officials said. In that strike, Tehran said that Iranian jihadists were hit, but Pakistan said that two children were killed and three women injured in a remote part of the country bordering Iran.

An Iranian official protested Thursday’s strike, according to the Fars news agency, which is close to the country’s security forces. “Iran demands an immediate explanation from the Pakistani authorities about this incident,” the agency quoted the official saying.

But Tehran also appeared to play down the impact of the strikes, saying that those killed weren’t Iranian.

Seven non-Iranian civilians—three women and four children—were killed in Pakistani strikes on an Iranian village close to the border with Pakistan, local Deputy Governor Alireza Marhamati said, according to Iranian state media.

Both Islamabad and Tehran have long accused each other of harboring militants. The Pakistani airstrike was against Pakistani separatists it said were present inside Iran.

Iran is closely allied to the regimes in Iraq and Syria and its airstrikes there this week didn’t provoke a military response. But Pakistan is a nuclear-armed state with a large army, making it a far riskier target. Islamabad also needs to show it can’t be bullied, as its main adversary is its giant neighbor to the east, India, which is also nuclear-armed.

“Pakistan’s retaliation appears to have been proportionate to Iran’s earlier strike, and notably it targeted only militants and not Iranian security forces,” said Michael Kugelman, South Asia Institute Director at the Wilson Center, a think tank in Washington. “With both sides even, so to speak, this provides an opening for de-escalation, if cooler heads prevail. But that’s a big if.”

Pakistan indicated that it doesn’t seek further conflict, saying it respects the sovereignty and territorial integrity of Iran.

“Iran is a brotherly country,” said Pakistan’s foreign ministry. “We have always emphasized dialogue and cooperation in confronting common challenges including the menace of terrorism and will continue to endeavor to find joint solutions.”

Ethnic Baloch insurgents frequently attack Pakistani security forces in the remote west of the country, and they have also targeted the Chinese presence in Pakistan. Those insurgents were the target Thursday inside Iran, including a group called the Baloch Liberation Army, according to the Pakistani security official.

—Benoit Faucon contributed to this article.

Write to Saeed Shah at saeed.shah@wsj.com