DOHA, Qatar—Qatar’s diplomacy in the Gaza war, including help in arranging a temporary cease-fire and hostage-release deal that took effect Friday, cements the ultrarich Muslim nation as Washington’s preferred interlocutor with extremist groups and pariah states—in the Middle East and increasingly around the world.

It is an improbable role that began to take shape some 30 years ago as the small Persian Gulf monarchy sought to secure itself in the midst of bigger neighbors by resolving regional disputes while winning the trust and gratitude of the U.S. and other Western governments. It has also hosted a major U.S. military base for two decades and purchased billions of dollars in arms from the U.S. and Europe.

The approach carries great peril, as Qatar’s willingness to talk to extremist groups has left it open to allegations from its neighbors and others that it supports terrorism, which it denies.

The past seven weeks of painstaking mediation, which Qatar launched hours after Hamas’s cross-border attack against Israel on Oct. 7, have again laid bare those tensions. Some U.S. lawmakers and former senior officials lambasted Qatar as a key Hamas backer, even as the Biden administration was pressing it to help secure the release of hundreds of abducted civilians and soldiers, including several American citizens.

Qatar opened a channel with Hamas leaders more than a decade ago, a step Qatari officials say came at the U.S.’s request. Qatar later allowed the group to open an office in Doha and provided hundreds of millions of dollars in aid for Gaza. Many in Israel are suspicious of Qatar’s relationship with the militants and fear it could thwart attempts to destroy the group.

Qatari officials said they have become used to having their motives and loyalties questioned over the years, but have only become more vocal in defending their position.

“The political leadership of Qatar is willing to take the risk” of maintaining contacts with parties that are shunned by the West, Majed Al Ansari, a Foreign Ministry spokesman and senior adviser to the country’s prime minister, said in a recent interview.

He added: “You can only have the high gains through taking the high risk, by doing these things.”

Qatar’s strategy put the Gulf state at especially high risk when Arab neighbors severed diplomatic and economic ties in 2017, with a green light from the Trump administration, and considered launching a ground invasion.

Saudi Arabia, Egypt and others had grown frustrated with Qatar’s independent foreign policy, including critical coverage by Al Jazeera television, which is based in Doha, and support for Muslim Brotherhood affiliates and revolutionary movements during the Arab Spring uprisings that toppled longtime dictators across the region.

The diplomatic rift and economic boycott ended after three years without extracting any meaningful concessions. Shaken but defiant, Qatar instead doubled down on efforts to mediate some of the world’s thorniest conflicts and cast itself as a neutral arbiter.

“The Qataris are going to do everything in their power to be indispensable to the United States. That’s the keystone of Qatar’s foreign policy,” said Patrick Theros, a former U.S. ambassador to Qatar. “That also means that they have to keep a visible distance from the United States sometimes, because then they can talk to the other guy.”

At the end of America’s 20-year war in Afghanistan, it was Qatar that hosted peace talks with the Taliban. The Islamist insurgents had opened an office in Doha in 2013 at the request of the U.S., which sought to reduce the Pakistani intelligence services’ influence over them.

When the Western-backed Kabul government collapsed in August 2021, Qatar helped evacuate tens of thousands of people from the country, including U.S. citizens and Afghans who had worked with the American military. It remains a key emissary to the Taliban, a U.S.-designated terrorist organization.

Following Russia’s invasion of Ukraine last year, Qatar maintained channels to the Kremlin that eventually allowed it to negotiate the return of children whom Russia had moved from Ukrainian territory it seized. Around the same time, it hosted U.S. talks with Venezuela about lifting sanctions in exchange for political changes.

Weeks before the Gaza war erupted, five Americans freed from prison in Iran landed in Doha on their way back to the U.S. as part of a Qatari-mediated deal that unfroze $6 billion in Iranian oil proceeds and aimed to restart nuclear talks. After last month’s Hamas attack on Israel, the U.S. and Qatar agreed to deny Iran access to the money in the midst of concern about Tehran’s longstanding funding of Hamas.

“Qatar is creating itself into a prickly Switzerland,” said David Roberts, the author of books about Qatar’s development and security policy in the Gulf, pointing to Doha’s effort to stay neutral while heavily arming itself against external threats.

The country, roughly the size of Connecticut, includes the largest American military base in the Middle East and over the years has purchased billions of dollars of weapons from the U.S., the U.K. and France. It built a national carrier, Qatar Airways, into one of the world’s largest long-haul carriers, and spent tens of billions of dollars to host last year’s FIFA soccer World Cup.

With a native population of around 300,000, Qatar wasn’t always the obvious choice for international mediation. In the early 1990s, it was a poor former British protectorate struggling to maintain its autonomy in the shadow of Saudi Arabia and Iran, having refused to join a confederation of other coastal emirates.

After Sheikh Hamad bin Khalifa Al Thani, the father of the current ruler, took over in a bloodless coup from his father, Qatar began exploiting its majority control of the world’s biggest natural-gas field. It funneled the resulting wealth into building a military base for U.S. troops that were pushed out of neighboring Saudi Arabia and launching Al Jazeera, a pan-Arab broadcaster that reported critically on the region.

Al Jazeera helped create Qatar’s maverick image but also repeatedly got it into hot water. During the Iraq War, President George W. Bush, in a meeting with British Prime Minister Tony Blair, reportedly contemplated bombing the Doha headquarters of the broadcaster, which was airing video of the fighting in the Iraqi city of Fallujah that the Pentagon said was misleading. The White House dismissed the report at the time, and the British government denied it.

Qatar’s small size and unassuming national profile lent it credibility as an honest broker. Its wealth helped lubricate its diplomacy, funding development programs in many of the countries where it conducts conflict resolution, and its small native population gave its government a relatively free hand in foreign policy without worrying much about domestic reactions.

For years, Sheikh Hamad bin Jassim Al Thani, who served as foreign minister and later prime minister, jetted across the Middle East, trying to mediate disputes. A success came in 2008, when he helped broker an agreement between Lebanese factions to avoid the outbreak of another civil war there.

A few years later, Qatar agreed to host Hamas’s exiled leadership after it closed its office in Damascus, Syria, following the outbreak of the Syrian civil war. For years, the Qataris have financed the provision of electricity in Hamas-controlled Gaza and supported the 100,000 poorest families there. Days before the Oct. 7 attack, they negotiated an increase in Israeli work permits for residents of Gaza.

“We put our money where our mouth is, and we literally pay for the sustainability of the agreements that happen here in Doha,” said Al Ansari, the adviser to Qatar’s prime minister.

He added: “Why are we able to mediate very strongly and have open channels of communication between Hamas and Israel? It’s because of the trust we have from both sides.”

Write to Stephen Kalin at stephen.kalin@wsj.com