Shell is holding early stage talks to acquire rival BP in what would be the largest oil deal in a generation, people familiar with the matter said.

Talks between company representatives are active, the people said, and BP is considering the approach carefully. Acquiring BP would put Shell on firmer footing to challenge larger competitors such as Exxon Mobil and Chevron . It would be a landmark combination of two so-called supermajor oil companies, a group of multinational behemoths that dominate the production of the world’s most important energy sources.

Potential terms of any deal couldn’t be learned and a tie-up is far from certain, the people familiar with the matter warned. The discussions are moving slowly, one of the people added.

BP is currently valued at around $80 billion. Taking into account a premium, a deal could end up as the largest corporate oil deal since the $83 billion megamerger that created Exxon Mobil at the turn of the century. It would also easily be the biggest M&A deal of the year, so far, in a market that has been rattled by President Trump’s trade war and other geopolitical tensions.

A Shell spokesman said: “As we have said many times before, we are sharply focused on capturing the value in Shell through continuing to focus on performance, discipline and simplification.”

A BP spokesman declined to comment.

Shell comes into the talks operating from a position of strength, with its stock sharply outperforming BP in recent years. Shell, which like BP is based in the U.K. but has operations around the world, has a market value of more than $200 billion.

BP has been the laggard among major oil companies after an ill-fated push away from fossil fuels into renewable energy. It has also suffered years of management upheaval and operational disasters .

Activist investor Elliott Investment Management, which owns more than 5% of BP’s shares, has pushed for changes at the energy company since at least February, underscoring the oil and gas producer’s exposure to a potential takeover bid from a rival.

BP has since adopted several measures to try to address investor frustrations. It announced plans earlier this year to boost oil and gas production and sharply cut investments in clean energy.

BP is also in the process of trying to sell its lubricants business that operates under the Castrol brand and has indicated it could unload at least part of its solar unit, Lightsource. More recently, BP said that its Chairman Helge Lund , a key architect of the company’s low-carbon strategy, plans to step down .

Shell meanwhile has focused on its most profitable operations, pledging to pump more oil and gas and rolling back green energy targets.

When asked publicly, Shell Chief Executive Officer Wael Sawan has said recently that the company’s bar for big dealmaking would be high. Shell in May announced a multibillion-dollar share buyback plan, the latest in a long series of big share repurchases. Shell has been working with bankers on a potential sale of its chemicals assets in Europe and the U.S., The Wall Street Journal previously reported.

For Shell, acquiring BP would take years of integration, complicated by culture clashes and possibly the sale of overlapping assets. But a deal could give Shell’s global trading business greater reach and bolster its dominance in areas like liquefied natural gas. Analysts and investors also see a good matchup in the companies’ Gulf of Mexico operations.

Acquiring BP would offer an opportunity for Shell to spread costs over a larger operating base and would box out rivals. Shell would also be more politically palatable to U.K. regulators who may oppose a foreign buyer from acquiring BP, a more than century-old company that traces its roots to oil exploration in Persia during the height of the British Empire.

Bloomberg News reported in early May that Shell was evaluating a potential BP deal.

A Shell-BP deal would be the latest in a wave of M&A activity across the energy landscape as the producers look to achieve greater economies of scale.

Chevron is still working to close its $53 billion megadeal for Hess, which has been held up due Exxon’s effort to challenge the deal’s legality.

Exxon last year closed a $60 billion deal to buy Pioneer Natural Resources. Diamondback Energy sealed a $26 billion deal for Endeavor Energy Resources to bolster its position in the Permian Basin.

Write to Ben Dummett at ben.dummett@wsj.com , Lauren Thomas at lauren.thomas@wsj.com and Jenny Strasburg at jenny.strasburg@wsj.com