The Slovenian, in the rainbow bands, attacked on the steep Rampe Saint-Hillaire with five kilometres to go, dropping everyone bar Vingegaard.
Van Der Poel and a group of six came back to the pair, and despite the Dutchman launching his sprint first, it was Pogacar who went over the top to claim his first stage win at this year's Tour de France after some great teamwork from Joao Almeida to keep it all together in the finale.
The pair are now level on time going into Wednesday's time trial, with Van Der Poel holding onto yellow for another day. Vingegaard remains third in GC, eight seconds in arrears of his stage four rivals.
After the stage, Pogacar said: “Everyone was at the limit. I tried an attack on the last climb, Jonas followed me and it all came back together.
"Joao did an amazing job for me with everyone attacking. It’s such a nice victory.
“To win at the Tour is incredible, and to win in this jersey. To have 100 wins is amazing."
Earlier in the day, as the riders traversed a 174.2-kilometre route between Amiens and Rouen, four riders got up the road to make the breakaway.
Former Tour stage winner Kasper Asgreen was the final rider to make it across as he joined Jonas Abrahamsen - a star of the 2024 edition - alongside French duo Lenny Martinez and Thomas Gachignard.
They were only given a maximum of just over two minutes on the run into Rouen, which was ridden at pace as the riders found a tailwind pushing them towards the city.
It was a calm day in the peloton before they reached the climbs at the end of the day, as yellow jersey wearer Van Der Poel’s team Alpecin-Deceuninck controlled the pace as the Dutchman looked for his second stage win in the first four days of the race.
Martinez, who had had a poor opening day to the tour as the last man over the line, was the smallest rider in the four up the road and with 47 kilometres to go, he naively attacked his breakaway compatriots on the first categorised climb of the stage. He was soon caught by the trio, and missed out on the King of the Mountain point on top of Cote Jacques Anquetil as Asgreen beat him in the sprint.
The stage was dedicated to French cycling icon Anquetil, who became the first man to win the race five times back in the 1950s and with the last 30 kilometres up and down all the way to the finish, a mini general classification scrap looked likely.
UAE Team Emirates, with Pogacar in their midsts, took control of the chase on the category three climb up the Cote de Belbeuf, with safety from crashes paramount to their strategy on the narrow roads through Normandy.
The pace from Pogacar’s squad was too much for Jayco-AlUlua’s GC hopeful Ben O’Connor, who was dropped from the front group with 12 kilometres to go, shipping nearly two minutes to the top contenders.
Only 25 riders remained onto the final climb with Visma Lease A Bike in control of the group, but it was Pogacar who launched on the 14% ramp. Only Jonas Vingegaard could go with him, but that didn’t last long with the Slovenian getting a gap. However, the pair crested the summit of the ramp together with just over five kilometres to go as the Dane gritted his teeth to catch his rival.
Five riders would come back to the pair, including Van Der Poel, Almeida, Remco Evenepoel, Oscar Onley and Jumbo Lease A Bike's Matteo Jorgensen. Both Jorgensen and Evenepoel tried attacks before the final 500 metres, but neither could get a gap.
Almeida was the one to keep it together, and he led out the sprint with 250 metres to go, and Pogacar made no mistake.
It was a selective group at the line after a technical finale, and it wasn't just O'Connor who lost time. Enric Mas and Primoz Roglic lost 32 seconds, whilst Florian Lipowitz lost 54 seconds.
Pogacar moved back into the King of the Mountains jersey, whilst Jonathan Milan increased his lead in the green jersey. Kevin Vaquelin retained the white Best Young Riders jersey after finishing 10th on the fourth stage.
Tomorrow sees the riders tackle a flat 33-kilometre time trial around the city of Caen, giving the race the first real time gaps in the general classification battle.