His time of 36 minutes and 42 seconds around the 33-kilometre course saw him leap up to second on the general classification.
Jonas Vingegaard was the main favourite to lose the most time, as Evenepoel clawed more than a minute back on the Danish two-time Tour champion. He would finish 12th on the day in a time of 38 minutes and three seconds.
Pogacar took the yellow jersey from Van der Poel after the Dutchman failed to match the Slovenian's time of 36 minutes and 58 seconds.
The world champion now leads Evenepoel by 42 seconds, with Kevin Vauquelin in third after an inspiring ride from the Frenchman. Vingegaard dropped down to fourth in the overall standings after the biggest shake-up so far at the Tour.
After the stage, Evenepoel said: "You never know because big guys like Edoardo can do this course well. I knew I had a good chance, I had the legs there and I didn't think I could go any faster.
"I pushed pretty steady... my strongest point was I kept the same pace at the end as I did in the first 10 kilometres... I paced it perfectly.
"Tadej did a very strong time trial as well, 16 seconds is pretty close... It shows he is in big form, and it's the reason why he is the guy to beat at this Tour de France."
Affini and Plapp impress
The time trial, a circuitous route from Caen back into the French city, was pan flat, and suited the strong time triallists.
With the GC contenders going out late in the day, it was up to Luke Plapp of Jayco-AlUla and Edoardo Affini to provide the early entertainment.
The Australian Plapp set the fastest time check through the first and second checkpoints, but it was Affini who would spend much of the afternoon in the hot seat.
The Italian would be there as the leaders set off. Remco Evenepoel, sitting ninth overall after four stages of the Tour, was the first of the big challengers on the day to set off, missing out by two seconds to Plapp’s fastest time. The wind had picked up throughout the day, and as he rode into a headwind on his gold time trial bike and with the rainbow bands on his back, he would have been happy with his initial efforts.
Jonas Vingegaard, third overall overnight, was 14 seconds back on the Belgian, whilst Pogacar was matching Evenepoel.
Van der Poel, not known for his time-trialling capabilities compared to the aforementioned trio, was level on time with Pogacar at the top of the general classification, but he had already lost 30 seconds at the first checkpoint, meaning his dream of another day in yellow was dashed early on.
All eyes turned to time check three, where Evenepoel was stretching his virtual lead over Affini, as he hunted the stage win. He was 11 seconds ahead of the European time trial champion with 8.2 kilometres to go.
The momentum was building for one rider, but it was turning into a struggle for Vingegaard, who was losing time on the road to both the Belgian and Pogacar.
Evenepoel would cut the finish line with a time of 36 minutes, 42 seconds - 33 seconds quicker than Affini with an average speed of 54km/h over the 33 kilometres. He had thrown down the challenge for the Tour’s two biggest stars.
Crisis for Vingegaard?
Vingegaard was shipping time in the closing stages, and he will be disappointed to lose more than a minute to Pogacar. He will now wait for his chance in the mountains next week for an opportunity to pounce on Pogacar.
The best young rider at the start of the day, Vauquelin, put in the time trial of his life to lift himself up to third in GC, finishing fifth on the stage, whilst there were good rides for Primoz Roglic and Florian Lipowitz. Joao Almeida also kept himself in contention, finishing eighth on the stage, and a valuable lieutenant for teammate Pogacar.
The biggest losers in the GC battle included Enric Mas, who lost nearly three minutes on the day.
Stage six of the Tour sees the sprinters come back to the fore with a relatively flat 201.5km run from Bayeux to Vire Normandie. The finish of Thursday's stage contains a ramp to the line, giving plenty an opportunity to snatch a stage win.