Barguil somehow held on to win a photo finish with Colombian Rigoberto Uran as Poland's Bartosz Huzarski took third on the 146.8km 16th stage from Graus to Sallent de Gallego.
Italian Nibali, meanwhile, struggled mightily on the climb to the finish as he was dropped by American Chris Horner with around 3km to go, losing 22 seconds on the RadioShack rider and seeing his overall lead trimmed to just 28 seconds with five stages remaining.
Movistar's Alejandro Valverde remains in third place overall but he also gained time on Nibali and is now just 1min 14secs behind the Giro d'Italia winner.
"I wasn't at my best, but it wasn't too big of a scare," said Nibali afterwards.
"It was a very difficult finish, above all for the headwind and I thought I would be better in the final kilometres but as you saw I lacked the strength.
"After three very hard days it is normal to feel a bit tired, but at least we have a rest day tomorrow and I can recuperate."
After two brutal stages through the Pyrenees over the weekend, the pack was split for the majority of the day with a large leading group of over 20 riders being held in check by the chasing peloton.
The excitement didn't truly start therefore until around 10km to go when, after a series of failed attempts, Barguil eventually pulled clear at the front and quickly opened up a 30-second gap.
However, unlike his break to victory on Friday, the 21-year-old didn't have enough in the tank to maintain his lead all the way to the finish and was caught by Uran about a kilometre from home.
The Sky rider then seemed certain for a first stage victory in the Tour of Spain but Barguil somehow surged past him in the final sprint to the line to take the win.
"It has been a great tour for me. When I attacked I had very good legs but I saw Uran was coming and I thought it best to wait a little and go for the sprint," said Barguil.
"To beat such a great rider and win gives me great confidence."
Further back Valverde, the 2009 winner, and Joaquim Rodriguez, who in July finished third on the Tour de France, were attempting to break away from Nibali on the final climb, but unlike in the previous two days the 2010 champion wasn't able to stay with them as the Spanish duo both gained 28 seconds on the leader.
And Horner was also able to escape the clutches of Nibali in the final stages to put himself very much in contention to win his first Tour of Spain come Madrid on Sunday.
Nibali will at least have a day to recover, though, with Tuesday being a rest day before the 189km ride from Calahorra to Burgos on Wednesday.
Leading results from the 16th stage of the Tour of Spain, a 146.8km ride from Graus to Sallent de Gallego on Monday:
1. Warren Barguil (FRA/ARG) 3hr 43min 31sec.
2. Rigoberto Uran (COL/SKY) at s.t.
3. Bartosz Huzarski (POL/APP) 0:03.
4. Dominik Nerz (GER/BMC) 0:08.
5. Jose Herrada (ESP/MOV) 0:20.
6. Mickal Cherel (FRA/ALM) 0:37.
7. Maciej Paterski (POL/CAN) s.t.
8. Andr Cardoso (POR/CJR) 0:40.
9. Amets Txurruka (ESP/CJR) 0:42.
10. Chris Anker Srensen (DEN/SAX) 0:45.
11. Juan Antonio Flecha (ESP/VAC) 1:06.
12. Ben Gastauer (LUX/ALM) 1:08.
13. Luis Angel Mate (ESP/COF) 1:09.
14. Egoi Martinez (ESP/EUS) 1:13.
15. Christian Meier (CAN/ORI) 1:29.
16. Juan Manuel Garate (ESP/BKN) 1:36.
17. Joaquim Rodriguez (ESP/KAT) 1:41.
18. Alejandro Valverde (ESP/MOV) 1:44.
19. Thibaut Pinot (FRA/FDJ) 1:47.
20. Chris Horner (USA/RSH) s.t.
22. Nicolas Roche (EIR/SAX) 2:03.
24. Domenico Pozzovivo (ITA/ALM) 2:04.
26. Vincenzo Nibali (ITA/AST) 2:09.
1. Vincenzo Nibali (ITA/AST) 64hr 06min 01sec.
2. Chris Horner (USA/RSH) at 0:28.
3. Alejandro Valverde (ESP/MOV) 1:14.
4. Joaquim Rodriguez (ESP/KAT) 2:29.
5. Domenico Pozzovivo (ITA/ALM) 3:38.
6. Nicolas Roche (EIR/SAX) 3:43.
7. Thibaut Pinot (FRA/FDJ) 4:37.
8. Leopold Knig (CZE/APP) 6:17.
9. Samuel Sanchez (ESP/EUS) 7:33.
10. Tanel Kangert (EST/AST) 9:21.
11. Daniel Moreno (ESP/KAT) 9:31.
12. David Arroyo (ESP/CJR) 11:22.
13. Jos Herrada (ESP/MOV) 11:42.
14. Michele Scarponi (ITA/LAM) 12:08.
15. Igor Anton (ESP/EUS) 14:44.
16. Dominik Nerz (GER/BMC) 15:33.
17. Robert Kiserlovski (CRO/RSH) 18:13.
18. Eros Capecchi (ITA/MOV) 19:16.
19. Andr Cardoso (POR/CJR) 19:58.
20. Chris Anker Srensen (DEN/SAX) 20:53.