Two clinical finishes sunk the Somerset club, who could consider themselves unfortunate to finish empty handed.
Yeovil should have taken the lead in the seventh minute when Kevin Betsy was left with a clear opportunity from a free-kick by Anthony Barry.
His six-yard shot was brilliantly saved by Dorus de Vries who deflected the ball for a corner.
Yeovil had striker Marcus Stewart injured and replaced on 18 minutes, but they continued to push forward with Betsy and Lloyd Owusu working well together up front.
However, when the deadlock was broken it was at the other end.
Paul Anderson, the right winger on loan from Liverpool, had been closely watched by full-back Nathan Jones.
But in a quick break, with Jones stranded up field, he broke past Scott Guyett and crossed for Jason Scotland to provide a good low finish from 12 yards.
At the start of the second half Yeovil got the goal they deserved when a free-kick on the left from Barry was headed back across goal by Guyett and Betsy scrambled in his first since joining on a loan spell from Bristol City.
Yeovil continued to create danger from free-kicks, but when the deciding goal came it was from Swansea.
On 79 minutes Ferrie Bodde wriggled past a Yeovil defender and beat goalkeeper Steve Mildenhall with a low angled drive from 12 yards.
Yeovil threw on another loan signing - James Walker from Norwich - in the closing stages and he almost scored a debut goal.
His close-range effort came down off the underside of the bar but refused to cross the line.