He struck in the 50th and 60th minutes and although Leon Knight pulled one back late on, the Terriers held out and the Swans slipped to a second successive defeat.
There was only one change in the home line-up following the defeat at Gillingham with Thomas Butler brought back into a wide midfield role at the expense of Leon Britton who was on the bench.
Huddersfield almost took the lead from a 12th minute free-kick taken by Andy Holdsworth.
Veteran Andy Booth nodded down into the danger area and a snap-shot from Danny Schofield struck the base of the post with Willy Gueret well beaten.
Andy Robinson had visiting keeper Matt Glennon stretching to keep out a fierce drive as the Swans started to find some attacking flare, but with Booth winning his aerial battle, the Terriers were soon back on the attack.
An incisive break by Taylor-Fletcher after Booth had won another header should have brought some reward but his shot from a good position whistled over the bar.
Tackles became ever more furious and David Mirfin became the third player booked as the first half drew to a close.
Five minutes into the second period Huddersfield took the lead with a well-worked goal.
Mark Hudson threaded the ball through to Taylor-Fletcher and as Gueret came out to block the striker flicked beyond him and into the corner of the net.
The Terriers doubled their lead on the hour when Schofield drifted through a hesitant Swansea defence before sliding a pass to Taylor-Fletcher who beat Gueret easily from close range.
Knight pulled one back in the 81st minute with his sixth strike of the season when he sent a low shot through a crowded goalmouth.
But despite piling on the pressure in the closing minutes it was Huddersfield who held firm for a deserved victory.