The on-loan Hull City striker appeared to knock the ball into the net with his hand after Izzy Iriekpen's shot was superbly saved by Keiren Westwood.
Several minutes of Carlisle complaints followed, but in truth a draw would not have been enough for the Cumbrians.
Swansea had taken a stunning lead after just 40 seconds to leave the hosts reeling.
Lee Trundle, an immense figure throughout, picked up Leon Britton's cross and spun and volleyed effortlessly into the roof of Westwood's net.
The tone was then set for the rest of the first half with the Swans dominating Carlisle's midfield and firing off shots at ease.
Better finishing would have put Swansea clear at the break and they were clearly taken aback by Carlisle's new attacking attitude after the break.
Johann Smith - who was woeful in the first half - was moved from left wing to striker and within seconds Swansea's defenders were struggling to deal with his pace.
The tactical switch paid off immediately when Smith scored a great individual effort.
Picking up Chris Lumsdon's cute pass, the teenage American ghosted between two defenders and rounded keeper Willy Gueret to finish into the empty net.
A pulsating, end-to-end game ensued with both sides going all out for a win that would edge them closer to the final play-off position.
Late on Smith should have set up Lumsdon for a simple finish, but instead fired weakly at Gueret before Duffy had the last word with seconds left to play.