The Swans had gone ahead in the 67th minute with an own goal by Gary McDonald, but that was cancelled out by a Craig Davies penalty ten minutes later.
It was not a vintage performance from Swansea, but they battled through for a hard-earned victory.
The Swans were without Paul Anderson, who suffered a calf strain during the draw at Nottingham Forest in midweek.
Guillem Bauza was relegated to the bench and joining the starting line-up were Thomas Butler and young midfielder Joe Allen.
Both sides packed midfield and there was very little goalmouth action as play settled down.
A snapshot by Lewis Alessandra in the 18th minute ruffled the side-netting as the visitors fashioned a rare strike on the home goal.
Davies was busy down the Swansea left, but Oldham were unable to create any clear-cut openings despite forcing the home side back on the defensive around the half-hour mark.
Swansea finally woke up a minute before the break when Allen went close and they also forced a couple of corners but were unable to carve out an opening.
Bauza replaced Allen on the hour as Swans manager Roberto Martinez looked for more fire up front after a lacklustre opening half.
The breakthrough for the Swans came in the 67th minute when Bauza worked Jason Scotland free.
The striker cut the ball back from the byline and Oldham midfielder McDonald slid it into his own net while trying to clear.
The introduction of Bauza had certainly sparked the Swans into life and he was the one player on the field who could find space and use the ball.
In the 75th minute, Oldham levelled after Davies was brought down by Ferrie Bodde after cutting across just inside the penalty area.
Davies took the spot-kick and easily beat Dorus de Vries in the home goal.
But Brandy came on in the 83rd minute and within 90 seconds pounced on a loose ball to rifle home beyond veteran keeper Mark Crossley for the winner.