Oldham picked up their first points of the season but Swansea are still waiting for a win after Lee Trundle's penalty miss at Boundary Park.
Trundle was denied by Oldham keeper Chris Howarth as the home team held on to the lead they had earned through half-time substitute Maheta Molango.
John Sheridan's Latics had lost 1-0 in their opening fixtures against Tranmere and Port Vale, but they played attacking football and were relieved to see Molango end their goal drought.
Swansea, expected to challenge for promotion after reaching the League One play-off final last season, have taken only one point from a possible nine.
Trundle's afternoon to forget started when he was off target with two shots from just inside the box.
Oldham went close on 14 minutes when Andy Liddell crossed to midfielder Gary McDonald whose shot was deflected against the outside of the post by Kristian O'Leary.
Home striker Chris Hall forced Willy Gueret into a save before Charlton tested the keeper from 25 yards.
But Swansea hit the woodwork themselves when Chris Howarth could only punch to ball to Trundle 12 yards out and the striker hit the bar.
Swansea had a great chance on 58 minutes when Trundle crossed to the far post but Leon Knight's effort was kept out and Rory Fallon slammed wide from close range.
Howarth also denied Knight before Oldham went in front at the midway point of the second half.
Charlton crossed from the left and Molango was unmarked to nod home his first goal for the club from inside the six-yard box.
Swansea's penalty came just five minutes later when Charlton tackled substitute Thomas Butler down the right and referee Graham Laws controversially deemed it to be a foul.
Trundle stepped up but Howarth dived correctly to his left and beat away the shot.