Madley decided Wayne Routledge had handled in the penalty area during stoppage time and Stoke equalised from the spot.
Afterwards Swansea manager Michael Laudrup claimed Madley owed him and his players an apology.
However, referee Andre Marriner, who was criticised by West Brom for awarding Chelsea a late penalty in the last round of top-flight action, has been put in charge of Saturday's fixture between Fulham and Swansea.
Source: PA
Source: PA