It has emerged that Swansea's players did not wear Kick It Out T-shirts ahead of Saturday's game with Wigan because not all the players wanted to.
It was not the only protest in the Premier League as Manchester United's Rio Ferdinand, Reading striker Jason Roberts and Manchester City's Joleon Lescott all refused to wear T-shirts, claiming not enough had been done to stamp out discrimination.
The issue had been highlighted after the under-21 game between Serbia and England in midweek, when England's Danny Rose was subjected to monkey chants and had stones thrown at him by the crowd.
Vorm said: "I think for the group to wear the T-shirts it has to be a unanimous decision and not everyone wanted to wear them.
"We are either all together or we're not. This has been a problem for a while and I don't think that's doing much to stop racism.
"Of course everybody is against racism, but it's not like we wear a T-shirt and then it's gone.
"They have to really do something to stop racism — it's not going to change because you wear a shirt."
Vorm was part of the Dutch side who were subjected to racist chanting during a training session at Euro 2012.
He also experienced prejudice while playing in Holland.
"In Holland the crowd calls you more names," he said. "There is much more respect here for the players than there is back home.
"They call you so many things during the game there.
"In the Euros we trained in front of 25,000 people in Poland and when we were doing the warm-up we could hear monkey noises behind the goal.
"That was during open training so it still happens, not necessarily in the UK but in Eastern European countries.
"It's been a problem for a while and I just hope after what happened to England Under-21s they're really going to do something about it, not just wear a T-shirt."
There have been calls for Fifa to come down heavily on Serbia following last week's ugly scenes.
Some have even suggested they should be banned from competing at tournaments and Vorm supports a such a hard-line stance.
"Punish the people who shout things like that severely," he added.
"It doesn't belong on a football pitch or even in the world.
"It happens quite a lot in Eastern European countries and they have to do something to make a point.
"I don't know if they can ban a country, but they need to make a statement.
"Maybe next time, if it happens again, they should be ruled out of the Euros, World Cup qualifying or whatever.
"They must do something that they'll feel, not just give them a fine."
Swansea manager Michael Laudrup said: "I think it's an individual thing. It's not a football issue, it's a social issue, and I can't tell the players they have to do it.
"I didn't want to have any influence because there are some things you just have to leave up to each person."
And Wigan boss Roberto Martinez said: "We wanted to show respect to the home team.
"The home team for whatever reason decided not to wear the T-shirts so I don't think it made sense for us to wear them.
"I would expect the same from Swansea if things had been the other way round and we had been at home.
"Swansea's players decided not to wear them and we respected the home team."