"I would say it is practically impossible," said Poyet, who guided the Seagulls to League One safety last season.
"I am very close to signing a four-year deal here and everything is agreed."
Poyet, who was appointed at the Withdean in November last year, still has a year to run on his first managerial contract but has been in talks over the new deal since April.
It is not known if Brighton would stand in the 42-year-old's way should Swansea make an approach, but Poyet admitted he was not surprised to have been linked with the job.
Poyet is not the only manager distancing himself from the speculation surrounding the Liberty Stadium post.
Exeter City boss Paul Tisdale, who was another name linked with the post last year, has refused to be drawn into talking about renewed speculation involving him.
Thriving Swans await new boss - Curtis
But Wigan Athletic assistant manager Graeme Jones, who followed manager Roberto Martinez to the Premier League last summer from Swansea, is being heavily linked with a return to south Wales, this time as manager.
Jones played a key role behind the scenes as the Swans ran away with the League One title in 2008 and flirted with the Championship play-offs the following season.
Former West Brom manager Tony Mowbray is also a name that is being touted around frequently.
Mowbray led the Baggies to the Premier League in style twice, before later leaving for Scottish giants Celtic.
But his time at Parkhead lasted less than a year and he was sacked at the tail end of last season.
Coaches Alan Curtis and Colin Pascoe are in charge of pre-season training at the Liberty Stadium while a replacement for Sousa is found.
Swansea chairman Huw Jenkins says he is keeping his options open, hinting that he may not necessarily go for a fourth successive rookie as he did with Sousa, Martinez and before that Kenny Jackett, the current Millwall boss.
Monk surprised at Sousa departure
"We've got to find somebody with the right personality to come in here and give us 100% and work with us to achieve what we want," Jenkins said.
"The number-one priority is finding somebody we can work with, somebody who is happy to work with Swansea City and realise the level where we are at.
"He will have the hard work and commitment and drive and enthusiasm needed to take us forward and achieve or even better what we did last year.
"Some people to be fair to them are quite loyal and committed to where they are and so you've just got to respect that.
"Then there are players in the game who are looking to start their managerial career and there's plenty of managers who have lost their jobs and are looking to restart, so there's a variety of candidates."
Jenkins did not put a timescale on the search for a new manager but said the club needed to refocus "quickly over the next few days".
Source: BBC Sport
Source: BBC Sport