The Iron have lost three of their last six league games, but are still second in League One.
The home defeat by Shrewsbury on 11 February ended a run of 26 league games without losing at Glanford Park.
"Let's get real, we are second in the league and that is after playing probably our worst football of the season," Swann told the club website.
"Do you honestly believe that when a player doesn't control the ball with his first touch of the game we should be on his back?"
He added: "The rumours of contract disputes, unrest in the team, a not-bothered attitude, is so far from the truth it hurts when I read it and certain people just want to derail our success whether knowingly, or not, which is even more annoying.
"We ask passionately that you all get behind them and give them that chance to recapture their form and yes, do not panic, because we can do it, we can get promoted and we can be proud of them."
The Iron were top of the table when they last won, against Port Vale on 28 January, but are now seven points behind leaders Sheffield United and level on points with third-placed Fleetwood.
Scunthorpe host the Cod Army on Saturday.

Adrian Simut, 35, travelled to Newport train station, where he thought he was meeting a 14-year-old girl called Sam.
But he was confronted by so-called "paedophile hunters" and later arrested, Newport Crown Court was told.
He admitted attempting to incite a child to engage in sexual activity and meeting a child after online grooming.
He also pleaded guilty at an earlier hearing to attempting to cause a child to watch a sexual act.
The court heard Simut started messaging the teenager in an online chat room in June, saying she looked "cute and beautiful".
But she was in fact an adult belonging to a group called Petronus.
The operation was captured on film by the BBC Wales Week In Week Out programme, which investigated the role of paedophile hunters in Wales.
The court heard that Romanian national Simut arranged to meet "Sam" in Newport and suggested she bring a friend for sexual activity.
Sentencing him, Judge Michael Fitton QC said: "You were the subject of an exchange conducted by those who are looking to attract paedophiles."
He was also made the subject of an indefinite sexual harm prevention order.
Paedophile hunting groups have drawn criticism from both the Home Office and police, who have said it was inappropriate for the public to conduct undercover work.
They have urged anyone with information to instead pass it to them instead.
Scunthorpe chairman Peter Swann has urged fans to keep faith in the team and "not get on players' backs".
[NEXT_CONCEPT]
A paedophile who travelled from London to south Wales has been jailed for three years.