McCall said he and some of his players were goaded by supporters following their 1-0 loss to the Lions at Wembley.
He questioned why stewards had not prevented the invasion.
"We're absolutely distraught, we wanted to clap our own fans, the last thing we need is people running around us and giving us verbals," said McCall.
"I think it's nonsense," he told BBC Radio Leeds. "Ten minutes later an army of stewards comes on when they should have been on straight after the final whistle and stood on the halfway line.
"I don't understand why people feel they need to come over to players, intimidate us and wind us up. I was close to clocking a couple.
"I'm not necessarily blaming the people who did it but where was the stewarding? Why march them out 10 minutes afterwards?
"They should have got to the halfway line straight away and stopped anyone from coming over."
Millwall boss Neil Harris encouraged his side's fans to leave the pitch to allow the team to enjoy their celebrations, while match-winner Steve Morison claimed they had "ruined" the moment.
"It's not ideal, it's not quite the place for a pitch invasion, I've got to be honest," Harris told BBC Radio London.
"I don't condone it whatsoever. I'm just hoping there was no disrespect towards Stuart and his team.
"For me, I wanted to get them off because I wanted to see my players enjoy it and I wanted to see my players take the admiration of the whole fan base, which happened in the end.
"I think it was overzealous, a lot of people wanting selfies and jumping about. I certainly don't condone it, but I'm hoping it was lighthearted."

Tynan, a former Manchester City player, died after being hit by a train at West Allerton station in Merseyside on Tuesday, British Transport Police said.
Tynan's death is not being treated as suspicious.
Her family paid tribute to a "vibrant, generous and fun-loving girl", who was "a dedicated athlete, never happier than when she had a ball at her feet".
Tynan began her career at Liverpool Feds, spent six years at Everton's Centre of Excellence and was playing for Women's Premier League side Fylde Ladies.
A family statement also said she was a "the most loving and caring daughter and sister anyone could wish for" and that she was the "ultimate team player".
It added: "Zoe always knew how to cheer anyone up, and was a loyal, straight-talking friend to many. She touched so many people's lives and will never be forgotten."
Tynan joined Manchester City in 2015, making one Women's FA Cup appearance before moving to Fylde.
Floral tributes have been left at the scene, according to the Liverpool Echo.
England internationals including Lucy Bronze and Casey Stoney have also paid tribute.
Fylde manager Luke Swindlehurst said: "We want to remember Zoe in the best possible way: a hugely talented player and an immensely likable character."
Tynan had appeared for England at various youth levels and was recently included in the Under-19 squad for a training camp at St George's Park.
The Football Association said it was "deeply saddened" by the death and Tynan's Under-19 coach Mo Marley described her as a "hugely-liked and popular member of the team".
Bradford City boss Stuart McCall said he was "close to clocking" Millwall fans who invaded the pitch at the end of the League One play-off final.
[NEXT_CONCEPT]
England Under-19 Women's and Fylde Ladies midfielder Zoe Tynan has died, aged 18.