People for the Ethical Treatment of Animals (Peta) wants the mascot to be replaced with a modern animal-friendly version.
The group has suggested a fish with the slogan Skegness - A Happy Plaice.
The mayor of Skegness said all suggestions were welcome but said he did not think there would be much support for Jolly to be axed.
The original Jolly Fisherman poster was commissioned by the Great Northern Railway to encourage visitors to take the train on their seaside breaks to Skegness.
The poster featured the slogan "Skegness is so bracing".
It was commissioned in 1908 for 12 guineas and has became synonymous with the Lincolnshire resort.
However, Dawn Carr, of Peta, said Jolly "evokes images of cruelty to animals".
"Fish being tricked into impaling themselves in the mouth, animals being pulled out of the ocean in giant nets where they are so crushed together the pressure pushes their stomachs out through their mouths and their eyes pop out."
"This is what the Jolly Fisherman represents," she said.
She said Peta's suggestion would give the resort an opportunity to rebrand itself as a progressive holiday destination.
"One where the mutilation and killing of sea animals is not being celebrated."
Skegness's mayor, Carl Macey, said the Jolly Fisherman was loved by adults and children alike.
He said: "Animal cruelty is something we don't want to be portrayed for, but I don't feel Jolly has ever come across in that manner.
"He is just accepted as happy and friendly and he does such a wonderful job for Skegness.
"It [Peta's suggestion] will go before council, but if it isn't broken we won't try and fix it."

The pair worked together at Watford and Northern Irishman Rodgers, 43, is the frontrunner for the Celtic job after two rounds of talks this week.
Celtic's largest shareholder Dermot Desmond said on Wednesday "in excess of six" candidates had been interviewed for the manager's position.
"There's no doubt about his qualities," said Warburton, 53, of Rodgers.
The Englishman was a coach with the Hornets during Rodgers' time as manager, with the latter subsequently having a spell in charge of Reading and then leading Swansea to promotion to the Premier League.
A move to Liverpool followed and Rodgers left Anfield in October after over three years in charge.
Warburton is preparing his Rangers side to face Hibernian in Saturday's Scottish Cup final.
The Englishman has already won the Championship and Challenge Cup in his first season in Glasgow.
And, after overcoming eventual Premiership champions Celtic in this season's Scottish Cup semi-final, Warburton's Rangers will face their Old Firm rivals in next season's top flight.

Media playback is not supported on this device
The news of Annan Athletic's 4-1 fourth-round win over Hamilton Academical was far-reaching.
Omar, the son of refugees from Iraqi Kurdistan, was featured in the local press there.
"There's a big following there now, my uncle tells me," said the Glasgow University student.
"A lot of them, when they are looking out for scores, will look out for Annan Athletic now."
Annan's dream ended in the fifth round though, when they lost 4-1 to Morton.
For more information about how to get into football, take a look at the Get Inspired page on the sport.

The 24-year-old, capped 32 times by England, has been out of action since November with a knee injury.
However, it is understood his recovery has progressed quicker than expected.
Vunipola could now be in contention to face Scotland on 11 March, or Ireland a week later.
He was one of England's outstanding performers throughout their unbeaten year of 2016, and his ball-carrying and dynamism has been missed by Eddie Jones' side.
There were initial fears Vunipola would miss the whole of England's Six Nations title defence, but it is understood he has looked sharp on his return to full training this week, and is targeting a quick return to international rugby.
Vunipola's older brother, Mako, recently made his own comeback from a knee problem, and came off the bench in the win over Italy.
An animal rights group is calling for the Skegness mascot, the Jolly Fisherman, to be replaced with a fish.
[NEXT_CONCEPT]
Rangers manager Mark Warburton believes Brendan Rodgers would be a "great choice" as Celtic's new manager.
[NEXT_CONCEPT]
Goal hero Rabin Omar made headlines when his club from the fourth tier of Scottish football dumped a Premiership side out of the Scottish Cup.
[NEXT_CONCEPT]
England number eight Billy Vunipola is set for a shock early return to action for his club Saracens against Newcastle on Sunday, boosting his hopes of a 2017 Six Nations call-up.