The poster, on Tamworth Road, in Sawley, Derbyshire, shows aliens beaming up a person into their spaceship with the text, "they'll take the fat ones first".
Combat Bullying, based in Nottinghamshire, said it would "further harm" those who are being bullied.
Fit4Less said it wanted to create a "light-hearted and humorous" advert.
It reads: "They're comingâ€¦ and when they arrive they'll take the FAT ones first!". It then has "save yourself!" with an arrow pointing to the gym's website.
One passer-by said the poster was "ridiculous" and he would "rip it down" because "being a big lad myself, it upsets you".
Many other people said it was "in poor taste" and "offensive".
Natalie Harvey, founder of the charity, said the poster has caused her concerns because it would "aid bullying".
"This week alone three cases have been brought to my attention of children being bullied because of weight issues," she said.
"If those children or the perpetrators saw this poster it would cause further harm for the children who are being bullied.
"The poster should be removed and replaced with something more tasteful to attract the gym goers."
Kerry Matthews, from the gym in Long Eaton, Derbyshire, said: "We really didn't mean it offensively.
"It was supposed to be a bit cheeky, hence the alien image, and grab people's attention in a couple of seconds.
"So many campaigns use ultra skinny people and that's not the reality."
Ms Matthews added that the gym wanted to get people talking about getting fitter.

The 35-year-old joined the Iron from Blackpool in 2013 and has made 119 league appearances for the League One side.
He helped them to a third-placed finish this season, before they were beaten by Millwall in the play-off semi-finals.
Bishop told the club website: "With the way the season finished, it's a sense of unfinished business and it was disappointing for all of us."

He told The Hollywood Reporter he was "super excited" and wanted "to put a different spin on it", but did not expand on what role he would play.
The film will see Jordan reunite with his Creed director Ryan Coogler.
"I'm looking forward to working with Ryan once again. It's gonna be a good time," he said.
He added he could not wait to work with Nyong'o, who won a best supporting actress Oscar for 12 Years a Slave in 2014, as well as Chadwick Boseman - who is currently on the big screen playing Black Panther in Captain America: Civil War.
Jordan played the Human Torch in last year's Fantastic Four flop reboot.  Nyong'o also played CGI character Maz Kanata in Star Wars: The Force Awakens.
Black Panther will be released in February 2018.
An anti-bullying charity has called for a gym billboard poster to be removed for being "offensive".
[NEXT_CONCEPT]
Scunthorpe midfielder Neal Bishop has signed a one-year contract extension.
[NEXT_CONCEPT]
Michael B Jordan has confirmed he and Oscar-winning actress Lupita Nyong'o will be among the cast of the upcoming Marvel superhero film Black Panther.