Many residents are "devastated" at the move, according to a local councillor.
But Shoreline Housing said 52% of locals backed the demolition of the East Marsh blocks and Comber Place maisonettes, while 42% were against it.
A total of 457 flats are affected and will start to be emptied in October, with residents rehomed over the next four years.
Councillor Steve Beasant, who represents people in the East Marsh ward, said: "Residents are devastated at the news.
"They actually thought that Shoreline was beginning to think again about the multi-storey flats."
Shoreline said it would carry out a study by April to find out if it is "affordable and viable" to provide suitable alternative older persons' accommodation.
The company added that its employees at the buildings would also be affected by the demolition decision. A review of staff structure is to be undertaken.
A Shoreline spokesman said: "We strongly believe this is in the best long-term interests of Shoreline so that we can ensure our continuing ability to invest in affordable modern accommodation across the borough for years to come."

Joe Acquaro was shot dead in Melbourne in the early hours of Tuesday. Investigators believe underworld bosses may have put a price on his head.
The lawyer was well-known in the city's Italian community and his body was found outside a cafe he owned.
Clients included crime bosses accused of extortion, drugs and arms offences.
A vicious gangland feud that claimed at least 28 lives in Melbourne ended more than a decade ago, but detectives say that underworld tensions still fester in Australia's second biggest city.

As reported in the Daventry Express, Sam Chambers, coach of Weedon Bec Primary School, sent his CV to the FA after England crashed out of Euro 2016.
"I feel like the perfect candidate as I have guided my school... to an unbeaten 2015-16 season," he wrote.
Ladbrokes has made him a 1,000/1 outsider for the job.
Roy Hodgson stood down as England manager after the shock 2-1 defeat to Iceland last week.
Jessica Bridge from the bookmaker said: "Stranger things have happened in the last 12 months, like Leicester City winning the Premier League, so it's not out of the realms of possibility.
"Sam enters the record books as one of the greatest sporting underdogs of all time."
Mr Chambers, who describes himself as a "sports enthusiast", faces stiff competition from current favourites Jurgen Klinsmann and Sam Allardyce, but says he has the passion to be a success in the job.
"I, along with millions across the nation, saw our beloved team fall to the minnows that are Iceland," he wrote in his covering letter to the FA.
"It is my intention, should I be given the opportunity, to re-kindle the enthusiasm of the nation for the round-ball game."
Elaborating on his school's impressive season, Mr Chambers added he had guided his side to an unbeaten campaign - even though they faced "mighty teams such as Woodford Halse CE Primary Academy".
He ends the letter with an optimistic "I look forward to hearing from you all!"
The FA is yet to comment on Mr Chambers' application.
Earlier this year, a seven-year-old boy failed in his quest to become Inverness Caledonian Thistle's new manager, despite a pledge that his headmaster would probably let him leave school early "for important matches and training".
Six high rise blocks in Grimsby are to be demolished meaning hundreds of people will have to move home.
[NEXT_CONCEPT]
Police in Australia believe a lawyer who represented some of the country's most notorious organised crime figures was the victim of a contract killing.
[NEXT_CONCEPT]
The coach of a primary school football team has applied for the England job, saying he is the "perfect candidate" to succeed Roy Hodgson.