At least five members of an oil exploration team were killed, a spokesman for the University of Maiduguri said.
Soldiers and members of a vigilante group also died in the ambush.
The high number of casualties will be a blow for the government, which insists the insurgency is all but defeated.
At least 20,000 people have been killed and thousands more abducted since Boko Haram launched its insurgency in 2009.
In the most notorious abduction case, it seized 276 girls from a boarding school in the north-eastern town of Chibok in 2014.
It has since freed more than 100 of the girls, in exchange for the release of its fighters detained by Nigeria's security forces.
Details of what happened on Tuesday remain unclear, with initial reports from the army suggesting the abducted geologists and surveyors, who worked at the University of Maiduguri, had been rescued.
On Tuesday, the army said the bodies of nine Nigerian soldiers and a civilian had been recovered.
But now the university has said at least five members of staff, including two academics and a driver, were killed when the heavily armed convoy made its way back to Maiduguri, in north-east Nigeria.
Several others remain missing.
"We got the impression our staff on the team were rescued because that was what the military spokesman said yesterday," Maiduguri University spokesman Dani Mamman said, quoted by AFP.
"But we were shocked when we were given four dead bodies. This means it  wasn't a rescue. We still have other staff that are yet to be accounted for."
Sources told the BBC that more than 40 people, most of them soldiers and volunteers from a vigilante group, had been killed in total - although other sources suggest the number killed may rise further still.
"The death toll keeps mounting," a source told AFP, adding: "More bodies are coming in."

French President Francois Hollande expressed "deep sadness" at the death of Huguette Dupeu in a statement.
Most of the dead in the raid were foreign tourists. Two of the attackers were killed by security forces.
A huge march against terrorism is planned for Sunday that will finish outside the Bardo museum.
President Hollande is among those expected at the event in the capital Tunis.
Lyse Doucet: Tunisia's test of transition
Cradle of 'Arab Spring' under threat
The museum's planned reopening earlier this week was postponed over security concerns.
Schoolchildren and students were allowed in on Friday, with the general public to be readmitted Monday.
The attack, claimed by Islamic State (IS), was the deadliest in Tunisia since the uprising which led to the overthrow of long-serving ruler  Zine al-Abidine Ben Ali in 2011.
Over 20 people have been arrested in connection with the attack, although just two gunmen are thought to have carried out the raid.
They are said to have been trained in Libya in an area controlled by IS.
The Tunisian Prime Minister, Habib Essid, has admitted "shortcomings" in the country's security system and has dismissed six police chiefs over the attack.

Ms Tiffney was last seen leaving her home in Edinburgh's Dean Village in May 2002.
A murder charge against her son Sean Flynn was found not proven by a jury at the High Court in Perth in 2005.
The remains were found in an area off the A198, near the entrance to Gosford House in Longniddry.
A cyclist called the emergency services when they made the discovery at about 18:30 on Sunday.
There are 40 police officers working on the find.
It is thought Ms Tiffney was murdered and her body disposed of in the Gosford House area.
The bones are still in situ and it is understood it will take a few days to recover them.
Det Supt Pat Campbell, of Police Scotland, said: "The recovery will be a painstaking process.
"We are consulting with a number of experts to ensure that the remains are recovered in a sensitive and careful manner and no evidence that helps us to establish what happened is damaged or missed.
"I understand the anxiety of those who may be waiting for news of a missing loved one, but whilst the identity of the remains is unknown we cannot speculate."
He added: "The death is being treated as unexplained and once the remains are recovered, there will be significant, detailed forensic analysis required to establish the cause of death and whether any crime has been committed."
More than 40 people have died during an attempt to free people during an ambush by militant Islam group Boko Haram, sources have told the BBC.
[NEXT_CONCEPT]
A Frenchwoman injured in the attack on Tunisia's national museum has died of her wounds, bringing the total killed in the assault by Islamists to 22.
[NEXT_CONCEPT]
Police investigating the discovery of human remains in East Lothian have spoken to the family of missing woman Louise Tiffney.