A report following an unannounced inspection said the overall number of self-harm incidents at HMP Woodhill was also "very high".
It said the prison's response to the deaths had "lacked rigour".
The National Offender Management Service said a new governor was "working hard" to improve things.
The high-security jail, which is mainly used to hold remand, newly-sentenced and short-term prisoners, was inspected in January.
The report said recorded levels of assaults were "very high and double what inspectors have seen in other local prisons".
But it said the "general environment in the prison was good" and the accommodation was "reasonable".
Nick Hardwick, chief inspector of prisons, said: "Woodhill is a prison that retains great potential, particularly in terms of the resources at its disposal.
"However, as we reported last time, outcomes for prisoners remained concerning and some were very disappointing."
Michael Spurr, chief executive of the National Offender Management Service, said the governor and his staff had a "unique set of challenges".
"Woodhill holds some of the most difficult and disruptive men in the prison estate," he said.
"The governor and his staff are working hard to reduce the incidents of violence and to improve rehabilitation."

The League One club have also signed Norwich City midfielder Ebou Adams on loan until January.
Adams, 21, signed from non-league Dartford in February 2016, will be making his second loan move from Carrow Road, having had a brief three-game stint with Braintree last season.
Jules, 20, did not play a first-team game for the Royals.
But he did make 10 Scottish Premiership appearances for Motherwell during a loan spell last season, and he was a member of the Scotland Under-20 squad that finished third at the recent Toulon tournament.
Shrewsbury have now made five summer signings, having previously brought in strikers Lenell John-Lewis and Arthur Gnahoua and midfielder Jon Nolan.
Find all the latest football transfers on our dedicated page.

America Luke Richardson sent about 4,000 messages to a 14-year-old girl from Essex and met her in a hotel, Greater Manchester Police (GMP) said.
The 22-year-old had previously been dismissed by the force after he stole uniform and visited schools without authorisation.
Richardson, of Salford, was sentenced at Manchester Crown Court.
He was also given a sexual harm prevention order and will be on the sex offenders' register for life.
He admitted six counts of sexual activity with a child, one of grooming and one of taking indecent photographs of a child.
GMP suspended Richardson in January 2014 after finding he had "used his role inappropriately" while entering a school without authorisation on an independent patrol.
But, after removing his uniform and warrant card from him, Richardson was again found to have gone to a sixth form college while wearing police uniform.
On searching his home, officers found a large quantity of police uniform to the value of Â£970 and arrested him on suspicion of theft.
Subsequent allegations of inappropriate contact with children prompted a new investigation in September 2015.
Richardson's confiscated phone and computer revealed 4,000 messages had been sent to a 14-year-old girl in Essex.
She revealed they had met up in a hotel booked by Richardson.
He was then charged with a number of sexual offences and remanded in custody before being formally dismissed in April.
Supt Mark Kenny said: "I am pleased that America Luke Richardson is now off the streets and unable to harm any more young girls.
"We are continuing to investigate the possibility that Richardson may have more victims and we are appealing for anyone who may have been approached inappropriately by him online, or in person, to come and speak to police."
Inspectors have raised concerns about the number of suicides at a Buckinghamshire prison, after four "self-inflicted deaths" last year.
[NEXT_CONCEPT]
Shrewsbury Town have signed teenage Reading defender Zak Jules for an undisclosed fee on a two-year contract.
[NEXT_CONCEPT]
A former special constable who groomed and engaged in sexual activity with a child has been jailed for five years.