Mark Drakeford has already apologised for the scandal on the Tawel Fan ward at Glan Clwyd Hospital, Bodelwyddan.
It followed a damning report into what went on with families saying patients were treated like animals in a zoo.
But he said the embarrassment to the Welsh government was the "least important part" of it all.
"The real focus is to make sure the families and the people who are closely connected to what went on is that we properly attend to what they need," Mr Drakeford told BBC Wales.
He added he is yet to meet any of them but he was "hugely impressed" by the way the families have acted and they are "looking for assurances" that there will not be a repeat of what went on.
"I'm very willing to speak to the families when the time is right," he said. "This is a really tough time for these families."
'It was probably a mistake keeping him in there'
Mr Drakeford said on Thursday he will consider putting Betsi Cadwaladr health board into special measures which could mean the Welsh government or another body taking over some or all of the board's functions.
In an interview with BBC Radio Wales on Friday, Mr Drakeford said he will receive advice via a "proper process" involving the Welsh government, Health Inspectorate Wales and the auditor general for Wales which will meet to review whether further intervention is needed.
"It isn't a matter of a politician shooting from the hip, making an off the cuff decision," he said.
"We get the experts involved together and we act on the proper advice that we are given."
Meanwhile, the boss of patient watchdog North Wales Community Health Council (CHC) said the health board needed a "greater level of external support and direction".
Chief officer Geoff Ryall-Harvey said: "The avalanche of adverse reports over the past few months has been unprecedented in my experience of 30 years as a CHC chief officer."

Artis Sterkis, 37, collapsed on the Peterhead-registered Starlight Rays.
He was taken to hospital by rescue helicopter after the incident in August 2011, but he never regained consciousness.
James Thores was fined at Elgin Sheriff Court after admitting a fishing vessel health and safety breach.
The £20,000 fine was reduced from £30,000 because of the guilty plea.
Starlight Rays featured in the BBC television series Trawlermen, and was performing stand-by duties for the oil industry when the incident happened in the Devenick field, 155 miles off Aberdeen.
An earlier Marine Accident Investigation Branch (MAIB) report said the pump's petrol engine ran for more than an hour in a compartment with no mechanical ventilation and little natural air circulation.
Speaking after Thursday's sentencing, Captain Bill Bennett, of the Maritime and Coastguard Agency, said: 'The dangers of using of a petrol engine in a confined space are well known."

Enninful, who has previously worked for Italian and American Vogue, will take over from Alexandra Shulman, who announced her departure in January.
Condé Nast International's Jonathan Newhouse said Enninful was "an influential figure in the communities of fashion, Hollywood and music".
He is currently the fashion and creative director at W Magazine, where he has worked since 2011.
The 45-year-old will start his new role on 1 August.
After the news was announced, designer Marc Jacobs tweeted: "Congratulations to our longtime friend @Edward_Enninful, the new editor of British Vogue! We can't wait to see what you do! ❤️"
Caroline Rush, chief executive of the British Fashion Council, told the BBC: "Anything that promotes diversity and gender equality in the industry is incredibly important, but there's no doubt that Edward's been appointed because of his talent."
"He brings a very distinct point of view in terms of the imagery that he has been associated with, and I think that's going to be really exciting to see how that is translated into the pages of Vogue."
Follow us on Facebook, on Twitter @BBCNewsEnts, or on Instagram at bbcnewsents. If you have a story suggestion email entertainment.news@bbc.co.uk.
The families of patients who suffered "institutional abuse" at a mental health ward in Denbighshire will have a chance to meet the health minister.
[NEXT_CONCEPT]
A fishing boat skipper and owner has been fined £20,000 after a crewman died from carbon monoxide poisoning while trying to pump out water.
[NEXT_CONCEPT]
British Vogue has confirmed Edward Enninful as its first male editor.