Chief inspector of schools Meilyr Rowlands told AMs it was partly due to the "unintended consequences" of how schools' performance is measured.
Another Estyn witness said more able learners' progress had been "stifled".
Ministers agreed they needed "to do more to push our most able students" and were introducing changes to raise standards "for all our young people".
The Welsh Assembly's Children, Young People and Education Committee was quizzing Estyn about its annual report, on Wednesday.
Mr Rowlands said that schools focused on a performance measure which requires pupils to get five GCSEs at grade A*-C, including English or Welsh and maths.
That, he said, encouraged schools to focus on the borderline between C and D grades "and possibly not giving enough support for those who could get As and A*s".
But he said that evidence from primary schools, further education and the PISA international tests also suggested that able and talented pupils were not given enough support in the education system.
"I don't think schools and the education system more generally have done enough," he said.
Claire Morgan, Strategic Director for Estyn, said the focus on particular performance indicators meant there is "some gaming, [and] early entry that hasn't helped to stretch the more able".
"I think this has stifled the progress of more able learners," she added.
A Welsh Government spokesman said ministers recognised "we need to do more to push our most able students to reach their full potential".
"Through programmes such as the Seren Network, we are already supporting our brightest students to reach our leading universities.
The spokesman said changes were being introduced to the system used to measure schools' performance for 15 to 16 year olds, following a review of qualifications for 14 to 19 year olds.
The changes were intended to "raise standards and extend opportunities for all our young people no matter what their background", he added.

The 20-year-old was shot in Malvern Road in Kilburn, north-west London, on Friday night and later pronounced dead at the scene.
It is believed she was standing with a group when they were "approached by a number of suspects possibly on pedal cycles", a Met Police spokesman said.
A shot was fired at the group and the woman was killed.
The victim's family has been notified and a post-mortem examination is due to take place later.

Rana Rahimpour was told by US authorities she could not travel under a visa waiver scheme.
Under new laws, dual citizens from several nationalities must instead apply for a visa at the US embassy.
But Ms Rahimpour said advice from the embassy on whether the rules had been implemented was unclear.
The UK foreign office website says the rules come in on 1 April, while the US State Department website says the authorities have "begun the process of implementing changes" to the waiver scheme.
Dual citizens fight new US visa rules
It was only after contacting the agency overseeing the waiver while at the airport, that Ms Rahimpour - a presenter at the BBC's Persian service - was told her nationality excluded her from travelling.
Two of her cousins were also prevented from flying. They had all been due to travel on the flight from Heathrow to a family birthday in the US.
Ms Rahimpour described the new rules as "very unfair", pointing out they restrict Iranians who have gained their nationality through marriage or parentage.
She tweeted: "My fully #British daughter can't attend her #American cousin's bday cos her mum was born in #Iran."
The new rules, which also affect Iraqi, Syrian and Sudanese dual citizens, have sparked anger.
They were enacted in the wake of raised security concerns following the Paris attacks, but a joint statement from European ambassadors warned of dual nationals being "disproportionately and unfairly affected".
More able children are not getting enough attention in Wales' education system, the schools watchdog has said.
[NEXT_CONCEPT]
A woman was shot dead by attackers who may have been on bicycles, police have said.
[NEXT_CONCEPT]
A BBC journalist with dual British-Iranian nationality has been prevented from flying to the US after falling foul of changes to visa rules.