Ms Yellen said current valuations were "quite high", but added she did not see any bubbles forming.
Chinese shares saw the biggest falls in the region with the Shanghai Composite closing down 2.8% at 4,112.21 - marking a third day of declines.
Hong Kong's Hang Seng was down 1.3% at 27,289.97 - falling for the sixth day.
Several companies on the mainland have announced share offerings through the week, leading investors to pull funds from other stocks.
Investors were also worried about reported new moves by regulators to cool China's market, such as capping the size of margin financing and limiting the type of stocks investors can buy with borrowed money.
In Japan, the benchmark Nikkei 225 closed down 1.23% at 19,291.99 after being closed all week for a holiday.
Australia's benchmark S&P/ASX 200 index closed down 0.82% at 5,645.70.
Shares in Australia had suffered their biggest one-day fall in more than two years on Wednesday following disappointing results from Australia's biggest banks.
Investors are concerned about the country's big four banks facing slower growth.
National Australia Bank (NAB), the last of the big banks to announce their results this week, reported a 5.4% rise in cash profit for the six months to March.
It also announced a 5.5bn Australian dollar ($4.4bn; Â£2.8bn) capital-raising exercise which will involve the selling of new shares.
NAB wants to demerge and float its troubled British bank, Clydesdale, and shore up its balance sheet.
The lender's capital-raising announcement has been reported by some of Australia's media as being the largest in the country's corporate history.
Also weighing on investor confidence were the latest figures on Australia's unemployment rate, which showed a slight rise to 6.2% in April.
In South Korea, the benchmark Kospi index closed down 0.65% at 2,091.00.

Jamie Taggart, from Argyll and Bute, failed to return from a plant-hunting trip to the northern mountainous area.
He has not been seen since 2 November, when his rucksack and passport were found at a guest house in Sapa.
Mr Taggart, 41, runs Linn Botanic Gardens in the village of Cove on the Rosneath peninsula and serves the area as a retained firefighter.
Local police and the British embassy in Hanoi have been informed about his disappearance.
Friend and fellow botanist Ian Sinclair, who has been liaising with the search parties, said they had been hampered by horrendous weather.
"The snow is continuing to melt, but there is a massive amount of snow damage to the forest such as collapsed trees," he said.
"The weather has been horrendous, with snow down to Sapa. This is extremely unusual as Sapa is upper tropical rainforest.
"A large group from the national park will be doing an intensive search once the snow melts."
A search due to begin on Tuesday will concentrate on four trails inside the forest of Lai Chau and Lao Cai, he said.
Family and friends of Mr Taggart in his hometown of Cove have issued an appeal for funds to help cover the cost of the search.
A Facebook page - Jamie Taggart Search - has also been set up.
Organisations and individuals have so far raised thousands of pounds, including six-year-old Annie Ferguson of Cove who donated her Â£14.50 savings.
The botanist's father, Jim Taggart, found out his son was missing when he failed to appear on a scheduled flight home to Scotland on 29 November.
He had arrived at a guest house in Sapa on 30 October and left on a motorbike taxi to explore the hills.
On 2 November his rucksack and passport were found at the accommodation.
Dr Taggart previously told BBC Radio Scotland that he had received a handful of text messages from his son before contact stopped.
"I don't think he had got lost. Either something happened to him on his first day on the hills or there is some explanation we can only guess at," he said.
A Foreign and Commonwealth Office spokesman said: "We are still in close contact with the local authorities and are providing consular assistance to the family at this difficult time."
Asian shares fell on Thursday, tracking losses seen on US markets after Federal Reserve chair Janet Yellen expressed concerns over share valuations.
[NEXT_CONCEPT]
The search for a Scots botanist missing in Vietnam is expected to resume later after being abandoned in heavy snow.