Sea Watch said men from the speedboat boarded the dinghy and beat migrants with sticks, causing a "mass panic".
At least four of the 150 people on board the dinghy fell into the water and drowned, Sea Watch said.
A spokesman for Libya's naval forces in Tripoli said he had not heard about the incident.
The motive for the attack was not clear, but Ruben Neugebauer, a spokesman for Sea Watch, said the attackers were attempting to steal the dinghy's engine.
Mr Neugebauer said that 25 people remained missing.
Sea Watch, a German organisation, said in a statement: "The violent intervention of the Libyan Coast Guard caused a situation of mass panic on board the rubber boat in distress.
"One tube of the rubber boat collapsed, causing the majority of the 150 people to slip into the water."
About 120 people were rescued and brought aboard the Sea Watch 2 ship. Four were unconscious and receiving treatment.
Italy's coast guard confirmed that Sea Watch had been involved in some of the nine rescue operations taking place on Friday.
Both the Libyan and Italian coast guards said they had received no information about the attack yet.
Italy has taken in more than 146,000 boat migrants so far this year and is on track to see more arrivals than last year, when 153,000 arrived.
In August, the Doctors Without Borders humanitarian group, which operates rescue ships off the coast of Libya, said it was attacked by armed men on a Libyan navy boat.
The Libyan navy said it thought the vessel was involved in people smuggling.

28 October 2015 Last updated at 14:19 GMT
Scott Kelly and Kjell Lindgren had to grease the station's big robot arm, re-route some cables and remove some insulation.
The spacewalk had to be delayed after Lindgren switched on the water flow in his suit too soon.
Mission Control had to make sure the suit would be safe, before they gave the all-clear.
Water is needed to cool the suit.
Kelly who's been living at the orbiting lab since March will stay until March 2016.
But early on Thursday morning this week, he will break the American record for NASA's longest single space shot.
That 215-day record - more than seven months - was set in 2007.
President Barack Obama already has sent his congratulations in a phone call last week- telling him his time in space is  ``nothing to sneeze at.''
Kelly replied ''It shouldn't be a problem getting to the end with enough energy and enthusiasm to complete the job. Records are made to be broken.''
Earlier this month, Kelly broke the U.S. record for the most time in space: 383 days and counting over four missions. It will be 522 days in total by the time he returns to Earth.
The pair will do a second spacewalk on Nov 6th.

25 May 2017 Last updated at 07:00 BST
He met Newsround at the world famous Chelsea Flower Show, where top gardeners show off their best designs and compete for awards.
George is an ambassador for the Royal Horticultural Society, which runs the show, and he even has his own blog.
Watch his video to find out what's so great about gardening.
At least four migrants have died after a speedboat labelled "Libyan Coast Guard" attacked a dinghy, according to a migrant rescue organisation.
[NEXT_CONCEPT]
Two astronauts on board the International Space Station have taken their first spacewalk outside - to do some repairs and maintenance.
[NEXT_CONCEPT]
Meet George - he wants more children like him to get outside and get into gardening.