"In full co-ordination with the government of Iraq, we're deploying a specialised expeditionary targeting force to assist Iraqi and Kurdish Peshmerga forces," Mr Carter said.
The operators would carry out raids, free hostages and capture IS leaders.
They would also conduct "unilateral operations" in Syria, he said.
Islamic State conquered large swathes of Iraq and Syria in the summer of 2014.
A US-led coalition of 65 countries is conducting a campaign against the militants in both countries.
Mr Carter made the announcement in remarks to the House Armed Services Committee in Washington on Tuesday.
He did not offer specifics on troop numbers, but said they would also be focused on securing Iraq's borders and building the capacity of the Iraqi government forces.
US special forces have already conducted operations in both Syria and Iraq - notably the killing of the IS oil chief Abu Sayaff in eastern Syria and the freeing of prisoners from the IS-held town of Hawija near Kirkuk in Iraq.
But this is a clear indication from the Pentagon that their role will become more significant.
The "specialised expeditionary targeting force", as the Iraq group is being called, will not only assist local forces but will also conduct further "unilateral" raids into Syria.
While this does not amount to significant numbers of boots on the ground, it is an escalation of America's role in direct combat.
Mr Carter said there was now an important opportunity to divide IS militants in Iraq from those in Syria.
In October, the US announced it would send up to 50 special forces soldiers to northern Syria as military advisers to Kurdish and rebel forces.

The European research centre announced its spoof discovery with pictures showing its scientists using The Force in everyday life.
It was one of many April Fools jokes seen on websites around the world.
Others included a selfie shoe, driverless pizza delivery and thought-powered web search.
"The Force is what gives a particle physicist his powers,"  said Cern scientist Ben Kenobi of the University of Mos Eisley, Tatooine in a press release.
Many scientists at the centre were already using The Force, said the release, to communicate over long distances, influence minds and for "lifting heavy things out of swamps".
Microsoft used the background image for its main Bing search page to tout its April Fools joke in which it claimed the search system could read palms and minds to ensure people got the results they really wanted.
Anyone could try thought-powered search by putting their right hand on a screen where the Bing homepage indicated.
"The innovative new search function intelligently analyses the electrical signals transmitted through the hand to calculate the relevant search term," said the software firm.
Many other firms and sites produced spoof announcements and products for 1 April.
Smartphone maker Samsung produced a spoof page for a Blade edge version of its Galaxy smartphone that, it claimed, was designed for cooks. The limited edition handset incorporates a diamond-edged blade so the phone can also be used to chop food when it is not being used for calls, texts, or browsing the web.
HTC joined in with fake product pages for the Re-Sok - the "world's first truly smart sock". The technology-enhanced footwear is engineered with GPS so pairs of socks can easily find each other and have an automatic hole warning system to alert owners when their intelligent footwear is running thin.
For its April Fools parody, Motorola went to the trouble of making a lavish video showing two craftsmen producing selfie-sticks out of wood and leather.
New York-based footwear maker Miz Mooz joined in and produced a pair of shoes that have smartphone docking ports in each toe so each one can be used to take selfies.
Other parodies included cab-calling firm Hailo introducing piggy-back rides around cities; Domino's pizza introducing a driverless pizza delivery system using autonomous scooters; BMW making the ultimate rugby mouth guard and Sony making a wearable add-on for the PlayStation so players can take part in games when they are swimming.

Crews were called to a building site in Bloomery Circle, Llanwern, at 21:15 BST on Wednesday.
The blaze burned for more than 10 hours before it was extinguished.
It is understood the burning material included a large amount of scrapped railway sleepers.
A Gwent Police spokesman said: "Officers are investigating the circumstances surrounding the fire... which is believed to have been deliberately set."
The US is to deploy a specialised force to Iraq to build pressure on Islamic State militants, US Defence Secretary Ashton Carter has said.
[NEXT_CONCEPT]
The Force - the mysterious energy field used by the Jedi in Star Wars - has been discovered by researchers at the Cern laboratory.
[NEXT_CONCEPT]
A fire involving about 100 tonnes of waste wood and rubbish in Newport was started deliberately, the police and fire service have said.