One of the men was arrested in the capital Berlin and the other in a town close to Cologne, in west Germany.
Police seized computers, mobile phones and sketches in the raids but did not find a "smoking gun", prosecutors' spokesman Martin Steltner said.
About 450 officers were involved in the raids.
One of the men detained, reported to be aged 35, had been living in a refugee shelter in the town of Attendorn, east of Cologne, and is wanted by the Algerian authorities for alleged links to IS.
Police said "investigations show that he has been trained militarily in Syria".
The man's wife, also wanted by Algeria, was detained at the same time but prosecutors said she was not a suspect in the German case.
The suspect held in Berlin was arrested on suspicion of falsifying identity documents, police said.
Two other Algerians in Germany, said by police to be "from the jihadist scene", were tracked down but not arrested.
"Our understanding is that the four men accused could have planned to carry out an attack together," Mr Steltner said.
German media said the possible targets included central Berlin tourist sites Checkpoint Charlie and Alexanderplatz.
Since January, the suspects had changed their mobile phones multiple times and communicated using instant messaging services, reports said.
Germany's security concerns have risen since the Paris attacks last year.
Officials declined to comment on a report in the Tagesspiegel newspaper, citing security sources, which said senior IS figures had ordered an attack on Germany.

The Warriors could have led by more than 11-9 at half-time.
"We had enough opportunities to be further ahead, and second half we made too many mistakes," Townsend said.
"Edinburgh took their chances. It took a bit of character and skill from us to get a chance to win near the end, but we weren't nearly good enough today."
Glasgow had the small consolation of winning the 1872 Cup for the first time in three years, prevailing 43-41 on aggregate after winning the away leg 25-12 at Murrayfield on Boxing Day.
"[That's no consolation] at the moment, but in future that's something on record," Townsend told BBC Scotland.
"Our players are obviously really disappointed, they wanted to finish the season with a win, and it shows how competitive these fixtures are.
"I thought Edinburgh played really well in defence, and really stopped us creating quick ball. That's a credit to them, and they took their chances really well as well, so they deserved to win today.
"Even though we were getting penalties a lot around the breakdown we didn't convert our opportunities with the line-outs we had close to the line."
After reaching the Pro12 semi-finals for the past four seasons - reaching the final in 2014, and winning the title in 2015 - Glasgow had to settle for sixth place in Townsend's final campaign, but reached the European Champions Cup quarter-finals for the first time.
Townsend addressed the Scotstoun crowd after the final whistle, thanking fans for their support, and will name his first Scotland squad on Monday as he takes over the national job for the forthcoming southern hemisphere tour to face Italy, Australia and Fiji.
"I'll miss it a lot," he reflected of his final Scotstoun experience as head coach. "We've got the best supporters in the world and we're just disappointed today we weren't able to give them the performance they deserve.
"Just now I'm still a bit down, but I'll miss coming to work, I'll miss working with this group of players and I'll enjoy coming back and watching, seeing how this club continues to progress, getting more and more people watching and also playing some really good rugby."
German police have arrested two Algerians suspected of planning an attack and having links to the militant group, Islamic State (IS).
[NEXT_CONCEPT]
Departing Glasgow coach Gregor Townsend rued his side's inability to take their chances after slipping to defeat by Edinburgh in his final game in charge.