The latest protests are a reaction to him sacking two finance ministers last week, further damaging confidence in the economy.
This comes on top of claims of widespread corruption, recent student protests and a succession battle in the governing African National Congress.
The marchers are using the hasthag #ZumaMustFall on social media.
The #ZumaMustFall hashtag is a modification of #FeesMustFall which was used to organise protests against a rise in university fees.
Before that, #RhodesMustFall was used to demand a statue of British colonialist Cecil Rhodes was removed from the University of Cape Town's campus.
"It feels like the start of something big," a woman in her 50s called Lianda told me.
It's the first protest she has been on since she saw friends and family killed during the 1976 Soweto uprising.
For her it was important to join the demonstrations with her teenage daughters "for the sake of their future". So along with thousands of others she came to register her displeasure at the ANC leader, who some increasingly see as a political liability.
But unlike the groundswell of protest that led to the recall of President Thabo Mbeke in 2008, Jacob Zuma still appears to have support within the National Executive Committee of the ANC.
Earlier this week, the party's top brass went on TV to deliver a message to a South African public still reeling from the events of the past week, that they still support their president.
Yet privately there is said to be considerable disquiet within the ANC.
This may not be an Arab Spring but watch this space. The discontent is growing louder.
The ANC has been in power since the end of apartheid in 1994 and won a landslide in general elections last year.
However, the economy is struggling, unemployment is around 25% and many accuse ANC officials of corruption.
Mr Zuma's decision to appoint the little known Des van Rooyen as finance minister, before replacing him four days later, was widely condemned.
The appointment led the rand to tumble to record lows, although it rose after he was replaced by the respected Pravin Gordhan on Sunday night.

The Hull 2017 Culture Company said fundraising had exceeded its public target of £18m, raised with investment from 61 partners.
The full programme of events is to be announced next week.
Martin Green, of Hull 2017, said the money was "a massive vote of confidence" in the city.
The chief executive and director of Hull 2017 said Hull was a "great city" and "the northern powerhouse argument was a good one".
Mr Green admitted his personal target for funding had been about £30m.
He was the man behind the London 2012 Olympic ceremonies, oversaw the torch relays ahead of the Olympic and Paralympic Games in 2012 and was the executive producer of the Tour de France's opening ceremony in Leeds.
About £22m will be spent on cultural events and another portion of the money will support events after 2017, the charity said.
More than 2,000 people have already signed up to volunteer during the year and 60,000 school students will take part in some way.

McManus netted from close range on the quarter-hour mark to open the scoring and Ross Forbes' deflected strike quickly doubled Morton's lead.
A shot by Kyle McAusland pulled Alloa back into the game before half-time.
But another McManus finish and one by Denny Johnstone late on ensured a comfortable win for the hosts.

A person claiming to be in a vehicle on the A414 at North Weald, Essex, called police at 08:30 BST on Wednesday to say it had earlier been in Ashford in Kent.
The lorry was found by Bedfordshire Police at Toddington Services, on the M1, in the afternoon.
Six people are being held in custody.
Supt Trevor Roe, from Essex Police, said it was a "positive outcome".
"We've seen with human trafficking....the events could turn out to be very tragic so I think it was important we reacted fairly quickly across our neighbouring forces to try to locate the vehicle," he said.
Thousands of protesters are marching in South Africa to demand President Jacob Zuma is sacked.
[NEXT_CONCEPT]
Hull City of Culture 2017 has raised £32m to produce the year-long festival, according to the charity set up to deliver it.
[NEXT_CONCEPT]
Declan McManus scored twice as Greenock Morton beat Scottish Championship bottom side Alloa Athletic, who remain 10 points adrift.
[NEXT_CONCEPT]
Four suspected illegal immigrants are on the run after a lorry containing ten people was discovered by police following a "distress" call from one of the people inside.