An area of the town near the swimming pool had been evacuated for most of the morning and early afternoon.
Other residents close by had been advised to open windows, stay indoors near walls and to avoid going outside.
An exclusion zone remains in force on the river while experts make safe the sea mine which was spotted on Monday.
It is thought to be a British mine, dating from the  1940s or 1950s.
The evacuation affected three streets, with wider safety advice issued to residents within 500m of the device.
There was also an exclusion zone at sea of about half a mile, and small boat owners were told not to sail during the operation.
Caledonian MacBrayne reported some disruption to ferry services, affecting sailings from Gourock and Rothesay.
Ch Insp Elliot Brown, the local area commander for Inverclyde, stressed that the evacuation had been a precautionary measure, and that there was no immediate danger to the community of Gourock.
Lt Cdr Tim Castrinoyannakis, the officer in charge of the Royal Navy's Northern Diving Unit, said that due to the deterioration of the mine it was not possible to conclude how much explosive material it still contained.
He said that because of this it was decided to move it to a safer place before dealing with it.
Inverclyde Council said: "Thanks to residents and businesses in Gourock for their patience while the cordon and safe evacuation procedures were carried out today."

Mark Gilmore was suspended in June 2014 by the county's police and crime commissioner (PCC) in connection with an investigation into the awarding of vehicle contracts in Northern Ireland.
PCC Mark Burns-Williamson said prosecutors had concluded there was no criminal case for the chief constable to answer.
However, Mr Gilmore will not return to his post immediately.
In a statement, Mr Burns-Williamson said: "There remains however a legal requirement for me to consider conduct matters in relation to the police standards of professional behaviour and an independent investigation will be carried out by Lancashire Police.
"While this conduct investigation takes place Mark Gilmore has agreed to work on a transition project for the National Police Chiefs' Council (NPCC) and to delegate day-to-day operational control of West Yorkshire Police to his deputy Dee Collins who has, in his absence, acted as the temporary chief constable."
The Police Service of Northern Ireland (PSNI) had arrested a total of seven men on suspicion of offences including bribery, misconduct in public office and procuring misconduct in public office.
Mr Gilmore, originally from Belfast, joined the Royal Ulster Constabulary, the predecessor of the PSNI, in 1983.
He was appointed Chief Constable of West Yorkshire in April 2013.

Sally Shalloe, 51, of Attenborough, Nottinghamshire, died when her bicycle was hit by David Staley's motorbike in Middleton Boulevard in April last year.
At Nottingham Crown Court, Staley, 32, admitted causing Ms Shalloe's death by dangerous driving.
Staley, of Arnold, Nottingham, is due to be sentenced on 13 May.
Ms Shalloe worked as a research fellow at the University of Nottingham, and husband Jim Hill said his wife previously produced studies on road safety.
"She was cycling to work that morning," said Mr Hill, who had been married to Ms Shalloe for 16 years.
"I had a policewoman walk into my work office and tell me she'd been killed. It was like being split in two with an axe.
"It's turned my life upside down completely. I've got a constant feeling of loss."
Staley was disqualified from the roads before being bailed ahead of his sentencing.
Residents are returning to their homes after an operation to move an unexploded mine discovered in the Firth of Clyde.
[NEXT_CONCEPT]
West Yorkshire's chief constable has had his suspension lifted.
[NEXT_CONCEPT]
A motorcyclist has pleaded guilty to causing the death of a former road safety worker who was killed in a crash on a busy Nottingham road.