Police said the elderly man was spotted in the water at Leith at about 22:00 on Wednesday.
He was taken to Edinburgh Royal Infirmary by ambulance for treatment, but later died.
A Police Scotland spokesman said inquiries into the circumstances surrounding this incident were ongoing.

The 42-year-old had regained the lead from Frenchman Armel Le Cléac'h, but was second as he passed the halfway mark just off New Zealand's south island.
Thomson's next milestone is to turn north at Cape Horn as he heads towards the finish port of Les Sables d'Olonne.
Gosport-based Thomson  is aiming to be the first Briton to win the race.
Media playback is not supported on this device
"There are a few hectic days to come and I will certainly be approaching them with caution to make sure that I keep Hugo Boss (Thomson's boat) in one piece," he said.
"It's over 1000 miles behind me to third place, so at least I have some breathing space behind so I can worry about looking forwards, which I think is really important".

The incident occurred in the 84th minute of Wednesday's Confederation Cup match between Sudan's Al Merreikh and Kawkab Marrakech of Morocco.
Caf is reviewing the incident and awaiting the match officials' report.
Kawkab won the match 2-0 - with two penalties - for a 2-1 aggregate win.
Caf director of communications Junior Binyam told BBC Sport: "Investigations are being carried out and the the case will be submitted to the disciplinary jury."
Kawkab were 1-0 up in Wednesday's second leg of the play-offs tie - with the score 1-1 on aggregate - when a number of Merreikh club officials entered the field.
In the ensuing melee, Al Merreikh's director football Hatim lashed out with his left arm and seemed to strike Otogo-Castane on the right side of his head.
After play resumed, Kawkab went on to win an 89th-minute penalty which they converted to earn their place in the group stage of Africa's second tier club competition.
A man has died after being pulled from the water near the Royal Yacht Britannia at Ocean Terminal in Edinburgh.
[NEXT_CONCEPT]
Welsh sailor Alex Thomson has passed the halfway point of the solo non-stop round-the-world race, the Vendee Globe.
[NEXT_CONCEPT]
Al Merreikh official Hatim Mohamed Ahmed is being investigated by the Confederation of African Football after he appeared to hit Gabonese referee Eric Otogo-Castane in the face.