Chartoff's daughters confirmed he died at home in Santa Monica surrounded by his family. He had pancreatic cancer.
"He was an amazing father and person," said Julie Chartoff, adding: "There are no words."
He won the best picture Oscar for the first Rocky film in 1977, which was written by its star Sylvester Stallone.
The producer shared the Academy Award - one of three for the 1976 film - with partner Irwin Winkler.
They both went on to produce all seven Rocky movies, including the forthcoming film Creed, which is currently in post-production.
Chartoff was also nominated for an Oscar in 1981 for Raging Bull, Martin Scorsese's darker take on the sport, which starred Robert De Niro as boxer Jake La Motta.
He produced films into his late 70s and early 80s, including 2014's The Gambler with Mark Wahlberg and 2013's Enders Game with Harrison Ford.
He also worked on Jane Fonda's They Shoot Horses, Don't They? and 1983 film The Right Stuff, which won four Oscars.
Chartoff and Winkler - who worked together until 1985 - convinced United Artists to make Rocky for less than $1 million (Â£646,000), after Stallone insisted on playing the lead, according to Variety.
The film went on to earn 10 Oscar nominations and take more than $200m (Â£129m).
Chartoff is survived by third wife Jenny Weyman and his five children.

Walter Oakley received a letter from the Russian and US leaders thanking him for his help in televising the meeting.
No agreements were signed at the summit but it was said to mark the end of the Cold War.
Mr Oakley from Low Fell, Gateshead, was holidaying when CNN asked him to help organise filming.
Newcastle auctioneers Anderson and Garland said the value is impossible to know.
Mr Oakley came across the letter again while researching a book about his family and has decided to sell it so more people can see and enjoy it.
His wife Diane said the letter has meant a lot to the family but it belongs in a collection or museum.
She said: "No matter how busy they were, they had the time to sit down and write the letter.
"He was Walt to both of them. He said they were both very nice guys, very easy to get along with. You wouldn't think they were the presidents of the two most powerful countries in the world."
Auctioneer Fred Wyrley-Birch said: "It's impossible to say how much the letter is worth because something like this has never been up for sale before to my knowledge.
"We'll start the bidding at Â£1,000 and see where it goes from there.
"Looking at the political situation between the US and Russia at the moment, it's a very interesting glimpse back to 27 years ago when there were increasingly friendly relations."
The letter will be auctioned during the Anderson and Garland's Fine Art sale from 22 to 24 March.

The 20-year-old has played seven games for the Swans since making his £1.75m move to the Liberty Stadium from League Two Exeter City in January 2015.
Grimes made his only Premier League start to date in September, having scored his first Swansea goal in the League Cup a month earlier.
His last appearance came in their 3-2 FA Cup exit against League Two Oxford.
Find all the latest football transfers on our dedicated page.
Oscar-winning producer Robert Chartoff, who was behind hit boxing films such as the Rocky series and Raging Bull, has died in California, aged 81.
[NEXT_CONCEPT]
A letter sent to a TV engineer by Mikhail Gorbachev and George Bush after their 1989 summit in Malta is being auctioned.
[NEXT_CONCEPT]
Blackburn Rovers have signed midfielder Matt Grimes from Premier League side Swansea on a three-month loan deal.