The US network said it was "not an easy decision" and had "enormous respect" for its creators and cast.
Set in the 1970s, the series told of a charismatic record executive trying to revive his struggling label.
Its ensemble cast included Bobby Cannavale in the lead role, Olivia Wilde as his wife and Sir Mick's son James as a rebellious British singer.
Sir Mick and Scorsese were among the show's executive producers, with the latter directing its pilot episode.
The series premiere in February was watched by 764,000 viewers, though the average audience subsequently dipped to 650,000.
"After careful consideration, we have decided not to proceed with a second season of Vinyl," HBO said in a statement.
"We have enormous respect for the creative team and cast for their hard work and passion on this project."
Ray Romano, another actor in the cast, told an Australian newspaper on Thursday the cancellation felt like "a knife in the guts".
Wilde, meanwhile, thanked her Twitter followers for their "kind words" and said she had been part of "something special".

Strathspey Steam Railway needs a crossing on the River Dulnain so it can offer journeys between Aviemore and Grantown-on-Spey.
The group runs locomotives along 10 miles (16km) of line between Aviemore and Broomhill.
It needs three more miles of track (4.8km) to link up with Grantown.

Lewis, 19, signed a three-year contract with the English Premier League club on Friday, having played three FA Cup games for them last season.
Manager Ray McKinnon has also given a contract to Deniz Mehmet following his departure from Port Vale.
Hibernian midfielder Sam Stanton has also made the move to United on a one-year loan deal.
The arrivals come with Cammy Bell, last season's first-choice goalkeeper, possibly leaving Tannadice to return to Kilmarnock.
Bell, 30, is believed to be keen on a move to the Scottish Premiership club.
McKinnon said of Lewis: "I expect him to challenge for the starting position.
"Harry comes here as one of the top young goalkeeping prospects in English football.
"It says so much about his ambition and desire that he has chosen to come up to Scotland to get first-team involvement.
"It is a coup for us to bring him to Tannadice and I am positive we will hear much more about him during his career."
Lewis, who has represented England at youth level, joined Southampton from hometown club Shrewsbury Town in 2015, hopes to challenge for starts with the Scottish Championship club.
"I am excited at the prospect of being involved in first-team football," he told United's website.
"I believe I am joining a winning team that will be challenging at the top of the league and want to play my part in helping the club secure the title."
Stanton becomes United's ninth summer signing and their fourth of the week.
The 24-year-old Mehmet returns to Scottish football after having played with Falkirk before joining Port Vale in January.
Dutch midfielder Jordie Briels also signed on a one-year contract after the 25-year-old left Fortuna Sittard in his homeland.
Sir Mick Jagger and Martin Scorsese's music business-based drama Vinyl has been cancelled by HBO after one season.
[NEXT_CONCEPT]
Railway enthusiasts have hired a powerful crane to lift the lower sections of a new bridge into place across a river in the Highlands.
[NEXT_CONCEPT]
Dundee United have signed their second goalkeeper in a week by bringing in Harry Lewis on loan from Southampton.