Media playback is not supported on this device
The team finished second but the final handover between Daniel Talbot and Adam Gemili, 18, was not completed in time.
Gemili said: "Maybe I went early or too hard. It's disappointing as we could have been in contention in the final."
"It looked like Gemili went a little bit early. There could have been a situation where Adam was just so amped up and excited by this crowd, this moment and this opportunity that he was running a little bit faster than the mark was made for."
Favourites Jamaica rested Usain Bolt but still won heat one, while, in the second, the United States recorded the fastest overall qualification time.
Britain, who won this event at the 2004 Games in Athens, were disqualified in Beijing four years ago and have now made a mess of baton changes in five of the last six major championships.
Talbot said: "I don't know what happened. I just couldn't catch Adam. I'm really gutted and it's devastating."
Christian Malcolm, who ran the first leg for Britain, described the outcome as "unfortunate". He added: "It's one of those things that happens but it's a fantastic performance to run the time we did. Daniel has come in at the last moment. This is a big stadium and it can be overwhelming."

Baxter, 24, played the last of his 93 league games for United in February, with the ex-Everton trainee then banned for an unspecified reason by the club.
In July 2015 he was suspended by the Football Association for failing an out-of-competition drugs test.
Midfielder Ryan Flynn, 24, who scored 13 goals in 153 league games, has also been released.
Meanwhile, Dean Hammond, who made 34 Blades appearances during his loan spell from Leicester City in the 2015-16 season, has activated a clause in his contract to remain at Bramall Lane on a permanent deal.
However, the 33-year-old has immediately been placed on the transfer list alongside Kieron Freeman, Kieran Wallace, James Wallace, Paul Coutts, Martyn Woolford and Diego De Girolamo.
Goalkeeper George Long has been offered a new deal while Alex Baptiste, David Edgar and Conor Sammon have left the club after their loan deals expired.
Mark Howard, Bob Harris, Callum McFadzean, Terry Kennedy, Harrison McGahey, Jay McEveley, Florent Cuvelier and Jamal Campbell-Ryce have all been released.
Great Britain will miss the men's Olympic 4x100m relay final after being disqualified in their semi-final.
[NEXT_CONCEPT]
Forward Jose Baxter, who was suspended twice in nine months, is among 10 players released by Sheffield United.