The European court of justice has declared the data retention directive illegal, torpedoing UK government schemes for the so-called "snooper's charter" of wide-ranging collection of phone and internet data.
In a judgment delivered on Tuesday, the ECJ, Europe's highest court, declared that the directive is "invalid" because it "interferes in a particularly serious manner with the fundamental rights to respect for private life and to the protection of personal data."
The decision, following requests from the Irish and Austrian courts, is a blow to Theresa May's plans to push through a data retention scheme which would collect and store data from UK citizens' internet and phone use for up to 12 months for later examination.
Any criminal cases which rested on acquisition of data through the directive could also be called into question, because the court decided that "the declaration of invalidity takes effect from the date on which the directive entered into force" – that is, 2006.
A Home Office spokesperson said: "We are considering the judgment and its implications carefully. The retention of communications data is absolutely fundamental to ensure law enforcement have the powers they need to investigate crime, protect the public and ensure national security.”
The ECJ said that it weighed the needs of law enforcement to be able to keep and access data, but judged that "by adopting the Data Retention Principle, the EU legislature has exceeded the limits imposed by compliance with the principle of proportionality."
The idea of a "snooper's charter" that would allow the government to order internet service providers and phone companies to collect and store data for up to two years has been a source of controversy within the UK's coalition government, with the deputy prime minister, Nick Clegg, telling the home secretary, Theresa May, in December 2012 that it was unworkable in.
The pressure group Privacy International called the ECJ ruling "strong and unequivocal", saying that "the right to privacy provides a fundamental barrier between the individual and powerful institutions, and laws allowing for indiscriminate, blanket retention on this scale are completely unacceptable.
"As the court states, it is not, and never was, proportionate to spy on the entire population of Europe. The types of data retained under this hastily enacted directive are incredibly revealing about our lives, including our daily activities and whom we have relationships with. It is right and overdue that this terrible directive was struck down."
European governments will now have to to draft new legislation on data scrutiny to help prevent serious crimes such as terrorism, while also narrowing the law's scope to conform with the verdict.
"Data retention for the purpose of investigating serious crimes is necessary and that remains the case," the German interior minister, Thomas de Maizière, told reporters after the ruling, urging quick agreement on more narrowly defined new legislation.
Germany, the most populous of the EU's 28 nations, did not implement the directive amid court challenges and domestic political differences.
The 2006 legislation required telecommunication firms to store phonecalls or online communication records for at least six months and up to two years. The data typically reveals who was involved in the communication, where it originated, when and how often but not its content.
Still, the Luxembourg-based court ruled the legislation provided "very precise information on private lives," including daily habits and social relationships that represented a "particularly serious interference with fundamental rights".
The Green European parliament lawmaker, Jan Philipp Albrecht, called the ruling "a major victory for civil rights in Europe" and insisted the data collection also "totally failed to lead to any noticeable improvement in law enforcement."
The EU court said also the legislation's failure to ensure the storage of communication data in the EU alone also represented a potential breach of privacy laws.
James Welch, legal director for Liberty, said: "Knowing the every detail of our communications creates an extremely intimate portrait of our lives. In this hugely welcome and significant judgment, the court criticised the blanket nature of the obligation to retain communications data and the lack of safeguards, particularly as to who can authorise access. Mass snooping of this type has no place in an internet-age democracy – our government must clean up its act.”