Australian internet firms are wiping some customer "metadata" immediately or within days, causing problems for law enforcement including an investigation into a potential child rape that had to be abandoned.
Examples provided to Fairfax Media of how metadata is used by the police include one case in which the Australian Federal Police had to ditch an investigation into an internet user who had said online that they planned to sexually assault a baby.
In June last year, Interpol provided IP address details relating to the online comment to the AFP, but the suspect could not be identified because the Australian internet provider kept such metadata for a maximum of seven days.
Details of the case were provided by Attorney-General George Brandis' office. The Abbott government is pushing for a scheme that would compel providers to keep such metadata – which includes the time and place of communications and certain internet usage but not the content – for at least two years.
"Both the AFP and ASIO have urgently advised me that inconsistent retention, and even unavailability, of this data is hampering investigations and, in some tragic cases, preventing perpetrators from being brought to justice," Senator Brandis told Fairfax Media.
"No responsible government can sit by while those who protect our community lose access to the tools they need to do their job."
Police and intelligence agencies say it is essential to most major investigations, including terrorism, child exploitation and cybercrime.
In another major child pornography case, called Operation Drakensburg, the AFP was unable to pursue nearly one third of 522 Australian IP addresses that may have accessed illegal material on a British website because the internet providers had not kept the metadata.
"In such cases, telecommunications data is the only avenue of inquiry that may establish the identity of offenders," an AFP spokeswoman said.
She said that in some cases, metadata was kept "for as little as less than a week, while others are storing data for up to two years".
John Stanton, CEO of the Communications Alliance, the peak body for the telecommunications industry, said it was "true there is great variability" in the lengths of time for which providers kept metadata.
But he said companies generally didn't keep data they didn't need for commercial purposes such as billing – and to do so might put them at odds with privacy laws.
"Historically it's been frowned upon under the Privacy Act to retain data you don't need," he said.
The Communications Alliance has indicated providers broadly keep phone data for at least two years already but says keeping data such as IP addresses, text message metadata and mobile phone location data would be an additional burden and cost.
Mr Stanton said the industry broadly regarded six months as sufficient for things like IP addresses, as the "vast majority" of requests by authorities involved data less than six months old.
Alastair MacGibbon, a former AFP officer now with the Centre for Internet Safety at the University of Canberra, said the telecommunications market had become "increasingly fragmented" with more providers and newer types of services tending to keep less data and for shorter periods.
And while privacy issues were important, victims of crime also had to be considered.
"Try telling a victim of crime that their matter wasn't able to be progressed because something as simple as metadata wasn't kept," he said.
He said the cost of keeping extra data had been "overplayed" by the industry.
