data storage and transmission

Mark Fowler recently sent me some stuff (thanks!) on an IDC report that says 280 exabytes of data was created, stored and replicated last year. That's 75% more than the year before, which in itself is interesting. They go on to say that we can't store all that because the figure exceeds storage capacity, but that doesn't matter because most of the data isn't worth storing.

So the amount of data in the world was doubling every 13 months when they wrote the report. A few years back it was doubling every 18 months, so even the rate of relative growth is increasing. Very soon it will double every year, and in ten years, it will probably double every month.

As I wrote earlier today, 1 exabyte is transmitted on the net every month now according to Cisco, 30EB/month by 2011 and up to 200EB/month by 2012. As data production rate accelerates, so will the amount of network capacity needed. Doubling network capacity every month seems ridiculously difficult to achieve.

This implies that we will start to see data crystallisation, with local creation, storage and transmission going through the roof but network capacity lagging so far behind that it will be impossible to replicate this data in other locations, and impossible to collect it globally, or even to index it globally.


I guess the real question is whether this matters. If the data is low grade stuff like local video footage or sensor data, perhaps there is no real problem if we can't network it all globally. But it certainly does put a technological damper on globalisation when we realise that the information world will start to crystallise down into ever-shrinking cells.