“Crazy” amounts of data are fueling new storage technology

2039 might seem like a long way off, but Ian Crawford is already planning for it.

It will be the 100th anniversary of the outbreak of the Second World War – a big year for his employer, the Imperial War Museum.

Mr Crawford is the museum’s chief information officer, overseeing a project to digitize its vast collection of images, sound and film.

With a collection of approximately 24,000 hours of film and video and 11 million photographs, this is a huge undertaking.

And in the period before 2039, material from the Second World War will be a priority.

Creating digital copies of these historical resources is vital because the original copies degrade over time and one day will be lost forever.

“When you have a single copy, you want to make sure your storage system is reliable,” says Ian Crawford.

The amount of data needed for such long-term storage is constantly growing, as the latest scanners can record documents and films in great detail.

“Its growth potential is really huge,” says Crawford.

“Now we’re looking at the objects themselves and scanning in 3D – that can generate very large files.”

This deluge of data is not limited to museums – it is everywhere.

Businesses are buying more space to back up data, hospitals need somewhere to store records, government needs a place to store more and more information.

“We continue to generate an insane amount of data,” says Simon Robinson, principal analyst at research firm Enterprise Strategy Group.

“Most organizations – it varies a lot – their data volume doubles every four to five years. And in some industries it’s growing much faster,” he says.

Data that needs to be kept for a long time is not stored in traditional data centers, those huge warehouses with racks of servers and flashing lights. These operations are designed for data that needs to be accessed and updated frequently.

Instead, the most popular long-term data storage method is tape. In particular, the format known as LTO (Linear Tape Open), the latest version is called LTO-9.

The cassettes themselves are not dissimilar to old VHS tapes, but slightly smaller and more angular.

Inside the cartridge is a kilometer of magnetic tape, capable of holding 18 terabytes of data.

That’s a lot – one tape can fit the same amount of data as almost 300 standard smartphones.

The Imperial War Museum in Duxford uses a tape system from Spectra Logic. The machine, the size of a large wardrobe, can hold up to 1500 LTO tapes.

Such LTO systems dominate the long-term storage market. They have been around for decades and have proven to be reliable.

It’s also quite cheap, which is important because generally customers want to pay as little as possible for long-term storage.

However, some believe it can be done better.

In a former wallpaper factory in Chiswick, west London, a start-up is developing a long-term storage system that uses lasers to burn tiny holograms into a light-sensitive polymer.

CEO Charlie Gale points out that with magnetic tape, data can only be stored on the surface, while holograms can store data in multiple layers.

“You can do things called multiplexing, where you can layer multiple sets of information into one space. That’s really kind of the superpower of what we’re doing. And we believe we can fit more information into a smaller space than ever before,” he says.

HoloMem polymer blocks can handle extreme temperatures without data corruption – between -14C to 160C.

In comparison, magnetic tape must be maintained between 16 C and 25 C, which means significant heating and cooling costs, especially in countries with extreme temperatures.

Tape also needs to be replaced after about 15 years, while polymer is good for at least 50 years.

Mr. Gale notes that because the laser chemically changes the polymer, the data cannot be manipulated after it is written.

A prototype of the Holomem system, which will be able to store and retrieve data, will be ready at the end of this year.

Mr Gale says the cost of the system has been kept down by using off-the-shelf, widely available components, including the laser – so he’s confident HoloMem will be able to match or beat the cost of magnetic tape.

HoloMem will have to be competitive as there is a formidable competitor looming over the market.

Through its research arm, Microsoft is developing its own long-term data storage system.

Like HoloMem, it decided it was time to move on from magnetic tape, but Microsoft chose glass as the storage material.

The system, called Project Silica, uses powerful lasers to create tiny structural changes in glass, called voxels, that can be used to store data. Voxels are incredibly small and can be layered.

Microsoft says that a 2mm-thick piece of glass the size of a DVD would be able to store more than seven terabytes of data.

The system places glass panes on racks where they can be accessed by tiny crab robots that zip along rails.

Cheap and durable glass is an attractive storage medium, says Richard Black, who heads Project Silica.

“It’s pretty much immune to temperature, humidity, particles and electromagnetic fields,” says Mr Black.

It could potentially preserve data for hundreds and possibly thousands of years.

Such a system could one day be integrated into Microsoft’s massive cloud business, Azure.

But that’s a bit off, as the system has years of development ahead of it.

At Duxford, the Imperial War Museum, like many organisations, has been experimenting with artificial intelligence. They recently tested whether AI could identify different Spitfire models in images from their image catalog.

Mr Crawford thinks artificial intelligence could be incredibly useful in cataloging its digital library, a job that would take humans hundreds of years.

AI’s ability to sift through vast amounts of data makes it even more important to preserve that data—it could hold something valuable.

“In the past, the business was to archive data in case they needed it. Now there’s a real business reason why they might want to go back and do some analysis,” Mr Robinson says.