UK Government’s digital history preserved and made searchable by Manchester firm

The archive of the UK central government’s online presence has been indexed and digitally archived to the cloud by Manchester tech firm MirrorWeb.

The cloud-native archiving company, which is based in the city centre, devised a new portal to create a more accessible, searchable and user-friendly resource for the public.

Comprising all captured government web-published content, The National Archives’ gigantic 120 terabyte (TB) web archive encompasses billions of web pages from 1996 to the present.

It took MirrorWeb just two weeks to transfer the data from 72 hard drives at The National Archives to internal hard drives, using devices called AWS Snowballs, before transferring more than two decades of government internet history to the cloud.

The four-year contract was awarded to MirrorWeb, which was tasked with both moving the data to the cloud using Amazon Web Services (AWS), as well as indexing it.

Indexing the data meant that MirrorWeb had to write a complete replacement for the UK Government Web Archives’ previous search functionality.

As a result, 1.4 billion documents were indexed and are now accessible and searchable to researchers, students and members of the public who need to use them, enabling them to view websites and social media content in their original form as well as search for content on specific topics.

John Sheridan, digital director at The National Archives, said: “We are preserving 1,000 years of British history and a big part of that is preserving the digital record of government today.

“MirrorWeb has brought some outstanding technical capabilities, in particular data migration, cloud computing, search, new ways of harvesting and crawling content and new ways of presenting content and making it available.”

He added: “I have been most impressed by MirrorWeb’s use of cloud computing technologies.

“For example, to index the entire 120TB collection they were able to spin up 1,000 node plus cluster of computers to process the entirety of that collection, and in just a couple of days.”

To carry out the indexing MirrorWeb built its own software, WarpPipe, which has the ability to index a large number of small files and indexed all The National Archives’ documents in just 10 hours.

Philip Clegg, chief technical officer at MirrorWeb, explained: “The files within The National Archives are relatively small, but in terms of numbers the volume is huge.

Philip Clegg

“This posed a problem for the big data processing tools already on the market, which were quoting us a timeframe of six to eight weeks.

“This is why we built WarpPipe, enabling the documents to be indexed in 10 hours.”

The history of the UK central government’s online presence can now be searched by any user.

The search functionality is provided by Elasticsearch, which was chosen because it improves on The National Archives’ previous search engine in terms of speed, flexibility and reliability.

The index will eventually be updated monthly, as opposed to quarterly, giving the end-user more up to date archive content.

Mr Clegg explained: “In under a second the public-facing website can bring up results from every UK government website which has been preserved and can be viewed just as it was for any chosen date.

“In this information age, it is vital that our digital history is preserved and this resource will help educate future generations to come.”

The 120TB of data was backed up in a data centre in The National Archives across 72 USB-3 hard drives.

MirrorWeb transferred the data using devices called AWS Snowballs which connect to the local network, copy and encrypt the data to internal hard drives, and can then be shipped to an AWS data centre for transfer into the cloud.

MirrorWeb used its two custom-built computers that allowed it to move data from up to 16 of the USB-3 hard drives at a time.