Accelerated evolution

It is now more than three years since the UK’s National Archives first launched its digital archive. But as David Thomas, the organisation’s head of government and technology, tells Enterprise Information, the innovation continues.

By Jessica Twentyman

Enterprise Information (EI): In recent years, digital technology has transformed the worlds of records management and archiving. How is the National Archives rising to that challenge?

David Thomas (DT): Over the next few years, the National Archives will gradually evolve from an institution whose main medium of operation is paper-based to one that mainly deals in electronic records. We already have a working digital archive, which makes records available on our website. This was launched in April 2003 and its launch marked the first time that a digital repository, successfully storing and making available electronic records of government held here at the National Archives, was available online to our many users around the world. The digital archive’s holdings include the records of a number of high-profile public inquiries, departmental websites and the records of parliamentary committees and royal commissions.

Electronic records in an enormous variety of formats are stored, including office documents, applications, databases, virtual-reality models and audio-visual materials. The challenge since then has been to ensure that the digital archive continually evolves in line with the way we’re headed as an organisation. We have a project board for the digital archive. That team is responsible for putting into action the ideas that arise both from internal discussions and from records managers all over the world.

In particular, we are currently working on three approaches that will enable us to make a greater number of records more accessible to our audiences worldwide: increased automation, enhanced search and the inclusion of social-networking capabilities.

EI: Let’s take those one by one. What exactly do you mean by automation?

DT: Well, this is where our ‘Seamless Flow Programme’ comes in. The fact is that most government records are now created electronically as a result of the widespread introduction of electronic records management systems. Previous legislation meant that the bulk of records were not transferred to the National Archives until they were 30 years old.

However, with the introduction of the Freedom of Information Act, the principle of ‘closed until 30’ disappeared in January 2005. We now need to make arrangements to select and preserve such records as soon as possible after their creation since, unlike paper, they are very vulnerable to corruption and loss.

At the moment, electronic records created in government departments are selected and transferred to The National Archives where they are stored in our digital archive. That process is quite labour-intensive. Since we expect the volume of electronic records transferred to The National Archives to increase dramatically over the next few years, the only way we can manage this is by automating as many of the processes as possible and ensuring that other processes – such as selection, exemption and identification of records – take place either at the point of creation in the government departments or as soon as possible after creation.

Automation will be the key to this. We already know that we will not be able to absorb the increased volume of electronic records simply by scaling up our existing semi-manual processes. Only a complete re-engineering of the workflows and a major investment in automation of the processes and their underlying technical infrastructure, as envisaged in the Seamless Flow Programme, will provide a solution.

The programme is about linking together existing components and automating manual processes. The process of developing the Seamless Flow approach will enable us to review and streamline other aspects of our architecture. To a large extent, we are focusing on an incrementally staged introduction, built where possible on existing technology components and adhering to open technology standards.

One of the major stages in the Seamless Flow Programme was the introduction of global search capabilities – and this is now completed.

EI: So how have you introduced these global search capabilities?

DT: We recently completed an implementation of new search technology from [enterprise search software vendor] Autonomy. This will enable the public to perform online searches for records far more easily. The system enables people to search more than 30 million online public records dating back 1,000 years, by subject, for the first time. The reason this is so innovative for us is that, until recently, if you wanted to search the entire contents of our website, there were six or seven places you could look, as the website is underpinned by many different databases.

First, you would have had to understand the structure of the databases we had on the website and, even then, you would have to know exactly what you were looking for. The new search tool brings this all together, so that users can use a single search bar on our website. The system is also much better configured to search by subject area. So, for example, if you’re looking for information on a particular town or a particular person, it will give you access to everything we hold on that town or person.

Not just that, but the results are organised into category – for example, family history, military history and so on. So if 20,000 hits come back, users can ‘drill down’ into whatever area seems most relevant to their search.

EI: Given the complexity that underlies that search technology, its implementation must have been quite a challenge. Into what areas have you put the most effort?

DT: Much of our effort has focused on categorisation, which is essential to ensure the quality of results that are returned for individual searches. That was particularly important to us because our website is used by the public and any search tool needs to cater to the needs of an audience with a very broad range of skills and expertise when it comes to using search engines. At the same time, it needed to be pretty sophisticated because of the huge volume and range of records it needed to search. We were attracted to Autonomy’s technology because of its ability to form an understanding of the indexed content. Rather than using the simple keyword-based technology that many other search platforms offer, Autonomy’s technology develops a conceptual and contextual understanding of the meaning of electronic data.

Throughout our implementation of the search technology, our focus has been on user-led classification of records and we have performed extensive public-usability tests. The next stage will be the introduction of personalisation capabilities. By that, I mean that the search engine is capable of recognising individuals, learning what their interests are and tailoring results accordingly. That will involve the addition of profiling and alerting functionality, so that users will be able to define their areas of interest and be informed of new material as and when it becomes available.

EI: And what stage are you at now with the implementation?

DT: The search engine is now live on site and that’s the first phase covered. But we’re constantly revisiting our categorisation effort and performing updates. That means that the team keeps running searches to make sure that the top 100 searches provide exactly what users need. We’re also using recommended links, so that if we’ve got a big release of records onto the website, or a particular area has generated a lot of publicity, it gets pushed to the top of results lists.

In essence, the ability to search records is nothing new in itself – the National Archives has always been about enabling people to search for information and making sure that the right information is easy to find. But the ability to search across databases and our move into personalisation takes this new implementation a huge step forward.

We plan to have a new version of our website up and running in the next few years and my belief is that the search engine will be central to its design and function – rather than the add-on that it is right now.

EI: How do you plan to incorporate social networking into your strategy for the digital archive?

DT: This is an entirely new area for us, but I think it’s one that will prove immensely valuable in the work that we do and in making sure that the public is more involved in that work. What we’re looking at is working on a wiki approach so that users can contribute to our collections.

That is a very radical move and it does, admittedly, make some people within the organisation a little nervous. There’s a good reason for that: colleagues here are accustomed to experts within the organisation generating catalogue information and content and publish it, rather than outsiders.

But we have over 115 kilometres of records in our holdings and we need to recognise that some of our users are far more expert in particular areas of these holdings than we are ourselves. So the idea is that wikis can provide a forum where these experts can contribute their ideas and information about specific areas of our holdings and on British history in general.

The beauty of this approach is that these contributions will be searchable, just like the content we generate internally. In effect, what we’ll be creating is our own Wikipedia, based on our holdings – but also expanding the range of expertise that our audience is able to access.

About the National Archives

The National Archives was formed in April 2003 by the mergers of the Public Record Office and the Historical Manuscripts Commission. Based at Kew in West London, it is responsible for looking after the records of UK central government and the British courts of law, and making sure everyone can see them. The collection is one of the largest in the world and spans an unbroken period from the 11th century to the present day.

The National Archives advises government departments and the wider public sector on best practice in records management, as well as selecting government records of enduring historical value to be preserved forever. It strives to make the collection as accessible as possible to the community at large and to heighten its profile both nationally and internationally. In particular, its education service is committed to sharing and interpreting records for the benefit of academics and students of all ages.

One of the National Archives’ major achievements in the past few years has been the development of electronic records management, which will eventually replace paper-based systems. The National Archives is at the forefront of this advance and will continue to develop it and advise others on its use. It has also pioneered the digitisation of records on paper and other traditional media so that they now can be seen online.

Take, for example, the Domesday Book, the oldest public record held at the National Archives: it is generally considered to be one of the UK’s finest national treasures, and yet a recent survey revealed that less than one per cent of the population has actually been to see the original at the National Archives’ museum.

For many, that is no longer an issue, with the August 2006 launch of Domesday Online, a website provided by the National Archives’ DocumentsOnline service. Through this venture, the National Archives hopes people will take the opportunity to visit Domesday Book from the comfort of their own homes and learn more about this iconic document.

The website, provided by The National Archives’ DocumentsOnline service, also contains useful information about the history of this 920-year-old document.

The most exciting element of the project, however, is that the public can now search the Domesday Book online by name of town, city or village. For a small fee (£3.50) it is also possible to purchase a copy of the original page featuring the place name and a translation of the entry into modern English.