Search for intelligence

With the volume of enterprise data doubling every two years, a new breed of intelligent search and navigation tools is helping make sense of the information deluge, as Gary Eastwood discovers.

By Gary Eastwood

Current media darling Google has already indexed more than eight-billion web pages but many more are believed to be hidden in corporate intranets and databases. It is thought that if all of these were indexed, the world-wide web would be 500 times larger than it is today.

But the experience of many IT managers is limited to using web-based search engines, often with little understanding of the differences between web, intranet and enterprise search – not to mention desktop search. But with a new generation of search, navigation and ‘information-access’ tools already being used in major organisations around the world, it seems that the information-retrieval processes we have become accustomed to are about to change.

Heralded by Win FS – Microsoft’s latest desktop file and search system – vendors like Endeca, Google, Convera, Kozoru, Fast Search & Transfer (FAST), APR Smartlogik and Yahoo!, to name a few, are realising that the next ‘battleground’ will be intelligent search and navigation.

While the terms used by vendors vary depending on their marketing strategy; ‘object oriented’, ‘guided’ or ‘faceted’ search and navigation tools provide a more ‘intelligent’ way for users to trawl through the realms of data that traditional search engines throw up in response to a keyword search.

“I have a lot of respect for Google, but type in ‘Einstein’ and you will get millions of documents back,” says Bjorn Olstad, CTO at enterprise-search-platform specialist FAST. “Is it about Einstein’s grandson, his theory of relativity or something else? There is a need to ‘dis-ambiguate’ search words.”

While typing keywords into a search box can be a quick and easy way of finding the right information, the technology assumes the seeker knows the exact name of the book, product, dataset or document they are looking for. If not, the search is likely to result in a vast number of ‘empty’ routes and paths that contain irrelevant information.

“The first step is to find the information, but that needs to be supported by an intelligent-navigation system. It’s like finding a needle in a haystack of haystacks,” explains Olstad.

With the burgeoning volume of digital information, Paul Sonderegger, search and navigation ‘evangelist’ at US-based vendor Endeca (and a former enterprise-search analyst at Forrester) believes that the current generation of algorithm-based search engines has now reached its technical limit.

“The likes of Autonomy, SAS Institute and Verity were one phrase enquirers based on the notion of finding a nugget of information,” he says. “But keyword search has been around since the 1970s, when it was determined by the technical constraints of the time. It was never optimised for how humans express – in language – the search terms they are looking for. Vendors have spent 30 years working on complex algorithms, but those efforts have now peaked in their ability to provide the information that people are looking for.”

The search box will be around for some time yet, because it provides a rapid and simple route to information – when the seeker knows exactly what they are looking for. If not, searching a large topic area for a specific item of information without knowing its exact name will throw up millions of hits.

“What do you then? You have to have some form of navigation,” says Sonderegger. “But that raises issues as everybody categorises things differently, which is a core problem. The classification system would be unclear and inscrutable to the vast majority of users. So, you need a classification that expresses the ‘attributes’ of the underlying content.”

Faceted navigation derives from the ‘heyday’ of library sciences in the 1920s, when an Indian librarian, Shiyali Ramamrita Ranganathan found that classification systems suffered from one problem: books could only be kept in one location, which meant that anyone trying to find a specific title first needed to understand the classification scheme. He realised that classifying books multi-dimensionally by exposing their facets or attributes (essentially sub-topics) and expressing each one as an interface, would allow each person to take any number of different paths to get to the same information, without an understanding of the actual classification system.

For example, someone looking for information on the price of aeroplane kerosene could search by terms like ‘airline costs’, ‘aviation fuel’, ‘the rising cost of air flight’, and so on – yet still reach the same information.

In a conceptual sense, a facet represents each face of a multi-dimensional shape such as a cube. If the centre of the cube represents ‘the price of aeroplane kerosene’, it is possible to access that information from each face, one of which represents ‘airline costs’, another ‘aviation fuel’, and so on.

Endeca applies that notion to the digital world by exposing the same information on a number of different facets. “It means that a user can follow any path that makes sense to them on that particular day and still end up with the same information,” says Sonderegger. “i-Tunes is a great example: you can search by song, band, album or year and because it supports multiple paths, you still get to the same underlying piece of content.”

Jeremy Bentley, CEO of enterprise-metadata specialist APR Smartlogik puts it more simply: “Whether you want to use the term ‘faceted’ or not, search and navigation today is about trying to organise a set of results from a search engine into some kind of order that makes it easier for the brain to handle,” he says. “How do you order 20,000 Google results?”

Currently, solutions consist of a keyword-search box (the preferred method of searching for some users) and advanced or guided navigation, which groups search results according to subject facets or attributes. Navigation is coupled with spotlighting, which is often a bar on the right-hand side of the search screen that makes recommendations based on its understanding of the context of the search.

A simple but effective demonstration of this is the web page of directory service Yell.com, which uses Smartlogik’s Semaphore Taxonomy Manager. A user typing in the phrase ‘printing services’ might then be asked if they mean ‘computer peripherals’, ‘stationary designers’ or ‘typesetters’. As a result, the user is guided to the target information via any way they choose.

“It can provide competitive advantage,” argues Bentley. “For example, BA’s portal will change shape depending on the search query – do you want cheap flights, company information or industry news? It puts information into the context that the user expects, making the portal – and therefore the company – look more intelligent.”

Such a powerful tool is inevitably finding its way into the enterprise, and advanced search and navigation tools that sit on top of existing data repositories, such as knowledge, content and records-management systems, are helping organisations to leverage their existing knowledge base.

“Databases, content-management systems and other data repositories were primarily designed to make it easy to enter and store large amounts of information,” says Sonderegger of Endeca. “They were not really designed with the idea of getting information back out again. The genius of faceted navigation is that it dramatically eases information retrieval, because it provides a very flexible and easy-to-use interface.”

Its proponents cite a number of business benefits from the more dogmatic claim that it will increase revenues, through to softer benefits such as speed of accessing information. Somewhere in between lies the claim that this new generation of search and navigation technology can influence strategic objectives by placing the right information in the hands of the business decision makers quickly and accurately.

For example, KANA, a knowledge-management (KM) specialist focused on the customer-service space, uses object-oriented search and navigation technology (an object essentially equates to a facet or an attribute) to provide customer-service agents with advanced information search and retrieval capabilities.

It does so by leveraging existing KM resources, as well as connecting into content-management repositories such as Interwoven, Documentum and FileNet. KANA terms this as ‘object-oriented KM’.

“It’s especially important in the customer-service area. If a customer navigates down a path on your website, for example, and doesn’t find what they are looking for, it can be very frustrating,” explains Charlie Isaacs, CTO at KANA.

Printer specialist Xerox is using Kana IQ in its customer-service centres, which need to provide localised content to a global customer base in more than 20 countries – in at least seven languages. “We needed to be multinational, multilingual and serve our customers from a single location, using the internet inside and outside the firewall,” says George Barnes, e-business manager at Xerox. “We manage close to 30,000 search sessions per day and that number continues to climb.”

Endeca’s customers, meanwhile, represent the gamut of vertical industries. Retailer Tesco Wine Warehouse, for example, uses the company’s customer-facing-guided-navigation technology to provide an improved customer experience.

“We wanted the site to function like a personal wine merchant, helping our customers find the right wine to suit their palate and their wallet,” explains Mike McNamara, CTO at Tesco.com. “Guided search enhances the customer experience and allows us to match more customers with offers. We expect to improve sales, increase average order size and generate repeat business.”

Sonderegger of Endeca is equally confident that faceted search will have a great impact on enterprise search. “As evidenced by Microsoft’s new WinFS filing system, the popularity of XML, web tagging and i-Tunes, there will be a change in the way people locate and find information,” he says. “And there is great cause to believe that guided navigation is already that next generation of search-navigation technology, because it takes advantage of human-search behaviour – an asset far richer than using a search box.”

The search box will be around for some time yet, as it currently provides the fastest route to known information. But as volumes of data continue to expand, intelligent search and navigation tools optimised for human-search behaviour should ensure that the proverbial needle in the haystack becomes easier to find.