The EII challenge

Is enterprise-information integration (EII) the new paradigm? By Gary Eastwood

Ipedo, a US-based enterprise-information-integration (EII) specialist, recently threw down a challenge to CIOs and IT directors in an attempt to prove the value of EII technology.

The company’s ‘3,2,1,0 Challenge’ offered 15 organisations the chance to allow its professional services team to integrate any three information sources chosen by the participant, in two days and to present the data within a single web-based application… and at zero cost and risk to the participating companies.

“With the EII challenge, we’re putting our money where our mouth is to prove how quickly we can bring the benefits to companies,” said Nick Zhang, CEO of Ipedo, when he launched the challenge.

Unfortunately, the results are so far unpublished. But the tale does provide an insight into the rapidly growing EII market and a neat analogy as to its status: full of confidence and bravado, but with relatively few real-life results so far.

EII is the buzzword of the moment, at least in data integration circles. No wonder. According to analysts IDC, an average of 48 applications and 14 databases are deployed throughout the typical Fortune 1000 company, with 70 per cent of the average IT budget devoted to data-integration projects.

This burgeoning amount of data, the advent of web-services standards, such as XML and SOAP, and a need for businesses to react rapidly to changing market conditions are combining to drive the

EII concept. As a result, analysts Aberdeen Group, for example, estimate the EII market to currently stand at $250m, while Meta Group expects adoption rates for EII solutions to eclipse those for extract, transform and load (ETL) tools by 2006.

Until recently, the technology has been pushed by a handful of specialist EII vendors, such as Metamatrix, Avaki, Ipedo, Actuate (through its acquisition of Nimble Technologies) and Composite Software.

Validation of the technology is also coming from moves by business intelligence (BI) vendors, such as Informatica, Cognos, Hyperion and Business Objects, which are eager to partner with EII vendors in order to integrate the two technologies, and by heavyweights such as IBM (which has re-badged its Websphere concept to ‘Websphere Information Integration’) and Microsoft (which is pushing the concept of ‘federated’ data through BizTalk and SQL Server 2005).

Proponents of EII claim that the technology can gather real-time information from disparate data sources throughout the enterprise – whether data in transactional systems, e-mails, digital media or data stored in relational databases – on an ad hoc basis and present it in a single application, while shielding the end user from the underlying complexities of locating, querying and joining data from these source systems. Essentially, EII creates a ‘virtual database’ for ad hoc queries.

“The cynics are saying that EII is just another marketing con, but that’s unfair,” says Steve Craggs, chair of Europe for the Integration Consortium, a non-profit, industry body responsible for influencing the direction of the integration industry. “The last ten years have seen a focus on integrating businesses and making more of what you’ve already got, and a number of things have sprung from that, including data integration, enterprise-content management, enterprise-application integration (EAI), and so on. There is also more focus on ‘information’ rather than ‘data’. EII provides the ability to get at all your information regardless of the data format or the systems it resides in.”

EII is a completely different approach to existing data-integration technologies such as EAI and ETL tools for data warehousing. EII accesses the data in situ, without the need to create a new store of integrated data. Because queries are run on an ad hoc basis, EII is also potentially cheaper than EAI, with less complexity.

“The concept of EII is to link disparate data sources, regardless of format and source, and make it look like it came from a single relational database. It uses structured and unstructured data to create a virtual database,” explains Hamish Morjaria, VP for EMEA at Metamatrix.

One advantage of EII over other forms of data integration is its ability to retrieve up-to-date information on demand that can support operational decision-making. According to a recent Ipedo survey of 60 US-based users of common data integration technologies, 57 per cent of respondents cite “better access to information” as their main driver for using EII, followed by a “a more flexible integration approach” (47 per cent).

“Organisations rely on diverse information from multiple sources – internal and external. In order to adjust to and utilise changes in the business environment (customers, competitors, technology, environment, politics, etc), applications and end users need to access and integrate diverse data and content as if it were a single resource, regardless of where the information resides. And they need to distribute, consolidate, and synchronise information across complex, multi-platform, multi-vendor IT environments,” explains Kim Jasper, marketing manager in Information Management Software, EMEA, IBM.

Compared to data warehousing, meanwhile, EII is designed to look at data wherever it is residing at that point in time. “A data warehouse gathers data from all parts of the company, brings it together and analyses it. It’s very efficient, but does it have all the up-to-date information? EII doesn’t move data around, it just looks at it in situ,” says Craggs.

However, EII technology complements existing data-integration technologies, rather than replaces it. “ETL is a better vehicle for large-scale movement of information (typically an overnight process concerning terabytes of data), whereas EAI transfers changes to data between applications. EII is complementary to both – it trades underlying data in transactional systems and presents them in a single view,” says Jeff Morris, director of product marketing at Actuate.

Cliff Longman, CTO of data warehouse specialist Kalido, agrees: “I’m not aware of one of our customers that uses EII without a data warehouse. Also, I can’t think of one customer that has all its data in a warehouse. EII is complementary to data warehousing – it sits on top and brings into play all the data sources not already in a data warehouse.”

Users are also recognising that EII can also be used as a rapid prototyping tool to justify building a data warehouse. “EII helps business users get information quicker and easier. As all the components are reusable, EII can be used for prototyping and then developing a data warehouse,” explains IBM’s Jasper.

However, Kalido’s Longman points out that one disadvantage of querying against operational systems, as EII does in order to get real-time data, is that of performance. “It can be dangerous to load operational systems with queries,” he explains. “It’s very difficult to plan capacity and performance if your systems are being hit by ad hoc queries all the time. EII tools are good for assembling a small view of recent data where data quality is good, which is probably only one per cent of business intelligence.”

Bert Oosterhof, director of technology, EMEA, Informatica (a partner of Composite Software), however, points out that there are ways to minimise this particular problem. “One disadvantage of running a BI tool directly on an operational system is that it uses CPUs – you want to put as less of a burden on the system as possible. Composite, for example, uses a sophisticated ‘optimiser’ to deal with queries across different systems, as well as caching technology – so rather than running similar queries across systems, users can access previously cached queries,” he says.

Despite these doubts, reference sites for EII implementations are steadily growing, and not just in the US. Mobile operator Orange, for example, recently chose Actuate’s EII solution in order to help it schedule and deliver over 200 daily reports. Large volumes of constantly updated data, stored in many different applications, meant that report production was a highly resource-intensive process.

But the use of EII has provided Orange users with web-based information delivery of reports and increased the volume and speed of report production, with 95 per cent of reports now scheduled delivered automatically, says Shaun Desmond, performance analysis manager at Orange. “When a user logs on, the system recognises their profile and delivers the appropriate information in the required format. This helps people at every link in the information chain to do their job better. Just in terms of my department, to produce the same volume and quality of reporting without Actuate would require a fifty per cent greater headcount.”

Until recently, EII was described as a “technology looking for an application”. It seems that the need to perform corporate-wide BI queries in real time, and on an ad hoc basis, could well be that application.