The ETA on ETL

Long-Term Market Drivers

• Continued Market Expansion and Growth: Although vendor consolidations will continue, the overall market will continue to grow as ETT vendors are now competing with each other, instead of with in-house programming staffs. In addition, data quality has been recognized as a major concern as the ramifications of “garbage in, garbage out” quickly become obvious in near real-time applications that require linking customer records across multiple, disparate database environments, and there will likely be even higher growth in this market segment. The demand for data marts to support analytic applications and the use of data augmentation services to enhance customer records with demographic, socio-economic, and lifestyle data will also contribute to this growth.
• Alliances and/or Acquisitions for a More Complete Solution: While we have divided the ETL space into data extraction, data cleansing, data transformation, and data augmentation, vendors realize that the implemented customer solution requires components of each and have moved to expand the scope of their offerings through their own technology, acquisitions, or partnerships. As the lines between EAI and ETL continue to blur, due in no small part to the need for near-real-time information, traditional ETL vendors are targeting EAI as well (sometimes as a cooperative partner).
• Data Augmentation and Data Enhancement: In addition to consolidating data from multiple sources within a data warehouse, it is often desirable to augment the internally collected data with external data such as customer demographics, which are available from a variety of third-party sources including Acxiom. This is especially useful when conducting “one-to-one” marketing campaigns or utilizing data mining for market segmentation and differentiation.
• Access to Non-database Data: While ODBC, the open database connectivity standard, was once considered state-of-the-art with its ability to access data in relational databases, newer standards such as OLE DB, used for accessing relational, non-relational, and semi-structured data including documents and spreadsheets, as well as the acceleration adoption of XML, will enable data warehouses to be populated from additional data sources.
• Database Vendors Bundle ETL Functionality: The bundling of its Data Transformation Services (DTS) with Microsoft SQL Server 7.0 in late 1998, the inclusion of Oracle Warehouse Builder with the Oracle Developer Suite in mid-2001, and the inclusion of IBM’s Information Integrator and Warehouse Manager within its DB2 UDB Data Warehouse Enterprise Edition software bundle were early examples of database vendors including ETL functionality within their core products. Not only will this continue to place price pressure on third-party data integration vendors, it will almost certainly place the database vendor’s own offerings on the long, if not the short list, in competitive evaluations.
• Expectations that Company Representatives are Familiar with Their Customers’ History: In general, consumers and organizations expect vendor representatives to be familiar with their prior interactions with the company, including purchase history and details of prior support center calls. This “360 degree” customer view has helped fuel the growth of both analytic and operational customer relationship management (CRM) systems and the need for data integration, data cleansing, and data augmentation products and services to populate the underlying database. Consumers are openly hostile about having to supply the same information twice, either over time or during the same phone call or Web site visit, and companies must recognize that asking consumers to do so because “the system requires it” has about the same negative effect as did telling them that “the computer made an error” two decades ago.

Offensive vs. Defensive Responses

Improved Data Access Drivers

Offensive: Even in cases where it may have become a checklist item, vendors will continue to emphasize the wide variety of data sources and targets that they can respectively access and populate, as well as their ability to work with XML-tagged data and message queuing systems. Vendors whose data access capabilities have been certified by enterprise application (EA) vendors should definitely highlight this certification when selling into accounts utilizing the EA system.

Defensive: Vendors lacking the ability to access a particular data source will focus on their overall access capabilities and stress any particular advantages they may possess in areas such as data transformation capabilities or data augmentation. They should downplay the data access weakness and propose that it can be easily resolved with a custom program that will export the selected data to a flat file that its product can then access. They should also consider partnering with specialists such as iWay, rather than developing this capability on their own.

Data Profiling Capabilities

Offensive: ETL vendors with data profiling capabilities should highlight the need for these data warehouse developers to understand the data contained in the organizations’ source systems and point to horror stories and implementation failures resulting from erroneously assuming that documentation reflected actual content.

Defensive: Vendors lacking data profiling capabilities will mention the capabilities of their partners; lacking such partners they will downplay the importance of data profiling, while actively seeking to form partnerships of their own.

EAI and ETL Convergence

Offensive: While most vendors now position themselves as data integration vendors, those with both EAI and ETL capabilities should focus on the flexibility of their platform and its applicability to a wide variety of problems and the ability to work directly with a wide range of sources including traditional databases, Web logs, XML documents, message queues, and software application packages.

Defensive: ETL vendors lacking some EAI capabilities should form partnerships with specialists that can fulfill these needs. They should claim that by focusing on their heritage capabilities, and partnering with others for capabilities they lack, they provide a best-of-breed solution to their customers.

Data Augmentation

Offensive: Vendors with strong data augmentation capabilities should focus on the value of, for example, enhancing customer records with demographic and psychographic information in order to better serve these customers and/or identify appropriate sales opportunities.

Defensive: Vendors that do not directly offer data augmentation capabilities should be ready to point to third parties that do and perhaps claim “that while concerns about individual privacy dissuaded them from directly entering this market, they formed partnerships to provide this option to their customers.”

Enterprise Information Integration (EII)

Offensive: Vendors actively targeting the EII market should explain their definition of the concept (and it seems to vary by vendor) and reference industry luminaries such as Bill Inmon whose use of the term “Adaptive Project Mart” is evidence of its credibility. At the same time, these vendors should carefully point out where EII fits in the overall data warehouse spectrum and the risks of “quick and dirty” data integration in order to set appropriate expectations.

Defensive: Vendors not actively targeting this market will dismiss it as a new buzzword and merely a reincarnation of the virtual data warehouse. They will rightly emphasize that quality decisions require quality data, which if drawn from multiple sources, must be “refined” to ensure consistency.