It’s a common business challenge: the business users require information quickly to inform important decisions, but the team they rely on for this need, primarily the IT team, cannot provide the information fast enough. Perhaps more than a challenge, it’s a business frustration, and a common one at that.
When inquiring about the company’s data architecture, I’m not surprised when they tell me: “we have an old but robust and functional system that houses relevant information from the company based on ETL processes that extract information from different systems and populate the necessary tables with information ready to be consumed.” It’s an Enterprise Data Warehouse (EDW), a robust and reliable system still implemented by many companies.
So, what happens when the business user requests information? Firstly, the IT teams create a data mart in another separate database. This is problematic not only due to the cost of maintaining multiple databases but also the integrity and consistency of the information stored comes into question since synchronizing various data marts with the EDW is not a simple process.
The real problem however is the delay between the time that the business users put in their request and the time it takes for it to provide that information, especially if that information is not in the EDW. IT teams have to locate the source from which this information can be obtained and incorporate it into existing ETL flows. This is a new ETL project that must be designed, developed and tested – accumulating further delays for the business user, and not to mention the need for several IT teams to be involved, i.e. those responsible for source data, ETL, EDW, data marts and reports. In short, we are talking about a process that can take up to 6 months to come to fruition and for the business users to obtain the information needed to extract value.
Swapping Robust & Functional, for Agile & Efficient
In a modern, data-driven enterprise this sort of delay and inflexibility is unacceptable, especially with the growing need to incorporate new sources of information. So what are the following objectives that modern enterprises need to adhere to in order to succeed:
- Provide an information system that transforms data into relevant information very quickly.
- Centralize all databases from the same point, including security aspects, documentation, governance and metadata.
- Provide real-time information without waiting for data marts to be refreshed with the latest EDW information.
- Improve the time to market with short and agile projects.
- Be prepared to incorporate new sources of information of different types, including unstructured sources.
It is important to note that I am not suggesting that companies do away with their EDW completely but simply that they rethink their “robust and functional system” and instead, consider the logical data warehouse approach. Take a look at the following diagrams to see how a data virtualization layer can aid the creation of a logical data warehouse architecture.
Figure 1: Current situation. Traditional Data Warehouse Architecture
Figure 2: New Architecture: Logical Data Warehouse based on a Data Virtualization Layer
Building a Logical Data Warehouse
For more information on how you can achieve these goals with Denodo, check out the following webinar entitled “The Big BI Dilemma – Bimodal Logical Data Warehouse to the Rescue!” featuring independent industry analyst and expert on logical data warehouses, Rick van der Lans.
