We are in the midst of a significant transformation in each and every sphere of business. We are witnessing an Industrial 4.0 revolution across the industrial sectors. The way products are getting manufactured is being transformed with automation, robotics, and 3D printing. The way services are getting rendered is changing with just-in-time and on-demand service delivery platforms. The focus of marketing is shifting from the “vantage point” to the “point of presence.” Sourcing is changing from just-in-time to predictive and collaborative. Manufacturing is shifting gears from dedicated plants to shared services. The world is going through a digital transformation in every sector, from cars to social, and from agriculture to healthcare.
As a result, business processes and workflows are undergoing dynamic changes, leading, in turn, to changes in the underlying data sources and data dependencies. Data sources are changing (one system to multiple inhouse and partner systems). The nature of data interfaces is changing with the prevalence of XML and JSON formats, APIs, and cloud data. The half-life of data structures and interfaces is shrinking fast, which is leading to a state of continuously changing data sources, including changes in velocity, variety, volume, and veracity, and affecting disparate data sources from business ecosystem partners across the globe. Data is coming in from new sources such as social networks, mobile devices, cloud and the IoT, in multiple forms and formats, and this further adds to the complexity.
Business and innovations are changing at a mind-boggling rate. “Data oil” for the decision engine is becoming increasingly foundational and critical for business success. The ever-increasing nature of data fluiditycalls for an agile and just-in-time “decision intelligence system,” an agile, on-demand, logical data integration mechanism, combined with a configurable BI system that is both flexible and robust.
Limitations of the Traditional Enterprise Data Warehouse
Many organizations rely on the traditional extract, transform, and load (ETL) model, which involves building an enterprise data warehouse (EDW) and a centralized data lake to serve the entire enterprise. As organizations continue to grow through mergers and expansion and evolve with changing business dynamics, the traditional EDW will be out-of-synch sooner than planned, and as the business ecosystem continues to change, due to new partnerships, new business models, and new service providers, the EDW chasm will only get wider.
To be agile and responsive to the new market scenarios, companies need access to data in real time, or near real time, with no internal organizational or geographic boundaries.
The Emergence of the Logical Data Warehouse (LDW)
The logical data warehouse (LDW) addresses this problem by disassociating reporting and analysis from hardwired data sources. It is a flexible architecture (not a product) that facilitates logical (not physical) integration of data across multiple data sources, systems, formats, and latencies. The type of data source no longer matters, whether it is a database, web service, JSON or XML file, NoSQL platform, or a big data deployment. It fulfills the primary objective of a single point of integrated data access for the enterprise. It looks and behaves like a single data source, but in reality, it is a virtual data repository.
Logical data warehouses offer numerous advantages, including:
- Seamless data access from visualization tools
- Fast provisioning for required data access
- Reduced inconsistencies across sources
- The insulation of business users and downstream systems from underlying data complexity
- The agility to connect
to new big data sources
- No more complex ETL and unnecessary data logistics
- No custom connections to data stores
- The ability to leverage
existing BI/Analytics data infrastructure
- Access to data at any granularity level
- Control data governance and compliance from a single point
Time-to-insight and time-to-response are the keys for business survival and growth. Logical data warehousing provides faster time-to-insight.
The Enabler of LDWs: Data Virtualization
The power that enables logical data warehousing is data virtualization. Data virtualization provides exhaustive data federation capabilities while hiding data-source complexities from downstream consumers and interfaces. This enables data architects to configure data views (data consumption end points) combining several disparate and distributed data sources, to fulfil business requests. It does this while continuing to leverage the existing, established data repositories within the system. It seamlessly connects with disparate data sources, collates required information, and publishes in the required format (ODBC or JDBC connection, API, web service, data extract, etc.).
Here are some of the illustrative use cases for LDWs that we see across industries:
Supply Chain and Logistics Optimization: Currently most transportation vehicles are connected to various devices to ensure the efficient tracking and delivering of goods to customers. Another scenario is last mile connectivity, which geo-local partners use to reach the final destination. In such a case, LDWs with data virtualization can seamlessly combine the shipment data and SLA from the head office with the last mile delivery partner and ensure timely delivery within the SLA. For example, if slow traffic or bad weather adversely affects movement, a connected logistics system could adjust itself to the current situation and modify downstream processes accordingly.
Production and Inventory optimization: Many companies have paid dearly, and continue to pay dearly, due to accumulated inventory pile-up in the market that is invisible to the manufacturing company. We have often seen this in the semiconductor, telecom, and automotive industries. Seamless integration of supply and sales and warranty and accounts receivables is a great antidote to the timely detection of such a problem. This seemingly impossible situation can now be addressed effortlessly with an LDW powered by data virtualization.
Conclusion
In this connected world, business processes, workflows, and interfaces will continue to rapidly change. As a result, data fluidity will continue to increase and accelerate, as businesses go through digital transformation. It is imperative for businesses to be data-agile and responsive to market conditions, so they can stay relevant and profitable.
The situation necessitates logical data warehousing enabled by data virtualization, enabling a fast provisioning capability for the enterprise. Logical data warehousing with data virtualization provides a win-win scenario for business and IT. It reduces the inherent friction between IT’s quest for stability and business’ need for agile, on-demand information.
In scenarios where there is a need for a true real-time response, with high velocity data streams (we mean hundreds of thousands of records per second), a high-velocity data engine can be deployed, enabling the fast provisioning of true real-time data through data virtualization. This results in faster implementation and lowers the total cost of ownership. (COMPEGENCE provides a high-velocity real-time data engine, powered by Denodo’s data virtualization platform, that easily handles the network scale and speed of solutions, handling hundreds of thousands of records a second with true real-time alerts and analytics).
