The pandemic caused considerable disruption, but as a result, it also accelerated cloud adoption. As we moved towards remote working and switched to online customer interactions, it became clear that the cloud was more than just a “nice to have,” it was the foundation for the “new normal.”
However, cloud environments do come with their own challenges, and many of these are related to data. Companies need to surmount these challenges immediately so that they can remain at the top of their game through these difficult times.
New Cloud Environments, New Challenges
It is not uncommon for companies to struggle during all three stages of the cloud adoption journey:
- The initial migration. Companies often find it challenging to take the first step: moving data to the cloud. Not only does this involve transforming the data in existing systems to new cloud models, but also continuing to run existing services during the migration.
- Accessing and integrating disparate data sources after the initial migration. Having migrated most of the data, companies often then struggle with doing business with hybrid models that need to serve a mix of on-premises data and cloud data.
- Dealing with more advanced cloud use. Companies with mature cloud infrastructures find that they now have to combine multiple types of cloud services such as public cloud services, internal private cloud services, and software-as-a-service (SaaS) implementations, further complicating data integration.
Compounding the issue, the traditional methods of replicating data using extract, transform, and load (ETL) processes, in which data is first extracted from a source, then transformed into the necessary format, and finally loaded onto a new location, have proven to be difficult and costly to modify. Also, because they deliver data in scheduled batches, they cannot deliver data in real time. It is not surprising that the limitations of this technique, mostly around complexity, performance, and security and governance, are becoming increasingly evident.
Data Virtualization and Cloud Adoption
Data virtualization complements ETL and other data integration methods, such as Enterprise Service Bus (ESB) and Enterprise Applications Integration (EAI), by enabling access to data without having to replicate it.
Let me explain how this is possible with a simple analogy to home entertainment services such as Netflix and Spotify. We don’t need to store the movies and music at home on DVDs, Blu Rays or CDs. Instead, we read the information about the film or music (the meta-data) to decide what they want to listen to or watch, and when we make our choice, that content is streamed in real-time from some unknown location in the cloud.
Data virtualization works like this, but for data, and instead of functioning at home, it functions within enterprises. With data virtualization, the data is kept at the source, and only when it is needed, it abstracted and consumed in a report, dashboard, or application, in real-time. In contrast, the bulk moving and copying of data in ETL and data warehousing models would be like filling your garage with boxes upon boxes of CDs and DVDs.
Data virtualization enables businesses to gain real-time data insights without having to move any data, and it also provides centralized security and data governance, regardless of the location and status of the various data sourced. If a user wanted to run a query, the data virtualization layer would hold details (metadata) about all of the available data. However, it would only be at runtime that they system would abstract and combine the actual data to furnish the request.
Now Is the Time
With cloud adoption playing such an important in the current and future data landscape, companies need to act immediately to overcome any data-related challenges. Fortunately, technologies like data virtualization can bring considerable agility to businesses, offering a proven, intelligent solution for overcoming cloud data challenges at all phases of the cloud journey.
Beyond work, Charles has a wife and two daughters and lives in Beaconsfield, near London. His hobbies include flying (he has a private pilot’s licence), yacht sailing and road cycling. He has also climbed and trekked the mountains of the Andes, the Alps, the Himalayas and Kilimanjaro.
- Denodo and Open Line: Easing Mergers and Modernization Initiatives - May 9, 2024
- Data Mesh and the Future of Financial Services - April 5, 2023
- Data Fabrics, Data Meshes and the role of Data Virtualisation - March 24, 2022