Recently, we have seen the rise of new technologies like big data, the Internet of things (IoT), and data lakes. But we have not seen many developments in the way that data gets delivered. Modernizing the data architectures is the need of the hour.
The Problems with Today’s Data Delivery Methods
Organizations still rely on traditional extract, transform, and load (ETL) processes, which have been developed 30 or 40 years ago. But these processes were built at a time when data volumes were relatively low and there was mostly structured data.
Today, with ever-faster data creation and ever-growing data volumes, ETL processes are starting to seem too slow, and with new data sources such as IoT data, social media data, and sensor data, they are also starting to break.
At the same time, the speed of business has increased significantly. Organizations can no longer base decisions on four-week-old data; even hour-old data might be out-of-date.
Unfortunately, organizations are spending so much time bringing data into a central repository that they struggle to derive insights from the data. This takes time and effort on the human resources side, and it also requires heavy investments in storage and maintenance. In terms of data architecture modernization, organizations are asking, “Is there any way to avoid physically moving data from place to place?”
Modernize Data Architectures with Data Virtualization
Data virtualization is a data integration and data management solution that leaves all of the data at the source, and it does not deliver data to the consumer until the very moment that the data is needed. When data virtualization does deliver data, it does so immediately.
Data virtualization is deployed in a layer between data storage and data consumers, separating the different technologies, and abstracting data consumers from the complexities of access. With data virtualization, data consumers do not have to worry about being experts in SQL or knowing other technical details. It is also very easy to add a source to a data virtualization layer implementation.
The data virtualization layer contains no actual data, only the metadata required to access the various sources. But with this critical metadata layer, organizations can easily expose information about data sources, in a structured way, to users and applications, for a strong data governance foundation, as well as strong data cataloging capabilities.
By providing a layer between consumers and data storage, data virtualization also serves as a strong security layer. Everybody accesses data through the data virtualization layer, so organizations can easily implement a centralized data security management layer that can include features like data masking.
Watch Us Live
I sat down with my colleague, Deb Mukherji, practice head – data and analytics (ASEAN), Tech Mahindra, and Alexander Hoehl, senior director business development (APAC), Denodo, to discuss this topic in the video below. We had a lively discussion that included Tech Mahindra’s critical role as a systems integrator, the extended benefits of data virtualization, and some of the ways that modern architecture can help to fight Covid19.
Modernization: The Time Is Now
Modernizing the data architecture cannot be put off indefinitely. If you have any questions about modernization, please ask them in the Comments field below, and I will forward them to the appropriate person.
- Modernizing Data Architectures - August 26, 2020