In this article, I will discuss the complexity of data migration when transitioning to a new system, based on the traditional ways of working. I will explain why data virtualization can play a role in taking away this complexity, for the most part and how you can take the first step towards a more flexible data architecture.
Transitioning from one system to another, be it an ERP, CRM, financial or order system, is one of the biggest hurdles for many organizations. When contemplating this transition, there are usually two options that companies choose between:
- Full migration from old to new
The first option involves data migration from the existing system to the new system. This complex operation involves moving the data and interpreting it across two very complicated data models; this can involve managing thousands of tables. Next, you need to develop the ETL processes to give all the new and old data their proper little place in the data warehouse. Adding to the mix, if you’re really lucky, data warehouse remodeling to accommodate these new data sets. This lift and shift approach has a knock on effect on your dashboards and reports which as a result need to be updated. This is a massive operation which could take up to several months or even years!
Full migration from old to new
- Combine two data streams into one information model
The other option is to feed both data streams from the old system and the new system into the data warehouse. Although this approach means you can avoid migrating the data from one system to another– you are essentially just shifting the complexity as you will still need to adjust the data model and reports to accommodate the new data sets.
Combine two data streams into one information model
The Problem With These “Phased” Approaches
Unfortunately, it’s very difficult to migrate important systems in one fell swoop, not to mention risky owing to the possibility of delays and inaccuracies in the data. Most companies opt for a “phased” approach across departments, regions user groups or functionalities, rather than the “all eggs in one basket” data migration approach. The problem with this gradual process is that the underlying data source, reports and dashboards tend to gradually change too, resulting in inaccuracies across the board.
If Only You Could Migrate Only The Data That is Required
Wouldn’t it be so much easier if you could choose to migrate only the data from your old system that is required for the new system to function? For example, the client-, product- or pricing data? Of course, it would, but it sounds too good to be true.
If Only You Could Migrate Only The Data That is Required
Data virtualization technology removes the need to physically duplicate data, and integrates data from various sources, virtually. With this approach you can create a “data hub” which delivers data from both the old and new systems, making it accessible to the business user from one virtual layer, without needing to move anything. You’ll still need to apply transformation rules to integrate the data, but it’s so much easier and faster to do this because it is all virtual. You’ll have a central location where all the integration takes place, instead of various complex data streams or slow, laborious data integration via ETL.
Another cool feature is that a data virtualization platform offers a means to easily provide insight into the data’s origin. This means that the business user can see where their information is coming from and who has touched the data in the process.
The First Step Towards a More Flexible Data Architecture
It’s true that data virtualization can make that data migration to a new system more straightforward and less time-consuming. But this simplicity needn’t be limited to this one project alone, once data virtualization has been implemented you will no longer need to combine operation systems physically, only virtually. Any time you add a new link to a data source, you are building on your data hub and slowly parting with your traditional data warehouse.
This means that your architecture is simplified, more flexible and enables you to respond to changes faster than ever.
This blog was penned by Jonathan Wisgerhof, Senior Architect, Kadenza
www.kadenza.nl
- How Much Time Could Your Company Save If You Said Goodbye to Data Migration? - January 30, 2019
- Get Ready for the General Data Protection Regulation (GDPR), with Data Virtualization - May 24, 2018
- Data Virtualization is a Revenue Generator - September 20, 2017