Data Migration – things I wish I’d known
Is this scenario familiar? You’re working through an intense programme of transformational change. It has taken a huge amount of effort over many months and, despite the usual programme gremlins, all seems on track to realise the benefits that you planned at the outset. The key stakeholders are engaged, the business has released the specialist resources you need to support the programme, (almost) when you needed them, your delivery partners are aligned and then, after all your careful planning, you find that the data from the old systems refuses to fit neatly into the new. Suddenly the programme that had been going so well is facing what frequently turns out to be a significant overrun.
The overlooked element of a programme
Data migration is arguably the most overlooked part of any large-scale programme of change. No part of a programme is straightforward, but many elements, such as defining the new operating model, specifying business requirements or designing the solution architecture, tend to grab more attention. Data migration, by contrast, just isn’t glamorous and is usually the squeezed and neglected part of the programme. That’s because it’s labour intensive, repetitive and requires meticulous attention to the minute detail of potentially millions of data items. At best, it is often underestimated and, in some worst-case scenarios, it’s typically regarded as that part of the programme from which time can pilfered to compensate for other, late-running workstreams.
It creates a frustrating paradox in that migrating data requires heavy duty input from business users, many of whom find it difficult to enter the mind-set required to work through the large amounts of data that flow through the migration process. Even worse, they have a day job that will always come first, particularly when there’s a priority operational issue to deal with. The net result is that it’s usually hard to get the business to fully support the data migration process, particularly when it comes to being accountable for the quality of the migration outputs.
Such attitudes typically lead to data migration being insufficiently well planned from the outset, with workstreams under-resourced and critical path activities being initiated too late in the programme. In such circumstances, it’s effectively a programme time-bomb and the effects on timelines and costs can be devastating. But the impact of bad data goes well beyond the programme itself; error-strewn data won’t support the processes designed into the new operating model, which puts ‘cut-over’ and ‘go-live’ dates at risk. It undermines confidence in the core system and ultimately delays the realisation of business benefits. There’s also the issue of having to keep the old systems on stream for longer than anticipated and this in itself can be the source of significant additional cost.
Data migrations require thorough due diligence and planning
Viewed at its simplest level, data migration is the process of transferring data between computer storage types or file formats. But, as outlined above, it is an order of magnitude more complex than this description suggests. Modern data migration projects require complete coordination across every technique, task and resource on the project, from extraction to go-live and archival. The question, of course, is how can this be achieved successfully given the different elements involved?
There is a compelling case for investing in a proper approach to data migration and, like any investment, this begins with thorough due diligence. Knowing what you’re dealing with, what you’ll need to put in and what the risks and returns are likely to be, will create a solid foundation from which a detailed plan of action can be developed. And, since no plan survives contact with the enemy (to coin a military phrase), due diligence should also help to define the people, processes and technology required to get the job done.
Thankfully, there are a number of specialist organisations out there that can provide the tools, techniques and resources to help to automate the process of physically shifting and/or merging data from its source to its destination. However, this is only part of the story and there’s so much more that needs to be done to ensure that this island of automation delivers benefits rather than more woes. There is a world of pain awaiting those who pay insufficient attention to the cleansing, harmonisation and verification of data, before and after any migration tools have done their bit.
Below are some of the aspects to think about as part of a meaningful due diligence process, that might just help to smooth the data migration path.
The complexity of data
Why is the cleansing, harmonisation and verification process so critical?
Because data, the way it is used and built over time, isn’t simple. Just think about how data is stored, used and managed within an organisation and how that can differ across, say, inter-group companies and even departments. For example, a data subject, e.g. a part number or a specific client, could have a different naming convention in different databases. Most organisations have numerous undocumented procedures and disconnected legacy systems, that create something of a data ‘backwater’. Invariably operational teams develop multiple workarounds in local systems (e.g. Excel), often they will introduce paper and manual workarounds into the process and of course, when it comes to data entry, many will have their own way of doing things. This leads to multiple data entries and poor data quality – the bane of any migration project. There can also be multiple copies of data, particularly where siloed operations exist and data gets out of sync and difficult to access.
Any one of these scenarios will mean that the ‘source’ data is likely to be of mediocre quality at best and where more are present, the challenge will be significant. It’s not unusual for organisations to simply ‘sleep walk’ into trouble, believing that the automated tools they have invested in will alone see them through. But data migration is perhaps the last bastion of manual effort when it comes to IT and there is no substitute for rolling up one’s sleeves and getting stuck in. It is a painstaking task, but taking a more structured approach to the preparation, harmonisation and verification of data, will ensure that any automated routines deployed are more effective and result in a higher quality, less expensive migration, done in shorter timescales.
Here are some of the things to consider as part of this:
- Before starting, ensure the business is fully engaged and key users are appointed to support the process – it’s their data and they must feel ownership of it. Resist the pressure to push on without both being in place.
- Commit to thorough cleansing work pre-migration, to provide a solid baseline for everything that will follow.
- Where data is being merged from multiple databases, harmonise it up front to avoid duplicate data clashes and ensure ways of working and processes are aligned.
- Export large data sets for detailed data comparison and use automated scripts to speed up the process.
- Report clearly on data mismatches so these can be analysed quickly by key users and corrective decisions made. Filter out the good data to leave just the errors for easier analysis.
- Provide support to key users so they are able to understand the context and consequences of reported data errors.
- Condense and collate the reported issues into actionable feedback for the organisation.
- Perform root cause analysis on reported data mismatches to troubleshoot the cause of the underlying problems.
- Establish feedback loops on the root cause analysis performed and provide support and advice to key users on how best to resolve the issues identified and carry out data corrections.
- Be prepared to repeat test cycles before committing to a go-live, but don’t be tempted to do this just because the business has failed to deliver on their allotted responsibilities. This comes back to ownership and they must check their own data for accuracy and quality.
Delivering a successful migration
Data migration may be considered a technical detail in many organisations but it is critical to the success of any technology enabled transformation programme. It is also more complex and difficult than most people imagine and its significance is frequently underestimated. As a result, the data migration workstream in any programme is regularly compromised and poorly executed by many organisations, resulting in costly overruns and delays.
Clearly then, any organisation that is undertaking a migration needs to invest in the right approach. This means doing a proper due diligence and from this, building a resilient plan of action. Getting the right specialist parties involved and investing in appropriate activities and processes, de-risks the overall migration, improves the quality of the end result and reduces costs.
Technology is arguably the key enabler of change for most organisations and data migration is an intrinsic element in this. It’s always a workstream that attracts a great deal of scrutiny for DAV consultants, for all the reasons outlined above. Over the years we have developed hands-on experience within the DAV team, in facilitating the migration of data and specific capability in the cleansing, harmonisation and verification activities that smooth the process. Working as part of the client’s team, we provide an effective bridge between the business and the specialist providers of automated data migration solutions, improving efficiency of the migration process and relieving pressure on key users.
Recognising the demands and pitfalls of the data migration process and gearing up to address/avoid these fully, will mitigate the risk of downstream problems and the impact these inevitably have on timescales and budgets. Forewarned is forearmed, as they say.