Data reconciliation is a process that is severely underestimated by many companies. It is the phase during any data migration in which the data gets verified. This process compares data between two sources, or between a source system and destination, to ensure that the data was transferred or received correctly, and that it remains consistent between them.
Data reconciliation and data comparison processes often happen before and after a data migration to make sure that the target and the source are the same. They are both important cross-checks and should be included in the process flow by all industries that deal with data.
The Importance of Data Reconciliation
A data migration process with no problems is a goal for all companies, and to ensure that this process will be successful, data reconciliation and data comparison needs to take place and be performed using the right approach.
During the data migration phase, data can get lost, network problems can occur, and the systems used by the company can have communication issues, so data values can get lost or arrive at the target changed, incorrect or duplicated.
So, data reconciliation is the step that helps users track and measure data transferred to make sure that it is correct and reliable. If the right approach for data reconciliation and data comparison is not taken by the organisation, errors and problems can go unnoticed, and decision-making by the company may be compromised by inaccurate and unreliable data.
Implementing a Proven Approach to Data Reconciliation
Most businesses require a simple data reconciliation mechanism as part of their data migration process. It’s not recommended to use Excel spreadsheets, as those require too much manual work and can make the data and information unreliable. Further, Excel is not an auditable tool, meaning users can make changes to data within spreadsheets and no record of those changes is kept or available anywhere.
To start, organisations need to adopt a reconciliation tool that will be simple to deploy and use, but still provides details of all of the actions that need to be taken by the users to correct the data. It’s best to go for tools that present data exceptions as a dashboard – a visual and simple view – to allow the organisation to review the data easily during the migration validation.
1. Reconciliation issues are identified
When data reconciliation problems get identified during the process, the best approach is to send a notification to the team that is responsible for the data – and other possible support teams inside the organisation – so they can get a detailed view of what happened and define the necessary actions to correct it.
2. The frequency of data reconciliation
The frequency of data reconciliation should be defined by the company according to their needs. But there are two ways of doing it:
- One time data reconciliation: it’s manually triggered and performs reconciliation when needed
- Scheduled data reconciliation: the reconciliation happens as scheduled (daily, weekly, monthly, etc.)
3. The levels of data reconciliation
More than one level of data reconciliation reporting should be utilized. The levels of data to reconcile can be:
- Pre-load: the pre-load reconciliation compares the data between the source and the transformed data. This level gives the confirmation that the right changes have been applied to the data.
- Post-load: the post-load reconciliation compares the transformed data and the target data to ensure that the right data has been loaded.
Also, the system used should allow the company to filter to reconcile the data at different levels. For example, reconcile data at the highest level, and if it doesn’t give the company the needed information they should be able to change the filters and reconcile the data at a mid-level. And then, if it still doesn’t work, they can filter and reconcile at a low level with the selected criteria.
4. Reconciliation reports
All the reconciliation reports should give the business a full view of the comparison4s made between the source data and the target data.
The systems should also allow users – according to their levels and their permissions to view and change the data – the permission to validate not only the migration but also the changes applied to the data. This step is crucial to keep the data migrated accurate and reliable.
