Objectives:
An effective data management involves a series of complex and interrelated processes that enable an organisation to use its data to achieve strategic objectives. Data management includes the ability to design data for applications, store and access it securely, share it appropriately, learn from it and ensure that it meets business needs. An assumption underlying claims about the value of data is that the data itself is reliable and trusted. In other words, of high quality.
However, many factors can undermine this assumption by contributing to poor quality data, lack of understanding of the effects of poor quality data on organisational success, poor planning, ‘siloed’ system design, inconsistent development processes, incomplete documentation, lack of standards, or lack of governance. Many organisations fail to define what makes data ‘fit for purpose’. All data management disciplines contribute to quality and high quality data that supports the organisation should be the goal of all data management disciplines. Since uninformed decisions or actions by anyone interacting with data can result in poor quality data, producing high quality data requires cross-functional commitment and co-ordination. Organisations and teams should be aware of this and should plan for high quality data, executing processes and projects in a way that takes into account risks related to unforeseen or unacceptable conditions in the data.
Activities carried out by our Team:
Defining High Quality data
Many people recognise poor quality data when they see it; fewer are able to define what is meant by high quality data. Before launching a data quality programme, it is useful to understand the business needs, define terms, identify weaknesses in the organisation and begin to build consensus on drivers and priorities for data quality improvement. A series of questions must be asked to understand the current state and assess organisational readiness for data quality improvement.
Defining a Data Quality strategy
Improving data quality requires a strategy that takes into account the work that needs to be done and how people will do it. Priorities on data quality must align with the business strategy. The adoption or development of a framework and methodology will help guide both strategy and tactics while providing means to measure progress and impact. A framework should include methods for:
- Understanding and prioritising business needs
- Identifying key data to meet business needs
- Defining business rules and data quality standards according to business requirements
- Assessing data according to expectations
- Sharing results and get feedback from stakeholders
- Prioritising and manage issues
- Identifying and prioritising opportunities for improvement
- Measuring, monitoring and writing reports on data quality
- Managing the metadata created through data quality processes
- Integrating data quality controls into business and technical processes
A framework should also explain how to organise data quality and how to use data quality tools.
Identify critical data and business rules
Not all data are of equal importance. Data Quality management activities should first focus on the organisation’s most important data: data that, if of higher quality, would provide greater value to the organisation and its clients. Data can be prioritised according to regulatory requirements, financial value and direct impact on clients.
Carrying out an initial Data Quality Assessment
Once the most critical business needs and the data that support them have been identified, the most important part of the data quality assessment is actually examining that data, interrogating it to understand its content and relationships, and comparing the actual data with the rules and expectations. The first time this is done, analysts will discover many things: undocumented relationships and dependencies within the data, implicit rules, redundant data, contradictory data, etc., as well as data that actually conforms to the rules.
Identifying and prioritise potential improvements
Having demonstrated that the improvement process can work, the next objective is to apply it strategically. This requires the identification and prioritisation of potential improvements.
Identification can be achieved by large-scale profiling of larger datasets to understand the breadth of existing issues. It can also be achieved by other means, such as interviewing stakeholders on the data issues that impact them and following up with analysis of the business impact of these issues. Ultimately, prioritisation requires a combination of data analysis and discussion with stakeholders.
Defining objectives for Data Quality improvement
The knowledge gained through preliminary evaluations forms the basis for specific objectives of the data quality programme. Improvement can take different forms, from simple remediation (e.g. correcting errors in records) to correcting causes at their root. Repair and improvement plans should take into account quick wins – problems that can be solved immediately at low cost – and longer-term strategic changes. The strategic objective of such plans should be to address the root causes of problems and, first and foremost, to put in place mechanisms to prevent problems.
Developing and provide Data Quality operations
Many data quality programmes are initiated through a series of improvement projects identified through data quality assessment results. In order to sustain data quality, a DQ programme should put in place a plan that allows the team to manage data quality rules and standards, monitor ongoing compliance of data with rules, identify and manage data quality issues and write reports on quality levels.