Five Steps to Better Data Quality

In the information age, data is the calorie that drives business decisions. Just as there are good calories and bad calories, not all data is equal. As more and more companies use data to drive their decision-making processes, getting quality data is crucial to their success.

Data science has been around since the 1980s, in the form of concepts such as data warehousing. If you scour the internet, looking for advice on getting better data quality, you’ll find long-standing concepts such as metrics of accuracy and timeliness along with recent tactical advice on how to sanitize data using Python or R. How do we better navigate through this dizzying array of information?

Here are five steps you can take today to improve the quality of your data.

Step 1: Define Your Usefulness Metrics

To help managers make better decisions faster or to help employees be more responsive, you must define what “useful data” looks like.

These are the most common metrics we use to define useful data:

• Accuracy
• Precision
• Completeness
• Validity
• Relevancy
• Timeliness
• Ability to be understood
• Trustworthiness

Step 2: Profiling

Profiling involves analyzing the data you’re working with in order to clarify its structure, content, relationships, and derivation rules. This is a crucial step for machine learning because users typically have an intuitive understanding of how data is interrelated. However, machines currently need precise instructions so it’s necessary to profile the data at hand and make it work for your user via data analysis software.

To begin, you should clarify how different data points are related to one another. How do you want to group and structure them? What rules do you want to apply to the data to derive for display purposes? These steps are typical when performing a data profile.

After you’ve performed in-depth profiling, you may continue to perform detailed profiling. This is because continuous detailed profiling helps determine the appropriate data for extraction and the appropriate filters to apply to your data set.

After you load the data into memory, you may want to continue profiling the data to ensure that it is correctly sanitized and transformed to comply with your requirements.

Step 3: Standardization

Standardizing data is another crucial step for improving quality. Standards help improve communication among teams.

Good communication requires the ability to deliver complex information clearly and concisely, with minimal confusion. This is true for conveying your data to your audience.

There are two types of standards: external and internal. External standards (as in outside your organization) are appropriate for commonly used data types like datetime. For example, if you wanted to represent datetime, you’d choose a widely accepted international standard like ISO-8601. I’d advise you not to invent your own standards needlessly and don’t choose obscure standards. Remember, your goal is to communicate data easily and effectively. Therefore, external standards such as ISO-8601 should be chosen wisely.

In some situations, you may need to create your own internal standards. It takes more work, but it is possible to do so. Internal standards also help improve communications within your company. They can also serve another purpose—for example, imagine that your business has a revolutionary process that allows you to ship twice as fast as your competitors. This is a huge competitive edge for you and will likely require the entire company to work within this revolutionary process; however, if the same vocabulary is consistently applied in the data as well then there is a good chance that your staff will stay on the same page and work within the new paradigm that has been created.

Step 4: Matching or Linking

If you have properly defined your data model and performed profiling, but your audience is still not getting the kind of useful insights they thought they would, then you need to add matching and linking capabilities.

Recall that we talked about relationships and the structure of your data earlier, in Step 2: Profiling. You need to show your audience the relationships you discover in your data. When the relationships are in place, your audience will be able to perform a wide array of operations on the data, rolling up, drilling down, and slicing and dicing the data as they need. In other words, they’ll have business intelligence via online analytic processing.

Imagine being able to analyze sales data that is tied to customers’ demographics, as well as product inventory. Now you will be able to predict trends and buy patterns based on product, transaction time, or demographics. It’s the same group of data but it can be analyzed in three different ways.

Step 5: Monitoring

The work of a good data analyst is never done. You need to constantly monitor the changes in the data you receive, analyze those changes and adjust your analysis accordingly. Changes may be brought about by a new competitor in the scene, or maybe there’s a change in regulation. Technology advances may also cause you to change your data analysis process.

Software and data can decay, too. Continuous profiling may lead you to change your policy in order to remain competitive. New standards may need to be introduced to meet those standards.

Monitoring data is crucial. If there is any doubt about the integrity of your data, it should be thoroughly checked before it is used. There are many tools available to help alleviate this workload; one example is software that sends notifications to departments responsible for collecting or sanitizing data whenever your monitoring software picks up anomalies such as wrongly inputted data.

Now that you know the five steps for improving your data quality control, take note of which ones you’re already doing well and put in place a quarterly review process to ensure that you’re continually evaluating your data quality control. This way, you’ll always be seeing where you stand and where you can improve.