Five data quality checks before you report a number

Why check data quality before reporting?

A number in a report is a promise. Donors, partners and your own managers will use it to make decisions. If the number is wrong, the decision may be wrong too. Errors also cost trust: one discovered mistake makes people doubt every other figure you report.

The good news is that you do not need perfect data. USAID's guidance on data quality standards says that quality must be matched to how the data will be used. It also warns managers not to let the ideal drive out the good. The aim is data that is good enough for the decision at hand, with clear notes on its limits.

What are the five data quality standards?

USAID's performance monitoring guide lists five standards. Many NGOs and donors use them. They are easy to remember and easy to check.

How do you check each one? Five quick checks

1. Check validity: read the indicator definition. Write down exactly what the indicator measures and who counts. Then look at the question or form that produces the number. Do they match? If your result is "increased household income" and your indicator is a proxy, such as roof type, state the assumption.
2. Check reliability: compare the tools and the people. Are all teams using the same version of the form? Did every enumerator receive the same training? Compare results from different data collectors for the same area. Large differences point to a problem in how people ask or record.
3. Check precision: look at the sample and the detail. Is the sample large enough to say what you want to say about each group? Are numbers rounded or estimated? If you cannot report by sex, age or location, say so. Our guide to disaggregated data explains what is needed.
4. Check integrity: trace a sample back to the source. Pick ten numbers from your report. Follow each one back to the monthly summary, then to the original form or register. Look for copy errors, double counting and missing records. Ask who might gain if the number were higher. Targets and pressure from above are common reasons for inflated figures.
5. Check timeliness: look at dates. When was the data collected, and when did it reach you? If it is old, say so, and flag any preliminary estimates until final data are available.

What do the checks look like in the field?

Some simple habits make these checks routine:

• Use digital forms with rules. Set required fields, allowed ranges and skip logic in your data collection tool. This prevents many errors at entry. Our guide on KoboToolbox shows how.
• Review data weekly. Look for duplicates, blank answers and odd values, such as a household of 40 people.
• Supervise a sample. A supervisor revisits a few households or listens to a few calls, then compares the answers.
• Keep a data log. Record corrections: what changed, why, who approved it.
• Protect the original records. Never overwrite raw data. Clean in a copy.

What should you do when you find a problem?

Do not hide it, and do not stop reporting. Decide which of these three fits:

1. Fix it. If the error is clear, such as a typing mistake, correct it and note the change.
2. Report it with a caveat. If the number is usable but weak, report it and add a short note on the limit. For example: "Based on 45 of 60 sites; 15 sites could not be reached."
3. Do not report it. If the number is misleading, leave it out and explain why. Plan how you will collect it better next time.

Then find the root cause. Was the indicator unclear? Was training too short? Were targets unrealistic? A fix at the root prevents the same problem next month. For remote work, see our guide on remote monitoring, and for the document that defines indicators, see indicator reference sheets.

Common mistakes to avoid

• Treating data quality as a once-a-year task. Checks are most useful while data is being collected, when errors are cheap to fix.
• Chasing perfection. Spending weeks on a small gap is rarely worth it. Match effort to the decision.
• Rewarding only good numbers. If staff are punished for bad results, they will hide them.
• Mixing definitions. A "beneficiary" counted by household in one report and by person in another will not add up.

How EMPO can help

EMPO is a trusted training and consultancy provider based in Yangon and one of Myanmar's leading names in practical NGO training, with 1,200+ people trained and 15 partner organisations. Our MEAL training covers indicators, data collection and data quality, online or in person across Myanmar, in English and Burmese. See our training services or contact us to plan a session for your team.

Sources

• Performance Monitoring and Evaluation TIPS 12: Data Quality Standards — USAID, 2009