Before you tune the AI model, inspect the data

In 2023, I published a note called Data-centric AI / Big data vs. Good data.

The title was deliberately simple. At the time, much of the public AI conversation focused on larger datasets and better models. The practical work kept pointing to a less exciting problem: the model often inherited inconsistent labels, unclear definitions, and decisions that nobody had documented.

I would phrase the conclusion more carefully today. More data can matter. Model choice can matter. But neither fixes a dataset whose meaning changes from one row, source, or annotator to the next.

Why teams tune the model first

Model changes are visible. You can change an algorithm, run an experiment, and compare a metric.

Data work is less tidy. It means tracing sources, reviewing ambiguous examples, asking subject-matter experts what a label actually means, and sometimes discovering that the original target was poorly defined.

That makes model tuning tempting even when the bottleneck is elsewhere.

The better first question is not “Which model should we try next?” It is:

What evidence tells us that the current data represents the decision we want the system to make?

Good data is not spotless data

“Good data” does not mean a dataset with no missing values or unusual records. Real operational data is rarely that clean.

It means the data is fit for a defined use:

• one row and one label have an agreed meaning;
• the source and transformations are traceable;
• important edge cases are represented;
• train and evaluation records are separated correctly;
• known gaps and exclusions are documented;
• errors can be connected back to a data decision.

A large dataset can fail all six checks. A smaller dataset can still be useful if its boundaries are understood.

A practical data-centric loop

The workflow I use is iterative:

1. Define the unit and target. Write down what one record represents and what the output will be used for.
2. Trace the source. Record where fields and labels came from, including transformations and manual decisions.
3. Find disagreement. Compare annotators, systems, or business rules. Disagreement is evidence that the definition needs work.
4. Inspect rare and ambiguous cases. Aggregate accuracy often hides the examples that matter most in production.
5. Turn decisions into checks. A resolved edge case should become a validation rule, test, or annotation guideline.
6. Run error analysis by slice. Review failures by source, class, time period, geography, or another meaningful business dimension.
7. Create a data feedback loop. Production errors should improve the dataset and its documentation, not disappear into a backlog.

This is not a one-time cleanup before “real” model development. It is part of model development.

Where domain experts enter

Subject-matter experts are not there only to approve the final demo.

They help define what counts as a valid label, which ambiguity is acceptable, which error is expensive, and when the system should return “unknown” instead of a confident answer.

That collaboration is especially important when the dataset describes clinical decisions, addresses and locations, customer operations, financial events, or any workflow where a technically valid value can still have the wrong business meaning.

What this means before building an AI feature

Before investing in an AI feature, I would inspect four things:

• the business decision the output should support;
• the provenance and consistency of the available data;
• the edge cases that could create false confidence;
• the feedback mechanism after the system is used.

Sometimes that review confirms that the data is ready for a prototype. Sometimes it reveals that a validation layer, labeling guide, or smaller data-quality project should come first.

Both outcomes are useful. Discovering that the foundation is weak before model development is cheaper than discovering it after a polished demo.

Evidence boundary

This note is a practitioner’s interpretation of data-centric AI, not a new research result or a claim that data quality is the only factor in model performance.

The original 2023 article discusses model-centric and data-centric approaches, the role of subject-matter experts, and examples of labeling inconsistency. You can read it on Medium or in the original Atlantbh publication.

Cover photograph by Alex Knight, reused from the original article.