Data Is Not Neutral: The Hidden Risks Behind AI Decisions

By Edwin Bernard, MSc, CEFA, CAITL (Certified Artificial Intelligence Transformation Leader, USAII) Hong Kong, November 2025

Introduction

AI is often seen as objective because it uses data and algorithms, not human judgment. This belief is not just wrong—it’s risky. AI systems, including Machine Learning (ML) models, are only as unbiased as the data they learn from. If that data is incomplete or skewed, these systems will reflect and amplify those biases.

AI systems are developed by humans and learn from data created and curated by people whose decisions are influenced by personal preferences, values, opinions, and unconscious biases. These biases seep into data collection, labelling, and algorithm design. Many AI models are trained on historical data that reflects existing societal inequalities and discrimination, so the AI learns and perpetuates these patterns.

It’s not just hard—it’s impossible—to collect a perfectly complete and representative dataset of the entire world and all its nuances. Even if we captured every possible scenario, rare cases would still be rare because they occur infrequently in reality. This structural imbalance means some groups will always be underrepresented, creating silent risks that models cannot overcome without intervention. Furthermore, what is considered “fair” or “unbiased” changes over time as societal norms evolve.

Building trustworthy AI requires more than technical accuracy—it demands ongoing vigilance and responsibility. In this article, we demonstrate how bias in training data impacts model performance using a simple example: applying Ordinary Least Squares (OLS) regression to a dataset of Engine Size and CO2 emissions, publicly available on Kaggle.com—a global online platform for data science and machine learning that hosts datasets, competitions, and tools for practitioners and researchers. Through two scenarios—one where Diesel cars dominate the training set and another where they are excluded—we show how bias distorts model assumptions and masks subgroup failures, even when overall metrics look strong.

The impact of biased data

OLS is a basic supervised machine learning algorithm for tasks that involve predicting numeric outcomes. It learns a linear relationship between input features (e.g., Engine size) and the target variable (CO2 emissions) by minimizing prediction errors. In formula form:

CO2 emission = intercept + slope x Engine size + prediction error

Model performance is commonly evaluated using:

• R2 (Coefficient of Determination): Indicates how much of the variation in the target (CO2 emissions) is explained by the model.
• MAE (Mean Absolute Error): The average size of prediction errors, measured in absolute terms.
• RMSE (Root Mean Squared Error): The average size of prediction errors, giving more weight to larger mistakes.

We split our dataset into 10% for training (770 observations) and 90% for testing on unseen data (6,615 observations).

Scenario 1: Diesel-Only Training Data
From the 10% training set (770 observations), we selected only Diesel cars (19 observations) and trained an OLS model. For comparison, we also trained a model on a random sample of the training set of the same size (19 observations).

This gives the following model parameters:

Model	intercept	slope	Formula
Diesel only	90.3	55.8	CO2 =   90.3 + 55.8 x Engine size + prediction error
Random	150.4	36.6	CO2 = 150.4 + 36.6 x Engine size + prediction error

The intercept and slope differ significantly. The Diesel-only model starts at a lower baseline but rises much faster with engine size, reflecting Diesel-specific patterns.

Model	Training Data Set
	R2	MAE	RMSE
Diesel only	63.3%	18.6	22.7
Random	65.8%	27.1	33.3

On their respective training sets, both models explain a similar amount of variation (R2 ≈ 63–66%). At first glance, they look equally good, but the Diesel-only model has lower errors because it fits its narrow subset better.

However, evaluating a model on unseen data is crucial because it shows how well the model generalizes beyond the examples it was trained on. A model might look accurate on its training set—even if that data is biased—but real-world predictions involve new cases. Testing on unseen data helps reveal overfitting and bias, ensuring the model is robust and reliable for broader applications.

Testing the models on the 90% unseen data (6,615 observations), and also specifically on Diesel cars within that 90% (156 observations), gives the following results:

Model	90% Unseen Data			Diesel within 90%
	R2	MAE	RMSE	R2	MAE	RMSE
Diesel only	45.7%	32.1	43.2	63.3%	20.5	25.5
Random	65.4%	27.2	34.5	50.6%	24.7	29.6

The Diesel-only model performs poorly on the full unseen dataset (R2 drops to 45.7%), while the random model maintains stronger generalization (R2 = 65.4%). Interestingly, the Diesel-only model still does well on Diesel cars within the unseen set, confirming its bias toward that subgroup.

This demonstrates how biased training data can distort model assumptions and limit generalization. The Diesel-only model appeared accurate on its training set, but its performance collapsed on the broader dataset. In contrast, the randomly trained model generalized better across all fuel types.

Scenario 2: No-Diesel Training Data
From the same 10%, we select a sample with no Diesel cars and also compare this to a random sample of equal size (751 observations). This results in the following model parameters:

Model	intercept	slope	Formula
No Diesel	136.9	36.0	CO2 = 136.9 + 36.0 x Engine size + prediction error
Random	136.4	36.2	CO2 = 136.4 + 36.2 x Engine size + prediction error

When Diesel cars are excluded from the training sample, the resulting OLS model closely matches the model trained on a random sample of the same size. Both intercept and slope are nearly identical, indicating that the absence of Diesel vehicles does not significantly distort the relationship between engine size and CO2 emissions for the remaining data. This suggests that bias becomes most problematic when the training set is dominated by a specific subgroup (e.g., Diesel-only), whereas excluding a subgroup entirely may have less impact if the remaining data is still representative of the overall population.

Testing the models on the 90% unseen data and on Diesel cars within that 90% results in:

Model	90% Unseen Data			Diesel within 90%
	R2	MAE	RMSE	R2	MAE	RMSE
No Diesel	72.4%	23.3	30.8	48.1%	25.1	30.4
Random	72.4%	23.3	30.8	48.3%	25.0	30.3

On the full unseen dataset, both models perform strongly (R2 = 72.4%). But when tested specifically on Diesel cars, performance collapses to ~48% R2—even for the random model that does include Diesel vehicles.

This gap highlights an important limitation: excluding Diesel vehicles—or including them in very small numbers, as in the random sample—leaves the model unable to capture Diesel-specific emission patterns, making predictions for this subgroup highly unreliable despite strong aggregate metrics. In other words, bias is not just about exclusion; severe underrepresentation can be equally damaging, and it is a structural reality in real-world data.

Critical Insight: Overall performance metrics can mask subgroup failures, creating a false sense of security when minority groups are underrepresented.

Why Underrepresentation Creates Hidden Risks

Underrepresentation is the most dangerous form of bias because its impact is silent and hard to detect—and it is virtually unavoidable. No dataset can perfectly capture the full diversity of real-world scenarios. Every training set is a tiny slice of reality, which means some groups will always be missing or underrepresented.

Underrepresentation is also not just a sampling artifact—it’s a structural reality. Even if we captured the entire universe of possibilities, rare scenarios remain rare because they occur less often in the real world.

When a model learns from data dominated by majority patterns:

• It overfits to those patterns.
• It underperforms on minority groups, as seen with Diesel cars when they are excluded or severely underrepresented.
• It creates systematic blind spots, which in real-world applications can lead to unfair or harmful outcomes.

The hidden nature of this risk makes it particularly dangerous:

• Small groups can be invisible in overall metrics. A model may appear accurate while failing badly for underrepresented segments.
• Training datasets are often tiny compared to all possible scenarios, and even if we captured the entire universe of possibilities, rare scenarios would still be rare. This structural imbalance makes underrepresentation an unavoidable and silent risk—only revealed when the model fails in production.

Conclusion

Absolute objectivity in AI is unattainable because data can never capture the full complexity of reality—there will always be missing information. We do not know what we do not know, and what we do know is biased. Underrepresentation is not merely a sampling flaw; it is a structural fact. Even if we had data covering every possible scenario, rare cases would remain rare because they occur infrequently in the real world.

Our OLS case study demonstrated how subgroup dominance or exclusion distorts model assumptions and hides failures behind strong aggregate metrics. When a subgroup dominates the training set, the model becomes overly tuned to that subgroup and fails to generalize. Conversely, when a subgroup is excluded or severely underrepresented, the model may appear strong overall yet perform poorly for that subgroup. This illustrates a critical risk: underrepresentation is a silent but structural problem, nearly as harmful as complete exclusion.

The goal should not be to eliminate bias—because that is impossible—but to reduce it through deliberate actions: building diverse datasets, applying fairness checks, and monitoring continuously. Ongoing monitoring in production is crucial to detect emerging bias issues early, before they lead to harmful outcomes. Trustworthy AI is not a one-time achievement but an ongoing commitment to continuous improvement and adaptation. It is dynamic, not static!