Variance

Variance


Variance describes how much a machine learning model reacts to small changes in its training data. A high-variance model produces noticeably different predictions when you drop a handful of examples from the training set. A low-variance model says roughly the same thing either way.

The idea only means something next to bias. Bias is the model sitting systematically in the wrong place, variance is the model being unstable. A simple model usually shows high bias and low variance, a very flexible one shows the reverse. Overfitting is what high variance looks like from the outside: the model memorises the training data, mistakes its noise for rules, and stumbles on anything new.

Several levers bring variance down in practice. Collecting more training data, adding regularization, reducing model complexity, and averaging predictions across several models through ensemble methods such as random forests are the usual ones.

A concrete case: a decision tree predicting return risk on an ecommerce site scores almost perfectly on the training set when its depth is left unbounded. The same model behaves erratically in production, because the tree has learned branches specific to individual orders. Cap the depth and training accuracy drops while live predictions steady out.

To measure it, teams watch the gap between training and validation scores. The wider that gap opens, the bigger the variance problem.

From generative AI strategy to custom agent development and retrieval architectures, we help you scale AI responsibly.
Discuss your AI project