Test Data

Test Data


Test data is the set held back from training and used once, after training is finished, to measure how the model performs. The separation is the whole point. The model learns nothing from these examples, it is only measured on them.

Data is usually split three ways. The training set teaches the parameters, the validation set drives hyperparameter choices and early stopping, and the test set is run once at the end. Because the validation set gets looked at repeatedly it becomes indirectly contaminated over time, which is why the headline number has to come from the test set.

The real risk in practice is leakage. If the same record sits in both training and test, or test examples are near-duplicates of training ones, the measurement looks better than reality. With time series data the split is made by date rather than at random, because putting a future record into training hands the model information it will not have in production.

A concrete case: a bank builds a model to score loan applications. Split randomly, the test score comes out high. Split by application date, the score drops noticeably, because the model is now predicting the next three months from the past alone. The second number is closer to real performance.

A test set is spent once. Looking at the result, changing the model and rerunning against the same set turns it into another validation set.

From generative AI strategy to custom agent development and retrieval architectures, we help you scale AI responsibly.
Discuss your AI project