Keep an evaluation set the model never saw: 150 of 1,000 in the example

A model evaluation set tests whether your trained model can handle data it has never seen before. Building a good one means picking examples and labels that truly represent the task you want your model to solve, and keeping them separate from all data used during training or tuning.

The evaluation set checks what the model still gets wrong

Hold out a portion of your labeled data—never used during model training or tuning—to see what still trips up your model after you think it is ready[1]. This is your evaluation (test) set, and its only job is to show how the model performs on new, unseen data. If you use it only at the very end, you get the closest thing to a real-world check: did your model learn patterns that generalize, or just memorize the training set?

The split itself is up to you; no source gives a rule or fixed percentage for how much data to hold out. What matters is that the evaluation set never influences training or hyperparameter choice[1]. Use the validation set for tuning—never the evaluation set. The evaluation data is for one purpose: a final, unbiased measure.

[1] Training, validation, and test data sets - Wikipedia

Method and Sources: See Training, validation, and test data sets - Wikipedia.

What makes an evaluation set trustworthy

A good evaluation set must match the real-world data your model will see. If your model will classify emails in English, your evaluation set should include emails with the same styles, lengths, and quirks as your target environment[1]. If the evaluation data is too easy, too hard, or unrepresentative, the results will mislead you.

Key factors:

• Distribution: The evaluation set should follow the same probability distribution as your training and validation sets[1].
• Label balance: Each class or outcome should be represented according to its real-world frequency, or evenly if you want to measure class-specific performance.
• Sample size: No source gives a fixed percentage; 70/15/15 for training/validation/evaluation is a common illustrative choice, and the right size depends on your data. For a small custom task, a 150-example evaluation set (50 per label when the labels are equally common) is illustrative; for critical tasks, larger is better.
• Beyond the random split: Gardner et al. (2020) show that a test set drawn the same way as the training data can leave a model's local decision boundaries untested, and propose contrast sets: small, hand-made changes to test examples that flip the label, written by the dataset's own authors [2].

If your process or data changes over time, so should your evaluation set. Periodically refreshing the evaluation data ensures your model is still evaluated against current data patterns[1].

A worked example: splitting data for a three-label task

Suppose you have 1,000 labeled customer support tickets, each tagged as billing, technical, or general. For this example, you want to split these tickets into three sets: one for training the model, one for tuning, and one for final evaluation.

A typical split (an assumption for this example, not a measured standard) allocates 70% of the data to training and 15% each to validation and evaluation. The example also assumes the three labels are equally common; with real data, give each split the same label mix as the whole set. That means:

• 700 tickets for training, with the same label mix as the whole set
• 150 tickets for validation, with the same label mix
• 150 tickets for evaluation, which at equal label counts is 50 from each category

You train the model using the 700 training tickets. The validation set is used to select hyperparameters—like the model's size or regularization method. Only after all tuning choices are done do you check the model on the 150 evaluation tickets. This step is where you get a true, unbiased measure of performance on data the model has never seen.

What happens if you build the evaluation set poorly

Pass rule: No overlap—if examples from your evaluation set leak into training or hyperparameter tuning, your model’s reported accuracy is inflated and won’t hold outside the lab[1]. This is classic data leakage: your model gets to “see the answers” it should never touch.

If the evaluation set is too small or misses the range of data your model faces in production, you may see high metrics that collapse on real-world deployment. For example, a model tuned only on common cases can easily miss rare but critical categories, and your test numbers will not warn you.

Avoiding these pitfalls means:

• Keep splits strictly separate, with no reused examples.
• Check the evaluation set for coverage—does it match the variety you expect in real use?
• Do not loop on evaluation results to tweak the model; treat evaluation as a final check, not a guide for more tuning[1].

Packages like easyml[3] automate strict splits and enforce these patterns, especially for common algorithms, but always verify that no feedback loop is introduced by how you run them.

Method and Sources: Wikipedia, “Training, validation, and test data sets”[1]; easyml abstract[3].

When and how to update the evaluation set

Over time, new types of data or changing distributions can make your original evaluation set obsolete. For example, if customer support tickets start including messages in new formats or languages, your evaluation set should be updated to reflect these changes[1].

The process:

1. Collect new data that matches your current real-world task.
2. Carefully separate new evaluation examples from all data used in retraining or tuning.
3. Repeat the split, keeping the evaluation set as a one-time, final test for each model version.

This cycle helps ensure your model’s reported performance remains meaningful as your environment evolves.

Questions readers ask

Why can’t I just use the training set to evaluate my model?

Using the training set inflates performance by measuring on data the model has already seen. This does not reflect how the model will perform on truly new data[1].

How big should my evaluation set be?

There is no single rule, but it should be large enough to confidently measure performance across all labels or categories you care about. For small custom tasks, 150 examples (with 50 per label) is a common illustrative split, but more is better if available.

What if my data is too small for three separate sets?

If data is very limited, techniques like cross-validation can reuse data more efficiently, but you must still ensure that no data used for evaluation is involved in any model tuning or training[1].

How the numbers work

These figures were computed for this article, not estimated by its writer. Inputs come from the numbered sources or are assumptions, labelled as such; change an assumption and the results change with it.

Sources

1. Training, validation, and test data sets - Wikipedia
2. Gardner, M. et al. (2020). Evaluating Models' Local Decision Boundaries via Contrast Sets. Findings of EMNLP 2020
3. Easyml: Easily Build and Evaluate Machine Learning Models