Fine-tuning BERT models for recipe classification
This project compared twenty-two fine-tuned configurations of BERT Base, BERT Large, and RoBERTa for classifying recipes as vegetarian or non-vegetarian from text alone.
The best held-out accuracy was 0.869, produced by BERT Base rather than either larger architecture. More interestingly, validation accuracy was almost unrelated to held-out accuracy across the twenty-two runs. That result changed how I would design the comparison if I repeated it.
Why the task is contextual
A vegetarian classifier cannot be reduced reliably to a list of prohibited words.
Vegetable broth and chicken stock occupy similar positions in an ingredient list but imply different labels. Gelatin, anchovy in a sauce, and lard in a pastry can each decide the classification. At the same time, a recipe description that says a dish pairs well with roast lamb does not make the dish itself non-vegetarian.
The relevant signal depends on context. That makes the problem well suited to a bidirectional language model.
The input consisted of each recipe's description concatenated with its tokenized ingredient list. Missing descriptions were filled as empty strings. The development data were split 80/20 into training and validation sets. Final performance was measured on a separate held-out set that the models never saw during tuning.
Three architectures
BERT Base has twelve transformer encoder layers, 768 hidden units, twelve attention heads, and about 110 million parameters.
BERT Large increases this to twenty-four layers, 1024 hidden units, sixteen attention heads, and roughly 340 million parameters.
RoBERTa retains the basic transformer architecture but changes the pretraining procedure, including the use of dynamic masking.
All three architectures were fine-tuned end to end with AdamW and cross-entropy loss. Early stopping was based on validation accuracy.
The larger models did not perform best on the held-out set. BERT Large occupied three of the bottom eleven positions. RoBERTa produced the highest single validation accuracy in the study but did not place in the top six on held-out performance.
One possible explanation is that twenty thousand short texts do not provide enough task-specific signal to make the added capacity useful. The larger models may have been more prone to overfitting.
There is an important confounder, though. The larger architectures were also tuned less extensively because each run cost more. Their lower performance cannot be attributed cleanly to model size alone.
The result that mattered most
Each of the twenty-two runs produced both a validation accuracy and a held-out accuracy. Treating those pairs as a small dataset reveals the most useful result from the project.
Validation accuracy against held-out accuracy
The Pearson correlation between validation and held-out accuracy is 0.09. The rank correlation is −0.09.
There is essentially no relationship.
The configuration with the highest validation accuracy, a RoBERTa run at 0.8600, placed seventh on the held-out set. The eventual winner had a validation accuracy of 0.8515, near the middle of the observed range. Even the lowest validation accuracy in the study, 0.8352, outperformed three runs with better validation values.
The validation set contained roughly four thousand recipes. For an accuracy near 0.85, the standard error of a single proportion is about 0.006.
That matters because many of the observed differences between configurations were only around one percentage point. The total spread of validation accuracies across all twenty-two runs was 0.0248.
Most of the apparent ranking was therefore too small to separate confidently from sampling noise.
I would use k-fold cross-validation if I repeated the study.
Five folds would reduce dependence on a single split and provide a variance estimate for each configuration. That would make it possible to distinguish meaningful model differences from ordinary sampling variation.
The study supports the conclusion that a fine-tuned BERT Base model performs around 0.86 on this task.
It does not support a precise ranking of the top several configurations.
What the tuning did establish
Some choices produced differences large enough to be informative despite the noisy ranking.
Sequence length was chosen from the observed token distribution. The maximum recipe length was 556 tokens, beyond BERT's 512-token ceiling. The mean was only 73.7 tokens, and the ninetieth percentile was 126.
A maximum length of 128 therefore retained about ninety percent of recipes without truncation while using much less memory than a length of 256. The single 256-token configuration placed eighth and did not justify the added cost.
Weight decay also produced a visible difference. A value of 0.001 appeared in the best run. Increasing it to 0.02 reduced held-out accuracy by about two percentage points, while removing weight decay produced validation curves consistent with stronger overfitting.
Batch size settled at 64. A batch size of 128 converged faster but generalized worse. A batch size of 32 cost more training time without a clear gain.
A static learning rate of 1e-5 performed best among the rates tested. A warm-up schedule smoothed early training but did not improve the final result. A rate of 1e-6 was too slow to converge within the epoch budget.
A useful repeat run
The winning configuration was run again with the same hyperparameters. This time it trained for nine epochs rather than five and finished fifteenth of twenty-two.
Training loss and validation accuracy, by epoch
In the winning run, training loss falls from 0.489 to 0.226. Validation accuracy peaks at epoch three, dips at epoch four, and recovers at epoch five without exceeding its earlier maximum. Early stopping with patience two ends training there.
The repeat run makes the weakness of the stopping signal visible.
Validation accuracy varies by only about four thousandths across epochs three to five. That is well inside the sampling noise already described. Whether the patience counter stops at epoch five or survives several more epochs can therefore depend on small changes from data shuffling and stochastic optimization.
This does not mean early stopping failed. It means the stopping rule inherited the instability of the metric it was monitoring.
What I would change
The first change would be cross-validation for model selection.
I would also test lighter architectures such as DistilBERT and ALBERT. The dataset is modest in size and the classification domain is narrow, so additional parameter count may not be useful.
Ensembling across random seeds would be another natural extension. The repeated winning configuration showed that seed-level variability was large enough to matter.
Finally, I would compare the validation and held-out data distributions directly. The weak correlation may be explained entirely by sampling noise, but a systematic difference between the two splits would produce the same pattern and should be ruled out empirically.