Fine-tuning BERT models for recipe classification
Deciding whether a recipe is vegetarian sounds like a lookup problem and is not one. The distinguishing evidence is often a single ingredient whose status depends on wording that a keyword list will not catch, which makes it a good test of whether contextual language models earn their cost. Twenty-two configurations of BERT Base, BERT Large, and RoBERTa were fine-tuned and compared. The best reached 0.869 on held-out data, and the comparison produced a more useful finding than the winner did.
Why the task is not a lookup
A vegetarian classifier built from a banned-ingredient list fails on the cases that matter. Vegetable broth and chicken stock are both broths, occupy the same syntactic position in an ingredient list, and differ by one modifier. Gelatin, anchovy in a sauce, and lard in a pastry are all decisive and all easy to miss. Meanwhile a description mentioning that a dish pairs well with roast lamb is not evidence about the dish itself. The signal is contextual, which is the condition under which a bidirectional transformer should outperform simpler methods, and the point of the exercise was to find out by how much and at what cost.
The input was the recipe description concatenated with its tokenised ingredient list, with missing descriptions filled as empty strings, split eighty-twenty into training and validation. Final evaluation was on a held-out set the models never saw, scored externally.
Three architectures
BERT Base has twelve transformer encoder layers, 768 hidden units, twelve attention heads, and roughly 110 million parameters. BERT Large has twenty-four layers, 1024 hidden units, sixteen heads, and about 340 million parameters, giving it substantially more capacity for long or syntactically complicated text. RoBERTa keeps BERT's architecture and changes the pre-training: dynamic masking, so that different tokens are hidden on different passes rather than a mask fixed once, which prevents the model from over-fitting to a single masking pattern. All were fine-tuned end to end with AdamW and cross-entropy loss, and all were regularised with early stopping on validation accuracy.
The larger models lost. BERT Large occupied three of the bottom eleven places, and RoBERTa, despite recording the single highest validation accuracy of any run, did not reach the top six on held-out data. The straightforward explanation is that a dataset of twenty thousand short texts does not supply enough signal to fit 340 million parameters, and that the additional capacity was spent memorising the training split. The more careful explanation is that the larger models were also tuned less thoroughly, because each run cost more, so their under-performance is confounded with the search budget they received. Both readings should be stated; only the first is flattering.
The finding worth keeping
The twenty-two runs form a small dataset in their own right, with a validation accuracy and a held-out accuracy for each. Plotting one against the other gives the result that has stayed with me from this project.
Validation accuracy against held-out accuracy
Across the 22 runs the Pearson correlation between validation accuracy and held-out accuracy is 0.09, and the rank correlation is −0.09. The relationship is not weak; it is absent. The run with the highest validation accuracy of all, a RoBERTa configuration at 0.8600, placed seventh on held-out data, and the winning run sat in the middle of the validation distribution at 0.8515. The lowest validation accuracy in the study, 0.8352, still beat three configurations with visibly better validation numbers.
The explanation is not mysterious, and it is worth spelling out because it generalises well beyond this task. Validation accuracy here was computed on a single twenty-percent split of roughly four thousand recipes. The standard error on an accuracy estimate near 0.85 from that many observations is close to 0.006, so a difference of a single percentage point between two configurations is inside the noise. The entire spread of validation accuracies across all 22 runs was 0.0248, or about four standard errors, which means most of the ordering produced by that column is measurement error rather than a real difference in model quality. Selecting a model on it is close to selecting at random among the reasonable candidates.
The right instrument was k-fold cross-validation, which was not used. Five folds would have cut the standard error on each configuration's estimate by more than half and, more importantly, would have supplied a variance estimate per configuration, making it possible to say whether any two of them actually differed. The study as conducted can establish that fine-tuned BERT Base performs around 0.86 on this task. It cannot establish which of its top ten configurations is best, and it would have been wrong to claim otherwise.
What the tuning did establish
Some choices produced differences large enough to survive the noise, and some were settled by evidence rather than by search.
Sequence length was decided by measurement. Tokenising the corpus gave a maximum length of 556 tokens, past BERT's 512-token ceiling, but a mean of only 73.7 and a ninetieth percentile of 126. A limit of 128 therefore covers ninety percent of recipes untruncated at a quarter of the memory cost of 256, and the one configuration run at 256 placed eighth without justifying its expense. That is the kind of decision worth making from the data rather than from a default.
Weight decay mattered in the expected direction and by a visible margin. The value 0.001 produced the best run; raising it to 0.02 cost roughly two points of held-out accuracy, and removing it entirely produced unstable validation curves consistent with over-fitting. Batch size settled at 64, with 128 converging faster and generalising worse, and 32 costing considerably more time for no gain. Learning rate settled at a static 1e-5; a linear scheduler with warm-up over the first ten percent of steps smoothed early training without improving the final result, and 1e-6 was too slow to converge inside the epoch budget.
An incident worth recording
The winning configuration was run a third time, with every hyperparameter identical. It trained for nine epochs rather than five and finished fifteenth of twenty-two.
Training loss and validation accuracy, by epoch
Training loss falls monotonically from 0.489 to 0.226 while validation accuracy peaks at epoch three, dips at epoch four, and recovers at epoch five without exceeding its earlier maximum. Early stopping with patience two halted training here. The divergence between the two curves after epoch three is over-fitting becoming visible in real time.
The two curves explain the incident. Validation accuracy is nearly flat across epochs three to five, varying by about four thousandths, which is well inside the noise established above. Whether the patience counter trips at epoch five or survives to epoch nine therefore depends on which side of that noise a particular epoch happens to land, and that in turn depends on data shuffling and the order of gradient updates. Early stopping was doing its job; it was simply being asked to make a decision on evidence too weak to support one, and it resolved the coin flip differently on a different run. The lesson is not that early stopping is unreliable. It is that a stopping rule inherits the precision of the quantity it monitors, and monitoring a noisy quantity produces a noisy rule.
What would change
Cross-validation, first and above everything else, both for selection and for the variance estimate it supplies. Beyond that, the natural extensions are lighter rather than heavier architectures, since DistilBERT and ALBERT are designed for precisely this situation of a modest dataset and a narrow domain, and ensembling across seeds, which would convert the run-to-run variability documented above from a nuisance into a resource. Finally, the gap between validation and held-out performance deserves direct investigation rather than inference: comparing the distributions of the two sets would establish whether the decorrelation is pure sampling noise, as argued here, or whether the held-out set differs systematically in ways that preprocessing could address.