A conformal prediction framework for multi-label movie genre classification
A multi-label classifier can be accurate on average and still give no indication of when its prediction is uncertain. This project paired a fully fine-tuned DistilBERT model with conformal prediction so that each film received a set of possible genres rather than a single thresholded prediction.
The goal was to obtain a formal coverage guarantee and then examine what that guarantee cost in practice. The selected model achieved 90.2% empirical coverage, but the average prediction set contained 8.62 of the 20 possible genres. The guarantee held. The sets were often too large to be very useful.
A multi-label problem
Genre assignment is naturally multi-label. A film can be both comedy and romance, or action and science fiction. In this dataset, films carry an average of a little more than two and a half genres.
The class distribution is also highly imbalanced. Drama appears in more than a thousand of the 2,386 films, while foreign, western, and television film appear only a few dozen times or fewer. A classifier trained on these data will learn common genres much more reliably than rare ones.
The model used textual metadata only. The title, overview, and tagline were concatenated into a single input. Overviews averaged about fifty-three words.
Under those conditions, uncertainty is unavoidable. A fixed probability threshold hides that uncertainty inside a binary decision. Conformal prediction instead returns a set and aims to guarantee that the true label set is contained within it at least a specified proportion of the time.
That guarantee is distribution-free and finite-sample. It does not guarantee that the resulting set will be small.
The mechanism
Conformal prediction requires a non-conformity score that measures how unusual an observation looks under the fitted model.
A calibration set is held out from training. Its observations are scored, a quantile of those scores is selected, and the quantile is converted into a threshold for new observations.
Three score functions were compared.
| Score | Definition | Behaviour |
|---|---|---|
| Maximum | maxy∈T (1 − py) | Determined by the true genre the model is least certain about. One difficult label can make the full observation look anomalous. |
| Sum | ∑y∈T (1 − py) | Adds uncertainty across all true genres. Its scale therefore increases with the number of genres. |
| Average | |T|−1 ∑y∈T (1 − py) | Averages uncertainty across the true set and removes the direct dependence on set size. |
The sum-based score performed best on held-out data. That is consistent with the objective because success required capturing the film's entire true genre set.
It also created a practical problem. Since the score adds one term for every true genre, conventional confidence levels produced very large thresholds. At the nominal 90% level, the procedure returned all twenty genres for every film. Coverage was perfect because the model excluded nothing.
The final approach treated the confidence parameter as something to tune against the project objective. That is a departure from the clean textbook interpretation of conformal prediction. The chosen value is empirical rather than a pre-specified miscoverage rate, and the write-up should be read with that distinction in mind.
Calibration and the price of coverage
What the confidence level buys, and what it costs
The browser panel runs a full conformal calibration. Six hundred calibration films are scored, the quantile is selected at the chosen α, and the resulting threshold is applied to six hundred held-out films.
One film's twenty genre probabilities are shown below the threshold. The probabilities in the browser are synthetic stand-ins. Their genre marginals match the training data, and the operating point at α = 0.615 is constructed to reproduce the reported coverage near 0.90, mean set size near 8.6, and micro-recall near 0.96.
As α moves toward 0.1 under the sum score, the prediction set expands until it contains all twenty genres. That reproduces the degenerate solution found in an earlier version of the project.
The maximum score behaves differently at the same α. This is why the score functions cannot be compared meaningfully under one shared confidence value.
The search
Fifteen configurations were compared across four main choices.
- non-conformity score
- confidence parameter
- depth of fine-tuning
- global versus class-conditional calibration
Two findings were much larger than the rest.
First, fine-tuning depth dominated the model-level choices. Training only the classification head, or freezing the lower transformer layers, produced scores around 0.31. Fully unfreezing DistilBERT moved the score to about 0.78.
Changes to dropout, learning rate, batch size, and epoch count were much smaller. In this dataset, the pretrained representations themselves needed to adapt to the language of film descriptions.
Second, class-conditional calibration performed worse than global calibration.
A separate quantile for each genre seems attractive because the classes are so imbalanced. In practice, the rare genres did not provide enough calibration examples for stable quantile estimates. Four class-conditional variants scored between 0.686 and 0.707, below every properly fine-tuned global variant.
Fifteen configurations, by held-out score
Coverage is annotated beside each bar. It stays between about 0.90 and 1.00 across the configurations even while the overall score changes by a factor of roughly two and a half.
Coverage was the easy requirement to satisfy. The meaningful difference was how large the prediction sets had to become in order to satisfy it.
What the selected model actually does
The best configuration used a fully fine-tuned DistilBERT with a global sum-based score at α = 0.615.
Its broader metrics make the tradeoff clear.
| Metric | Value | Reading |
|---|---|---|
| Average set size | 8.62 | Nearly nine of twenty genres returned per film |
| Hamming loss | 0.3138 | About 31% of the 20 label decisions per film are wrong |
| Micro precision | 0.2849 | Roughly seven in ten predicted genres are not actually present |
| Micro recall | 0.9568 | Almost every true genre is captured |
| Micro F1 | 0.4390 | Summary of the precision-recall imbalance |
| Macro precision | 0.2083 | Lower when each genre is weighted equally, showing heavier overprediction of rare genres |
| Macro recall | 0.8384 | Most rare true genres are still captured |
| Macro F1 | 0.3212 |
Micro-recall is near 0.96 while micro-precision is near 0.28. The model captures almost every true genre, but it does so by including many genres that are not present.
The conformal guarantee is therefore working as intended. The problem is not that coverage failed. The problem is that coverage alone is too weak an objective for this application.
A useful prediction set needs to penalize unnecessary inclusion as well as missed labels.
A negative result on calibration
Temperature scaling was tested as a possible explanation for the large prediction sets.
If the model's logits were poorly calibrated, rescaling them before the sigmoid could sharpen the probabilities and perhaps reduce set size at the same coverage level.
The fitted temperature was 1.0, which means no rescaling was preferred.
The logits were already well calibrated with respect to the binary cross-entropy objective used for training. The overprediction therefore appears to come from the non-conformity score rather than from ordinary probability miscalibration.
That null result is useful because it rules out the simplest explanation.
Limitations and next steps
The classifier used only title, overview, and tagline. Other available metadata such as runtime and budget may contain additional genre signal.
The choice of α was empirical rather than fixed in advance. That weakens the usual interpretation of the conformal guarantee and should be treated as a methodological limitation.
The larger issue is the size of the prediction sets. None of the variants tested here solved it.
The most useful next step would be a non-conformity score that places an explicit cost on unnecessary inclusion as well as on omission. The calibration objective would then be closer to what a user of the genre labels actually needs.
Cross-conformal or jackknife-style calibration could also use the limited data more efficiently than a single held-out calibration split.