A conformal prediction framework for multi-label movie genre classification
A classifier that returns a single best guess is easy to evaluate and hard to trust. This project pairs a fully fine-tuned DistilBERT with conformal prediction so that the model returns a set of genres carrying a coverage guarantee, and then asks what that guarantee costs. The answer, in this instance, is a great deal: the calibrated model captures nearly every true genre and does so by predicting far too many.
A problem with two ways to be wrong
Genre assignment is multi-label by nature. A film is not a comedy or a romance; it is frequently both, occasionally neither in any useful sense, and the label sets in this dataset average a little over two and a half genres per film. The distribution across the twenty available genres is also severely unbalanced: drama appears in more than a thousand of the 2,386 films, while foreign, western, and television film appear a few dozen times or fewer. A classifier trained on this will learn the common genres well and the rare ones barely at all.
The task was scored on textual metadata only, the title, overview, and tagline concatenated into a single input string, with the overviews averaging about fifty-three words. Under those conditions a model will be genuinely uncertain about a large fraction of films, and the interesting question is not how to hide that uncertainty behind a threshold of 0.5 but how to report it.
Conformal prediction replaces the point prediction with a set, and attaches a guarantee: over repeated use, the set contains the truth at least a specified fraction of the time. The guarantee is distribution-free and holds in finite samples. What it does not promise is that the set will be small.
The mechanism
The procedure needs a non-conformity score, a number saying how surprising an observation is under the fitted model. A calibration set held out from training is scored, a quantile of those scores is taken at the desired confidence level, and that quantile is converted into a probability threshold applied to new films. Three score functions were compared, each aggregating the model's uncertainty across a film's true genres differently:
| Score | Definition | Behaviour |
|---|---|---|
| Maximum | maxy∈T (1 − py) | Governed entirely by the single true genre the model is least sure of. One difficult label makes the whole film look anomalous. |
| Sum | ∑y∈T (1 − py) | Accumulates uncertainty over all true genres, so a film with four moderately uncertain labels scores worse than one with a single uncertain label. Scales with set size. |
| Average | |T|−1 ∑y∈T (1 − py) | Between the two, removing the dependence on how many genres a film has. |
The sum-based score performed best on held-out data, which is unsurprising given that the objective rewarded capturing a film's entire label set rather than any individual label. It also has a consequence that took some experimentation to appreciate: because the score adds a term for every true genre, its scale depends on how many genres a film has, and the quantile at conventional confidence levels lands somewhere useless. At the nominal ninety percent level the procedure returns every one of the twenty genres for every film, achieving perfect coverage by refusing to exclude anything.
The remedy was to treat the confidence level as a tuning parameter rather than a specification, and to select it against the actual objective. That is a departure from textbook conformal prediction, where α is a commitment made in advance, and it should be described as what it is: a threshold chosen empirically, which yields calibrated set sizes but forfeits the clean interpretation of α as a miscoverage rate.
Calibration, and the price of coverage
What the confidence level buys, and what it costs
A full conformal calibration, run in the browser: 600 calibration films are scored, the quantile is taken at the chosen α, and the resulting threshold is applied to 600 held-out films. Beneath, one film's twenty genre probabilities are shown against that threshold. The mechanism is exactly the one used in the project; the probabilities are synthetic stand-ins, drawn so that the genre marginals match the training set and the operating point at α = 0.615 reproduces the reported coverage of 0.90, mean set size near 8.6, and micro-recall near 0.96.
Drag α down toward 0.1 under the sum score and watch the prediction set swell to all twenty genres, which is the degenerate solution version V2 found: perfect coverage, no information. Switch to the maximum score and the same α behaves quite differently, which is why the three score functions could not share a confidence level.
The search
Fifteen configurations were run, varying the score function, the confidence level, the depth of fine-tuning, and the calibration scheme. Two findings emerged clearly enough to be worth stating separately from the leaderboard arithmetic.
The first is that fine-tuning depth dominated everything else. Freezing the transformer and training only the classification head, or freezing only the lower layers, produced scores around 0.31, in the same range as the degenerate configurations. Unfreezing DistilBERT entirely moved the score to 0.78. No adjustment to dropout, learning rate, batch size, or epochs came close to that difference. The domain language of film synopses is far enough from DistilBERT's pre-training distribution that the representations themselves had to move.
The second is that class-conditional calibration underperformed the global scheme, which was not the expected result. Computing a separate quantile per genre should in principle handle the severe class imbalance better, allowing rare genres their own threshold. In practice the rare genres have too few calibration examples to estimate a quantile from, and four class-conditional variants landed between 0.686 and 0.707, below every properly fine-tuned global variant.
Fifteen configurations, by held-out score
Empirical coverage is annotated beside each bar. Note that coverage is nearly constant across the whole range, between 0.90 and 1.00, while the score varies by a factor of two and a half. Coverage was the easy part; every configuration hit its target. What separates them is the size of the sets they needed in order to hit it.
What the calibrated model actually does
The best configuration, a fully fine-tuned DistilBERT with a global sum-based score at α = 0.615, was examined beyond the leaderboard metric. The diagnosis is unambiguous and worth reporting in full, because it is the substantive result of the project.
| Metric | Value | Reading |
|---|---|---|
| Average set size | 8.62 | Nearly nine of twenty genres returned per film |
| Hamming loss | 0.3138 | About 31% of the 20 label decisions per film are wrong |
| Micro precision | 0.2849 | Roughly seven in ten predicted genres are not actually present |
| Micro recall | 0.9568 | Almost every true genre is captured |
| Micro F1 | 0.4390 | The imbalance between the two, in one number |
| Macro precision | 0.2083 | Worse when every genre counts equally, so rare genres are over-predicted most |
| Macro recall | 0.8384 | Rare genres are still mostly captured |
| Macro F1 | 0.3212 |
Recall near 0.96 against precision near 0.28 is not a model that is uncertain; it is a model that has been instructed to be inclusive and has complied. The coverage guarantee was met exactly as advertised, and the sum-based score met it in the cheapest available way, by widening the sets until the true labels could not fall out. Nothing about this is a failure of conformal prediction, which delivered precisely what it promises. It is a demonstration that a coverage guarantee is a weak constraint on its own, and that a useful prediction set requires a score function which penalises inclusion as well as omission.
A negative result on calibration
Given the over-prediction, temperature scaling was the natural next instrument: rescale the logits before the sigmoid and the probability estimates sharpen, potentially shrinking the sets at fixed coverage. Fitting the temperature on the calibration set returned an optimal value of 1.0, which is to say no rescaling at all. The model's logits were already well calibrated with respect to the binary cross-entropy objective it was trained on, and the over-prediction is therefore not a calibration failure in the probabilistic sense. It is a property of the non-conformity score. Reporting the null result matters here, because it rules out the cheap explanation and points at the right one.
Limitations, and what would come next
The model saw only title, overview, and tagline, ignoring budget, runtime, and the rest of the available metadata, any of which carries genre signal. The choice of α was empirical rather than principled, which is a real cost to the interpretation. And the central problem, prediction sets too large to act on, is unaddressed by anything tried here. The productive direction is a non-conformity score that is asymmetric in the way the application is: one that charges for including a genre as well as for missing one, so that the calibration procedure is optimising something closer to what a user of the labels would actually want. Cross-conformal or jackknife-style calibration would also use the training data more efficiently than a single held-out split, which matters at 2,386 films.