A borrowed spam filter fails here — and the obvious way to measure that is wrong

THE OBVIOUS MEASUREMENT SAYS THE TRANSFER FAILS COMPLETELY, AND IT IS WRONG. A filter trained on 5,574 labelled SMS messages, applied to 1,956 YouTube comments at the default 0.5 cutoff, scores 53.2% accuracy against a majority-class rate of 51.4%, catching 9.5% of the spam. That reads as total failure and it is the number you get by running the obvious code. It is measuring the wrong thing.

THE DEFAULT CUTOFF ENCODES THE SPAM RATE OF THE CORPUS THE MODEL LEFT BEHIND. SMS is 13.4% spam and these comments are 51.4% — a factor of 3.8. A model trained where spam is rare puts most of its probability mass below 0.5, so almost nothing clears the bar in a population where spam is common. Move the cutoff to the target's own rate — which needs an estimate of how much spam you have, not labels saying which comments are spam — and accuracy goes from 53.2% to 69.4%, and recall from 9.5% to 70.5%. The model never changed and neither did its predictions.

WHAT ACTUALLY TRANSFERRED IS MODEST, AND AUC IS HOW YOU SEE IT. The borrowed filter's AUC on these comments is 0.774. AUC reads the ranking alone and is unaffected by the threshold, which is exactly why it is the honest measure when a model moves between populations — accuracy at a fixed cutoff conflates how well the model separates the classes with a decision rule calibrated somewhere else.

A FILTER TRAINED ON THE TARGET'S OWN COMMENTS SCORES 0.982 AGAINST 0.774 BORROWED. Trained on four videos and tested on the fifth, held out entirely, the in-domain filter reaches 93.3% accuracy. The gap to the borrowed filter is the cost of using someone else's corpus, and it is large.

ABOUT 25 OF YOUR OWN LABELS BEAT ALL 5,574 BORROWED ONES. Training on a random sample of in-domain comments and testing on a held-out video, accuracy passes the best the borrowed corpus achieves at roughly 25 labels — about one 223th of the borrowed set. For a team deciding whether to label their own data, that is the number that matters, and an afternoon covers it.

SHUFFLING THE SPLIT WOULD HAVE ADDED 2.0 POINTS FOR NOTHING. Pooled five-fold cross-validation over all the comments scores 95.3% against 93.3% leave-one-video-out. Comments under one video share vocabulary, campaigns and posting bots, so a shuffled split trains on the same video it tests on and measures memorisation as much as detection. The gap is small here and rests on five videos, but its direction is the one that flatters.

THE RANKING IS STABLE ACROSS VIDEOS AND THE DECISION IS NOT. Held-out AUC varies only from 0.983 to 0.990 across the five videos, while accuracy at the same fixed cutoff spans 89.4% to 95.9% — a range of 6.5 points. It is the threshold lesson again at a smaller scale: what a model knows travels between populations more reliably than where you should cut it, and a single accuracy figure conflates the two.