The badge we pulled the same evening

At ReelUse, the AI picks the best moments from a long recording and turns them into clips. We wanted to help customers start with the strongest one, so we shipped a Top Pick badge on the clip the system scored highest.

We withdrew it that same evening.

What the score was really measuring

Once it was live, we looked at what the badge was doing. The score behind it mostly ranked clips against each other. It didn't measure whether a clip was actually good. And over a quarter of videos got no Top Pick at all, for reasons that had nothing to do with quality.

So the badge was saying "this is the best one" when what it meant was "this scored higher than its neighbours". Sometimes it said nothing at all. Neither is a promise I want a customer to rely on.

Why we didn't leave it up

A label like that is a claim. Customers would post the Top Pick first and trust it more than the others. If the claim isn't reliably true, it costs more trust than it earns. Pulling it fast was cheaper than explaining it later.

How we judge quality instead

This is why clip quality at ReelUse is judged by people, not by the model's own score. We run blind reviews: real clips cut from the same recordings, rated Good or Weak by reviewers who don't know which model picked them. The material is our own recordings plus one customer who opted in. Customer content is never used without consent.

It also taught us to repeat our tests. Run the same recording twice and only about half the clips come back the same. One comparison is noise. Decisions need repeated human review.

What I'd tell another AI PM

Before you put a model's score in front of a customer, ask what it actually measures. If you can't say "this is better" and back it with human judgement, don't let the interface say it either.