← All journal entries

When the answer key was wrong - a lesson from Rankle

Rankle is a simple idea. Four items, one scale, put them in order from lowest to highest. Planets by size, animals by top speed, drinks by sugar. You tap to swap, you submit, and only then do you see the real numbers.

That last part matters. The numbers are hidden while you play, because ranking things is trivial if you are shown the answer. So the game asks you to trust it. And for a while, it did not deserve that trust.

One in six

A proper audit of the puzzle pool turned up something embarrassing. Eleven of the first sixty-seven puzzles had their answers stored in the wrong order. The game checks your answer against the order the items are saved in, not against the numbers shown afterwards. So a player could reason correctly, get marked wrong, and then be shown the very numbers that proved they were right.

The worst one asked for the rocky planets from smallest to largest. The stored answer was Mercury, Venus, Earth, Mars. The real order is Mercury, Mars, Venus, Earth. A couple of others had ties in them, which made them unanswerable as well as wrong. A few had numbers that were simply off, like a comic book character given the wrong debut year.

That is the worst way a puzzle game can fail. A hard puzzle is fine. An unfair one makes people stop coming back, and they never tell you why.

The fix was boring on purpose

Every broken puzzle got repaired or replaced. A few were beyond saving, where the premise itself was a coin flip, and those went in the bin.

The real fix was a small checker that runs before every build. It reads every puzzle, looks at the four values, and refuses to build the site if they are not in order, if any two are tied or too close to call, if a label gives the answer away, or if the same item turns up twice. It is a few dozen lines. It has no opinions. It just counts.

Before trusting it, I put one of the old mistakes back in on purpose to make sure it actually caught it. It did.

Never let a language model do the ordering

The bigger lesson is about which jobs to hand to AI. Language models are good at suggesting interesting topics and ballpark numbers. They are not reliable at the mechanical part, which is putting four numbers in order or checking that a word really contains a set of letters. The same thing turned up on Cluster, where a model happily claimed that a word contained letters it did not.

So the rule now is simple. Ideas can come from anywhere. Anything that can be checked by counting gets checked by code, every time, before a player ever sees it.

- Chris