JevMan

playopper.ai

The curator's review

Take the models out and there is no game being played: nobody steers Pac-Man here but the models, answering at every junction with odds for each direction inside a two-second clock. That is what makes it a benchmark and a spectacle at once. The leaderboard is oddly riveting. At the top two entries are statistically tied on score yet miles apart on cost per game, and one model survives half as long as the rest for a third of the price. You can watch any model's run frame by frame, or step into the maze and play against one yourself.

Six AI models played 100 games each of classic Pac-Man against the arcade ghosts. A real leaderboard by mean score, survival time, latency and cost per game, plus a watch mode for any model's run and a mode to play against one yourself.

Open on opper.ai

The full review

JevMan is a Pac-Man benchmark for decision models, and the rules are what make it worth a look. Each model plays 100 real-time games against the classic arcade ghosts, with three lives or five minutes, and at every junction the game sends the maze as JSON and asks for a direction. The model has two seconds to answer. Only decision models are ranked, meaning ones that reply with odds for every option in a single pass without writing any text. Chat models and plain rule-based baselines are shown at their score for reference, marked Chat or Base, rather than given a rank.

The leaderboard reports mean score, survival time, latency and cost per game, and it is honest about ties: a score marked =1 means two entries sit within each other's margin of error. That honesty is where the interesting reading is. Six named models are listed, and the top two cannot be told apart on score but are very far apart on what a game costs. One entry survives about half as long as the others for roughly a third of the price. A rival on the same page was too slow to rank at all: Claude Haiku 5.5 was run by the site's own team, and two thirds of its answers arrived too late, so backup moves played instead.

Two things on the page are for you rather than the models. Watch mode replays any model's run frame by frame, and a play-versus mode lets you steer Pac-Man yourself against one of them. The benchmark is open source, and any model behind an HTTP endpoint can enter: the repo has a 34-line example endpoint, one command runs the games under the leaderboard's rules, and a pull request adds the result, with CI replaying every submitted game to check its score. Community entries are labelled with who ran them, and their latency and cost are as reported.

It is a benchmark pretending to be an arcade, or the other way round, and it works as both. The thing to look at is the cost column next to the score column.