TIS-100 Bench: LLMs Push the Pareto Frontier in Programming Puzzle Games
Suite. All 22 campaign puzzles, all 25 TIS-NET puzzles, and eight tournament puzzles from 2018: 19 easy (10 minutes), 17 medium (30), and 19 hard (45). Every model has one scored episode per puzzle. Time limits differ by difficulty, so difficulty and search time can’t be separated.
Scoring. Each model–puzzle pair contributes one selected attempt, with no best-of-attempts scoring, so solve rates are pass@1: a puzzle counts as solved when that one episode saved a program that replays in a fresh game. Unsolved puzzles stay in every average and score zero.
New frontier points. The reference is the community leaderboard as of October 15, 2026. Ties don’t count. Models found 154 candidate points on 23 puzzles. Our count pools and deduplicates them across models and keeps the 133 on 18 puzzles with fresh-game replay and 100,000 passing tests. Testing is strong evidence of correctness, not proof.
Record proximity. The community record divided by the model’s best score, averaged across the three scores and all puzzles, ×100. Each score can come from a different program.
Limits. One episode per pair compares the configurations as tested and doesn’t measure repeated-run variability. Spread between puzzles is variation across tasks. Public puzzles may overlap with training data. Mechanic groups are our own labels.
Programs. Every model Pareto point’s program is public under the MIT License, in TIS-100 save format and named like the community leaderboard’s files (
Resources. Costs are estimated from recorded October 2026 API list prices, including cache discounts. Tokens accumulate across responses, so repeated input counts again, and tokenizers differ. Game calls count individual dispatched tools, including errors, and exclude the automatic final submission.
