RandomBench

How well can language models pick numbers at random?

Each model was asked many times for a random integer from 0 to 99, in a fresh conversation every time. A model that gives the same answer every time scores 0.01, and a perfect random number generator scores about 1.

Method and limitations Code and data on GitHub Official leaderboard on Kaggle

Models
39
from 4 providers
Answers
34,231
each in a fresh conversation
Most random
0.146
GPT-5.4. A perfect source scores about 1.
Most common answer
42
28.3% of the average model's answers

What the models answer

Share of answers at each value, averaged so every model counts the same

The tallest bars are 42 (28.3%), 47 (28.0%), 73 (20.6%), 37 (5.1%), and 57 (3.6%).

Ranking

Rank# Model Replies Score Answers, 0 to 99
1GPT-5.4 OpenAI8230.146 ±0.016
2–3Gemini 2.5 Flash Google6210.108 ±0.011
2–3Gemini 4 Neon Google9350.105 ±0.008
4Grok 4.6 xAI8900.082 ±0.010
5–6GPT-6.2 Astra OpenAI8950.063 ±0.004
5–6GPT-6.2 Sol OpenAI8360.059 ±0.005
7–11GPT-6 Astra OpenAI9910.045 ±0.003
7–11GPT-6.1 Sol OpenAI8860.042 ±0.003
7–12Claude Opus 5.5 Anthropic8660.041 ±0.003
7–12GPT-5.6 Terra OpenAI8670.041 ±0.002
7–17Gemini 2.5 Pro Google4200.037 ±0.005
11–16GPT-5.4 mini OpenAI8390.036 ±0.001
9–17Gemini 3.1 Pro (preview) Google10450.036 ±0.003
11–17Grok 4.7 xAI9070.035 ±0.002
11–17Gemini 3.8 Flash Google6240.034 ±0.002
12–17gpt-oss-120b OpenAI9350.033 ±0.001
11–17GPT-5.4 nano OpenAI8970.032 ±0.003
18–20Gemini 3.6 Flash Google8870.027 ±0.001
18–23GPT-5.6 Luna OpenAI8930.026 ±0.002
18–24GPT-5.5 OpenAI10140.025 ±0.002
19–25Claude Opus 4.5 Anthropic8790.024 ±0.001
19–26Claude Haiku 5.5 Anthropic9490.022 ±0.002
19–26Gemini 3 Flash (preview) Google9460.022 ±0.001
20–27Gemini 4 Argon Google9580.022 ±0.001
22–28Gemma 4 26B A4B Google8990.021 ±0.001
21–28Claude Opus 4.7 Anthropic9080.021 ±0.001
24–30Gemma 4 31B Google8710.020 ±0.001
25–31Claude Opus 5 Anthropic9370.019 ±0.001
27–31GPT-6 Luna OpenAI10450.018 ±0.001
27–31Claude Opus 4.8 Anthropic8660.018 ±0.002
28–31GPT-6 Sol OpenAI9330.017 ±0.001
32Claude Sonnet 4.6 Anthropic8210.015 ±0.001
33–35Claude Sonnet 5 Anthropic8590.013 ±0.001
33–35Claude Sonnet 5.5 Anthropic9510.012 ±0.000
33–35Claude Haiku 4.5 Anthropic8320.012 ±0.000
36–37GPT-5.6 Sol OpenAI9870.011 ±0.000
36–37Gemini 3.5 Flash-Lite Google8420.011 ±0.000
38–39Claude Opus 4.6 Anthropic10430.010 ±0.000
38–39Gemini 3.1 Flash-Lite (preview) Google8180.010 ±0.000

Answers draws how often the model gave each value, with its favorite in orange. The bars follow the square root of the counts, so a value given a quarter as often as the favorite is half as tall, and one given only once still shows.

Replies counts every call that got a reply, valid or not. Calls that failed, such as those the quota refused, aren't counted. Every score's 95% interval comes from 10,000 bootstrap resamples of the model's replies, so a model with fewer replies gets a wider interval, not a lower score.

A rank is a range where scores are too close to call. A model ranks below every model whose interval lies wholly above its own, and above every model whose interval lies wholly below it. Where two intervals overlap, either model could be ahead. Each range compares the models two at a time, so with this many models, an end of a range may now and then be off by one.

The bias checks on a share of the answers, such as Multiples of 10, are lenient toward avoidance. A model that never answers a multiple of 10 is only 10% away from the ideal share, so it scores 0.900, and one that answers nothing else scores 0.100.

What stands out

No model comes close. Even the most random model scores less than 0.2, and the top of its interval is less than 0.2 too. A perfect random number generator scores about 1.

Three numbers dominate. All but five models favor 42, 47, or 73, and those three numbers make up more than 70% of all the answers. Gemini 3.1 Flash-Lite (preview) and Claude Opus 4.6 have answered nothing but 42, and five more models give their favorite more than 85% of the time: Gemini 3.5 Flash-Lite, GPT-5.6 Sol, Claude Haiku 4.5, and Claude Sonnet 5 and 5.5. Two of the exceptions barely qualify: GPT-5.4 favors 57 over 47, and gpt-oss-120b favors 57 over 42, each by less than 2% of its answers. The other three are the newest models here, and they favor the 60s. GPT-6.2 Astra and GPT-6.2 Sol both favor 63, and Gemini 4 Neon favors 68. Each answers from 60 to 69 more than 35% of the time, where no older model does even 20%. All three rank among the six most random models.

Whole regions go unused. More than 95% of all answers lie between 20 and 79, and more than 90% end in 2, 3, or 7. No model has ever answered 0 or 99. Gemini 4 Neon strays furthest: fewer than a third of its answers end in 2, 3, or 7, the only model under half, and more end in 8 than in any other digit. Only GPT-5.4, Gemini 2.5 Flash, and Grok 4.6, three of the four most random models, have answered above 89, and only GPT-5.4 and Grok 4.6 below 10. Only GPT-5.4 and Grok 4.6 have answered with a repeated digit, like 77, more than once, and Gemini 2.5 Flash and gpt-oss-120b have each given a single 77. Only GPT-5.4, GPT-5.5, and GPT-5.6 Terra have answered with a multiple of 10. Every other model has avoided all of these, and multiples of 5 too, so they all get the same score on Multiples of 10, Multiples of 5, Edges, and Repeated Digits. Here the identical scores are the finding.

Newer isn't more random. GPT-5.4 ranks first, ahead of every later GPT, from GPT-5.5 to GPT-6.2. Gemini 2.5 Flash, one of the oldest models here, has the second-highest score, ahead of every newer Gemini. Grok 4.7 is less random than Grok 4.6, GPT-6 Luna than GPT-5.6 Luna, and Claude Sonnet 5 and 5.5 than Sonnet 4.6. Claude Opus 5.5 is the most random Claude, ahead of the newer Sonnet 5.5 and Haiku 5.5. Some go the other way: each Sol model is more random than the one before, from GPT-5.6 Sol to GPT-6.2 Sol. Gemini 4 Neon is more random than Gemini 4 Argon, and Claude Haiku 5.5 than Haiku 4.5. None of these comparisons has overlapping intervals, except Gemini 4 Neon's with Gemini 2.5 Flash, whose scores are too close to tell apart.

Reasoning doesn't fix it. Eleven models reason at length by default, spending a median of more than 50 output tokens on a two-digit answer, and Gemini 2.5 Pro, Gemini 3.1 Pro (preview), and Grok 4.6 more than 200. None of the eleven scores above 0.12 on Uniformity, where a fair pick would score close to 1. Among Google's models, those that spend more tokens tend to score higher, but among OpenAI's it goes the other way: its three most random models, GPT-5.4, GPT-6.2 Astra, and GPT-6.2 Sol, spend 4 tokens on every reply. And it isn't a clean comparison, since reasoning goes with provider: all eleven are from Google or xAI, or are gpt-oss-120b.

The wording matters. Each model was asked in six wordings, and its score pools them, so a model can score higher by switching favorites between wordings than by straying from one within them. Claude Sonnet 4.6 has given only two numbers: 47 every time in four of the wordings, 42 every time in the one that asks for an integer chosen uniformly at random, so that every value is equally likely, and a mix of the two in the sixth. Yet it scores clearly above Claude Sonnet 5, Sonnet 5.5, and Haiku 4.5, which keep one favorite in every wording but stray from it more often. The uniform wording helps GPT-5.4 most: asked that way, it scores more than 0.4 on Uniformity, more than three times its score on the other five wordings together, and it scores higher that way in every one of its runs. No other model's score even triples, and the uniform wording raises Uniformity for only about half of the models.

Nearly every reply is valid. The one big exception is Claude Opus 4.8 when it's asked to draw a raffle winner. It declines nearly every time, saying it can't actually draw a random number, and suggests a better way, such as random.org, a spreadsheet's RANDBETWEEN, or Python. Some of those replies warn that its own pick would favor numbers like 37, 42, or 73, and a few offer a casual pick anyway. Its other replies are all valid. Grok 4.6 now and then adds a confidence score after its answer, as in 47\n\confidence{80}, and the second number makes the reply invalid. Claude Opus 5.5 has slipped ten times. Four of its replies were blank. Five write out its thinking, four of them before the number and one with no number at all. And one names the range along with its pick: I chose 47 at random from 0–99, inclusive. gpt-oss-120b has slipped twice, once with an empty reply and once with a refusal: I’m sorry, but I can’t comply with that. Every other model's replies have all been valid.

Answers can shift between days. The intervals on the page assume a model answers the same way each time it's asked, but Claude Opus 4.5 and Claude Haiku 4.5 scored clearly differently on November 19 to 21 than on November 15 and 16. Opus 4.5 gave 47 in more than 60% of its answers on November 15, but in fewer than 55% on November 19 and 20, and its score rose. Haiku 4.5 gave 42 in fewer than 90% of its answers on November 15, but in more than 95% on November 20, and its score fell. Each shift goes the same way in at least five of the six wordings. Even so, the models drawn on both sets of days rank in nearly the same order on each.