SEEPLOREAsk ChatGPT a question and a long answer pours out in seconds. But the AI isn't thinking the whole sentence through before it writes. It just repeats one tiny job very fast: guess the next character. By the end of this post you'll know what that means — and why AI sometimes says things that sound right but aren't.
A fill-in-the-blank game
"Once upon a time, ___." What goes in the blank? You probably think of "there was a" or "in a small village" first. Something like "refrigerator" hardly comes to mind. That means your brain is already weighing how likely each word is.
AI does the same thing. The difference is that it doesn't pick just one answer — it gives every candidate a score in percent. I built a small AI that writes Korean fairy tales, one character at a time. When I give it the Korean opening for "once upon a time" and ask for the next character, it answers something like: this one 20%, that one 12%, another 10%… All the percentages add up to 100%. Think of it like a weather forecast's "30% chance of rain." Spaces and periods count as characters too.
How did the AI learn those percentages?
A brand-new AI knows nothing. It gives every character about the same score, so if you ask it to write, you get random gibberish.
So we make it practice on text people wrote. One sentence like "Once upon a time, an honest woodcutter lived in a mountain village" already turns into lots of little quiz questions: given everything up to here, what comes next? The answer is written right there in the sentence, so no teacher is needed to mark it.
Before training, the AI gave the right character in one of these questions a score of 0.0% — it ranked 835th among the candidates. So we nudged the roughly 110,000 numbers inside the AI a tiny bit toward the right answer. After 8 nudges on that one question, the score had climbed to 51%. The real practice bundled 64 questions at a time and repeated the nudging 2,000 times.
The fun part: nobody taught it grammar. It only practiced guessing, yet it started putting spaces in the right places and ending sentences the way Korean stories do.
Why not always pick the top character?
Once every candidate has a score, the AI has to choose one. Picking the winner every time sounds sensible, but in practice the text starts repeating itself. It keeps getting pulled toward the most common phrases.
So the AI picks like drawing lots. A character at 30% gets chosen about three times in ten, and even low-scoring characters win now and then. The seeplore title The AI That Writes One Letter at a Time shows this as a slot machine: the bigger a character's percentage, the wider its slot on the reel, so when you pull the lever the reel usually stops on a wide one.
There's one more setting, called temperature. Turn it down and the top character wins more often — safe but predictable writing. Turn it up and surprises show up more often — fresh, but sometimes nonsense.
The chosen character is added to the end, and the AI looks at the now slightly longer text and guesses again. It's a bit like playing a word-chain game with yourself. That's also why ChatGPT gives slightly different answers to the same question: if one character in the middle comes out differently, everything after it changes.
Looking back for clues
Some characters almost always follow certain others, so the AI has to look at what came before. It's like answering a test question by hunting for clues in the earlier sentences.
In the picture below, lines run from the blank (the yellow square) back to earlier characters. The brighter the line, the more the AI used that character as a clue. This small AI can look back up to 64 characters. The part that gathers these clues is called "attention," and big AIs like ChatGPT have it too.
Why AI makes up convincing things
Now the weak spot is easy to see. The AI only picks a likely next character. It never checks whether what it says is true.
My AI practiced on 40 fairy tales, and the stories it wrote don't appear anywhere in those 40. The longest stretch that exactly matched an original was only 7 characters. Sometimes characters from different stories showed up in the same tale. Believable, but made up.
When ChatGPT confidently states something false, it's for the same reason. This is called a "hallucination," and turning the temperature down doesn't make it disappear. So for homework or anything important, always double-check with a reliable source.
A small AI vs. ChatGPT
My AI has about 110,000 of these scoring numbers and has read about 85,000 characters. According to the GPT-3 paper from 2020, GPT-3 had about 175 billion of them and read about 300 billion tokens (small chunks of text). The sizes aren't even close, but the job is the same: look back, give percentages, pick one, add it on.
Most AIs released since then haven't published these numbers, so we don't know exactly how big they are. The idea, though, is exactly what you've just seen.
Once you know how it works, you'll want to try it. In the "try it yourself" part of the title, change the opening and the temperature and pull the lever. And if you'd like to see more of how an AI learns, AI by Signals shows a small AI guessing handwritten digits, getting them wrong and fixing itself.