In the spring of 2017, Aidan Gomez was twenty years old, interning at Google, and sleeping in the office.
At three in the morning, he and Ashish Vaswani were still working in a Google office. Their eight-person team had spent twelve weeks pushing toward a paper deadline, sometimes sleeping on couches behind a curtain patterned like neurons.
They weren't building a chatbot. They were trying to make machines better at translation.
Google Translate had already made a large jump the year before. Its newer neural system translated whole sentences instead of stitching together short phrases. The translations sounded more natural because the model could use more of the sentence as context.
But the architecture underneath it had a problem. It read language in a queue.
The paper that came out of those late nights was called Attention Is All You Need. It introduced the Transformer, the architecture behind GPT, BERT, and the line of work that eventually produced ChatGPT.
The strange part is that the team found it while working on translation.
Before neural machine translation, systems broke sentences into smaller pieces and tried to assemble a translation from statistical matches. Neural translation changed the job into one end-to-end model: read a sentence, build an internal representation, then generate it in another language.
One influential 2014 system used two stacked LSTMs. The first encoded the source sentence into a fixed-dimensional state. The second decoded the translation from it.
Think of it as sending a whole meeting through one summary note. One person hears everything and writes the note. A second person never hears the meeting, but has to reconstruct it in another language from that note alone.
Researchers suspected that the single note would become a bottleneck as sentences grew. The source could contain a name at the start, a pronoun near the end, and several words in between that changed what the pronoun meant.
There was another problem. RNNs read sequentially. To process word 20, the network first needed the state for word 19, which needed word 18, all the way back through the sentence.
That was a bad match for GPUs and TPUs. The hardware was built to perform many calculations together, while the model kept handing it a queue. Google's own later explanation says the sequential structure made RNNs slow and difficult to train, especially as sentences grew.
Translation quality was improving. The architecture was fighting the hardware.
The first breakthrough wasn't the Transformer. It was attention.
In a paper first posted in 2014, Bahdanau and colleagues gave the decoder a way to look back across the encoded source words while producing each translated word.
If the model was about to translate banque, it could put more weight on bank than on the rest of the sentence. For the next output word, it could look somewhere else. The model learned which part of the source mattered at that moment. This loosened the fixed-vector bottleneck, and attention became part of strong translation systems. Google's 2016 GNMT used it between a deep LSTM encoder and decoder.
Attention helped the decoder find the right information. The LSTMs around it still moved through words sequentially.
So the Google team tried the more dangerous version of the idea: remove the recurrent layers.
The Transformer replaced recurrent layers with self-attention.
Normal encoder-decoder attention lets the decoder inspect the source. In Transformer encoder self-attention, every source position can attend to every other source position.
The decoder is different. Its masked self-attention only sees earlier output positions, while encoder-decoder attention lets it inspect the encoded source.
Each token produces a query, a key, and a value. The query represents what this token is looking for. Keys represent what the other tokens offer. Their match decides how much of each value flows into the new representation.
Take this sentence:
The animal didn't cross the street because it was tired.
To represent it, the model can give more weight to animal than street. Another attention head can learn a different relationship at the same time, such as which word describes the action or which words form a phrase.
The original Transformer used multiple attention heads, feed-forward layers, residual connections, normalization, and positional encodings. So the title is catchy rather than literal.
Attention wasn't the only computation in the network. It was the mechanism that replaced recurrence and let information move directly between positions.
That direct path changed training. All positions within an encoder layer, and all masked target positions within a decoder layer, could be computed in parallel. The layers themselves still ran in sequence.
The team tested it where they had started: translation.
The training result was the real shock. The base Transformer completed 100,000 training steps in twelve hours on one machine with eight NVIDIA P100 GPUs. In that time, it reached a new state of the art for English-to-German translation.
The larger English-to-French model trained for 3.5 days on the same eight-GPU setup and set a new single-model record. Google summarized the improvement as training up to an order of magnitude faster than the recurrent and convolutional models it compared against.
The paper also estimated the raw training computation. On English-to-German, its base model used roughly 23 times less compute than the listed ConvS2S ensemble. On English-to-French, the big Transformer used roughly 52 times less. Those ratios come from the paper's FLOP estimates. Wall-clock speed still varies by model and hardware.
That difference changes what researchers can try. A model that occupies the hardware for half a day can be adjusted and trained again. A model that needs days or weeks makes every experiment expensive. The Transformer shortened that loop while producing better translations.
It also handled relationships that translation systems regularly got wrong. Google showed two sentences using the pronoun it. In one, it referred to an animal. In the other, it referred to a street. The current Google Translate model got them wrong in French, while the Transformer translated both correctly.
They had removed the queue, cut the training cost, and improved the translation at the same time.
That combination is what let the result grow beyond the problem that produced it. Parallel training later made Transformers attractive for scaling, alongside better hardware, more data, improved optimization, and distributed training.
One caveat gets lost in the usual retelling: generation is still sequential for an autoregressive model. It generates a token, appends it, then generates the next one. The Transformer removed the training bottleneck that came from recurrent state. It didn't make every stage of text generation parallel.
The original Transformer was still an encoder-decoder trained for translation. The team hadn't accidentally trained ChatGPT on a translation dataset.
What they had built was a better way for words to exchange context, and that mechanism wasn't limited to moving between English and French. Several later ideas had to stack on top before it became an assistant.
In 2018, OpenAI showed that a Transformer language model could first learn from a large body of unlabeled text, then be fine-tuned for specific tasks.
The first GPT paper reported new state-of-the-art results on 9 of the 12 datasets it studied.
GPT stands for Generative Pre-trained Transformer. It showed that a decoder-style Transformer could be generatively pre-trained on unlabeled text, then fine-tuned across different understanding tasks with relatively small task-specific changes.
Google's BERT took a different branch in the same year. It used the Transformer encoder and trained with masked-language modeling plus next-sentence prediction, so its representations could condition on context from both sides.
GPT used a causal decoder-style stack. BERT used an encoder. Neither copied the complete encoder-decoder architecture from the 2017 paper.
The architecture was becoming a reusable language engine.
Then scale changed what the interface could be. GPT-3, published in 2020, had 175 billion parameters and was tested on tasks described directly through text, often with only a few examples and no parameter updates.
The paper showed strong results across translation, question answering, cloze tasks, and several forms of on-the-fly adaptation, while also documenting tasks where it still struggled.
GPT-3 showed that many tasks could be specified in context through instructions, examples, or formatting without updating the model's weights.
A prompt was becoming a kind of temporary program. Reliable general-purpose instruction following still needed another step.
A raw language model learns to predict text. People want an assistant that follows an instruction, admits uncertainty, respects a constraint, and doesn't continue every prompt as if it were a random page from the internet.
More parameters didn't automatically fix that gap.
InstructGPT used labeler demonstrations for supervised fine-tuning, human rankings to train a reward model, and PPO for reinforcement learning from human feedback.
On the paper's held-out API prompt distribution, labelers preferred the 1.3 billion parameter InstructGPT model over the 175 billion parameter GPT-3 baseline, despite it being more than 100 times smaller.
That result explains why the history of LLMs can't be told as architecture plus scale alone. Transformer parallelism supported scaling. Generative pre-training enabled broad transfer. GPT-3 demonstrated stronger in-context learning at scale. Instruction tuning and human feedback improved behavior on the prompts OpenAI evaluated.
OpenAI released ChatGPT on November 30, 2022. It was fine-tuned for dialogue from a GPT-3.5-series model trained earlier in 2022, using an RLHF process similar to InstructGPT with dialogue-specific data collection.
OpenAI described it as a model that could answer follow-up questions, challenge incorrect premises, admit mistakes, and reject some inappropriate requests.
The chat box looks like the simplest part of the system. It may have been the part that changed the public relationship with AI the most.
You didn't need to choose a classifier, prepare a labeled dataset, or call a task-specific endpoint. You could describe the job in the same language you used to describe it to another person.
The Transformer paper solved a sequence-modeling problem. Its larger consequence was that natural language became a practical interface for computation.
Before this wave, software usually forced the user to learn its structure. You found the right screen, filled the right form, clicked the right action, or wrote code against the right API.
Now I can write, "Read these logs, find the likely failure, and explain which service I should inspect first." The model can summarize, classify, translate, generate code, call tools, or ask for missing context through one interface.
The software still has structure underneath. The APIs, permissions, data models, and failure modes haven't disappeared. I work on agents, so I see the gap every day. A model can understand the sentence and still call the wrong tool, act for the wrong tenant, or produce something that looks complete while missing half the requirements.
Attention made relationships between tokens easier to learn. It didn't give a model truth, judgment, identity, or safe execution. Those are separate engineering problems, and we're still working through them.
That takes me back to the Google office in 2017.
The eight researchers were exhausted because they were trying to beat other translation systems before a conference deadline. The paper they submitted described English-to-German and English-to-French benchmarks, not a universal assistant.
But the architecture was already pointing past translation. It had replaced the part that forced language through a queue. Words could connect directly, training could use parallel hardware properly, and the same machinery could be reused anywhere context mattered.
Then the rest of the chain happened. GPT turned the decoder side into a pre-trained language model. BERT used the encoder side to learn from both directions. GPT-3 showed that examples and instructions could live inside the prompt. InstructGPT added human preferences. ChatGPT wrapped the result in a conversation.
So I wouldn't call the Transformer an accidental chatbot. Too many separate discoveries, datasets, training runs, and engineering decisions came after it.
The discovery was more useful than the original task.
They were trying to make translation faster and better. They removed recurrence to do it, and that gave later researchers an architecture they could keep scaling and reshaping.
Five years later, the descendant of that translator was answering questions in a chat box.
Now I can write a sentence to operate software.