If you're a Product Manager working on AI products, you don't need to become an ML researcher.
But you do need to understand what happens inside an LLM.
Because sooner or later, you'll have to answer questions like:
You don't need to understand every mathematical detail behind a Transformer.
You need a good mental model.
That's what this article is about.
What exactly is an LLM?
LLM stands for Large Language Model.
At a high level, an LLM is a machine-learning model trained on large amounts of data to learn patterns in language and generate outputs based on its input.
The simplest mental model is:
«An LLM takes tokens as input and predicts what token should come next.»
For example:
The capital of France is
The model might assign probabilities to possible next tokens:
Paris 92%
London 2%
Berlin 1%
Madrid 1%
...
It selects a token, adds it to the sequence, and predicts the next token.
This process continues until the model generates the response.
So when an LLM writes a paragraph, it isn't necessarily creating the entire paragraph in one shot.
It's generating a sequence of tokens.
That simple idea explains a surprisingly large part of how LLMs work.
The Big Picture
Before diving into the details, here's the entire process:
USER
|
v
"Explain RAG"
|
v
Tokenization
|
v
Tokens
|
v
Embeddings
|
v
+----------------+
| Transformer |
| |
| Self-Attention |
| | |
| MLP |
| | |
| Many Layers |
+-------+--------+
|
v
Logits
|
v
Probability
Distribution
|
v
Token Selection
|
v
Next Token
|
+------+
|
v
Repeat Generation
|
v
RESPONSE
Now let's break this down.
When you type:
Product management is interesting.
the model doesn't directly receive those words as normal human-readable text.
The text is first converted into tokens.
A tokenizer might represent it conceptually as:
["Product", " management", " is", " interesting", "."]
But tokens aren't necessarily complete words.
A word can be split into multiple tokens.
internationalization
could be represented as several smaller pieces.
The exact result depends on the tokenizer and model.
This is why:
«A token is not necessarily equal to a word.»
And this matters a lot in real products.
Why should a Product Manager care about tokens?
Because tokens affect:
Imagine your application handles:
10,000 users
×
2,000 input tokens
×
10 requests per day
That's:
200,000,000 input tokens/day
Suddenly, tokenization isn't just an ML concept.
It's a product economics problem.
Neural networks work with numbers.
So the tokens need to be converted into numerical representations.
This is where embeddings come in.
Conceptually:
"product"
|
v
[0.21, -0.73, 0.44, 0.18, ...]
The actual vectors are much larger than this example.
You can think of an embedding as a numerical representation that allows the model to work with relationships between pieces of information.
For example, concepts that occur in similar contexts can have useful relationships in the model's representation space.
vector space
apple
*
/
/
* fruit
car
\
*
vehicle
This is only an intuition.
Real embedding spaces are high-dimensional and considerably more complicated.
The important idea is:
«Embeddings convert discrete information into numerical representations that neural networks can process.»
Now we reach the most important part.
Modern LLMs are largely built around the Transformer architecture.
The Transformer architecture was introduced in the 2017 research paper:
"Attention Is All You Need" (https://arxiv.org/abs/1706.03762).
The paper introduced an architecture based heavily on attention mechanisms rather than the recurrent architectures commonly used in earlier sequence models.
Today, Transformer-based architectures are fundamental to modern generative AI.
But remember:
«Transformer ≠ LLM»
A Transformer is an architecture.
An LLM is a language model that can be built using a Transformer-based architecture.
A simplified Transformer block looks something like this:
Input
|
v
+------------------+
| Self-Attention |
+------------------+
|
v
Residual Connection
|
v
Normalization
|
v
+------------------+
| Feed Forward |
| Network (MLP) |
+------------------+
|
v
Residual Connection
|
v
Normalization
|
v
Output
Two components are especially important:
Let's start with attention.
Consider this sentence:
«"The developer put the laptop on the table because it was broken."»
What does "it" refer to?
A language model needs to understand relationships between different parts of the sequence.
Self-attention allows the model to determine which tokens are relevant to one another.
Instead of processing every token completely independently, the model can calculate relationships between tokens.
A simplified mental model is:
^
|
What does "it"
refer to?
|
+------------------+----------------+
| |
laptop table
The model uses attention mechanisms to build contextual representations.
You'll frequently hear:
or:
Q = Query
K = Key
V = Value
The simplified attention equation is:
softmax(QKᵀ / √dₖ)V
As a Product Manager, you don't need to derive this equation.
The intuition is more useful:
Query
|
+----> Compare with Keys
|
v
Attention Scores
|
v
Weighted Values
|
v
New Representation
You can think of it as the model asking:
«"Which other pieces of the context are relevant to this token?"»
Imagine you're building an AI customer-support assistant.
A customer says:
«"I bought the phone two weeks ago. The battery is already failing. Can I get a replacement?"»
The model needs to connect several pieces of information:
phone
|
+---- purchased two weeks ago
|
+---- battery failing
|
+---- asking about replacement
Attention helps the model build contextual relationships between these tokens.
This is one of the reasons Transformer-based models are so powerful for language tasks.
Transformers generally don't rely on a single attention mechanism.
They use multiple attention heads.
Input
|
+-----------+-----------+
| | |
v v v
Head 1 Head 2 Head 3
| | |
v v v
Pattern A Pattern B Pattern C
| | |
+-----------+-----------+
|
v
Combined
|
v
Output
Different heads can learn different relationships during training.
We shouldn't think of them as manually assigned roles.
The model learns useful representations from the training process.
Consider:
Dog bites man.
and:
Man bites dog.
Same words.
Very different meaning.
So the model needs information about the position/order of tokens.
Transformer architectures therefore use mechanisms for representing positional information.
You may encounter terms such as:
The exact technique depends on the model architecture.
The key idea is simple:
«The model needs to know where tokens occur in the sequence.»
After attention, Transformer blocks also contain feed-forward neural networks, often called MLPs.
A simplified view:
Token Representation
|
v
Linear Layer
|
v
Activation
|
v
Linear Layer
|
v
Output
A useful mental model is:
«Attention allows tokens to exchange contextual information, while the feed-forward network performs additional nonlinear transformations on those representations.»
These operations are repeated across many layers.
One Transformer block isn't the whole model.
LLMs contain many layers.
Input Embeddings
|
v
+----------------+
| Transformer 1 |
+----------------+
|
v
+----------------+
| Transformer 2 |
+----------------+
|
v
+----------------+
| Transformer 3 |
+----------------+
|
v
...
|
v
+----------------+
| Transformer N |
+----------------+
|
v
Final Representation
Each layer transforms the representation further.
This repeated computation is one reason large language models require significant computational resources.
You've probably heard statements like:
«"This is a 7B model."»
«"This model has 70B parameters."»
The "B" means billion.
Parameters are learned numerical values inside the model.
Very roughly:
Model
|
+-- Weights
|
+-- Biases
|
+-- Other learned parameters
During training, these parameters are adjusted so the model becomes better at its objective.
Why does model size matter?
Larger models generally require more resources.
That can affect:
But:
«Bigger does not automatically mean better for your product.»
A smaller model might be preferable when your application needs:
This is an important AI PM trade-off.
Now we get to training.
A simplified training pipeline looks like:
Large Dataset
|
v
Data Processing
|
v
Tokenization
|
v
Training Examples
|
v
Transformer Model
|
v
Prediction
|
v
Calculate Loss
|
v
Backpropagation
|
v
Update Parameters
|
+----------------+
|
v
Repeat
This process happens an enormous number of times.
One of the fundamental training objectives for autoregressive language models is next-token prediction.
The product manager wrote a
The model tries to predict the next token.
Maybe:
PRD 0.50
document 0.20
strategy 0.10
...
The actual training data tells the model what the target token should be.
The model's prediction is compared with the target.
The resulting error contributes to the training loss.
The parameters are then adjusted.
This happens repeatedly across massive amounts of training data.
Loss is a numerical measure of how far the model's prediction was from the desired target.
Simplified:
Prediction
|
v
Compare with target
|
v
Loss
|
v
Calculate gradients
|
v
Update parameters
For language models, cross-entropy loss is commonly used for next-token prediction.
You don't need to memorize the mathematical derivation to understand the product implications.
«Training uses errors to update the model's parameters.»
Backpropagation calculates how the model's parameters contributed to the error.
A simplified mental model:
Prediction
|
v
Error
|
v
Gradients
|
v
Parameter Updates
|
v
Better Future Predictions
Modern model training also uses optimization algorithms to determine how those parameters should be updated.
Again, the important thing for a PM is understanding the role of the process rather than memorizing every equation.
A pretrained model isn't automatically a great conversational assistant.
Modern AI systems can involve additional stages such as:
Large-Scale Training
|
v
Pretraining
|
v
Base Model
|
v
Instruction Tuning
|
v
Alignment / Preference Optimization
|
v
Safety & Evaluation
|
v
Useful Model
The exact pipeline differs between model providers and model families.
Additional techniques may include:
This distinction is extremely important.
Training
Training means:
«Adjusting the model's parameters using data.»
Data
↓
Prediction
↓
Loss
↓
Backpropagation
↓
Parameter Update
Inference
Inference means:
«Using the trained model to generate an output.»
Prompt
↓
Model
↓
Prediction
↓
Output
Think of it simply as:
TRAINING
"Learn"
↓
INFERENCE
"Use what you learned"
For most AI Product Managers, inference will be much more relevant to day-to-day product decisions than training a foundation model from scratch.
Let's say you ask:
«"Explain product-market fit in simple terms."»
A simplified inference flow is:
User
|
v
Application
|
+-- System Instructions
+-- Conversation History
+-- User Prompt
|
v
Tokenizer
|
v
Tokens
|
v
Embeddings
|
v
Transformer Layers
|
v
Logits
|
v
Probability Distribution
|
v
Token Selection
|
v
Next Token
|
+------> Repeat
|
v
Final Response
This is the basic lifecycle of an LLM request.
At the end of the model's computation, it produces scores called logits for possible next tokens.
Imagine:
Token Score
Paris 8.2
London 3.1
Berlin 2.7
Madrid 2.2
...
These raw scores can be transformed into probabilities using softmax.
Logits
|
v
Softmax
|
v
Probabilities
|
v
Token Selection
The model then chooses a token according to the decoding strategy.
Temperature is one of the generation settings you'll encounter when working with LLM APIs.
Generally:
Lower temperature
A → 90%
B → 5%
C → 2%
D → 1%
versus:
Higher temperature
A → 45%
B → 25%
C → 20%
D → 10%
These numbers are illustrative, not actual model behavior.
Product implication
For:
Invoice extraction
you probably want controlled and consistent outputs.
Creative writing
you may want more variation.
So generation parameters can directly affect the product experience.
Suppose the model generates:
The product is successful because...
The
↓
The product
↓
The product is
↓
The product is successful
↓
The product is successful because
↓
...
Each generated token becomes part of the context for the next prediction.
This is also why LLM applications can stream responses.
Instead of waiting for:
[4 seconds]
↓
Entire response
the application can display:
The...
The product...
The product is...
The product is successful...
Streaming can make an application feel much faster, even if total generation time doesn't change.
That's a UX decision, not merely an engineering optimization.
You've probably seen:
«"This model supports a 128K context window."»
The context window represents how much context the model can process within a particular interaction, according to that model's limits.
That context may include:
System instructions
Conversation history
User prompt
Retrieved documents
Tool results
Generated output
+--------------------------------------+
| CONTEXT WINDOW |
| |
| System Instructions |
| Conversation History |
| User Input |
| Retrieved Information |
| Tool Results |
| Model Output |
| |
+--------------------------------------+
Why should a PM care about context windows?
Because context affects:
And here's an important distinction:
«Being able to fit information into the context window doesn't mean the model will use all of it effectively.»
More context can also mean:
So:
«"Can we fit the document?"»
and
«"Can the model effectively use the document?"»
are two different questions.
During autoregressive generation, the model repeatedly needs information from previous tokens.
Recomputing everything from scratch would be inefficient.
Inference systems therefore commonly use a Key-Value cache, or KV cache, to reuse attention-related information from previously processed tokens.
Previous Tokens
|
v
Key / Value Computation
|
v
KV Cache
|
v
New Token
|
v
Reuse Cached Information
KV caching matters for:
For a technical PM working on AI infrastructure, this is an especially useful concept to understand.
Here's a common misconception:
«"The LLM searches the internet every time I ask a question."»
A basic LLM doesn't necessarily do that.
Its parameters contain patterns learned during training.
That learned information isn't equivalent to a traditional database.
This distinction becomes very important when building enterprise AI applications.
Imagine your company has:
Product Documentation
Pricing
Employee Handbook
Customer Policies
Internal Wiki
Support Articles
You want your AI assistant to answer questions about them.
Simply having trained the foundation model on general internet data doesn't mean it knows your company's latest internal information.
This is where RAG becomes useful.
RAG stands for:
Retrieval-Augmented Generation.
The basic idea:
«Retrieve relevant information and give it to the LLM as context before generating the answer.»
A simplified architecture:
User Question
|
v
Query Processing
|
v
Retrieval Layer
|
+--------+--------+
| |
v v
Vector Search Keyword Search
| |
+--------+--------+
|
v
Relevant Docs
|
v
Context Builder
|
v
LLM
|
v
Answer
Suppose a customer asks:
«"What's our current refund policy?"»
Your base model may not know your company's latest policy.
Instead:
Question
|
v
Search company knowledge
|
v
Retrieve relevant policy
|
v
Add policy to prompt
|
v
LLM
|
v
Answer
The model doesn't permanently learn the document.
The application supplies the information at inference time.
Production RAG systems can be more sophisticated:
User
|
v
+---------------+
| Query Process |
+-------+-------+
|
v
+---------------+
| Retrieval |
+-------+-------+
|
+------------+------------+
| |
v v
Vector Search Keyword Search
| |
+------------+------------+
|
v
+---------------+
| Reranker |
+-------+-------+
|
v
Relevant Context
|
v
+---------------+
| LLM |
+-------+-------+
|
v
Answer
This creates several PM questions:
These aren't purely engineering questions.
They're product decisions.
This is one of the most common questions in AI product development.
RAG
Provide external information at runtime.
Question
↓
Retrieve Information
↓
LLM
↓
Answer
Fine-tuning
Further train the model to specialize its behavior.
Base Model
↓
Specialized Dataset
↓
Fine-Tuning
↓
Specialized Model
A simplified rule of thumb:
Requirement| Often worth considering
Frequently changing information| RAG
Company knowledge| RAG
Document-grounded answers| RAG
Need citations| RAG
Specific response style| Fine-tuning may help
Specialized task behavior| Fine-tuning may help
Consistent formatting| Fine-tuning may help
In some systems, you may use both.
The correct choice depends on the problem you're solving.
This is one of the most important concepts for AI PMs.
An LLM isn't inherently a fact-checking database.
It's generating outputs based on learned patterns and the information available to it.
Therefore, it can generate something that sounds extremely convincing but is incorrect.
User:
Who wrote the fictional book XYZ?
LLM:
XYZ was written by John Smith in 1987.
The answer sounds plausible.
But it could be completely invented.
This behavior is commonly called a hallucination.
There isn't one magic solution.
Production systems can combine:
Better instructions
Clearly define what the model should and shouldn't do.
Give the model relevant source material.
Grounding
Require responses to rely on provided information.
Structured outputs
Constrain the expected response format.
Tool calling
Let the model retrieve information from reliable systems.
Guardrails
Validate or block problematic outputs.
Evaluations
Continuously test the system against representative examples.
Human review
For high-risk workflows, keep a human in the loop.
An LLM by itself doesn't automatically have access to your:
But your application can provide tools.
User
|
v
LLM
|
+------------+------------+
| | |
v v v
Search DB Check Order Create Ticket
| | |
+------------+------------+
|
v
LLM
|
v
Response
The model can determine that a tool is needed.
The application executes it.
The tool result is returned.
The model then uses that result to continue the interaction.
This is one of the foundations of modern AI agents.
An LLM and an AI agent aren't the same thing.
A basic LLM application:
User
|
v
LLM
|
v
Answer
An agentic system:
User
|
v
Agent
|
v
LLM
|
v
Decide what to do
|
v
Tool
|
v
Observe Result
|
v
LLM
|
v
Decide Next Step
|
v
Tool
|
v
...
|
v
Final Answer
The LLM provides much of the language and reasoning capability.
The surrounding application provides:
This distinction is important when designing AI products.
This is probably the most important architecture to understand as an AI PM.
USER
|
v
+----------------+
| Frontend |
+-------+--------+
|
v
+----------------+
| API Gateway |
+-------+--------+
|
v
+----------------+
| AI Orchestrator|
+-------+--------+
|
+------------+------------+
| | |
v v v
Prompt RAG Tools
Manager
| | |
+------------+------------+
|
v
+----------------+
| LLM Gateway |
+-------+--------+
|
+------------+------------+
| | |
v v v
Model A Model B Model C
| | |
+------------+------------+
|
v
+----------------+
| Guardrails & |
| Validation |
+-------+--------+
|
v
Response
Notice something:
The LLM is only one component.
A production AI application may also need:
«Calling an LLM API is easy. Building a reliable AI product is much harder.»
The model's API price is only part of the equation.
Your total AI cost could include:
Input Tokens
Output Tokens
Embedding Calls
Reranking
LLM Calls
Tool Calls
Vector Database
Compute
Storage
Monitoring
Consider an AI support assistant:
User Question
|
v
Embedding
|
v
Vector Search
|
v
Reranking
|
v
LLM
|
v
Tool Call
|
v
LLM Again
One user interaction can therefore involve multiple computational steps.
That's why AI unit economics are important for Product Managers.
Imagine two applications.
Application A
Question
|
Wait 8 seconds
|
Complete answer
Application B
Question
|
First token in 1 second
|
Streaming...
|
Complete answer in 8 seconds
The total generation time could be similar.
But the perceived experience can be very different.
That's why AI products may track:
AI performance isn't just an infrastructure metric.
It's part of the user experience.
Imagine you have three models:
Model A
High capability
High cost
High latency
Model B
Good capability
Medium cost
Medium latency
Model C
Lower capability
Low cost
Low latency
Which one should your product use?
There's no universal answer.
It depends on the use case.
For a high-value enterprise workflow, higher capability may justify higher costs.
For a high-volume consumer feature, latency and cost might matter more.
For a simple classification task, using the most powerful model available may be unnecessary.
The better question is:
«Which model provides enough quality for this particular user problem at an acceptable cost and latency?»
That's a product question.
When evaluating models, don't look at only one benchmark.
For a real product, you might care about:
Quality
├── Accuracy
├── Factuality
├── Reasoning
├── Instruction Following
├── Safety
├── Consistency
└── Structured Output
Performance
├── Latency
├── Throughput
└── Reliability
Economics
├── Input Cost
├── Output Cost
└── Infrastructure Cost
A model can perform extremely well on a benchmark and still perform poorly for your particular product.
That's why your own evaluation dataset matters.
Suppose you're building an AI customer-support assistant.
You can create a dataset like:
Question
Expected Behavior
Expected Answer Characteristics
Safety Requirements
Example:
Question:
Can I return this product after 30 days?
Expected behavior:
Use the company's actual return policy
and provide the relevant source.
You can then test the system against hundreds or thousands of similar scenarios.
Possible evaluation dimensions include:
This becomes something like automated testing for your AI system.
Prompt engineering is useful.
But production AI systems require much more than a clever prompt.
Think of the stack like this:
User Experience
|
v
Product Workflow
|
v
Prompt / Instructions
|
v
Context / RAG
|
v
Tools
|
v
Model
|
v
Infrastructure
|
v
Evaluation
If your AI feature isn't working, changing the prompt might not solve the real problem.
This is why AI PMs should understand the whole system.
You don't need to implement every component yourself.
But you should understand how the pieces fit together.
+-----------------------------------+
| USER PROBLEM |
+-----------------------------------+
| PRODUCT UX |
+-----------------------------------+
| AI APPLICATION LAYER |
| RAG | Tools | Agents | Memory |
+-----------------------------------+
| LLM LAYER |
| Tokens | Attention | Transformer |
+-----------------------------------+
| MODEL INFRASTRUCTURE |
| GPUs | Serving | Cache | APIs |
+-----------------------------------+
Your job as a PM is to make decisions across these layers.
I would break the learning path into five levels.
Level 1 — Fundamentals
Know:
Level 2 — AI Product Development
Level 3 — Technical AI PM
Understand:
Level 4 — Production AI
Level 5 — AI Product Leadership
Eventually learn:
You don't need Level 5 knowledge to get your first AI PM role.
But knowing the roadmap is useful.
Let's put everything together.
Imagine we're building an AI customer-support assistant.
A customer asks:
«"Where is my order?"»
The architecture could look like:
Customer
|
v
Chat Interface
|
v
Backend
|
v
AI Orchestrator
|
v
LLM
|
|--- "I need order information"
|
v
Order API
|
v
Order Status
|
v
LLM
|
v
Natural Language Response
|
v
Customer
The LLM doesn't necessarily know the customer's order status.
It needs to retrieve that information from the order system.
This is an important distinction:
«The LLM reasons over information. Your application connects it to the systems that contain the information.»
Production AI requires thinking about failure modes.
Hallucination
The model invents an order status.
Possible mitigation: Make the order system the source of truth.
Authorization failure
The system exposes another customer's information.
Possible mitigation: Strong authentication, authorization and tool-level permissions.
Slow API
The order service takes five seconds.
Possible mitigation: Optimize the backend and design the UX around latency.
Excessive context
The application sends the entire conversation on every request.
Possible mitigation: Context management, summarization and appropriate retrieval.
Excessive cost
The system uses an expensive model for every request.
Possible mitigation: Model routing, smaller models for simpler tasks, caching and request optimization.
Prompt injection
A malicious input attempts to manipulate the model or its tools.
Possible mitigation: Defense-in-depth security, permission boundaries, tool authorization, validation and adversarial testing.
If you're becoming an AI Product Manager, this is probably the most useful mindset:
«Don't think of an LLM as a magical brain. Think of it as one component inside a larger probabilistic software system.»
The model is incredibly powerful.
But it isn't perfect.
It doesn't automatically know your company's private information.
It doesn't automatically verify every statement.
It doesn't automatically understand your business rules.
It doesn't automatically have access to your APIs.
And it doesn't automatically produce reliable production behavior.
The surrounding architecture matters just as much.
If you remember only one diagram from this article, remember this:
USER
|
v
PROMPT
|
v
TOKENIZATION
|
v
TOKENS
|
v
EMBEDDINGS
|
v
POSITIONAL INFORMATION
|
v
+-----------------------+
| TRANSFORMER |
| |
| Self-Attention |
| | |
| MLP |
| | |
| Many Layers |
+-----------+-----------+
|
v
LOGITS
|
v
SOFTMAX
|
v
TOKEN SELECTION
|
v
NEXT TOKEN
|
+-----------+
|
v
Repeat Generation
|
v
RESPONSE
And a production AI application:
User
|
v
Product Experience
|
v
Application Logic
|
+------ RAG
|
+------ Tools
|
+------ Memory
|
v
LLM
|
v
Guardrails
|
v
Evaluation
|
v
Response
That second diagram is the one I would keep in mind as a Product Manager.
Understanding LLMs doesn't mean memorizing every equation behind a Transformer.
For a Product Manager, the goal is to understand enough to answer:
«What is technically possible?»
«What will it cost?»
«How fast will it be?»
«How reliable will it be?»
«What can go wrong?»
«What architecture do we need?»
«And, most importantly, does this actually solve a user problem?»
The simplest LLM mental model is:
Text
↓
Tokens
↓
Embeddings
↓
Transformer
↓
Attention
↓
Probability Distribution
↓
Next Token
↓
Repeat
↓
Response
And the production AI product is:
User
↓
Product Experience
↓
Application Logic
↓
Context / RAG
↓
Tools / APIs
↓
LLM
↓
Guardrails
↓
Evaluation
↓
Response
Once you understand these two flows, concepts such as RAG, AI agents, LLM gateways, model routing, prompt engineering, fine-tuning, AI evaluation and AI infrastructure become much easier to understand.
And that's the level of technical depth I'd recommend for an aspiring AI Product Manager:
Know enough to understand the technology, challenge assumptions, work effectively with engineers, and make better product decisions — without trying to become a foundation-model researcher.
Further Reading