There is a moment in every LLM project where someone says the
quiet thing out loud. "The context window is a million tokens
now. Why are we still building a retrieval pipeline? Just put
the whole codebase in."
It is a reasonable question. It has an answer, and the answer is
on the price sheet rather than in the architecture diagram.
OpenAI announced GPT-6 Astra on 3 September 2026. It takes text
and image input, returns text, and carries a 1M token context window. The launch list price for the standard tier is
Put the two specs next to each other. One million tokens of
context. Ten dollars per million input tokens. Fill the window
and you have spent $10 before the model has emitted a single
token of answer.
A giant context window is not a replacement for retrieval. It is
a way to pay for the retrieval you did not do.
OpenAI published its own benchmark results at launch. Those are
vendor-reported rather than independently verified, and worth
reading in that light: GPQA Diamond at 96%, FrontierMath Tier 4
v2 at 97.6%, ARC-AGI-3 at 98.6%.
None of those numbers tell you whether to fill the window. The
price sheet does.
Every number below is the published per-token price multiplied
out. Nobody's invoice was consulted.
Standard tier, one request that fills the context window:
The input side is 250 times the output side. For an ordinary
request shape, that ratio runs the other way and output is where
your bill lives. The moment you fill a million-token window,
that reverses and it is not close.
Give the thing traffic. A thousand requests a day, each one
stuffing the window:
On the fast tier it is $20 per filled window, so double it.
The retrieved version of the same feature sends about 8,000
tokens of prompt instead:
Same model. Same question. Same answer length, so the output
half of the bill is identical either way. The entire difference
lives on the input side, and it is 125x.
If your corpus is genuinely fixed across requests, check whether
cached input pricing applies to your account before you accept
the $10 as your real number. Caching helps most exactly where the
same bytes go up over and over. It does nothing for a corpus that
changes per request, and it does not change any of the other
four reasons below.
Cost is the loudest argument and it is not the only one.
Relevance is a thing you can inspect. When you retrieve the
top twelve chunks, you have a list. You can read it. You can
check whether the chunk that answers the question is in it. When
you stuff a million tokens, you have a haystack and a hope.
Failure becomes two separable questions. A wrong answer from
a retrieval pipeline splits cleanly: was the right chunk
retrieved, and did the model use it. Those have different fixes.
Bad retrieval means your chunking, your embedding model, or your
query is wrong. Good retrieval with a bad answer means your
prompt is wrong. A wrong answer out of a stuffed window is one
undifferentiated problem, and the only lever you have is to
rewrite the instructions and try again.
Latency follows the input. A million tokens has to be sent
and processed before the first token of the answer comes back.
Eight thousand does not. You do not need a benchmark to know
which of those a user waiting on a spinner prefers.
Citations. This is the one that changes what you can ship.
Retrieved chunks carry ids. Those ids go into the prompt, come
back in the answer, get rendered as sources in your UI, and get
written to your logs. Six months later somebody asks why the
system told a customer the wrong refund policy, and you can
answer, because you know which paragraph of which document
version was in front of the model. Stuff the window and the
honest answer to that question is "all of it".
I am deliberately not making a claim here about answer quality
degrading over long contexts. That is contested, it depends on
the model, and I have not measured it on Astra. The four
arguments above hold without it.
Here is the whole pipeline in TypeScript. It is about a hundred
lines. Set that against the $300,000 a month that the
thousand-requests-a-day arithmetic above produces, and the
build-versus-buy conversation gets short.
Start by documents off disk. Swap this for your database,
your S3 bucket, your Git repo.
import { readdir, readFile } from "node:fs/promises";
import { join } from "node:path";
export type Doc = { id: string; text: string };
export async function loadDocs(dir: string): Promise<Doc[]> {
const names = await readdir(dir);
return Promise.all(
names
.filter((n) => n.endsWith(".md"))
.map(async (n) => ({
id: n,
text: await readFile(join(dir, n), "utf8"),
})),
);
}
Then chunk. Fixed-size windows with overlap, because the
sentence that answers the question has a habit of landing exactly
on a boundary.
export type Chunk = { id: string; docId: string; text: string };
export function chunkDoc(
doc: Doc,
size = 2000,
overlap = 250,
): Chunk[] {
const out: Chunk[] = [];
const step = size - overlap;
for (let i = 0; i < doc.text.length; i += step) {
const text = doc.text.slice(i, i + size);
if (text.trim().length > 0) {
out.push({ id: `${doc.id}#${i}`, docId: doc.id, text });
}
if (i + size >= doc.text.length) break;
}
return out;
}
The chunk id
is the character offset inside the document. That
is the citation trail, and it costs one string template to keep.
Do not throw it away.
Character-count chunking is the version you start with. Splitting
on headings or paragraphs beats it on prose that has structure,
and both beat splitting on a fixed token count that cuts
mid-sentence.
One embedding call per batch of chunks, at index time. The batch
size is not decoration. Hand the endpoint your whole corpus in a
single call and the first real run fails on the array limit, so
the loop is part of the code, not an optimisation for later.
import OpenAI from "openai";
export const client = new OpenAI();
export async function embed(
texts: string[],
batch = 256,
): Promise<number[][]> {
const out: number[][] = [];
for (let i = 0; i < texts.length; i += batch) {
const res = await client.embeddings.create({
model: "text-embedding-3-small",
input: texts.slice(i, i + batch),
});
out.push(...res.data.map((d) => d.embedding));
}
return out;
}
The cost shape here matters more than the number. Embedding is
paid once per document version and amortised across every query
that document ever serves. The $10 window fill is paid once per
request, forever. There is no constant to quote for the first
half: it is corpus size times update frequency divided by query
volume. A corpus that changes hourly and one that changes yearly
land orders of magnitude apart, so compute it from the pricing page with your own numbers.
export type Indexed = Chunk & { vector: number[] };
export async function buildIndex(
chunks: Chunk[],
): Promise<Indexed[]> {
const vectors = await embed(chunks.map((c) => c.text));
return chunks.map((c, i) => ({ ...c, vector: vectors[i] }));
}
As a rough rule of thumb, an in-memory array holds up into the
tens of thousands of chunks. Measure it on your own data rather
than taking that number from me. Past the point where it stops
holding up, reach for a vector store. Do not reach for one on day
one to look serious.
Cosine similarity, sorted, sliced.
export function cosine(a: number[], b: number[]): number {
let dot = 0;
let na = 0;
let nb = 0;
for (let i = 0; i < a.length; i++) {
dot += a[i] * b[i];
na += a[i] * a[i];
nb += b[i] * b[i];
}
return dot / (Math.sqrt(na) * Math.sqrt(nb));
}
If your embedding model returns unit vectors, the dot product
alone would do the job. The full cosine costs two extra loops and
is correct either way, which is worth more than the loops cost
you at index time.
export async function retrieve(
index: Indexed[],
query: string,
k = 12,
): Promise<Indexed[]> {
const [q] = await embed([query]);
return index
.map((c) => ({ c, score: cosine(q, c.vector) }))
.sort((a, b) => b.score - a.score)
.slice(0, k)
.map((r) => r.c);
}
k
is the dial that connects quality to the invoice. Twelve
chunks of 2,000 characters is roughly 6,000 tokens, which is
about six cents of input on Astra's standard tier. Going to
k = 24
doubles that to twelve cents and still uses just over 1%
of the window. Tune it on your own evaluation set. The point is
that you have a dial. A stuffed window has one setting.
Then assemble the prompt so the answer can cite its sources.
export function buildContext(hits: Indexed[]): string {
return hits
.map((h, i) => `[${i + 1}] source=${h.id}\n${h.text}`)
.join("\n\n");
}
export const SYSTEM = [
"Answer only from the numbered sources below.",
"Cite the source number after every claim, like [3].",
"If the sources do not contain the answer, say so.",
].join("\n");
That system prompt is doing real work. It turns "the model said
something" into "the model said something and pointed at the
paragraph it came from", which is the difference between a demo
and something you can put in front of a customer.
The part that ends the argument in your team's planning meeting.
const IN_PER_M = 10; // launch list price, USD per 1M input
export const estTokens = (s: string) => Math.ceil(s.length / 4);
export function inputUSD(tokens: number): number {
return (tokens / 1e6) * IN_PER_M;
}
Four characters per token is close enough to make the decision
and wrong enough that nobody should quote it in a budget. Run the
model's own tokeniser before that number leaves your team.
const MODEL = process.env.OPENAI_MODEL!;
const docs = await loadDocs("./corpus");
const index = await buildIndex(docs.flatMap((d) => chunkDoc(d)));
const question = "Why did checkout retries double in July?";
const hits = await retrieve(index, question, 12);
const context = buildContext(hits);
const stuffed = docs.reduce((n, d) => n + estTokens(d.text), 0);
const sent = estTokens(SYSTEM + context + question);
console.log("stuff ", stuffed, inputUSD(stuffed).toFixed(2));
console.log("retrieve", sent, inputUSD(sent).toFixed(2));
Set OPENAI_MODEL
to whatever the current model id is on the
pricing page. Then run it against your own corpus and read the
two lines. If your documents fill the window, the first line is
1000000 and 10.00. The second, with the twelve 2,000-character
chunks these settings produce, lands near 6100 and 0.06. Give it
a real system prompt and answer-format instructions and you are
at 8,000 tokens and eight cents. Either way the ratio between
the two lines is the only thing you need to read.
Send the retrieved version:
const res = await client.chat.completions.create({
model: MODEL,
messages: [
{ role: "system", content: SYSTEM },
{ role: "user", content: `${context}\n\nQ: ${question}` },
],
});
console.log(res.choices[0].message.content);
console.log("sources:", hits.map((h) => h.id));
That last line is the whole citation trail. Log it next to the
answer and next to the cost, and every future question about
this system has an answer in your database.
The window is not a mistake. There is work it is built for, and
pretending otherwise would be as lazy as the "just put it all
in" argument.
Two of those cases are about volume. If you need one answer about
one large document, once, ten dollars is cheaper than an
afternoon of your time building an index. And if the whole corpus
is 20,000 tokens, retrieval is machinery around a problem you do
not have. Send it all.
The other two are about the shape of the task. Summarise this
whole book. Find every place in this repository that touches the
payment flow. Retrieval finds passages that resemble a query, and
neither of those has one. Top-k over a codebase will miss the
file that matters because it never mentioned the words you
searched for. Long agentic runs land in the same category for a
different reason: when a missed file turns into a pile of paid
steps built on top of a wrong turn, buying the full context up
front can be the cheaper option.
What those share is low request volume, or a task whose failure
mode costs more than the window does. High-volume question
answering over a document corpus is the opposite of that on both
counts, and it is also the single most common thing teams build.
Read the spec again with the price sheet next to it. One million
tokens at $10 per million. Read that as a spec and you see
capacity somebody handed you. Read it as a price and you see ten
dollars of per-call budget, where every token that goes in is a
token you chose to pay for.
Take your largest prompt in production and print its token count
next to its dollar cost, in the log line, at the point of the
call. Most teams have never looked at that number and are
surprised by it.
Then, if you are stuffing anything large, build the hundred lines
above against your real corpus and print both figures. You do not
need a decision meeting after that. You need the two numbers, and
they will make the decision look obvious in retrospect.
Chunking, embeddings, top-k and the citation trail are the parts
of an LLM system that decide whether your answers are grounded
and whether your invoice is survivable. That is what my book AI That Reads covers, in TypeScript, from a flat file on disk
It is book 2 of AI in TypeScript, a five-book series that runs from your first LLM call through to agents you can leave running in production.