{"slug": "filling-gpt-6-astra-s-1m-token-window-costs-10-a-call", "title": "Filling GPT-6 Astra's 1M-Token Window Costs $10 a Call", "summary": "OpenAI's GPT-6 Astra, launched September 3, 2026, offers a 1M-token context window at $10 per million input tokens, making a fully filled window cost $10 per request. An engineer argues that stuffing the window is not a substitute for retrieval, citing cost, relevance, debuggability, latency, and citation benefits of retrieval pipelines.", "body_md": "There is a moment in every LLM project where someone says the\n\nquiet thing out loud. \"The context window is a million tokens\n\nnow. Why are we still building a retrieval pipeline? Just put\n\nthe whole codebase in.\"\n\nIt is a reasonable question. It has an answer, and the answer is\n\non the price sheet rather than in the architecture diagram.\n\nOpenAI announced GPT-6 Astra on 3 September 2026. It takes text\n\nand image input, returns text, and carries a **1M token context\nwindow**. The launch list price for the standard tier is\n\nPut the two specs next to each other. One million tokens of\n\ncontext. Ten dollars per million input tokens. Fill the window\n\nand you have spent $10 before the model has emitted a single\n\ntoken of answer.\n\nA giant context window is not a replacement for retrieval. It is\n\na way to pay for the retrieval you did not do.\n\nOpenAI published its own benchmark results at launch. Those are\n\nvendor-reported rather than independently verified, and worth\n\nreading in that light: GPQA Diamond at 96%, FrontierMath Tier 4\n\nv2 at 97.6%, ARC-AGI-3 at 98.6%.\n\nNone of those numbers tell you whether to fill the window. The\n\nprice sheet does.\n\nEvery number below is the published per-token price multiplied\n\nout. Nobody's invoice was consulted.\n\nStandard tier, one request that fills the context window:\n\nThe input side is 250 times the output side. For an ordinary\n\nrequest shape, that ratio runs the other way and output is where\n\nyour bill lives. The moment you fill a million-token window,\n\nthat reverses and it is not close.\n\nGive the thing traffic. A thousand requests a day, each one\n\nstuffing the window:\n\nOn the fast tier it is $20 per filled window, so double it.\n\nThe retrieved version of the same feature sends about 8,000\n\ntokens of prompt instead:\n\nSame model. Same question. Same answer length, so the output\n\nhalf of the bill is identical either way. The entire difference\n\nlives on the input side, and it is 125x.\n\nIf your corpus is genuinely fixed across requests, check whether\n\ncached input pricing applies to your account before you accept\n\nthe $10 as your real number. Caching helps most exactly where the\n\nsame bytes go up over and over. It does nothing for a corpus that\n\nchanges per request, and it does not change any of the other\n\nfour reasons below.\n\nCost is the loudest argument and it is not the only one.\n\n**Relevance is a thing you can inspect.** When you retrieve the\n\ntop twelve chunks, you have a list. You can read it. You can\n\ncheck whether the chunk that answers the question is in it. When\n\nyou stuff a million tokens, you have a haystack and a hope.\n\n**Failure becomes two separable questions.** A wrong answer from\n\na retrieval pipeline splits cleanly: was the right chunk\n\nretrieved, and did the model use it. Those have different fixes.\n\nBad retrieval means your chunking, your embedding model, or your\n\nquery is wrong. Good retrieval with a bad answer means your\n\nprompt is wrong. A wrong answer out of a stuffed window is one\n\nundifferentiated problem, and the only lever you have is to\n\nrewrite the instructions and try again.\n\n**Latency follows the input.** A million tokens has to be sent\n\nand processed before the first token of the answer comes back.\n\nEight thousand does not. You do not need a benchmark to know\n\nwhich of those a user waiting on a spinner prefers.\n\n**Citations.** This is the one that changes what you can ship.\n\nRetrieved chunks carry ids. Those ids go into the prompt, come\n\nback in the answer, get rendered as sources in your UI, and get\n\nwritten to your logs. Six months later somebody asks why the\n\nsystem told a customer the wrong refund policy, and you can\n\nanswer, because you know which paragraph of which document\n\nversion was in front of the model. Stuff the window and the\n\nhonest answer to that question is \"all of it\".\n\nI am deliberately not making a claim here about answer quality\n\ndegrading over long contexts. That is contested, it depends on\n\nthe model, and I have not measured it on Astra. The four\n\narguments above hold without it.\n\nHere is the whole pipeline in TypeScript. It is about a hundred\n\nlines. Set that against the $300,000 a month that the\n\nthousand-requests-a-day arithmetic above produces, and the\n\nbuild-versus-buy conversation gets short.\n\nStart by loading documents off disk. Swap this for your database,\n\nyour S3 bucket, your Git repo.\n\n``` js\nimport { readdir, readFile } from \"node:fs/promises\";\nimport { join } from \"node:path\";\n\nexport type Doc = { id: string; text: string };\n\nexport async function loadDocs(dir: string): Promise<Doc[]> {\n  const names = await readdir(dir);\n  return Promise.all(\n    names\n      .filter((n) => n.endsWith(\".md\"))\n      .map(async (n) => ({\n        id: n,\n        text: await readFile(join(dir, n), \"utf8\"),\n      })),\n  );\n}\n```\n\nThen chunk. Fixed-size windows with overlap, because the\n\nsentence that answers the question has a habit of landing exactly\n\non a boundary.\n\n```\nexport type Chunk = { id: string; docId: string; text: string };\n\nexport function chunkDoc(\n  doc: Doc,\n  size = 2000,\n  overlap = 250,\n): Chunk[] {\n  const out: Chunk[] = [];\n  const step = size - overlap;\n  for (let i = 0; i < doc.text.length; i += step) {\n    const text = doc.text.slice(i, i + size);\n    if (text.trim().length > 0) {\n      out.push({ id: `${doc.id}#${i}`, docId: doc.id, text });\n    }\n    if (i + size >= doc.text.length) break;\n  }\n  return out;\n}\n```\n\nThe chunk `id`\n\nis the character offset inside the document. That\n\nis the citation trail, and it costs one string template to keep.\n\nDo not throw it away.\n\nCharacter-count chunking is the version you start with. Splitting\n\non headings or paragraphs beats it on prose that has structure,\n\nand both beat splitting on a fixed token count that cuts\n\nmid-sentence.\n\nOne embedding call per batch of chunks, at index time. The batch\n\nsize is not decoration. Hand the endpoint your whole corpus in a\n\nsingle call and the first real run fails on the array limit, so\n\nthe loop is part of the code, not an optimisation for later.\n\n``` python\nimport OpenAI from \"openai\";\n\nexport const client = new OpenAI();\n\nexport async function embed(\n  texts: string[],\n  batch = 256,\n): Promise<number[][]> {\n  const out: number[][] = [];\n  for (let i = 0; i < texts.length; i += batch) {\n    const res = await client.embeddings.create({\n      model: \"text-embedding-3-small\",\n      input: texts.slice(i, i + batch),\n    });\n    out.push(...res.data.map((d) => d.embedding));\n  }\n  return out;\n}\n```\n\nThe cost shape here matters more than the number. Embedding is\n\npaid once per document version and amortised across every query\n\nthat document ever serves. The $10 window fill is paid once per\n\nrequest, forever. There is no constant to quote for the first\n\nhalf: it is corpus size times update frequency divided by query\n\nvolume. A corpus that changes hourly and one that changes yearly\n\nland orders of magnitude apart, so compute it from the [pricing\npage](https://openai.com/api/pricing/) with your own numbers.\n\n```\nexport type Indexed = Chunk & { vector: number[] };\n\nexport async function buildIndex(\n  chunks: Chunk[],\n): Promise<Indexed[]> {\n  const vectors = await embed(chunks.map((c) => c.text));\n  return chunks.map((c, i) => ({ ...c, vector: vectors[i] }));\n}\n```\n\nAs a rough rule of thumb, an in-memory array holds up into the\n\ntens of thousands of chunks. Measure it on your own data rather\n\nthan taking that number from me. Past the point where it stops\n\nholding up, reach for a vector store. Do not reach for one on day\n\none to look serious.\n\nCosine similarity, sorted, sliced.\n\n``` js\nexport function cosine(a: number[], b: number[]): number {\n  let dot = 0;\n  let na = 0;\n  let nb = 0;\n  for (let i = 0; i < a.length; i++) {\n    dot += a[i] * b[i];\n    na += a[i] * a[i];\n    nb += b[i] * b[i];\n  }\n  return dot / (Math.sqrt(na) * Math.sqrt(nb));\n}\n```\n\nIf your embedding model returns unit vectors, the dot product\n\nalone would do the job. The full cosine costs two extra loops and\n\nis correct either way, which is worth more than the loops cost\n\nyou at index time.\n\n```\nexport async function retrieve(\n  index: Indexed[],\n  query: string,\n  k = 12,\n): Promise<Indexed[]> {\n  const [q] = await embed([query]);\n  return index\n    .map((c) => ({ c, score: cosine(q, c.vector) }))\n    .sort((a, b) => b.score - a.score)\n    .slice(0, k)\n    .map((r) => r.c);\n}\n```\n\n`k`\n\nis the dial that connects quality to the invoice. Twelve\n\nchunks of 2,000 characters is roughly 6,000 tokens, which is\n\nabout six cents of input on Astra's standard tier. Going to\n\n`k = 24`\n\ndoubles that to twelve cents and still uses just over 1%\n\nof the window. Tune it on your own evaluation set. The point is\n\nthat you *have* a dial. A stuffed window has one setting.\n\nThen assemble the prompt so the answer can cite its sources.\n\n```\nexport function buildContext(hits: Indexed[]): string {\n  return hits\n    .map((h, i) => `[${i + 1}] source=${h.id}\\n${h.text}`)\n    .join(\"\\n\\n\");\n}\n\nexport const SYSTEM = [\n  \"Answer only from the numbered sources below.\",\n  \"Cite the source number after every claim, like [3].\",\n  \"If the sources do not contain the answer, say so.\",\n].join(\"\\n\");\n```\n\nThat system prompt is doing real work. It turns \"the model said\n\nsomething\" into \"the model said something and pointed at the\n\nparagraph it came from\", which is the difference between a demo\n\nand something you can put in front of a customer.\n\nThe part that ends the argument in your team's planning meeting.\n\n``` js\nconst IN_PER_M = 10; // launch list price, USD per 1M input\n\nexport const estTokens = (s: string) => Math.ceil(s.length / 4);\n\nexport function inputUSD(tokens: number): number {\n  return (tokens / 1e6) * IN_PER_M;\n}\n```\n\nFour characters per token is close enough to make the decision\n\nand wrong enough that nobody should quote it in a budget. Run the\n\nmodel's own tokeniser before that number leaves your team.\n\n``` js\nconst MODEL = process.env.OPENAI_MODEL!;\n\nconst docs = await loadDocs(\"./corpus\");\nconst index = await buildIndex(docs.flatMap((d) => chunkDoc(d)));\n\nconst question = \"Why did checkout retries double in July?\";\nconst hits = await retrieve(index, question, 12);\nconst context = buildContext(hits);\n\nconst stuffed = docs.reduce((n, d) => n + estTokens(d.text), 0);\nconst sent = estTokens(SYSTEM + context + question);\n\nconsole.log(\"stuff \", stuffed, inputUSD(stuffed).toFixed(2));\nconsole.log(\"retrieve\", sent, inputUSD(sent).toFixed(2));\n```\n\nSet `OPENAI_MODEL`\n\nto whatever the current model id is on the\n\npricing page. Then run it against your own corpus and read the\n\ntwo lines. If your documents fill the window, the first line is\n\n1000000 and 10.00. The second, with the twelve 2,000-character\n\nchunks these settings produce, lands near 6100 and 0.06. Give it\n\na real system prompt and answer-format instructions and you are\n\nat 8,000 tokens and eight cents. Either way the ratio between\n\nthe two lines is the only thing you need to read.\n\nSend the retrieved version:\n\n``` js\nconst res = await client.chat.completions.create({\n  model: MODEL,\n  messages: [\n    { role: \"system\", content: SYSTEM },\n    { role: \"user\", content: `${context}\\n\\nQ: ${question}` },\n  ],\n});\n\nconsole.log(res.choices[0].message.content);\nconsole.log(\"sources:\", hits.map((h) => h.id));\n```\n\nThat last line is the whole citation trail. Log it next to the\n\nanswer and next to the cost, and every future question about\n\nthis system has an answer in your database.\n\nThe window is not a mistake. There is work it is built for, and\n\npretending otherwise would be as lazy as the \"just put it all\n\nin\" argument.\n\nTwo of those cases are about volume. If you need one answer about\n\none large document, once, ten dollars is cheaper than an\n\nafternoon of your time building an index. And if the whole corpus\n\nis 20,000 tokens, retrieval is machinery around a problem you do\n\nnot have. Send it all.\n\nThe other two are about the shape of the task. Summarise this\n\nwhole book. Find every place in this repository that touches the\n\npayment flow. Retrieval finds passages that resemble a query, and\n\nneither of those has one. Top-k over a codebase will miss the\n\nfile that matters because it never mentioned the words you\n\nsearched for. Long agentic runs land in the same category for a\n\ndifferent reason: when a missed file turns into a pile of paid\n\nsteps built on top of a wrong turn, buying the full context up\n\nfront can be the cheaper option.\n\nWhat those share is low request volume, or a task whose failure\n\nmode costs more than the window does. High-volume question\n\nanswering over a document corpus is the opposite of that on both\n\ncounts, and it is also the single most common thing teams build.\n\nRead the spec again with the price sheet next to it. One million\n\ntokens at $10 per million. Read that as a spec and you see\n\ncapacity somebody handed you. Read it as a price and you see ten\n\ndollars of per-call budget, where every token that goes in is a\n\ntoken you chose to pay for.\n\nTake your largest prompt in production and print its token count\n\nnext to its dollar cost, in the log line, at the point of the\n\ncall. Most teams have never looked at that number and are\n\nsurprised by it.\n\nThen, if you are stuffing anything large, build the hundred lines\n\nabove against your real corpus and print both figures. You do not\n\nneed a decision meeting after that. You need the two numbers, and\n\nthey will make the decision look obvious in retrospect.\n\nChunking, embeddings, top-k and the citation trail are the parts\n\nof an LLM system that decide whether your answers are grounded\n\nand whether your invoice is survivable. That is what my book *AI\nThat Reads* covers, in TypeScript, from a flat file on disk\n\nIt is book 2 of *AI in TypeScript*, a five-book series that runs from your first LLM call through to agents you can leave running in production.", "url": "https://wpnews.pro/news/filling-gpt-6-astra-s-1m-token-window-costs-10-a-call", "canonical_source": "https://dev.to/gabrielanhaia/filling-gpt-6-astras-1m-token-window-costs-10-a-call-1m46", "published_at": "2026-09-03 21:53:40+00:00", "updated_at": "2026-09-03 22:24:48.482705+00:00", "lang": "en", "topics": ["large-language-models", "ai-products", "ai-infrastructure", "developer-tools"], "entities": ["OpenAI", "GPT-6 Astra"], "alternates": {"html": "https://wpnews.pro/news/filling-gpt-6-astra-s-1m-token-window-costs-10-a-call", "markdown": "https://wpnews.pro/news/filling-gpt-6-astra-s-1m-token-window-costs-10-a-call.md", "text": "https://wpnews.pro/news/filling-gpt-6-astra-s-1m-token-window-costs-10-a-call.txt", "jsonld": "https://wpnews.pro/news/filling-gpt-6-astra-s-1m-token-window-costs-10-a-call.jsonld"}}