# The Future of Large Language Models > Source: > Published: 2026-06-16 19:31:12+00:00 We are building an autonomous research agent that turns a vague question into a structured plan, gathers evidence across multiple calls, and synthesizes a markdown report. This is the practical future of LLMs: not monolithic chat, but small, orchestrated reasoning loops that leverage long context and tool use. Because Oxlo.ai charges a flat rate per request instead of per token ([see pricing](https://oxlo.ai/pricing)), running multi-step agent workflows like this stays predictable even when prompts grow. `pip install openai` We point the OpenAI SDK at Oxlo.ai. If you want to experiment later, Oxlo.ai also offers reasoning specialists such as DeepSeek R1 671B MoE and Kimi K2.6, but Llama 3.3 70B is a solid general-purpose default for this pipeline. ``` python from openai import OpenAI client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY") ``` The system prompt forces the model to stay in character and emit structured output. We keep it strict so downstream parsing stays reliable. ``` SYSTEM_PROMPT = """You are a research agent. Your job is to help a user investigate a complex topic. When asked to plan, return exactly one sub-question per line, no bullets, no numbers. When asked to answer a sub-question, return a concise, factual paragraph with citations if possible. When asked to synthesize, return a markdown report with an H1 title, an executive summary, and detailed sections.""" ``` We send the user query to the model and ask for a list of sub-questions. We split the response on newlines to get discrete tasks. ``` python from openai import OpenAI client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY") SYSTEM_PROMPT = """You are a research agent. Your job is to help a user investigate a complex topic. When asked to plan, return exactly one sub-question per line, no bullets, no numbers. When asked to answer a sub-question, return a concise, factual paragraph with citations if possible. When asked to synthesize, return a markdown report with an H1 title, an executive summary, and detailed sections.""" def generate_plan(user_query: str) -> list[str]: planning_prompt = ( f"User question: {user_query}\n\n" "Generate exactly 3 focused sub-questions that will help answer the user question. " "Return one per line, no numbering." ) response = client.chat.completions.create( model="llama-3.3-70b", messages=[ {"role": "system", "content": SYSTEM_PROMPT}, {"role": "user", "content": planning_prompt}, ], ) raw = response.choices[0].message.content.strip() return [line.strip() for line in raw.splitlines() if line.strip()] # Example plan = generate_plan("What are the trade-offs between retrieval-augmented generation and long-context LLMs?") print(plan) ``` We loop over the plan and call the model once per sub-question. On Oxlo.ai, each call costs the same flat amount regardless of prompt length, so expanding context here does not explode the bill. ``` python from openai import OpenAI client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY") SYSTEM_PROMPT = """You are a research agent. Your job is to help a user investigate a complex topic. When asked to plan, return exactly one sub-question per line, no bullets, no numbers. When asked to answer a sub-question, return a concise, factual paragraph with citations if possible. When asked to synthesize, return a markdown report with an H1 title, an executive summary, and detailed sections.""" def gather_evidence(sub_questions: list[str]) -> dict[str, str]: evidence = {} for idx, question in enumerate(sub_questions, 1): answer_prompt = f"Sub-question {idx}: {question}\n\nAnswer concisely." response = client.chat.completions.create( model="llama-3.3-70b", messages=[ {"role": "system", "content": SYSTEM_PROMPT}, {"role": "user", "content": answer_prompt}, ], ) evidence[question] = response.choices[0].message.content.strip() return evidence # Assuming 'plan' from Step 3 answers = gather_evidence(plan) for q, a in answers.items(): print(f"Q: {q}\nA: {a}\n") ``` Finally, we feed the collected evidence back into the model with a synthesis prompt. This demonstrates the long-context strength of modern LLMs: condensing multiple reasoning steps into a coherent deliverable. ``` python from openai import OpenAI client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY") SYSTEM_PROMPT = """You are a research agent. Your job is to help a user investigate a complex topic. When asked to plan, return exactly one sub-question per line, no bullets, no numbers. When asked to answer a sub-question, return a concise, factual paragraph with citations if possible. When asked to synthesize, return a markdown report with an H1 title, an executive summary, and detailed sections.""" def synthesize(user_query: str, evidence: dict[str, str]) -> str: evidence_block = "\n\n".join([f"Sub-question: {q}\nAnswer: {a}" for q, a in evidence.items()]) synthesis_prompt = ( f"Original question: {user_query}\n\n" f"Evidence collected:\n\n{evidence_block}\n\n" "Synthesize the above into a final markdown report." ) response = client.chat.completions.create( model="llama-3.3-70b", messages=[ {"role": "system", "content": SYSTEM_PROMPT}, {"role": "user", "content": synthesis_prompt}, ], ) return response.choices[0].message.content.strip() # Assuming 'query' and 'answers' from previous steps report = synthesize("What are the trade-offs between retrieval-augmented generation and long-context LLMs?", answers) print(report) ``` Here is the complete script. I run it on the topic above. Because Oxlo.ai has no cold starts on popular models, the multi-turn pipeline executes immediately. ``` python from openai import OpenAI client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY") SYSTEM_PROMPT = """You are a research agent. Your job is to help a user investigate a complex topic. When asked to plan, return exactly one sub-question per line, no bullets, no numbers. When asked to answer a sub-question, return a concise, factual paragraph with citations if possible. When asked to synthesize, return a markdown report with an H1 title, an executive summary, and detailed sections.""" def generate_plan(user_query: str) -> list[str]: planning_prompt = ( f"User question: {user_query}\n\n" "Generate exactly 3 focused sub-questions that will help answer the user question. " "Return one per line, no numbering." ) response = client.chat.completions.create( model="llama-3.3-70b", messages=[ {"role": "system", "content": SYSTEM_PROMPT}, {"role": "user", "content": planning_prompt}, ], ) raw = response.choices[0].message.content.strip() return [line.strip() for line in raw.splitlines() if line.strip()] def gather_evidence(sub_questions: list[str]) -> dict[str, str]: evidence = {} for idx, question in enumerate(sub_questions, 1): answer_prompt = f"Sub-question {idx}: {question}\n\nAnswer concisely." response = client.chat.completions.create( model="llama-3.3-70b", messages=[ {"role": "system", "content": SYSTEM_PROMPT}, {"role": "user", "content": answer_prompt}, ], ) evidence[question] = response.choices[0].message.content.strip() return evidence def synthesize(user_query: str, evidence: dict[str, str]) -> str: evidence_block = "\n\n".join([f"Sub-question: {q}\nAnswer: {a}" for q, a in evidence.items()]) synthesis_prompt = ( f"Original question: {user_query}\n\n" f"Evidence collected:\n\n{evidence_block}\n\n" "Synthesize the above into a final markdown report." ) response = client.chat.completions.create( model="llama-3.3-70b", messages=[ {"role": "system", "content": SYSTEM_PROMPT}, {"role": "user", "content": synthesis_prompt}, ], ) return response.choices[0].message.content.strip() if __name__ == "__main__": query = "What are the trade-offs between retrieval-augmented generation and long-context LLMs?" plan = generate_plan(query) answers = gather_evidence(plan) report = synthesize(query, answers) print(report) ``` Example output: ``` # Trade-offs Between Retrieval-Augmented Generation and Long-Context LLMs ## Executive Summary Retrieval-augmented generation (RAG) and long-context LLMs both aim to ground model outputs in external knowledge, but they differ in cost structure, latency, and accuracy dynamics. ## Detailed Analysis ### Cost and Infrastructure RAG requires vector databases, embedding pipelines, and chunking strategies. Long-context models eliminate much of that infrastructure but demand larger GPU memory and longer inference times per request. ### Accuracy and Hallucination RAG pinpoints specific source snippets, which reduces hallucination for fact-heavy queries. Long-context models can lose signal in the middle of a huge prompt unless trained with strong attention mechanisms. ### Latency RAG adds a retrieval round-trip. Long-context models process everything in a single forward pass, though total time can still be high for 100K+ token windows. ## Conclusion Hybrid architectures are emerging: use RAG for initial filtering, then feed a smaller, relevant corpus into a long-context model for synthesis. ``` Swap Llama 3.3 70B for Kimi K2.6 or DeepSeek V3.2 if you want stronger reasoning in the synthesis step. You can also replace the simulated evidence loop with real tool calls using Oxlo.ai's function calling support, feeding live search results or database rows into the same pipeline.