Thursdays with Koog: Prompts Everywhere

Mark Murphy's 'Thursdays with Koog' column explains that in the AIAgent framework, the user's instruction is only a 'user message' within a larger prompt that includes a system prompt and conversation history, and that AIAgent handles assembling these automatically. For multi-turn interactions, PromptExecutor offers more control, allowing developers to build a complete Prompt using Koog's DSL and pass it to execute(), whereas AIAgent's callback approach for subsequent user messages is considered awkward. The column highlights the trade-offs between using AIAgent's convenience and PromptExecutor's flexibility for managing prompts in coding agents.

Last Thursday, we explored the differences between LLMClient, PromptExecutor, and AIAgent https://pac.commonsware.com/archive/thursdays-with-koog-promptexecutor-vs-aiagent/ in the AIAgent , mostly for its tool support.It turns out that how you work with prompts differs depending on which of these layers you work with. You have a richer API at lower layers, mostly because AIAgent is designed to handle much of prompt management for you. As users of coding agents, we are used to thinking of "the prompt" as being "what we tell the agent to do". So, for example: knosh "Make me a sandwich" obligatory XKCD shout-out https://xkcd.com/149/ Here, Make me a sandwich is a prompt. It is a fairly lousy prompt, in that coding agents generally are ill-suited for meal prep, but, hey. While that is a prompt, it is not the prompt that gets sent to the LLM. Make me a sandwich is a "user message": something you the user is sending to the LLM. With a one-shot agent like Knosh, that is the one and only user message. However, most LLM clients are "multi-turn", with multiple user messages interspersed with LLM replies. Plus, many LLM clients have a "system prompt": a message that is supplied by the agent's developers , that forms the first text in the prompt sent to the LLM. You can envision this in the form of a chat transcript: | You are a helpful coding agent. Make me a sandwich. < Sorry, Dave, I can't do that. Would you like me to open the pod bay door instead? Um, I don't have a "pod bay door", and my name is not Dave. < I like to think that all my users are Dave, each in their own way. Are you sure that you are coding agent? < Well, you asked for a sandwich. I assumed that the normal "coding agent" guidance was part of some other... mythos. Here, | represents the system prompt, prefixes user messages, and < prefixes assistant messages replies from the LLM . The good news is: AIAgent handles assembling all that for you. An available constructor parameter on AIAgent is systemPrompt , where you supply that initial message. Knosh assembles this from multiple user-supplied sources: AGENTS.md files prompt commandTo send a user message to the LLM, you simply call run on the AIAgent , and it attaches that to the system prompt and sends the whole thing to the LLM. For a one-shot single-turn agent harness like Knosh, you just get back the LLM's response. For a multi-turn agent, AIAgent will keep track of the LLM's response and add it to the growing transcript that represents the overall prompt. If the LLM's response represents a tool call, AIAgent will invoke the tool, attach that result to the overall prompt, and send the whole thing back to the LLM. However, this leads to two challenges: If the first user message is supplied to run , how does AIAgent get further messages from the user? What happens if the prompt gets large? Knosh neatly avoids those by having only one turn. This is good, because while dealing the large prompts via Koog's history compression https://docs.koog.ai/history-compression/ is reasonable, Koog's callback-centric approach to future user messages is awkward IMHO. I might talk about that awkwardness in some future issue. One way to avoid that awkwardness is to not use AIAgent , and instead use PromptExecutor . Turn management becomes straightforward: execute to send a prompt to the LLM, and the return value is what the LLM sent back execute to send the revised prompt to the LLM, and the return value is what the LLM sent backKoog has a prompt DSL to assemble prompts from various pieces system prompts, user messages, assistant responses . You wind up with a Prompt that represents the whole exchange between user and assistant, which you pass to execute : val response = promptExecutor.execute prompt = prompt "demo" { user "Summarize this." }, model = OpenAIModels.Chat.GPT4o the above code snippet comes courtesy of the Koog documentation https://docs.koog.ai/prompts/prompt-executors/ running-a-prompt What you lose is tool call support. Technically speaking, you can blend that in — AIAgent does that, using an underlying PromptExecutor — but that's going to be a fair bit of work. In fact, more details about "that fair bit of work" is what I plan to talk about next week: how you set up tools and how they tie into this whole prompt thing.