Thursdays with Koog: Status and Metrics

Knosh 0.3.0, an open-source coding agent built on Koog's AIAgent framework, now displays a Braille-dot loading animation that advances with each tool call and status event, improving user feedback during slow local LLM responses. The update leverages Koog's StatusFeature and MetricsFeature to track agent activity, addressing the poor UX of waiting for locally-hosted models like Ollama.

As we have seen in previous "Thursdays with Koog" issues https://pac.commonsware.com/archive/ , Knosh https://knosh.commonsware.com uses Koog's https://docs.koog.ai/ AIAgent to send a prompt and get a final response, with the AIAgent framework handling tool calls. However, especially for a local Ollama model, getting a response may take some time. Not only are locally-hosted LLMs usually slower than top-tier frontier models, Ollama might have to load the model into memory, if it has not been used in a while. So, we execute our prompt and we wait and we wait and we wait and we wait and we wait and we wait and we finally get a response. This makes for a fairly lousy UX. Many TUI coding agent harnesses do something to indicate that the harness is in communication with the LLM and is not just stuck. One popular pattern is to use Braille dots to simulate a loading state https://github.com/edisoncks/cli-loading-indicator/ . I wanted to add that to Knosh, with the caveat that I wanted the animation to advance as messages went back and forth between Knosh and the LLM. Just having the animation animate based on the passage of time would provide limited benefit for the user. Sometimes, you can get a coding agent harness to display the "thinking" messages. As a user, you tend to think of your interaction with the agent as being a message that you send out "make the button 50% larger and a slightly darker shade of purple" and a message that you get back. Sometimes, you get multiple messages displayed before control fully returns to you, with thinking periods interspersed. What is really happening during those thinking periods is that the LLM is talking to itself. One of the techniques used to get better results out of LLMs is encouraging them to talk through what they are doing. Their generated output then becomes input in an updated prompt, and that helps steer them towards better solutions. This is especially true in planning situations, as a plan will evolve over many such "thinking" exchanges. This approach was not part of the original APIs created by OpenAI and others. To accommodate this, especially with arbitrary LLM clients, they added notation to indicate "thinking" messages not designed to be shown to users , much in the way that there is notation to identify tool call requests. And, they added "streaming" mode as a way of getting many messages back from a single REST call, with the call ending when the LLM indicates that it is done with its turn and control returns to the user. Koog supports both simple request-and-response and streaming modes. My original vision was to have Knosh use the streaming mode, and update the Braille dots animation with each message. That was going to be just complicated enough that I elected to skip that and stick with the simple request-and-response approach. Besides, not all models support streaming, especially in the local-model space, so I did not want to rely upon it. What I could do, though, was find out about tool calls. The Koog API supports the notion of "installing" features into the AIAgent . These features hook into the internal pipeline that AIAgent uses for processing inbound and outbound messages. You get an installFeatures DSL when you construct the AIAgent as a trailing lambda expression, and in there you can call install to install these optional features: return AIAgent // lots of parameters go here { if onMetrics = null { install MetricsFeature.Feature { callback = onMetrics } } if onStatus = null { install StatusFeature.Feature { callback = onStatus } } } } from Knosh 0.3.0 https://codeberg.org/commonsguy/knosh/src/tag/0.3.0/lib/knosh-agents/src/main/kotlin/com/commonsware/knosh/agents/AIAgentFactory.kt L315-L354 The StatusFeature lets you register a callback for AgentStatus events. Specifically, you can find out about Starting , CallingModel , and RunningTool events. The Knosh CLI uses that to update a StatusLine , advancing the Braille-dots animation on each status change, while also printing a message about what the status change is: / Returns the human-readable label shown on the status line for this AgentStatus . The label carries no spinner frame. @return the label text @receiver the status to describe / internal fun AgentStatus.label : String = when this { AgentStatus.Starting - "starting…" AgentStatus.CallingModel - "calling model…" is AgentStatus.RunningTool - "running tool: $toolName" } / A single-line, in-place terminal status indicator rendered to stderr. Each call to update advances an 8-frame Braille spinner by one position cycling back to the start after the eighth frame and rewrites the line as a carriage return, the frame, the label, then a clear-to-end-of-line sequence, with no trailing newline, so successive updates overwrite one another. clear erases the line. The spinner advances independently of the label: the frame is purely a function of how many updates have occurred. Rendering is dependency-free raw ANSI; callers must construct a StatusLine only when stderr is an interactive terminal. @param write the sink that receives raw output fragments; defaults to writing to and flushing System.err / internal class StatusLine private val write: String - Unit = { fragment - System.err.print fragment System.err.flush } { private var frameIndex = 0 / Advances the spinner by one frame and rewrites the status line to show status . @param status the current agent status to display / fun update status: AgentStatus { val frame = SPINNER FRAMES frameIndex % SPINNER FRAMES.size frameIndex++ write "\r$frame ${status.label }$CLEAR TO END OF LINE" } / Erases the status line, leaving the cursor at the start of an empty line. / fun clear { write "\r$CLEAR TO END OF LINE" } } from Knosh 0.3.0 https://codeberg.org/commonsguy/knosh/src/tag/0.3.0/knosh-cli/src/main/kotlin/com/commonsware/knosh/app/StatusLine.kt The other feature that Knosh uses is MetricsFeature . This registers a callback to be invoked when the full request-and-response has completed. You can find out: The first two can be tracked directly by Koog. The latter two require the LLM to supply that information as part of the REST API, and that is not always supported. If we get that data, Knosh will include it in the output, so you get a sense for how expensive that particular request was. If you pull up Knosh in your favorite IDE and try looking up this Feature system, you will see that it appears to be tied to a GraphAIAgent , a specific implementation of the AIAgent abstract class. Next week, we will learn a bit about Koog's strategy system and what strategy Knosh employs.