Generative UI is the next evolution of AI interaction, moving beyond text to create dynamic, agent-driven user interfaces. We explored three key patterns - Static (AG-UI), Declarative (A2UI), and Open-ended UI - and used the CopilotKit framework to build two demos: a Tic-Tac-Toe assistant, and a 2D RPG with LangGraph NPC.
Today’s AI agents primarily communicate through text. While revolutionary, this often forces humans to interact with a conversational interface, even for tasks that would be more efficient or intuitive within a traditional graphical user interface.
Generative UI is the paradigm shift that enables AI agents to dynamically generate the UI elements required to complete a task. Instead of the user describing what they see, the agent reads the UI state directly and acts through it - clicking buttons, moving characters, proposing decisions.
This shift blends the power of LLMs with familiar interactive components and is critical for implementing human-in-the-loop workflows, where the agent can propose an action but requires human confirmation via an interactive UI before execution. This ensures control and a better user experience.
Three core patterns exist for building Generative UI:
The different patterns are not mutually exclusive and can be complementary, e.g. both AG-UI and A2UI are used by the CopilotKit framework for implementing Generative UI.
The Generative UI approach is seeing rapid development from several frameworks:
We picked CopilotKit for building our two demos. It is an open-source (MIT) framework with 30k+ stars on GitHub that connects AI agents to your frontend. It is the company behind the AG-UI protocol.
Its core concepts are:
It supports React/Next.js and Angular, and works with agent backends like LangGraph or Microsoft Agent Framework. CopilotKit is partnered with Microsoft and Google for development on the AG-UI and A2UI protocols.
To gain hands-on experience with Generative UI, we have developed two demos with CopilotKit: a Tic-Tac-Toe Assistant and a 2D RPG. They demonstrate how a front-end application can expose its state and functionality (tools) to an agent, allowing the agent to perform actions that are visible and interactive for the user.
In our first demo we built a tic-tac-toe assistant that coaches the player at the immensely sophisticated game of tic-tac-toe. The agent can suggest moves (with an accept/reject dialog for human-in-the-loop), annotate the board with threats and opportunities, and render a custom "Coach Card" via generated HTML. The demo illustrates key Generative UI features by applying the patterns discussed above:
proposeNextMove
tool, and the result is rendered as an "Accept / Reject" dialog (renderAndWaitForResponse
), requiring user confirmation before the move is made.analyzeBoard
tool, which returns data used to render overlay symbols (threats, opportunities) directly onto the existing UI components (Board.tsx
/Square.tsx
).renderCoachCard
tool, which generates fully custom HTML for the "Coach Card" component, rendered using a secure mechanism (dangerouslySetInnerHTML
with DOMPurify).useAgentContext
hook is used to publish the current board state, player, and winner status, giving the agent a "readable state" of the application.The application is structured as a Next.js app, with all frontend logic (tools, UI) in the browser, communicating with the backend API route (/api/copilotkit
) which hosts the CopilotKit’s built-in agent and handles the model interaction (Azure OpenAI gpt-5-mini
via @ai-sdk/azure
).
This demo validated that exposing React state as readables and defining actions as tools is a clean, working pattern. The built-in agent made setup fast - readables and the instructions prop on the chat sidebar just worked.
For the second demo we wanted to go deeper: custom system prompts, richer agent logic, and a more complex scenario. We built a 2D RPG where the player walks around a medieval world collecting bananas, berries, and crystals. An NPC - powered by a LangGraph agent - acts as a quest giver. The agent can: All of this runs through CopilotKit's readables and actions - the agent never touches the DOM directly; it just reads structured state and calls named functions.
Nothing is scripted. The NPC reads the world state, decides what to ask for, and reacts to whatever the player actually does.
The NPC feels more real because we can freely talk to it and it makes decisions on the fly - exactly the kind of dynamic interaction that's hard to achieve with traditional game scripting.
Summary of what we've learned from the demos.
useCopilotReadable
) and do (useCopilotAction
) right next to the React components that own the state. CopilotKit handles serialization, transport, and tool binding. It feels like a natural extension of React's component model - state flows down, actions flow up, and the agent plugs into that loop.useCopilotReadable
values appear in the model's context automatically. With a custom LangGraph agent, they arrive in state.copilotkit.context
- but you have to manually inject them into your system message. Actions worked immediately; readables silently did nothing until we traced the issue.
instructions
prop is ignored with custom agents.Couldn't we just append state to the user's message and define function-calling tools on our own? Yes - but CopilotKit gives you:
For a quick prototype, rolling your own is fine. For anything that will grow - multiple agents, generative UI patterns, shared state - a framework pays off. Generative UI represents an exciting step forward in AI-powered applications, moving beyond simple text generation to dynamic, context-aware user interaction. Our experience with CopilotKit in developing the two demos validated the effectiveness of the AG-UI and open-ended patterns for creating engaging and powerful Human-in-the-Loop experiences. As frameworks mature, Generative UI is poised to become a core part of how developers integrate AI into modern web applications.