Orchestrating Parallel Intelligence: Building a Multi-Agent AI Grading System with LangGraph

The AI engineering ecosystem is shifting from monolithic LLM prompts to multi-agent, graph-based architectures. Developers are breaking complex tasks into discrete nodes within stateful directed graphs, using concepts like state management with reducers and parallel execution patterns such as Scatter-Gather to build enterprise-grade AI workflows.

The era of the monolithic, zero-shot Large Language Model LLM prompt is fading. In its place, the AI engineering ecosystem is rapidly adopting multi-agent, graph-based architectures. Building robust AI applications no longer relies on asking an LLM to perform complex, multi-faceted tasks in a single breath. Instead, developers are breaking down complex cognitive tasks into discrete, manageable nodes within a stateful, directed graph. To transition from building simple chatbots to architecting enterprise-grade AI workflows, engineers must master a specific set of theoretical concepts: graph-based execution models, state management with reducers, structured output enforcement, and distributed architectural patterns like Scatter-Gather . At the heart of modern LLM orchestration frameworks is the application of graph theory — specifically, Directed Graphs. Rather than executing a simple linear script, tasks are mapped out as a mathematical graph comprising Nodes and Edges . In an LLM graph, a node represents a specific functional execution. A node is entirely ignorant of the broader application; its only job is to receive a specific input, perform a transformation usually via an LLM inference call or a deterministic Python function , and return an output. By isolating work into nodes, we achieve prompt isolation . Instead of forcing an LLM to balance three different persona instructions in one massive prompt, which often leads to attention dilution and hallucination, a node contains a singular, hyper-focused prompt. Edges are the directional pathways that connect nodes. They dictate the flow of execution. Edges can be: This graph-based approach allows for acyclic execution Directed Acyclic Graphs where data flows from start to finish without loops, as well as cyclic execution, which is the foundational theory behind autonomous AI agents that must “think, act, observe, and repeat” until a condition is met. If nodes are isolated workers, they need a secure, predictable way to share information. This is solved through a centralized State . In state-driven LLM frameworks, the graph maintains a global state object often defined by a rigid schema . Every time a node executes, it reads from this state, performs its task, and returns a state update. The framework then merges this update into the global state before passing it to the next node. To prevent chaotic execution, the state must be rigidly typed. By defining the exact structure of the state e.g., using Python TypedDict or data classes , the application ensures that every node knows exactly what data is available and what data it is expected to output. If a node requires a text input string and a historical context list, the strict schema guarantees these variables are present, preventing runtime crashes during expensive LLM calls. One of the most powerful features of graph-based orchestration is parallel execution. If three nodes do not depend on each other’s outputs, they can be executed simultaneously. However, this introduces a classic computer science problem: State Collisions Race Conditions . Imagine a state schema that holds a list of evaluation scores. If three independent LLM nodes execute in parallel and all try to write a new score to the state at the exact same millisecond, how does the framework know how to handle the update? Without intervention, the last node to finish would simply overwrite the work of the other two. This is solved using Reducers . A reducer is a function that specifies exactly how concurrent state updates should be merged. Instead of a simple overwrite operation, variables within the state are annotated with specific reduction logic. The most common reducer is the add or append operator. When the framework receives state updates from three parallel nodes simultaneously, the reducer intercepts them and gracefully appends all three outputs into a single, unified list. Update TypeMechanismTheoretical Use CaseOverwrite Default Replaces the old value with the new value.Updating a boolean flag or a final summary string. Append/Add Reducer Concatenates or mathematically adds new values to existing ones.Collecting parallel scores, aggregating multiple feedback strings, or maintaining a conversation history. Reducers are what make parallel LLM execution mathematically safe and programmatically predictable. LLMs are inherently stochastic, probabilistic text generators. By default, they output unstructured natural language. This is fundamentally incompatible with programmatic workflows that require specific data types like integers for scoring or specific JSON structures for database insertion . Historically, engineers used prompt engineering to beg the LLM to output valid JSON e.g., “Return ONLY valid JSON and no other text” . This was brittle and prone to parsing errors when the LLM inevitably included a conversational prefix like “Here is your JSON:” . Modern frameworks solve this through Structured Outputs , leveraging underlying LLM function-calling or tool-calling capabilities. By passing a strict schema such as a Pydantic model directly into the API request, the LLM is constrained at the inference level. The schema acts as a structural contract: When an LLM is wrapped in a structured output parser, the application code no longer deals with raw, unpredictable strings. It receives cleanly parsed, strongly typed objects, bridging the gap between natural language reasoning and traditional deterministic programming. The culmination of state graphs, reducers, and structured outputs enables a highly advanced distributed system pattern known as Scatter-Gather or Fan-Out / Fan-In . When a complex analytical task is required, forcing a single LLM to analyze multiple distinct criteria sequentially requires a massive context window and dilutes the model’s attention mechanism. In the Fan-Out phase, the control flow splits. A single initial state is broadcast scattered to multiple independent nodes simultaneously. Each node is given a narrow, specialized objective. Because these nodes operate independently, they do not block one another. They run concurrently, utilizing asynchronous network requests. Once the parallel nodes complete their isolated tasks, the graph control flow converges. The framework utilizes the predefined reducers to safely merge the disparate outputs gathering them into a single, updated state object. Finally, a convergence node the Fan-In node executes. It takes the newly merged, comprehensive state — which now contains all the granular insights from the parallel nodes — and synthesizes it into a final output. The provided code uses LangGraph to orchestrate this exact “Fan-Out / Fan-In” architecture. We start with a user’s essay, fan out to three separate evaluators Language, Analysis, Thought working in parallel, and then fan in to a final evaluator to aggregate the results. Here is the cell-by-cell breakdown of the code in Microsoft Fabric Notebook https://www.microsoft.com/en-in/microsoft-fabric . pip install -q langgraph langchain-groq python-dotenv typing-extensions Import necessary librariesfrom langgraph.graph import StateGraph, START, ENDfrom langchain groq import ChatGroqfrom typing import TypedDict, Annotatedfrom pydantic import BaseModel, Fieldimport operatorfrom IPython.display import Image Initialize the Groq modelmodel = ChatGroq model='llama-3.3-70b-versatile', temperature=0 First, we install our core dependencies. langgraph is the primary framework we use to build cyclical, stateful agent architectures. langchain-groq is the integration package connecting us to Groq’s cloud infrastructure. Groq uses specialized LPU Language Processing Unit hardware to serve LLMs at blazing-fast speeds, which is essential for multi-agent workflows where multiple LLM calls happen simultaneously. pydantic is a data validation library for Python. We import BaseModel and Field from Pydantic, which are critical for forcing the LLM to reply in a strict data format rather than rambling text. We import TypedDict and Annotated to structure our memory, and operator to help us build our Reducer logic. Finally, we instantiate our “ Brain ”: the llama-3.3–70b-versatile model via Groq. Crucially, we set the temperature=0 because we want highly deterministic, analytical grading, not creative fiction. class EvaluationSchema BaseModel : feedback: str = Field description='Detailed feedback for the essay' score: int = Field description='Score out of 10', ge=0, le=10 structured model = model.with structured output EvaluationSchema class UPSCState TypedDict : essay: str language feedback: str analysis feedback: str clarity feedback: str overall feedback: str individual scores: Annotated list int , operator.add avg score: floatdef evaluate language state: UPSCState : prompt = f'Evaluate the language quality of the following essay and provide a feedback and assign a score out of 10 \n {state "essay" }' output = structured model.invoke prompt return {'language feedback': output.feedback, 'individual scores': output.score }def evaluate analysis state: UPSCState : prompt = f'Evaluate the depth of analysis of the following essay and provide a feedback and assign a score out of 10 \n {state "essay" }' output = structured model.invoke prompt return {'analysis feedback': output.feedback, 'individual scores': output.score }def evaluate thought state: UPSCState : prompt = f'Evaluate the clarity of thought of the following essay and provide a feedback and assign a score out of 10 \n {state "essay" }' output = structured model.invoke prompt return {'clarity feedback': output.feedback, 'individual scores': output.score }def final evaluation state: UPSCState : summary feedback prompt = f'Based on the following feedbacks create a summarized feedback \n language feedback - {state "language feedback" } \n depth of analysis feedback - {state "analysis feedback" } \n clarity of thought feedback - {state "clarity feedback" }' We use the standard model here not the structured one to get raw text back overall feedback = model.invoke prompt .content avg calculate avg score = sum state 'individual scores' / len state 'individual scores' return {'overall feedback': overall feedback, 'avg score': avg score} This cell is the architectural marvel of our application. It establishes how our agents think, remember, and act. Enforcing Structured Output: LLMs naturally output raw, unpredictable text. If we want to calculate an average score later, we cannot have the AI output a string like ”I give this essay an eight out of ten.” We need standard data types. By defining EvaluationSchema with Pydantic, we instruct the LLM to output a precise JSON object containing exactly two keys: a string called feedback and an integer called score strictly bounded between 0 and 10 . We then create a structured model which is a specialized version of our LLM physically bound to this strict schema. The Agentic Memory State : This is our agent’s Short-Term Memory scratchpad. We define exactly what our graph will remember. The UPSCState dictates that our application will keep track of the original essay, the feedback strings from our different agents, and the calculated average score. Pay close attention to individual scores . Because our three evaluators will run in parallel, if they all try to write a score to a standard Python list, they will overwrite each other due to race conditions. By using Annotated with operator.add , we create a Reducer . We are explicitly telling the state: Whenever a node provides a new list of scores, do not overwrite the existing list; append the new items to it . The Worker Nodes: These three functions represent our specialized, narrow-focus agents. Each one reads the target essay from the state, constructs a highly specific prompt aimed at its unique domain language, analysis, or clarity , and invokes the structured model . They then return a dictionary updating their specific feedback column in the state. Importantly, they wrap their integer score in a list output.score , which the state’s reducer will safely add to the main individual scores pool. The Manager Node: Once the parallel tasks are complete, this manager node takes over. It reads the specific feedback generated by the three subordinate workers and weaves them into a comprehensive summary. Notice that it uses the standard model.invoke instead of the structured one, because here we want rich, unstructured text. Finally, it leverages standard Python math to calculate the avg score from the successfully populated individual scores list, updating the state one last time. Build the Graphgraph = StateGraph UPSCState Nodesgraph.add node 'evaluate language', evaluate language graph.add node 'evaluate analysis', evaluate analysis graph.add node 'evaluate thought', evaluate thought graph.add node 'final evaluation', final evaluation Edgesgraph.add edge START, 'evaluate language' graph.add edge START, 'evaluate analysis' graph.add edge START, 'evaluate thought' graph.add edge 'evaluate language', 'final evaluation' graph.add edge 'evaluate analysis', 'final evaluation' graph.add edge 'evaluate thought', 'final evaluation' graph.add edge 'final evaluation', END Compile the graphworkflow = graph.compile Display the graphImage workflow.get graph .draw mermaid png We have our workers nodes and our memory state , but right now, they are just isolated functions. This cell acts as the choreographer. Binding Memory: We initialize the StateGraph and physically bind our UPSCState memory structure to it. Every node in this graph will now share this exact scratchpad. Registering Nodes: We use add node to tell LangGraph about our Python functions. Drawing Edges The Magic of Parallelism : The magic happens in the edges. Notice how START connects to evaluate language , evaluate analysis , and evaluate thought simultaneously?. This specific topology tells LangGraph to execute all three workers in parallel . The Fan-In: We then draw edges from all three of those evaluators down to final evaluation . LangGraph’s backend engine is smart enough to know that final evaluation must wait until all three incoming edge requirements have resolved before it triggers. Compilation: Finally, workflow.compile fuses these rules into an executable application, and we generate a Mermaid diagram to visually verify our topology. {'essay': "India in the Age of AI\nAs the world enters a transformative era defined by artificial intelligence AI , India stands at a critical juncture — one where it can either emerge as a global leader in AI innovation or risk falling behind in the technology race. The age of AI brings with it immense promise as well as unprecedented challenges, and how India navigates this landscape will shape its socio-economic and geopolitical future.\n\nIndia's strengths in the AI domain are rooted in its vast pool of skilled engineers, a thriving IT industry, and a growing startup ecosystem. With over 5 million STEM graduates annually and a burgeoning base of AI researchers, India possesses the intellectual capital required to build cutting-edge AI systems. Institutions like IITs, IIITs, and IISc have begun fostering AI research, while private players such as TCS, Infosys, and Wipro are integrating AI into their global services. In 2020, the government launched the National AI Strategy AI for All with a focus on inclusive growth, aiming to leverage AI in healthcare, agriculture, education, and smart mobility.\n\nOne of the most promising applications of AI in India lies in agriculture, where predictive analytics can guide farmers on optimal sowing times, weather forecasts, and pest control. In healthcare, AI-powered diagnostics can help address India’s doctor-patient ratio crisis, particularly in rural areas. Educational platforms are increasingly using AI to personalize learning paths, while smart governance tools are helping improve public service delivery and fraud detection.\n\nHowever, the path to AI-led growth is riddled with challenges. Chief among them is the digital divide. While metropolitan cities may embrace AI-driven solutions, rural India continues to struggle with basic internet access and digital literacy. The risk of job displacement due to automation also looms large, especially for low-skilled workers. Without effective skilling and re-skilling programs, AI could exacerbate existing socio-economic inequalities.\n\nAnother pressing concern is data privacy and ethics. As AI systems rely heavily on vast datasets, ensuring that personal data is used transparently and responsibly becomes vital. India is still shaping its data protection laws, and in the absence of a strong regulatory framework, AI systems may risk misuse or bias.\n\nTo harness AI responsibly, India must adopt a multi-stakeholder approach involving the government, academia, industry, and civil society. Policies should promote open datasets, encourage responsible innovation, and ensure ethical AI practices. There is also a need for international collaboration, particularly with countries leading in AI research, to gain strategic advantage and ensure interoperability in global systems.\n\nIndia’s demographic dividend, when paired with responsible AI adoption, can unlock massive economic growth, improve governance, and uplift marginalized communities. But this vision will only materialize if AI is seen not merely as a tool for automation, but as an enabler of human-centered development.\n\nIn conclusion, India in the age of AI is a story in the making — one of opportunity, responsibility, and transformation. The decisions we make today will not just determine India’s AI trajectory, but also its future as an inclusive, equitable, and innovation-driven society.", 'language feedback': "The essay demonstrates a high level of language quality, with clear and concise writing, effective use of transitions, and proper grammar and spelling. The author provides a well-structured argument, supported by relevant examples and evidence. The use of technical terms, such as 'predictive analytics' and 'AI-powered diagnostics', is accurate and appropriate. However, there are some areas for improvement, such as varying sentence structure and using more nuanced vocabulary. Additionally, some paragraphs could be more tightly focused and condensed. Overall, the essay is well-written and effectively conveys the author's message.", 'analysis feedback': "The essay provides a comprehensive analysis of India's position in the age of AI, highlighting both the opportunities and challenges. The writer demonstrates a good understanding of the subject matter, citing relevant examples and statistics to support their arguments. The essay is well-structured, with a clear introduction, body, and conclusion. However, the analysis could be deeper, with more nuanced discussions of the complexities involved. Additionally, the writer could have provided more concrete policy recommendations and explored the potential risks and unintended consequences of AI adoption. Overall, the essay demonstrates a good level of critical thinking and writing skills, but could benefit from more rigorous analysis and attention to detail.", 'clarity feedback': "The essay demonstrates a clear and well-structured thought process, effectively exploring India's potential in the age of AI. The writer provides a balanced view of the opportunities and challenges, highlighting the country's strengths and weaknesses. The use of specific examples, such as the application of AI in agriculture and healthcare, adds depth to the discussion. However, some points, like the digital divide and data privacy concerns, could be further elaborated. The conclusion effectively summarizes the main ideas and emphasizes the importance of responsible AI adoption. Overall, the essay showcases a strong clarity of thought, but could benefit from more nuanced analysis and supporting evidence.", 'overall feedback': "Here is a summarized feedback:\n\nThe essay is well-written, effectively conveying the author's message with clear and concise language, proper grammar and spelling, and a well-structured argument. The author demonstrates a good understanding of the subject matter, providing a comprehensive analysis of India's position in the age of AI, and showcasing a clear and well-structured thought process. However, there are areas for improvement, including:\n\n Varying sentence structure and using more nuanced vocabulary\n Providing more concrete policy recommendations and exploring potential risks and unintended consequences of AI adoption\n Elaborating on certain points, such as the digital divide and data privacy concerns\n Conducting more rigorous analysis and attention to detail to deepen the discussion\n Condensing and focusing some paragraphs for better clarity\n\nOverall, the essay demonstrates strong language quality, a good level of critical thinking and writing skills, and a clear thought process, but could benefit from more nuanced analysis, supporting evidence, and attention to detail to take it to the next level.", 'individual scores': 8, 9, 8 , 'avg score': 8.333333333333334} Testing the workflowessay = """India in the Age of AIAs the world enters a transformative era defined by artificial intelligence AI , India stands at a critical juncture — one where it can either emerge as a global leader in AI innovation or risk falling behind in the technology race. The age of AI brings with it immense promise as well as unprecedented challenges, and how India navigates this landscape will shape its socio-economic and geopolitical future.India's strengths in the AI domain are rooted in its vast pool of skilled engineers, a thriving IT industry, and a growing startup ecosystem. With over 5 million STEM graduates annually and a burgeoning base of AI researchers, India possesses the intellectual capital required to build cutting-edge AI systems. Institutions like IITs, IIITs, and IISc have begun fostering AI research, while private players such as TCS, Infosys, and Wipro are integrating AI into their global services. In 2020, the government launched the National AI Strategy AI for All with a focus on inclusive growth, aiming to leverage AI in healthcare, agriculture, education, and smart mobility.One of the most promising applications of AI in India lies in agriculture, where predictive analytics can guide farmers on optimal sowing times, weather forecasts, and pest control. In healthcare, AI-powered diagnostics can help address India’s doctor-patient ratio crisis, particularly in rural areas. Educational platforms are increasingly using AI to personalize learning paths, while smart governance tools are helping improve public service delivery and fraud detection.However, the path to AI-led growth is riddled with challenges. Chief among them is the digital divide. While metropolitan cities may embrace AI-driven solutions, rural India continues to struggle with basic internet access and digital literacy. The risk of job displacement due to automation also looms large, especially for low-skilled workers. Without effective skilling and re-skilling programs, AI could exacerbate existing socio-economic inequalities.Another pressing concern is data privacy and ethics. As AI systems rely heavily on vast datasets, ensuring that personal data is used transparently and responsibly becomes vital. India is still shaping its data protection laws, and in the absence of a strong regulatory framework, AI systems may risk misuse or bias.To harness AI responsibly, India must adopt a multi-stakeholder approach involving the government, academia, industry, and civil society. Policies should promote open datasets, encourage responsible innovation, and ensure ethical AI practices. There is also a need for international collaboration, particularly with countries leading in AI research, to gain strategic advantage and ensure interoperability in global systems.India’s demographic dividend, when paired with responsible AI adoption, can unlock massive economic growth, improve governance, and uplift marginalized communities. But this vision will only materialize if AI is seen not merely as a tool for automation, but as an enabler of human-centered development.In conclusion, India in the age of AI is a story in the making — one of opportunity, responsibility, and transformation. The decisions we make today will not just determine India’s AI trajectory, but also its future as an inclusive, equitable, and innovation-driven society."""initial state = { 'essay': essay}final state = workflow.invoke initial state print final state This is the ignition switch. We load the user’s text into the essay key of our initial state dictionary. By calling workflow.invoke initial state , the system springs into action. The graph branches out, querying the Groq LLM three times concurrently for structured feedback, safely aggregating the integer scores into the state via our reducer, and then passes the combined context to the final node for summarization. The resulting output printed to our terminal will be a highly structured, deeply analyzed evaluation. Because we used parallel specialized agents, this process takes a fraction of the time and hallucinates far less than asking a single LLM to perform the entire grading rubric in one massive prompt. We are no longer programming software by writing explicit lines of hardcoded logic; we are orchestrating intelligence by defining goals, providing toolsets, and designing cognitive architectures. The transition from standard prompting to agentic graphs is an evolution from software as a tool to software as an entity. By mastering state management, reducers, structured JSON outputs, and node-based parallel routing in LangGraph, we are learning the intricate dance of memory and planning required to build the autonomous synthetic workforces of the future. Hey, I am Sandip Palit , from Kolkata, India. I love to explore what’s new in the Data Science space and share it with the community. I am a Fabric Super User , and in this Agentic AI using Microsoft Fabric Playlist, I will share my learnings and hands-on projects on Agentic AI. Thank You for reading this article. Please feel free to share your thoughts in the comments section. Orchestrating Parallel Intelligence: Building a Multi-Agent AI Grading System with LangGraph https://pub.towardsai.net/orchestrating-parallel-intelligence-building-a-multi-agent-ai-grading-system-with-langgraph-9468daf9dd29 was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.