New advancements in Generative AI

A developer highlights the shift in generative AI tooling toward agentic workflows, local execution, and structured outputs. The post demonstrates how native structured outputs with JSON schemas and local models like Llama 3.1 via Ollama are now practical for many tasks, reducing reliance on third-party APIs.

If you spent last year fine-tuning LLMs just to get a JSON payload back without markdown backticks wrapping the entire thing, you probably noticed the goalposts moved. We are past the era where generative AI is just a chatbot API you paste into a React app. The tooling has shifted toward agentic workflows, local execution, and structured inputs that actually respect your schemas. If you haven't looked at the ecosystem in the last six months, your mental model is likely outdated. Here is what actually matters right now for a working developer, minus the hype cycle. Remember writing regex to scrape a markdown block out of a gpt-3.5 response because the model ignored your system prompt about raw JSON? That was exhausting. The biggest quiet win in recent tooling is native structured outputs. Major providers and open-source runtimes now let you pass a JSON schema directly to the inference endpoint. The model's token selection is constrained at the logit level so it literally cannot output invalid data. Here is what this looks like using the modern OpenAI SDK with Pydantic. If the model tries to return a string where an integer belongs, the API errors out before it even hits your network layer. python import os from openai import OpenAI from pydantic import BaseModel, Field client = OpenAI api key=os.environ.get "OPENAI API KEY" class CodeReview BaseModel : summary: str = Field description="One sentence summary of the code quality" bug count: int = Field description="Number of bugs found" refactor suggestions: list str = Field description="List of specific improvements" completion = client.beta.chat.completions.parse model="gpt-4o-2024-08-06", messages= {"role": "system", "content": "You are a senior code reviewer."}, {"role": "user", "content": "Review this: const x = eval userInput ; "} , response format=CodeReview, review = completion.choices.message.parsed print f"Bugs found: {review.bug count}" print review.refactor suggestions The trip-up here: if you're using older open-source models via Ollama or vLLM, you still have to pass grammar files like GBNF or rely on library-level constraints like Instructor. Don't assume response format={"type": "json object"} guarantees your schema fields exist. It just guarantees valid JSON. Always use the Pydantic parsing features if you want actual schema compliance. Running models locally used to mean watching your fans spin at maximum velocity while a 7B parameter model took forty seconds to explain a stack trace. That has changed. With the proliferation of quantized formats like GGUF and engines like Ollama and llama.cpp, running models like Llama 3.1 8B or Mistral 7B on an Apple Silicon Mac or a decent consumer GPU is genuinely fast. For many CRUD-adjacent tasks—classification, text extraction, simple entity recognition—you don't need to ship user data to a third-party API anymore. Here is a quick Node.js script using the standard fetch API to hit a local Ollama instance running Llama 3.1: js async function summarizeLocally text { const response = await fetch 'http://localhost:11434/api/generate', { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify { model: 'llama3.1', prompt: Summarize this error log in one sentence: ${text} , stream: false } } ; if response.ok { throw new Error Local inference failed: ${response.statusText} ; } const data = await response.json ; return data.response; } summarizeLocally "TypeError: Cannot read properties of undefined reading 'map' at UserList.jsx:42" .then console.log .catch console.error ; The catch: Context windows and instruction following on smaller local models are still brittle. If your prompt relies on complex multi-step reasoning, an 8B model will hallucinate steps that a larger frontier model handles effortlessly. Match the model size to the actual complexity of the task, not your desire to keep everything on your local machine. Six months ago, people were building agents using massive, opinionated frameworks that abstracted everything away behind twelve layers of classes. Half the time, you spent more time debugging the framework's state machine than getting the AI to do anything useful. The trend now is raw, simple agentic loops. An agent is essentially just a while loop that calls an LLM, checks if the model wants to call a tool, executes that tool, and feeds the result back into the context. You don't need a heavy abstraction for this. You just need function calling and basic control flow. python import json import requests def get current weather location: str : Stub for an actual weather API call return json.dumps {"location": location, "temperature": "72", "unit": "fahrenheit"} available tools = { "get current weather": get current weather } The actual agent loop is remarkably dumb and simple def run agent loop initial prompt : messages = {"role": "user", "content": initial prompt} for in range 5 : Hard limit to prevent infinite loops response = client.chat.completions.create model="gpt-4o-mini", messages=messages, tools= { "type": "function", "function": { "name": "get current weather", "parameters": { "type": "object", "properties": {"location": {"type": "string"}}, "required": "location" } } } response message = response.choices 0 .message messages.append response message if not response message.tool calls: return response message.content for tool call in response message.tool calls: function name = tool call.function.name function to call = available tools function name function args = json.loads tool call.function.arguments tool output = function to call function args messages.append { "tool call id": tool call.id, "role": "tool", "name": function name, "content": tool output, } The classic gotcha here is token bloat. As the loop iterates, the history grows. If your tool returns a massive JSON payload or a 500-line log file, your context window fills up instantly, your API costs spike, and the model starts losing the plot. Always truncate or summarize tool outputs before shoving them back into the message array. Pick one part of your current stack that involves messy text parsing, manual categorization, or repetitive data extraction. Spin up a local Ollama instance or grab an API key, write a 30-line script using structured outputs to solve it, and see where the model actually fails.