New advancements in Generative AI

If you spent last year fine-tuning LLMs just to get a JSON payload back without markdown backticks wrapping the entire thing, you probably noticed the goalposts moved.

We are past the era where generative AI is just a chatbot API you paste into a React app. The tooling has shifted toward agentic workflows, local execution, and structured inputs that actually respect your schemas. If you haven't looked at the ecosystem in the last six months, your mental model is likely outdated.

Here is what actually matters right now for a working developer, minus the hype cycle.

Remember writing regex to scrape a markdown block out of a gpt-3.5

response because the model ignored your system prompt about raw JSON? That was exhausting.

The biggest quiet win in recent tooling is native structured outputs. Major providers and open-source runtimes now let you pass a JSON schema directly to the inference endpoint. The model's token selection is constrained at the logit level so it literally cannot output invalid data.

Here is what this looks like using the modern OpenAI SDK with Pydantic. If the model tries to return a string where an integer belongs, the API errors out before it even hits your network layer.

import os
from openai import OpenAI
from pydantic import BaseModel, Field

client = OpenAI(api_key=os.environ.get("OPENAI_API_KEY"))

class CodeReview(BaseModel):
    summary: str = Field(description="One sentence summary of the code quality")
    bug_count: int = Field(description="Number of bugs found")
    refactor_suggestions: list[str] = Field(description="List of specific improvements")

completion = client.beta.chat.completions.parse(
    model="gpt-4o-2024-08-06",
    messages=[
        {"role": "system", "content": "You are a senior code reviewer."},
        {"role": "user", "content": "Review this: `const x = eval(userInput);`"}
    ],
    response_format=CodeReview,
)

review = completion.choices.message.parsed
print(f"Bugs found: {review.bug_count}")
print(review.refactor_suggestions)

The trip-up here: if you're using older open-source models via Ollama or vLLM, you still have to pass grammar files (like GBNF) or rely on library-level constraints like Instructor. Don't assume response_format={"type": "json_object"}

guarantees your schema fields exist. It just guarantees valid JSON. Always use the Pydantic parsing features if you want actual schema compliance.

Running models locally used to mean watching your fans spin at maximum velocity while a 7B parameter model took forty seconds to explain a stack trace.

That has changed. With the proliferation of quantized formats like GGUF and engines like Ollama and llama.cpp, running models like Llama 3.1 8B or Mistral 7B on an Apple Silicon Mac or a decent consumer GPU is genuinely fast. For many CRUD-adjacent tasks—classification, text extraction, simple entity recognition—you don't need to ship user data to a third-party API anymore.

Here is a quick Node.js script using the standard fetch

API to hit a local Ollama instance running Llama 3.1:

async function summarizeLocally(text) {
  const response = await fetch('http://localhost:11434/api/generate', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({
      model: 'llama3.1',
      prompt: `Summarize this error log in one sentence: ${text}`,
      stream: false
    })
  });

  if (!response.ok) {
    throw new Error(`Local inference failed: ${response.statusText}`);
  }

  const data = await response.json();
  return data.response;
}

summarizeLocally("TypeError: Cannot read properties of undefined (reading 'map') at UserList.jsx:42")
  .then(console.log)
  .catch(console.error);

The catch: Context windows and instruction following on smaller local models are still brittle. If your prompt relies on complex multi-step reasoning, an 8B model will hallucinate steps that a larger frontier model handles effortlessly. Match the model size to the actual complexity of the task, not your desire to keep everything on your local machine.

Six months ago, people were building agents using massive, opinionated frameworks that abstracted everything away behind twelve layers of classes. Half the time, you spent more time debugging the framework's state machine than getting the AI to do anything useful.

The trend now is raw, simple agentic loops. An agent is essentially just a while

loop that calls an LLM, checks if the model wants to call a tool, executes that tool, and feeds the result back into the context.

You don't need a heavy abstraction for this. You just need function calling and basic control flow.

import json
import requests

def get_current_weather(location: str):
    return json.dumps({"location": location, "temperature": "72", "unit": "fahrenheit"})

available_tools = {
    "get_current_weather": get_current_weather
}

def run_agent_loop(initial_prompt):
    messages = [{"role": "user", "content": initial_prompt}]

    for _ in range(5): # Hard limit to prevent infinite loops
        response = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=messages,
            tools=[{
                "type": "function",
                "function": {
                    "name": "get_current_weather",
                    "parameters": {
                        "type": "object",
                        "properties": {"location": {"type": "string"}},
                        "required": ["location"]
                    }
                }
            }]
        )

        response_message = response.choices[0].message
        messages.append(response_message)

        if not response_message.tool_calls:
            return response_message.content

        for tool_call in response_message.tool_calls:
            function_name = tool_call.function.name
            function_to_call = available_tools[function_name]
            function_args = json.loads(tool_call.function.arguments)

            tool_output = function_to_call(**function_args)

            messages.append({
                "tool_call_id": tool_call.id,
                "role": "tool",
                "name": function_name,
                "content": tool_output,
            })

The classic gotcha here is token bloat. As the loop iterates, the history grows. If your tool returns a massive JSON payload or a 500-line log file, your context window fills up instantly, your API costs spike, and the model starts losing the plot. Always truncate or summarize tool outputs before shoving them back into the message array.

Pick one part of your current stack that involves messy text parsing, manual categorization, or repetitive data extraction. Spin up a local Ollama instance or grab an API key, write a 30-line script using structured outputs to solve it, and see where the model actually fails.