# Multi-Model System Design: When One Model Isn't Enough

> Source: <https://www.glukhov.org/llm-architecture/model-routing/multi-model-system-design/>
> Published: 2026-06-15 00:00:00+00:00

# Multi-Model System Design: When One Model Isn't Enough

Pick the simplest pattern that works.

Single-model systems are simple. Multi-model systems are powerful. The challenge isn’t choosing models — it’s designing the architecture that orchestrates them.

A multi-model system isn’t about having more models. It’s about having the right model for the right task at the right time.

## Architecture patterns

Five patterns cover most use cases:

| Pattern | Complexity | When to use | Tradeoff |
|---|---|---|---|
| Single Model | Lowest | Prototyping, simple tasks | Limited capability |
| Sequential | Low | Multi-step workflows | Higher latency |
| Parallel | Medium | Independent tasks | Higher cost |
| Hierarchical | High | Complex reasoning | Complex orchestration |
| Ensemble | Highest | Critical decisions | Highest cost |

Pick the simplest one that works. Complexity is real, and it compounds.

## Sequential architecture

Process tasks through a chain of models, each specializing in a step.

### Pattern 1: Pipeline

Pipeline pattern — each model’s output feeds the next:

``` python
class ModelPipeline:
    def __init__(self):
        self.models = [
            {"model": "qwen2.5-1.5b", "task": "classify"},
            {"model": "qwen2.5-7b", "task": "extract"},
            {"model": "qwen2.5-32b", "task": "reason"},
        ]

    def process(self, input: str) -> str:
        current = input
        for model_config in self.models:
            current = self.call_model(
                model_config["model"],
                self.create_prompt(model_config["task"], current)
            )
        return current
```

Latency adds up. Three models in sequence means three times the latency. Only use this when each step actually needs a different model.

### Pattern 2: Router

Router pattern — classify the task, route to the specialist:

``` python
class ModelRouter:
    def __init__(self):
        self.classifier = "qwen2.5-1.5b"
        self.specialists = {
            "code": "qwen2.5-coder-7b",
            "math": "qwen2.5-32b",
            "creative": "claude-sonnet-4",
            "general": "qwen2.5-7b",
        }

    def route(self, prompt: str) -> str:
        task_type = self.classify(prompt)
        model = self.specialists.get(task_type, self.specialists["general"])
        return self.call_model(model, prompt)
```

The classifier is the weak link. If it misclassifies, you route to the wrong model and lose quality. Use a classifier that’s good enough — even a small one works if the categories are clear.

## Parallel architecture

Process independent tasks simultaneously.

### Pattern 1: Fan-Out

Fan-out — run the same prompt through multiple models:

``` python
import asyncio

class ModelFanOut:
    def __init__(self):
        self.models = [
            "qwen2.5-7b",
            "qwen2.5-32b",
            "claude-sonnet-4",
        ]

    async def process(self, prompt: str) -> list[str]:
        tasks = [self.call_model(model, prompt) for model in self.models]
        return await asyncio.gather(*tasks)
```

Useful for comparison, A/B testing, or when you want to pick the best output. Expensive, but the quality gain is worth it for critical decisions.

### Pattern 2: Voting

Voting — combine outputs through consensus:

``` python
class ModelVoting:
    def __init__(self):
        self.models = [
            "qwen2.5-7b",
            "qwen2.5-32b",
            "claude-sonnet-4",
        ]

    def vote(self, prompt: str) -> str:
        responses = [self.call_model(model, prompt) for model in self.models]
        from collections import Counter
        votes = Counter(responses)
        return votes.most_common(1)[0][0]
```

Majority voting works for classification. For generation tasks, it’s harder — you need semantic similarity, not exact matches.

## Hierarchical architecture

Use models at different levels of abstraction.

### Pattern 1: Planner-Executor

Planner-executor — a strong model plans, smaller models execute:

``` python
class PlannerExecutor:
    def __init__(self):
        self.planner = "qwen2.5-32b"
        self.executors = {
            "code": "qwen2.5-coder-7b",
            "search": "qwen2.5-7b",
            "math": "qwen2.5-7b",
        }

    def process(self, task: str) -> str:
        plan = self.call_model(self.planner, f"Plan: {task}")
        results = []
        for step in self.parse_plan(plan):
            executor = self.executors.get(step["type"], "qwen2.5-7b")
            result = self.call_model(executor, step["prompt"])
            results.append(result)
        return self.call_model(self.planner, f"Synthesize: {results}")
```

The planner does the heavy lifting. The executors handle specific tasks. This pattern works well when the planning step is expensive but the execution steps are cheap.

### Pattern 2: Supervisor-Worker

Supervisor-worker — a supervisor delegates and reviews:

``` python
class SupervisorWorker:
    def __init__(self):
        self.supervisor = "qwen2.5-32b"
        self.workers = ["qwen2.5-7b", "qwen2.5-coder-7b"]

    def process(self, task: str) -> str:
        assignments = self.call_model(self.supervisor, f"Assign: {task}")
        results = []
        for assignment in self.parse_assignments(assignments):
            result = self.call_model(
                assignment["worker"], assignment["task"]
            )
            results.append(result)
        return self.call_model(self.supervisor, f"Review: {results}")
```

The supervisor is the bottleneck. It plans, delegates, and reviews. Make sure it’s fast enough, or the whole system slows down.

## Ensemble architecture

Combine multiple models for critical decisions.

### Pattern 1: Weighted Ensemble

Weighted ensemble — score each model’s output, pick the highest:

``` python
class WeightedEnsemble:
    def __init__(self):
        self.models = {
            "qwen2.5-32b": 0.5,
            "claude-sonnet-4": 0.3,
            "qwen2.5-7b": 0.2,
        }

    def decide(self, prompt: str) -> str:
        responses = {
            model: self.call_model(model, prompt)
            for model in self.models
        }
        scores = {}
        for model, response in responses.items():
            score = self.evaluate(response) * self.models[model]
            scores[response] = scores.get(response, 0) + score
        return max(scores, key=scores.get)
```

Weights reflect your confidence in each model. Adjust them based on actual performance, not benchmarks.

### Pattern 2: Consensus Ensemble

Consensus ensemble — require agreement, escalate if there isn’t any:

``` python
class ConsensusEnsemble:
    def __init__(self, threshold: float = 0.7):
        self.threshold = threshold
        self.models = [
            "qwen2.5-32b",
            "claude-sonnet-4",
            "qwen2.5-7b",
        ]

    def decide(self, prompt: str) -> str:
        responses = [
            self.call_model(model, prompt)
            for model in self.models
        ]
        from collections import Counter
        votes = Counter(responses)
        max_votes = max(votes.values())

        if max_votes / len(self.models) >= self.threshold:
            return votes.most_common(1)[0][0]

        return self.call_model("qwen2.5-32b", prompt)
```

The threshold controls how strict consensus is. 0.7 means two-thirds agreement. Lower it for faster decisions, raise it for higher confidence.

## When multi-model systems make sense

Multi-model systems make sense when you have mixed workloads, need high quality for critical decisions, or are optimizing for cost or latency.

They don’t make sense when all tasks are similar complexity, you’re prototyping, or simplicity matters more than optimization.

The rule of thumb: start with one model. Add more when you hit a real constraint — cost, latency, or quality. Don’t architect complexity before you need it.

## Tradeoffs

| Pattern | Cost | Latency | Quality | Complexity |
|---|---|---|---|---|
| Single Model | Lowest | Lowest | Variable | Lowest |
| Sequential | Medium | High | High | Medium |
| Parallel | High | Low | High | Medium |
| Hierarchical | High | High | Highest | High |
| Ensemble | Highest | Medium | Highest | Highest |

Every pattern trades something. Pick the one that matches your constraints.

## Related

[Model Routing Strategies](https://www.glukhov.org/llm-architecture/model-routing/model-routing-strategies/)— capability-based, cost-aware, latency-aware routing[Cost Optimization for LLM Systems](https://www.glukhov.org/llm-architecture/cost-optimization/cost-optimization-for-llm-systems/)— token budgeting, fallback models, caching[LLM Guardrails in Practice](https://www.glukhov.org/llm-architecture/guardrails/llm-guardrails-in-practice/)— input validation, output filtering, safety[LLM Architecture](https://www.glukhov.org/llm-architecture/)— system design pillar: routing, cost, guardrails, and orchestration