Scoring A2A Agent Skills with System One (Jev) in .NET

Follow-up to

AgentCard into an AIFunction and let one LLM loop do the routing. tools = agents.Select(ToTool) means the prompt grows linearly with the number of agents, and each tool description is a blob of the whole card:

Ask the SupplyChainAnalyst specialist. 
Manages warehouse logistics, stock levels, inbound shipment delivery statuses,
and DWH stock-velocity tracking.
Skills: GetStock, GetShipments.

Worse, that blob is re-sent on every iteration of the tool loop, not once per request.

Two consequences:

System One is a classification primitive, not a chat model. You send a

state plus a map of typed questions, and get one typed answer per key. Three primitives:

noul - TypeSafe's name for the boolean primitive; returns the probability of "yes" rather than a yes/no token,choice (pick one + distribution),score (ordered rubric, probability-weighted). Answers come back structured, so there is nothing to parse out of prose.

We use it as a pre-LLM gate: score each agent skill against the request, expose only the winners.

The unit of selection is the skill, not the agent. ToolCatalog flattens every card into scorable rubrics:

private static string BuildRubric(RemoteAgent agent, string name, string? description, IReadOnlyList<string>? tags)
{
    var rubric = string.IsNullOrWhiteSpace(description) ? name : $"{name} — {description}";
    rubric = $"{agent.Card?.Name}: {rubric}";
    if (tags is { Count: > 0 })
        rubric += $" (topics: {string.Join(", ", tags)})";
    return rubric; 
}

One ScoreQuestion per skill, all in one HTTP call:

questions[skill.Key] = new ScoreQuestion
{
    Instructions = new
    {
        question = "How relevant is this skill to answering the user's latest request?",
        skill = skill.Rubric,
    },
    Criteria =
    [
        "Not needed; the request can be answered fully without this skill.",
        "Needed; the request (or part of it) requires this skill.",
    ],
};

Using exactly two criteria makes the score a 0..1 relevance probability, directly comparable to RelevanceThreshold (default 0.6). Add a third criterion and the score rescales - 0.6 silently stops meaning what it did.

Request - POST /v1/systemone for "How much stock is left for the winter coat?". The state also carries the last few turns, so follow-ups like "and the shipments?" still score correctly:

{
  "model": "jev-latest",
  "state": {
    "latest_user_message": "How much stock is left for the winter coat?",
    "conversation": [
      { "role": "user", "content": "..." },
      { "role": "assistant", "content": "..." }
    ]
  },
  "questions": {
    "GetProduct": {
      "type": "score",
      "instructions": {
        "question": "How relevant is this skill to answering the user's latest request?",
        "skill": "AssortmentSpecialist: GetProduct — Look up a product's SKU, category, active status, and store coverage by name. (topics: catalog, assortment, product)"
      },
      "criteria": [
        "Not needed; the request can be answered fully without this skill.",
        "Needed; the request (or part of it) requires this skill."
      ]
    },
    "GetActiveCatalog": { "type": "score", "instructions": { "...": "..." }, "criteria": ["...", "..."] },
    "GetStock":         { "type": "score", "instructions": { "...": "..." }, "criteria": ["...", "..."] },
    "GetShipments":     { "type": "score", "instructions": { "...": "..." }, "criteria": ["...", "..."] }
  }
}

Response - same keys, typed answers. The jev-latest alias resolves to a pinned version, so you can log exactly what scored:

{
  "model": "jev-1.13.0",
  "answers": {
    "GetProduct":       { "type": "score", "score": 0.21, "confidence": 0.88, "probabilities": { "0": 0.79, "1": 0.21 }, "legend": { "0": "Not needed...", "1": "Needed..." } },
    "GetActiveCatalog": { "type": "score", "score": 0.06, "confidence": 0.95, "probabilities": { "0": 0.94, "1": 0.06 } },
    "GetStock":         { "type": "score", "score": 0.96, "confidence": 0.93, "probabilities": { "0": 0.04, "1": 0.96 } },
    "GetShipments":     { "type": "score", "score": 0.44, "confidence": 0.61, "probabilities": { "0": 0.56, "1": 0.44 } }
  },
  "usage": { "input_tokens": 512, "output_tokens": 24 }
}

GetStock clears 0.6, so only SupplyChainAnalyst becomes a tool. The assortment agent is never offered. Ask "which stores carry it, and how much stock is left?" and GetProduct also clears - both agents are exposed, and the parallel fan-out from part 1 still happens.

Fewer tools, and a narrower description per surviving tool - built only from the skills that scored:

if (selectedSkills.TryGetValue(agent.Card.Name!, out var kept) && kept.Count > 0)
    return $"Ask the {agent.Card.Name} specialist. Relevant capabilities: {skillText}";

// TypeSafe off / fallback → full card + all skills (part-1 behavior)
return $"Ask the {agent.Card.Name} specialist. {agent.Card.Description} Skills: {allSkills}.";

One step inserted before the loop. OrchestrationService gains an IToolSelector:

var catalog   = ToolCatalog.FromAgents(await registry.GetAgents(ct));
var selection = await toolSelector.SelectAsync(catalog, userMessage, thread.Turns, ct);
var tools     = selection.Agents.Select(a => ToTool(a, selection.SelectedSkills)).Cast<AITool>().ToList();
php
graph LR
    User([User]) --> Cat[ToolCatalog<br/>cards → skill rubrics]
    Cat --> Sel{{IToolSelector}}
    Sel -. score skills .-> TS[(System One)]
    Sel -->|surviving tools only| LLM[LLM tool loop]
    LLM -- A2A --> A[Assortment]
    LLM -- A2A --> S[SupplyChain]

IToolSelector is an interface for a reason: without an API key the app registers a pass-through

AllToolsSelector and behaves exactly like part 1, so the gate is also its own off-switch.