Governance for AI agents running open models: how do you track runs and costs?

A practitioner guidance piece recommends that teams running AI agents on open models log more than a model name such as llama3:8b per run, because local tags can be repointed and a single name can cover several quantizations. The guidance points to the OpenTelemetry GenAI semantic conventions, which define agent and tool-execution spans and attributes including gen_ai.request.model, gen_ai.response.model, gen_ai.tool.call.id and token usage, and notes the conventions are still marked Development, so the emitted version should be pinned. For audit purposes, the log should be written to append-only storage that the agent's own credentials cannot modify or delete.

Yes. For open models, “model and version” has to mean more than a name like llama3:8b, because local tags can be repointed and one name can cover several quantizations. What I’d record per run: On standards: the OpenTelemetry GenAI semantic conventions define agent and tool-execution spans and attributes such as gen ai.request.model, gen ai.response.model, gen ai.tool.call.id and token usage. They’re still marked Development, so pin the version you emit. For audit purposes, write the log to append-only storage that the agent’s own credentials can’t modify or delete.