An AI-augmented product, released — same version as three months ago. But is it really same?
Since then, the frontier model behind it got migrated, the RAG was refreshed 50 times, the system prompt "improved", a tool permission widened to close a support issue. And so on. All of it silently changes behavior — sometimes noticeably, sometimes not (until too late?). Was any of it re-checked? You bet - most of the time it wasn't. Does it still work as intended?
The AI scene moves in months, not the yearly cycles the industry was used to. The leading model companies are hungry for success — and still count as startups.
So you put cutting-edge AI capabilities into your product? You may have silently introduced risk.
Now, management standards/regulations do exist — but they predate the RAG-and-agents era, target only the risky use cases, etc. Nothing stable invented yet for day-to-day AI change management. I'm used to dynamic environments, but this one makes me a bit uncomfortable. Thoughts?