{"slug": "ai-products-generate-more-signal-than-teams-know-how-to-use", "title": "AI products generate more signal than teams know how to use", "summary": "AI products generate more usable signal than previous software, but most teams fail to leverage it due to fragmented data across engineering, product, and customer success. The author argues that holistic observability—combining agent traces, user feedback, and sentiment analysis—can yield clear insights for improving retention and revenue.", "body_md": "07/15/2026\n\nAI products generate more usable signal than any software before them, yet (almost) nobody is actually using it.\n\nI think that’s a mistake, and a surprisingly expensive one at that.\n\n>\n\nFor the last twenty years, we built products around indirect signals. A user clicked here, viewed that page, hovered over this button but didn't press it. We became really good at inferring user intent and behavior from these breadcrumbs! Unless you were running surveys, this was your only data point for understanding users.\n\nBut with AI products, users now interact via natural language. This new form of user input is far more useful. We can read exactly what users want to accomplish, and whether or not they’re satisfied.\n\nWhen you pair this with other signals (thumbs up/down votes, online evals, technical metrics, etc), you get a clear relationship between the performance of your agent and revenue.\n\nBut it feels like almost nobody is actually putting the data together. The average team has some basic APM software and agent tracing set up. But when it comes to analyzing user intent, churn signals, retention, and customer satisfaction, the data falls relatively short.\n\n>\n\nTwo reasons.\n\nThe first and largest reason is that data is really fragmented across teams:\n\n- Engineering owns tracing and technical metrics.\n\n- Product owns user feedback and analytics.\n\n- Customer Success owns churn signals and revenue expansion.\n\nNobody is actually in charge of analyzing these in tandem 1. But if you dig deeper, you'll find that all of this data leads to interlinked relationships between agent performance and revenue.\n\nThe second reason is that it simply takes a lot of effort to collect and effectively analyze this data. And if you’re not careful, it can get expensive fast.\n\n>\n\nWhen you do put this into practice and truly capitalize on these data points, the relationships become a lot more clear:\n\nIt seems like the average user spends 3 turns asking the agent to rewrite the email in the user’s tone. We can probably boost adoption of our email service if we improve this!\n\nWow, 35% of users who encounter problem XYZ tend to never come back! We can probably cut churn if we investigate that problem.\n\nOver 430 users have asked the agent to search the internet, but the agent doesn’t search capabilities yet. We should add this to our product offering!\n\nI’ve started calling this “holistic observability”. More than just tracing your agent, you collect a suite of data points:\n\n- agent traces and conversations\n\n- user votes/feedback (simple thumbs up/down data is enough)\n\n- online evals/monitoring over traces\n\n- topic clustering over user prompts\n\n- sentiment analysis on user prompts (frustration, confusion, etc)\n\nEventually, signals start to show themselves. The result is clear and actionable insights for teams across the organization. Customer success can clearly identify churn signals and opportunities for revenue expansion in existing accounts. Product teams have a clear understanding of use cases, and which ones are underserved. Engineering teams get insight into real agent performance that extends beyond simple error tracking.\n\nMy strongest opinion is that in AI startups, agent observability should not remain an engineering-only concern. CEOs, product leaders, and customer-facing teams should be reading this data, too.\n\nWe spent the last twenty years trying to infer what users wanted from clicks. Now they're telling us directly.\n\nIt would be strange not to listen.\n\nPerhaps the only roles I’ve seen do this well are founders or engineers at extremely early startups (since engineers tend to fill all of these roles at early startups). [↩](#user-content-fnref-1)", "url": "https://wpnews.pro/news/ai-products-generate-more-signal-than-teams-know-how-to-use", "canonical_source": "https://www.armank.dev/thoughts/2", "published_at": "2026-07-16 16:26:43+00:00", "updated_at": "2026-07-16 16:55:29.319475+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-products", "ai-tools", "ai-agents", "developer-tools"], "entities": [], "alternates": {"html": "https://wpnews.pro/news/ai-products-generate-more-signal-than-teams-know-how-to-use", "markdown": "https://wpnews.pro/news/ai-products-generate-more-signal-than-teams-know-how-to-use.md", "text": "https://wpnews.pro/news/ai-products-generate-more-signal-than-teams-know-how-to-use.txt", "jsonld": "https://wpnews.pro/news/ai-products-generate-more-signal-than-teams-know-how-to-use.jsonld"}}