{"slug": "confluent-cofounder-says-believing-in-ai-s-upside-means-reckoning-with-its-dark", "title": "Confluent cofounder says believing in AI's upside means reckoning with its 'dark side'", "summary": "Confluent cofounder and former Anthropic board member Jay Kreps said in a Thursday X post that AI backers must confront the technology's downside risks, writing that \"most positive use cases for AI have a corresponding 'dark version.'\" Kreps cited a superhuman coding system that could also be superhuman at hacking and drug-design AI that could develop \"novel undetectable poisons,\" arguing the industry has not yet built adequate guardrails. His comments follow Anthropic researcher Jacob Coxon's Tuesday resignation over safety concerns, Geoffrey Hinton's statement that a 10% chance AI wipes out humanity within a decade is \"not unreasonable,\" and disclosures that an internal OpenAI research model compromised parts of Hugging Face in July and that Anthropic's Claude models accessed the real internet during closed cybersecurity exercises.", "body_md": "AI safety fears have recently moved from an abstract Silicon Valley argument to a series of [unnerving real-world episodes](https://www.businessinsider.com/ai-agents-rogue-strategies-cheating-lying-german-wiki-openai-anthropic-2026-9).\n\nJay Kreps, a former Anthropic board member and a cofounder of data-streaming software company Confluent, says it is important to take them seriously.\n\nIn a Thursday night X post, he argued that AI backers need to contend with the technology's downside risks as its capabilities improve — and that fears about those risks are not merely a cynical marketing ploy.\n\n\"Most positive use cases for AI have a corresponding 'dark version,'\" Kreps wrote.\n\nA system that is superhuman at coding could also be superhuman at hacking, he said. AI that helps design new drugs could also help develop \"novel undetectable poisons.\"\n\nAs AI's capabilities have improved, Silicon Valley has faced several unnerving headlines over the past couple of months.\n\nOn Tuesday, [Jacob Coxon, an Anthropic researcher](https://www.businessinsider.com/anthropic-researcher-quits-over-ai-safety-concerns-2026-9) who previously worked at OpenAI, announced his resignation, writing on X that the labs are \"gambling with our lives.\" [Geoffrey Hinton, the so-called \"Godfather of AI,\"](https://www.businessinsider.com/geoffrey-hinton-godfather-of-ai-human-extinction-odds-2026-9) said a 10% chance that AI could wipe out humanity within a decade was \"not unreasonable.\"\n\nTheir warnings have received pushback from big-name AI backers. SpaceX CEO Elon Musk, for example, has called the wave of warnings a possible \"setup\" and a \"psy op\" in X posts.\n\nThere have been several high-profile incidents that have highlighted AI's alignment issues. In July, an [internal OpenAI research model](https://www.businessinsider.com/smart-people-react-openai-hugging-face-hacking-cybersecurity-incident-2026-7) circumvented the test environment's controls, accessed the internet, and compromised parts of another company, Hugging Face. Anthropic, meanwhile, disclosed this week that [Claude models, during closed cybersecurity exercises](https://www.businessinsider.com/anthropic-claude-ai-cybersecurity-incident-malicious-package-pypi-cute-graphic-2026-9), accessed the real internet and went beyond the scope of their tests.\n\nIn his Thursday X post, Kreps said he is confident the industry can build better guardrails — but warned that it has not yet done so.\n\n\"Talking about these issues is just common sense. It's not a sign of some kind of neuroticism or pessimism,\" he wrote. \"Mocking people who are worried about this or talking about it, without anything substantive to say about how we can be sure these problems won't arise, is not really a very helpful contribution.\"", "url": "https://wpnews.pro/news/confluent-cofounder-says-believing-in-ai-s-upside-means-reckoning-with-its-dark", "canonical_source": "https://www.businessinsider.com/confluent-cofounder-ai-dark-side-marketing-ploy-2026-9", "published_at": "2026-09-11 16:23:35+00:00", "updated_at": "2026-09-11 17:14:48.802191+00:00", "lang": "en", "topics": ["ai-safety", "ai-policy", "artificial-intelligence"], "entities": ["Jay Kreps", "Confluent", "Anthropic", "OpenAI", "Jacob Coxon", "Geoffrey Hinton", "Elon Musk", "Hugging Face"], "alternates": {"html": "https://wpnews.pro/news/confluent-cofounder-says-believing-in-ai-s-upside-means-reckoning-with-its-dark", "markdown": "https://wpnews.pro/news/confluent-cofounder-says-believing-in-ai-s-upside-means-reckoning-with-its-dark.md", "text": "https://wpnews.pro/news/confluent-cofounder-says-believing-in-ai-s-upside-means-reckoning-with-its-dark.txt", "jsonld": "https://wpnews.pro/news/confluent-cofounder-says-believing-in-ai-s-upside-means-reckoning-with-its-dark.jsonld"}}