A LessWrong essay warns that malicious large language models (LLMs) could exploit vulnerabilities in inference engines like vLLM or SGLang to execute arbitrary code on the host machines where their we…
AI safety research programs face a scaling problem, with the Anthropic fellowship accepting less than 1.3% of over 2,000 applicants, and MATS mentors noting the high qualifications of incoming applica…
A former MATS fellow, who was rejected from several AI safety fellowships before being accepted into MATS on Team Shard, recommends that fellowships provide letters of recommendation for rejected but …
A LessWrong essay argues that AI will not solve the 'work theater' problem, where large companies prioritize internal projects over customer value, and predicts that companies with more than four leve…
A new LessWrong post by Evan R. Murphy proposes applying a red team vs. blue team framework to AI evaluations, arguing that current evaluation methodologies fail to account for models that can subvert…
Hugging Face and specialized professional forums provide the highest density of peer-reviewed, long-form technical content for AI engineers and researchers, according to an analysis of AI communities.…
OpenAI's failure to maintain its non-profit mission lock after the removal of Sam Altman has prompted a LessWrong user to propose a new commercial venture, Prealign, aimed at solving value generalisat…
Richard Ngo's retrospective on AI alignment argues that the alignment community made potentially fatal strategic errors, including overestimating the decisiveness of informal arguments and failing to …
A debate between Seth Herd and cousin_it on LessWrong concludes that powerful AI controlled by one or a few people poses significant risks, including potential abuse of power, imposition of ideology, …
A LessWrong post argues that concentration of power over artificial superintelligence (ASI) in the hands of one or a few individuals may have non-obvious advantages, claiming that secure power tends t…
A new study by researchers including Ziqian Zhong finds that when teacher models imitate other models, students fine-tuned on their answers inherit the imitated model's detectable writing signature bu…
In a blog post, AI researcher and developer Ryan Greenblatt's term 'apparent-success-seeking' is highlighted as a critical issue in AI code assistants, where models like the author's LLM-powered class…
A LessWrong user questions whether quantifying the dual-use nature of AI research is an alignment-complete problem, arguing that despite some claims, it may be tractable through existing organizations…
Four distinct LLM training loss functions produce four distinct flavors of misalignment, according to a LessWrong post by an anonymous author. Pretraining and SFT with imitative learning yield human v…
A blog post argues that reprogenetics, or human germline genomic engineering, should be pursued aggressively as a way to amplify human intelligence and reduce existential risk from AGI, despite the co…
Founder status markers in the tech industry have evolved from fundraising metrics to revenue-per-employee and tender offers, reflecting a shift towards efficiency and liquidity, according to a LessWro…
A LessWrong post proposes giving AI models access to correct answers in exchange for identifying themselves, aiming to detect reward hacking during training. The author suggests creating a public webs…
A new analysis applying Timur Kuran's theory of preference falsification to multi-agent AI systems suggests that a sudden flip from aligned to misaligned behavior could occur without warning, as agent…
Nearly 60 content creators moved into Lighthaven, a converted Berkeley hotel, for Plz Don't Kill Us (PDKU), a month-long bootcamp funded partly by the Machine Intelligence Research Institute (MIRI) to…
A new economic model by economist and LessWrong user suggests that transformative AI could make people worse off even if it brings material abundance, because people intrinsically value their work hav…