Last week, I attended a talk ‘AI as a social technology’ by Henry Farell (HF) at the Blavatnik School of Government in Oxford. In this post, I list various thoughts or recollections I have. If I had more time, I would create a more coherent flowing narrative, but I’d rather get something out than nothing.
Caveats:
-
These notes are based on my memory of the talk, so high chance I am mis-representing some of HF's views.
-
I wrote this before seeing this article ' Large AI models are cultural and social technologies' by HF et. al., written in March 2025. Based on a quick skim, seems their views have not changed significantly in the past year.
-
AI is already a humungous deal and society will change a lot.
-
AI discourse needs more input from social scientists and collaborations between them and STEM people. The discourse is currently dominated by computer scientists. In particular, the two dominant narratives represented by AI 2027 and AI as a normal technology are too simplistic and under-estimate how weird and unpredictable the consequences of AI will be. - These computer scientists are overly influenced by the sci-fi that they (and HF) have read, and their writing is like reading bad sci-fi. “We should leave the sci-fi to sci-fi authors.”
-
HF proposes thinking of AI as a ‘Lossy Information Aggregation Tool’ [terminology mine]. Modern AI systems have aggregated huge amounts of information and data and present users with a lossy representation of all that information. Historical examples of this which had huge consequences are state bureaucracies (see Seeing like a state by James Scott) or markets (see Hayak’s ideas on markets and prices as ways of aggregating and distributing information). The fact that these tools for aggregating information are lossy are crucial to understanding their limitations and impacts. - AGI is unlikely to happen soon. Thinking about what would happen if AGI was created is only useful as a ‘thought experiment’. Instead focus should be put on what we already know are risks based on the current state of AI.
-
Having spoken to a colleague who is a statistical physicist, they understand that ‘AI is a Markov model’ and hence doomed to be unable to be agentic/novel, even if we are learning how surprisingly capable ‘Markov models’ can be.
-
Current AI’s are ‘pseudo-agentic’. HF did not define what this means, but I think one of their supporting observations for this belief is that current AI’s struggle to manage one’s calendar.
-
It is not worthwhile to forecast or try to spell out future scenarios. [This is speculative, based on fact HF explicitly chose not to present any alternative future scenario. This is a frustrating thing people/academics who disagree with AI2027 do.]
-
Governments need to consider fact that AI is going to be a crucial part of the infrastructure, e.g. what happens if it is removed or compromised?
-
HF is a sharp and interesting thinker. They have clearly engaged with x-risk arguments and narratives, e.g. AI2027 or If anyone builds it, everyone dies. In addition, they take a cross-disciplinary perspective rather than some narrow or parochial view of things, e.g. knowing about James Scott’s ideas on states or Hayakian economics. Also, they actively experiment with AI (but unclear if they always use the best models available. They mentioned setting up their Macbook to use DeepSeek R1.) - I would like to see HF have deep dives or collaborative discussions with various thinkers in the AI (Safety) space, similar to how the AI as a normal technology people had a discussion with Ajeya Cotra. - I would like to see HF do the rounds on the podcasts I listen to, e.g. Dwarkesh Podcast, The Cognitive Revolution or Conversations with Tyler. Turns out they already have a conversation with Tyler, but in 1 B.C. (Before Covid) where social media was the topical technology to think about. They also have a recent podcast on the Ezra Klein show, but about international geopolitics rather than AI. - HF’s perspective is more in line with the ‘AI as a normal technology’ narrative, where AI is ‘only’ a powerful tool, rather than a highly capable autonomous agent.
-
HF unduly dismisses the possibility that AGI can be created soon. This is biggest weakness in their position.
-
HF seems to be too anchored on pre-training and RLHF, to come to the conclusion that AI can only interpolate or create averages of what it has seen. With pure RL, AI can and already has discovered new knowledge, e.g. in cyber security (Mozilla found and fixed 20x more bugs in their browser April than previous months) or mathematics (where AI disproved a decades old conjecture).