cd /news/artificial-intelligence/iter-ca-update-14 · home › topics › artificial-intelligence › article
[ARTICLE · art-170224] src=buttondown.com ↗ pub= topic=artificial-intelligence verified=true sentiment=· neutral

iter.ca update #14

Independent researcher and blogger iter.ca published a post on October 29 interpreting attention head outputs with natural language autoencoders (NLAs), part of ongoing work on interpreting changes to a model's residual stream. The author flagged a possible flaw in the NLA approach in which attention heads attend to the first token most of the time — in Qwen3 chats, always the `<|im_start|>` token — and is considering filtering out cases where a head mostly acts as an attention sink. The update also covers a Bayesian network modeling tool built with the Amp Code agent harness and a planned interactive AI x-risk model for questions such as how an intervention would change the author's P(doom).

key takeaway

Attention heads in Qwen3 chats often attend mostly to the first token, `<|im_start|>`, so NLA interpretations should filter out cases where a head is just acting as an attention sink.

by read4 min views36 publishedNov 6, 2026
iter.ca update #14
Image: Buttondown (auto-discovered)

It’s time for the fourteenth iter.ca update email. This is the only email whose number corresponds to the number of lines in a sonnet! It's been a month since my last update; hopefully I'll be able to deliver more frequent updates to you in the future.

posts #

- [Interpreting attention head outputs with NLAs](https://www.lesswrong.com/posts/nNYc5qddHXwdk5kix/interpreting-attention-head-outputs-with-nlas) on October 29
- [Some interesting internal documents from Anthropic litigation](https://www.lesswrong.com/posts/em5wnA7gpQSvfsuxQ/some-interesting-internal-documents-from-anthropic) on October 14
- [Don't run your own mailserver](https://iter.ca/post/mailserver-bad/) on October 8 ([Claudeslop version](https://www.youtube.com/watch?v=txFRuT_XJOg) )
- [Anthropic's corporate structure](https://www.lesswrong.com/posts/Hb5aCFtLuHc5uvAgX/anthropic-s-corporate-structure) on October 7

research #

I’ve done some more natural language autoencoder stuff; my post talks about that. It’s still pretty incomplete though; I hope to understand this kind of thing better. I’m very interested in interpreting changes to the residual stream. I’ve been thinking about generalizing this to making a model that can interpret arbitrary change at any layer, but I want to get better at this simpler case first.

I think there might be a problem with the NLA from that post where attention heads attend to the first token most of the time (Qwen3 doesn’t have an explicit attention sink, but for chats the first token is always <|im_start|>). Maybe I should filter out cases where the attention head is mostly just attending to the first token as a sink? I’m not sure what the best way to handle this is though.

SWE arc #

I tried using Amp Code (an agent harness) and I like working with it. It's opinionated in some ways that I don't like, but I like it for doing web development stuff. I really like the way agents talking between threads works. Overall I find it pretty fun to use! I ended up doing some software development with it:

  • I started working on redesigning the homepage for my website. You can take a look at the current WIP version (mostly made by Claude) but I still have to fix some things. It works a lot better on desktop than mobile for now. I’m not sure if this redesign is actually good though. Let me know if you have strong thoughts!
  • I created a Bayesian network modeling tool ! I think it's pretty neat; I'll write about it more soon, and why I think BNs are interesting.

London #

I've done some London-maxxing! I finally went to the Isle of Dogs and wandered around for a bit. I also went to some other places in London.

misc #

  • I changed my favorite color from cyan to yellow. I still like cyan, but yellow just speaks to me more now.
  • I’m doing Just Write , although I haven’t actually written anything for it yet (probably I’ll do something related to Bayesian networks).

looking forward #

I’m going to keep working on NLA stuff as detailed above; hopefully I get some more interesting results soon!

Also, I’m going to write about how Bayesian networks can be useful for modeling things. I’ve tried doing AI x-risk modeling using BNs; making a good model is pretty hard; it would be cool if I could make a good interactive model where you can see what I think would happen if you changed various things (e.g. answering “How would a specific intervention change my P(doom)?”, or “Are worlds where an AI company unilaterally s better or worse?”). You can take a look at my current modeling work but keep in mind that it’s still a draft and I don’t endorse it yet.

I also tried creating a PACER account so I could read some more AI-related court filings. Unfortunately they don’t seem to like foreign credit cards (and I’m a mere Canadian with no US credit card), so I’ll have to wait until I get the verification code mailed to me to read some more court filings.

My current plan is to be back in Toronto on December 15th. I'm not quite sure what I'm going to be up to once I get back to Toronto yet! I’ve considered visiting Montreal when I’m back in Canada but it might be too cold for that. I might go to 40c3 in Hamburg. I also might end up in SF for some time.

── more in #artificial-intelligence 4 stories · sorted by recency
── more on @iter.ca 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/iter-ca-update-14] indexed:0 read:4min 2026-11-06 · —