{"slug": "how-ai-headshot-generators-actually-work-lora-15-selfies-and-the-overfitting", "title": "How AI headshot generators actually work: LoRA, 15 selfies, and the overfitting problem", "summary": "An engineer explains how AI headshot generators work under the hood, detailing the use of LoRA (Low-Rank Adaptation) to fine-tune diffusion models on a user's selfies. The post highlights the overfitting problem that occurs when training on similar photos, leading to generic or same-looking results, and suggests using diverse photos to improve output quality.", "body_md": "Every AI headshot product works roughly the same way under the hood, and once you know the shape of it, most of the weird results stop being weird.\n\nYou upload 10 to 20 selfies. Twenty minutes later you get 100 portraits back. Some look exactly like you. Some look like your cousin. One looks like you wearing a jacket you've never owned, in an office you've never been to, and that one is usually the most interesting failure.\n\nHere's what's happening in between.\n\n`sks person, wearing a blue shirt, indoor lighting`\n\n.`sks person, corporate headshot, grey backdrop, softbox lighting`\n\n, a few hundred times with different seeds.Stage 3 is where all the quality lives, so that's the one worth understanding.\n\nA diffusion model has a few billion parameters. Fine-tuning all of them for one person's face would take hours on serious hardware and produce a multi-gigabyte file per customer. Nobody's running a business on that.\n\nLoRA, Low-Rank Adaptation, sidesteps it. Instead of updating a big weight matrix `W`\n\n, you freeze `W`\n\nand learn two skinny matrices `A`\n\nand `B`\n\nwhose product has the same shape. At inference you compute `W + BA`\n\n. If `W`\n\nis 1024x1024 and you pick rank 16, then `A`\n\nis 16x1024 and `B`\n\nis 1024x16. That's 32,768 trained numbers instead of 1,048,576, so about 3%.\n\nThese adapters get injected into the attention layers, which is where the model decides what a thing looks like rather than where it goes. Train for a few hundred steps, ship a file that's a few megabytes, load it in milliseconds.\n\nThe rank is the knob. Low rank, say 4 to 8, and the adapter doesn't have enough capacity to hold your specific face, so you get a generic person who vaguely resembles you. High rank, 64 and up, and it has enough capacity to memorise your training set wholesale, which sounds good and isn't.\n\nThis surprises people. More data is supposed to be better.\n\nThe problem is that the adapter learns everything your photos have in common, and it has no way to know which of those things you consider \"your face\".\n\nUpload 15 selfies taken in the same week and 12 of them will have the same haircut, the same lighting from the same window, and the same three t-shirts. The model learns `sks person`\n\nmeans a face **plus** that lighting **plus** those shirts. Then you ask for a corporate headshot and it fights itself, because half of what it learned about you is a grey marl t-shirt in a bedroom.\n\nThat's overfitting, and on faces it shows up in a specific way: the generated images look great and they all look like the same photo. Same angle, same expression, same background tone. The model isn't generating your face in new situations. It's reconstructing your training set with slight variation.\n\nFifteen genuinely different photos beat 100 near-duplicates, every time. Different days, different rooms, different clothes, a couple of different angles.\n\nThe opposite failure is more common in the cheap tools. You get 100 clean, well-lit, professional portraits of somebody who is nearly you.\n\nTwo things usually cause it.\n\n**Undertrained adapter.** The trigger token never fully bound to your face, so the base model's idea of \"a person\" is doing most of the work. You get a composite of you and the average face in the model's training data.\n\n**Prior preservation pulling too hard.** To stop the adapter from destroying the model's general concept of \"person\", training usually mixes in generated images of random people alongside yours. Too much of that and your identity gets regularised away.\n\nThere's a measurable version of this. Take a face embedding model, generate an embedding for your input photos and for each output, and compute cosine similarity. Anything above about 0.65 reads as clearly the same person. Below 0.5 and most people looking at it will say \"that's not quite you\", even if they can't say why.\n\nIf a product isn't filtering its outputs on something like that score, you're doing the filtering by hand, which is what it feels like when you get 100 images back and 12 are usable.\n\nThe practical version, given all of the above:\n\nRoughly 15 photos, ideally from more than one occasion. The person who takes 15 selfies in one sitting in one room gets noticeably worse results than the person who digs 15 out of their camera roll from the last year.\n\nIf you want to see what the pipeline does with your photos before committing, most of the products have a free tier that runs a smaller version. You can put a handful of selfies through [a free generator](https://www.betterpic.io/free-tools/free-ai-headshot-generator) and see the identity-drift problem for yourself, which is a faster education than reading about it.\n\nThe paid versions of [AI headshots](https://www.betterpic.io/ai-headshots) mostly differ in how much they spend on stage 4, the generation and filtering, rather than on anything exotic in stage 3. Everyone's doing LoRA or something close to it. The difference is how many candidates get generated and how aggressively the bad ones are thrown away before you see them.\n\nWhich is also why the outputs vary so much between products at the same price. If you want to see the difference in practice, it's worth looking at [BetterPic side by side with Aragon](https://www.betterpic.io/compare/aragon) rather than trusting anyone's sample gallery, including ours. Sample galleries are cherry-picked by definition.\n\nThe model has no concept of \"your face\" as separate from \"the pixels in your uploads\". Everything consistent across your training images becomes part of your identity as far as the adapter is concerned.\n\nSo the quality of your result is decided almost entirely before you hit upload.\n\n*Disclosure: I work on BetterPic, one of the tools linked above.*", "url": "https://wpnews.pro/news/how-ai-headshot-generators-actually-work-lora-15-selfies-and-the-overfitting", "canonical_source": "https://dev.to/miriam_alonso_01/how-ai-headshot-generators-actually-work-lora-15-selfies-and-the-overfitting-problem-43p", "published_at": "2026-08-20 13:04:01+00:00", "updated_at": "2026-08-20 13:16:48.864706+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "generative-ai", "ai-products", "ai-tools"], "entities": ["LoRA", "diffusion model", "sks person"], "alternates": {"html": "https://wpnews.pro/news/how-ai-headshot-generators-actually-work-lora-15-selfies-and-the-overfitting", "markdown": "https://wpnews.pro/news/how-ai-headshot-generators-actually-work-lora-15-selfies-and-the-overfitting.md", "text": "https://wpnews.pro/news/how-ai-headshot-generators-actually-work-lora-15-selfies-and-the-overfitting.txt", "jsonld": "https://wpnews.pro/news/how-ai-headshot-generators-actually-work-lora-15-selfies-and-the-overfitting.jsonld"}}