{"slug": "show-hn-test-product-ideas-on-synthetic-customers-open-source-ssr-skill", "title": "Show HN: Test product ideas on synthetic customers (open-source SSR skill)", "summary": "An open-source Agent Skill called synthetic-consumer-research lets Claude Code, Codex, Cursor, Google Antigravity, Gemini CLI and other agents run synthetic consumer surveys using Semantic Similarity Rating (SSR), the method from a 2025 paper by Maier, Aslak, Fiaschi et al. of PyMC Labs and Colgate-Palmolive. Tested on 57 real product surveys with 9,300 people, SSR reached 90% of human test-retest reliability and matched the real rating spread with a KS similarity of 0.88, versus 0.26 for asking an LLM directly for a 1-to-5 rating. The skill generates personas, queries each one in a separate LLM call, converts answers into probability distributions over ratings 1 to 5, and reports mean rating, top-2-box share, spread, demographic breakdowns and the most positive and negative responses.", "body_md": "**Test a product idea on 100 simulated customers in a few minutes.** This is an open [Agent Skill](https://agentskills.io) (`SKILL.md`) that works in **Claude Code, Codex, Cursor, Google Antigravity, Gemini CLI** and any other agent that supports skills, and from a plain terminal too. It runs a synthetic consumer survey. It gives LLM personas a product concept, collects their answers in their own words, and turns those answers into realistic 1 to 5 purchase intent ratings with **Semantic Similarity Rating (SSR)**.\n\nSSR comes from the 2025 paper [*LLMs Reproduce Human Purchase Intent via Semantic Similarity Elicitation of Likert Ratings*](https://arxiv.org/abs/2510.08338) (Maier, Aslak, Fiaschi et al., PyMC Labs and Colgate-Palmolive). It was tested on 57 real product surveys with 9,300 people. SSR reached **90% of human test-retest reliability**: the consistency real people have when they answer the same survey twice. Its rating spread matched the real one with a **KS similarity of 0.88**.\n\nYou describe a product. The skill then:\n\n1. **Creates synthetic consumers:** personas with realistic ages, incomes, jobs and locations for your target market.\n2. **Asks each one the survey question separately,** for example \"How likely are you to purchase the product?\". Each persona gets its own fresh LLM call, so answers don't copy each other.\n3. **Scores every answer with SSR.** Each answer becomes a probability for each rating from 1 to 5, not a single guessed number.\n4. **Writes a report:**  - the mean rating;\n  - the top-2-box share (people rating 4 or 5);\n  - the spread of ratings;\n  - results by age or income group;\n  - the most negative and most positive answers, which show what puts people off and what sells.\n\nUse it for concept testing, price checks, landing page and ad copy, feature ideas, or picking between two product versions before you spend money on a real survey.\n\nAlso known as: **AI focus group**, **synthetic panel**, **silicon sampling**, **LLM survey simulation**, **AI customer personas**, **digital twin customers** and **synthetic respondents**.\n\nAsk an LLM \"rate 1 to 5\" and it plays it safe: nearly every answer is a 3 or a 4. The average can look fine, but the spread is unrealistic. In the paper, this direct method matched the real rating spread with a KS similarity of only **0.26**.\n\nSSR lets the model answer like a person (\"I like it, but $29 is a lot for me right now\"). It then measures how close that sentence is in meaning to reference sentences for each rating. That keeps the uncertainty in the answer and gives realistic spreads (**0.88**).\n\n**Any agent, one command** (uses the open [`skills`](https://skills.sh) installer, which detects Claude Code, Codex, Cursor, Antigravity, Gemini CLI and more):\n\n```\nnpx skills add AryamanSi17/synthetic-consumer-research\n```\n\n**Or install by hand.** Copy the folder `skills/synthetic-consumer-research` into your agent's skills folder:\n\n| Agent | Personal (all projects) | One project only | \n|---|---|---|\n| Claude Code | `~/.claude/skills/` | `.claude/skills/` | \n| Codex | `~/.agents/skills/` (older versions:`~/.codex/skills/` ) | `.agents/skills/` | \n| Cursor | `~/.cursor/skills/` | `.cursor/skills/` or`.agents/skills/` | \n| Google Antigravity | `~/.gemini/antigravity/skills/` | `.agents/skills/` | \n| Gemini CLI | `~/.gemini/skills/` | `.gemini/skills/` | \n\n```\ngit clone https://github.com/AryamanSi17/synthetic-consumer-research\ncp -r synthetic-consumer-research/skills/synthetic-consumer-research ~/.claude/skills/   # change the path for your agent\n```\n\n`.agents/skills/` inside a project is the shared folder that Codex, Cursor and Antigravity all read, so one copy there covers all three.\n\n**Claude Code plugin:**\n\n```\n/plugin marketplace add AryamanSi17/synthetic-consumer-research\n/plugin install synthetic-consumer-research@synthetic-consumer-research\n```\n\n**Embeddings.** Pick one:\n\n- **OpenAI `text-embedding-3-small` (recommended, as in the paper):** set`OPENAI_API_KEY` . No extra package is needed.\n- **Free and local:**`pip install sentence-transformers` . It uses`all-MiniLM-L6-v2` .\n\nThe scripts are plain Python with no other dependencies. To collect answers, `simulate.py` finds the agent command line tool on its own: `claude`, `codex` or `gemini`. Anywhere else, including inside Cursor or Antigravity, pass any tool that reads a prompt on stdin, for example `--llm-cmd 'llm -m gpt-4o'`.\n\nIn your agent (Claude Code, Codex, Cursor, Antigravity), just ask:\n\nTest this product idea on 100 synthetic customers in the US: a $29 cold brew bottle that makes café-style cold brew overnight.\n\nThe agent follows the skill: it writes the concept and personas, runs the survey, scores it and explains the results.\n\nOr run it yourself:\n\n```\ncd skills/synthetic-consumer-research\npython3 scripts/simulate.py --concept ../../examples/concept.md --personas ../../examples/personas.jsonl --samples 2\npython3 scripts/ssr.py responses.jsonl --by age_band --out results\n```\n\nAlready have real survey results? Add `--human 12,30,80,150,90` (the counts for ratings 1 to 5) to measure how close the synthetic survey gets. This is the best way to check the setup before trusting it.\n\nThis is from the included [example](https://github.com/AryamanSi17/synthetic-consumer-research/blob/main/examples): 8 personas with 2 answers each, scored with the free local model. See the full [report](https://github.com/AryamanSi17/synthetic-consumer-research/blob/main/examples/results/report.md).\n\n```\nMean rating: 3.33 / 5\nTop-2-box (rated 4 or 5): 49%\n\n  1  ###                             10.1%\n  2  ######                          21.3%\n  3  ######                          20.0%\n  4  #######                         22.4%\n  5  ########                        26.2%\n```\n\nThe answers are often more useful than the score:\n\n\"On a fixed income, $29 for a bottle is a lot to spend on something I can already make for pennies.\" (65+, retired)\n\n\"I buy cold brew most days and a bottle that makes it at home for much less is appealing. I'd want to hear from people who've used it, though.\" (35-44, high income)\n\nThis tiny run is only a demonstration. Use 100 or more personas for real decisions. The local model packs scores closer together than OpenAI embeddings do, so use OpenAI when it matters.\n\nFor each answer and each set of five reference sentences (one sentence per rating, 1 to 5):\n\n1. **Embed** the answer and the five reference sentences.\n2. **Take the cosine similarity** of the answer to each reference sentence:`s1 ... s5` .\n3. **Subtract the smallest similarity** from all five. All the reference sentences are about the same topic, so the raw similarities are bunched together. Subtracting the minimum spreads them out.\n4. **Normalise** so the five numbers sum to 1. That gives the probability of each rating.\n\n```\np(r) = (s_r − s_min) / Σ (s_k − s_min)\n```\n\nThe skill averages six different reference sets, which makes the result less dependent on any one wording. Optional temperature sharpens or flattens the result: `p(r)^(1/T)`. The survey result is the average over all answers.\n\nThe reference sentences live in [`anchors/`](https://github.com/AryamanSi17/synthetic-consumer-research/blob/main/skills/synthetic-consumer-research/anchors), with one file for purchase intent and one generic agree/disagree file. Edit them or add your own JSON for other questions.\n\n| Method | Matches real concept ranking (correlation attainment) | Matches real rating spread (KS similarity) | \n|---|---|---|\n| Ask for a number directly | about 80% | 0.26 | \n| Free text, then an LLM picks a number | 85% | 0.72 | \n| **Free text + SSR** | **about 90%** | **0.88** | \n\nThese figures are for GPT-4o, with the same pattern for Gemini 2.0 Flash. Other findings:\n\n- **Demographic personas matter.** Without them, correlation dropped from 92% to 50%.\n- **Age and income effects were reproduced well.** Gender and region effects were not.\n- **Text descriptions** work almost as well as concept images.\n\n- **This is a first filter, not real customers.** Use it to rank ideas and find objections quickly. Confirm important decisions with real people.\n- **Narrow validation.** The method was proven on personal care products in the United States. Other categories, countries and languages are less tested.\n- **Training data limits.** The model only knows its training data, so new categories, local brands and cultural habits may be off.\n- **Unreliable splits.** Gender and region splits weren't reliable in the paper.\n- **Unpublished reference sentences.** The paper's own sentences aren't published. The ones here follow its rules (1 = won't buy, 3 = unsure, 5 = will buy), and results move slightly with the wording.\n\n**What is synthetic consumer research?**\nUsing AI personas to answer survey questions as customers would, so you can test ideas before paying for a real panel.\n\n**What is a Likert scale?**\nThe familiar 1 to 5 survey scale, from \"definitely would not buy\" to \"definitely would buy\".\n\n**What is Semantic Similarity Rating (SSR)?**\nA way to turn free-text answers into Likert ratings by comparing their meaning with reference sentences, using embeddings. It was introduced in arXiv:2510.08338.\n\n**Which agents does it work with?**\nClaude Code, Codex, Cursor, Google Antigravity, Gemini CLI and any other agent that reads `SKILL.md` skills. The scripts also run on their own: `simulate.py --llm-cmd` accepts any LLM command line tool, and `ssr.py` scores answers from any source.\n\n**How much does it cost?**\nOne short LLM call per answer, plus embeddings that cost a fraction of a cent per 1,000 answers. The local embedding model is free.\n\n**Can I score real open-ended survey answers?**\nYes. Put them in a JSONL file with a `response` field and run `ssr.py`.\n\n- **Method:** Benjamin F. Maier, Ulf Aslak, Luca Fiaschi, Nina Rismal, Kemble Fletcher, Christian C. Luhmann, Robbie Dow, Kli Pappas and Thomas V. Wiecki.\n- **Official reference implementation:**[pymc-labs/semantic-similarity-rating](https://github.com/pymc-labs/semantic-similarity-rating) , Apache 2.0. This skill is an independent, dependency-free version of the same formula, plus persona simulation and reporting.\n\n```\n@article{maier2025ssr,\n  title   = {LLMs Reproduce Human Purchase Intent via Semantic Similarity Elicitation of Likert Ratings},\n  author  = {Maier, Benjamin F. and Aslak, Ulf and Fiaschi, Luca and Rismal, Nina and Fletcher, Kemble and Luhmann, Christian C. and Dow, Robbie and Pappas, Kli and Wiecki, Thomas V.},\n  journal = {arXiv preprint arXiv:2510.08338},\n  year    = {2025}\n}\n```\n\nMade by [Aryaman Sinha](https://github.com/AryamanSi17) · Instagram [@ai_aryaman](https://www.instagram.com/ai_aryaman/)", "url": "https://wpnews.pro/news/show-hn-test-product-ideas-on-synthetic-customers-open-source-ssr-skill", "canonical_source": "https://github.com/AryamanSi17/synthetic-consumer-research", "published_at": "2026-11-21 18:27:14+00:00", "updated_at": "2026-11-21 18:43:35.337685+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "large-language-models", "ai-research", "developer-tools"], "entities": ["Claude Code", "Codex", "Cursor", "Google Antigravity", "Gemini CLI", "PyMC Labs", "Colgate-Palmolive", "Semantic Similarity Rating"], "also_reported_by": [], "takeaway": "Scoring free-text persona answers by semantic similarity to reference sentences per rating yields a KS similarity of 0.88 to real survey spreads, versus 0.26 when asking an LLM directly for a 1-to-5 rating.", "recommendation": null, "alternates": {"html": "https://wpnews.pro/news/show-hn-test-product-ideas-on-synthetic-customers-open-source-ssr-skill", "markdown": "https://wpnews.pro/news/show-hn-test-product-ideas-on-synthetic-customers-open-source-ssr-skill.md", "text": "https://wpnews.pro/news/show-hn-test-product-ideas-on-synthetic-customers-open-source-ssr-skill.txt", "jsonld": "https://wpnews.pro/news/show-hn-test-product-ideas-on-synthetic-customers-open-source-ssr-skill.jsonld"}}