Open source tool for analyzing your social media data (want to help me make it better)?

Developer Chris Soria released cat-vader, an open-source tool for analyzing social media data that integrates multi-label LLM labeling with platform-specific ingestion, and is seeking community feedback to improve the pipeline. The tool currently supports Bluesky analytics and offers provider-agnostic LLM backends with ensemble voting, positioning itself as a practical solution for researchers and analysts.

for now: Where cat-vader sits in the ecosystem and why it’s interesting cat-vader is effectively a “data → fetch → label multi-label → analyze” pipeline with optional category discovery and provider-agnostic LLM backends plus ensemble voting . Your writeup also highlights a very “social platforms” reality: Bluesky’s API reports views = 0 , so engagement analysis ends up centered on likes/replies rather than reach . Chris Soria https://christophersoria.com/posts/2026/03/catvader-bluesky-analysis/ That combination platform ingestion + schema-driven multi-label LLM labeling + downstream analysis has a lot of surface area to improve—and plenty of adjacent projects to borrow patterns from. Similar projects/attempts worth studying what to borrow 1 LLM-assisted labeling + review workflows human-in-the-loop These tools focus on the practical “label at scale, then audit/correct” loop—exactly what social media coding needs. Label Studio prompt-centric + automation / prelabeling : strong patterns for prelabel → human verify → export , plus prompt workflows for reducing copy/paste overhead. Label Studio https://labelstud.io/blog/automate-data-labeling-with-llms-and-prompt-interface/ Argilla + spaCy-LLM tutorials: good reference architecture for “LLM suggestions stored alongside records” and iterative improvement of label quality. Argilla https://docs.v1.argilla.io/en/latest/tutorials/notebooks/labelling-spacy-llm.html Prodigy + LLM recipes: practical “correct the LLM” workflows designed for annotation speed paid tool, but the docs have reusable concepts . prodi.gy https://prodi.gy/docs/large-language-models What to borrow for cat-vader - A first-class review UI loop even a lightweight one: streamlit/gradio where users can correct ambiguous labels and export a “gold” set. - Treat the LLM as a suggestion engine + uncertainty flagger , not just the final labeler. 2 Synthetic data + scalable LLM pipeline orchestration Distilabel Argilla : opinionated framework for building reliable, scalable pipelines for synthetic data and AI feedback. Useful patterns for batching, caching, evaluation hooks, and reproducibility. GitHub https://github.com/argilla-io/distilabel LLM Tool: a “research-friendly” end-to-end pipeline for annotating text datasets, tracking benchmarks, and training classifiers. GitHub https://github.com/antoinelemor/LLM Tool What to borrow - Pipeline primitives: stages , artifacts , run manifests , cached intermediate outputs , evaluation summaries . 3 Weak supervision + ensembles conceptual match to “multi-model vote” Snorkel data programming : classic approach to combining multiple noisy labeling sources heuristics, models into a better latent label estimate. PMC https://pmc.ncbi.nlm.nih.gov/articles/PMC7075849/ Prompted weak supervision Alfred : prompts as labeling functions; relevant if you expand “auto category discovery” into “prompt library of heuristics.” ACL Anthology https://aclanthology.org/2023.acl-demo.46.pdf What to borrow - Instead of plain majority vote, add an optional “learned combiner” even a simple Dawid–Skene-style weighting so stronger models or historically reliable models count more on specific categories. 4 Social-media-specific NLP baselines non-LLM, but critical for validation TweetNLP: packaged classifiers for social-media tasks topic, sentiment, emotion, hate/offensive, irony . PyPI https://pypi.org/project/tweetnlp/ BERTweet: canonical tweet-domain pretraining baseline. GitHub https://github.com/VinAIResearch/BERTweet TweetEval benchmark: widely used evaluation suite for tweet classification tasks. Hugging Face https://huggingface.co/cardiffnlp/twitter-roberta-base-irony What to borrow - Use at least one strong non-LLM baseline or task-specific transformer as a sanity check for stability and drift: “Do LLM labels behave wildly differently than a tuned domain model on the same data?” 5 Bluesky / AT Protocol ingestion patterns - Official Bluesky developer docs SDK pointers . Bluesky Documentation https://docs.bsky.app/docs/get-started - Community-maintained atproto Python SDK broad coverage of the protocol . GitHub https://github.com/MarshalX/atproto What to borrow - Standard handling for pagination/cursors , session reuse , rate limits , and consistent “record schemas” across endpoints. Common pitfalls that show up in your exact use case A Engagement modeling pitfalls No impressions / reach on Bluesky views = 0 means “likes” conflate content appeal with distribution mechanics you can’t observe. In your writeup you correctly restrict conclusions to likes/replies. Chris Soria https://christophersoria.com/posts/2026/03/catvader-bluesky-analysis/ - Heavy-tailed engagement: your approach uses log likes + 1 and shows that adding account fixed effects jumps R² from ~18% to ~61.9% . That’s a good illustration of “creator identity dominates.” Chris Soria https://christophersoria.com/posts/2026/03/catvader-bluesky-analysis/ - Simpson’s paradox / composition effects: your “economy tanks engagement” disappears once you control for who posts—exactly the kind of thing social media analysis constantly hits. Chris Soria https://christophersoria.com/posts/2026/03/catvader-bluesky-analysis/ Implication for cat-vader: the library should make “composition controls” fixed effects, stratified reports easy and default-ish. B LLM labeling pitfalls - Prompt + schema choices can create systematic label bias e.g., majority-label bias in-context , especially for multi-label setups where “Other” and borderline categories are frequent. arXiv https://arxiv.org/html/2312.16549v1 - Small category sets can look stable while actually drifting label semantics shift run-to-run . C Product pitfalls practical adoption Your repo already emphasizes best practices like “detailed category descriptions,” and notes that chain-of-thought / step-back prompting didn’t consistently help. GitHub https://github.com/chrissoria/catvader Suggestions for cat-vader prioritized, concrete 1 Tighten “first 15 minutes” usability docs + API consistency Why: this is the 1 adoption lever for open source tooling. Unify parameter naming across README + posts + code. Your writeup example uses sm posts=250 , but the public API reference is sm limit . That’s the kind of mismatch that creates immediate friction. Chris Soria https://christophersoria.com/posts/2026/03/catvader-bluesky-analysis/ Update repo docs that still reference CatLLM paths/names. ARCHITECTURE.md describes src/catllm/ and modules like summarize , while the cat-vader changelog states summarize was removed and the package renamed. CONTRIBUTING.md is titled “Contributing to CatLLM” and points to cat-llm commands/URLs. GitHub https://github.com/chrissoria/catvader/blob/main/ARCHITECTURE.md Make supported platforms list consistent. README lists "threads" , "bluesky" , "reddit" , "mastodon" , "youtube" , while the changelog includes LinkedIn support. Choose one source of truth and generate the others from it. GitHub https://github.com/chrissoria/catvader Deliverable idea: a single docs/reference.md generated from docstrings or vice versa so the examples can’t drift. 2 Fix credential handling and remove machine-specific paths In social media.py there is a hardcoded ENV PATH pointing to a local “Documents/Important Docs/…” location, and error messages instruct setting env vars there. That will break for essentially everyone except you. GitHub https://github.com/chrissoria/catvader/blob/main/src/catvader/ social media.py What to do This is a high-impact, low-effort PR. 3 Make ingestion robust pagination, rate limits, retries, provenance For Bluesky you already handle cursor pagination and optionally authenticate; good. GitHub https://github.com/chrissoria/catvader/blob/main/src/catvader/ social media.py Next improvements that matter in real-world runs: Add standardized retry/backoff 429/5xx across all platforms with jitter. Persist provenance columns endpoint used, fetched at, cursor/page, auth used yes/no . Normalize schema across platforms e.g., unify “quote/repost/share” semantics per platform and document what is missing or always-zero—Bluesky views/shares are always 0 . GitHub https://github.com/chrissoria/catvader/blob/main/src/catvader/ social media.py 4 Treat labeling as an experiment: add “auditability” by default Right now cat-vader supports multi-model ensemble voting and returns per-model outputs + consensus columns. That’s a strong base. GitHub https://github.com/chrissoria/catvader Add these defaults so users can trust results: - Store the prompt + schema + model config used model name, provider, temperature/creativity, thinking budget in a run manifest JSON saved next to outputs. - Add a “disagreement report”: - rows where models disagree, - categories with low agreement, - “most confusing pairs” A vs B . - Add a small “gold set” evaluator : - user supplies 100 hand-coded posts → cat-vader outputs precision/recall per label + calibration plots even basic . This matches what labeling platforms and weak supervision systems emphasize: the workflow is label → check → refine , not “label once.” Label Studio https://labelstud.io/blog/automate-data-labeling-with-llms-and-prompt-interface/ 5 Improve category discovery so it produces reusable taxonomies Your approach descriptions per category + “auto” category discovery is good. GitHub https://github.com/chrissoria/catvader Two upgrades make it more “research-grade”: This mirrors shared tasks that treat social labels as multi-label and often hierarchical. ceur-ws.org https://ceur-ws.org/Vol-4038/paper 147.pdf 6 Ship opinionated “analysis helpers” for the exact effect you found Your headline result—identity dominates engagement variance R² ~61.9% with fixed effects —is compelling because it’s the right model for this setting. Chris Soria https://christophersoria.com/posts/2026/03/catvader-bluesky-analysis/ Codify that into the package: Also: cat-vader already started adding useful covariates like day , month , hour , n posts that day , post length , contains url , contains image . Make those first-class in analysis helpers. GitHub https://github.com/chrissoria/catvader A practical “next PRs” roadmap Highest leverage adoption + correctness - Remove hardcoded ENV PATH ; standardize dotenv/env handling. GitHub https://github.com/chrissoria/catvader/blob/main/src/catvader/ social media.py - Fix doc drift sm posts vs sm limit , CatLLM references, supported platforms list . Chris Soria https://christophersoria.com/posts/2026/03/catvader-bluesky-analysis/ Next trust + research usefulness - Add run manifests + disagreement reports + “gold set” evaluator. GitHub https://github.com/chrissoria/catvader - Add engagement helpers that default to fixed-effects comparisons codify your main analytic insight . Chris Soria https://christophersoria.com/posts/2026/03/catvader-bluesky-analysis/ Later scale + extensibility - Optional learned ensemble combiner Snorkel-style weighting . arXiv https://arxiv.org/abs/1711.10160 - Small review UI loop Label Studio/Argilla-inspired patterns . Label Studio https://labelstud.io/blog/automate-data-labeling-with-llms-and-prompt-interface/ Reading list directly relevant to your tool Snorkel / weak supervision core ensemble theory : Ratner et al. Snorkel . PMC https://pmc.ncbi.nlm.nih.gov/articles/PMC7075849/ Prompted weak supervision: Alfred prompted labeling functions . ACL Anthology https://aclanthology.org/2023.acl-demo.46.pdf LLM-in-the-loop labeling workflows: Label Studio prompt-centric workflow; Argilla + spaCy-LLM tutorial. Label Studio https://labelstud.io/blog/automate-data-labeling-with-llms-and-prompt-interface/ Social media NLP baselines: TweetNLP + BERTweet + TweetEval references. PyPI https://pypi.org/project/tweetnlp/ Bluesky ingestion: Bluesky docs + atproto Python SDK. Bluesky Documentation https://docs.bsky.app/docs/get-started A “north star” framing for cat-vader If cat-vader becomes the tool where a researcher can: - fetch posts from a platform, - label them with auditable multi-label outputs, - review disagreements quickly, - produce fixed-effects-aware engagement results by default, …then it’s not just “LLM classification,” it’s a reproducible social media coding workbench —and your Bluesky result identity dominates; content still matters within identity becomes a built-in, repeatable analysis template. Chris Soria https://christophersoria.com/posts/2026/03/catvader-bluesky-analysis/