A system design for using Instagram and TikTok signals to source and evaluate pre-seed consumer companies, without pretending the platforms offer a commercial video firehose.
| Working-scale hypothesis | Estimate |
|---|---|
| Accounts tracked | 50,000 |
| Videos assessed per month | 500,000 |
| Marginal platform cost | ≈ $2,200/month |
| Partner-reviewed leads | ≈ 80/month |
| Marginal cost per reviewed lead | ≈ $28 |
| Share spent on model inference | ≈ 21% |
All six are working assumptions; the first phase replaces them with observed values.
Access is the defining constraint. Neither platform offers an official commercial interface for broadly enumerating arbitrary public accounts and videos. Official access exists, but it is narrow.Invert discovery. Build a bounded candidate universe from storefronts, app listings, ad-transparency surfaces, domains, and the fund's existing network; then resolve each company forward to its social handles.Separate discovery from consented diligence. OAuth and founder-uploaded originals provide the best evidence, but only after a company is already known.Use models according to their strengths. A cheap native-video model extracts structured facts. A stronger reasoning model, given only validated evidence and a short list of read-only tools, investigates each shortlisted company and writes the dossier. A human decides.Treat attention as a budget. The scheduler spends roughly 70% exploiting known signals, 20% exploring uncertain accounts, and 10% on a randomized audit stream that the ranking cannot influence.Try to kill the idea first. Before queues, databases, or workflow engines, manually test about 100 companies. Stop if the system cannot find five previously unknown, meeting-worthy companiesorcannot obtain the evidence safely and map accounts accurately.
Before the architecture, here is the output it exists to produce. LumenSkin is a fictional skincare brand, invented to show what a partner sees in the weekly digest and how the system arrived at it. Real leads will look like this; the company will not.
A partner has two hours a week for sourcing. Twenty companies arrive in a Monday digest. For each one they need to answer three questions in under a minute: why now, what is the evidence, and what could make this wrong? The card is built around exactly those questions. There is no composite score because a score hides the answer to all three.
LumenSkin entered the candidate universe when its storefront appeared in a weekly Shopify-store crawl. It was linked to its TikTok account through the link in its bio, and sat in the dormant tier for two months with ~4,000 followers and nothing unusual happening. Then, within a month, four things happened:
| Date | Observation | Where it came from |
|---|---|---|
| 19 Jul | First active ad creative for the brand appears | TikTok's public ad-transparency listing |
| 9 Aug onward | Engagement rate rises to 3.1× its own 90-day average and stays there for 11 days | |
| Metadata polls; a change-point test that fires only on a sustained break, not a single viral post | ||
| last 6 videos | "Restock" or "sold out" said or shown in 4 of them | The video model's structured read of speech and on-screen text, with timestamps |
| 2 of the 3 best videos | Made with a paid creator, not by the brand itself | Same read; flagged as an explanation to check, not a verdict |
LumenSkin · investigate
Why now:engagement broke sharply against its own baseline three weeks after its first paid ad. A 4,100-follower account is behaving like a much larger one, and sell-out language keeps recurring.
Traction:3.1× baseline engagement, sustained 11 days.(open the chart)Commercial intent:first active ad creative observed 19 Jul. Intent only; no spend or performance data exists for it.Demand language:"restock" / "sold out" in 4 of the last 6 videos.(jump to the two moments)Counter-evidence:two of the three strongest videos are a paid creator collaboration; the break may be bought reach rather than organic demand.Coverage:
full· last video read Tue 04:11 · team/execution evidence:not enough to judge· thesis version: consumer 4.2
Three things to notice. Every line links to the underlying video moment or chart, so the partner can check rather than trust. The counter-evidence is on the card, not buried; the system's job is to make the partner's judgment faster, not to argue for the company. And "not enough to judge" is stated as such; an unknown never quietly becomes a zero.
Open the evidence, watchlist it, reject it with a one-line reason, or push it into the fund's CRM. Whatever they choose is recorded and improves next week's ranking. It never becomes an automated investment decision.
Precision: of the 20 cards in a digest, how many does the partner accept?*(the weekly product metric)*Novelty: was the company already known to the fund? The system earns its keep only on the ones that were not.Conversion: accepted → meeting taken → diligence → investment, tracked over months.Recall: what did the ranking miss? Estimated from a random 10% of fetches the ranking is not allowed to influence.
PERMITTED CANDIDATE SOURCES
fund CRM / referrals · storefronts · app listings · ad creatives · domains
│
▼
COMPANY + HANDLE RESOLUTION
deterministic evidence → human review if uncertain
│
▼
SCHEDULER + HARD BUDGET GOVERNOR
70% exploit / 20% explore / 10% audit
│
▼
METADATA POLL → SELECTIVE MEDIA FETCH
│
▼
NATIVE-VIDEO PERCEPTION
cheap semantic pass → evidence-grade re-read where needed
│
▼
EVIDENCE STORE + VERSIONED THESIS SCORING
│
▼
DOSSIER: BOUNDED INVESTIGATION LOOP → CRITIC
read-only tools, step + cost caps, fail → human queue
│
▼
SLACK DIGEST / REVIEW APP / CRM HANDOFF
│
└── analyst verdicts ──┐
│
┌──────────────────────────────────────────────┘
│ feed back into the scheduler's exploit share
└► but never into the randomized audit stream
SHORTLISTED FOUNDER ── OAuth / direct original upload ──► consented diligence
The access constraintRequirements and non-goalsInvert the searchCore entitiesThe serving interfaceHigh-level designVideo perceptionThe scheduler is the productScoring without trainingEvaluation when labels arrive years lateUnit economicsDeployment and graceful degradationHow this failsSources
Neither platform offers an official interface that lets a commercial product broadly enumerate arbitrary public accounts and videos. Official access is real but narrow: creator-authorized data, known-handle enrichment, limited discovery surfaces, research-only corpora, and region-bounded ad-transparency data.
| Access path | Useful role | Critical limitation or gate |
|---|---|---|
| Instagram: authorized Professional account | ||
2known Professional usernamemedia_url
can be absent because of audio/copyright/download settings3456789The two US cases usually cited say less than they are made to say. Meta v. Bright Data was a contract ruling on the facts before that court 10, and hiQ v. LinkedIn raised early-stage questions under the US anti-hacking statute (the CFAA) before hiQ lost on contract and settled under an injunction 11. Neither grants a general right to collect and commercially analyze public social media, and neither settles copyright, music, publicity rights, privacy law, platform terms, or non-US jurisdictions.
Scaled collection therefore proceeds only under a counsel-approved source policy written per provider and region. “Public” is not a lawful basis, and a business-account label is a filter, not a permission. Each source records its basis, purpose, fields, and retention; raw media expires after 30–90 days; and deletion propagates to transcripts, embeddings, evidence frames, model uploads, caches, logs, and CRM exports. 12
At the proposed operating configuration, using batch inference and low media resolution, a 30-second video costs roughly $0.0002 to understand versus an illustrative $0.0016 vendor record price to acquire: about 8×, or roughly 3.5× at non-batch list price.
That ratio is a hypothesis, not a procurement quote. The architecture does not require it to be exact. Both costs are fractions of a cent, while almost everything fetchable is irrelevant to a consumer-startup search. The scarce resources are permitted access and well-allocated attention, not raw model tokens.
Expand requirements, constraints, and explicit exclusions
F1 · Discover: find commercially active consumer companies without depending on a platform-wide search endpoint.F2 · Resolve: connect a social account to a company, domain, app listing, legal entity where available, and founders/operators.F3 · Comprehend: extract spoken audio, on-screen text, product/category, claims, price points, calls to action, and production signals.F4 · Detect change: identify inflection against an account's own history, with a cold-start path for new accounts.F5 · Evaluate: assess against a fund-specific, versioned thesis with attributable evidence.F6 · Surface: publish a weekly digest and high-confidence inflection alerts; capture explicit verdicts.F7 · Explain: every claim points to an evidence record: source URL/ID, capture time, access class, content hash, extraction version, and deletion status.F8 · Map trends: deferred to v2; it requires a separate sample frame and clustering evaluation.F9 · Integrate: push accepted leads into the fund's existing CRM.
N1 · Cost ceiling: hard monthly and per-stage caps, cost-per-record telemetry, and a kill switch.N2 · Freshness: target inflection detection within 48 hours of observation; weekly digest; urgent alerts within an hour of detection.N3 · Supply resilience: losing one provider degrades coverage rather than silently stopping the product.N4 · Auditability: reconstruct retained knowledge and the thesis version active at the time. When source media expires, replay becomes metadata-only.N5 · Honest degradation: every digest states what is stale or missing.N6 · Privacy: purpose and access basis per field, media/derived-data expiry, and deletion from day one.N7 · Tenant isolation: thesis, scores, evidence, and leads are tenant-scoped.
The system does not make investment recommendations, value companies, touch private/logged-in data without explicit authorization, or claim coverage of deep tech, B2B, and companies without a visible consumer/commercial footprint. It compresses evidence for a human decision-maker.
The intuitive design begins inside Instagram and TikTok: crawl hashtags, sounds, and search results, then hope companies fall out. It fails structurally. There is no commercial account-listing endpoint, available discovery surfaces are limited, and nearly everything fetched is not a company.
Instead, create a bounded, measurable candidate universe and treat the social handle as an attribute of the company.
A · PLATFORM-FIRST (the intuitive design) B · COMPANY-FIRST (the inversion)
───────────────────────────────────────── ──────────────────────────────────────────
Instagram + TikTok Storefronts · app-store listings ·
hundreds of millions of accounts active ad creatives · domains · fund CRM
│ │
▼ ▼
Crawl hashtags / sounds / search Candidate company spine
through whichever vendor still works a bounded, measurable universe
│ │
▼ ▼
Hope a real company falls out Resolve forward to the social handle
link-in-bio · bio domain · ad-creative
match · name search → ~50k tracked accounts
Cost scales with the haystack. Cost scales with the shortlist.
>99% of what you pay to fetch is Registries are enumerable by construction,
not a company. though they carry gaps and lag.
This is not a claim of completeness. Storefront, app-store, ad, and domain datasets have coverage gaps, lag, and composition bias. The pilot measures unique yield, overlap, and bias by source.
An early brand with a newly observed active creative has revealed commercial intent before it necessarily has a press mention, database record, or large following. It is a binary intent signal, not spend, performance, or proof of stage.
Discovery uses the fund's own pipeline, referrals, storefront/app/domain/ad seeds, and a controlled exploration sample. Consented diligence uses OAuth, creator-authorized aggregators, or direct upload of originals after a company is shortlisted.
Consent cannot carry the top of the funnel: connected founders already know about the fund and form a self-selected sample. Its value is higher-fidelity diligence and a possible data flywheel. A free benchmark report can motivate connection; over time, connected first-party accounts can replace collected records as the cohort base. The pilot measures whether this substitution actually occurs.
Expand the domain model and audit semantics
Two choices matter more than the field list: facts are observations rather than overwritten columns, and every acquisition attempt is itself observable.
| Entity | Holds | Important invariant |
|---|---|---|
Company |
||
| Canonical identity, domain, legal entity, category, app listings, resolution confidence | Created from company evidence, not asserted by a video | |
SocialAccount |
||
| Platform, handle, type, follower history, company link and link confidence | Many-to-one; uncertain links go to review | |
ContentItem |
||
| Platform ID, publication time, duration, media pointer, metric snapshots | Metrics are time-stamped observations | |
Extraction |
||
| Structured perception result plus model, prompt, and schema versions | Re-runnable; old extractions are retained while policy permits | |
Observation |
||
Subject, predicate, value, source, observed_at , valid_from |
||
| Separates when something became true from when the system learned it | ||
Signal |
||
| Velocity, trend break, anomaly, cohort-relative measures | Reproducible computation | |
Thesis |
||
| Hard filters, weighted dimensions, exemplars, effective dates | Data and version history, not scattered code | |
Score |
||
| Company, thesis version, sub-scores, total, provenance | Never overwritten | |
Lead |
||
| Surfaced candidate, state, assignee, digest/alert | Unit of fund workflow | |
Verdict |
||
| Analyst decision, reason, timestamp | Explicit feedback; missing feedback is not negative | |
SourceRun |
||
| Provider, cost, records, conformance, outcome | Makes provider economics and health measurable | |
DossierRun |
||
| Tool calls made, facts retrieved, cost, critic verdict, revision count | Every investigation loop is replayable and capped |
This supports the question a partner will eventually ask: “Why did we not see this company in March?” The answer is scoped by retention rights. Retained facts, extractions, and scores can be replayed; expired/deleted media leaves a metadata-only audit trail.
Expand the proposed read-mostly, provenance-bearing interface
| Method and path | Purpose |
|---|---|
GET /v1/leads |
|
| Ranked/sorted candidates filtered by thesis, time, status, category, and coverage | |
GET /v1/companies/{id} |
|
| Resolved dossier and current evidence dimensions | |
GET /v1/companies/{id}/timeline |
|
| Point-in-time replay within retention rights | |
GET /v1/companies/{id}/evidence |
|
| Source, capture time, access class, hash, extraction version, deletion status | |
POST /v1/companies/{id}/memo |
|
| Draft investment-committee memo; asynchronous because it runs the bounded evidence-gathering loop described in §6 | |
POST/PATCH /v1/theses |
|
| Create a thesis or mint a new immutable version | |
POST /v1/leads/{id}/verdict |
|
| Analyst decision and reason code | |
GET /v1/trends |
|
| v2 only: concept clusters and member companies | |
GET /v1/coverage |
|
| Provider health, degradation state, and stale/missing coverage | |
POST /v1/watchlist |
|
| Force a company/handle into the hot tier |
Events worth interrupting someone for: lead.surfaced
, company.inflection
, and coverage.degraded
.
Every claim combines value, status/uncertainty, and provenance. The example below is intentionally annotated pseudo-JSON; the wire format is strict JSON.
{
"lead_id": "ld_8f21",
"company": {
"id": "co_44c1",
"name": "LumenSkin",
"resolution_confidence": 0.94,
"resolved_via": ["link_in_bio", "ad_library"]
},
"score": {
"thesis_version": "consumer_v4.2",
"components": [
{
"name": "traction_inflection",
"value": 91,
"method": "cusum",
"evidence": ["sig_velocity_9931"]
},
{
"name": "product_differentiation",
"value": 64,
"method": "rubric_llm",
"evidence": ["vid_77a2@00:12", "vid_81b0@00:04"]
},
{
"name": "execution_evidence",
"status": "insufficient_evidence",
"evidence": []
}
]
},
"coverage": {
"level": "full",
"last_full_fetch": "2026-08-20T04:11Z"
}
}
Missing evidence never silently becomes zero. Zero is a judgment; absence is not.
The v1 product is a company pipeline. Trend mapping is explicitly v2 because it requires a different sample frame and clustering evaluation; a corpus selected for company ranking cannot honestly establish that a concept is “rising.” Vector indexing arrives with that second product, not before.
The pipeline runs top to bottom, from supply → control → perception and identity → assessment → delivery, as drawn on the cover.
Supply: provider adapters normalize records and report success, cost, latency, freshness, and schema conformance. Adapters reduce integration cost; they do not make correlated platform/legal risk disappear.Scheduler: owns the fetch decision and every spend cap.Workflow: the pilot runs on scripts. A durable workflow engine is adopted only when jobs become multi-day, resumable, and externally flaky. Before that, a queue plus idempotent state machine is enough.Perception: turns media into structured observations; no downstream tool consumes raw creator instructions.Entity resolution: deterministic evidence can auto-link only above a calibrated precision threshold. Model-proposed links always require human review; uncertain candidates are queued, not dropped.Storage: Postgres as system of record; S3-compatible object storage with aggressive lifecycle rules. No separate vector database in v1.Assessment: reproducible arithmetic for quantitative signals; a stronger reasoning model, running a bounded investigation loop, for qualitative thesis fit and the dossier.Serving: Slack for habit, a review surface for depth, and the fund's existing CRM for adoption.
Most of this pipeline is a workflow: fixed stages, fixed order, the model filling a schema at each one. That is a choice, not an omission. Every stage that touches creator media is tool-less, because captions, audio, and frames are untrusted input and a model with tools is a model that can be talked into using them. The scheduler is a stated policy with a spend governor, because an agent deciding what to crawl would make cost unbounded and would quietly break the randomized audit stream. Agency is reserved for the one step whose path cannot be written in advance: diligence on a shortlisted company.
The dossier is a bounded investigation, not a summary. Which checks matter depends on the company. For the LumenSkin example, the relevant checks include whether the creator collaboration was disclosed as paid, whether the storefront's product count grew, and how app-store review velocity compares with its cohort. So the reasoning model runs a short loop with a small, read-only tool set: query the timeline, extractions, and cohort statistics in the evidence store, and call the same metered enrichment adapters the supply layer already uses (domain records, app-store listings, ad-transparency lookups). Hard limits: at most a dozen tool calls, a per-dossier cost cap, no raw media, no open web, every retrieved fact carries its provenance pointer into the card.
A separate critic checks the card before a partner sees it. A second prompt, distinct from the writer and ideally using a different model, verifies that every claim points to evidence, counter-evidence is present, unknowns are marked as insufficient rather than scored, and no prohibited proxies (appearance, accent, demographics) leaked in. One revision round; a second failure routes the card to a human queue rather than to the digest. This is what turns the document's promises about provenance and counter-evidence into something enforced rather than hoped for.
Not proposed: multi-agent swarms. The task decomposes into sequential stages, not independent research threads, and parallel agents cost several times the tokens while making verification harder because each one saw only a slice.
Each stage records provider, exact model, prompt version, output schema, parameters, token use, and cost. Changes are eval-gated. The interface does not hide provider capabilities such as media resolution. Model selection also does not magically solve data residency: storage, logs, workflow history, providers, and delivery surfaces all matter.
Expand model selection, token assumptions, cascade, and security
Google's 2.5-generation video documentation describes approximately 1 FPS, 66 tokens per low-resolution frame, 258 per default-resolution frame, and 32 audio tokens per second. 13 These assumptions are generation-specific and must be remeasured before implementation.
The 2.5 Flash-Lite model accepts video, but its model page does not enumerate media_resolution
as a parameter. Before writing the pipeline, call count_tokens
on representative clips and verify that media_resolution: low
is honored end to end. If not, select the cheapest model that does honor it and rerun every cost number.
| One 30-second clip | Approximate cost | Media work owned by the system |
|---|---|---|
| Gemini 2.5 Flash-Lite, low, 3k input / 400 output | $0.00046 | Retrieve, validate/hash, upload, delete |
| Same via Batch API | $0.00023 | Same path; asynchronous, half price |
| Gemini 2.5 Flash, default, ~9k input | $0.0030 | Same path |
| Claude Sonnet 5, five frames + transcript | $0.015 | Frame selection, transcode, duplicate detection, speech-to-text, on-screen text recognition |
| TwelveLabs index + analysis | $0.021 | Separate index and lifecycle |
Prices as of August 2026. 14 15
Native ingestion does not mean “no pipeline.” It collapses a large custom ASR/OCR/keyframe subsystem into a thinner path plus an exception lane for unsupported codecs, corrupt media, truncated downloads, expired URLs, and provider-file deletion.
Tier 1: semantic pass. Run cheaply across ingested video. Determine whether it represents a commercial entity, category, product, call to action, and coarse signals.Tier 2: evidence pass. Promote roughly 5% when a price, claim, URL, or other dossier-bearing detail needs an evidence-grade reread. At 1 FPS and low resolution, tier 1 can miss rapid edits and fine text.
The promotion percentage is a budget assumption, not a universal sampling rate.
{
"is_commercial_entity": true,
"entity_kind": "dtc_brand",
"product": {
"category": "skincare/serum",
"price_point": 34.0,
"currency": "USD",
"evidence_ts": ["00:07", "00:19"]
},
"claims": [
{ "text": "…", "kind": "efficacy", "ts": "00:11" }
],
"commerce_signals": {
"cta": "link_in_bio",
"sold_out_mentioned": true
},
"field_status": {
"price_point": "observed",
"founder_present": "inferred",
"url_shown": "missing"
},
"corroborating_items": 3
}
There is deliberately no model-generated confidence: 0.86
. A self-reported number is not calibrated. Per-field observed | inferred | missing
status and corroboration counts are actionable.
Creator media is untrusted data. The perception stage has no tools, fills a strict schema, sanitizes URLs/text, and cannot follow instructions embedded in captions, audio, or frames. Adversarial multimodal prompt-injection cases belong in the release gate. Derived transcripts and embeddings follow the same retention/deletion policy as their source.
Once per-video comprehension is tiny, advantage comes from choosing the right 500,000 videos rather than processing the wrong five million.
The operating rule is poll broadly and shallowly; fetch deeply only on evidence. Accounts move between hot, watch, and dormant tiers. Promotions can come from an account-level change detector or lower-cost external evidence: a newly observed creative, app-store rank movement, domain launch, or press event.
┌─────────────────────────┐
│ tier assignment │
│ hot / watch / dormant │
└────────────┬────────────┘
│
▼
cheap metadata observation
│
┌─────────────────┴─────────────────┐
│ │
▼ ▼
account/cohort change external trigger
│ │
└─────────────────┬─────────────────┘
▼
budget-gated media fetch
│
▼
perception + assessment
│
▼
promote / hold / demote
If scores determine which accounts receive more observation, early favorites accumulate evidence and remain favorites. Ranking quality becomes impossible to estimate because ignored accounts never become observable.
Partition the fetch budget:
| Allocation | Purpose |
|---|---|
| ~70% exploit | |
| Score/evidence-driven accounts likely to change the partner shortlist | |
| ~20% explore | |
| New, thin-history, or out-of-cohort accounts the current score does not understand | |
| ~10% randomized audit | |
| Coverage/recall estimate; untouched by scores and analyst verdicts |
Every fetch records the policy that selected it. Start with fixed quotas. Consider an adaptive allocation policy (a contextual bandit) only after unbiased verdict data exists.
A change detector needs history; a CUSUM test against an account's own 90-day baseline cannot serve a new account. Cold-start accounts use age-normalized post velocity and outlier-resistant cohort percentiles (median and median absolute deviation within follower band and category) until enough history exists. The 48-hour freshness objective begins when the system starts observing, not when the company was created.
Expand the rules-first score, thesis model, and bias controls
A learned ranker is premature today: high-quality labels are scarce, delayed, and selected by prior human attention. Rules-first is a sequencing decision, not a belief that arithmetic is inherently truthful. The verdict/audit streams are designed so a learned ranker can become defensible later.
| Dimension | Method | Rationale |
|---|---|---|
| Traction inflection | Change-point detector (CUSUM) after warm-up | Detects a sustained break from the account's own baseline |
| Cohort-relative growth | Robust cohort normalization | Avoids comparing raw engagement across unlike accounts |
| Commercial intent | Counting verified extracted fields | CTA, restock/sell-out language, price points, SKU breadth |
| Engagement anomaly | Heuristics with alternative explanations | Keeps suspicious patterns visible without alleging fraud |
| Product differentiation | Evidence-citing rubric model | Qualitative judgment against a written, versioned definition |
| Execution evidence | Evidence-citing rubric model | Shipping cadence, customer specificity, domain expertise, verifiable prior work |
| Thesis fit | Few-shot rubric model | Positive and negative exemplars from the fund's own decisions |
Appearance, accent, charisma, and demographic proxies are not investment evidence. The system scores only dated, attributable execution evidence: what the team demonstrably did.
Every dimension reports evidence coverage. insufficient_evidence
is distinct from zero.
A thesis is a versioned record containing hard filters, weighted dimensions, natural-language definitions, and positive/negative exemplars. In-context examples provide immediate, auditable personalization without fine-tuning. Whether negative exemplars materially improve ranking is a hypothesis tested by ablation, not asserted as fact.
An engagement anomaly suppresses confidence and exposes evidence plus benign alternatives: giveaway, paid boost, collaboration, or a single viral post. The system never makes an automated fraud allegation.
Expand the three evaluation horizons and backtest design
Whether a company was a good investment may take three to seven years to observe. That is not a usable product loop. Separate extraction correctness from investment judgment.
MINUTES DAYS MONTHS / YEARS
──────── ──── ──────────────
Golden videos + entities → Analyst verdicts → Fund-action funnel
Field-level extraction eval explicit reason codes meeting / diligence / invest
Prompt/model/vendor gates precision of top 20 + novelty proxy momentum kept separate
randomized audit stream estimates what ranking missed
- About 300 hand-labeled videos and 100 hand-resolved companies.
- Stratified by platform, language, duration, edit style, category, and hard negatives.
- Double-label a subset and report agreement rather than assuming ground truth is obvious.
- Measure category/claim extraction, price/OCR fidelity, timestamp tolerance, account resolution, latency, and cost.
- Include prompt injection, unsupported media, and source-schema drift.
- Dossier golden cases: planted unsupported claims, missing counter-evidence, and prohibited proxies that the critic must catch before a card ships.
The weekly digest captures explicit accept/reject/insufficient-evidence reasons. Precision of the top 20 (how many digest items the partner accepts) is useful but biased: only shown items receive judgment, and a non-click is not a negative. Read it beside:
- randomized audit performance,
- novelty rate: previously unknown to the fund,
- meeting-request and meeting-taken conversion,
- reject reasons and coverage,
- partner time per accepted lead.
Press, hiring, launch, or fundraising at 90–180 days are proxy momentum, recorded point-in-time. The primary product funnel is:
previously unknown → partner accepted → meeting taken → diligence → investment
Replay approximately 50 known 2024–25 cases plus matched same-vintage controls that did not raise, using only features available at the historical cutoff. Controls prevent survivorship theater; the cutoff stops the backtest from quietly using information that did not exist yet. Do this once as a cold-start sanity check rather than industrializing expensive historical social data.
Expand the assumptions, reconciliation, and 10× scenario
Illustrative monthly envelope: 50,000 tracked accounts and 500,000 videos.
| Stage | Volume assumption | Monthly cost |
|---|---|---|
| Metadata polls | 1.5M polls; least-sourced price | $150 |
| Media acquisition | 500k vendor records at ~$0.0016 | $800 |
| Perception tier 1 | 500k batch/low-resolution videos | $100 |
| Perception tier 2 | 25k default-resolution rereads | $75 |
| Entity resolution and enrichment | ~8k companies | $200 |
| Dossiers | ~250 surfaced candidates; bounded investigation loop + critic at ~$1.20 | $300 |
| Infrastructure, storage, egress | Postgres, object storage, workflow, compute | $600 |
| Total | ||
| $2,225 ≈ $2,200 |
The three model lines total $475, or about 21%; the dossier loop, not video comprehension, is most of it. A 20-item weekly digest means roughly 80 partner-reviewed leads/month, so marginal platform cost is about $28 per reviewed lead. The system may surface roughly 250 candidates through digest and alerts, but dividing by a count nobody reviewed would create a vanity denominator.
- The $0.0016 acquisition input is a marketplace per-record price. End-to-end cost per successfully assessed video (including valid media bytes, retries, failures, egress, and missing-media records) is unmeasured.
- The metadata-poll price is the least-supported line and varies by provider.
- Infrastructure is an estimate, not a quote.
- These are marginal platform costs, not all-in economics. Human review, annotation, vendor minimums, legal/procurement, engineering, and maintenance are excluded.
The first phase measures each unknown directly.
At five million videos/month, the estimate is roughly $16.5k:
- metadata polls: ~$1.5k,
- media acquisition: ~$8k,
- perception: ~$1.75k,
- entity resolution/enrichment: ~$2k,
- infrastructure: ~$3k,
- dossiers: still about ~$300 because partner review capacity, not video volume, bounds the number produced.
Entity resolution becomes human-bound before compute-bound. Time-series signal data may need partitioned or analytics-oriented storage. Inference falls to around 12% because dossier volume is pinned by human capacity while acquisition scales with volume.
Expand the operational design, failure modes, and user-visible degradation
Choose one cloud rather than pretend to hedge clouds: AWS in this design. ECS Fargate, RDS Postgres, S3 lifecycle policies, a managed durable workflow engine once justified, managed secrets, Terraform, and two environments. No Kubernetes for a ten-seat product.
OpenTelemetry reports cost per record beside latency, freshness, success rate, and errors.
| Failure | Detection | Response |
|---|---|---|
| Provider unavailable, enjoined, or blocks access | Source-run success, freshness, sentinel coverage | Circuit-break; use an independently verified source if available; degrade visibly |
| Platform markup/schema changes | Schema conformance and field-fill shifts | Quarantine adapter; alert and patch |
| Rate limits tighten | 429 ratio and completion lag | Backoff; demote tiers; spend on fewer accounts |
| Model changes/deprecates | Canary/golden-set regression | Eval-gated migration; never automatic upgrade |
| Cost spikes | Per-stage cost telemetry | Stage cap trips before monthly cap; kill switch |
| Entity resolution drifts | Confidence and disagreement distributions | Widen human-review band; never widen auto-merge |
| Engagement anomalies shift | Cohort distribution | Recalibrate; surface uncertainty |
| Prompt injection/unsafe media | Schema/moderation/adversarial flags | Tool-less extraction; quarantine; the dossier loop sees validated fields only and its tools are read-only |
| Dossier loop overruns or wanders | Step count, per-dossier cost, critic rejections | Hard-stop at the cap; card routed to human queue, never silently shipped |
| State | Behavior | What the partner sees |
|---|---|---|
full |
||
| Sources healthy; media fetch active | Normal digest | |
metadata_only |
||
| No new media; prior extractions and current metadata | “No new video since Tue; scores use cached extractions” | |
cached_only |
||
| Prior state only; discovery stopped | “Discovery d since Tue; this is not a quiet market” | |
halted |
||
| Kill switch or legal hold | Explicit stop notice |
Silent thinness destroys historical trust. Coverage is therefore a product field, not an observability footnote.
The timing problem. By the time a social breakout is visible, other investors may already be circling. If the detector surfaces only obvious companies, this is an expensive newsletter.A structural blind spot. Social-native discovery favors consumer, creator-led, and commercially expressive companies. It is not a general startup-discovery engine.Label starvation. Component evals prove the machine works; they do not prove the investment thesis is right. Genuine outcomes remain slow and sparse.Vendor fragility. Provider adapters do not protect against correlated platform, contractual, or regulatory failure. Consented data reduces dependency but cannot replace unknown-company discovery.Adversarial drift. If the signals become known and valuable, founders will optimize for them. The signal set must be refreshed rather than tuned once.The “so what?” problem. A fund that takes no new meetings gains nothing from twenty weekly leads. The review habit, reason codes, CRM handoff, and partner time are part of the architecture.
The two decisions I hold most loosely are:
- whether company/commerce-first seeding covers enough of the fund's real opportunity set, and
- whether pre-breakout detection is statistically real at the available data density rather than noise with extra steps.
Both have explicit pilot tests. Everything else is a hypothesis whose falsification condition should remain visible.
Platform, pricing, and legal claims last checked 22 August 2026.
Instagram API with Instagram Login: authorized Professional-account access and review tiers.Meta documentation
[
2. ]Instagram Business Discovery: known-handle fields and cases where media_url
is omitted. Meta documentation
[ 3. ]Instagram Hashtag Search and Public Content Access: 30 unique hashtags per seven days and approved-use requirements. Hashtag Search · Public Content Access
[ 4. ]TikTok Display API: creator-authorized profile and video metadata. TikTok documentation
[ 5. ]TikTok Data Portability: consented EEA/UK export; posts/profile data may include media download URLs. TikTok documentation
[ 6. ]Research-only corpora: commercial-user exclusion for TikTok Research API and controlled Meta Content Library access. TikTok FAQ · Meta Content Library via ICPSR
[ 7. ]Ad transparency: active ads versus richer topic/region-bounded transparency. Meta Ad Library · TikTok Commercial Content API
[ 8. ]Provider pricing/limitations: marketplace record prices and examples where media download is excluded. Bright Data Instagram scraper · Apify Instagram scraper
[ 9. ]Platform terms: automated collection restrictions. Instagram Terms · TikTok US Terms
[ 10. ]* Meta Platforms v. Bright Data:* January 2024 summary-judgment order addressing contract claims on the facts before the court.
[ 11. ]* hiQ Labs v. LinkedIn:* Ninth Circuit preliminary-injunction opinion on CFAA authorization.
[ 12. ]Public data and privacy: joint regulator statement that publicly accessible personal information remains subject to privacy law. ICO joint statement (PDF)
[ 13. ]Gemini video understanding/media resolution: sampling, tokenization, fast-action caveat, and generation-specific resolution behavior. Video understanding · Media resolution
[ 14. ]Gemini pricing: Flash-Lite/Flash input-output rates and Batch discount. Google pricing
[ 15. ]Anthropic model pricing and modalities: Sonnet 5 and Opus 5 pricing and supported input types as of the checked date. Anthropic pricing