Synthetic Data Generation for Financial AI Research with NVIDIA NeMo

NVIDIA released a pipeline combining NeMo Data Designer, NeMo Curator, and Nemotron models to generate 500,000 unique synthetic financial news headlines across 13 categories for AI research. The iterative approach addresses data imbalance by generating, deduplicating, and adjusting category weights over 82 iterations, producing a dataset for trading, risk modeling, and surveillance applications.

Fine-tuning LLMs for financial natural language processing NLP is constrained by limited, imbalanced data. Real-world financial news overrepresents earnings and stock movements, while rarer events such as credit-rating changes, product approvals, and labor issues are harder to capture at scale. Synthetic generation can help fill those gaps for trading research, risk modeling, and surveillance, but diversity requires more than a single large generation run. This post walks through an iterative pipeline for generating 500,000 unique financial news headlines across 12 topics plus an “Other” category. The workflow combines NVIDIA NeMo Data Designer https://nvidia-nemo.github.io/DataDesigner/latest/ for structured generation, NVIDIA NeMo Curator https://github.com/NVIDIA/NeMo-Curator for semantic deduplication, and NVIDIA Nemotron models for high-throughput headline synthesis. It also adds a farthest-from-centroid few-shot strategy to steer each round toward novel outputs. In a naive run of 50,000 headlines, 65% were removed as near-duplicates. Instead of scaling a single batch, the pipeline iterates: generate, filter, deduplicate globally, select distinctive few-shot examples, correct category weights, and repeat until the corpus reaches the target size. Our full run produced 502,536 unique headlines across 13 categories in 82 iterations, using approximately 6 days of compute on a single 8-way NVIDIA B200 node with checkpointing, crash recovery, and SLURM job chaining. By the end, you’ll understand how to reproduce the generation-deduplication loop, adapt it for your own financial research workflows, or start directly from the open-sourced dataset. Reproducibility Versions Component | Version | |---|---| | NeMo Curator | 1.0.0rc0.dev0 | | NeMo Data Designer | 0.1.5 | | vLLM | 0.12.0 | | Model | | Table 1. Software and model versions used in this guide Hardware: Single 8-way NVIDIA B200 node. GPUs 0–3 dedicated to vLLM inference 4-way tensor parallelism, 448 concurrent requests ; GPUs 4–7 ran NeMo Curator semantic deduplication with Ray. Pipeline parameters: 35K headlines per batch 50K in early iterations , 90% cosine-similarity deduplication threshold, 500 K-means clusters, 3 few-shot examples per category, 80% cross-iteration similarity cutoff for example selection. Checkpointing: Each iteration persisted corpus state, category weights, few-shot examples, metrics, and used-example embeddings for crash recovery and SLURM job chaining. Pipeline overview Financial headline generation fails as a single-pass problem because real-world feeds are imbalanced: earnings reports and stock movements dominate, while credit-rating changes, product approvals, and other rarer events are sparse. In a 50K-headline baseline, semantic deduplication retained only 17,348 unique headlines, showing that larger batches add many near-duplicates rather than proportionally more coverage. The iterative pipeline addresses this with a closed loop. Each iteration generates a category-weighted batch, filters malformed outputs, deduplicates against the full accumulated corpus, selects diverse few-shot examples, and adjusts category weights before the next iteration. Figure 1, below, illustrates the overall architecture. The key design decision is global deduplication . Each batch is compared with all previously retained headlines, not just with itself, preventing cross-batch duplicates and preserving semantic uniqueness across the 500K corpus. Step 1. Generate headlines with NeMo Data Designer We serve Nemotron 3 Nano 30B parameters, 3B active via vLLM with 4-way tensor parallelism and 448 concurrent requests. The model’s Mixture-of-Experts MoE architecture balances throughput and quality, using 3B active parameters per forward pass within a 30B-parameter model. NeMo Data Designer orchestrates generation through a declarative configuration: a weighted category sampler across 12 topics plus “Other,” and an LLM column that generates headlines conditioned on the sampled category and current few-shot examples. Category weights are updated after each iteration through distribution correction Step 5 . Data Designer configuration: config builder = DataDesignerConfigBuilder Model config for local vLLM-served Nemotron local model = ModelConfig alias="local-nemotron", model="nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8", provider="local-vllm", inference parameters=InferenceParameters max parallel requests=448, temperature=0.95, top p=0.95, , config builder.add model config local model Category sampler with DYNAMIC weights updated each iteration config builder.add column SamplerColumnConfig name="category", sampler type=SamplerType.CATEGORY, params=CategorySamplerParams values=CATEGORIES, 12 topics + "Other" weights=current weights, Adjusted after each iteration , LLM column for headline generation with few-shot examples config builder.add column LLMTextColumnConfig name="headline", model alias="local-nemotron", system prompt="You are a financial news headline writer.", prompt=f"""Generate a realistic financial news headline \ for the category: {{{{ category }}}} {format examples for prompt examples by category } Generate a single headline for the "{{{{ category }}}}" category. Headline:""", We use high-diversity sampling temperature=0.95, frequency penalty 0.3, presence penalty 0.2 and rely on downstream filtering and semantic deduplication to remove malformed or redundant outputs. Prompt structure The prompt includes a role-setting system prompt, few-shot examples grouped by category, and an instruction for the sampled category. Data Designer injects the category at runtime through Jinja templating, and the few-shot examples are refreshed each iteration Step 4 . Prompt structure simplified : SYSTEM: You are a financial news headline writer. Generate realistic, professional financial news headlines. Output ONLY the headline text, nothing else. USER: Generate a realistic financial news headline for the category: {{ category }} Examples by category: - Earnings: "Google Parent Alphabet Posts $1.84 EPS, $63.5B Revenue" - Earnings: "Myriad Genetics Beats Q4 Estimates with 15% Revenue Surge" - Earnings: "Codec Solutions Posts Q3 Adjusted EPS $0.31, Surpasses ..." - Price Targets: "J.P. Morgan Cuts Target for Thermo Fisher, Downgrades ..." - Price Targets: "Mitsubishi UFJ Secures Buy Rating on Mitsui & Co. ..." - Price Targets: "Credence Resources Upgrades Target to $12.50 ..." - Stock price movement: "Equities Rise Broadly, Dow Gains 180 Points ..." - ... 3 examples x 12 topics + Other = up to 39 few-shot examples Generate a single headline for the "{{ category }}" category. The headline should be 10-25 words long, realistic, and match the style of professional financial news. Headline: Step 2. Quality filtering Before semantic deduplication, a lightweight rule-based pass removes malformed outputs such as truncated, runaway, or garbled headlines. Filter | Threshold | Purpose | |---|---|---| | Minimum word count | = 5 words | Remove truncated or fragment outputs | | Minimum char count | = 25 characters | Remove very short outputs | | Maximum word count | <= 100 words | Remove runaway generation | | Maximum char count | <= 1,000 characters | Catch edge cases | | Bracket count | < 5 brackets | Remove formatting artifacts | | Special character ratio | <= 25% | Remove garbled outputs | Table 2. Quality filters applied to generated headlines, with thresholds and rationale for each In practice, these filters removed fewer than 1% of headlines per batch, so most quality control comes from semantic deduplication in the next step. Step 3. Semantic deduplication with NeMo Curator This is the core quality mechanism. After each batch, the pipeline combines newly generated headlines with the accumulated corpus and runs NeMo Curator’s TextSemanticDeduplicationWorkflow . Headlines are embedded with all-MiniLM-L6-v2 , clustered with K-means, and compared within clusters using pairwise cosine similarity. Any headline above the 90% similarity threshold is removed. NeMo Curator semantic deduplication configuration: workflow = TextSemanticDeduplicationWorkflow input path=input path, output path=output path, cache path=cache path, perform removal=True, Embedding parameters text field="headline", model identifier="sentence-transformers/all-MiniLM-L6-v2", embedding max seq length=512, embedding model inference batch size=1024, Deduplication parameters n clusters=500, Higher = smaller clusters = faster pairwise eps=0.10, eps = 1 - similarity threshold 0.90 id field="id", Among duplicates, keep the more representative copy ranking strategy=RankingStrategy metadata cols= "cosine dist to cent" , ascending=True, Keep the most representative closest to centroid , pairwise batch size=4096, input filetype="parquet", output filetype="parquet", workflow.run Why 500 clusters? Pairwise similarity is expensive because comparisons scale quadratically within each cluster. At the 500K-headline endpoint, 500 clusters keep the average cluster near ~1,000 headlines and total comparisons near ~500M. Using 13 clusters, one per category, would push total comparisons toward ~19B. The higher cluster count keeps global deduplication tractable as the corpus grows. Similarity distribution monitoring After each iteration, the pipeline plots an ECDF Empirical Cumulative Distribution Function of pairwise cosine similarities. The red line marks the 0.90 deduplication threshold: a steep rise near 1.0 signals many near-duplicates, while a smoother left-shifted curve indicates broader coverage. Figure 2, below, overlays representative iterations 5, 20, 40, 60, and 82. Early curves skew right because the model repeats common patterns from limited few-shot examples. Later curves shift left and stabilize, showing that updated few-shot examples continue to push generation toward novel content even as the corpus approaches 500K headlines. Step 4. Extract diverse few-shot examples for the next iteration After deduplication, the pipeline selects few-shot examples for the next batch. The goal is to steer the model away from patterns it has already produced. Farthest-from-centroid selection K-means clustering from the deduplication step gives each headline a cluster centroid. We rank headlines by cosine distance from that centroid and select farthest examples, which are atypical within their semantic neighborhood and useful for prompting novel outputs. Cross-iteration semantic filtering Distance alone is not enough. The pipeline also compares each candidate against all previously used few-shot examples and rejects any candidate with =80% cosine similarity. This prevents semantically repeated prompt signals across iterations. Table 3, below, summarizes the selection priority. Three examples are selected per category, for 39 total per iteration, and the used-example set plus embeddings are checkpointed for recovery. Priority | Strategy | Purpose | |---|---|---| | 1. Primary | Farthest from K-means centroid | Most atypical = most informative few-shot signal | | 2. Semantic filter | < 80% cosine similarity to all used examples | Prevents semantic repetition across iterations | | 3. Fallback | Random sample from category pool | Ensures coverage when outliers are exhausted | | 4. Last resort | Seed examples manually curated | Cold-start baseline for hard-to-fill categories | Table 3. Few-shot example selection priority Semantic similarity check for few-shot candidate filtering: python def is semantically distinct candidate embedding, used embeddings, threshold=0.80 : """Reject candidates = threshold similar to ANY used example.""" if used embeddings.size == 0: return True candidate norm = candidate embedding / np.linalg.norm candidate embedding + 1e-9 used norms = used embeddings / np.linalg.norm used embeddings, axis=1, keepdims=True + 1e-9 similarities = np.dot used norms, candidate norm return np.max similarities < threshold Table 4, below, shows how selected examples become more specific by the final iteration. Category | Iteration 1 Example | Iteration 82 Example | | Earnings | Google Parent Alphabet Posts $1.84 EPS, $63.5B Revenue, Exceeds Forecasts | ThermoGlow Posts Q2 Beat, Lifts FY Guidance on Record Orders for Ultra-Low-VOC Automotive Adhesive | | M&A | Macy’s to Acquire Nordstrom’s Home Division in $1.8B Deal | Siemens Agrees to Acquire Mentor Graphics in $3.5B Deal, EU Demands Divestiture of Overlapping EDA Tools | | Credit Ratings | S&P Upgrades Moody’s Outlook for Major European Bank to Stable | Fitch Downgrades Argentina’s State Oil Company YPF to B-, Citing Fiscal Pressures and Falling Production | Table 4. Few-shot examples from iteration 1 early and iteration 82 final . Later examples feature more specific entities, niche financial instruments, and longer, more detailed structures, reflecting the pipeline’s drift toward frontier content. Step 5. Dynamic distribution correction Weighted sampling alone doesn’t keep categories on target because the model favors easier classes such as “Other” and “Stock Movement.” After each iteration, the pipeline compares target and actual category proportions, boosts underrepresented categories, clamps extremes, and normalizes weights for the next batch. Weight adjustment algorithm: python def compute adjusted weights current dist, target dist, floor=0.005 : adjusted = for category in CATEGORIES: target = target dist.get category, 0.01 current = current dist.get category, 0.0 if current 0: ratio = target / current 1 = under-represented ratio = max 0.2, min 5.0, ratio Clamp to avoid extremes weight = target ratio else: weight = target 2.0 Missing category: boost adjusted.append max floor, weight total = sum adjusted return w / total for w in adjusted Normalize to 1.0 By iteration 82, rare categories such as “Credit Ratings” and “Product Approval” closely match their 1% targets, while “Other” remains overrepresented. The correction loop still reduces model bias enough to preserve coverage across all categories. Results Pipeline run summary: The 500K target is slightly overshot because generation proceeds in fixed 35K batches and halts only after the threshold is crossed; the excess 2,536 examples are trimmed from the over-represented “Other” category. Total unique headlines: 502,536 target: 500,000 Total iterations: 82 Batch size: 35,000 headlines per iteration Total generated pre-deduplication : ~2.87M headlines Overall deduplication rate: ~82% cumulative Net new per iteration: ~5,000–6,000 unique headlines Categories: 12 broad topics + Other Similarity threshold: 90% cosine similarity Few-shot examples per iteration: 39 3 per category Deduplication efficiency Semantic deduplication was the dominant filter, and its role shifted over the run. In iteration 1, seeded by only one manual example per category, the pipeline retained 17,348 of 50K headlines; the remaining 32,652 65% were near-duplicates. Iterations 2–3 used farthest-from-centroid examples from previous outputs and maintained higher yields of 15K–17K new headlines per 50K batch, though cross-batch collisions began rising as the corpus grew. From iteration 4 onward, we reduced batch size from 50K to 35K once the deduplication drop rate exceeded ~80%. Larger SLURM jobs were mostly producing headlines discarded downstream, while smaller 35K batches delivered similar net-new yield with less wasted compute. After the corpus passed ~100K headlines, each new headline had to be distinct from a much larger pool. By iteration 40+, yield stabilized at ~5K–6K new headlines per 35K batch, or roughly 15%, with about 30K discarded per batch. Table 4 shows the transition: early duplicates are mostly within-batch repetition, while later duplicates are mostly collisions with the accumulated corpus. The steady nonzero yield near 500K shows that evolving few-shot examples continued to steer generation into novel regions. Iteration | Batch Size | Net New | Discarded | Yield | Corpus Size After | |---|---|---|---|---|---| | 1 | 50,000 | 17,348 | 32,652 | 35% | 17,348 | | 2 | 50,000 | 17,199 | 32,801 | 34% | 34,547 | | 3 | 50,000 | 15,209 | 34,791 | 30% | 49,756 | | 5 | 50,000 | 16,917 | 33,083 | 34% | 80,028 | | 42 | 35,000 | ~5,700 | ~29,300 | 16% | 268,724 | | 82 | 35,000 | ~5,155 | ~29,845 | 15% | 502,536 | Table 5. Deduplication yield across representative iterations. Yield drops from ~35% early, small corpus to ~15% late, large corpus as cross-batch collisions dominate Downstream application: Model distillation The generated dataset supports the AI Model Distillation for Financial Data developer example https://build.nvidia.com/nvidia/ai-model-distillation-for-financial-data , where a 49B–70B teacher model labels synthetic headlines and smaller 1B, 3B, and 8B student models are fine-tuned with LoRA. With 25K labeled examples, a 3B student reaches 95% of the teacher’s F1-score on this financial headline classification task. Dataset diversity matters here: semantic uniqueness and category balance expose the student to rare financial events and edge cases that are difficult to collect from real-world sources. A practical path Four principles emerged for iterative synthetic data generation with global semantic duplication: Iterate, don’t batch. A single 500K generation pass produces mostly duplicates. Eighty-two smaller iterations with evolving prompts yield a far more diverse corpus. Deduplicate globally, not locally. Each iteration must deduplicate against the full accumulated dataset to prevent cross-batch duplicates. Few-shot examples are the steering wheel. Farthest-from-centroid selection with semantic similarity filtering is the most impactful lever for generation diversity. Distribution correction is essential. Without dynamic weight adjustment, models over-generate popular categories and starve rare ones. Getting started Use the dataset — Download FinHeadlineMix on Hugging Face https://huggingface.co/datasets/nvidia/FinHeadlineMix and use it directly for headline classification, fine-tuning, or as training data for your own distillation pipeline. Run the recipe — Reproduce or extend this pipeline using NeMo Data Designer https://github.com/NVIDIA/NeMo-Curator for structured synthetic generation and NeMo Curator https://github.com/NVIDIA/NeMo-Curator for scalable semantic deduplication. Train a distilled model — Follow the AI Model Distillation for Financial Data https://developer.nvidia.com/blog/build-efficient-financial-data-workflows-with-ai-model-distillation/ developer example to fine-tune a compact student model on this dataset. Going further Nemotron-3-Nano-30B-A3B on NVIDIA Build https://build.nvidia.com/nvidia/nemotron-3-nano-30b-a3b — the MoE model used for generation, approved for distillation workflows Scaling Laws for Task-Specific LLM Distillation https://arxiv.org/abs/2606.24747 — research paper leveraging the produced FinHeadlineMix dataset https://huggingface.co/datasets/nvidia/FinHeadlineMix to derive distillation scaling laws Build Efficient Financial Data Workflows with AI Model Distillation https://developer.nvidia.com/blog/build-efficient-financial-data-workflows-with-ai-model-distillation/ — companion blog How to Build License-Compliant Synthetic Data Pipelines https://developer.nvidia.com/blog/how-to-build-license-compliant-synthetic-data-pipelines-for-ai-model-distillation/ — related reading