How to Use FocalPrompt #
Getting Started
Build an ordered inference scenario
Add system, developer, user, and assistant message cards in the exact order sent to the model. Instruction messages must come first and at least one user message is required.
1. Label foci in Analyse messages; keep chat as Retain
Foci are grounded only in **Analyse** messages. **Retain** messages stay unchanged across experimental arms. Their optional **Input name** identifies a batch input column, not the message itself.
For single runs, the editor text is used directly; you can leave Input name blank. Message IDs are managed automatically and preserved with saved analysis.
Tag the Analyse content into points of focus or "foci" and aim for close to **100% coverage**. The model can propose foci, but review the exact message-relative spans before running experiments.
2. Predict prospective focus
Before sampling, the baseline model uses the full scenario and retained chat to predict a 100% budget across the labelled foci. Each focus gets a short justification. No generated outputs are provided.
3. Generate baseline outputs and inspect noise
Default: 10 independent outputs at temperature 0.7. Both settings are configurable. Ordered roles, retained input and any strict JSON Schema contract stay intact. Inspect pairwise cosine distances, the projected similarity map and candidate output groups. Grouping is exploratory; ten outputs cannot establish the true number of modes. 4. Assess retrospective focus (optional)
The same model independently assesses each output against the full scenario. Each assessment allocates 100% and explains each focus. The prediction is withheld to avoid anchoring. Compare the per-output results and their mean with the prospective budget, using percentage-point differences. These are behavioural self-reports, not transformer attention weights.
5. Ablate and compare
Once baseline outputs are ready, delete each verified focus span, sample the ablated scenarios and run the existing permutation tests with Benjamini–Hochberg correction. Step 4 is optional and can be completed later to compare observed shifts against prospective and retrospective budgets without rerunning ablation. Raw shifts, q-values and baseline noise remain available.
Changing the scenario, foci, baseline model, count or temperature requires a new prediction. Partial sampling and assessments can be retried without repeating successful calls. Export the workspace to preserve the complete run.
Continue exploring
The results report, rewrite sliders, order sensitivity, task quality evaluation, batch analysis and agent builder remain available. A separate single-output assessment is also available. The report's reported-focus series uses the retrospective mean after step 4.
Target Focus Mix
Use sliders to set one global target focus mix across all Analyse messages. Rewrite them together, then generate and reassess an output from the rewritten scenario to compare target against reported focus. FocalPrompt detects whether removing each focus shifts the model's behaviour in semantic embedding space. It does not measure correctness, quality, or safety, and it does not tell you what to delete.
A non-significant result is a failure to detect a shift at this sample size, not evidence that the text does nothing. Short structural instructions can matter greatly while barely moving embeddings.
Note: Cost and runtime increase with the number of foci and with baseline and ablated sample counts.
How this works #
Batch Analysis
7. Batch Analysis
Set a retained message's **Input name** to a column name such as `customer_message`. Each row's value replaces that message's **entire content**, not a placeholder inside it. Analyse messages and the output contract are shared unchanged across rows.
Add rows manually or upload a CSV containing `output` (the recorded response) plus one column for every named input. Multiple varying messages need distinct input names, with a non-blank value for each in every row. Leave Input name blank on retained messages that should stay fixed across rows.
Agent Builder
8. Build Optimized Agents
The Agent Builder tab allows you to build optimized agents for specific inputs. The system uses LLM assessment to select relevant foci and generate new outputs for comparison.
Best Practices
- Aim for 100% coverage of your prompt with foci whose text matches the prompt verbatim
- Mark varying input messages as Retain and give each one a unique
input_name - Use ablation analysis to test which deletions produce a detectable behavioural shift
- Treat non-significant foci as undetected, not as unused
- Test prompt adjustments with the focus sliders before finalizing
Inference scenario #
Messages are sent in this order. Analyse messages can contain foci; Retain messages remain unchanged in every experimental arm.
Output contract Optional strict JSON Schema; preserved across every arm.
1. Label the prompt foci #
Tip: A focus is a semantic/experimental unit — it may be contiguous, multi-span, nested, or overlapping.
Select text and use **Tag as Focus**, or select a focus then **Add another span** (Shift+tag also works on desktop).
Structured XML/Markdown sections can seed coarse foci without replacing finer ones.
No foci defined yet. Click "Auto-Detect Foci" or "Add Focus Manually" to get started.
2. Predict focus before generating #
The model identifies the current request from the full scenario and retained chat, then predicts a 100% budget across the labelled foci. Each focus is marked direct, background or inactive, with a justification tied to this request. Retained chat informs the prediction but does not receive a separate budget share.
Steps 2–4 use the baseline model selected below. Prediction and retrospective assessment use temperature 0.2; baseline generation uses the configurable temperature in step 3.
Label foci, then predict their contribution before sampling outputs.
3. Sample baseline outputs #
Independent completions of the unchanged scenario. Inspect baseline noise, similarities and possible response modes before ablation. Predictions are never included in generation requests.
Default: 10 outputs at temperature 0.7.
Inspect or generate a separate single output #
4. Assess retrospective focus (optional) #
This step is optional for ablation and can be completed afterwards. Each baseline output gets an independent 100% focus self-assessment at temperature 0.2, with justifications grounded in that output. Compare their average with the prospective prediction. These are behavioural self-reports, not internal attention measurements. The model selector here applies only to the optional single-output assessment.
Complete steps 2 and 3, then assess every baseline output.
5. Ablation — compare revealed sensitivity #
Requires the complete baseline outputs from step 3; retrospective assessment and baseline diagnostics are not prerequisites. Leave-one-focus-out ablation with permutation testing detects whether removing each focus shifts behaviour in embedding space — not correctness, quality, or safety, and not a delete recommendation. Complete step 4 whenever you want to add the reported vs revealed comparison to the **Results report** below.
Experiment configuration
Advanced #
Click "Run Ablation Analysis" to test whether deleting each focus shifts the model's behaviour. Results open as an insight report (Overview first).
6. Focus order sensitivity #
Measures behavioural sensitivity to **focus ordering** while keeping
semantic content constant (other messages, roles, and the output contract preserved).
Reuses Experiment B baseline samples. Distinct from LOO ablation (content removal)
and from shuffle-order robustness on each focus card (LOO + remaining reorder).
Not mechanistic attention.
Run ablation (step 5) first to reuse baseline samples.
Complete Experiment B, then run order sensitivity here.
## 7. Task quality evaluation (LLM)
Score how well each **Experiment B** output meets your task criteria
(quality / instruction-following). Evaluates baseline samples (full prompt) and
ablated samples (one focus removed per run) — separate single outputs are excluded.
Separate from **behavioral difference** (whether ablation changed outputs)
and from **reported focus** (Experiment A).
Judge 1 · Self-assessment
Uses the model recorded in the ablation run.
Judge 2 · Another model
Both judges score the same sampled outputs against the same criteria and original task context. Each judges independently; scores are shown separately. Adding a judge adds evaluation cost.
Sampling keeps at least one baseline and one sample per ablated focus. Each judge scores up to four outputs per request. Temporary connection or service failures retry automatically up to twice; saved scores are retained.
Run ablation (step 5) first. Baseline and ablated samples from that run will be scored here against your criteria.
Enter criteria above, then click Evaluate.
8. Jev dynamic focus experiment #
Separate experiment · typesafe-ai/jev Jev evaluates every labelled focus against the complete scenario and retained user input, before any output exists. It decides which instructions to include; the generation model below writes the response. Your main experiment stays available.
Jev is an evaluation model, so it appears here rather than in the chat model picker. It returns probabilities, not prose explanations or a 100% focus allocation. Gateway lists an input-token price; promotions or credits may change your charge. Current pricing.
Compare the full prompt with selected foci in source order, plus selected foci in Jev’s order when enabled. All arms use the same model, sample count and generation temperature. Retained messages, unlabelled text and the output contract are preserved. Ordering stays within each message; overlapping and multi-span foci keep their positions.
Label foci in step 1, then select and preview here. Steps 2–7 are not required.
9. Baseline + singleton focus analysis #
No foci → each focus individually → all foci → each focus removed. Separate influence, sufficiency and necessity using the same semantic comparisons as ablation.
All retained messages, the output contract and text outside labelled spans are preserved. Label foci in step 1; other steps are optional. Matching full-prompt and leave-one-out samples are reused automatically.
1. Data Input #
Configure per-row inputs
Batch Analysis uses the scenario from **Prompt Analysis**. For each message that varies by row, choose **Retain** and set its **Input name** to a CSV column name, such as `customer_message`. Each value replaces the entire message, not a placeholder within it.
Required CSV columns: output. Every row must contain a non-blank value for each column.
output is the recorded model response for each row. The prompt column is reserved for legacy files. Missing, blank, and unused columns are reported explicitly.
Add Pair Manually
Fill in the configured input fields below and the recorded model **output** for this row. The field names match your CSV columns.
Prompt (Applied to All Pairs)
Enter the prompt that was used to generate all outputs. This shared prompt does not configure per-row inputs; use Input name on retained messages in Prompt Analysis for those.
No pairs added yet. Upload a CSV file or add pairs manually.
1. Label the prompt foci #
No foci defined yet. Click "Auto-Detect from First Prompt" or "Import from Prompt Analysis" to get started.
3. Run Analysis #
Each pair is its own permutation experiment: the scenario is sampled, then each grounded Analyse focus is deleted and sampled. Retain messages are never ablated. This may take a while.
Experiment configuration
Advanced #
4. Results #
Run batch analysis to see results.
1. Enter Chat Content #
1. Label the prompt foci #
💡 Define the foci that the agent can select from, or import from Prompt Analysis tab.
No foci defined yet. Click "Auto-Detect Foci" or "Import from Prompt Analysis" to get started.
3. Assess Chat & Select Foci #
Enter chat content and define foci, then click "Assess Chat & Select Foci" to see which foci are relevant.
4. Generate Response #
After assessing chat and selecting foci, click "Generate Response" to see the constructed prompt and agent output.
5. Batch Agent Building #
💡 Import pairs (input/output) to automatically build optimized agents for each input. The system uses LLM assessment (same as single agent builder) to select relevant foci and generate new outputs for comparison.
No batch data imported yet. Click "Import from Batch Analysis" to get started.