How to Use FocalPrompt

Getting Started

1. Tag Your Prompt into Foci

Start by tagging your prompt into points of focus or "foci". You want to aim for as close to 100% coverage as possible for a comprehensive analysis. Our model can help you detect foci in your prompt but is unlikely to take you to full coverage.

2. Define Dynamic Inputs

Dynamic parts of your prompt, where the input will vary, can be tagged as "dynamic" in section 2 "Define Foci". Dynamic inputs include:

  • Chat inputs - Variable conversation content
  • RAG - Retrieval-Augmented Generation context
  • Tools - Tool execution results

This will allow you to provide matched inputs and outputs in the batch analysis section.

3. Generate Output

You can then generate a single output for the prompt as provided. This will use the individual instance of any dynamic inputs currently provided in the prompt.

Prompt Analysis

4. Assess Focus Distribution

Section 4 will then assess the single output and your prompt foci to determine where the LLM paid most attention in producing that output.

5. Adjust Intended Focus

You can use sliders to adjust the intended focus, and our model will attempt to rewrite your prompt to reflect the intended focus. You can generate an output using the new suggested prompt to test it.

6. Ablation Analysis

Next you can run an "Ablation Analysis" where we sequentially remove individual prompt foci to see how this changes the output. We also run the prompt as it is 20 times to take a "noise" measurement. Given LLMs are non-deterministic, we select a temperature of 0.7 to assess the relative variance of output with no change in input. We then see whether removing an individual point of focus from your prompt changes the output more than just the random noise in your prompt.

💡 Tip: You may wish to reconsider prompt foci that do not have an impact above the prompt's background noise level.

⚠️ Note: The cost of the ablation analysis and the length of time it runs will increase with the number of prompt foci you have identified.

Batch Analysis

7. Batch Analysis

In the Batch Analysis tab we can assess dynamic inputs and outputs, and how the different parts of the prompt input are influencing the output.

You can manually add dynamic input, output combinations or upload in bulk in a CSV file with the appropriate format.

Agent Builder

8. Build Optimized Agents

The Agent Builder tab allows you to build optimized agents for specific inputs. The system uses LLM assessment to select relevant foci and generate new outputs for comparison.

Best Practices

  • Aim for 100% coverage of your prompt with foci
  • Tag dynamic inputs appropriately to enable batch analysis
  • Use ablation analysis to identify which foci actually matter
  • Remove or reconsider foci that don't impact output above noise level
  • Test prompt adjustments with the focus sliders before finalizing

1. Enter Your Prompt

2. Define Foci

💡 Tip: Select text in the prompt above and click "Tag as Focus" to manually add focus points.

No foci defined yet. Click "Auto-Detect Foci" or "Add Focus Manually" to get started.

3. Enter or Generate Output

4. Assess Focus Distribution

Click "Assess Focus" to see the results.

6. Ablation Analysis

Ablation analysis measures how much each focus section contributes to the output by removing one focus at a time and comparing the results to the baseline.

Click "Run Ablation Analysis" to see how each focus influences the output.

1. Data Input

CSV format: input,output (or chat_content,suggested_message). Case-insensitive column names. The prompt will be defined separately for all pairs.

Add Pair Manually

Prompt (Applied to All Pairs)

Enter the prompt that was used to generate all outputs. This prompt will be applied to all input-output pairs.

No pairs added yet. Upload a CSV file or add pairs manually.

2. Define Foci

No foci defined yet. Click "Auto-Detect from First Prompt" or "Import from Prompt Analysis" to get started.

3. Run Analysis

This will run ablation analysis on each pair. Noise is calculated once for the entire batch (20 baseline samples) and applied to all pairs, significantly reducing costs. This may take a while.

4. Results

Run batch analysis to see results.

1. Enter Chat Content

2. Define Foci

💡 Define the foci that the agent can select from, or import from Prompt Analysis tab.

No foci defined yet. Click "Auto-Detect Foci" or "Import from Prompt Analysis" to get started.

3. Assess Chat & Select Foci

Enter chat content and define foci, then click "Assess Chat & Select Foci" to see which foci are relevant.

4. Generate Response

After assessing chat and selecting foci, click "Generate Response" to see the constructed prompt and agent output.

5. Batch Agent Building

💡 Import pairs (input/output) to automatically build optimized agents for each input. The system uses LLM assessment (same as single agent builder) to select relevant foci and generate new outputs for comparison.

No batch data imported yet. Click "Import from Batch Analysis" to get started.