Change something. Re-run the same checks. Read the delta honestly.
An experiment compares one frozen prompt set — the same prompts, the same engines, the same market — before and after a change. It reports what moved, how large the sample was on each side, and what it cannot account for.
- Frozen by design
- The prompt set is locked at baseline. Editing it invalidates the comparison, and the product says so.
- Stated window
- The evaluation window is chosen up front and printed on the face of the result.
- Refusal threshold
- Below 20 valid checks per side, no verdict is declared.
If the questions change, the answer to “did it work?” is unknowable.
The single most common way an AI visibility before-and-after becomes meaningless is that the prompt set drifted between the two measurements. So AnswerRank locks it.
- The prompt set, engine list and market are captured at baseline and reused verbatim for every follow-up run.
- Adding or removing a prompt mid-experiment is blocked. If the set genuinely has to change, you start a new experiment rather than quietly rewriting the old one.
- Only valid checks enter either side, so a provider outage during the after period shrinks the sample instead of manufacturing a decline.
- Both sides report their own n. A comparison of 64 checks against 9 is not presented as though it were symmetric.
| Element | Status |
|---|---|
| Prompt set | Frozen |
| Engine list | Frozen |
| Market and language | Frozen |
| Scoring weights | Frozen |
| The answers themselves | Free to vary |
| Everything outside your site | Uncontrolled |
Six states, and the discipline they enforce.
The states exist so that the boundary between before and after is a recorded fact rather than something reconstructed from memory when someone asks whether the work paid off.
- 01
Draft
draft
You define the hypothesis, the prompt set, the engines and the market. Nothing is measured yet.
- 02
Baseline
baseline
The frozen set runs repeatedly before any change lands, building the before side with enough valid checks to be worth comparing against.
- 03
Implemented
implemented
You mark the change as shipped. The date becomes the boundary, and it is recorded on the experiment rather than inferred later.
- 04
Measuring
measuring
The identical set re-runs across the evaluation window. Partial results are visible, and labelled as partial.
- 05
Completed
completed
The window closes and the delta is reported with both sample sizes and a confidence assessment.
- 06
Abandoned
abandoned
Stopped without a verdict — the change was reverted, the prompt set shifted, or the comparison stopped being fair. Recorded, not deleted.
You choose the window before you look at the result.
Deciding when to stop measuring after you have seen the numbers is how a neutral result becomes a success story. The window is set at baseline and printed on the result, so the shape of the measurement is visible to whoever reads it.
n = 64 valid
n = 61 valid
- AI visibility
- 27% → 41%
- Recommendation rate
- 9% → 16%
- Evaluation window
- 14 days
- Prompt set
- Frozen · 32
Evidence suggests the change is associated with the movement above. Competitor activity, model updates and index refreshes during the window are uncontrolled.
How to pick a window
- Long enough for the change to be fetched and reflected — a page published yesterday may not have been read yet.
- Long enough to accumulate the valid checks each side needs, which depends on how many prompts and engines the set covers.
- Short enough that the competitive landscape has not meaningfully turned over underneath you.
- Fixed in advance, and reported alongside the delta so nobody has to ask how the number was framed.
Confidence on the result uses the same assessment as every other figure in the product: 80 valid checks, 3 engines and 3 distinct runs are required before anything is called high confidence.
Below 20 valid checks per side, there is no verdict.
This is the feature most likely to disappoint someone in a hurry, and it is the one we are least willing to remove. A confident-looking arrow drawn on six observations is worse than an honest blank.
Not enough data to declare a result (baseline has 14, follow-up has 11; 20 valid checks needed per side). The deltas below are directional only.
- Before
- 48
- After
- 52
- Baseline n
- 14 valid
- Follow-up n
- 11 valid
The numbers are still shown, because hiding them would be its own kind of dishonesty. What is withheld is the conclusion.
Why the threshold sits where it does
- AI answers are non-deterministic. Two runs of the same prompt can differ for reasons that have nothing to do with your site.
- A small set of checks can swing several points on ordinary variance alone, which is enough to invent a win that was never there.
- The threshold applies per side, so a well-sampled baseline cannot rescue a thin follow-up.
- When the sample is too small, the product says so in plain language rather than shrinking a caveat into grey six-point type.
The same guard runs everywhere a comparison appears, including inside exported reports — so a client-facing PDF cannot claim something the dashboard refused to claim.
What an experiment cannot control.
You are measuring a system you do not own, in a market that keeps moving. Naming what is uncontrolled is the difference between evidence and a claim.
Model and product updates
Providers ship new model versions, change retrieval behaviour and adjust answer formats without notice. A shift on one engine and not the others is often a clue that this happened.
Competitor activity
Rivals publish, get covered and update their own pages during your window. You are not measuring in a sealed room.
Index and crawl timing
A page you published may not have been fetched yet. Absence of movement inside a short window can mean nothing has been read, not that nothing worked.
Ordinary answer variance
The same prompt asked twice can produce different answers. This is exactly why the comparison needs a sample rather than a screenshot.
An experiment is evidence, not proof
A completed experiment supports a statement like “the change is associated with a 12-point movement across this frozen prompt set over 14 days”. It does not support “the change caused the movement”, and AnswerRank will not print that sentence for you.
Repeating an experiment, or seeing the same association hold across several engines and several runs, strengthens the case. It never converts it into a causal claim, because the mechanism remains unobservable from outside the provider.
Start with an honest baseline.
Measure where you are today, then test whether anything you change moves it.
No signup required for your first visibility check.
Every figure AnswerRank reports carries its sample size and how it was collected.
Read the methodology