This simulated case follows an AI ad-optimisation pilot that looked positive early and negative later. Intake can show that the team cannot explain the reversal and that success criteria were never agreed. That does not prove the model failed. Stage 2 would inspect the test design, control group, configuration changes, market movement and guardrail metrics. The directional decision is FIX: pause, establish experiment governance and run one controlled re-test with pre-registered thresholds and a stop rule. If a clean re-run cannot be secured on acceptable terms, the default becomes STOP as “unproven”, not “failed”.
What does this case show? Without a protected control, pre-agreed thresholds and a single owner, neither positive nor negative pilot results support a defensible AI decision.
A mid-sized free-to-play mobile game studio earns most of its revenue from advertising. Ad frequency is a sensitive lever. More ads can lift short-term revenue but may damage retention, ratings and lifetime value. A vendor offers machine learning to set individualised ad-frequency caps.
A paid pilot began with an apparent revenue uplift and ended below baseline. Should the company continue, restructure or terminate it?
The model changes the ad-configuration layer, but performance signals originate upstream in ad demand and infrastructure, and downstream in player behaviour. The full chain must be instrumented to interpret the result.
The following items may be reported during the initial intake or interviews. They are not treated as measured performance values.
Initial intake, interviews and limited document signals may make the following hypothesis a priority. It is not presented as a confirmed root cause until it is tested against operational evidence.
The working hypothesis is that the organisation cannot yet run a trustworthy experiment on this revenue lever. The visible symptom is a pilot that appears to have lost money. The deeper issue may be a contaminated test, no pre-committed success criteria and fragmented ownership. On that evidence, the result is not proof that the AI failed. It is proof that the organisation cannot yet tell whether it worked.
The following are value hypotheses, not achieved or validated outcomes.
Customer value: Players experience ad frequency directly. A governed system should protect retention, stability and perceived fairness while testing revenue value.
Internal value: A clean decision avoids scaling an unproven claim and avoids killing a potentially useful lever. It also strengthens vendor accountability and future experiment quality.
Scale from the early uplift or stop from the final decline.
The AI role, human decision boundary, data readiness and measurement logic are designed together. No scale decision is made without pilot evidence.
This table is not an investment decision or a definitive ranking. Priority and evidence readiness are reassessed during Stage 2.
| Opportunity hypothesis | AI role | Value connection | Evidence readiness | Directional priority |
|---|---|---|---|---|
| Experiment-governance standard | Explain + Govern | Make future AI results decidable | Leadership ownership and process change required | Highest; prerequisite |
| Causal pilot monitoring and change log | Detect + Explain | Separate model effect from other changes | Joined data and frozen control required | High; re-test foundation |
| Segment-level ad-cap optimisation | Predict + Recommend | Revenue with player-experience guardrails | Clean experiment and raw-data access required | Conditional; re-test only |
| Portfolio-wide automatic rollout | Execute | Scale revenue optimisation | Current evidence is contaminated | Not recommended |
The design below is not a solution commitment. It is a pilot hypothesis to be validated.
Controlled AI Ad-Frequency Re-Test
The following are readiness hypotheses that require document and process review.
This plan is adapted to data access and client scope. Stage 1 alone does not include implementation or outcome validation.
| Period | Purpose | Main actions |
|---|---|---|
| Days 1–30 | Governance and baseline | Name one experiment owner, write the holdout and change-freeze standard, define success and stop thresholds, and complete infrastructure hygiene checks. |
| Days 31–60 | Controlled re-test | Negotiate the re-run, freeze unrelated changes, run treatment versus a locked holdout in the same markets and window. |
| Days 61–90 | Decision gate | Evaluate only against the pre-registered thresholds and produce a one-page Scale or Stop memo for management. |
The treatment meets the pre-registered revenue threshold and no player-experience or stability guardrail is breached.
The direction is positive, but uncertainty, sample size or implementation quality prevents a scale decision.
The threshold is missed in a clean test, or the organisation cannot secure an interpretable re-test on acceptable terms.
Transferable lesson: Before asking whether an AI initiative works, ask whether the organisation would be able to tell.
Assess the available evidence, business value and implementation conditions together.