Ask the people running your AI pilot what would make you stop it. If the room goes quiet, the pilot has no success criteria, and it will end the way most pilots end: not with a decision, but with attention drifting elsewhere. The fix is not a bigger project plan. It is one page, written before the first vendor call, that states your AI pilot success criteria plainly enough that anyone in the company could read it in three minutes and say whether the pilot is on track. We call that page the pilot charter. This article covers what goes on it, why each field matters, and the mistakes that make charters useless. We have written separately about how to run a pilot without disrupting the business. This is about what to write before the running starts.
Why AI Pilot Success Criteria Have to Come First
A pilot is an experiment, and an experiment without a stated hypothesis cannot fail. It can only be reinterpreted. If you start a customer-service AI pilot without saying what a good result looks like, then whatever happens after eight weeks becomes the result. Response times dropped a little? Success. Agents disliked it? Learning experience. Nobody looked at the numbers? Inconclusive, let's extend it. Every outcome is survivable, which means no outcome produces a decision.
The charter forces the decision to be designed in advance. It also protects the people doing the work. A team running a pilot against written criteria knows what they are accountable for and, just as important, what they are not. Without it, a pilot that was supposed to test one thing quietly becomes a referendum on AI in general, and the team carries that weight.
For a leader who thinks of company resources as something held in trust, the charter is also the stewardship document. It is where you write down, before spending the money, how you will know whether the spend was worth it.
The Eight Fields on the Page
The charter is deliberately short. Eight fields, each a name, a number, or a date that you supply. If a field cannot be filled in honestly, that is information: the pilot is not ready to start.
1. The outcome
One sentence describing the business result the pilot exists to test, written in operational terms. Not "explore AI in customer service" but "reduce the time our team spends answering repeat billing questions." The outcome names the work that changes, not the technology that does it.
2. The baseline number
The current measured value of whatever the outcome is about, with the date it was measured and where the number came from. This is the field most pilots skip, and skipping it is fatal, because without a baseline the target in the next field has nothing to be compared against.
Do not guess the baseline. If you do not know how many hours your team currently spends on repeat billing questions, the honest charter leaves the line blank and the pilot's first week is spent measuring it. A blank baseline that gets filled in with real data is worth far more than a plausible-sounding estimate that nobody can defend when the pilot ends.
3. The target
The value of the baseline number that would make the pilot a clear success, and the value below which it is a clear failure. Two numbers, not one. The space between them is where judgment will be needed, and it is better to admit that now than to discover it later.
4. The timeline
A start date and an end date. Pilots that run until they succeed are not pilots. Eight to twelve weeks is common for a mid-market operational pilot, but the right length is whatever gives the baseline number enough time to move if it is going to. The end date is a commitment to decide, not a deadline for the technology to work.
5. The owner
One name. Not a committee, not a department, and not the vendor. The owner is the person who will stand in front of leadership on the end date and report the baseline, the result, and a recommendation. If no one on your team can own it, you are not ready to pilot; you are ready to hire or to wait.
6. The review dates
Two or three dates between start and end when the owner reports the current number against the baseline. Reviews are short and factual. Their purpose is to catch a pilot that is drifting or a measurement that is broken while there is still time to correct it, not to re-litigate whether the pilot should exist.
7. The exit criterion
The condition under which the pilot stops early. Usually it is a floor on the number ("if repeat-question handling time has not moved by the second review") or a risk event ("if the tool exposes customer data outside the approved system"). Writing the exit criterion is what answers the question at the top of this article. When someone asks what would make you stop, you point at this line.
8. The fallback
What happens to the work if the pilot stops or fails. In most cases the answer is "the team goes back to the current process," and writing that down is reassuring in itself. It makes clear to everyone that the pilot is reversible, which is what gives people permission to report honestly when it is not working.
Choosing AI Pilot Success Criteria That Hold Up
A few principles make the difference between criteria that produce a decision and criteria that produce a debate.
Measure the business, not the tool. Accuracy scores, adoption percentages, and usage counts describe the technology. They are useful diagnostics, but the success criterion should describe the operational outcome, because that is what you are paying for. A tool with 95% accuracy that no one trusts enough to act on has not changed the business.
Use a number you already collect, or can collect cheaply. If measuring the outcome requires building a new reporting system, the pilot has two projects in it. Pick a metric your existing systems already produce, even if it is a proxy.
Set the failure threshold as carefully as the success threshold. Leaders tend to spend their energy on the target and leave failure undefined. But the failure line is what makes the end-date decision clean. Without it, every mediocre result becomes "promising."
Write for the skeptic. Imagine the person on your leadership team who is least enthusiastic about AI reading the charter. If they would accept the criteria as fair before the pilot starts, they will accept the result after it ends. That is worth more than any amount of persuasion later. Our piece on setting measurable goals for an AI strategy covers the same discipline at the strategy level.
Common Mistakes Mid-Market Leaders Make With Pilot Charters
Letting the vendor write it. A vendor's charter will measure what the tool does well. Yours should measure what your business needs. Accept the vendor's input on what is technically feasible and nothing else.
Starting the pilot and promising to write the charter "once we see how it goes." The charter written after week two reflects what the pilot is already doing, not what it was supposed to test. Written afterward, it is documentation, not criteria.
Choosing an outcome that is really three outcomes. "Improve customer service" contains response time, resolution rate, satisfaction, and cost. Pick one for the pilot. The others can be the second pilot.
Treating the end date as negotiable. Extensions are sometimes right, but they should be a deliberate decision recorded on the charter, with a reason, not a quiet slide. Each extension without a reason makes the next one easier.
Confusing a failed pilot with a failed idea. A pilot that hits its exit criterion has done its job: it produced a decision cheaply. The fallback field exists so that stopping is normal, not shameful.
How AI with Renew Approaches This
The charter is a standard we hold ourselves to in implementation work, and we will be candid that it is a discipline, not an effortless habit. Our assessment engagements map where a company currently is; the charter belongs to the pilot that follows, and we encourage every client to write one whether or not we are the ones running the pilot. We would rather see a charter with a blank baseline and a plan to measure it than one with confident numbers nobody can source. Our interest here is simple to state: pilots with clear criteria either prove their value or end quickly, and both outcomes are better for the client than a pilot that drifts. We sell implementation work after assessments, so a decisive pilot serves us too. What we can promise is that we will say so when the criteria you have written would not survive the skeptic.
The One Next Step
Write the charter before you start. To make that faster, we have published the AI Pilot Charter as a one-page template with a filled-in fictional example and a short how-to, in our Insights library. Download it, fill in the eight fields with your own names, numbers, and dates, and leave the baseline blank if you do not yet have the number. That blank line is the first honest piece of your AI pilot success criteria, and the pilot is not ready until it is filled.