The number of cases, how they were selected and who checked the expected answers. Difficult cases must be represented.
Our answer
We agree a test set suited to the process and its risks. Your subject experts review the expected results. We record the sample size, coverage and limitations.
03
What is the error rate, with its confidence interval?
An observed error rate, sample size and uncertainty range, compared with a threshold agreed in advance.
Our answer
We define what counts as an error before testing. Where a statistical error rate is appropriate, we report it with the sample size and a 95% confidence interval. Small samples stay visibly uncertain.
Named providers, agreed locations, retention rules and the actual settings or contractual terms for model training.
Our answer
We map the data flow and agree providers, hosting locations, access and retention before project data is shared. Training use is checked against the selected service and contract.
06
What happens when the model is updated or retired?
An owner, a change process, regression checks and a usable fallback. A working demo is not an operating plan.
Our answer
The handover identifies dependencies, change owners and evaluation checks. We agree how updates are tested and when the workflow should fall back to a previous version or human handling.
07
Which decisions stay with our people, and how is that enforced?
Explicit approval points, limited permissions and controls in the workflow itself.
Our answer
We define the actions an agent may take and the decisions that require approval. Permissions, review gates and traceable actions are part of the implementation.
08
What does it cost to run at full volume, every month?
A volume-based estimate covering models, infrastructure, integrations, review and maintenance, with the uncertain assumptions exposed.
Our answer
We estimate operating cost against your expected workload. The decision includes model usage, hosting, human review, support and the effect of higher volume.
09
Who owns the prompts, code and test sets when you leave?
Written ownership and licence terms, a list of third-party dependencies and a handover the client can use.
Our answer
We agree ownership, licences, access and handover in the engagement terms. The deliverables identify the code, prompts, evaluations and operating notes included in scope.
10
What result would make you recommend that we stop?
Clear stop criteria established before delivery, including insufficient value, unacceptable errors or unmanageable operating cost.
Our answer
We agree acceptance and stop criteria at the start. The final recommendation can be scale, improve or stop. A pilot must support a decision, including a decision not to continue.