Buying AI for gaming operations: evidence, permissions and acceptance tests

Internal document assistant evaluation pack

Set acceptance thresholds from the use case and risk before testing. Keep untested capabilities and unavailable evidence explicit.

Choose one job with an observable outcome

Write a bounded task such as locating the current withdrawal procedure and drafting an internal response with the source reference. Identify the authorised document set, intended user, allowed output and actions the system must never take. Record the existing workflow and its review effort before comparing suppliers. Broad claims about transforming operations make poor acceptance criteria. A useful proposal lets a reviewer distinguish a correct answer, a justified abstention, an unsupported answer and an unauthorised action without asking whether the demo felt impressive.

Use a framework as a process aid

NIST’s generative-AI profile is a cross-sector companion to AI RMF 1.0, intended to help organisations address trustworthiness across design, development, use and evaluation. The framework is voluntary. Referencing it in a proposal does not certify a product or prove compliance with gambling, privacy or other obligations. Ask the vendor to show the concrete evaluation, incident and change-management evidence relevant to your proposed use. Keep those artifacts tied to the actual service and deployment rather than accepting a framework logo as the answer.

NIST — Generative Artificial Intelligence Profile, AI 600-1

Build a test set the supplier cannot curate away

Prepare a protected set of synthetic or appropriately authorised questions representing ordinary work, ambiguous requests, missing evidence and conflicting document versions. Agree the reference answer or acceptable abstention before running the comparison. Keep a holdout set out of configuration work and use the same task conditions across candidates. Record correctness, supported citations, material omissions, unnecessary escalation and review time separately. A single accuracy percentage hides which failures occurred. If the set is small or unrepresentative, state that limitation rather than presenting the result as general performance.

Test untrusted documents and connected permissions

OWASP describes both direct prompt injection and instructions carried through external content. It notes that retrieval augmentation does not fully remove this risk and recommends measures including least privilege, external-content separation and human approval for high-risk actions. For our internal-document example, put a harmless adversarial instruction in a synthetic source and verify the application’s actual access boundary. An answer-generation test is insufficient if a connected tool can send messages, change records or retrieve data beyond the user’s authority.

OWASP Gen AI Security Project — Prompt injection

Ask where your data and evaluation records go

Document the exact service configuration, data categories sent, retention settings, deletion process, access roles, subprocessors and contractual treatment of training use. Request evidence for those answers from the vendor’s current documentation and contract; do not infer them from another product sold by the same company. Use synthetic material until the appropriate owners authorise the real data flow. Keep evaluation logs useful but limited: record the case reference, version, decision and necessary evidence without copying sensitive customer information into broad procurement reports.

Calculate cost around an accepted outcome

Count the actual input, retrieval, output, tool and infrastructure charges in the proposed workflow, together with reviewer time and rework. Define accepted output before measuring the denominator. If ten drafts require substantial correction, their low generation price is not the cost of ten completed tasks. Compare with a measured baseline using the same work and quality threshold. Separate unavailable costs from zero, and label any planning assumptions. This guide supplies no vendor price, savings percentage or return forecast because those depend on the actual offer and workload.

Make rollout reversible and changes reviewable

Start with the bounded authority that passed the tests, with a named owner and a way to disable the integration. Save the model or service version where exposed, prompts, retrieval configuration, evaluation set version and approval record. Re-run relevant cases after material changes, including changed source documents or permissions. Ask how the supplier announces changes it controls and how you retrieve your records on exit. Escalate consequential decisions through the authorised human process; adding new actions is a new acceptance scope, not a silent extension of an old answer-quality test.