Define an acceptance contract
Write down the exact model identifier, request format, traffic pattern, and data restrictions. Separate essential features from conveniences. A large catalogue cannot compensate for a missing tool format or an unacceptable retention policy.
Before testing, define what passing means: valid required fields, correct answers on known examples, and completion within your chosen deadline. Use workload-specific targets rather than adopting another application’s latency threshold.
Identify the accountable operator
Distinguish the model developer from a reseller, gateway, or hosting operator. Confirm who bills you, handles support, and processes prompts. Familiar model names do not establish official affiliation or identical deployment behavior.
Keep an evidence record with the operator’s legal name, terms URL, date checked, support channel, and unresolved questions. Ask whether upstream routing can change and how changes affecting data handling or model versions are communicated.
Run a reproducible comparison
Use the same non-sensitive examples and settings across candidates. Include normal requests, long inputs, invalid parameters, and interrupted streams. Repeat trials at relevant times; one successful demonstration is not evidence of dependable operation.
Record one row per attempt: timestamp, endpoint, requested and reported model, settings, request identifier, result, error, first-response time, total duration, and billed usage. Mark correctness separately from speed so fast failures cannot appear successful.
Make a reversible decision
Choose only after essential checks pass and unexplained billing differences are resolved. Start with a modest balance, verify usage export and credential revocation, and keep the original test records for comparison.
Write down the rejection reasons as well as the winner. Recheck after routing, model versions, ownership, or terms change; a previously acceptable service can stop meeting your requirements without moving significantly in this index.