Define an acceptance contract

Write down the exact model identifier, request format, traffic pattern, and data restrictions. Separate essential features from conveniences. A large catalogue cannot compensate for a missing tool format or an unacceptable retention policy.

Before testing, define what passing means: valid required fields, correct answers on known examples, and completion within your chosen deadline. Use workload-specific targets rather than adopting another application’s latency threshold.

Identify the accountable operator

Distinguish the model developer from a reseller, gateway, or hosting operator. Confirm who bills you, handles support, and processes prompts. Familiar model names do not establish official affiliation or identical deployment behavior.

Keep an evidence record with the operator’s legal name, terms URL, date checked, support channel, and unresolved questions. Ask whether upstream routing can change and how changes affecting data handling or model versions are communicated.

Run a reproducible comparison

Use the same non-sensitive examples and settings across candidates. Include normal requests, long inputs, invalid parameters, and interrupted streams. Repeat trials at relevant times; one successful demonstration is not evidence of dependable operation.

Record one row per attempt: timestamp, endpoint, requested and reported model, settings, request identifier, result, error, first-response time, total duration, and billed usage. Mark correctness separately from speed so fast failures cannot appear successful.

Make a reversible decision

Choose only after essential checks pass and unexplained billing differences are resolved. Start with a modest balance, verify usage export and credential revocation, and keep the original test records for comparison.

Write down the rejection reasons as well as the winner. Recheck after routing, model versions, ownership, or terms change; a previously acceptable service can stop meeting your requirements without moving significantly in this index.