Test Chinese AI models on company documents by giving each candidate the same questions, source files and reference answers. Score whether the answer is correct, whether its cited passage supports it, and how much human checking it requires. Include Chinese and English documents, superseded instructions and questions the files cannot answer.
A journalist once asked me who was winning the AI race, the US or China. In my March LinkedIn post, I questioned what winning meant. For a business choosing a document assistant, that question becomes very specific: can a colleague find the right instruction and use it with confidence? A national ranking gives the buyer little help with that decision.
My name is Thomas Derksen, founder of Asiabits in Shanghai. The comparison I would want to see starts with a familiar working document. Put a maintenance manual or supplier specification in front of the system, then ask a question whose answer an experienced employee can verify. Keep that employee involved when choosing the questions and judging the responses.

What should a document test actually measure?
A document test should measure whether an employee can reach a correct, supported answer with an acceptable amount of checking. Define the useful result before selecting a model. A purchasing team might need the delivery condition from an approved specification. A service team might need the applicable maintenance procedure for a particular machine revision.
Write the expected answer beside each question and identify the source passage that establishes it. Include any condition that changes the answer, such as the equipment version or the date an instruction became effective. This gives the reviewer a stable reference. Otherwise, an answer that sounds plausible can receive a good score simply because nobody has checked the underlying file.
Keep separate scores for correctness and support. A correct answer with an unrelated citation still creates work for the employee. A faithful quotation can also be wrong for the task when it comes from a superseded document. Record these as different errors so the technical team knows which part of the application needs attention.
Measure the complete working session as well. Include the time spent opening the source, checking a table and correcting a response. Ask the same employee to complete some questions using the existing search process. That creates a practical comparison with today's work. Choose acceptance requirements according to the consequence of a wrong answer and the review the task already requires.
How do search and answer generation differ?
Search selects the passages an assistant can use; answer generation turns those passages into a response. In a retrieval-based application, the team needs to inspect both. A weak search result can leave the answering model without the evidence it needs, even when that model writes clear English or Chinese.
Alibaba Cloud's Qwen team provides a useful example of the search layer. Its Qwen3 Embedding repository documents embedding and reranking models for text retrieval, including multilingual uses. Embeddings represent text numerically so related passages can be found. Reranking then helps order candidate passages against the question. These components give developers ways to assemble document search.
During a pilot, ask the supplier to show the retrieved passages before the generated answer. If the relevant paragraph is missing, investigate extraction, document splitting and search. If the right paragraph is present but the response changes its meaning, investigate the answering stage. This simple separation makes a supplier conversation much more productive than repeatedly changing the wording of the question.
For a small document set, compare retrieval with giving the model the relevant text directly. Keep the question and reference answer unchanged. The comparison can reveal whether the retrieval stage is introducing a problem. Our guide to running Chinese models on company servers covers the separate infrastructure decision. Use the deployment you are actually considering when measuring response time.
How should Chinese and English files be compared?
Chinese and English files should be tested with equivalent business questions and the terminology employees actually use. Include questions in one language whose answer appears in the other. A bilingual assistant needs to preserve part numbers, units and conditions while explaining the relevant passage clearly to its user.
Build a short terminology list with the people who read the documents. A component can have a formal Chinese name, an English catalogue term and an internal abbreviation. Include these variations in separate questions. Record whether the system finds the same correct passage. This tests the vocabulary gap that a polished demonstration with carefully chosen wording can avoid.
Tables deserve their own cases. Ask for a value that depends on both a row and a column, then inspect the extracted text. Check scanned files separately from digitally created PDFs. If the file-processing stage loses a table heading, the answering model may receive an incomplete instruction. Keep the original page visible to the reviewer so that extraction errors can be identified directly.
Our own research illustrates why exact names matter. In our database of 7,000+ robotics companies, the current research collection contains 7,320 company records worldwide. A question about one manufacturer must resolve to the right record before any product comparison is useful. Apply that discipline to your documents: retain original product identifiers and document titles alongside translated explanations. The Asiabits robotics directory supports that initial company research.
Which difficult cases belong in the pilot?
A useful pilot includes conflicting versions, missing answers and questions that combine conditions from several passages. These cases show how the application behaves when a quick keyword match is insufficient. Start with the exceptions employees already encounter, then make the expected handling explicit in the reference answers.
Place an old instruction and its replacement in the test collection. Give both clear dates and status labels. Ask a question affected by the change, and check which document the assistant uses. Repeat the test after adding a revised file. The team should be able to see how the application updates its searchable collection and removes obsolete material from the answer path.
Add a question whose answer is absent from the supplied documents. The acceptable response should identify the missing information and give the employee a useful next step, such as consulting the responsible engineering team. Judge that response against the task. An invented setting for a machine is a materially different error from failing to find a routine administrative detail.
Test access restrictions through the application itself. Use separate test accounts with different document permissions and verify what each can retrieve. A model's ability to explain a file does not establish who should be able to read it. Ask the supplier to demonstrate how a removed permission affects search results, source previews and stored conversation history.
Keep some questions out of the supplier's development set. Once the application has been adjusted, use these unseen questions to check whether the improvement carries across to other documents. Preserve the original results. That gives the buyer a clear comparison after changes to the model, document extraction or search settings.
For example, create a fictional maintenance file with two equipment revisions and a table of inspection intervals. Ask which interval applies to the later revision, then ask the same question using the Chinese equipment name. Add an older file with a different interval and repeat. Because the example is invented for testing, the team can change its facts and verify whether the answer changes with them. Use this controlled exercise alongside representative company files.
Give reviewers a consistent way to record partial success. An answer might identify the right procedure but omit the condition that makes it applicable. Save that omission explicitly rather than reducing the assessment to a single satisfaction score. It will tell the supplier what needs to improve and let the buyer decide whether the remaining checking work is acceptable.
What should you ask a Chinese AI supplier to deliver?
Ask the supplier for a reproducible test record showing the exact application, model release, document collection and results. A colleague should be able to rerun a question and inspect the supporting passage. Agree who owns the document updates and who responds when employees report an incorrect answer.
Request a record for each tested question: the question itself, the retrieved passages, the final answer, the reviewer's assessment and the elapsed time. Include the source document version. This makes it possible to compare a model change with a search change and to identify whether an improvement reduced the employee's checking work.
Before extending the pilot, choose a bounded operating task and assign responsibility for reviewing its output. An internal document assistant can begin by helping a trained colleague find evidence. If later versions will update records or send messages, those actions need their own tests and permissions. Our guide to evaluating Chinese AI agents examines that wider workflow.
Asiabits prepares technology discussions and visits around the buyer's practical questions. Through our company visits in China, teams can discuss the product with the people building it and compare its proposed role in their business. Bring a representative document, a clear reference answer and the operating question you need to settle. Those inputs give a demonstration a useful purpose.
Frequently asked questions
How should a company compare Qwen and DeepSeek on documents?
Give the candidate applications identical documents, questions and reference answers. Compare correctness, supporting passages and employee review time, using the exact deployment each supplier proposes.
What is retrieval in a document assistant?
Retrieval selects passages from a document collection for the answering model to use. Inspect those passages separately from the generated response to distinguish search failures from answer errors.
How do you test a bilingual Chinese and English AI assistant?
Include cross-language questions and the component names employees actually use. Check that translated answers preserve identifiers, units and the conditions attached to each instruction.
What should happen when the documents contain no answer?
The assistant should identify the missing information and direct the employee to the appropriate source or responsible person. Include such questions in the test set and check that the system avoids inventing an answer.
Get These Insights Every Monday
Join 18,000+ professionals reading Asiabits. Free, every Monday, straight from Shanghai.
Subscribe Free →
