How to evaluate enterprise AI search using your own knowledge

A polished demonstration answers the vendor's question

Your evaluation needs to answer yours. Can the service find the information your people need, respect the permissions attached to it and show enough source evidence for someone to judge the answer?

That cannot be established with a generic collection of perfect documents. Use an approved sample that resembles the information environment the service is expected to support.

Build a small but awkward knowledge set

Include current and superseded documents, duplicate versions, specialist language, short and long files, email attachments, restricted material and at least one question the sources cannot answer. Record the owner, date, version and expected access for each item.

If production information cannot be used safely, create representative material that preserves the structure, terminology and permission differences without exposing sensitive content.

Write the questions before running the test

Ask intended users for questions that reflect real work. Include exact lookups, conceptual questions, questions that require more than one source and questions where current information must take precedence.

For each question, record what a good result should contain, which source or sources should support it, who is allowed to see them and what an acceptable failure looks like. This prevents a confident-sounding answer from being mistaken for a correct one.

Test retrieval before judging the AI response

First inspect what the search service retrieved. If the correct document never reached the model, changing the prompt will not solve the real problem. Check whether the service found the right material, ranked it sensibly and excluded irrelevant or forbidden sources.

Then assess the answer: does it stay within the retrieved evidence, identify uncertainty, show the source and avoid filling a gap with plausible invention?

Permissions and change are part of search quality

Run the same question as users with different access. Check the original source, the indexed copy, retrieval and the final answer. A reassuring interface is not proof that the permission path works.

Replace or withdraw a document during the evaluation. Measure how long the old content remains available and what happens to previous answers or cached material.

Test failure on purpose

• Ask a question with no supporting source
• Use conflicting current and superseded material
• Remove access after content has been indexed
• Make a source unavailable during retrieval
• Try exact identifiers and unusual terminology
• Check unsupported or poorly extracted file types

A mandatory permission or evidence failure should not be hidden inside one attractive weighted score.

Example: record the result, not just a score

This illustrative test uses fictional documents. It is not a customer result or a completed test of Pūnaha.

Question: Who must approve a purchase?

Test documents: Current Purchasing Policy v2 requires manager approval; superseded v1 does not.

Expected result: For an account allowed to access v2, identify manager approval and cite v2.

Failure conditions: Use v1, invent an approver or omit the supporting source.

Observed result: Not run.

Repeat with an account that cannot access v2. The system should not disclose its restricted contents; an “insufficient accessible evidence” response may be appropriate.

Record the actual answer, source version, test-account role and date before deciding whether the test passes.

Run a bounded evaluation

Agree the users, content, questions, measures, time period and decision in advance. The decision may be to proceed to a pilot, change the requirements, retain an existing service or stop.

Pūnaha supports configurable knowledge preparation and retrieval, including semantic chunks, document structure and tables of contents, parent–child passages, wiki-style knowledge articles, generated questions, summaries, entities and relationships, and structured tables. Selected methods prepare searchable representations of your information, with source references and customer-controlled deployment options. Hot Desk does not present public Pūnaha customer references or universal performance claims. Read more about Pūnaha enterprise search or discuss an independent AI evaluation.

A note about this article

This is general information from Hot Desk Consultancy Services Limited. It does not assess a particular product, organisation or information environment and is not security, privacy, legal or procurement advice.