How do we check that generated prose contains the right information, rather than merely sounding like a good answer?

Ask an agent for a refund status when the authoritative tool says pending. “Your refund is complete” answers the question but contradicts the tool. “We appreciate your patience” sounds polite but gives no status. “Pending” preserves the fact but might need explanation to help the customer.

We need three checks: coverage asks whether the requested information is present; grounding asks whether evidence supports the claims; communication quality asks whether the response explains them clearly and usefully. A good answer brings those properties together.

The same problem appears in lucidRESUME: “helped build” becomes “led”, a prototype becomes production, and an intended improvement becomes a measured result.

This is a practical extension of my Ten Commandments of LLM Use: keep state and actions in code, and use models to interpret language. Here we compare code, embeddings and models to see what each can check.

NLP means natural-language processing: software working with language. NLI means natural-language inference: checking the relationship between statements. LLMs are general text models. No model training is needed here.

The examples focus on model operations. Microsoft.Extensions.AI supplies .NET chat and embedding APIs; OllamaSharp connects them to local models. Configured clients and setup are in the CLI demo. The research document gives the background; NLI remains experimental in the résumé application.

Choose a check for the property you need

There is a spectrum of machinery available, but it is not a ladder on which bigger always means better. Embeddings and NLI answer different questions. A decision model can be larger than a general-purpose writer. Compare the task, output contract and observed cost together.

Use code for explicit refund state, retrieval for possible supporting passages, and a semantic check for “helped” becoming “led”. A larger model may help explain a complex failure; its factual claims still need checking.

Embedding similarity is especially easy to overinterpret. “The refund is pending” and “The refund is complete” concern the same transaction and may be close in embedding space. Related meaning does not establish the same state. Use similarity to find candidates, then check the relationship that matters.

An embedding is a list of numbers representing a text for comparison. You do not interpret each number yourself. The .NET embedding generator can produce both vectors in one call:

var pair = await embeddings.GenerateAsync([
    "Your refund is pending.",
    "Your refund is complete."
]);

var similarity = ArticleSamples.Cosine(pair[0].Vector, pair[1].Vector);

Here embeddings is an IEmbeddingGenerator<string, Embedding<float>> configured with EmbeddingGemma. Cosine is the demo's comparison helper. It measures vector closeness, not agreement about status.

This pair produced about 0.716 in the local sample, despite the different statuses. That is an illustration, not a threshold to adopt. Useful score ranges depend on the model and task. Compare vectors from the same model and keep that model identity with cached vectors.

Escalate unresolved language checks when useful. A larger judge must not waive a known amount mismatch. Cache source analysis and retain the evidence and policy behind each judgement.

Check answer presence separately

For “What is the refund status?”, the coverage obligation is to state the current status, or explicitly explain that it cannot be determined. Keep those outcomes distinct: reporting unknown is a useful response, but it has not supplied the missing status.

The second response passes coverage even when it fails grounding. The third might sound good while failing coverage. That separation makes the diagnostic useful: “No status in response” is something the writer can repair.

For a multi-part question, store individual obligations. “What did you implement, and what was the measured result?” needs a contribution and a result, or an explicit measurement limitation. A long answer about architecture should not hide the missing outcome.

NLI can inspect an output as its premise, with a coverage hypothesis such as “The response specifies the current refund status.” A decision model can instead choose answered, explicitly_unknown or missing using definitions supplied by the application. Both need evaluation: recognising what a response contains is still an inference task.

Coverage checks must describe the requested information, not presume its correct value. An answer saying “complete” still contains a status; grounding checks whether that status is supported.

Give the response something precise to say

A structured agent has more than a conversation transcript. It has tool results, retrieved passages, workflow state and a proposed response. Keep those sources distinguishable:

{
  "userQuestion": "Has my refund completed?",
  "evidence": {
    "source": "refund-tool",
    "orderId": "A17",
    "status": "pending",
    "observedAt": "2026-10-14T08:15:00Z"
  },
  "proposedClaim": "Your refund for order A17 is complete."
}

Code can reject complete against pending directly. Use that explicit state check first. A model becomes useful when the prose expresses the same error indirectly: “The money is already on its way back to your account” may imply a transition the tool never confirmed.

If a writer supplies structured claims alongside its prose, compare those directly:

var factsMatch = draft.OrderId == evidence.OrderId
    && draft.Amount == evidence.Amount
    && draft.Status == evidence.Status;

A changed status fails that check. For a simple message, a template can render the authoritative status directly. Free prose needs another check: its words might disagree with otherwise correct structured fields.

Choose the authoritative evidence for the particular claim. An old chat message saying “done” must not outweigh the latest tool status. A customer asking for a refund establishes a request, not execution. And a successful response from a tool establishes only what that response actually reports.

This extends the Reduced RAG approach: retrieve source passages and reduce them to a relevant packet before asking a model to judge or write. SeekStorm and rank fusion help find candidate passages. Their ranking cannot establish that a proposed assertion follows from those passages.

NLP can describe the account

Consider this synthetic source:

I helped the team add a C# retry queue to billing. Its effect has not yet been measured.

Useful analysis separates the contribution from its qualification:

Traditional NLP can split text into sentences and words, identify names, and analyse which words belong together. More specialised analysis tries to identify the action and its participants. That is useful for separating “I helped” from “the team delivered”, or keeping “not measured” attached to the outcome. spaCy documents these linguistic tools; PropBank is a foundation for describing actions and participants.

For the agent, the useful questions are concrete: Who performed the action? Was it attempted, completed or planned? Does “not” modify the result? Does “we” refer to the person or a team?

Store extracted fields with the original spans and confidence. An extractor might attach “not measured” incorrectly or mistake a product name for a person. Its interpretation should be reviewable alongside the text.

In lucidRESUME, accepted prose and skills belong to the complete career record. In a support agent, the equivalent record might contain a reviewed customer statement and the relevant tool observation. Analyse changed records once, then reuse them. DoomSummarizer uses the same broad idea of retaining useful analysis before synthesis.

NLI asks whether one statement follows from another

Natural-language inference takes a premise, the text we are checking against, and a hypothesis, the statement we want to test. A three-way model returns one of three labels:

• Entailment: the text supports the statement.
• Contradiction: the statement conflicts with the text.
• Neutral: the text does not tell us enough either way.

Use the accepted evidence as the premise and the proposed claim as the hypothesis. Direction matters: “I led the migration” supports participation, but “I participated in the migration” does not establish leadership.

Here are intended labels for synthetic examples, rather than promised model outputs:

Neutral is essential. Missing evidence does not establish falsehood. The refund might have completed, but this record does not tell us. An agent should check or qualify the response instead of confidently announcing either completion or failure.

NLI also cannot authenticate the premise. A false source can support a faithful rewrite. Provenance and source selection remain necessary.

A small NLI example

The CLI uses a three-way DeBERTa NLI model. It scores those labels rather than composing an answer. The .NET example exposes that operation directly:

var scores = await nli.CompareAsync(
    evidence: "The refund is pending. It is not complete.",
    claim: "The refund is complete.");

var needsReview = scores.Entailment < 0.80;

NliCli is the demo adapter, not an API built into .NET. It runs the Python model worker and returns the three scores. Its contract is two strings in, three scores out.

The input order is the important part: evidence first, claim second. The output exposes Entailment, Neutral and Contradiction. The worker preserves the full input; it refuses oversized passages rather than silently removing a qualification. This small adapter loads a worker for each call; the corpus runner batches examples into one process. Reuse a loaded model for frequent checks.

0.80 is an illustrative review threshold. The scores describe the model's distribution over its labels, not a guarantee that a response has an 80% chance of being correct. Evaluate false approvals and false alarms on your own examples before choosing a threshold.

Watch model names carefully. The similarly named zero-shot v2 model has entailment and not_entailment outputs. It cannot supply separate neutral and contradiction probabilities. Check the model configuration rather than assuming every NLI-derived classifier has three labels.

Split longer responses into claims, retaining relevant qualifications in each evidence packet. A second sentence explaining prototype status may be essential to judging the first.

SummaC explores NLI-based inconsistency detection in summarisation and shows why granularity matters. That motivates sentence or passage checks here; its summarisation results do not establish a résumé or agent acceptance threshold.

Decision models let us define the next step

Entailment is a useful relationship. A workflow needs an action too.

If the source says a refund is pending, we can revise the response to report pending. If the source has no status, we need another observation. Both situations reject “complete”, but they call for different responses.

Nimble exposes typed decisions through Ollama's System One API. Supply state, instructions and allowed choices. It returns a selection and probabilities rather than generating an explanation we must parse.

For answer presence, define the choices you actually need:

var decision = await nimble.ClassifyAsync(
    text: "Question: What is my refund status? Output: Your refund is complete.",
    instructions: "Check answer presence only, not truth or style.",
    criteria: new Dictionary<string, string>
    {
        ["answered"] = "The output states a refund status, even if false.",
        ["unknown"] = "It explicitly cannot determine the status.",
        ["missing"] = "It does neither."
    });

var statusIsPresent = decision.SelectedCandidate == "answered";

DecisionChecks is the demo's thin adapter around lucidRESUME's validated Nimble provider. The unfamiliar part is the criteria: we define the meaning of each allowed choice. The model returns a SelectedCandidate and a probability for each choice, rather than free-form advice. Ollama 0.35 or newer and an installed Nimble model are required.

Here answered is expected: the output contains a status, even if the evidence says it is wrong. It should proceed to the grounding check, not directly to “good answer”. unknown and missing describe different omissions. Choosing a label does not authorise a refund or change workflow state.

The original comparison harness also asks whether a response should be presented, revised or followed by a question. Those choices describe workflow policy. Their names and definitions come from us; the model selects among them. A typed output gives us a clean contract, not a guarantee of correct judgement.

These decisions can also guide an interview. An account that names an action but lacks personal ownership suggests “Which part did you implement?” An account that claims improvement without a measurement suggests “How did you observe the improvement?” Store the reviewed answer back in the same record. In the résumé workflow, those questions stay optional and generation can use the evidence already available.

Mix in an LLM where language needs interpretation

An LLM can propose the obligations for a complicated user question, extract candidate claims with spans, or revise a response after a missing field is identified. Keep that decomposition separate from the output being checked. Letting the writer redefine the obligations after seeing its own answer makes omissions easier to conceal.

For example, a structured writer request can carry both the available facts and the required coverage:

{
  "obligations": ["personal contribution", "observed result or limitation"],
  "facts": ["helped add a C# retry queue", "effect not measured"],
  "instruction": "Answer both obligations using these facts. Preserve ownership."
}

The writer has room to produce natural prose. Afterwards, coverage checks inspect the resulting text and support checks compare its claims with the original facts. An LLM can also review clarity or tone under a separate rubric, but that review does not replace either check.

A general LLM can perform the same classification. The unfamiliar .NET operation here is typed structured output: ask for a response matching a C# type.

var judgement = await writer.GetResponseAsync<CoverageJudgement>([
    new(ChatRole.System,
        "Check coverage only. Answered: status stated, even if false. " +
        "Unknown: explicitly unavailable. Missing: neither. Ignore truth and style."),
    new(ChatRole.User,
        "Question: What is my refund status? Output: Your refund is complete.")
]);

var missing = judgement.Result.Coverage == CoverageLabel.Missing;

writer is an IChatClient, configured here with Qwen3.8 27B on an Ollama server. CoverageJudgement is the demo record containing a CoverageLabel enum: Answered, Unknown or Missing. GetResponseAsync<T> uses that type to request structured output and deserialize the result. It saves parsing an explanation for a yes/no answer; it cannot make a wrong judgement right.

The LLM generates a typed response; Nimble returns a scored choice among our criteria. We do not invent confidence probabilities for the generative result. Both can still choose incorrectly. The Python baseline runs the earlier three-label corpus; this C# sample shows the application-specific choice.

What the CLI actually found

The support probe contains 12 synthetic examples, balanced across the three intended NLI labels. They cover refund state, assisted work, prototype deployment, metric direction, publication evidence and qualifications. We kept the same prompts, fixtures and classification settings when adding two larger general LLMs from the LAN server.

These timings describe different work. NLI ran on the local Apple M5 CPU, excluding loading. The other models ran on the same LAN server; Nimble answered two questions per request, while each LLM generated one classification. First requests took roughly 8 seconds for Nimble, 16 seconds for Qwen and 31 seconds for Gemma, including any model load. Median times mostly reflect later requests. This is a small probe, not a representative accuracy or hardware benchmark.

NLI's most instructive error was treating a planned cache improvement as achieved: its top label was entailment at about 0.64. Nimble chose neutral at about 0.53. Neither is confident support. The larger LLMs classified that case as neutral. They still confused missing evidence with contradiction in other cases.

An illustrative 0.80 probability gate and 0.20 margin would abstain on that NLI error, but accept other misclassifications. These are not calibrated thresholds. The generative LLM reports contain labels, not invented confidence scores. Nimble's separate next-response choices matched 10/12 expected actions; those actions were not tested for the other models.

The six coverage examples reveal another distinction:

Each hypothesis tests a particular piece of information, including a separate case for an explicit unknown. Nimble and both larger LLMs called the polished non-answer a contradiction rather than the fixture's neutral label. That loses an exact label match, but correctly identifies that the output does not support the requested coverage finding. NLI instead missed the explicit-unknown finding.

The larger models handle this task much better than the original 4B baseline. That small-model result should not stand in for general LLM capability. The trade-off in these runs is higher latency, not an absence of capability. All LLM corpus runs used temperature zero, thinking disabled and the same three-label schema; they do not evaluate reasoning-enabled configurations or communication quality.

The saved reports include every mismatch. New LLM reports also retain model digests, quantisation, server version and generation settings. The demo README contains the setup and rerun commands.

Apply the Ten Commandments examples

The examples in the earlier article give us concrete questions to test:

A policy answer can pass coverage and fail support. A completion message needs a send event. A price needs a code comparison. A recommendation needs a real book record. Decision models help interpret prose; the earlier boundaries still apply.

Combine checks without combining their meaning

Decision models are a convenient fit for bounded workflow choices. The larger LLMs also handled coverage well, with longer response times in these runs. This probe does not establish a universal factual judge, and it does not evaluate how well any model writes an explanation.

I would arrange the response path like this:

flowchart TD
    E[Scoped evidence and current tool state] --> W[Compose proposed response]
    W --> C[Check identities numbers and state in code]
    C --> N[Check coverage and support separately]
    N --> G[Review communication if needed]
    G --> P[Present supported useful response]
    N --> R[Revise using available evidence]
    N --> Q[Ask or obtain missing evidence]
    R --> C

Bound revision attempts. Unresolved prose can fall back to source wording or explain missing information; it should not keep rewriting until a checker agrees.

Measure omissions, unsupported claims and communication quality separately. Test changes in actor, time, units, negation and deployment status. Keep permissions and authoritative state in code: a model's label cannot authorise a refund.

The aim is to give the writer precise accounts and catch changed meaning, while leaving room for language that helps its reader. A bigger model is one option for interpretation; it is never a replacement for the fact we already know.