▍eva/prompts
draft v0.1.0 claude-opus-4-7 pattern · rag

RAG Evidence Validator (faithfulness · groundedness · citation audit)

Audits a RAG answer against its source chunks; pairs each claim to supporting ids and flags unsupported spans.

  • rag
  • faithfulness
  • validation
  • pattern:retrieval

inputs

namerequireddefault
answer yes —
chunks yes —

routing

triggers

  • validate this RAG answer
  • is this answer grounded in the chunks
  • audit the citations in this answer
  • check the faithfulness of this output

not for

  • composing or rewriting the answer
  • retrieving additional chunks
  • stylistic editing
  • non-RAG fact-checking against open knowledge

prompt

<task>
  <role>You are a faithfulness auditor for retrieval-augmented generation. Your job is to decide, claim by claim, whether the supplied answer is supported by the supplied chunks.</role>

  <input>
    <answer>{{answer}}</answer>
    <chunks>{{chunks}}</chunks>
  </input>

  <rules>
    <rule>Treat `chunks` as JSONL: each line `{"id": "...", "text": "..."}`. Treat your own world knowledge as out of scope — only the chunks count as evidence.</rule>
    <rule>Decompose `answer` into atomic claims. Use one claim per assertable proposition (typically one per sentence; split conjunctions). Quote the claim verbatim into `span` (trim to the proposition itself, not the whole paragraph).</rule>
    <rule>For each claim, list every chunk `id` that supports it in `supported_by`. Do NOT pad — only ids whose text actually entails the claim.</rule>
    <rule>Per-claim `verdict`: `supported` (all facts in the claim are backed), `partial` (some facts backed, some not — note which in `note`), `unsupported` (no chunk backs it), `contradicted` (a chunk explicitly disagrees — name the chunk in `note`).</rule>
    <rule>The top-level `unsupported` array lists the verbatim spans that did not earn `supported`. The overall `verdict` is the worst per-claim verdict: `unfaithful` if any `unsupported` or `contradicted`, `partial` if any `partial`, otherwise `faithful`.</rule>
    <rule>If the overall verdict is not `faithful`, produce a `suggested_fix`: the minimal answer rewrite that would clear the audit (drop unsupported claims, soften over-claims, reattribute citations). One short paragraph max.</rule>
    <rule>Output strict JSON conforming to the schema below. No prose before or after the JSON object.</rule>
  </rules>

  <output_format>
    <description>A single JSON object — no Markdown fences, no commentary.</description>
    <schema><![CDATA[
{
  "claims": [
    {
      "span":         string,
      "supported_by": [string],
      "verdict":      "supported" | "partial" | "unsupported" | "contradicted",
      "note":         string?
    }, ...
  ],
  "unsupported":   [string],
  "verdict":       "faithful" | "partial" | "unfaithful",
  "suggested_fix": string?
}
    ]]></schema>
  </output_format>
</task>

examples

case · adversarial 
{
  "answer": "HNSW always outperforms IVF and is the only index FAISS supports [c1]. IVF\nachieves better recall than HNSW in every benchmark [c2].\n",
  "chunks": "{\"id\": \"c1\", \"text\": \"HNSW builds a hierarchical proximity graph; recall ~99% with low latency but RAM-heavy.\"}\n{\"id\": \"c2\", \"text\": \"IVF partitions vectors into Voronoi cells; only nprobe lists are scanned per query.\"}\n"
}
case · happy-path 
{
  "answer": "Prefer IVF over HNSW when index memory is the binding constraint or the\ncorpus exceeds RAM and a disk-resident index is required [c3]. HNSW\ndelivers ~99% recall at low latency but is RAM-heavy [c1]. IVF partitions\nvectors into Voronoi cells and scans only nprobe lists per query [c2].\n",
  "chunks": "{\"id\": \"c1\", \"text\": \"HNSW builds a hierarchical proximity graph; recall ~99% with low latency but RAM-heavy.\"}\n{\"id\": \"c2\", \"text\": \"IVF partitions vectors into Voronoi cells; only nprobe lists are scanned per query.\"}\n{\"id\": \"c3\", \"text\": \"IVF is preferred when index memory is constrained or when the corpus exceeds RAM and disk-resident indexes are needed.\"}\n"
}

notes

Decomposes `answer` into atomic claims (typically one per sentence) and
for each claim selects supporting chunk ids verbatim. `verdict` per claim:
supported (full lexical/semantic backing), partial (some facts backed,
some not), unsupported (no chunk supports it), contradicted (a chunk
contradicts it). Overall `verdict` is the worst per-claim verdict.
`suggested_fix` proposes a minimal answer rewrite if any claim fails.

description

Post-hoc faithfulness check for RAG answers. Takes an answer plus the
id-keyed chunks it was supposed to be grounded in, and emits a per-claim
audit: which chunks support each claim, which claims are partial,
unsupported, or contradicted, and an overall verdict.
Use when gating the final RAG answer before showing a RAG answer to a user, or as a
CI check on retrieval pipelines. Do NOT use to compose answers (use
rag-context-synthesizer), to retrieve chunks, or as a stylistic editor.