RAG Audit Trail: Know What Your AI Knew

TL;DR: Fabra tracks exactly what data your LLM used for every decision. Full lineage, context replay, and compliance-ready audit trails — not just read-only wrappers.

The Problem

Your RAG application retrieves documents, assembles context, and calls an LLM. But when something goes wrong:

• "Why did the AI say that?" — You can't reproduce the exact context that was assembled
• "What documents were used?" — Your retriever logs don't capture what actually made it into the prompt
• "Was the data fresh?" — No tracking of when features and documents were last updated
• "Can we audit this decision?" — Regulators ask, and you have no answer

Most RAG frameworks are read-only wrappers — they query external stores but don't track what was retrieved. When the auditor asks "what did your AI know when it decided?", they have no answer.

The Solution: Write Path Ownership

Fabra owns the write path. We ingest, index, track freshness, and serve — not just query. This enables:

Capability	Read-Only Frameworks	Fabra
Track retrieved documents	❌ No	✅ Yes
Track feature values used	❌ No	✅ Yes
Replay exact context	❌ No	✅ Yes
Freshness timestamps	❌ No	✅ Yes
Compliance audit export	❌ No	✅ Yes

How It Works

1. Every Context Gets a UUIDv7 ID

When you assemble context, Fabra automatically assigns a time-sortable unique identifier:

from fabra import FeatureStore, context, ContextItem

store = FeatureStore()

@context(store, name="support_context", max_tokens=4000)
async def build_support_context(user_id: str, query: str) -> list[ContextItem]:
    docs = await search_docs(query)
    tier = await store.get_feature("user_tier", user_id)

    return [
        ContextItem(content=f"User tier: {tier}", priority=0),
        ContextItem(content=str(docs), priority=1),
    ]

# Call the context
ctx = await build_support_context("user123", "how do I reset my password?")

print(ctx.id)       # 01912345-6789-7abc-def0-123456789abc (UUIDv7)
print(ctx.lineage)  # Full data provenance

2. Full Lineage Tracking

Every context assembly records:

{
    "context_id": "01912345-6789-7abc-def0-123456789abc",
    "timestamp": "2024-01-15T10:30:00Z",

    # What features were used
    "features_used": [
        {
            "feature_name": "user_tier",
            "entity_id": "user123",
            "value": "premium",
            "freshness_ms": 150,
            "source": "cache"
        }
    ],

    # What retrievers were called
    "retrievers_used": [
        {
            "retriever_name": "search_docs",
            "query": "how do I reset my password?",
            "results_count": 5,
            "latency_ms": 45
        }
    ],

    # Assembly statistics
    "items_included": 4,
    "items_dropped": 1,
    "token_usage": 1847,
    "freshness_status": "guaranteed"
}

3. Context Replay

Reproduce any historical context by ID:

# Get the exact context that was assembled
historical_ctx = await store.get_context_at("01912345-6789-7abc-def0-123456789abc")

print(historical_ctx.content)   # Exact prompt content
print(historical_ctx.lineage)   # Full data provenance

4. CLI Tools

# Show a context by ID
fabra context show 01912345-6789-7abc-def0-123456789abc

# List recent contexts
fabra context list --limit 10

# Export for audit
fabra context export 01912345-6789-7abc-def0-123456789abc --format json

Use Cases

Incident Investigation

When a user reports a bad AI response:

# Find all contexts for this user in the incident window
contexts = await store.list_contexts(
    start=incident_time - timedelta(hours=1),
    end=incident_time + timedelta(hours=1),
)

# Examine what data was used
for ctx in contexts:
    full_ctx = await store.get_context_at(ctx["context_id"])
    print(f"Features: {[f['feature_name'] for f in full_ctx.lineage.features_used]}")
    print(f"Freshness: {full_ctx.lineage.stalest_feature_ms}ms")

Regulatory Compliance

Export context lineage for auditors:

# Export all contexts for a time period
for ctx_id in $(curl -fsS "http://127.0.0.1:8000/v1/contexts?start=2024-01-01T00:00:00Z&end=2024-01-31T23:59:59Z&limit=10000" | jq -r '.[].context_id'); do
    fabra context export $ctx_id --format json -o "audit/$ctx_id.json"
done

Debugging Stale Data

Identify when stale data affected decisions:

ctx = await build_context(user_id, query)

# Find the slowest/stalest feature
slowest = max(ctx.lineage.features_used, key=lambda f: f.freshness_ms)
print(f"Stalest feature: {slowest.feature_name} ({slowest.freshness_ms}ms)")

Quick Start

pip install "fabra-ai[ui]"

from fabra import FeatureStore, context, ContextItem
from fabra.retrieval import retriever

store = FeatureStore()

@retriever(index="docs", top_k=5)
async def search_docs(query: str):
    pass  # Auto-wired to pgvector

@context(store, max_tokens=4000)
async def chat_context(user_id: str, query: str) -> list[ContextItem]:
    docs = await search_docs(query)
    tier = await store.get_feature("user_tier", user_id)
    return [
        ContextItem(content=f"User tier: {tier}", priority=0),
        ContextItem(content=str(docs), priority=1),
    ]

# Every call is tracked
ctx = await chat_context("user123", "help me")
print(f"Context ID: {ctx.id}")
print(f"Lineage: {ctx.lineage}")

FAQ

Q: What is a RAG audit trail? A: A complete record of what data (documents, features, retrieved results) was used to assemble context for each LLM call. Fabra tracks this automatically with full lineage.

Q: How do I know what my AI knew when it made a decision? A: Use store.get_context_at(context_id) to replay the exact context that was assembled. Every context gets a UUIDv7 ID for retrieval.

Q: Can I use this for GDPR/SOC2 compliance? A: Yes. Export context lineage with fabra context export. Every AI decision traces back through the data that informed it.

Q: How is this different from LangChain? A: LangChain is a framework that queries external stores. Fabra is infrastructure that owns the write path — we track what was retrieved, not just how to retrieve it.

Q: Does lineage tracking add latency? A: Minimal. Lineage is recorded asynchronously after context assembly completes. Context is still returned immediately.

Next Steps

• Context Accountability — Deep dive into lineage tracking
• Compliance Guide — GDPR, SOC2, and regulated industries
• Context Store — Full RAG infrastructure docs
• Quickstart — Get started in 30 seconds

---

Also Need ML Features?

Fabra includes a Feature Store for serving ML features — user personalization, risk scores, recommendations. Same infrastructure, same deployment.

Feature Store → | Feast vs Fabra →

---

You Might Also Search For

• "rag audit trail"
• "what did ai know compliance"
• "llm decision audit"
• "context replay llm"
• "ai explainability debugging"