Skip to content
Blueprint Catalog

    ↑↓ move · Enter open · Esc close

    11 — Technology category · TR-110.60

    AI Observability & Evaluation

    Tracing what models and agents were asked, what they did and what it cost, and scoring their output against test sets and live traffic so quality changes show up in measurements rather than in user complaints.

    • LLM Observability
    • LLM Evaluation
    • Evals
    • Agent Tracing

    In scope

    • Traces of prompts, tool calls and agent steps
    • Offline evaluation against curated test sets
    • Online scoring, user feedback and human review queues
    • Token, cost and latency tracking per model call

    Out of scope

    • General application performance monitoring (see TR-90.30)
    • Training, experiment tracking and model registry (see TR-110.20)
    • Serving and scaling the model endpoint itself (see TR-110.50)

    Example technologies · 6

    Well-known technologies in this category, named as facts. Not a shortlist or an endorsement.

    In the catalog · 4

    Realizes · 5

    Elsewhere in AI & Machine Learning