# LLM Evals For The Entire Team. Not Just Engineers. Postman for AI evaluation. Connect via API, simulate conversations, and test entire AI workflows — not just prompts. No CSVs. No waiting on engineering. Evals ran to date[-21,916,950,212+] ## HOW IT WORKS ### Experiment without the engineering bottleneck. 1. 01 ### Connect any AI app in minutes. Point to any endpoint like Postman. Send requests, tweak prompts, see results live. 2. 02 ### Define your golden dataset and metrics. Upload test cases, pick the metrics that matter, set thresholds for "good". 3. 03 ### Run experiments on your whole app, not just a prompt. Run your dataset against your live app — not a playground. Catch multi-turn failures. 4. 04 ### Ship confidently, without the bottleneck. Wire evals into CI. Required checks block regressions before they merge. ### Connect Any Endpoint Point at any AI app like Postman. No SDK, no code changes. ```http POST https://api.your-app.com/v1/chat ``` **Params Headers Body Auth JSON** ```json { "model": "gpt-4o", "messages": [ { "role": "user", "content": "How do I dispute a charge?" } ] } ``` **Response 200 OK 412 ms 1.2 KB** ```json { "id": "chatcmpl-9f2a…", "output": "To dispute a charge, open…", "latency_ms": 412 } ``` ### Golden Dataset & Metrics Upload test cases, pick the metrics that matter, and set thresholds. | Input | Expected Output | Tag | |------------------------------------|------------------------------------------------|---------| | How do I cancel my subscription? | Go to Settings → Billing and click Cancel… | billing | | Why was my card declined? | A decline usually happens for one of three… | payments | | Can I get a refund for last month? | Refunds are processed within 5–7 business… | refunds | | What's covered under the basic plan? | The basic plan includes core access to… | plans | **Metrics 4 / 50+** - Answer Relevancy ≥ 0.85 - Faithfulness ≥ 0.80 - Hallucination ≤ 0.10 - Task Completion ≥ 0.75 - Tone ≥ 0.70 ### Side-by-Side Experiments Change a model, prompt, or pipeline step. See what improved and what regressed. | Metric | Baseline | Experiment | Δ | |----------------------|----------|------------|--------| | Answer Relevancy | 0.82 | 0.91 | +0.09 | | Faithfulness | 0.74 | 0.88 | +0.14 | | Hallucination | 0.18 | 0.06 | -0.12 | | Task Completion | 0.69 | 0.84 | +0.15 | | Tone | 0.81 | 0.79 | -0.02 | ### Ship Only When Evals Pass Required eval checks on every PR. Quality regressions block the merge. ### PLATFORM ## Testing you'll actually want to run. **Simulated user** I need to return a jacket **AI agent** I can help with that. Do you have the order #? **Simulated user** Order #8847-AX. **AI agent** Found it. Navy jacket from May 5. **Simulated user** Yes, that's the one. **AI agent** Return initiated. Label sent to your inbox. ### SIMULATION RESULTS - **100% 96% 99% 91% 94% 86% pass rate** - Passed **2,064** - Failed **336** - Avg turns **5.2** - P50 latency **1.2s** - Coherence **0.91** - Hallucinations **4.2%** ### Multi-turn conversation testing Simulate full conversations end-to-end and catch failures that only surface across multiple exchanges. **VARIANT DIFF** - **prompt v1.2→v1.3 temp 0.7→0.4** | Metric | v1.2 BASELINE | v1.3 VARIANT | WINNER | |---------------------|----------------|---------------|--------| | Faithfulness | 0.78 | 0.91 | +0.13 | | Relevance | 0.84 | 0.88 | +0.04 | | Tone match | 0.72 | 0.69 | -0.03 | | Coherence | 0.81 | 0.93 | +0.12 | | AVG | **0.79** | **0.85** | +0.07 | ### Side-by-side experiments Change any variable — model, prompt, system logic — and compare results across every metric and pipeline step. ### METRIC ALIGNMENT - **n=1,240 3 evaluators aligned** - **CONFUSION MATRIX vs. human label** **HUMAN PASS** - **842 true pos** - **56 false pos** - **EVAL FAIL** - **38 false neg** - **304 true neg** - **PRECISION 0.94** - **RECALL 0.96** - **F1 0.95** - **κ 0.84** ### OVERALL **92%** ### Alignment metrics with humans Compare metric scores against human annotations to surface false positives and negatives. - evals.yml on: push, pull_request PASS **94%** **TRIGGER** **COMMIT** **DURATION** **EVAL SCORE** - prompt edit main#a3c91e1 m 04s → **0.91** +0.03 ## Automated evals on every change Think GitHub actions for evals. Product managers and domain experts can tweak prompts, and evaluations will run automatically. ### MCP-native workflow Evaluate, iterate, and ship without leaving your favorite IDE. **METRICS** ## Metrics your org can rally behind. Powered by DeepEval. - 50+ research-backed eval metrics used by teams at OpenAI, Google, and Microsoft — from hallucination and faithfulness to tone, safety, and task completion. | Metric | Score | |-------------------------|--------| | Safety | 0.97 | | JSON Correctness | 0.95 | | Faithfulness | 0.93 | | Summarization | 0.91 | | Answer Relevancy | 0.88 | | Role Adherence | 0.86 | | RAGAS | 0.83 | | G-Eval | 0.80 | | Contextual Recall | 0.78 | | Coherence | 0.74 | | Task Completion | 0.71 | | Knowledge Retention | 0.67 | Warning | Tone Consistency | 0.63 | Warning | Latency | 0.59 | Warning | Verbosity | 0.56 | Warning | Groundedness | 0.53 | Warning | Coverage | 0.50 | Warning | Tool Correctness | 0.42 | Failed | Hallucination | 0.31 | Failed | Bias | 0.19 | Failed | Toxicity | 0.08 | Failed ## INTEGRATIONS ### Works with your stack. All of it. ## More on LLM evaluation - [LLM evaluation metrics guide](/content/blog/llm-evaluation-metrics-everything-you-need-for-llm-evaluation/index.html) - [Why LLM-as-a-judge works](/content/blog/why-llm-as-a-judge-is-the-best-llm-evaluation-method/index.html) - [What makes a good eval](/content/knowledge-base/playbook/what-makes-a-good-eval/index.html) - [Compare LLM evaluation tools](/content/knowledge-base/compare/best-llm-evaluation-tools/index.html) - [How Humach shipped voice AI 200% faster](/content/case-study/humach/index.html)