GEDD turns domain-expert judgment into one failure codebook, then derives your evaluation, your training reward and your response gate from it.
-
Updated
Nov 20, 2026 - Python
GEDD turns domain-expert judgment into one failure codebook, then derives your evaluation, your training reward and your response gate from it.
Evaluate and trace tool-calling LLM agents, with one guarantee: a tool failure can never look like an empty result.
Evaluation framework for testing LLM outputs locally. Define prompt templates and custom scorers, run evals against OpenAI or Anthropic models, store results in SQLite, and browse via Rich CLI dashboard or FastAPI UI. Lightweight, self-contained, extensible Python library.
CLI tool for evaluating and comparing LLM prompts. Define test cases in YAML, run against OpenAI and Anthropic models, score outputs, and generate HTML reports. A lightweight, local-first alternative to cloud eval platforms.
Observability layer for multi-step AI pipelines — traces execution, auto-diagnoses root causes, and builds a growing eval dataset from human feedback
CLI tool for defining and running LLM evaluation suites against any OpenAI-compatible endpoint. YAML test cases, multi-metric scoring (exact match, regex, LLM judge), and rich terminal reports. Built for fast feedback loops in CI/CD.
Self-hosted evaluation harness for AI agents using rubric-based LLM-as-judge scoring plus deterministic checks. Web UI for test case management, rubric definition, trace inspection, and side-by-side agent comparison.
Evaluate RAG pipelines end-to-end. Auto-generates Q&A benchmarks from your documents, runs them through your retrieval system, and scores answers on faithfulness, relevance, and context recall using LLM-as-judge. Local vector store, pluggable LLM backend.
Open-source evaluation framework for LLM agents. Run head-to-head A/B tests, score with LLM-as-judge rubrics, and visualize results in a Streamlit dashboard. Model-agnostic, self-hosted, zero external infrastructure.
Python tool for extracting validated, schema-defined JSON from documents (PDF, HTML, Markdown, plaintext) using LLMs, with source grounding and a built-in eval harness. Works with Anthropic Claude or OpenAI.
Self-hosted evaluation framework for tool-calling LLM agents. Define tasks in YAML with expected tool calls and scoring rubrics, run against Claude or GPT-4, get scored reports via CLI and FastAPI dashboard. Lightweight, local-first, no lock-in.
Lightweight CLI for versioning prompts and running eval suites. Score outputs with deterministic matching or LLM-as-judge, compare prompt versions with rich terminal diffs. No infra, git-friendly, local-first.
Self-hosted evaluation framework for MCP servers. Define YAML test suites, run agents against MCP tools, score with LLM-as-judge rubrics, and monitor results in a FastAPI dashboard with tool-call traces and regression detection.
Lightweight Python evaluation framework for LLM prompts. Define test cases in YAML, run against OpenAI and Anthropic models, score outputs with LLM-as-judge, view results in a web dashboard. Built for regression testing, prompt comparison, and quality assurance.
Open-source evaluation framework for AI agents. Define test suites with rubrics, run your agent, get LLM-as-judge scores against criteria, inspect full execution traces, and diff runs to catch behavioral regressions.
Define YAML rubrics, run agents through test scenarios, get LLM-judged per-criterion scores with full trajectory traces, and analyze results in an interactive web dashboard.
A FastAPI WebSocket service and CI-friendly CLI for running LangSmith evaluations on demand — bundles LLM-as-judge (Claude) and heuristic evaluators, syncs datasets idempotently, and gates deploys via threshold checks.
Self-hostable LLM evaluation framework for measuring model performance across configurable skills. Run YAML-defined benchmarks against OpenAI/Anthropic models, score with LLM-as-judge, compare results in a CLI and web dashboard.
Web studio for evaluating LLM agents with rubric-based scoring, LLM-as-judge validation, and side-by-side run diffs. Trace agent execution, compare performance, and iterate with full visibility into tool calls and LLM interactions.
LLM eval framework built from scratch in Python — accuracy, consistency, code-based, and model-based grading for a Claude-powered classifier.
To associate your repository with the eval-framework topic, visit your repo's landing page and select "manage topics."