Skip to content

Research eval tooling and replay options - #741

Merged
JacobStephens2 merged 1 commit into
mainfrom
evals
Oct 17, 2026
Merged

JacobStephens2 merged 1 commit into
mainfrom
evals

Conversation

@JacobStephens2

Copy link
Copy Markdown
Owner

Summary

Compare eval tooling for #698 across pre-change replays, production Run scores, and trace viewing, including a hand-rolled option.

docs/research/
├── eval-tools.md            # Langfuse, Harness exports, and initial alternatives
└── eval-tools-landscape.md  # Wider search, revised shortlist, and decision inputs

The companion recommends extending the existing replay runner, with promptfoo as the leading packaged alternative. It adds local viewer options and keeps hosted choices conditional on the unresolved data-hosting decision. References #698; this PR records research rather than selecting or implementing a dependency.

Evidence

  • Before: add evals -integrate with langfuse / oss eval tool? #698 identified Langfuse as a candidate without a settled tool or hosting choice.
    After: Primary-source comparisons cover subscription auth, licences, deployment, Harness coverage, grading, and trace formats, with live checks explicitly identified.
  • Validation: checked all 110 external links in the landscape note, local file links in both notes, private transcript exclusions, and git diff --cached --check.

Merge Danger

Door: two-way

Blast Radius: Documentation

Recommendations and pricing are dated; runtime integrations and host resource use remain untested.

@JacobStephens2
JacobStephens2 merged commit c949973 into main Oct 17, 2026
5 of 9 checks passed
@JacobStephens2
JacobStephens2 deleted the evals branch October 17, 2026 02:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant