You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
add evals -integrate with langfuse / oss eval tool? #698
The goal is to know whether changes to the agents make them better or worse before those changes enter production, then analyze production traces to understand accuracy and quality. Efficiency and other useful evaluation dimensions matter too, with accuracy and quality the priority.
Production Run scores, initially limited to facts known when a Run ends. Outcomes and repair/CI friction alone are insufficient evidence of quality; accuracy and quality need explicit evaluation.
The first replay cases and authoritative catch/grading criteria, the comparison baseline, and the pass/regression bar.
Replay isolation and reproducibility, including reconstructing historical states without their later fixes and using the existing authenticated Harnesses.
Which production quality judgments are captured automatically, which require a human, and how judgments are retained and compared.
The storage/tool choice and import format, private Security evidence handling, and whether this work should be published as a Spec with separate Tickets.
Langfuse is a candidate rather than an accepted dependency. Keep this in needs-triage: the purpose and rollout order are settled, while the implementation contract still needs shaping.
Initial eval-tool research: Langfuse, Harness exports, deployment/auth constraints, and the initial alternatives.
Wider landscape and revised decision inputs: commercial and local platforms, replay runners, native skill evals, session tools, and trace standards. Conference notes informed the questions; private recordings and Session content are excluded.
The maintainer also clarified that handrolling is acceptable. The research recommendation, rather than an accepted implementation decision, is:
Pre-change replays: maintain the existing Multi-family review: have another model family review a Run's work #429 runner with reusable case/arm/trial definitions, explicit graders, versioned JSON results and a small comparison report. promptfoo is the leading packaged alternative around the same CLI/workspace setup; native Claude plugin eval is a narrower skill-testing option.
Production Run quality: start with domain-specific outcome graders and calibrated CLI judges. Keep accuracy, incorrect findings, completion and efficiency separate. MLflow or Phoenix can provide richer score/dataset/trace workflows if needed.
Trace viewing: add Raindrop Workshop to the local shortlist. Langfuse remains credible for an integrated workflow on suitable separate infrastructure or approved hosted storage; Braintrust and LangSmith are conditional hosted alternatives.
On the format question: Session logs contain much of the evidence needed for traces, but require correlation and span mapping. OTel/OTLP, GenAI conventions, OpenInference, ATIF and code-attribution Agent Trace have different roles. Standardizing export does not guarantee identical rendering or recover missing source fields.
The earlier categorical claims that only MLflow/Phoenix have native local options and that Harbor specifically requires Docker are qualified in the companion. Local deployment shape has been checked; memory use and actual thirdshift integrations have not been benchmarked.
Still to settle: the initial cases and grading bar, reproducibility/isolation, the implementation boundary, and where non-Security data may live. The hosting question remains unanswered; Security sessions and derived eval evidence stay off third-party eval services. Coordinate a dedicated production viewer with #700/#711.
Validation: 110 landscape-source URLs checked, local links checked in both notes, private transcript exclusions checked, and whitespace validation passed. This is a documentation-only PR; #698 remains open for the implementation decision.
Each stays needs-triage until its Tickets and blocking links are published with to-tickets. #700 and #977 go first, since #978's Grader Ticket and #981's Session-trace Tickets wait on theirs.
Closing as done. The grilling (docs/grill-with-docs/done/698-evals/) produced four Specs, and all are closed: #700 Session traces, #977 Experiments, #978 Scoring run and #981 Run metrics. Anything still wanted for a Grafana-style viewer belongs in #914 or #813.
Activity
[-]add evals[/-][+]add evals -integrate with langfuse / oss eval tool?[/+]JacobStephens2 commentedon Oct 17, 2026
Triage Notes
Maintainer direction:
The goal is to know whether changes to the agents make them better or worse before those changes enter production, then analyze production traces to understand accuracy and quality. Efficiency and other useful evaluation dimensions matter too, with accuracy and quality the priority.
The maintainer agreed to this ordering:
Still to settle:
Langfuse is a candidate rather than an accepted dependency. Keep this in needs-triage: the purpose and rollout order are settled, while the implementation contract still needs shaping.
JacobStephens2 commentedon Oct 17, 2026
Research is ready for review in #741.
The maintainer also clarified that handrolling is acceptable. The research recommendation, rather than an accepted implementation decision, is:
On the format question: Session logs contain much of the evidence needed for traces, but require correlation and span mapping. OTel/OTLP, GenAI conventions, OpenInference, ATIF and code-attribution Agent Trace have different roles. Standardizing export does not guarantee identical rendering or recover missing source fields.
The earlier categorical claims that only MLflow/Phoenix have native local options and that Harbor specifically requires Docker are qualified in the companion. Local deployment shape has been checked; memory use and actual thirdshift integrations have not been benchmarked.
Still to settle: the initial cases and grading bar, reproducibility/isolation, the implementation boundary, and where non-Security data may live. The hosting question remains unanswered; Security sessions and derived eval evidence stay off third-party eval services. Coordinate a dedicated production viewer with #700/#711.
Validation: 110 landscape-source URLs checked, local links checked in both notes, private transcript exclusions checked, and whitespace validation passed. This is a documentation-only PR; #698 remains open for the implementation decision.
JacobStephens2 commentedon Oct 18, 2026
the implementation of the eval system isn't live yet.
Split the #698 eval grilling by round, and add its research notes
Merge pull request #971 from JacobStephens2/evals
Merge pull request #973 from JacobStephens2/eval-terms-round-7
JacobStephens2 commentedon Oct 21, 2026
The grilling is done. Its decision record is
.scratch/grill-with-docs/done/698-evals/, and it became four Specs:Each stays
needs-triageuntil its Tickets and blocking links are published withto-tickets. #700 and #977 go first, since #978's Grader Ticket and #981's Session-trace Tickets wait on theirs.Move the #698 eval grilling into done, with its Specs listed
Merge pull request #1042 from JacobStephens2/grilling-698-done
JacobStephens2 commentedon Oct 25, 2026
Closing as done. The grilling (
docs/grill-with-docs/done/698-evals/) produced four Specs, and all are closed: #700 Session traces, #977 Experiments, #978 Scoring run and #981 Run metrics. Anything still wanted for a Grafana-style viewer belongs in #914 or #813.