Repository navigation
Conversation
hugegraph-llm-eval scores retrieval, answers and property graphs without the application: domain contracts, provider-neutral judge ports, answer/retrieval/ graph evaluators, and an in-process engine with ordered results, bounded concurrency, retries, deadline, cancellation, partial results and per-call cost accounting. Results are structured records that callers can persist or feed back into a repair loop. Supports Python 3.12-3.14 and the dependency ranges introduced by apache#389. - base install depends on Pydantic only; graph, rouge and openai are extras with lazy vendor imports - one run-scoped limiter caps provider calls at max_samples x max_concurrency; nested evaluator batches share it, and cancellation and the deadline are checked before every judge call - examples/all_metrics.py runs all sixteen metrics through both entry points with a deterministic stand-in judge, and against a real model when configured - input builders and an OpenAI-compatible judge factory keep callers away from the artifact contract - CI builds the wheel and sdist outside the workspace, tests the installed distribution instead of the source tree, and runs the examples
…racts Keep semantic pair alternatives until global assignment, validate schema endpoints and null keys, and correct graph-faithfulness examples. Share the judge limiter, retain billed retry usage, stop unpriceable budgeted batches, support ID-only ranking, and preserve atomic claim evidence. Add regressions for the independently reproduced review findings and clarify metric, budget, installation and custom-judge boundaries.
Let evaluation run on a Claude or ChatGPT/Codex subscription instead of an API key. - ClaudeCodeJudge runs one tool-free headless `claude -p` per call and authenticates as the local CLI does; scrub_provider_env and setting_sources force the logged-in subscription. - CodexAppServerJudge shares one `codex app-server` process and runs each call as an ephemeral, read-only thread with shell, apps, plugins and user MCP servers disabled. Output schemas are normalised to OpenAI strict mode. - Both report tokens but no cost, record the CLI version in the judge descriptor, and reuse AttemptLoopJudge, the retry/backoff/deadline loop extracted from OpenAICompatibleJudge without changing its behaviour. Also drop contracts with no runtime consumer: the observation/span records, EvaluationDefinition, MetricDefinition, EvaluationRunSummary, unused identifier and reference types, graph schema constants, PersistenceError and FailureCategory.OBSERVATION_OVERFLOW. SpanContextRef, still carried by EvaluationOptions and records, moves to contracts/domain/trace.py.
Make Codex auth selection explicit, preserve nested safe defaults, fail closed when MCP isolation cannot be verified, and reject Claude CLI arguments that override controlled flags. Add regressions and document API-key versus subscription-account usage.
…e faithfulness prompt - Fix the Ruff and ty findings in the Codex judge and its tests, and keep its tool-feature defaults immutable. - Drop evaluation options, run fields and the trace/correlation contracts that nothing reads. - Make Codex auth_mode="subscription" check account/read at startup and fail closed unless the app-server is signed in with a ChatGPT account. - Rewrite the graph faithfulness prompt natively in Chinese and English. It no longer appends a Chinese override to a legacy template, carries no literal placeholders or foreign-language examples, defines out_v as the edge source and in_v as the target, and judges labels by meaning rather than literal wording. - Trim judge comments and docstrings that restate the code.
Reproduce and fix long evidence/finding overflows, duplicate composite keys, unbounded graph/reference/context workloads, and unstable entity recall inventories. Keep global indices, shared limits, and attempt usage across batches; expose explicit matching strategies without changing the maximum-weight default. Add synthetic public-API regressions and document input and scoring boundaries. Remove the temporary offline validation tooling workflow.
Review finding: subscription authentication can exceed a judge deadlineOn head |
Review finding: async cancellation does not interrupt an in-flight judge
This is distinct from |
Review finding: interrupted Codex turn is not explicitly stoppedIn |
|
Follow-up: the interrupt fix can stop a turn once its ID is available. If interruption or timeout occurs while the |
|
The deadline and interruption findings are addressed in Validation: 251 evaluation SDK tests passed with the graph, ROUGE and OpenAI extras installed; Ruff checks, formatting and |
Review finding: an unacknowledged turn can be retried and billed twiceIf |
|
Addressed in Regression coverage includes an actual delayed fake app-server acknowledgement with three configured attempts and a simulated connection loss. The full evaluation suite passed: 252 tests; Ruff and formatting checks passed. The original turn may still run if its acknowledgement never arrives, as documented; no live provider was used. |
Review follow-up: retry after an unconfirmed interruptThere is a related unknown-outcome case after a turn ID is known: a timeout sends |
Review finding: app-server startup can exceed the judge deadline
|
Independent review summaryCurrent head: Score: 9.5/10. Five independent reviews at the earlier head covered the SDK as a whole, adversarial risks, metrics and matching, execution and budgets, and judge integrations; each scored its reviewed scope 9.5/10. The subsequent judge deadline, interruption and uncertain-outcome fixes were independently rechecked by two reviewers, scoring their follow-up scopes 9.5/10 and 10/10. The unchanged sections were not claimed to have received a second full review at the new head. Review findings were published before fixes. Subscription authentication and model inference now share the deadline; Codex connection startup shares one deadline; active or unacknowledged turns with unknown outcomes are not automatically retried. The documentation describes cooperative cancellation and the remaining case where a start acknowledgement is unavailable and provider work may continue. Validation: 254 evaluation SDK tests passed locally with graph, ROUGE and OpenAI extras; Ruff and formatting checks passed. The previous head's CI passed; CI for this head has not reported yet. No live model provider or real HugeGraph integration was tested. The acknowledged-start interruption is best-effort and cannot establish that remote billing has stopped. |
bitflicker64
left a comment
There was a problem hiding this comment.
Blocking: yes. Summary: zh ROUGE-L counts whitespace and Chinese quotes as tokens, empty semantic and evidence inputs score 0.0 and 1.0 instead of not_applicable, and the Codex and OpenAI judges break the user's Codex login and reasoning-model requests.
| if language == "zh": | ||
| import jieba # type: ignore[import-untyped] | ||
|
|
||
| return list(jieba.cut(normalize_answer(text, language))) |
There was a problem hiding this comment.
Important: jieba.cut keeps the spaces that normalize_answer leaves as tokens, so zh ROUGE-L matches whitespace: _chinese_rouge("苹果 香蕉 橙子", "苹果 葡萄 西瓜") scores F1 0.6, against 0.3333 for the same words without spaces. Drop whitespace-only tokens before returning.
| from typing import cast | ||
|
|
||
| # Chinese punctuation set | ||
| _CHINESE_PUNCTUATION = set(",。!?、;:''()【】《》〈〉…—~·「」『』〔〕") |
There was a problem hiding this comment.
Important: the two quote characters in this set are ASCII ', so Chinese quotes “ ” ‘ ’ survive normalization and become zh ROUGE-L tokens: _chinese_rouge("“北京”是首都", "北京是首都") scores F1 0.75 for identical content. Add “”‘’ to the set.
| ) | ||
| candidate = _items(value.candidate, self._spec.kind) | ||
| reference = _items(value.reference, self._spec.kind) | ||
| if not candidate or not reference: |
There was a problem hiding this comment.
Important: when candidate and reference both have no items, such as edges: [] on a vertex-only chunk for the triple metric, the record completes with precision, recall and F1 of 0.0, scoring exact agreement as total failure. Return not_applicable (or 1.0) when both sides are empty.
| session = self.session(context) | ||
| evidence = value.reference.evidence | ||
| artifact = subject(evaluation_input, "retrieval_reference") | ||
| if not evidence: |
There was a problem hiding this comment.
Important: a sample without reference evidence gets evidence_recall_llm 1.0 with no judge call, whatever was retrieved, and retrieval_input defaults reference_evidence to (), so such samples inflate averages. Return not_applicable_record here, as context entity recall does for an empty inventory.
| auth_file = source_home / "auth.json" | ||
| if auth_file.is_file(): | ||
| target_auth = self._isolated_codex_home / "auth.json" | ||
| shutil.copyfile(auth_file, target_auth) |
There was a problem hiding this comment.
Important: Codex refresh tokens are single-use, so a token refresh inside this private CODEX_HOME leaves the user's auth.json holding a used token and forces a new codex login, and a restarted app-server recopies the stale file. Keep auth in the user's CODEX_HOME and isolate through config overrides, or copy a refreshed auth.json back before cleanup.
| ), | ||
| "response_format": request.response_type, | ||
| "temperature": self._settings.temperature, | ||
| "max_tokens": self._settings.max_tokens, |
There was a problem hiding this comment.
Important: every request sends max_tokens and temperature=0, which OpenAI rejects with a 400 for reasoning models (o-series, gpt-5), classified as permanent provider_request_rejected, and openai_judge offers no way to change either. Let callers send max_completion_tokens instead and omit temperature and seed.
| 每个 candidate index 必须恰好出现在 matches 或 false_positives,每个 reference index 必须恰好出现在 | ||
| matches 或 false_negatives;不得返回 tp、fp、fn 或其他键。""", | ||
| }[kind] | ||
| return f"{historical}\n\n---\n\n{protocol}" |
There was a problem hiding this comment.
Minor: the historical templates are appended unrendered, so the opt-in default prompts send literal {question}, {context} and {statements} slots and few-shot JSON written as {{...}}; retrieval_judge_prompt at line 152 does the same. Render the templates (unescape braces, drop the placeholder tail) before appending the protocol.
Summary
This is the ASF-specific continuation of #405 and tracks the original source PR hugegraph#87. A separate branch avoids adding unrelated ASF changes to the source PR. The original PR remains open for reference.
Validation