Skip to content

feat(eval): add standalone evaluation SDK under GraphRAG - #409

Open
imbajin wants to merge 29 commits into
apache:mainfrom
hugegraph:feat/eval-sdk-asf-20261108
Open

imbajin wants to merge 29 commits into
apache:mainfrom
hugegraph:feat/eval-sdk-asf-20261108

Conversation

@imbajin

@imbajin imbajin commented Nov 8, 2026

Copy link
Copy Markdown
Member

Summary

  • Add an independently installable evaluation SDK for retrieval, answer and property-graph quality, with injectable judge integrations.
  • Add SDK examples, tests and CI coverage.
  • Merge current ASF main and resolve documentation conflicts while keeping the evaluation module guidance.

This is the ASF-specific continuation of #405 and tracks the original source PR hugegraph#87. A separate branch avoids adding unrelated ASF changes to the source PR. The original PR remains open for reference.

Validation

  • Evaluation SDK tests with graph, rouge and OpenAI extras: 246 passed (Python 3.12).
  • Repository CI helper tests: 57 passed.
  • Ruff check for the evaluation module and changed CI helper scripts passed.
  • No live model service or external HugeGraph deployment was exercised.

guoyueqi and others added 26 commits October 29, 2026 11:10
hugegraph-llm-eval scores retrieval, answers and property graphs without the
application: domain contracts, provider-neutral judge ports, answer/retrieval/
graph evaluators, and an in-process engine with ordered results, bounded
concurrency, retries, deadline, cancellation, partial results and per-call cost
accounting. Results are structured records that callers can persist or feed back
into a repair loop.

Supports Python 3.12-3.14 and the dependency ranges introduced by apache#389.

- base install depends on Pydantic only; graph, rouge and openai are extras with
  lazy vendor imports
- one run-scoped limiter caps provider calls at max_samples x max_concurrency;
  nested evaluator batches share it, and cancellation and the deadline are
  checked before every judge call
- examples/all_metrics.py runs all sixteen metrics through both entry points
  with a deterministic stand-in judge, and against a real model when configured
- input builders and an OpenAI-compatible judge factory keep callers away from
  the artifact contract
- CI builds the wheel and sdist outside the workspace, tests the installed
  distribution instead of the source tree, and runs the examples
…racts

Keep semantic pair alternatives until global assignment, validate schema
endpoints and null keys, and correct graph-faithfulness examples. Share
the judge limiter, retain billed retry usage, stop unpriceable budgeted
batches, support ID-only ranking, and preserve atomic claim evidence.

Add regressions for the independently reproduced review findings and
clarify metric, budget, installation and custom-judge boundaries.
Let evaluation run on a Claude or ChatGPT/Codex subscription instead of an
API key.

- ClaudeCodeJudge runs one tool-free headless `claude -p` per call and
  authenticates as the local CLI does; scrub_provider_env and
  setting_sources force the logged-in subscription.
- CodexAppServerJudge shares one `codex app-server` process and runs each
  call as an ephemeral, read-only thread with shell, apps, plugins and user
  MCP servers disabled. Output schemas are normalised to OpenAI strict mode.
- Both report tokens but no cost, record the CLI version in the judge
  descriptor, and reuse AttemptLoopJudge, the retry/backoff/deadline loop
  extracted from OpenAICompatibleJudge without changing its behaviour.

Also drop contracts with no runtime consumer: the observation/span records,
EvaluationDefinition, MetricDefinition, EvaluationRunSummary, unused
identifier and reference types, graph schema constants, PersistenceError and
FailureCategory.OBSERVATION_OVERFLOW. SpanContextRef, still carried by
EvaluationOptions and records, moves to contracts/domain/trace.py.
Make Codex auth selection explicit, preserve nested safe defaults, fail closed when MCP isolation cannot be verified, and reject Claude CLI arguments that override controlled flags. Add regressions and document API-key versus subscription-account usage.
…e faithfulness prompt

- Fix the Ruff and ty findings in the Codex judge and its tests, and keep
  its tool-feature defaults immutable.
- Drop evaluation options, run fields and the trace/correlation contracts
  that nothing reads.
- Make Codex auth_mode="subscription" check account/read at startup and
  fail closed unless the app-server is signed in with a ChatGPT account.
- Rewrite the graph faithfulness prompt natively in Chinese and English.
  It no longer appends a Chinese override to a legacy template, carries no
  literal placeholders or foreign-language examples, defines out_v as the
  edge source and in_v as the target, and judges labels by meaning rather
  than literal wording.
- Trim judge comments and docstrings that restate the code.
Reproduce and fix long evidence/finding overflows, duplicate composite keys,
unbounded graph/reference/context workloads, and unstable entity recall inventories.
Keep global indices, shared limits, and attempt usage across batches; expose
explicit matching strategies without changing the maximum-weight default.
Add synthetic public-API regressions and document input and scoring boundaries.
Remove the temporary offline validation tooling workflow.
@imbajin

imbajin commented Nov 8, 2026

Copy link
Copy Markdown
Member Author

Review finding: subscription authentication can exceed a judge deadline

On head aefc19fefde8ce57d66d1e8cf62c1e8a6092c70d, ClaudeCodeJudge._attempt calls subscription authentication before applying the remaining judge deadline to the model process. Authentication uses a separate timeout of up to 30 seconds, and waiting for the authentication lock is also unbounded. A short-deadline request can therefore spend longer than its remaining budget before model inference starts. Authentication and inference should share the remaining time, including lock wait; I am preparing a regression test and fix.

@imbajin

imbajin commented Nov 8, 2026

Copy link
Copy Markdown
Member Author

Review finding: async cancellation does not interrupt an in-flight judge

aevaluate and aevaluate_many use asyncio.to_thread: cancelling the asyncio task sets a cooperative cancellation token, but JudgeSession checks that token before a judge call and does not interrupt one already in progress. An in-flight Claude subprocess (or another synchronous provider call) can therefore continue until it completes or its timeout expires, potentially continuing provider usage after the caller receives CancelledError. The current aevaluate docstring says cancellation cancels the underlying run without describing this limitation.

This is distinct from KeyboardInterrupt in subprocess.run: Python kills and waits for its direct child on that exception. A robust in-flight cancellation mechanism would require passing cancellation into the judge/provider lifecycle; until then the API documentation should make the cooperative boundary explicit.

@imbajin

imbajin commented Nov 8, 2026

Copy link
Copy Markdown
Member Author

Review finding: interrupted Codex turn is not explicitly stopped

In CodexAppServerJudge._await_turn, a call timeout sends turn/interrupt, but a KeyboardInterrupt while waiting for the turn escapes to the caller without that request. The surrounding cleanup only unwatches and unsubscribes from the thread; it does not explicitly stop the already-started turn. With a shared long-lived app-server, this can leave the turn running after the caller has stopped waiting. I am adding interruption handling and a regression test.

@imbajin

imbajin commented Nov 8, 2026

Copy link
Copy Markdown
Member Author

Follow-up: the interrupt fix can stop a turn once its ID is available. If interruption or timeout occurs while the turn/start response is still pending, the ID is unavailable; closing the shared app-server would disrupt other concurrent evaluations and is not a proven guarantee of stopping remote billing. The README now states this remaining best-effort boundary explicitly.

@imbajin

imbajin commented Nov 8, 2026

Copy link
Copy Markdown
Member Author

The deadline and interruption findings are addressed in ea64317. Subscription authentication lock wait, authentication and inference now share the remaining timeout; a started Codex turn is explicitly interrupted if its wait is interrupted after its ID is known. Cancellation documentation now states the cooperative in-flight boundary and the start-acknowledgement window where an ID is not yet available.

Validation: 251 evaluation SDK tests passed with the graph, ROUGE and OpenAI extras installed; Ruff checks, formatting and git diff --check passed. The added fake app-server test reproduces the delayed start acknowledgement boundary. No live provider integration was run.

@imbajin

imbajin commented Nov 8, 2026

Copy link
Copy Markdown
Member Author

Review finding: an unacknowledged turn can be retried and billed twice

If turn/start has been sent but its response arrives after the timeout, the existing code classifies the timeout as retryable. The server may already have accepted the first turn, so the default retry policy can start additional paid turns while the first outcome is unknown. A start request with an unknown outcome should fail without automatic retry; I am adding a regression test for multiple configured attempts.

@imbajin

imbajin commented Nov 8, 2026

Copy link
Copy Markdown
Member Author

Addressed in 77c7727: a timed-out or disconnected turn/start with an unknown outcome is now non-retryable, so the judge does not automatically start another potentially billable turn. An explicit server rejection retains its existing classification, and the timeout budget is checked before sending the request.

Regression coverage includes an actual delayed fake app-server acknowledgement with three configured attempts and a simulated connection loss. The full evaluation suite passed: 252 tests; Ruff and formatting checks passed. The original turn may still run if its acknowledgement never arrives, as documented; no live provider was used.

@imbajin

imbajin commented Nov 8, 2026

Copy link
Copy Markdown
Member Author

Review follow-up: retry after an unconfirmed interrupt

There is a related unknown-outcome case after a turn ID is known: a timeout sends turn/interrupt asynchronously, without waiting for confirmation, and a connection loss likewise leaves the turn's final state unknown. Automatic retry can therefore overlap or repeat a billable turn before the first one is known to have stopped. I am updating these unknown-outcome paths to avoid automatic retry; explicit completed provider failures remain eligible for their existing retry policy.

@imbajin

imbajin commented Nov 8, 2026

Copy link
Copy Markdown
Member Author

Review finding: app-server startup can exceed the judge deadline

CodexAppServerJudge._connection waits indefinitely for the shared connection lock, and _start gives each initialization RPC the original timeout. Contention or multiple slow startup stages can therefore exceed the remaining judge deadline before turn/start. I am propagating one absolute startup deadline through lock acquisition and each initialization RPC and adding focused regression tests.

@imbajin

imbajin commented Nov 8, 2026

Copy link
Copy Markdown
Member Author

Independent review summary

Current head: 3336a564948a5bc5521af43e66b4d4f56ebb6bfe against base 207fe2cfea4fec38afb9dca3332f5f613ed9d09f.

Score: 9.5/10. Five independent reviews at the earlier head covered the SDK as a whole, adversarial risks, metrics and matching, execution and budgets, and judge integrations; each scored its reviewed scope 9.5/10. The subsequent judge deadline, interruption and uncertain-outcome fixes were independently rechecked by two reviewers, scoring their follow-up scopes 9.5/10 and 10/10. The unchanged sections were not claimed to have received a second full review at the new head.

Review findings were published before fixes. Subscription authentication and model inference now share the deadline; Codex connection startup shares one deadline; active or unacknowledged turns with unknown outcomes are not automatically retried. The documentation describes cooperative cancellation and the remaining case where a start acknowledgement is unavailable and provider work may continue.

Validation: 254 evaluation SDK tests passed locally with graph, ROUGE and OpenAI extras; Ruff and formatting checks passed. The previous head's CI passed; CI for this head has not reported yet. No live model provider or real HugeGraph integration was tested. The acknowledged-start interruption is best-effort and cannot establish that remote billing has stopped.

@bitflicker64 bitflicker64 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: yes. Summary: zh ROUGE-L counts whitespace and Chinese quotes as tokens, empty semantic and evidence inputs score 0.0 and 1.0 instead of not_applicable, and the Codex and OpenAI judges break the user's Codex login and reasoning-model requests.

if language == "zh":
import jieba # type: ignore[import-untyped]

return list(jieba.cut(normalize_answer(text, language)))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Important: jieba.cut keeps the spaces that normalize_answer leaves as tokens, so zh ROUGE-L matches whitespace: _chinese_rouge("苹果 香蕉 橙子", "苹果 葡萄 西瓜") scores F1 0.6, against 0.3333 for the same words without spaces. Drop whitespace-only tokens before returning.

from typing import cast

# Chinese punctuation set
_CHINESE_PUNCTUATION = set(",。!?、;:''()【】《》〈〉…—~·「」『』〔〕")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Important: the two quote characters in this set are ASCII ', so Chinese quotes “ ” ‘ ’ survive normalization and become zh ROUGE-L tokens: _chinese_rouge("“北京”是首都", "北京是首都") scores F1 0.75 for identical content. Add “”‘’ to the set.

)
candidate = _items(value.candidate, self._spec.kind)
reference = _items(value.reference, self._spec.kind)
if not candidate or not reference:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Important: when candidate and reference both have no items, such as edges: [] on a vertex-only chunk for the triple metric, the record completes with precision, recall and F1 of 0.0, scoring exact agreement as total failure. Return not_applicable (or 1.0) when both sides are empty.

session = self.session(context)
evidence = value.reference.evidence
artifact = subject(evaluation_input, "retrieval_reference")
if not evidence:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Important: a sample without reference evidence gets evidence_recall_llm 1.0 with no judge call, whatever was retrieved, and retrieval_input defaults reference_evidence to (), so such samples inflate averages. Return not_applicable_record here, as context entity recall does for an empty inventory.

auth_file = source_home / "auth.json"
if auth_file.is_file():
target_auth = self._isolated_codex_home / "auth.json"
shutil.copyfile(auth_file, target_auth)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Important: Codex refresh tokens are single-use, so a token refresh inside this private CODEX_HOME leaves the user's auth.json holding a used token and forces a new codex login, and a restarted app-server recopies the stale file. Keep auth in the user's CODEX_HOME and isolate through config overrides, or copy a refreshed auth.json back before cleanup.

),
"response_format": request.response_type,
"temperature": self._settings.temperature,
"max_tokens": self._settings.max_tokens,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Important: every request sends max_tokens and temperature=0, which OpenAI rejects with a 400 for reasoning models (o-series, gpt-5), classified as permanent provider_request_rejected, and openai_judge offers no way to change either. Let callers send max_completion_tokens instead and omit temperature and seed.

每个 candidate index 必须恰好出现在 matches 或 false_positives,每个 reference index 必须恰好出现在
matches 或 false_negatives;不得返回 tp、fp、fn 或其他键。""",
}[kind]
return f"{historical}\n\n---\n\n{protocol}"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor: the historical templates are appended unrendered, so the opt-in default prompts send literal {question}, {context} and {statements} slots and few-shot JSON written as {{...}}; retrieval_judge_prompt at line 152 does the same. Render the templates (unescape braces, drop the placeholder tail) before appending the protocol.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants