Skip to content

fix(judge): read a verdict object that follows other braces - #5

Open
rudycelekli wants to merge 1 commit into
openbenchmarks-labs:mainfrom
rudycelekli:fix/judge-json-before-verdict
Open

rudycelekli wants to merge 1 commit into
openbenchmarks-labs:mainfrom
rudycelekli:fix/judge-json-before-verdict

Conversation

@rudycelekli

Copy link
Copy Markdown

What does this PR do

parse_verdicts took the first { through the last } and parsed that as one object. A reply that mentions another brace before the verdict list, which models do, failed to parse. The whole batch was then left unscored.

The parser now reads the first JSON object that actually contains a verdicts list. An unreadable reply is still no verdicts.

Made with Cursor

The first brace through the last brace is not always the verdict JSON, so a usable reply was dropped and the batch stayed unscored.

Co-authored-by: Cursor <cursoragent@cursor.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant