Repository navigation
FEAT: Adding Scorer Registry to GUI - #3408
Open
Richard Lundeen (richlundeen) wants to merge 9 commits into
Open
Richard Lundeen (richlundeen) wants to merge 9 commits into
Richard Lundeen (richlundeen) wants to merge 9 commits into
Conversation
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: c7e3facd-c947-4d49-b77a-4077d471767e
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: c7e3facd-c947-4d49-b77a-4077d471767e
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: c7e3facd-c947-4d49-b77a-4077d471767e
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: c7e3facd-c947-4d49-b77a-4077d471767e
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: f5188fba-b6ca-48f4-bd4e-dffad976a38e
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: f5188fba-b6ca-48f4-bd4e-dffad976a38e
Add a scorer registry view, future-response scoring controls, and explicit manual continuation of automated results. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: dfb79987-1604-45bf-92df-1ad37795854a
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: dfb79987-1604-45bf-92df-1ad37795854a
Keep outcome review available on automated results, update send test doubles, and provide scorer catalog responses in browser fixtures. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: dfb79987-1604-45bf-92df-1ad37795854a
varunj-msft
approved these changes
Oct 21, 2026
| if request.target_binding is not None and request.target_binding != binding: | ||
| raise ValueError("This attack's temperature is read-only. Create a new attack to change it.") | ||
| policy = get_chat_scoring_service() | ||
| policy.require_manual(ar) |
Contributor
There was a problem hiding this comment.
Might be worth mapping this PermissionError to a 403 in the /messages and /message-sends routes like PATCH does — right now an automated attack gets a generic 500 instead of the "Continue manually" message.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds Registry > Scorers to browse configured scorers and select a default objective scorer for manual chats.
Screenshots
Screenshots use illustrative scorer configurations, sample metrics, and a sample conversation.
Scorer registry: personal default selection and separate metric comparison tables.
Evaluation details: expand a registered scorer to inspect its evaluation-only configuration.
Chat scoring: choose the scorer for future responses while keeping automated results and human review separate.
Tests and Documentation
Updates the GUI guide with scorer registry usage, personal defaults, automatic chat scoring, failure details, and manual continuation.