[al3xanndru]

Problem

AE-2513: when a legacy trajectory evaluator's prompt lacks a required
placeholder, loading the evaluator fails with:

Prompt must contain {ExpectedAgentBehavior} and {{AgentRunHistory}} placeholders

The check (LegacyTrajectoryEvaluator.validate_prompt_placeholder) looks for {{ExpectedAgentBehavior}} with
double braces. A user who follows the message literally and writes {ExpectedAgentBehavior} still fails validation.
The DRI triage of SRE-679533 flagged this. In that incident, uip agent eval evaluator add had written a prompt with
{{ExpectedBehavior}}; the root-cause fix is in UiPath/cli.

Change

• The error message now reads Prompt must contain {{ExpectedAgentBehavior}} and {{AgentRunHistory}} placeholders.
  The validation logic is unchanged.
• uipath version bumped from 2.14.44 to 2.14.45 (pyproject.toml and uv.lock), per repo convention. 2.14.45 is
  not on PyPI yet.

Tests

• New test_legacy_trajectory_prompt_error_names_double_brace_placeholders in
  tests/evaluators/test_legacy_trajectory_evaluator.py. It is parametrized over a prompt that uses {{ExpectedBehavior}}
  (the CLI's old default) and a prompt missing {{AgentRunHistory}}, and asserts that the ValidationError names both
  placeholders with double braces. Both cases fail on the old message and pass with the fix.
• pytest tests/evaluators/: 554 passed.
• ruff check, ruff format --check and mypy on the changed files: clean.

Risks / open questions

• Only the message text changes. A caller that matches the old message text exactly would break; none was found in
  this repo.
• The PR title uses the DRI-loop form (AE-2513: …). If commitlint requires conventional commits, retitle it to
  fix(eval): … [AE-2513].

---

Drafted by the wabbit-dri draft-fixes loop for AE-2513 (draft #1), for a human to review: not ready to merge as is.

🤖 Generated with Claude Code

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud