Repository navigation
feat: agent simulations - #141
TEJASNARAYANS wants to merge 25 commits into
Conversation
 [High impact] The PR adds several workspace-wide agent workflows that run in both chat and voice paths. The PR still needs fixes for analysis jobs overwriting newer results, ignored credential choices, and skipped failure rules before merging. Findings
|
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
… export functionality with workspace_id filtering
…e digest handling
…handling and streamline tag management
… adjust tagging for worker image
…splay success values correctly
…lizing centralized option label functions
… config coercion and improving option label normalization
… generic options type
…alization and strict equality checks
…ric option matching logic
…ation run summary to include failed trial errors
…g.version for improved version handling
| def _reload_cancelled() -> bool: | ||
| db.expire(run, ["metric_clusters"]) | ||
| db.refresh(run) | ||
| return metric_clusters_raw_is_cancelled(run.metric_clusters) |
There was a problem hiding this comment.
Cancelling and immediately regenerating an analysis can let the old job continue. _reload_cancelled checks only metric_clusters.status, which regeneration changes back to "running". If the old job is waiting for an LLM reply, it misses the cancellation and runs alongside its replacement. Both jobs can then overwrite the same results, while charging for both sets of calls.
Give each analysis a unique generation ID and reject writes from older jobs.
| state = generate_simulation_metric_clusters( | ||
| db, | ||
| run, | ||
| provider=provider_enum, | ||
| model=model_str, | ||
| on_progress=on_progress, | ||
| max_llm_calls=max_llm_calls, | ||
| is_cancelled=_reload_cancelled, |
There was a problem hiding this comment.
The selected credential_id never reaches the clustering calls. The worker uses it to choose the provider and model, but passes only those values to generate_simulation_metric_clusters. The shared LLM helper then chooses the default credential instead.
If an organization has two credentials for the same provider, clustering uses the wrong key or gateway. It can fail when the model is available only through the selected credential. Pass the credential through every clustering call, as the prompt-suggestion step already does.
| source_rows=source_rows, | ||
| metrics=metrics, | ||
| policies={}, | ||
| on_progress=on_progress, | ||
| max_llm_calls=max_llm_calls, | ||
| is_cancelled=is_cancelled, | ||
| failure_predicate=simulation_failure_predicate(run, metrics_by_id), |
There was a problem hiding this comment.
Simulation success rules can use user_insight metrics, but this call sends them through a helper that always filters them out. A run that fails only those rules gets no failure analysis or prompt suggestions. It is instead told that no completed trial missed a rule.
Let the simulation path cluster every metric that has a success rule, rather than applying the call-import quality-only filter.
| clusters = payload.get("metric_clusters") | ||
| if isinstance(clusters, dict) and clusters.get("status") == "completed": | ||
| lines.extend(["", "04 Failure Diagnostics"]) | ||
| for group in clusters.get("groups") or []: | ||
| if not isinstance(group, dict): | ||
| continue | ||
| lines.append(f"{group.get('metric_name')}: {group.get('flagged_count')} flagged") | ||
| for cluster in group.get("clusters") or []: | ||
| if isinstance(cluster, dict): | ||
| lines.append(f" - {cluster.get('label')} ({cluster.get('count')})") | ||
| improvements = payload.get("prompt_improvements") | ||
| if isinstance(improvements, dict) and improvements.get("status") == "completed": | ||
| lines.extend(["", "05 Prompt Improvement Recommendations"]) |
There was a problem hiding this comment.
If WeasyPrint is unavailable or fails, _simulation_plain_text_lines ignores the selected sections. The downloaded PDF includes the summary, completed failure diagnostics, and prompt suggestions even when the user unchecked them.
Apply the same section checks used by the HTML report so users get the limited report they requested.
What Changed?
search_knowledge_basetool, download/view fixes, serialized ingest with advisory locks.efficientai-worker-harborimage (Python 3.12 + harbor extra), workspace capability checks on simulation/KB/tool routes.Why?
Teams need to batch-evaluate agents against scenarios and personas with the same metrics and gates used in production, without manual call scripting. Harbor runs trials at scale; the platform owns export, scoring, and artifacts.
KB + tools let simulations and voice agents use grounded answers and mocked or live integrations. Workspace scoping and metric filters prevent cross-workspace leaks (scenarios, personas, private metrics, rubrics).
How to Test?
docker compose build worker-harbor && docker compose up worker-harbor(or deployedefficientai-worker-harborimage) → run queues onharbor-sim.ready→ chat/voice search returns chunks; download original in a non-default workspace (link includesworkspace_id).mock_responseis set; live tools when not mocked.make test-parallel(or targeted harbor/KB/tool/metrics modules).Release Label
minor — backward-compatible features (simulations, KB, tools, metric generation); no intentional breaking API changes.
Checklist