user-simulation

Agent4Edu: Generating Learner Response Data by LLM-based Agents for Intelligent Education Systems (AAAI 2025, Oral Presentation)

Official Code for SIGIR 2022 "A Multi-task Based Neural Model to Simulate Users in Goal Oriented Dialogue Systems". User Simulator generates user-side utterance, predicts user's next action and satisfaction level.

"Simulating User Satisfaction for the Evaluation of Task-oriented Dialogue Systems" in SIGIR'21

In-Context Learning User Simulators for Task-Oriented Dialog Systems

Official codebase for the paper "MIMESIS: Learning User Simulators as Training Environments for Interactive Agents"

Scenario Testing for AI Agents

τ-Rec: verifiable benchmarks for agentic conversational recommenders. The original movies benchmark (ACM RecSys 2026 Resource Track) and τ-Rec-Books, a controlled books benchmark on Wikidata (ECIR 2027 submission).

Modeling user satisfaction dynamics using a discrete Hawkes process

Knowledge-state-aware user simulator for evaluating information calibration in LLM assistants (arXiv:2608.17150)

Code for paper "IvCDS: An End-to-end Driver Simulator For Personal In-vehicle Conversational Assistant"

Toolkit and benchmark suite for evaluating user simulators in IR: behavioral realism vs. tester reliability, over one canonical session schema — SIGIR 2026

LLM customer agents simulate a real randomized A/B test; measuring how faithfully the simulation reproduces the real treatment effects.

Simulated users with feelings, for testing AI companion chatbots

AssistantEval: a verifiable benchmark for personal AI assistants

Multi-agent beta testing framework where LLaVA vision agents form committees to test web apps. Agents analyze screenshots, discuss actions, vote on consensus, and catch bugs + security vulnerabilities. Playwright browser automation, Streamlit dashboard, FastAPI backend.

Evaluation harness for multi-turn RAG: per-turn rewrite and retrieval checks, conversation-level evals, user simulation, and calibrated LLM judges. Built on IBM’s MTRAG benchmark with LangGraph, Qdrant and Langfuse.

AutoSim: a self-improving user simulator for conversational search evaluation. Code, models, and protocol for the SIGIR-AP 2026 paper.

LLM-based User Simulation Agent for the AgentSociety Challenge. Features Chain-of-Thought reasoning, Iterative RAG, and User Persona extraction. Achieved +5.06% review generation quality over baseline.