Abstract
In this project I work on a line of user simulators for interactive and conversational search. SimIIR 2.0 extends the original SimIIR toolkit with user-type models, Markov-based interaction, and richer query generation. SimIIR 3 updates the framework for new search tasks and LLM-based pipelines. UXSim adds a hybrid layer that joins rule-based modules with an LLM-powered agent and a web interface, so we can simulate more nuanced user behaviour on real search systems.
Why I care about user simulators
I like to think about search systems from the user side first.
If we want to test new interfaces, ranking models, or agentic search flows, real users are ideal but very costly and slow to recruit. Classic offline metrics are fast, but they ignore how people actually search across a session.
User simulators sit in the middle: they let us run full search sessions in a controlled way, while still keeping some link to real behaviour. This project is my attempt to build a clean stack for that: starting from SimIIR 2.0, moving to SimIIR 3, and then adding UXSim as a bridge to LLM-based agents and modern search UIs.
SimIIR 2.0: a modern base for interactive search simulation
SimIIR 2.0
grew out of the DFG project SiNIR and is the first framework where I played a central role.
The goal was simple: update the original SimIIR toolkit to support modern Python and datasets, model different user types (exploratory, lookup, etc.), and move from fixed rules to Markov-based interaction.
In SimIIR 2.0, a simulated user is a small pipeline: query generator, click model, stopping rule, and so on. Each part can be swapped out, which makes it easy to test new ideas. The Markov models take the current state in the session and decide what to do next: reformulate, scroll, click, or stop.
We used this design to study how different user types react to changes in ranking and interface, plug in new query generators, and run full-session simulations on digital library and web search test beds.
SimIIR 2.0 is open source and lives on GitHub, so others can extend it and drop in their own user models.
SimIIR 3: towards conversational and LLM-aware simulation
As conversational search and LLM-based pipelines became more common, we needed a framework that could handle more complex interactions and system setups.
SimIIR 3
is the next step in that line.
I joined SimIIR 3 as a co-author and helped drive the design towards interactive and conversational search scenarios, cleaner integration with PyTerrier and neural rankers, and support for more flexible query and response generation modules, including LLM-based ones.
The core idea stays the same: user behaviour is built from small blocks (querying, examining, judging, stopping), but the environment can now be a traditional ranked list, a conversational assistant, or a mix of both.
SimIIR 3 is also meant as a community framework: it has a public repository, a standard layout for simulations, and is used in tutorials and workshops on user simulation.
UXSim: hybrid user search simulation
With
UXSim
I wanted to take a more direct step into hybrid simulation.
Traditional simulators are stable and easy to control, but they can feel too rigid. LLM agents are flexible and can handle rich context, but they sometimes drift or act in strange ways. UXSim tries to join the best parts of both.
The framework has three main ideas. First, an orchestration policy: an orchestration module decides when to call a classic simulator component (for example, a click model) and when to call an LLM-based module. This allows the simulation to stay grounded while still using natural language for complex steps.
Second, a cognitive agent over the interface: UXSim wraps the target system behind an interface abstraction. The cognitive agent sees a simple, declarative view of the UI (results, snippets, buttons) and then issues actions like "click result 3" or "scroll". These actions are then applied to the real UI or a faithful mock.
Third, a web UI for running and sharing simulations: a small web interface lets others configure scenarios, run simulations, and inspect traces without touching the code. This lowers the entry bar and makes it easier to share runs with others who may not have direct access to the underlying systems.
In short, UXSim is where I explore hybrid user simulation: classic components set a solid base, and the LLM layer adds flexibility for more nuanced, task-aware behaviour.
How these pieces fit together
The three frameworks play different roles in my research. SimIIR 2.0 is a clean, modular base for interactive search simulation. SimIIR 3 extends this base to conversational settings and LLM-aware environments. UXSim explores hybrid simulations that join classic modules with a cognitive LLM agent and a real UI layer.
Together they form a small stack: from principled, controllable user models, to more open-ended agents, all aimed at stress-testing search systems in a way that respects how users actually search.
Citation
If you found this work useful in your own research, please consider citing the following.
@inproceedings{zerhoudi2022simiir2,
title = {The SimIIR 2.0 Framework: User Types, Markov Model-Based Interaction Simulation, and Advanced Query Generation},
author = {Zerhoudi, Saber and Günther, Sebastian and Plassmeier, Kim and Borst, Timo and Seifert, Christin and Hagen, Matthias and Granitzer, Michael},
booktitle = {Proceedings of the 31st ACM International Conference on Information and Knowledge Management (CIKM '22)},
year = {2022},
doi = {10.1145/3511808.3557711}
}
@inproceedings{azzopardi2024simiir3,
title = {SimIIR 3: A Framework for the Simulation of Interactive and Conversational Information Retrieval},
author = {Azzopardi, Leif and Breuer, Timo and Engelmann, Björn and Kreutz, Christin and MacAvaney, Sean and Maxwell, David and Parry, Andrew and Roegiest, Adam and Wang, Xi and Zerhoudi, Saber},
booktitle = {Proceedings of the 2024 Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region (SIGIR-AP '24)},
year = {2024},
doi = {10.1145/3673791.3698427}
}
@inproceedings{zerhoudi2025uxsim,
title = {UXSim: Towards a Hybrid User Search Simulation},
author = {Zerhoudi, Saber and Granitzer, Michael},
booktitle = {Proceedings of the 34th ACM International Conference on Information and Knowledge Management (CIKM '25)},
year = {2025},
doi = {10.1145/3746252.3761640}
}