Search is an inherently interactive, non-deterministic and user-dependent process. This means that there are many different possible sequences of interactions which could be taken (some ending in success and others ending in failure). Simulation provides a powerful tool for low-cost, repeatable and reproducible evaluations which explore a large range of different possibilities - and enables the analysis of IR systems, interfaces, user behaviour and user strategies. To run a simulation, a model of the user is formalised, and then used, for example, as the basis of a metric, to create a test collection, or generate interaction data. In this talk, I will give an overview of various methods that we have developed in order to: (1) create simulated test collections which enable more extensive evaluations, as well as enable the evaluation on new collections without the expense of costly user judgements, and (2) create user interaction data, which enables a range of different user strategies/behaviours to be compared and contrasted in a systematic manner.
Dr. Leif Azzopardi is a Chancellor’s Fellow in Data Science and Associate Professor at the University of Strathclyde, Glasgow within Department of Computer and Information Science. He leads the Interactive Information Retrieval group within Strathclyde’s iSchool. His research focuses on examining the influence and impact of search technology on people and society and is heavily underpinned by theory. He has made numerous contributions in: (i) the development of statistical language models for document, sentence, expert retrieval, (ii) the simulation and evaluation of users and their interactions, (iii) the analysis of systems and retrieval bias using retrievability theory and the (iv) the formalisation of search and search behaviour using economic theory. He has given numerous keynotes, invited talks and tutorials through out the world on retrievability, search economics, and simulation. He is co-author of the Tango with Django which has seen over 1.5 million visitors. And more recently he has been co-developing resources for IR research with Lucene while co-creating evaluation resources for Technology Assisted Reviews as part of the CLEF eHealth Track 2017.
He is an honorary lecturer at the University of Glasgow (where he was previously a Senior Lecturer) and an honorary Adjunct Associate Professor at Queensland University of Technology. He received his Ph.D. in Computing Science from the University of Paisley in 2006, under the supervision of Prof. Mark Girolami and Prof. Keith van Rijsbergen. Prior to that he received a First Class Honours Degree in Information Science from the University of Newcastle, Australia, 2001.