xv
Preface
According to the cognitive viewpoint of information retrieval (IR) research, IR systems are essen-
tially interactive and thus users should not be abstracted out of system evaluations. To understand
the role of users in IR and to evaluate systems with users, interactive IR (IIR) researchers usually
design user studies in dierent contexts (i.e., controlled laboratory and naturalistic settings) to
examine various elements, facets, and stages of information search and evaluation process. Since
user study approach has been widely applied in search behavior and IR systems evaluation studies,
an eective evaluation framework of IIR user studies is critical for sharpening the instruments of
IIR research and improving the reliability and validity of the conclusions drawn from IIR user
studies. Despite the longstanding emphasis on evaluation in the IR community, there is still a lack
of evaluation of user studies as such, which hinders the fusion of the rigor of classical IR experi-
mentation and the richness of users’ search interactions. is book is an attempt to highlight this
research gap, synthesize literature around it, and engage the readers in thinking through some of
the ways forward.
We start by presenting a faceted framework for user study evaluation based on a system-
atic review of the state-of-the-art IIR research papers published in ACM SIGIR conferences
(2000–2018), CHIIR conferences (2016–2018), Journal of the Association for Information Science and
Technology (JASIST), Information Processing and Management (IP&M), CHI conferences, and ACM
Transactions on Information Systems (ACM TOIS) in Chapters 2 and 3. Note that we only included
the user studies where researchers proposed IIR-related research questions, recruited participants,
and clearly articulated the major components of study design (e.g., experimental system, task, test
collection), aiming to ensure that the detailed information concerning user study design can be
properly collected and analyzed.
Within the proposed faceted evaluation framework, in Chapter 4 we identify three major
types of research focuses based on the literature review in Chapter 2 (i.e., understanding user be-
havior and experience, system/interface features evaluation, and meta-evaluation of evaluation met-
rics) and explain how dierent facets and factors are employed and combined to jointly support the
explorations on dierent research problems. Also, we highlight the under-reported user study facets
(e.g., language, participant incentives, pilot study), which may signicantly aect the results and the
associated ndings of research. en, in Chapter 5 we employ the proposed faceted framework and
examine the strengths and weaknesses of a series of IIR user studies by (1) evaluating the value and
quality of each individual facet values (e.g., task type, system components), and (2) evaluating the
connections and “collaborations” among dierent facets (e.g., the relationship between participant