xiii
Contents
Preface ............................................................ xv
Acknowledgments .................................................. xvii
1 Introduction ....................................................... 1
1.1 Background: User Study in IIR Research ............................ 2
1.2 Aim of the Work ............................................... 2
1.3 Systematic Review and User Study Evaluation ........................ 3
1.4 Implications of the Faceted Framework .............................. 4
2 Interactive Information Retrieval ...................................... 7
2.1 Background ................................................... 7
2.2 Understanding User Behavior and Experience ....................... 10
2.3 System/Interface Features Evaluation .............................. 12
2.4 Meta-Evaluation of Evaluation Metrics ............................ 13
2.5 Summary .................................................... 14
3 Methodology: Paper Selection and Coding Scheme ....................... 15
3.1 Paper Selection: SIGIR, CHIIR, CHI, JASIST, IP&M, and TOIS ....... 17
3.2 Coding Scheme ............................................... 20
4 Faceted Framework of IIR User Studies ................................. 23
4.1 Structure of the Faceted Framework ............................... 23
4.1.1 Research Focus and Variables .............................. 25
4.1.2 Participant and Task ..................................... 27
4.1.3 System Feature ......................................... 31
4.1.4 Study Procedure and Experimental Design ................... 32
4.1.5 User Behavior, Experience, Data Analysis, and Results .......... 34
4.1.6 Example of Faceted Annotation ............................ 35
4.2 Under-Reported Facets and Factors ............................... 37
4.3 Summary .................................................... 40
5 Evaluating IIR User Studies of Dierent Types ........................... 41
5.1 Understanding User Behavior and Experience ....................... 42