35
dierent behaviors, inferring and predicting subsequent behavior and experience, evaluating a new
system or search interface feature with users), researchers have to decide the most appropriate
statistics and models (e.g., parametric and nonparametric correlation analysis, linear and nonlinear
regression models, machine learning techniques) in order to fully exploit the potential of the data
collected via user studies and properly answer research questions with relevant empirical evidences.
More detailed discussions regarding the connections between research focus, core components of
study procedures, and data analyses will be presented in Chapter 5.
4.1.6 EXAMPLE OF FACETED ANNOTATION
e previous subsections explain the main facets respectively using recently published, the-state-
of-the-art IIR user studies as examples. To further illustrate our faceted framework, this subsection
presents a representative example of user studies (Bron et al. 2013), and shows how dierent facets,
dimensions, and factors of this study are annotated within our framework. Two more examples
about other types of IIR user studies are presented in the Appendix.
User Study Example: Bron et al. (2013)
Research Focus and Variables
[Research Focus] IIR evaluation: evaluating interface feature; [Keywords (directly ex-
tracted from the paper)] search interface preferences, search behavior aggregated search,
multi-session search tasks; [Independent Variable] interface (tabbed or blended), task;
[Dependent Variable] mouse hover, search moves (paginate, bookmark, lter, change tab,
querying, view documents, bookmark delete, unique queries);
Participant
[Recruitment] longitudinal study: recruiting students from a course (for suitable work
task); controlled lab study: 42 students recruited on campus; [Controlled lab] Yes; [Field/
Large-scale] Yes; [Sample Size] longitudinal study: 25 participants; lab study: 42 partic-
ipants (2 ltered); [Gender Composition] longitudinal study: 12 males and 13 females.
lab study: 12 males, 30 females. [Participant Occupation] student; [Age] longitudinal
study: median = 23; lab study: median = 19; [Education Background/Level] longitudinal
study: postgraduate level in the area of media studies; lab study: undergraduate students;
[Participants’ Native Language(s)] no information; [Language Used in Study] English;
[Regular Incentives] longitudinal study: availability of unique sources. Lab study: no
information. [Extra Incentives/Bonus] no information; [Length of Study] longitudinal
study: 4 weeks; lab study: 1.5 h.
4.1 STRUCTURE OF THE FACETED FRAMEWORK
36 4. FACETED FRAMEWORK OF INTERACTIVE IR USER STUDIES
Task
[Task Source] longitudinal study: assigned work task in class (i.e., a writing project);
lab study: no information; [Search Task Type] longitudinal study: user-generated search
tasks based on the given writing project (work task); lab study: factual amorphous and
factual specic tasks; [Search Task Topic] longitudinal study: in historical context of the
1950s or 1920s, explain the emanicipatory role of a famous female television/lm per-
sonality. lab study: no information; [Number of Tasks] longitudinal study: 1 work task;
lab study: 6 search tasks; [Number of tasks/Person] longitudinal study: 1 work task; lab
study: 3 search tasks; [Time Length/Task] longitudinal study: 4 weeks; lab study: 10 min;
[Work Task Type] intellectual amorphous task; [Did Work Task] longitudinal study:
Yes; [Answer Search Task] lab study: N.A. [Evaluation Task] lab study: collect at least 5
items deemed to be relevant for each task.
Study Procedure and Experimental Design
[Task/Session Feature Controlled] lab study: task order was xed; [Task Rotation]
no information; [Pilot Study] no information; [Pre-study Training] lab session: each
occasion of the study started with a brief introduction to the whole research project and a
viewing of a 5-min tutorial video explaining the use of the experimental search displays.
[Actual Task Completion Time] no information; [Quality Control/Data Filtering Cri-
teria] lab study: for two of the subjects a technical failure prevented recording the pre-ex-
periment questionnaire data, and hence their data were excluded from further analysis.
[Experimental Design] longitudinal study: no clear intervention; lab study: mixed design.
System Features
[Study Interface Element Varied] blended or tabbed; [Other System/Context Feature
Varied] N.A.; [Study Apparatus] desktop computer; [Search Collection/Corpus] six
collections from several archives and library materials; [Ranking Algorithm] BM25;
[non-traditional IR System Assistance Tool] blended and tabbed interfaces.
Behavioral and Search Experience Measures
[Search Behavior Measures] longitudinal study: mouse hover; lab study: search moves
(paginate, bookmark, lter, change tab, queries, view documents, delete bookmark, unique
queries); [Instrument for Collecting Search Behavioral Data] experimental system;
[Relevance Judgment] lab study: participants evaluated documents and collected at
least 5 relevant items; [Instrument for Collecting User Judgment] experimental system;
[Search and System Performance Measures] N.A.; [Neuro-physiological Measures]
N.A.; [Instruments for Capturing Neuro-physiological Measures] N.A.; [Oine In-
..................Content has been hidden....................
You can't read the all page of ebook, please click here login for view all page.