41
CHAPTER 5
Evaluating IIR User Studies of
Dierent Types
Chapter 4 covered in detail the main facets of IIR user studies and some of the underlying connec-
tions among these facets. To support our arguments around the proposed framework, we use the-
state-of-the-art user studies selected from multiple IR and HCI venues as examples to illustrate the
facets and the associated subfacets that emerged from our paper coding process.
In this chapter, the faceted framework is applied in evaluating IIR user studies of dier-
ent types. e evaluation of IIR user studies should consider not only the individual facets and
components, but also the role and eects of the connections and “collaborations” among dierent
facets. As discussed in previous chapters, the ideal situation of user study design is an eective
balance among dierent facets and components. e eectiveness of a multi-facet balance refers to
the extent to which the combination of a series of facet values can properly address the proposed
research problem(s).
A study design compromise made in some aspects is usually for the purpose of obtaining
benets from other aspects. For instance, although in-situ or real-time judgment of document
usefulness and search satisfaction may be intrusive to the search process (and thereby may aect
the realism of search behavior data), it may help guarantee the accuracy of usefulness judgments
as participants can make judgments in real-time searching instead of trying to recall their past
search experiences from their limited short-term memory in a post-search questionnaire or inter-
view. us, despite the limitation, this design decision can be a good choice for a research on the
comparison between in-situ and post-search usefulness judgments (meta-evaluation of usefulness
judgment metrics). However, the limitation caused by this decision could become a major problem
if the research goal is to understand users’ web search strategies in local steps (due to the intru-
siveness). erefore, the benets and limitations of a user study design (i.e., facet values and the
potential connections among them) should be evaluated based upon the research questions it seeks
to address. In this chapter, we apply the faceted framework in IIR user study evaluation based on
the following two premises:
• the strengths and weaknesses of a user study should be evaluated with respect to its
ability in answering the proposed research question(s); and
• a user study can be deconstructed as a combination of several interrelated facet values.
..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset