23
CHAPTER 4
Faceted Framework of IIR User
Studies
Faceted approach as a widely used tool in conceptual analysis and library science classication
research can help researchers deconstruct a complex concept or entity into several specic, interre-
lated facets and dimensions and thereby can facilitate theorization, typology development, as well
as the associated empirical studies. One of the most relevant faceted frameworks in the area of IIR
is Li and Belkin’s (2008) faceted classication scheme of task in information seeking and retrieval.
is chapter presents the faceted framework which was gradually developed and iteratively revised
during the paper coding process, as our response to the very rst research goal that we proposed in
Chapter 1: develop a faceted framework of user study characterization and evaluation and identify
easily ignored user-side facets and factors. To fully address this problem, we rst discuss the main
facets and the associated subfacets (e.g., research focus, task, system feature) here as they jointly
serve as the basis for user study design. en, we identify a series of easily ignored, under-reported
facets that may signicantly aect the process and result of user study.
In the following sections, the identied user study facets or dimensions will be discussed in
detail and be illustrated with the associated examples or representative user studies from selected
venues. e nal version of faceted user study framework is shown in Figure 4.1, and will be
explained piece by piece in this chapter. Built upon the collection of recently published IIR user
studies from a variety of top venues, our faceted framework can oer a comprehensive picture of
the-state-of-the-art user study practice and facilitate faceted analysis (e.g., generating standard
metadata of user studies for summarization, comparative analysis, and evaluation).
4.1 STRUCTURE OF THE FACETED FRAMEWORK
Despite the uniqueness of every user study (especially in terms of the specic techniques and sys-
tem features), it is still possible and also critical for researchers to extract main common facets and
subfacets from a pool of specic IIR user studies and to gure out how these facets are dened,
designed, and manipulated according to the nature of the proposed research problems and goals.
We believe that a comprehensive, faceted framework of user study can serve as a checklist for IIR
researchers to reect on when they design, report, and evaluate various user studies. Based on our
qualitative coding, iterative discussion and continuous revision of the coding scheme, we developed
a faceted framework of IIR user study which consists of nine main facets: (1) the root or starting