Why theory matters
D.I.K. Sjøberg*,†; G.R. Bergersen*; T. Dybå† * University of Oslo, Norway
† SINTEF ICT, Trondheim, Norway
Keywords
Theory building; Constructs; Propositions; Skill; Expertise reversal; Cognitive load theory
Introduction
Data without theory is blind, but theory without data is mere intellectual play.
Paraphrased from Kant
It is relatively easy to generate and acquire much data from software engineering (SE) activities. The challenge is to obtain meaning from the data that represents something true, rather than spurious. To increase knowledge and insight, more theories should be built and used.
• Theories help predict what will happen in the future.
• Theories explain why things happen, that is, causes and effects as opposed to only identifying correlations.
• Theories help reduce the apparent complexity of the world.
• Theories summarize, condense and accumulate knowledge.
How to Use Theory
More than 10 years ago, we conducted an experiment on the effect of two different control styles in Java programs. The “centralized control style” was supposed to represent poor object-oriented programming. The alternative, the “delegated control style,” was supposed to represent good object-oriented programming. Among the 156 consultants who participated, we found no difference between the control styles regarding the time spent on solving the given tasks, given that the solutions were correct.
Does this imply that type of control style does not matter? No, when digging into the data, we found that only the senior consultants performed consistently better on the “good” solution. For the junior consultants, the result was reversed; they consistently performed better on the “poor” solution.
So, we found the opposite effect of the control style depending on the category of consultants. That one style benefitted juniors and another one benefitted seniors, did that happen by coincidence in our experiment or did we encounter a more general phenomenon described in an existing theory? First, we searched through a literature review on the use of theories in SE experiments conducted by Hannay et al., but found no relevant theories. Then we searched in the scientific psychology literature and found Sweller’s cognitive load theory (admittedly, after searching quite a long time). It states that the cognitive load of novices in learning processes may require other strategies than those that may benefit experts. One effect of the cognitive load theory is the “the expertise reversal effect,” a term coined by Kalyuga et al. to denote the phenomenon that dictates that the best instructional technique may depend on the skills of the learner.
If we replace “instructional technique” with “control style” and consider senior consultants as experts and junior consultants as novices, the expertise reversal effect may also be applied to our case with control style.
How to Build Theory
To make SE a more mature, scientific discipline, we need theories that describe SE phenomena. Theory building may involve adapting an existing theory or developing a theory from scratch. Either case is certainly challenging, but one needs to be brave enough to start theorizing and then seek critique for the proposed theories. Through successive improvements made by the community, the theories will become better and better.
Even though it has hardly been discussed in the SE literature, could it be that the reversal effect that we found in the aforementioned experiment is a general phenomenon in SE? To start theorizing on this issue, let us propose a theory on the reversal effect of SE technology according to the elements recommended in Sjøberg et al.
• Constructs (the basic concepts of the theory)
• Propositions (the interactions among the constructs)
• Explanations (the reasons for the claimed interactions)
• Scope (the circumstances under which the constructs, propositions, and explanations are valid)
Following is the theory on the reversal effect described according to this framework.
Constructs
• Software development skill (the ability of a person to develop software)
• SE technology (method, technique, tool, language, etc. used to support software development)
• Performance (the quality gained and the time saved from using the technology)
• SE technology difficulty (how difficult it is to master the technology to exploit its potential; the more difficult the technology is, the better skills and/or more practice is needed to master it)
• Competing SE technologies (technologies that are supposed to support the same kind of SE activities)
Propositions
• P1: Given that one masters two competing SE technologies, using the most difficult SE technology offers the highest development performance.
• P2: Given two competing SE technologies, T1 with little difficulty and T2 with great difficulty: T1 gives higher development performance than T2 for low-skilled developers; T2 gives higher development performance than T1 for high-skilled developers; see Fig. 1.
Explanation
P1 may be explained by the general case that a more sophisticated technology may have many powerful features that lead to increased performance, but at the same time may be difficult to use. For example, a helicopter may transport more people and goods longer and faster, and across more diverse terrain than a bike, but a helicopter is also more difficult to use.
P2 may be explained by the general case that if you have not mastered the sophisticated technology, you will benefit more from using a less sophisticated technology. When asked to solve an unfamiliar problem, powerful features of the technology may require the understanding of certain abstractions. If one has no idea of what these abstractions represent, solving the problem becomes even harder. In those cases, one will benefit from stepping through the details that are available rather than dealing with abstractions that one does not understand. Think of a helicopter versus a bicycle if one does not know how to operate either. The pedals have a clear connection with the movement of the rear wheel. The operations of a helicopter are much more abstract and thus more difficult to operate without pre-established skills.
Scope
A theory should represent knowledge that is somewhat widely applicable, but still not trivial, or one that follows from a definition. For example, stating in a theory that “software development is a human activity” provides no new knowledge. In disciplines that involve human behavior, one would hardly find such universal theories as Einstein’s general theory of relativity in physics. Given the many context factors that affect software development, it is a tough challenge to define an appropriate scope for an SE theory. In the case of the theory proposed here, we consider the scope to include only technologies that are either in use in industry or are likely to be useful on the basis of evidence from experiments in an artificial setting.
Kurt Lewin stated more than 60 years ago, “there is nothing so practical as a good theory.” Particularly in an engineering discipline like SE, theory should have practical applications. For example, the theory proposed herein could be used in cost-benefit considerations: if an organization has low-skilled (or high-skilled) developers, an easy (or difficult) SE technology should be used. Of course, there are many trade-offs, but then one should consider whether it is sensible to let the high-skilled developers construct a system if only the low-skilled developers will perform maintenance in the future.
In Summary: Find a Theory or Build One Yourself
So, the message here is that if you have obtained a set of data, you should interpret your data in the context of theory. If you do not find such a theory, use your data as a starting point for building theory yourself.
Note, paraphrasing Bunge, “premature theorizing is likely to be wrong—but not sterile—and that a long deferred beginning of theorizing is worse than any number of failures.” Our proposed theory is obviously not generally valid. There are clearly cases where no reversal effect exists; that is, between two competing technologies, one of them may give better performance for all skill levels. A more refined theory may classify SE technologies into two more categories: easy to use technologies that give increased performance for all skill levels (this is a mega success) and more difficult to use technologies that give decreased performance for all skill levels (there are many examples of such failures).
Furthermore, a construct like efficiency or cost-benefit may be included in the theory. Two technologies may have similar efficiency if one of them is twice as difficult to learn as the other, one but also gives twice the performance. However, measuring difficulty and performance on a ratio scale may be difficult. Further theorizing and critiquing are therefore required.
It is the task of the research community to collect or produce more data that may strengthen, refine or refute a proposed theory. But first someone has to propose the initial theory. Why not you?