6.3.1. One independent variable

When a single independent variable is examined, there are two main types of experimental design, depending on whether the participants take part in one or in all of the conditions. In a between-subject design (or independent measures), every person takes part in only one condition, whereas in a within-subject design (repeated-measures design), every participant takes part in all the conditions.

The choice of the experimental design depends first on the type of independent variable examined. When the independent variable cannot be manipulated by researchers and corresponds to an inherent characteristic of the individual, such as mother tongue or intellectual abilities, we can only use a between-subject design. A person’s characteristic can correspond to only one of the modalities of the variable. When the independent variable can be manipulated by the researcher, it is possible to implement either a between-subject or a within-subject design.

Different factors come into play when choosing a between-subject or a within-subject design. The first important factor is the control of external variables. In a between-subject design, where the conditions do not contain the same individuals, it is necessary to ensure that the characteristics of the individuals that may influence the dependent variable, are distributed evenly between the conditions. Imagine an experiment comparing sentence comprehension in a spoken or a written modality. An external variable could be the intellectual level of the participants, measured through their IQ score. If the participants in the spoken condition have a higher IQ than those in the written condition, then the results of the experiment could be influenced by the IQ level in addition to the modality of presentation.

In order to control external variables, one solution could be to assign participants to the conditions randomly. For an assignment to qualify as random, every person should have the same chance of being assigned to a condition and that assignment should not depend on the characteristics of other participants. In this case, the assignment criterion should be the chance factor, so that every time a person showed up, a coin is tossed to decide which condition they should be included in. In our example, a person with a high IQ would have the same chance as a person with a lower IQ of belonging to each condition. In real life, such a random assignment is difficult to implement, since it could lead to a highly unequal number of participants between conditions. It is therefore preferable to use block randomization, in which all the conditions appear once within a block, before moving on to the following block. Within each block, the order of the conditions is random. For example, for a design with three conditions, the first three participants could be assigned to conditions 1, then 3, then 2, the next three to conditions 2, 3, then 1, and so on. We cannot rule out the fact that even by using such randomization, the participants of the different groups may systematically differ on some points. However, this method is effective, particularly for large samples.

Another way of controlling external variables, in the case of a between-subject design, would be to assign similar individuals to each condition. In this case, the relevant characteristics have to be defined beforehand, so that the groups of participants are equivalent. Going back to our example, it would be desirable for the two groups of participants not to differ in terms of IQ. Of course, it would be impossible to only recruit individuals with the same IQ score. What would be more feasible, is to make sure that the IQ scores of the two groups belong to a similar range and on average, individuals in the first group have a similar IQ to those in the second group. This type of assignment is only doable when the criteria that need to be taken into account are few and easily measurable. When the number of criteria increases, it becomes very difficult, not to say impossible, to recruit similar participants. For example, we can easily imagine the difficulty of finding homogeneous groups of 20-year-old bilingual French–Portuguese participants with a similar IQ. When multiple constraints must be met, it is necessary to turn to a within-subject design.

In a within-subject design, participants work as their own control, since their characteristics follow them from one condition to another. In our example, IQ would no longer be a problem, since a person with a high or a low IQ would be tested in both conditions of modality presentation. As a consequence, the effects of modality presentation could no longer be attributed to the characteristics of the participants. In this respect, within-subject designs resolve the difficulties associated with building even groups of participants. On the other hand, they show a risk of spill-over effects, that is, participating in one condition may influence the responses given in a condition presented afterwards.

There are different kinds of spill-over effects. One of them, the order effect correponds to the fact that the order of participation in the different conditions could become an involuntary factor within the experiment. In short, depending on the order in which the conditions are presented, the results may differ. For example, this effect was shown in the case of scalar implicatures. Children derive more scalar implicatures linked to words such as some, namely some but not all, when the conditions with the words some and all are alternated, producing a contrast effect which encourages the derivation of the implicature, when compared to an experiment in which all the items with some are presented in a block, followed by another block containing all the items with all (Skordos and Papafragou 2012)¹¹. Another spill-over effect is that of learning, in the specific case where participation in one condition improves performance in the other condition. In the example we are analyzing, this would mean that simply carrying out the comprehension task in one modality condition, could improve the performance in the second condition. This would, of course, be the case if the sentences were the same in the two conditions. It is for this very reason that different sentences should be presented. We will return to this in the section on experimental material. This could also be the case if the participants developed specific strategies in the first condition, which could then be reused in the second condition. The effect of fatigue is yet another example of a spill-over effect producing opposite results to those from the learning effect. As conditions go by, performance levels decrease, due to the fatigue or weariness participants start to experience.

In order to overcome these different spill-over effects, two counterbalancing solutions can be implemented. The first way of counterbalancing would imply randomly modifying the order of presentation of the different items in the experiment, all conditions combined. The second way of counterbalancing would be to modify the order of presentation of the items within the conditions, and to modify the order of presentation of the conditions themselves. For an independent variable with two modalities, half of the participants would first take part in condition 1, then in condition 2, whereas the other half would take part in condition 2, then in condition 1. For example, half of the participants would complete the task in the spoken condition before moving on to the written condition. As the number of modalities for the independent variable increases, it very quickly becomes difficult to counterbalance the conditions. With three modalities, there would be six possible orders, and 24 possible orders for four modalities. It is therefore necessary to resort to partial counterbalancing, also known as Latin square design. In this type of design, rather than presenting all the possible combinations, we choose a portion of these so that each condition appears in a possible position, as illustrated below.

We should nonetheless note that there are situations where counterbalancing conditions is not convenient. For example, this would be the case for an independent variable involving a process difficult to cancel once activated. For example, Gillioz et al. (2012) studied the influence of different factors on the construction of emotional inferences. One of these factors corresponded to the simulation process, by which the readers imagined being characters of a story, in order to understand it from the inside. In this experiment, the simulation was manipulated through a within-subject design, by giving no specific instructions to the participants in the first part of the experiment, before specifically asking them to follow a simulation strategy in the second part. Here, a counterbalancing of the simulation conditions was not feasible, since adopting a simulation strategy during reading is difficult to cancel at a simple request. Therefore, a decision was made to present the conditions in the same order, at the risk of unintentionally inducing order effects into the results. Another solution would have been to manipulate this variable between the participants, while being careful to build even groups, as previously explained.

In summary, it is possible to build between-subject or within-subject designs. In order to control the external variables, within-subject designs should be used instead of between-subject designs whenever this is possible.

6.3.2. Several independent variables: factorial designs

An experiment can also be used to simultaneously test the role of several independent variables, by using a factorial design. This type of design makes it possible to test the hypotheses related to each independent variable, as well as to observe the joint influence of the independent variables, for instance, to determine whether the influence of a variable depends on the modality of another variable.

The simplest factorial design contains two variables, with two modalities each. It can be presented in a simplified way as a 2x2 design. In this type of design, the modalities of the two variables are combined to produce four conditions. Imagine that you want to extend the study about the modality of presentation (spoken or written) on sentence comprehension, by adding another independent variable, such as sentence complexity. The experiment should present spoken and written sentences in order to study the first variable. In addition, complex sentences and simple sentences should be used for studying the second variable. Combining these variables would result in the four conditions described below:

In a factorial design, every independent variable can correspond to a between-subject or within-subject measurement. To continue with our example, one possibility would be to follow a 2x2 design with independent measurements, in which participants only take part in one condition. In this case, the two independent variables would be between-subjects. Alternatively, it would be possible to follow a 2x2 repeated-measures design, in which the participants take part in all the conditions. Here, the independent variables would both be within-subjects. Finally, it would be possible to set up a mixed 2x2 design, in which one of the variables is between-subject and the other within-subject. In this case, participants would see two of the four conditions presented above, either a single type of sentence presented in spoken and written modalities, or the two types of sentences presented in a single modality.

When using factorial designs, different effects can be observed: the main effects and the interaction effects. The main effects correspond to the general effect of an independent variable on the dependent variable. In the example above, since there are two variables, two main effects can be observed. The first main effect would correspond to the presentation modality (spoken vs. written) effect on sentence comprehension, regardless of the type of sentences involved. For example, it may be that in general, participants better understand written sentences. The second main effect would correspond to the type of sentence effect on comprehension, regardless of the presentation modality. It is probable that simple sentences are generally better understood than complex ones. Finally, the interaction effect corresponds to the effect of one variable depending on the modality of the other variable. In the case of our example, a possible interaction effect would be that simple sentences are equally well understood in the spoken and written modalities, whereas complex sentences are better understood in writing than when spoken. Thus, the presentation modality effect would depend on the complexity of the sentence, since it would only be observed in the case of complex sentences.

In a factorial design, the number of modalities for every variable may vary, as well as the number of independent variables examined. For example, in a 2x3 design, two variables would be manipulated, one with two modalities and the second with three modalities. A 2x2x2 design would include three variables with two modalities each. Manipulating more than three variables in the same experiment is, however, not recommended, since the effects of interactions from such designs can become very complex to interpret.

6.4.1. Experimental items

The nature of the experimental material developed for an experiment depends on the independent variables formulated in the research hypothesis, as well as on the task chosen. In most of the examples discussed in the previous chapters, we have seen that the participants had to judge, recall, read or even react to linguistic stimuli belonging to one and/or the other conditions of the experiment. These linguistic stimuli, called items, are selected so as to manipulate the independent variable and to control the external variables.

Before going further, it is important to understand the concept of an item. An item is an element for which a response is recorded in an experiment. For example, in a lexical decision task, an item corresponds to a series of letters that the participants have to categorize either as a word or a non-word. In an acceptability judgment task, an item could be a sentence whose acceptability participants have to judge on a scale; in a comprehension, reading or recall task, an item may correspond to a sentence or a short passage. Experiments contain many items, in order to collect a reliable measurement of the process investigated. Testing multiple items, as well as testing multiple participants, decreases the portion of error that is attributed to the specific characteristics of the items or the participants.

Experimental items are constructed so as to vary the properties investigated, while keeping the other properties as stable as possible. For example, an experiment on scalar implicatures could contrast two conditions: on one hand, the sentence “some kids like chocolate”, which gives rise to the implicature, and, on the other hand, the sentence “all kids like chocolate”, represents another condition. In the same way that the use of a repeated-measures design decreases the biases related to participants, repeating an item through all the conditions may decrease the biases this could entail. For this reason, items are developed so that they can appear in the different conditions whenever possible. Later, we will discuss how to distribute the items across the different conditions, but it is already useful to take this point into account when building the experimental material.

To illustrate these principles, let us go back to the example on the influence of presentation modality and complexity on sentence comprehension. Sentence complexity could be operationalized by the presence of a relative clause with a subject pronoun (less complex), or an object pronoun (more complex), for instance. In order to manipulate the presentation modality, half of the sentences would have to be presented orally and the other half in writing. The dependent variables of this experiment could be the number of correct answers given to verification questions, following the presentation of the sentence, as well as the response time needed to provide such answers.

Following these criteria, the experimental items of this experiment could take the form:

1. (1a) The woman who follows the man carries an umbrella.
2. (1b) The woman whom the man follows carries an umbrella.
3. (2a) Elegantly, the courtier who adores his sweetheart picks her up to kiss her.
4. (2b) Elegantly, the courtier whom his sweetheart adores, picks her up to kiss her.

The two sentences in each pair clearly differ on the role of the relative pronoun. However, the two pairs of sentences also differ in structure, which can be problematic. The second pair of sentences is more syntactically complex than the first one. It also contains less frequent vocabulary, which could entail comprehension difficulties. In this case, the items would vary in relation to other complexity aspects than those investigated (the role of the relative pronoun), which increases the possibility of external variable involvement. In order to avoid the intrusion of unwanted external variables, it is preferable to build a homogeneous set of items, by establishing criteria relating to their structure, their style, language register, etc. applicable to all items.

In experiments where the material corresponds to sentences or texts, compliance with these criteria can be checked by means of a pretest, where people are asked to give their opinion about some of the material’s features. These people can be colleagues or people with the same general characteristics as the participants in the experiment. Performing a pretest may sometimes seem superfluous depending on the criteria applied to the items, but it is important to keep in mind that the judgment of researchers, albeit informed and justified, is not always shared by others, especially by the participants. For example, imagine an experiment on the influence of the emotional connotation of a text on the type of information drawn from it. In order to carry out this study, it would be necessary to choose or build texts conveying positive or negative emotions, as well as texts conveying neutral information. The judgment of a single person would be problematic in this case, because many parameters can influence the evaluation of the emotions conveyed by a text. These parameters may differ from respondent to respondent, but they may also differ in their relative importance regarding the attribution of emotions to texts. In this case, it would be compulsory to pretest the items and to obtain the evaluations of different people, in order to make sure that the experimental material is valid.

When the experimental material is a word list, numerous databases are available for obtaining the different relevant characteristics of the words. Different researchers and research groups have drawn up lists of existing databases for preparing experimental material. For example, this is the case for websites such as The Language Goldmine¹², Experimental Linguistics in the Field¹³, the Postdam Research Institute for Multilingualism¹⁴ or OpenLexicon¹⁵, which we encourage you to consult, in order to become familiar with the various types of available information for creating experimental material. These websites also list resources specifically tailored to the choice of suitable non-words or standardized images following different criteria.

Lexical databases list many objective word characteristics, such as the number of syllables, phonemes and orthographic neighbors, or their frequency in the language. Different frequency indicators are often accessible, one being calculated on the basis of a corpus of texts and the other on the basis of a corpus of film subtitles, which better reflect word frequency in the spoken modality. Studies have shown that the frequencies drawn from the subtitle corpus predicted word reading time better than their written frequency (New et al. 2007; Brysbaert and New 2009).

It is also possible to access more subjective data, such as the age of word acquisition, their familiarity, their emotional valence, their concrete nature, their imagery value or their subjective frequency. These different characteristics are assessed on the basis of judgments performed by native speakers. As a result, this data is only available for a limited word sample in certain languages.

In English, we can turn to databases such as CELEX¹⁶ (Baayen et al. 1995) or the MRC Psycholinguistic Database¹⁷ (Coltheart 1981). The latter gathers information drawn from different sources, in order to provide data relating to 26 different linguistic properties. Subtlex-US¹⁸ (Brysbaert and New 2009) and Subtlex-UK¹⁹ (van Heuven et al. 2014) contain frequencies based on film subtitles. Subtlex databases also exist for Dutch (Keuleers et al. 2010), Chinese (Cai and Brysbaert 2010), German (Brysbaert et al. 2011), Greek (Dimitropoulou et al. 2010), Spanish (Cuetos et al. 2011), Italian (Crepaldi et al. 2015), Portuguese (Soares et al. 2015), Polish (Mandera et al. 2014) and French, the latter being accessible on Lexique3²⁰ (New 2006).

For 10 years, data relating to naming and lexical decision times for tens of thousands of words and non-words in different languages have been collected and made available to researchers. This data is accessible via the Lexicon Project in different languages; for example, in English, they can be found in The English Lexicon Project²¹ (Balota et al. 2007) or in the British Lexicon Project²² (Keuleers et al. 2012), in French in the French Lexicon Project (Ferrand et al. 2010), in Dutch (Brysbaert et al. 2016, Keuleers et al. 2010), in Chinese (Sze et al. 2016) or in Spanish (Aguasvivas et al. 2018).

Databases specifically related to children’s lexicon are also available in different languages. In addition to the Subtlex databases which also include information on TV programs for children, statistics based on text corpora for children such as the American Heritage Word-Frequency Book (Carroll et al. 1971), or corpora of interactions with children like the CHILDES database (MacWhinney 2000) are available. TheChildFreq tool²³ (Bååth 2010) also makes it possible to search the CHILDES database in order to retrieve information relating to the interactions of American and British children.

However, it is possible that the research question involves examining a variable for which there is no published standard. It is then up to the researchers to select and validate the chosen words by implementing a pretest. For example, such a pretest was necessary for a study aimed at determining the influence of the context on the activation of color in mental representation, conducted by Connell and Lynott (2009). In this experiment, the participants read sentences describing objects whose typical color or alternative color was implied by the context. For example, one sentence described either a bear in the forest, which activated the brown color, or a bear at the North Pole, which activated the white color. Another story described either a ripe banana, which activated the yellow color, or an unripe banana, which activated the green color. In order to build this experimental material, it was first necessary to identify the objects, animals or plants which could take up different colors. A pretest was then conducted to determine the typical color of these objects and their alternative color, which were activated in most participants.

6.4.2. Filler items

Once the experimental items have been prepared, the filler items can be created so that the experiment is complete. The experiment includes this type of item for two reasons. First, they are essential for all tasks requiring a YES/NO response from the participants. In these tasks, the independent variable is generally manipulated for the items associated with the YES response. For participants to have the opportunity to answer NO to some of the elements presented to them, filler items associated with the NO response need to be added. In a lexical decision task, for example, filler items are non-words. In a verification task, they correspond to the elements which have not been presented in the text.

Secondly, filler items make it possible to conceal the real purpose of the task from participants. We have already mentioned the fact that it is necessary to test naive people, for the results to not be biased. By concealing the goals of the task, we aim to reduce the risk of participants suspecting which independent variables are being manipulated. For example, in the study by Connell and Lynott (2009) presented above, filler items were sentences describing objects which had no typical color associated with them. In the filler items, it is also possible to manipulate a completely different variable other than the one actually being investigated, in order to divert the participants’ attention. For example, in the study by Zufferey et al. (2015b) on the understanding of connectives by learners, some of the filler items included obvious grammatical errors, such as an incorrect subject–verb agreement. The number of filler items should generally be equal to or greater than the number of experimental items (Havik et al. 2009; Jegerski 2014).

6.4.3. Other aspects of the material

Experiments on language comprehension, implementing reading paradigms such as self-paced reading or eye-tracking, require participants to be presented with comprehension questions after reading some items, usually fillers. These comprehension questions aim to ensure that participants carry out the task properly by following the instructions, reading and trying to understand the sentences or short texts presented to them. Reading time or eye movement measurements obtained from participants who did not really process the text, but simply pressed the response keys only to move forward within the experiment, would be purposeless and may even risk preventing the demonstration of the effect. Comprehension questions are generally simple questions, expecting a YES/NO type of response. It is important to note that the degree of simplicity of these questions may influence the natural reading process, as participants sometimes develop strategies for answering these questions (Havik et al. 2009; Jegerski 2014).

6.4.4. The concept of lists

As discussed above, we encourage the use of repeated-measures designs, since they make it possible to assign the same participants to different experimental conditions. In the same way as participants, the items selected for the experiment represent only a sample of all possible items. These items also have their own characteristics which can influence the results, depending on the words or the types of sentences chosen. In order to control this influence, within-item designs should be implemented whenever possible. In other words, the same item should appear in all conditions. However, in most experiments, it is preferable to avoid having a participant see the same item in more than one condition, so as to prevent the familiarity effect. Indeed, the answer given during the second run risks being influenced by the fact that the item has already been seen and processed. For every item to be presented in each condition and for every participant to see every item only in one condition, it is necessary to organize items as lists. Every participant should see no more than one list during the experiment.

Let us go back to our fictitious example of an experiment on how the presentation modality might impact sentence comprehension. For this experiment, we imagine that we have created 40 items (40 sentences having the same structure). In order to set up a within-subject and within-item design, all of the items should appear once in their written form and once in their spoken form. Every participant should also be exposed to items in their spoken form and others in their written form. It would therefore be necessary to set up two lists of items. The first would contain items 1–20 in the written form and items 21–40 in the spoken form, whereas the second list would contain items 1–20 in the spoken form and 21–40 in the written form.

If we add the variable related to sentence complexity to this experiment, and we want to follow a within-subject and within-item design, every item and every participant should be confronted with the four possible conditions (all variables combined). In this case, we would have to create four lists according to the following model:

6.4.5. Number of items to be included in an experiment

An important question when creating experimental items concerns the number of items that must be included in the experiment. It is impossible to answer this question in a definite manner, as it depends on the effect size, the task implemented and the characteristics of the material. The experiment by Connell and Lynott (2009), for example, contained only 10 items, due to the very rare specificities of the words used, namely the fact of representing an object with a clearly defined typical/atypical color. Conversely, some studies implementing lexical decision tasks include more than a hundred, and sometimes even several hundred items (e.g. Carreiras et al. 2005; Perea et al. 2015). For a long time, choosing the number of items was decided on the basis of what was regularly done in a specific field. Recently, it has been suggested to target a minimum of 1,600 observations per condition when measuring reaction time in repeated-measures designs (Brysbaert and Stevens 2018). This number of observations is the product of the number of participants and the number of experimental items per condition, representing 40 participants seeing 40 items, or 20 participants seeing 80 items, for example. The criteria to be considered when choosing these numbers depends on the task: for simple tasks, processing 80–100 experimental items and 80–100 filler items poses no problem. For more complex tasks, the accumulation of items may induce fatigue effects which should preferably be avoided. In such cases, it might be better to test fewer items on more participants.

6.5.1. Instructions

Every experiment begins by clearly explaining to the participants how the task will unfold and what is expected from them. As we will see below, a task is made up of different trials, which are repeated a certain number of times and for which the participants have to perform the same action. For example, in a lexical decision task, a trial corresponds to the categorization of a string of letters into words and non-words. The instructions have to make it clear to participants how to give their answers. It is also essential to ask participants to keep their fingers on the answer keys throughout the experiment, in order to be able to react as quickly as possible. For a lexical decision task, the instructions could be as follows:

In this experiment, you will perform a lexical decision task. This means that we will present you with strings of letters and that you will have to decide, for each of them, whether they form an existing word in English or not.

The experiment will proceed as follows. First, you will see the message “Are you ready?” on the screen. When you are ready, you can press the YES key. This will bring up a fixation point in the center of the screen, for half a second. Please fix this point. The fixation point will then be replaced by a string of letters. At that moment, you will have to decide as rapidly and as accurately as possible, whether this string of letters is a word that exists in English or not. If it is a word, press the YES key. If it is not a word, press the NO key. The experiment will consist of 10 training trials, then 60 trials. It should last about 15 minutes. At the end of the experiment, you will see a message indicating that the experiment is over.

Please place your forefingers on the YES and NO keys and keep them on these keys throughout the experiment. From the moment you start a trial, be sure to give your answer rapidly and accurately. You can take a break at any time when you see the message “Are you ready?”.

If you have any questions, you can ask the person in charge of the experiment now.

6.5.2. Experimental trials

After presenting the instructions, the task begins. As we saw above, a task is divided into trials, each corresponding to an item. Depending on the task, the trials vary, but some characteristics remain the same, namely how items are presented and how responses are recorded. For online tasks, it is customary to precede every trial with a message such as “Ready to continue?”, to which participants reply YES in order to start the trial. This enables participants to prepare for the task and allows them to take a break when needed during the experiment. As soon as the trial begins, the elements are presented and responses to the different elements are recorded. When the experiment requires the recording of reaction times or reading times of single words, the presentation of the item (word or phrase) is generally preceded by a fixation point at the location where the item will appear, in order to attract the attention of the participants and reduce variations in the data collected. When recording the reading time of sentences or text segments, the use of a fixation point is not compulsory. On the other hand, in eye-tracking experiments, the accuracy of the measurement is very important and every trial begins with a fixation point. Figure 6.2 illustrates the different types of trials for some of the tasks presented in previous chapters.

The construction of experimental trials also requires defining how long the items will be presented. Sometimes, the participants set their own pace, as in self-paced reading tasks, in which pressing a key determines the progression in the experiment. In other cases, for example, in experiments based on a priming effect, it is necessary to precisely define the duration allocated to the presentation of the prime, and the time lapsed until the presentation of the target.

In order to prevent possible spill-over effects, the presentation of the items has to be randomized. In other words, for every participant, the order for presenting the items is established randomly. It is then highly unlikely that two participants will see the items in the same order. Randomization also makes it possible to avoid some items being systematically presented at the beginning or at the end of the task, which could lead to learning or fatigue effects, or cause the processing of a certain item to be regularly influenced by the one preceding it.

Depending on the number of items and the type of experimental design, the number of trials may be very high, which could induce loss of attention or fatigue in the participants, and thus jeopardize the quality of the data collected. One solution to this problem may be to divide the trials into blocks, so that the experiment can be segmented into shorter portions. The use of blocks can also be convenient for presenting all the items in one condition, before moving on to another condition. For example, to study oral and written comprehension, it is preferable to present the sentences in only one modality before moving on to the other, since having to constantly switch from one modality to the other could pose additional difficulties to participants. When using blocks, it is advisable to try to counterbalance not only the order of presentation of the items within the blocks, but also the blocks themselves.

For participants to get used to the task, the experiment begins with the presentation of training items, during which the experimenter can verify the correct understanding of the instructions and re-explain them if necessary. The experiment ends when all the items have been presented. At this point, the experimenter usually thanks the participants and answers any questions they may have. It may also be useful to ask the participants for feedback on their perception of the experiment and to survey their intuitions on the nature of the question being investigated, in order to determine whether their behavior may have been influenced in a way that could affect the experiment’s quality.

6.9.1. Questions

1. 1) Transform the following hypothesis into an operational hypothesis, then schematize it by including the external variables that you consider the most important, in terms of items and participants: “A person’s accent influences the credibility of what he or she says.”
2. 2) Choose how to counterbalance the conditions in the following situations:
   1. a) An experiment studying the influence of sentence complexity on the comprehension of anaphora in children.
   2. b) An experiment studying the influence of a concurrent task in working memory (participants must remember strings of letters in parallel with the reading task) on the construction of predictive inferences while reading.
3. 3) An experiment aims to study the influence of grammatical gender on the representation of common nouns. Based on Borodistky et al. (2003; see section 2.5), you ask French-speaking participants to choose associations of common nouns and first names, which can either be of the same gender or a different one.
   1. a) How do you choose common nouns and first names? What are the variables to be controlled?
   2. b) Choose the words to create a dozen pairs in French.
   3. c) Create lists to implement a repeated-measures design. Every item should be presented in the different conditions without the participants seeing the same item several times.
4. 4) Write the instructions for a self-paced reading experiment comprising of 40 items. Every item corresponds to a 5-sentence short story describing a situation in everyday life, and whose fourth sentence is the target sentence. Stories are sometimes followed by questions.
5. 5) Write the free and informed consent form for that same experiment.

6.9.2. Answer key

1. 1) There are different ways of approaching this question. Here, we will follow the assumption that people give less credibility to statements made by a person speaking with a foreign accent than by a person without an accent. To study this question, we suggest recording statements made by speakers with or without a foreign accent, to present to participants and ask them to assess, on a scale from 1 to 10, the truthfulness of such statements. By comparing the scores obtained in the different conditions, it would be possible to determine a connection, if existent, between foreign accent and credibility. Let us imagine that we will test native American-English speakers.

For this study, we can identify different variables to control. First of all, at the item level, the veracity of the information to be evaluated should be checked. It would be appropriate to present true and false statements, so that participants can respond to a full range of credibility. Secondly, the structure and complexity of the items should be kept constant, so that these parameters do not influence the truthfulness judgment. Thirdly, it might be necessary to present as many statements as possible, that are equally uttered by male and female speakers, in order to control a potential influence of the speaker’s gender on credibility. It is possible that the participants consider men more credible than women, due to the existence of certain stereotypes in society. The last very important aspect to check would be the speaker’s accent. It would be appropriate to vary the accents in order to generalize the results.

As far as participants are concerned, the main variable that could interfere with the variables being investigated in the study, relates to their general attitude towards people with a strong foreign accent. This is undoubtedly influenced by their general attitude towards foreigners, and it might be useful to measure this variable in order to take it into account when analyzing the results. The attitude towards speakers with foreign accents may also be influenced by a subject speaking one or more languages, as bilingual people probably tend to be more tolerant of a foreign accent than monolingual people. Finally, the level of mistrust in relation to a statement certainly varies from one person to another, and this would also be a variable to be kept under control.

The best solution for this experiment would be to build a within-subject design, that is, a design where every participant sees all the conditions of the experiment, and a within-item design, in other words, a design where every statement is presented either with or without a foreign accent.

The operational hypothesis, as well as the variables to be checked, can be schematized as follows:

1. 2) a) The conditions of this study are related to the complexity of the sentences presented. This means that children see more complex sentences and less complex sentences. In order to counterbalance the conditions, the best solution would be to present the sentences in the two conditions randomly, without separating the conditions themselves. Every child could see one or more complex sentences, before seeing one or more less complex ones, and then see complex sentences again, and so on.
   1. b) In this experiment, the conditions relate to the manipulation of the working memory. This implies that, for half of the sentences in the experiment, participants only have to read them, whereas for the other half of the sentences, a working memory task has to be performed in parallel with the reading task. In this case, the most appropriate technique would be to separate the experiment into two blocks, depending on the working memory condition. The order of presentation of the blocks would then have to be alternated among participants. Furthermore, the different sentences should be presented in one block or the other, and the presentation order of the sentences should be random.
2. 3) a) In order to answer this question, it is necessary to identify which variables can influence the memorization of a pair, consisting of a common noun and a first name. One possible variable is whether the common noun represents a living being, or not. The pair COW-AGATHE seems intuitively easier to remember than the pair SPOON-AGATHE. It would therefore be appropriate to decide to only test pairs with inanimate objects. A second variable that can influence the memorization of pairs is the frequency or length of the different words included in the task. For French common nouns, we can consult Lexique to find out their frequency and their length. Concerning first names, there are statistics provided by the national statistical institutes, on the classification of first names during the last decades. These may help to control to what extent participants are familiar with such names (depending on their age group).
   1. b) The pairs should be made up of male and female common nouns and male and female first names. For the example, we searched on Lexique for common nouns with a length oscillating between three and four syllables, and a frequency between 10 and 100 appearances per million (in books and movies). We chose the following nouns: ambulance, batterie, caméra, cigarette, pharmacie, télévision, ascenseur, canapé, escalier, hélicoptère, magazine and pantalon. In order to choose the appropriate first names, we consulted the classification of the most widely spread first names in France for the period 1995-2000, on the National Institute of Statistics and Economic Studies²⁶ website. We retained the first names Manon, Camille, Pauline, Marie, Chloé, Sarah, Thomas, Clément, Maxime, Lucas, Quentin and Julien.
   2. c) In order for each item to be presented in each condition, it should appear once associated with a male name, and once with a female name. It is therefore necessary to create two lists, which should each present half of the male and female common nouns associated with a male first name, and the other half with a female first name. One possibility would correspond to the following lists:

List 1:

ambulance-Thomas, batterie-Maxime, caméra-Julien, cigarette-Manon, pharmacie-Camille, télévision-Chloé, ascenseur-Clément, canapé-Lucas, escalier-Quentin, hélicoptère-Pauline, magazine-Sarah, pantalon-Marie.

List 2:

ambulance-Manon, batterie-Camille, caméra-Chloé, cigarette-Thomas, pharmacie-Maxime, télévision-Julien, ascenseur-Pauline, canapé-Sarah, escalier-Marie, hélicoptère-Clément, magazine-Lucas, pantalon-Quentin.

1. 4) In this experiment, you will read short stories that are five sentences long, describing situations in everyday life. The goal is to read these stories in a natural way, as you would have if you were at home. Before each story, you will see the message “Ready to continue?”. When you are ready, press the YES button to bring up the first sentence in the story. Read the sentence, then press YES to go to the next sentence and so on until the end of the story. Some stories will be followed by simple questions about the story. If such a question appears, you must answer the question as quickly and as accurately as possible, by pressing either YES or NO. It is very important to read each story without stopping. If you want to take a break during the experiment, it is possible to do so when you see the message “Ready to continue?”. Please keep your fingers on the YES and NO keys during the whole experiment, so that you can easily progress through the stories and answer the questions. The experiment will start with some training stories. If you have questions, you can ask the experimenter. The experiment will last between 20 and 30 minutes.
2. 5) In this study, we are interested in the process of reading comprehension. Please read the explanation of the experiment you are going to take part in, as well as the risks and benefits it may present, before deciding to participate.

If you agree to take part in this study, you will complete a reading task presented on a computer screen. It will take between 20 and 30 minutes.

You will not get any direct benefit from this experiment, but it will allow us to improve our knowledge about the comprehension processes at work while reading texts. As compensation, you will receive 10 Euros.

There is no direct risk associated with your participation in this experiment, except that of feeling bored. Participation involves an investment of 20–30 minutes of your time.

You are free to accept or refuse to take part in the study. You can now choose not to take part in it. If you choose to participate, you can still withdraw from the study at any time, without any need for justification. If you take part in the study and decide to withdraw from it following your participation, you can ask for your data to be deleted. In all cases, the 10 Euros compensation will be given to you.

All the data obtained during the experiment will be treated in strict confidence. You will only be identified by a randomly assigned number, and neither your name nor any means of identification will appear anywhere. No data identifying you will be used in the publications or presentations which result from this study.

At any time, you may ask questions or request further details from Ms. X, Address, Phone No.