So far, we have concentrated on argument mining from written texts, considering the linguistic and operational dimensions of this recent research area. This is the main aim of this book. However, argumentation is an everyday life process that occurs in many situations where the textual dimension is just one dimension, probably the most important one since it conveys most of the semantic content. The non-verbal dimensions frequently play a crucial role, which may support, contradict or introduce a new perspective on the argumentation being developed. Media analysis shows that for some areas such as debates, the non-verbal aspects of argumentation may be as important as the verbal ones.

In this chapter, we show a number of non-verbal dimensions that contribute to the construction and the evaluation of an argumentation. These should be taken into account in argument mining even if they cannot in general be mined in a way similar to textual data. The sound and visual dimensions of our everyday life are considered under different angles and their contribution to argumentation is underlined and illustrated. The interactions of these non-verbal dimensions to argumentation may be quite complex and the evaluation of their impact may be quite subjective. Nevertheless, such an analysis remains crucial for a correct evaluation of an argumentation in its global context.

This chapter is an introduction to non-verbal argumentation. It is not a comprehensive synthesis and does not offer any theoretical development or any solution to this problem, which is hardly emerging.

9.1. The text and its additions

Let us first consider the dimensions that are the closest to written texts, i.e. the various pictures and icons texts may include in a number of genres.

9.1.1. Text, pictures and icons

Argumentation may be found in standard texts such as news, political or business analysis, dissertations or juridical deliberations and decisions. These standard texts are in general purely written texts with no non-textual additions. Documents follow well-defined structures and the language is well mastered.

Argumentation can also be found in for example, do-it-yourself documents where users are guided and warned against errors. Argumentation plays a central role in blogs and forums which are forms of written debates using a language close to the oral one. In these latter classes of documents, the purely textual part is frequently associated with pictures, diagrams or icons which complement the text or underline a certain feature. They have in general a moderate impact on the text content. Pictures and diagrams convey meaning which is often more direct to access than long and verbose texts. Icons have a more direct and simple meaning, with a widely accepted underlying meaning.

9.1.2. Transcriptions of oral debates

Argument mining in debates, mediation, deliberations and other oral forms of this type is in general carried out from transcripts. These transcripts are frequently poor in punctuation and in forms which are typical of oral communication such as hesitation, laugh and even repetitions, and, obviously, they do not include any form of intonation, prosody, or speed of speech. It is however of much importance to consider these dimensions since, for example:

• – oral stress on a particular word or group of words indicates its importance and may change the strength of the support or attack being developed;
• – taking into account intonation may indicate that a statement is a question and not an affirmation, in case there is no explicit linguistic mark such as an interrogative pronoun;
• – various forms of hesitations may indicate doubt or uncertainty and can weaken the strength of the statement being developed;
• – an increase in the speech speed may indicate that this is a point that needs not be discussed, which is commonly admitted, or, conversely that the utterer does not want to see debated;

• – a decrease in the speech speed on a text portion may indicate either a hesitation or a form of insistence, with the focus shifted on that text portion;
• – similarly, the intensity of the sound produced by the speaker, e.g. speaking louder, may indicate a form of insistence, among a few other possibilities;
• – laughter and other such forms are in general quite ambiguous and depend on what is uttered and who is speaking.

These parameters could be indicated in textual transcriptions. For each of them, some precise attributes would have to be specified with a granularity that is sufficient. This will necessarily increase the text transcription workload, but the quality of the resulting analysis would be much higher [LEM 12].

Several other vocal features, which are frequent and crucial in debates, are more difficult to capture and their characterization is more subjective such as authority, irony, teasing or disdain. It is also frequent that speakers interrupt each other or speak in parallel to stress their point of view or to show their lack of interest in the other party’s perspectives. As a consequence, at the moment, oral transcriptions remain relatively imperfect and partial. Argumentation analysis carried out from these transcriptions should therefore be made with care.

9.2. Argumentation and visual aspects

Non-verbal behaviors, including gazes, facial expressions, gestures and body postures [RIC 08], influence the way a speaker and what is said are perceived. These non-verbal elements of communication, and by extension of argumentation, play a role in dialogues and other types of debates.

Debates frequently take place on TV or on similar media. The contents of the debate and who is talking is clearly central. However, the way the debate is presented to the public and the way speakers and moderators behave may deeply alter several of its features, among which the strength of the arguments, the persuasion effects and elements such as ad hominem or expertise considerations. For example, in a TV debate, camera shots and camera angles have an implicit meaning that may alter what the current speaker is saying. Suppose that while speaker A is speaking, the camera is pointing to A’s opponent B or at the audience that is laughing or yawning, then what A is saying may be judged to be not as credible as it should have been if the camera had just pointed to him. Speaker A may be supposed to be incompetent or not serious. The same camera can also point to the public, capturing the facial expression of some skeptical listeners. Annotation schemes exist to analyze non-verbal cues, such as the MUMIN coding scheme [ALL 07]. Among other things, the MUMIN framework allows the analysis of who is looking at whom during a conversation, and it could be also used to describe a camera movement.

Besides what the camera shows, the environment of the debate is important: the lighting, the visual perspective, the type of seats, etc. These elements are part of the visual rhetorical dimensions of argumentation. The credibility of the speaker is reinforced by the way he speaks (e.g. clearly, with no hesitation, in a way that is understandable by most listeners), the way he behaves (making gestures or not), the way he is dressed, how and when he looks at the camera or at his opponent, etc. These features are well known to politicians, journalists or businessmen who wish to convince others of their position, perspective, seriousness and commitment.

The background of the room is also important: the type of light, its intensity, the colors of the seats, tables and other furniture, the overall organization of the room, the distance or proximity with the audience, etc. These elements contribute to establishing an atmosphere that may inspire trust in the speaker’s ideas or, conversely, suspicion or worry. These elements have been investigated in detail by specialists of communication. They have an intrinsic value, but also depend on the listeners, their level of education, their expectations and their judgment and perception of the speaker.

Besides media, other visual elements are crucial in argumentation. For example, when arguing for a product, its packaging, the colors that are used and even the shape of the package are all important, in particular for food products. These simple visual elements argue much more efficiently and directly than any discourse based on the attractive properties of a product, even if they lead to incorrect evaluations of this product.

Similarly, impressive buildings like justice courts or presidential palaces strongly suggest that the activities realized in these buildings and the competence of its occupants is in harmony with the impressive character of the buildings. A court of justice set in an old, small and ugly building will not have the same effect even if the quality of the judgments which are made are the same as if they were done in a more impressive building.

There are many other visual elements that contribute to an argumentation in our everyday life: company logos, building entrance designs, etc. All these visual elements, either pictures or videos can be annotated, however their interpretation in an argumentation model is complex and contextual.

9.3. Argumentation and sound aspects

Sounds taken in isolation as well as music extracts offer a major contribution to non-verbal argumentation. Sounds are used, for example, to establish a transition between radio programs or to identify them (jingles). They can be artificial sounds or concrete sounds such as clocks or whistles. Music is used, for example, in shops, films, sport competitions or in movie trailers. Music used for argumentation purposes is often a short extract of a longer musical work, an extract that includes significant elements such as a melody or a specific rhythm. It can also be a dedicated short piece of music composed for that purpose. This is a frequent case in advertising. Music was also, via non-verbal rhetoric, a powerful means of argumentation, in particular during the Baroque period. Similarly to visual aspects, music has a large diversity of applications in argumentation. Some of the features of music in argumentation and rhetoric are underlined as illustrations to show the wide range of roles music may play.

Music does not convey meaning as language does, it however conveys emotional and symbolic contents that are very important in rhetoric and argumentation. These aspects are often paired with the rational part of arguments in a number of areas and applications. Since music is a more elaborated system than isolated sounds, we focus on music viewed as an organized system of sounds.

9.3.1. Music and rationality

Since the Greek period, a number of authors have investigated the structure of music from a rational and scientific point of view. Till the Renaissance, music was part of the Quadrivium together with geometry, arithmetics and astronomy. The other three “liberal” arts, the Trivium, included grammar, rhetorics and dialectics. Music was the closest discipline to the Trivium. Saint Augustine (354–430, in the Confessions and De Musica) and Boece (470–525, in the Consolations) show that music is a science, via the development of a rational analysis of music based on numbers and proportions, supposed to manage the harmony of movements, including movements of planets. At that period, music was considered not only as a mathematical object describing the structure of melodies and rhythms, with a strong explicative power, but also as a form of abstraction reflecting creativity and perfection. These considerations give an idea of the impact music can have when associated with an argumentative text.

The above elements on music and cognition show that investigating the role of musical activities in rhetoric and argumentation raises several difficulties:

• – First, domains in which music operates are not totally clear: psychological, perceptual or emotional. How, then, do these domains interact with textual argumentation?
• – A large proportion of musical knowledge that is part of our culture is non-verbal. How is it possible to describe the concrete impact of our musical and sound cultural heritage?
• – Next, it is not clear whether musical models can be given a Tarsky-style semantics, independently of any human usage and perception. How is it possible to account for such an abstract and culture-dependent system?
• – Finally, music is a multidimensional system that requires several layers of knowledge description and abstraction. Each layer is coherent and reflects a different perspective and system. It is not clear whether these systems elaborate on each other and are consistent or if they are inconsistent or even just unrelated.

9.3.2. Main features of musical structure: musical knowledge representation

Music has an organization that is quite different from natural language. The four main levels are, informally:

1. 1) the melody level that accounts for the structure of the “horizontal” dimension of music;
2. 2) the harmony level, which develops the “vertical” level of music in close connection with the melody, by specifying the structure and the sequencing of chords;
3. 3) the polyphony level, which develops the organization of layers of melodies realized by different voices or instruments. Polyphony, which is typical of western music, must observe the rules of harmony and melody construction;
4. 4) the form level, comparable to the discourse or document levels in language, which specifies various musical organizations (e.g. fugue, sonata, scherzo) at a more global level.

Music has many other components that are not discussed here such as timbre and instrumentation, meter and rhythm, dynamics and accentuation, phrasing and articulations, and, in more contemporary music, note clusters, groupings, series, etc. Each of these levels plays an important role in rhetoric and argumentation.

A number of musical elements have been developed through the centuries and have acquired a strong expressive and symbolic power in our western culture. These became major figures of sound that were easily recognized by the audience of the past centuries; they are still used today in various types of music with approximately the same underlying “meaning”. These forms are based on principles or expressive constants that borrow from different perceptual considerations and language metaphors.

The main music parameters that must be considered to develop a rhetoric of music and music affect are [SAI 14]:

• – modes and tonalities, claimed to have specific colors and to introduce specific moods. For example: the tonality C major is happy, with some idea of fight, pure and innocent, naïve and gentle, in contrast: C minor is sad, grave and serious;
• – melodic intervals and profiles of musical motives, in particular in relation with spatial metaphors, also claimed to suggest specific feelings and moods. For example, minor third (1) ascending: sad and painful, (2) descending: calm; major third (1) ascending: happy, joyful, (2) descending: sad;

• – melodic development, variations and alternations and musical figures. They are complex systems that must be taken into account. For example, ascending melodic movements (Anabasis, Ascencio) are frequently realized by means of adjacent notes or small intervals, this figure has a positive orientation similarly to orientation metaphors in language where going up is positive;
• – rhythm in musical motive, meter, tempo and tempo variations and specific forms of rhythm. These are parameters that can be associated with an immediate perception. Rhythms may be figurative: rhythmic hesitations where a regular rhythm includes gaps with missing notes or a rhythm that progressively disintegrates suggests uncertainty, doubts or anger. Syncopation may also provoke the same kind of feeling because of the instability it suggests with respect to the meter;
• – nuances, accentuation and articulation indications, which are as important as in spoken discourse. For example, a loud sound will have a different effect than a very soft one;
• – harmony, expressiveness of chords, sequences of chords: these have a role quite similar to tonalities illustrated above;
• – timbre, instrumentation and registration for organ: timbre is having a more and more important role in our contemporary civilization. It is formally defined by a weighted combination of harmonics from the fundamental sound. These combinations introduce colors that have an effect on the listener;
• – symbolic aspects of forms, proportions in musical motives and numerology. Although these features are more difficult to perceive, they nevertheless have an impact on the listener. For example, a well-balanced melody may entail confidence, whereas a more chaotic structure may entail suspicion or discomfort.

A number of these parameters have received annotation guidelines and norms in music analysis and in information retrieval. It is clear that these parameters largely interact and the moods, affects or feelings produced by a parameter can be further refined, transformed or modified by other parameters. Their association with text is an active research topic. Most of these are still preliminary and under testing. An overview of a number of projects can be found at http://recherche.ircam.fr/equipes/analyse-synthese/peeters/main_techno.php.

9.4. Impact of non-verbal aspects on argument strength and on argument schemes

We have seen in the previous sections, via a number of examples, that non-verbal aspects can have a major influence on argument strength. A sound, music extract or a visual element may reinforce a support or an attack if it has the same tone: positive for supports, negative for attacks. The opposite can also be observed where, for example, a visual aspect contradicts in some way a support or an attack. In that case, these latter are weakened.

It is difficult to precisely evaluate the impact in terms of strength of non-verbal elements since they are objects that are very different from linguistic objects. They have a much more contextual or personal dimension than language whose objects, words and constructions, are more normalized in terms of semantic and pragmatic impact.

Non-verbal elements also have an influence on argument schemes. For example, an argument scheme from expert opinion can be affected if in a video the expert A is arguing with opponents or an audience which is laughing or yawning. In that case, a shift could be observed to an argument scheme based on ad hominem considerations, which deeply affects the validity of the argumentation being carried out.

The opposite situation can also be observed for a speaker a priori judged to be not very competent. If an audience listens to him with a great attention and approves his arguments, then a shift to an argument scheme based on best explanation can be foreseen.

9.5. Ethical aspects

Ethical aspects related to non-verbal aspects of argumentation are important. While the “textual” part of argumentation essentially relies on language and meaning, which is relatively objective, the non-verbal aspects cannot in general be measured easily and introduce a large part of subjectivity and possible manipulations of an audience.

As illustrated above, a serious argument can be turned into a fallacious argument just by using non-verbal features. Conversely, fallacious arguments may seem, due to a good non-verbal “packaging”, to be very well founded and perfectly acceptable. These situations may occur in a large diversity of situations, including those where business, political decisions and advertising are involved. These are in fact part of any human behavior and can seldom be avoided. To circumvent these problems, the rules for a cooperative form of argumentation, as those given in [EEM 01], seem a good approach among a few others. Besides these cooperative rules, research is ongoing in psychological circles on the impact of visual and sound elements on human behavior, which are central to the concerns advocated here.