Pragmatic Labeling Based on Body Language


At present, the study of syntax and semantics has yielded certain results in the field of computational linguistics, while pragmatics has not been paid as much attention. The lack of paralinguistic research, the research of body language, is obvious. Among numerous definitions of body language we will use two in this paper: firstly, gestures, postures and face expressions accompanying speech or occurring independently, and secondly, it is a language used by deaf and mute people, or sign language.

Body language, in its first meaning plays an auxiliary role in language understanding but is also of vital importance for disambiguation. Body language in its second meaning is the only communication tool for mute and deaf people. In order to help them to become full members of the society and to remove communication barriers, scholars should pay more attention to the research of sign language.

The first step of natural language processing is its formalization. Usually tagging or labeling is meant by formalization. It is not a challenge for text processing anymore, but for body language processing one should deal with video, that is why there has been little research conducted in the field of computational linguistics so far. To date, most of body language research is highly descriptive and lacks formality, thus cannot be used in computational linguistics.

In this paper we are using the methods of componential analysis to perform formal description and tagging of body language. Componential analysis splits the unit of interest (here it is a gesture) into several components. For body language analysis these components include the part of the body involved into gesture production, the place, the direction of the motion, etc. Using this method for body language processing has the following advantages:


  • Objectiveness: personal characteristics or cultural background of the signer cannot influence the analysis and description;
  • Adaptability: the components used are relatively independent and are not correlated, thus if existing components appear to not be enough to describe some gestures, new components can be added and it will not affect the existing ones. Vice verse, if existing components are redundant for the development aim, some of them may be omitted, which will not influence the result (for instance, sign language tagging requires more components comparing to body language, because every movement is very detailed and needs to be described very precisely, for body language tagging it is not needed to use as many components);   
  • Formal aspect is very strong: research results may be presented as a binary tagset which can be used in applications development directly.


  • Universality: for componential analysis we do not need to take the meaning of the gesture into consideration, thus it may be used to process body language and sign language of any country;

This paper is not only in depth research of body language, but it is also a useful theoretical and reference material for further research and development. The results of the work have been used in several state and Tsinghua University projects.


The paper will be posted when possible.