[Link]

html

Visual Communication 94, Feather River Inn, Blairsden, California

VISEMICS: A PROPOSAL FOR A MARRIAGE
BETWEEN SEMIOTICS AND VISUAL COMMUNICATION
In a review of the contributions of various philosophers to semiotics, Thomas Sebeok,
founder and director of the Center for Language and Smiotic Studies at Indiana University
observed that Bacon, for one, "did not commit the vulgar error of identifying language
with communication." (1991, p. 71) But if language is not identifiable with
communication, then what is? It is the primary thesis of this essay that visuals are
equally as important as verbal language in communication, if not more so, and that
Sebeok is right in asserting that privileging verbal language is narrow minded, if not
vulgar.
The five questions driving this investigation focus on the relationship between semiotics
and visual communication, the relative importance of visual and verbal communication
systems, and the place visual communication fits within the larger world of sign systems.
1. What is semiotics and what elements/aspects of it are most useful in analyses of visual
communication?
Semiotics is the science of signs and semiotic analysis is used in studies of sign processes
in such fields as communication, cognition, linguistics, anthropology, marketing,
medicine and cellular biology. The contemporary focus that most concerns us at this
conference is on the communication aspects of semiotics, particularly the elements
relevant to visual communication.
The communication focus is reflected in the definition of semiotics given by Jakobson:
"the exchange of any messages whatever and of the system of signs which underlie
them." (Sebeok, 1991, p. 60) Fiske adds another dimension to this definition--the
generation of meaning. (1990, p. 42) In other words, messages are made of signs and
conveyed through sign systems called codes; the more we share the same codes in a
communication exchange, the closer our meanings will be. Presumably that is true for
visual as well as verbal communication and a comparison of those two communication
repertoires will be an important thread throughout this paper.
But what is a sign? A sign is anything that stands for something else --i.e. a sign stands
for an object or concept. (Hopes, 1991, p. 141; Eco, 1986, p. 15) The "stands for" process
is the point where meaning is created both through encoding (by the source) and
decoding (by the receiver--or "reader" in semiotic analysis) as in these stylized
representations of people. The Swiss linguist Ferdinand de Saussure developed a similar
construct which he described as a sound or image-- called a signifier--and the concept
for which it stands--called the signified. (Fiske, 1990, p. 43-44) Because of his
background as a linguist, Saussure saw the relationship between these two as arbitrary,
in other words, the link between the sign, or expression, and what it stands for is
understood by convention. This arbitrariness is true in most spoken and written
language--such as the names for these individuals depicted in these sketches--however,
that may not be so for other types of signs such as visuals that provide stylized cues to
stimulate recognition through resemblance.
The American C. S. Peirce, who is characterized as the founder of American semiotics,
disagrees with Saussure on the arbitrariness condition. Because Peirce is first of all a
cognitive philosopher, he defines signs in a broader way than language and focuses on
how they are logically or semiotically linked to their objects. Peirce's foundational
contributions to semiotics have shaped the field, as Berger explains, to largely focus on
meaning and modes of cognition (the codes needed to understand a text). (1982, p. 39.)
To Peirce signs are of three types--icons, indexes, and symbols. (Hoopes, 1991, p. 239)
Symbols are arbitrary, but icons and indexes are, in semiotician language, "motivated,"
that is they are more likely to resemble their object in some way, rather than being
arbitrary. Peirce defines an icon as similar to its subject--a representation such as a
drawing or photograph where likeness or resemblance is a determining characteristic.
An index is physically connected with its object--as an indication that something exists
or has occurred, such as a footprint that means someone just walked by or smoke that
means there is a fire. A symbol such as a leaf on a flag is arbitrarily linked with its
object--something that has acquired meaning through convention. Symbols, like most
spoken and written words, are arbitrary.
Peirce's categorization of signs provides a richer context for understanding visuals and
how they convey meaning than Saussure's signified and signifier--although Saussure's
dyadic relationship is included in Peirce's categories and it is at the base of all semiotic
analysis.
Another important element, particuarly in a comparison of visual and verbal
communication, is the notion of code or sign system. A code functions as a system of
rules. Language is a code and linguistic competence, i.e. understanding the elements and
structure of language, is foremost on the agenda of most modern schools. The concept of
semiotic code, however, is broader than just language and includes such sign systems as
kinesics or body language, braille, sign language, and algebra and chemistry. In this
advertisement, for example, the complex message about gender and religion is being
communicated through a manipulation of color codes. Nature can even be seen as a
system of coded signs and Eco makes the argument that the roots of semiotic
interpretation lie far back in time with hunters and trackers who could read the signs of
nature.
Visual communication has various types of codes that help determine the message. The
argument can be made that visual communication operates, to some degree, similar to a
language sign system in that the basic characteristics of language are similar--elements,
syntax, and grammar. For example, the visuals used in graphic design, photography,
video and film contain discrete sign elements (shots, typography, drawings, etc.) which
are variously called semes or signemes; syntax which structures the links between the
elements (composition in a still form and transitions in film); and rules about their use
which form a type of grammar --visual or aesthetic codes that carry their own meaning.
(i.e. estabishing shots, reaction shots, etc.).
Also important is the notion of the interpretant, another concept which is important to Perice's
semiotics and particularly relevant to an understanding of how visual communication
operates. It was mentioned earlier that the sign/object relationship is the basis of semiotics.
Peirce adds the concept of interpretant to that dyad to create a triadic construct of
sign/object/interpretant. By interpretant he means the idea contained in the concept as it is
decoded or a subsequent thought to which the sign gives rise. (Hoopes, 1991, p. 34) He
explains that a thought is a sign requiring interpretation by a subsequent thought in order to
achieve meaning. Our memories, which provide the foundation for the meaning process, are,
as Sebeok puts it, "reservoirs of interpretants." (1991, p. 131)
For example the word /dog/ and a picture of the animal both stand for some concept of
"dogness." In addition, there are interpretations imposed on this concept of horseness based
on our personal experiences and also on additional information and description that
accompanies the sign. Other verbal interpretants for the word /dog/ could be puppy, bitch,
hound, or faithful companion-- as well as equus, cheval...or even frankfurter or ugly woman.
Each further interepretation tends to elaborate on the concept and make it richer. (Sebeok,
1991, p. 18-10)
This review of the basic concepts of semiotics should make it clear that many of these
concepts are linked inextricably to visual communication. Even though some semiotics
studies are primarily focused on literature and language, and, as Deely notes language
has a prileged role in semiotics (1990, p. 27), it is inconceivable to try to explain
semiotics without referring to visual communication factors and functions and it makes
a lot of sense to use semiotics to explain how meaning is produced visually.
Finally to conclude this discussion of the basics of semiotics, it should be noted that what we
are talking about is an action process. As Deely explained in his book on semiotics, Peirce's
primary contribution was the full development of semiotics as a distinct body of knowledge
and this discipline rests on the notion of "a dynamic view of signification as a process."
(1990, p. 23)
2. Which is primary--verbal or visual communication?
It is important to realize that the semiotic process, in Peirce's view, was more centrally
concerned with thinking than with communicating. Peirce was a student of philosophy of
knowledge and he was most concerned with an analysis of thought rather than language.
Inference Peirce's primary contribution to cognition--and only incidentally to
communication--was his proposition that "all thinking is the inferential interpretation of
signs" (Hoopes, 1991, p. 11). In other words, a thought is a sign interpretation, an idea which
provides the link between cognition and communication. He explained that the meaning
process--finding the signified--is an infinite process of interpretation and, furthermore,
that to interpret means to define a relationship, i.e. what is involved in the something-stands-
for-something-else construct. (Eco, 1986, p. 2, 44). So we have at the base of Peircian
semiotics a thinking process based on inference which results in interpretation.
The reason Peirce's concept of interpretant is so important to us in visual communication is
that visual meaning is more open for interpretation than verbal. Contrary to
conventional wisdom and the views of some leading semioticians and most linguists, the
interpretation of a visual message may be more complex and more demanding of the
decoder because of the inferential dimension than is the interpretation of a verbal
message. The complexity of this type of inferential interpretation is illustrated in this use
of an upscale car to stand for the flaunting of achievement. In order to make sense of
this ad, you have to recognize the car and the fact that it stands for accomplishment, the
school and the fact that it is cueing a class reunion, and all the emotional context
associated with the reunion situation.
The "Primary" Debate Peirce, and particularly Sebeok, present us with a theory of
semiotics that is not only broader than a language-based sign system, it also treats both
nonverbal and visual communication with the respect due to these seminal
communication systems.
Saussure, however, is a linguist and although he admits that signs can be other than
words, his work privileges language as the most important sign system. Likewise, the
Soviet school of semiotics (Ivanov, Lotman, and Toporov among others) calls language a
"primary modeling system," because it uses natural language as its base, and all other
sign systems as "secondary." Natural language, in other words, is the primary
substructure for all other sign systems. (Sebeok, 1991, p. 50)
Other semioticians disagree and point to development learning as a better source for the
primary designation: as infants we develop visual skills before language skills; in this
sense visual communication should be considered primary rather than secondary.
Sebeok also disagrees with the language-as-primary theory and points out that language
evolved first as an adaptive function principally to enhance imitative signaling--the
evolutionary focus was more on language-as-modeling-system than on speech-as-
communication. Furthermore, at the time language evolved, homo habilis, the earliest
known species, had very sophisticated visual and nonverbal sign system repertoires but
a primitive verbal system limited to gutturals rather than articulate, linear speech.
Sebeok argues that "properly speaking, language itself is a secondary modeling system."
He is comparing language based sign systems with other visual and nonverbal systems
which are equally as complex, as well as antecedent to language. He concludes that "the
general belief that language replaced the cruder systems is totally wrong." (Sebeok,
1991, p. 57, 71)
I would like to suggest that visual interpretation is more complex than verbal
interpretation and therefore more appropriately designated as a primary sign system.
The reason is because the learner has to manage more of the interpretative function with
visuals than with language because of their different learning protocols. Verbal signs are
arbitrary and verbal messages are highly conventionalized, so we are taught to speak
and read these signs through an extensive educational program. Visual signs are more
often iconographic or indexical and those forms of interpretation are, as Sebeok noted in
a discussion of nonverbal communication, largely "wired in" rather than arbitrary and
conventional. (1991, p. 65) Even visual signs that are arbitrary, such as this plateful of
symbols, are not formally taught to us.
We acquire visual competence through development and experience rather than
training--in other words, for our basic nonverbal and visual communication skills we
are largely self-taught and that includes such skills as making sense of MTV, watching
multiple channels simultaneously, and negotiating freeways as we puzzle out a map.
Even though we may be self taught, our skill levels in certain areas where development
is important can be extremely high.
However, we have lost visual skills, too. Bill McKibbon in his book The Age of Missing
Information makes the point that our ancestors were much more tuned in to the land, to
weather, to animal behavior, and to nature than we are. Their additional sophistication in
understanding natural signs was largely a product of experience--with some tutoring from
wiser adults. (McKibbon, 1992)
Today we learn a basic survival set of visual and nonverbal skills with only minimal help
from mom and dad--we learn more on our own than we do with language whose initial
learning is directed by parents and later by schools. We do need training if we want to
develop more complex or specialized visual skills in order to track something in the
woods, to puzzle out the images of the cells we see in a microscope, to read a negative or
X-ray, or to understand a Picasso painting.
Furthermore the lack of a system for assigning conventional meanings for visual
communication suggests that visual information is subject to more active personal
interpretation--more so than with language. The interpretation of visual information,
like semiotic approaches to meaning interpretation, are highly subjective and highly
projective, which puts more demands on the receiver. Because of the resemblance factor
for icon interpretation and the experience factor for index interpretation, the formal
training needed may be less than for language--although the life experience may be
more demanding-- but regardless, the visual and nonverbal systems operate relatively
untutored in our society, at least in comparison to language. With visuals we are much
more on our own, both in learning and in interpreting, and that's why I believe visual
learning in our contemporary society is equally as challenging an accomplishment as
verbal learning.
3. Why is semiotics, with its roots in philosophy and linguistics, particularly useful in
analyzing visual communication?
Peirce's statement quoted earlier that semiotics is the study of an "inferential interpretation
process" introduces another concept that is important in understanding how visual
communication works and that is inference. Visual and verbal comunication differ in their
interpretive logic. Most language-based models of logic focus on induction and deduction as
platforms for reasoning. Visual thinking, however, is more easily analyzed by abduction, a
process which Peirce referred to as hypothesis building and Eco and Sebeok refer to as "the
conjectural paradigm." Deduction and induction are syllogistic; abduction reasons from what
Hoopes calls "statistical inference" which relies more on hypothesis testing or educated
guessing than rules of logic. (1991, p. 85)
Abduction operates largely from a nonverbal platform although words can be and are often
used in support. In medicine, for example, doctors use words to probe for subjective symptons
but rely on their own senses to probe for objective signs which are compared with the doctor's
knowledge of past cases. Medical diagnosis is a good example of abductive reasoning
supported by induction.
Nowhere is the power of abductive thinking made clearer than in Eco and Sebeok's book, The
Sign of Three, which compares Peirce's approach to thinking with the reasoning of two great
detectives Conan Doyle's Sherlock Holmes and Poe's Dupin (1983). A detective, of course,
detects things or discovers their existence; an inspector is someone who inspects, or looks.
The reasoning approach used by dectectives in crime fiction is very similar to the interpretive
process of abduction which has been described as "educated guessing toward a hypothesis."
In both cases the process begins with observations and then proceeds in a back-and-forth
process of developing hypotheses and comparing the observations with information known
and filed in memory. Whether it be solving a crime or tracking an animal through the forest, it
is the same sign interpretation process described by Peirce and it begins with observation--
observations which are usually visual and often indexical.
And this type of search for meaning is infinite, in Peirce's view, because every signifier can be
translated into other signifiers and interpretants through an endless process of inference
chaining.
Now ask yourself, how does a semiotician develop an expert reading of something? Some
critics look upon this process as something more closely aligned to tea leaf reading because
they don't understand how abduction works. Here's an example of visual semiotics: Ask
yourself how it works. What are its visemes? How does the visual syntax structure your
process of creating meaning? What do the visual codes or grammar contribute to your
interpretation? And how did the chaining of interpretants work?
Now analyze your interpretation process in making sense of visuals such as this. Did you use
syllogisms or observation testing? If Eco and Sebeok are right, then you will observe different
cues which bring forth information from your own mental filing cabinet and you will develop
hypotheses about various aspects of the meaning until the picture comes together. In other
words, the interpretative method of semiotics based on its use of abduction, is more closely
aligned with the way we process information visually which, to answer the second question, is
why semiotics is such a useful tool in visual communication research and theory building.
4. Where does viscomm fit in the panoply of communication sign systems?
One objective of this paper is to defend the notion that visual communication is a
primary sign system and it deserves the respect of scholars and philosophers. As part of
that argument, it might be helpful to sort out the various categories of sign systems and
locate visual communication relative to other systems.
The first premise of this model, and it is a heretical idea--at least to linguists--is that
human language is only one of many sign systems and it is certainly not "primary." In
other words, there is a large world of communication sciences disciplines and language-
based sign systems are only a part of that universe. As indicated in a quote from Sebeok,
we shouldn't think of communication as based solely or primarily on language: "Our
habit of thinking of communication as consisting exclusively of language has delayed the
study of communication." (Sebeok, 1991, p. 57). So we will begin with a model of
communication sign systems that does not begin with language.
Several writers have surveyed the scope of semiotic studies and this analysis is based on Eco's
review in A Theory of Semiotics (1979) and Sebeok's discussion in A Sign is Just a Sign
(1991). At the first level are a number of different types of semiotic systems that Sebeok
has identified as functioning in a communicative way. The variety of the communication
systems that are not language based should illustrate convincingly Sebeok's point about
the "vulgar error."
For example, protosemiotics has recently been suggested as a term to use in describing
the sign systems discussed in analyzing the solar system. Zoosemiosis includes animal
communication--all those puzzling questions we ask about how bats fly, how flocks of
birds turn synchronously, the meaning of dolphin and whale sounds, and the mapping
communicated by the honeybee's dance, as suggested in this visual.
Under the general area of biosemiosis, endosemiotics studies communication at the molecular
level. One might ask how cells can be described as communicating organisms however
molecular and cellular biologists and chemists are involved in puzzling out signaling systems
used both inside a cell as proteins and acids are moved around, and between cells as they
create complex molecular and cellular structures. Likewise, the neural, genetic, and metabolic
codes all operate through semiosis, as well as the processes studied in chemical, thermal,
structural, and electrical engineering. Other biosemiotic areas include phytosemiotics
(vegetable) with its photosynthesis processes and mycosemiotics (fungi).
Anthroposemiosis includes human communication and this, of course, is the area we are most
concerned with. Interestingly, human communication seems far less well developed as a
theoretical field than the scientific areas on the left of the model.
In trying to model human communication there are several points of puzzlement. For
example, is language only verbal or are there other types of sign systems which could be
called languages--visual, kinesics, sign language, braille, etc? The whole nonverbal area
is huge and grossly underestimated in studies of human communication. The problem
with nonverbal communication, however, is a semantic one; there is no way to refer to
this area other than in opposition to language which inherently privileges verbal
communication. Sebeok observes that "the human's rich repertoire of nonverbal
messages--by shared contrast with language--never constituted a unified field of study
and therefore lacks a positive integrative label." In another place he laments that there
is "as yet no universally agreed upon global designation for studies of nonverbal signs."
(Sebeok, 1991, p. 23, 60)
The model shown here is a preliminary attempt at depicting the important divisions in the
world of communication sign systems and locating them relative to one another. It is not
inclusive and, at this point, it is only a starting point for discussion. I welcome your
suggestions.
5. Visemics: Is it time for a discipline of our own?
The model, plus the previous discussion of the ties between semiotics and visual
communication argues for greater respect for visual communication; and this leads to a
discussion of the nature of the visual communication field. Is it time to declare visual
communication a discipline?
What exactly is included in a study of visual communication? One of the biggest reasons
this area gets less respect than language is because it is so loosely articulated. If we build
on a language-based model summarized earlier, then we would include 1. visual
elements or "visemes," 2. syntax (composition, transitions) and 3. grammar (the visual
aesthetic codes). As Berger points out, "The notion that film, television programs and
other cultural phenomena are like languages in many respects is central to semiological
analysis." (Berger, 1984, p. 190) However well developed the linguistic model may be, is
it adequate as an analytical frame for visual communication? If not, what's missing?
Furthermore, is the name right? While visual communication does not have the semantic
limitations that plague the area of nonverbal communication, it still may not be an effective
designator because of the implied parallelism. /Visual communication/ is a common phrase
but the expression /verbal communication/ is less often used--which suggests that the idea of
communication may contain verbal as a root concept, thus there's no need to designate it.
What the phrase /visual communication/ does is separate out visuals as a subset of the broader
language-based field of communication and we've already determined that visual
communication is broader than that.
Perhaps what we need is a name like kinesics or linguistics--a term that summarizes, clarifies,
and claims an area of study. Is there anything parallel to linguistics that carries the semantic
power of ownership--vististics, visive, visivics, visus, or visemics, for example. The last
phrase is my preference because it acknowledges the fundamental importance of semiotics
and the concept of sign systems. Or is such phrase making even necessary?
Conclusion
This paper suggests that semiotics is a very useful methodological tool for analyzing
visual communication because the thinking processes it proposes are parallel to the
interpretive processes used in creating and understanding visuals. Furthermore, it
argues that it is more logical to designate visual communication as a primary system
than language based upon what we know about developmental communication and the
complex demands of visual interpretation. Finally, since the two areas of visual
communication and semiotics are so closely linked, the time might be right for
announcing a new discipline with a name that joins the two--such as visemics.
References
Berger, Arthur Asa. 1982. Media Analysis Techniques. Newbury Park CA: Sage.
Berger, Arthur Asa. 1984. Signs in Contemproary Culture. New York: Longman.
Deely, John. 1990. Basics of Semiotics. Bloomington: Indiana University Press.
Eco, Umberto. 1979. A Theory of Semiotics. Bloomington: Indiana University Press, Midland
Book Edition.
Eco, Umberto. 1986. Semiotics and the Philosophy of Language. Bloomington: Indiana
University Press, Midland Book Edition.
Eco, Umberto and Thomas A Sebeok, ed. 1983. The Sign of Three. Bloomington: Indiana
University Press.
Hoopes, James. Peirce on Signs. 1991. Chapel Hill NC: The Unviersity of North Carolina
Press.
Sebeok, Thomas A. 1991. A Sign is Just a Sign. Bloomington: Indiana University Press.
clickhere to return to Professor Moriarty's home page
click here to return to the visual communication research page