Cinematic techniques in avatar-centric communication during a multi-user online simulation Number:7,386,799 from the United States Patent and Trademark Office (PTO) owispatent

Title: Cinematic techniques in avatar-centric communication during a multi-user online simulation

Abstract: A method and apparatus for avatar-centric communication, expression and display during a multi-user online simulation are described. Various techniques are introduced to increase the richness and realism of avatar interactions, including the use of "Chat Balloons", "Chat Props", cinematic techniques, avatar non-verbal communication, and gaze control.

Patent Number: 7,386,799 Issued on 06/10/2008 to Clanton,   et al.

---

Inventors:	Clanton; Charles H. (San Francisco, CA), Ventrella; Jeffrey J. (San Francisco, CA), Paiz; Fernando J. (Millbrae, CA)
Assignee:	Forterra Systems, Inc. (San Mateo, CA)
Appl. No.:	10/409,745
Filed:	April 8, 2003

---

Related U.S. Patent Documents

---

	Application Number	Filing Date	Patent Number	Issue Date
	60428545	Nov., 2002

---

Current U.S. Class:	715/758 ; 715/753; 715/861
Field of Search:	715/715,753,757,758,706,861,863

---

References Cited [Referenced By]

---

U.S. Patent Documents

5261041	November 1993	Susman
5267154	November 1993	Takeuchi et al.
5347306	September 1994	Nitta
5483630	January 1996	Unuma et al.
5546518	August 1996	Blossom et al.
5577185	November 1996	Tunnell et al.
5736982	April 1998	Suzuki et al.
5742329	April 1998	Masunaga et al.
5880731	March 1999	Liles et al.
5907328	May 1999	Brush, II et al.
6119147	September 2000	Toomey et al.
6154222	November 2000	Haratsch et al.
6285392	September 2001	Satoda et al.
6330022	December 2001	Seligmann
6331853	December 2001	Miyashita
6331861	December 2001	Gever et al.
6346956	February 2002	Matsuda
6359622	March 2002	Hayes-Roth
6396509	May 2002	Cheng
6404438	June 2002	Hatlelid et al.
6466213	October 2002	Bickmore et al.
6466215	October 2002	Matsuda et al.
6483878	November 2002	Yonezawa et al.
6535215	March 2003	DeWitt et al.
6559863	May 2003	Megiddo
6570555	May 2003	Prevost et al.
6753857	June 2004	Matsuura et al.
6772195	August 2004	Hatlelid et al.
6853398	February 2005	Malzbender et al.
7006098	February 2006	Bickmore et al.
7007235	February 2006	Hussein et al.
7116284	October 2006	Kato et al.
7124372	October 2006	Brin
7139767	November 2006	Taylor et al.
7146095	December 2006	Asami
2002/0005865	January 2002	Hayes-Roth
2002/0008716	January 2002	Colburn et al.
2002/0097267	July 2002	Dinan et al.
2002/0140732	October 2002	Tveskov
2002/0175990	November 2002	Martino et al.
2003/0117487	June 2003	Monroe
2003/0151658	August 2003	Smith
2003/0156135	August 2003	Lucarelli
2003/0233650	December 2003	Zaner et al.
2004/0053690	March 2004	Fogel et al.
2004/0100553	May 2004	Allen et al.
2004/0191477	September 2004	Davis et al.
2004/0196360	October 2004	Hillis et al.
2005/0005247	January 2005	Kamachi et al.
2005/0015725	January 2005	Matsuda
2005/0086605	April 2005	Ferrer et al.
2006/0064645	March 2006	Neven et al.
2006/0074689	April 2006	Cossato et al.
2006/0170945	August 2006	Bill
2007/0101276	May 2007	Yuen
2007/0159523	July 2007	Hillis et al.

Other References

AVJ Natural Cat, "Cat95", retrieved on Mar. 11, 2003 from the Internet at <URL:http://web.archive.org/web/*/http://www.naturalcat.com, 5 pages. cited by other . Lauder, J., "To Deceive Is To Enchant: Programmable Animation," Game Developer, May 2000, pp. 15-16, 18 and 20. cited by other.

Primary Examiner: Lo; Weilun
Assistant Examiner: Chuong; Truc T.
Attorney, Agent or Firm: Perkins Coie LLP

---

Parent Case Text

---

This application claims the benefit of U.S. Provisional Patent application No. 60/428,545, filed on Nov. 21, 2002, and entitled, "Avatar-Centric Communication, Expression and Visual Depiction," which is incorporated herein by reference.

---

Claims

---

The invention claimed is:

1. A method of displaying a user-specified conversation involving a plurality avatars in a simulation environment during a simulation in which each of a plurality of users participate, each of the users using a separate one of a plurality of processing systems on a network to control a separate one of the avatars, the method comprising: storing data defining a prop to facilitate the user-specified conversation between the avatars, the prop including a plurality of associated slots, in proximity to each other, at which an avatar can be placed to facilitate the conversation, and a plurality of viewpoints in proximity to the plurality of associated slots and defined relative to the prop, each of the viewpoints to provide a view during the simulation, at least one of the viewpoints to provide a view directed to one of the slots; and placing the prop in the simulation environment during the simulation; placing each of the plurality of avatars in a separate one of the slots in the prop during the simulation; generating a view of a first avatar controlled by a first user from a first viewpoint of the plurality of viewpoints during the simulation; and automatically changing the view from the first viewpoint to a second viewpoint of the plurality of viewpoints, in response to a user-initiated action of a second avatar controlled by a second user, where the second viewpoint differs from the first viewpoint in field of view or distance to subject or both, such as to give the second viewpoint a different zoom from the first viewpoint, to emphasize an element of non-verbal avatar communication.

2. A method comprising: generating a view of a first avatar from one of a plurality of selectable viewpoints, during a simulation in which a plurality of users participate, each of the users using a separate one of a plurality of processing systems on a network to control a separate one of a plurality of avatars, wherein the first avatar is controlled by a first user, and wherein said generating includes causing the first avatar to be displayed to a second user in a first pose, the second user controlling a second avatar in the simulation; detecting user input that specifies a change in viewpoint, from the first user; and automatically changing a pose of the first avatar, as displayed to the second user, in response to the user input from the first user that specifies the change in viewpoint; including causing the first avatar to be displayed to the second user in a second pose in response to the user input, wherein the second pose differs from the first pose in a characteristic including a hand or body movement or gesture of the first avatar excluding the first avatar's head or eye position.

---

Description

---

COPYRIGHT NOTICE

A portion of the disclosure of this patent document contains material which is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure, as it appears in the patent and Trademark Office patent file or records, but otherwise reserves all copyright rights whatsoever.

FIELD OF THE INVENTION

At least one embodiment of the present invention pertains to network-based simulations, and more particularly, to techniques for avatar-centric communication, expression and display during a multi-user online simulation.

BACKGROUND

Computer games and arcade games often feature animated, user-controlled characters which represent human users and which appear human or humanoid. These characters are referred to as "avatars". Currently, there is growing interest in creating an on-line community in which people are represented by avatars and can interact with each other in a virtual world (a simulated environment) through their avatars in a realistic manner. Ideally, the virtual world will provide sufficient "richness" so that the avatars can interact with each other and their virtual environment in much the same way people interact in the real world. The availability of the Internet makes such a virtual world potentially accessible to millions of users. Such a virtual world may impact many areas of everyday life, including communications, entertainment, commerce, and education, to name just a few. The usefulness and success of an avatar-based virtual community will depend largely on the sophistication and realism of the avatars and the ways in which they can interact. Users will want to use and participate in such applications only if their avatars are realistic and sophisticated in their capabilities.

While users of a virtual world may want to engage in various activities, such as racing a car or flying around the world in a plane, one of the most compelling and desired activities is communicating with other users. Thus, one of the principal features common to known three-dimensional (3D) worlds is the ability for different users of the world to communicate with one another, through text chat. In known 3D worlds, conversation has been presented in a way that is no different from online text conversations without a 3D world, using instant messaging and email, where text is presented in a 2D window, separating it from the 3D world. Known 3D virtual worlds do not provide avatars with body language, facial and gestural expression, or 3D symbolic visuals for user-to-user communication. To provide users with a rich user experience in a virtual world, it is desirable to make avatars' faces and bodies important components in user-to-user avatar communication, as in the real world.

BRIEF DESCRIPTION OF THE DRAWINGS

One or more embodiments of the present invention are illustrated by way of example and not limitation in the figures of the accompanying drawings, in which like references indicate similar elements and in which:

FIG. 1 illustrates a network environment in which the techniques can be implemented;

FIG. 2 is a high-level block diagram of a processing system that may represent any of the machines shown in FIG. 1;

FIG. 3 is a block diagram of software on both a client and on a server to implement the avatar-related features and techniques described herein;

FIG. 4 shows an example of a computer screen display showing two avatars engaged in a conversation in a 3D virtual world during a simulation, where Chat Balloons are used to represent the avatars' speech;

FIGS. 5A through 5C illustrate the use of Chat Balloon priority;

FIG. 6 shows the use of chat columns for Chat Balloons;

FIGS. 7A and 7B illustrate the water wheel metaphor for displaying Chat Balloons;

FIGS. 8A and 8B illustrate the word-at-a-time chat feature;

FIG. 9 shows visual connector between two Chat Balloons to identify a reply;

FIG. 10 shows an example of a Chat Prop;

FIG. 11 schematically illustrates the relationship between slots, Chat Props, and the virtual world;

FIGS. 12A and 12B show a Stage chat prop;

FIGS. 13A and 13B illustrate the use of an Auto-Chat Prop;

FIGS. 14A through 14C show an example of the use of multiple cameras in the same chat prop;

FIG. 15 illustrates how a user can select from among cameras in a chat prop by pressing an arrow key on the keyboard;

FIGS. 16A through 16C show an example of the Stage chat prop;

FIGS. 17A through 17C illustrate an example of a Love Seat chat prop;

FIGS. 18A through 18F show a set of camera views of a Spa Chat Prop;

FIG. 19 shows a treatment slot in the Spa Chat Prop;

FIG. 20 illustrates the use of a Cut To camera shot during avatar chat;

FIG. 21 shows the use of vectors to determine the position of an Intimate Camera;

FIG. 22 shows a laugh emotion of an avatar;

FIG. 23 shows a wave emotion;

FIG. 24A shows a bubbling moodicon emotion;

FIG. 24B shows a two-person traveling moodicon emotion;

FIGS. 25 and 26 show two different bubbling moodicon emotions;

FIG. 27 illustrates how the Smiley language allows concurrent display of avatar emotions with their associated text;

FIG. 28 shows the use of `GiveMic to change another avatar's role in a Chat Prop;

FIG. 29 illustrates the use of "gravity wells` to assist a user in controlling avatar gaze; and

FIG. 30 illustrates an example of a screen display indicating that a user is currently interacting with a graphical user interface and not paying attention to the virtual world.

DETAILED DESCRIPTION

A method and apparatus for avatar-centric communication, expression and display during a multi-user online simulation are described. Note that in this description, references to "one embodiment" or "an embodiment" mean that the feature being referred to is included in at least one embodiment of the present invention. Further, separate references to "one embodiment" or "an embodiment" in this description do not necessarily refer to the same embodiment; however, such embodiments are also not mutually exclusive unless so stated, and except as will be readily apparent to those skilled in the art from the description. For example, a feature, structure, act, etc. described in one embodiment may also be included in other embodiments. Thus, the present invention can include a variety of combinations and/or integrations of the embodiments described herein.

Various techniques are introduced below to increase the richness and realism of avatar interactions during a multi-user online simulation. As will be described, these techniques include the use of "Chat Balloons", "Chat Props", cinematic techniques, avatar non-verbal communication, and gaze control.

I. Network Environment

Before considering the avatar-centric techniques introduced herein, refer to FIG. 1, which illustrates a network environment in which the techniques can be implemented. The environment includes a server 1 which is coupled to a number N of clients 2 via a network 3. Although they are referred to separately herein, the server 1 and the clients 2 may be considered part of the network 3. Although only two clients 2 are shown, the number N of clients 2 is not restricted to any particular number for purposes of the present invention. Each of the clients 2 may be a conventional computer system, such as a personal computer (PC), a workstation, a set-top box, a console game system, or even a hand-held device such as a personal digital assistant (PDA), personal information manager (PIM), or the like. The server 1 may be, for example, a conventional server-class computer system or PC, or a combination of two or more such devices. The network 3 may be the Internet, a campus intranet, a wide area network (WAN), a local area network (LAN), or any other type of network or internetwork or combination thereof. Note that the present invention can be applied to networks that use any of a variety of communication techniques, including datagram based networks (e.g., the Internet), connection based (e.g., X.25) networks, virtual circuit based, e.g., Asynchronous Transfer Mode (ATM) networks, etc.

Note also that while FIG. 1 shows a client-server topology, in other embodiments, the techniques described below may be implemented using other topologies, such as a peer-to-peer topology. In addition, it will be recognized that certain aspects of the techniques described herein are normally implemented within a single machine or can be implemented in a single machine. Nonetheless, a client-server topology is assumed henceforth, to facilitate description.

During a simulation, each of the clients 2 maintains data representing a set of objects, representing real-world objects. These objects include avatars controlled by, and representing, the human users of the clients 2. Each machine maintains the same objects, however, the states of the objects may be computed at different times on each machine. Hence, in one embodiment, the server 1 maintains the "true" state of each object, i.e., the state which is considered to be the correct state of the object, and the server periodically distributes information regarding the true state of each object (including avatars) to the clients 2, based on inputs received from the clients 2. Note that in certain embodiments, the server 1 may be implemented in the same machine as one or more of the clients 2, at least for purposes of maintaining the true state of an object. However, for purposes of explanation, it is henceforth assumed that the server 1 is implemented in a separate machine on the network 3, as illustrated in FIG. 1. In addition, there may actually be more than one server 1; for example one server may maintain the true state of certain objects, while another server or servers maintain the true state of other objects or perform other server functions.

FIG. 2 is a high-level block diagram of a processing system that may represent any of the machines shown in FIG. 1. The illustrated system includes one or more central processing units (CPUs) 21 (e.g., a microprocessor), read-only memory (ROM) 22, random access memory (RAM) 23, and a mass storage device 24, each connected to a bus system 31. The bus system 31 may include one or more buses connected to each other through various bridges, controllers and/or adapters, such as are well-known in the art. For example, the bus system 31 may include a system bus that is connected through an adapter and/or bus controller to one or more expansion buses, such as a Peripheral Component Interconnect (PCI) bus. Also coupled to the bus system 31 are a monitor (display device) 25, audio speakers 26, a keyboard 27, a pointing device 28, a simulation input device 29, and a communication device 30.

Mass storage device 24 may include any suitable device for storing large volumes of data, such as magnetic disk or tape, magneto-optical (MO) storage device, or any of various types of Digital Versatile Disk (DVD) or compact disk (CD) based storage device, flash memory etc. The monitor 25 may be any device suitable for displaying alphanumeric, graphical and/or video data to a user, such as a cathode ray tube (CRT), a liquid crystal display (LCD), or the like, and associated controllers. The pointing device 28 may be any suitable device for enabling a user to position a cursor or pointer on the monitor 25, such as a mouse, trackball, touchpad, joystick, or the like. The simulation input device 29 may be any device which receives user inputs to allow a user to control objects in a simulation (including avatars), such as a joystick or the like. Note, however, that the keyboard 27 and/or the pointing device 28 may also serve this same purpose, such that the simulation input device 29 may be omitted in certain embodiments. The communication device 30 may be any device suitable for enabling the computer system to communicate data with another processing system over a network via a communication link 32, such as a conventional telephone modem, a wireless modem, a cable modem, an Integrated Services Digital Network (ISDN) adapter, a Digital Subscriber Line (DSL) adapter, an Ethernet adapter, or the like.

Some of the above-mentioned components may not be present in certain embodiments of the invention, and certain embodiments of the invention may include additional or substitute components not mentioned above. For example, the server 1 may not require components such as a monitor, speakers, keyboard, pointing device, or simulation input device, if it does not receive any direct, local user input.

The techniques introduced herein can be implemented in software, which may be stored in, for example, RAM 23, mass storage device 24, ROM 22, or a combination thereof, in one or more of the processing systems represented in FIG. 1.

FIG. 3 is a block diagram of software, on both a client 2 and on a server 1 (see FIG. 1), that can be used to implement the avatar-related features and techniques described below. In the illustrated embodiment, the server software includes a chat system 312 and a world simulation 313. The world simulation 313 includes a Chat Prop system 314. The client software includes an avatar control system 316; a user input unit 317; a local view manager 318; an emotion parser 319; a Chat Balloon display unit 320, a cinematographer unit 321; and Chat Balloon priorities data 322. The client and server software are described further below (section "VII.").

II. Terminology

Various terms are used in this document to describe embodiments of the invention. Accordingly, the following definitions and explanations apply to terms used herein:

TABLE-US-00001 Auto-Chat Prop An invisible, furniture-less Chat Prop (see definition of "Chat Prop" below) that holds two or more avatars in a conversation. Automatically An action done "automatically" is an action initiated without any user input intended to cause such action. Avatar-centric The general term for all techniques used for facilitating Communication online communication among users, which is centered around avatar body language, in-world Chat Balloons, camera behavior, and expressions, typically coordinated by Chat Props. Bubbling A moodicon (see definition of "moodicon" below) that moodicon moves upward from the pilot avatar. Camera A user viewpoint during a simulation, i.e., a location within the virtual world from which the user can view the virtual world; a virtual camera, unless expressly stated otherwise. Chat Prop A prop that facilitates avatar-based chat between users; typically, a set of one or more furniture items around which avatars can sit to hold a conversation (e.g. a living room with a sofa and two chairs). Various automated features help to facilitate conversation, such as cameras, gaze, and Chat Balloon behaviors. Chat Balloon A comic-book-like bubble or other similar shape containing text, which appears above an avatar's head when its user types text. This is distributed and so other users can see this. Conversational Any act by an avatar that relates to, or occurs within Act the context of, user-to-user communication through avatars, such as speech, a facial expression, a body gesture, or a moodicon by an avatar. Facial Expression A user-triggered avatar animation involving the face, head, and neck. Gaze As a verb: to look at someone or something. As a noun: an act or state of gazing. Gesture A user-triggered avatar animation involving the body, not including the face. Goal Orientation A user-input controlled direction towards which the avatar attempts to gaze and/or walk (also used for driving vehicles). LookAt A state in which an avatar's gaze is locked onto another avatar's head, or some other object of interest. LookAt target An avatar or object of interest on which another avatar has locked its gaze, using LookAt. Moodicon A symbolic graphical object that is emitted from an avatar's head. Objective Gaze Gaze behaviors that appear the same to all users (though they may be delayed in time by different amounts so the combination of behaviors may look somewhat different). Pilot avatar A given user's avatar. Shadow avatar An avatar controlled by someone other than the user. Smiley A word in the Smiley language and its associated emotion. Subjective Gaze Gaze behaviors that are not (necessarily) shared among users, i.e., may not appear the same to all users. The purpose of these is typically to enhance the subjective social experience of the user. Targeted emotion An emotion that only applies when the pilot avatar has a LookAt target. Toggle emotion An emotion that continues until turned off, e.g. `handup in the stage continues until `handdown. Two-person A moodicon that moves from the pilot avatar to a moodicon shadow avatar. User A human being who operates a machine in which techniques described below are implemented.

III. Overview of Described Solutions

Described below are various avatar-related techniques, organized for purposes of description into the following three topic areas:

A. Cinematics, Chat Props and Chat Balloons in Avatar-Centric Communication

B. Avatar-Centric Expressive Communication

C. Avatar Sentient Gaze

IV. Cinematics, Chat Props and Chat Balloons in Avatar-Centric Communication

In previous 3D (virtual) worlds, conversation has been presented in a way that is no different from online text conversations without a 3D world, using instant messaging and email. The presence of avatars has essentially been ignored in the design of communication. Displaying chat text in a 2D window separates it from the 3D world, and thus it cannot be used as an "extension of body language". The techniques introduced herein address these issues by coordinating various communicative elements in-world, within a comprehensive structural system called Chat Props, which are discussed in greater detail below.

The techniques introduced herein embed textual conversation into the 3D space of the avatars, and use cinematic virtual cameras to facilitate conversation and add drama to these conversations. These techniques include the following features: Chat Balloons--Includes controlling Chat Balloon placement, movement, appearance, and font, for facilitating group conversation. Chat Props--Includes: 1) the use of furniture (such as benches and card tables) to position avatars and provide a set of cameras that the user can easily select; and 2) a facility (without furniture) for placing avatars in specific locations around a conversation area, as participants arrive spontaneously (Auto-Chat Props). Cinematics--The use of virtual "cameras" in cinematic ways to facilitate communication and increase the emotional impact of avatar conversations. A. Chat Balloons

In comic books and advertisements, bubble-like shapes called "balloons" are used to contain text representing speech of an illustrated character. These balloons typically have a "tail" pointing to the speaker. In the context of the present invention, these balloons are referred to as "Chat Balloons". In a virtual world, Chat Balloons must be designed to make it clear who is speaking, and what he or she is saying. In addition to designing for these needs, the techniques described herein provide additional visual information, as described below. FIG. 4 shows an example of a computer screen display showing two avatars engaged in a conversation in a 3D virtual world during a simulation, where Chat Balloons are used to represent the avatars' (user-specified) speech.

Chat Balloon Priority

While viewing a group of avatars in the distance, it is useful to be able to tell if that group is engaged in conversation. However, since they are in the distance, it is usually less important to be able to read what is being said or to know which avatar in a group is saying it. But for speaking avatars close to the user's avatar (the pilot avatar), it is important to know who said what, and in what chronological order. Some speakers may be so important that their chat should stand out above everyone else's, e.g. a lecturer on a stage. To solve this problem, a system of Chat Balloon priorities can be used. In certain embodiments of the invention, there are four levels of priority, having the following characteristics:

1) No priority

At this priority level, Chat Balloons are not visible. This keeps irrelevant chatting from interrupting a specific high-priority conversation or social event. For instance, this priority can be used for a stage, to keep inter-stage conversations as clear as possible, without the background interference of Chat Balloons from avatars standing outside of the stage.

2) Low priority

Chat Balloons of distant avatars are generally assigned low priority and are visible if the avatars are visible. Chat Balloons of low priority are displayed in 3D, i.e., they obey laws of perspective in the sense that they appear smaller as their distance to the camera increases, while Chat Balloons of nearby avatars are larger and more legible. Low priority (3D) Chat Balloons are displayed as `billboards`--flat objects positioned in 3D space and oriented so that they always face the camera. More distant Chat Balloons are more translucent than nearby Chat Balloons, and distant Chat Balloons may overlap one another while nearby larger Chat Balloons avoid overlapping from the camera's (user's) viewpoint. So, distance Chat Balloons are small, translucent, and may overlap; their purpose is to show a conversation happening, not to make it legible. This approach saves screen space for nearby Chat Balloons which are larger, more opaque, and non-overlapping so that they are easier to read.

3) High priority

Nearby avatars that are part of a "Chat Group" (described below) are assigned high-priority. This Chat Group includes the user's avatar, all avatars within a certain distance, and all avatars in a Chat Group with any avatar with whom the user's avatar is in a Chat Group. This means that avatars can be some distance away but ensures that the user can share the same conversational context with all his chatmates. Chat Balloons of high priority are not displayed in 3D but are instead displayed on the 2D screen, overlaid on the 3D scene. They are rendered so as to ensure they are legible, since avatars in a Chat Prop together share a single conversation. Chat Balloons of high priority are arranged in vertical columns on the screen and are opaque. More specifically, the screen is divided into a predetermined number of (e.g., seven) columns for placement of Chat Balloons, so up to that number of avatars can chat with high-priority balloons at the same time. Each column contains the Chat Balloons of a different avatar. The columns are allocated to the most recent speakers, such that an avatar who remains silent for some time may lose its balloon column to an avatar who has just recently started speaking. The transition from low priority to high priority based on distance is tuned so that the transition is subtle and non-distracting. Low priority Chat Balloons are not allowed to occlude high priority balloons.

4) Super-high priority

Special avatars can have Chat Balloons of super-high priority, which are always on-screen and impervious to any other competition for attention by other Chat Balloons. For example, these Chat Balloons may have an opaque white background, a strong border, and black text to ensure that the "speech" it contains is always legible. For example, a Presenter (e.g., a master of ceremonies) on a stage Chat Prop may have super-high priority Chat Balloons. These balloons are always on the screen of every member of the Chat Prop whether or not the avatar of the Presenter is visible. Super-high priority Chat Balloons are not allowed to be occluded by other balloons or objects in the world.

FIGS. 5A through 5C illustrate the use of Chat Balloon priority. In this scene, avatars have gathered into two Chat Groups, one Chat Group 51 in the foreground and another Chat Group 52 in the background. The user's avatar is in the foreground Chat Group 51, and the user's view of the world is illustrated schematically by the translucent screen 53 on the left. Both Chat Groups are within the camera's view and so they show up on the screen 53, along with their Chat Balloons. The user's Chat Group 51 has high-priority balloons, while the Chat Balloons of the background Chat Group 52 are low-priority. There are many factors that can be used to determine Chat Balloon priority, but this illustration only shows Chat Balloon priority as a result of the speaker being in or out of the user's chat group.

In-World vs. On-Screen

One way to think of Chat Balloon priority is to consider low-priority balloons as "in-world" and high-priority (and super-high priority) balloons as "on-screen". Low priority Chat Balloons are positioned in the 3D world above their speakers' heads. They are rendered as flat billboards (always facing the camera) causing them to appear 2D, yet still positioned in appropriate locations in the world. Low-priority balloons are translucent and may overlap with each other. They behave roughly like (flat) helium balloons floating above their speakers' heads. Low-priority Chat Balloons may be illegible, because they may be occluded by other balloons or other objects, or they may simply be too far away.

High-priority balloons appear similar to low-priority balloons, except they are always clearly visible and legible. To enable legibility, they have the following attributes: They are opaque They do not overlap They are arranged from left to right according to the left-to-right arrangement of the corresponding avatars' heads They are arranged vertically roughly according to chronology within the conversation They are shown "in front of" everything else in the world--nothing can overlap them

Transitioning Priority

In FIG. 5B, one of the distant avatars 55 approaches the user's Chat Group. Her Chat Balloon 56 floats above her head as she walks. In FIG. 5C, she joins the user's Chat Group 51. After this point, her Chat Balloons are rendered as high priority balloons. The switch from low to high priority appears fairly smooth from the camera's point of view, even though the geometry is mapped from the 3D world to the 2D screen, along with the shifting required to arrange the other high priority Chat Balloons. The differences between low priority Chat Balloons and high priority Chat Balloons can be further seen from FIG. 6.

Providing a smooth visual transition from high to low priority can be accomplished, in part, by mapping the four 2D screen coordinates of the corners of the high (2D) priority balloon to 3D positions in the world, approximately over the head of the associated avatar, such that the visual result to the user at the time of the transition is minimal or non-existent. In addition, while in the low priority (3D) state, the actual 3D size of a Chat Balloon can change as a function of a change in its distance to the camera. It increases in size as a function of distance while less than a predetermined critical distance to the camera, so as to make its apparent size either the same to the user as if it were a high-priority balloon, or to appear to shrink by some pre-determined amount. When the distance from the low priority Chat Balloon to the camera is larger than this critical distance, the Chat Balloon's actual size (as opposed to apparent size) remains constant, such that the Chat Balloon behaves like a normal 3D object displayed using 3D perspective geometry and appears to recede as a function of distance in a natural way.

Chat Columns--Vertical Registration of Group Chat Balloons

In the above-described priority system, each user whose avatar is in a Chat Prop (described further below) actively converses with users controlling nearby avatars which have high priority Chat Balloons, all of which are arranged in chat columns. The left-to-right arrangement of the chat columns matches the left-to-right arrangement of the avatars on the screen. This approach is shown in FIG. 6. The Chat Balloons of avatars 64, 65 and 66 in the user's Chat Group 67 are high priority and, therefore, are placed in chat columns 61, 62 and 63, respectively. In contrast, the low priority Chat Balloons 68 of the distant Chat Group (the Chat Group to which the user does not belong) are not arranged into chat columns in this manner.

Because there may be more avatars than available chat columns, and because columns may not always appear directly over their associated avatars, a number of techniques can be used to clearly identify the correspondence between an avatar and its associated Chat Balloons, such as the following (note that not all of these techniques need be implemented in any given embodiment of the invention): Chat Balloons are created when a user begins typing text; they first appear immediately over the speaking avatar, quickly increase to full size, and float upward towards the appropriate chat column and height. Whenever the left-to-right arrangement of avatars on screen changes, the Chat Balloon left-to-right arrangement shifts accordingly, with ease-in, ease-out motions. The balloons also ignore rapid shifts in arrangement caused by small motions, such as when two avatars are nearly occluding each other. Left-to-right arrangement can also be affected by the user changing cameras or avatars moving around. For quick camera cuts to facial expressions, temporary balloon adjustments would be disruptive. So temporary cut-away cameras freeze the Chat Balloon column assignments. When avatars move around, the columns change, but this change occurs with some hesitation to avoid bouncing back and fourth when two avatars are nearly in each others way. Chat Balloons fill up with words much like a balloon being blown up with helium, as words are added. When the amount of text reaches a critical maximum, the Chat Balloon is "released", and a new Chat Balloon is created to accept the continuing flow of new words. Multiple Chat Balloons accumulate in a vertical stack as a result. These vertically-stacked Chat Balloons avoid collisions with each other. Consecutive balloons are connected by an umbilical cord-like connector, much like what is used in comic books. There is a limit to the number of vertically-stacked Chat Balloons, thus, the highest (oldest) Chat Balloon will rise and disappear off the top of the screen when a fresh balloon is created, thus keeping the number within the limit. The lowest (newest) Chat Balloon has a tail, which points towards the source of the chat (the avatar's head position). When a user completes the typing of a word, a small dash, or "bullet" flies from the source avatar, through the tail, and to the base of the Chat Balloon. These bullets visually represent "speech" from the avatar's mouth to the balloon, and provide a subtle visual dynamic that helps to visualize group conversation rate, speech overlap, etc. Quiet and subtle sound effects occur at key Chat Balloon events, to accompany visual effects, to help users sense conversational dynamics, often on an unconscious level. These sound effects (and their associated events) may signal: Initial creation of a Chat Balloon Completion of words, accompanied by bullets Erasing or backspacing letters of a word "releasing" a Chat Balloon--e.g., caused by the user hitting the return key time-out release of upper-most balloon, as a result of reaching maximum lifespan

Chat Balloon Customization and Individuality

A user can change certain visual characteristics of his Chat Balloons to personalize them, using a graphical user interface on the client. This customization also adds variety which makes it easier to identify the balloons of a specific avatar. The Chat Balloon features that a user can customize may include, for example: border color, text font, text color, background color, and choices of sound effects.

Water Wheel Metaphor

If Chat Balloons were to disappear too soon after creation, users might miss something. If the order of Chat Balloons is not visualized as they rise, then it becomes difficult to follow the chronology of the conversation. To address these two problems, the concept of a "water wheel" is used, as illustrated in FIGS. 7A and 7B. The waterwheel metaphor is a way to explain the behavior of balloons as they assume their positions within the vertical columns and as they stack vertically over avatar heads. FIG. 7B illustrates the water wheel 71 at a point in time just after that of FIG. 7A. The water wheel 71 rotates about an imaginary axis 72, as if due to water flowing over it. However, since Chat Balloons can be thought of as being filled with imaginary helium, the water wheel moves upward, rather than downward. As a user fills up balloons causing them to be released automatically, or hits the return key to release the balloon, the Chat Balloon 73 moves up and joins the waterwheel. With every Chat Balloon release within the Chat Group, the water wheel rises one notch, maintaining its chronological order with balloons from other speakers. The goal of the waterwheel is to keep Chat Balloons on the screen as long as possible to be read while ensuring that older Chat Balloons are always higher than younger balloons. So, Chat Balloons do not rise according to some simulated physics (like a helium balloon), but rather, according to a mechanism that slows their ascent while maintaining order.

Interleaving Expressions with Words

Users must be able to follow a conversation and understand the relationship of avatar expressions in connection to their words. This is one of the purposes fulfilled by word-at-a-time" Chat Balloon text entry. In certain embodiments of the invention, users can type words, mixed with triggering avatar expressions, for a dynamic mix of verbal and non-verbal communication.

To properly interleave emotional expression and chat, in the system of the present invention, chat is normally transmitted across the network in units of what is commonly considered the primary unit of meaning in language: the word. Thus, words that trigger expressive "body language" appear in their correct place in the sentence with the other words of the conversation. The word-at-a-time feature also allows conversations to occur in a more parallel fashion, much as they tend to do in person. Space or punctuation ends a word, sending it to the Chat Balloon. FIGS. 8A and 8B illustrate the word-at-a-time chat feature, and more specifically, two avatars speaking at the same time and responding to each other as the speech occurs.

Hitting the return key releases the currently-forming Chat Balloon. New users may not know the conventions used for avatar chat, so they may not hit return to free the balloon or a space or punctuation after a word to send it. To fulfill their intent, both balloons and words can be sent automatically after a specified delay. In certain embodiments, pressing return also pushes balloons up so that it is possible for a user to get rid of a balloon containing something previously said, to avoid confusion if someone new approaches and a new conversation is starting.

Chat Balloons provide a shared conversational experience. Since all Chat Balloon behavior is distributed, Chat Balloons appear the same to all users in a shared conversation. Because of this, each user can determine whether he needs to slow down to give others a chance to read what they have written. Social conventions are allowed to emerge among chatters, because they share the same visual information.

To help clarify meaning and reference previously entered text, users can click on another avatar's Chat Balloon to create a reply to that particular balloon. This creates a visual connector between the two balloons, as shown in FIG. 9.

B. Chat Props

In the real world, conversations normally have a context or subject. Conversations also often have "flavors" and meaning. Talking on a ridge overlooking the Grand Canyon feels very different from talking in an intimate coffee house; a dance club or bar feels different yet again. A virtual world may contain all of these situations and contexts. Consequently, introduced herein is a special type of virtual furniture, called Chat Props, for avatars to sit on to talk in these contexts. A Chat Prop is a prop that defines the context for a an avatar conversation. An example of a Chat Prop is shown in FIG. 10, which shows a living room Chat Prop.

The primary purpose of Chat Props is to position avatars so that it is possible to establish effective camera views and to provide convenient relative positioning of avatars for viewing expressions. Because Chat Props ensure that avatars are in known locations, cameras can be pre-positioned within the Chat Prop in such a way as to frame the conversation and heighten drama. Then a user can cycle through the pre-positioned camera positions quickly rather than having to take the time to position the camera manually.

Examples of Chat Props are: a single chair; a bench with two seats; two chairs facing one another; a bench facing two chairs; or a stage with standing slots for a Master of Ceremonies, a guest, and 12 audience seats divided by an aisle into two teams. Any grouping of furniture and its environment can be built as a Chat Prop so as to group avatars into a single conversational unit. Thus, all of the avatars in a Chat Prop see each other's balloons as high priority by default (although this can change under certain conditions as described below).

Avatar Slots

A Chat Prop has a fixed number of slots in known locations. A slot is typically a seat, such as one of the two seat areas on a small sofa or bench. However, a slot can also be a location to stand, or even a location floating in the air on a beam of light--it all depends on context of the Chat Prop. Each slot can be occupied by only one avatar. When an avatar approaches a Chat Prop, its user can choose to join the Chat Prop at any available slot.

Operation of Avatar Slots

Referring now to FIG. 11, avatar slots are now further explained with reference to a schematically-illustrated sample Chat Prop 112. The avatar slots 111 are pre-defined positions in the Chat Prop 112 with headings relative to the Chat Prop 112 itself. An avatar slot 111 can be thought of as a local frame of reference (coordinate system) containing a position and an orientation. The Chat Prop 112 also has a frame of reference (position and orientation) 114 within the simulated "world" 113, and all of the slots 111 in the Chat Prop 112 are defined relative to the frame of reference of the Chat Prop. Further, the simulated "world" 113 has its own frame of reference 115.

So, for instance, a Chat Prop in the form of a card table might be designed to have four seats surrounding the table in positions corresponding to four chairs and headings corresponding to the facing of the chairs inward. When the pilot avatar approaches the Chat Prop, the user can choose to join the Chat Prop. That choice could be either general (i.e., join the nearest available seat) or specific to a slot (e.g., clicking on the seat to join, clicking on a pop-up menu over the seat, etc). On joining, a brief animation is played of the avatar sitting down, and a variety of events can happen, such as camera changes, gaze changes as avatars acknowledge the newcomer, and other things, depending on the nature of the Chat Prop.

Designing Chat Props

Chat Props can be designed using a variety of software and modeling tools, such as: C++ for the lower-level engine; a 3D modeling and animation package for designing Chat Prop visuals, camera placements, and avatar animations (e.g., 3D Studio Max); and a scripting environment for higher-level architecture, Chat Prop specification, and event-triggered behavior. The combination of these design tools (especially the scripting level) empowers designers and programmers to build a large variety of Chat Props and invent many variations.

Specification of Chat Balloon Priority in a Chat Prop

Generally, when in a Chat Prop, the Chat Balloons of all the other avatars in the Chat Prop and any avatars standing close enough to share a Chat Group are high-priority. But, by using a general scripting environment one could design specialized Chat Props with alternative priority schemes for the Chat Balloons of other avatars. These priorities can be set for avatars in the other slots as well as avatars within specific regions within some distance of the Chat Prop. The regions may be enclosed in spheres defined with a center and a radius. There can be up to a predetermined number of regions, which are ordered in priority. The regions can be reset at any time, such as when a camera changes (see section on Cinematics). The following are examples of using prioritization in this manner: A Stage Chat Prop is illustrated in FIGS. 12A and 12B. The Stage Chat Prop has an audience consisting of two teams of six chairs each. The teams can have discussions where only Chat Balloons of teammates are visible. At other times, when the stage is being used for a fashion show, the region of the stage can be set to high priority for the avatars in the stage Chat Prop, so the Chat Balloons of any avatar on the stage are more legible, even though those avatars have not joined the Chat Prop. Similarly, the Chat Balloons of avatars behind the stage can be made invisible to the avatars in the stage Chat Prop. The Chat Balloons of the avatar in the Presenter (the presenter) slot on the stage can be set to the highest priority so that they are always visible, with an opaque white background easily distinguished from other Chat Balloons.

Chat Props in Motion

The Chat Props described above are static furniture elements positioned in a predetermined configuration. But Chat Props can also be animated or mobile in a number of ways, as illustrated in the following examples: 1. "Swing Set" paradigm: animated Chat Prop, stationary camera) The swing set Chat Prop simulates a swing set commonly found in a children's playground. In this type of Chat Prop, Chat Balloon column assignment stays fixed (to avoid confusing motion from swinging avatar positions). The assignment of Chat Balloons to columns does not immediately change, so Chat Balloon