Providing conference data in a network communications system based on client or server information examined during a conference Number:7,426,191 from the United States Patent and Trademark Office (PTO) owispatent An improved networked computer communications system handles arbitrary streams of data, and transports at varying speeds those streams where intermediate updates can be dropped if they are obsoleted by later arriving data updates, optimizing the utilization of network and node resources. Complex buffering by system server software allows distributed, parallel, or redundant processing, transmission, and storage for performance, reliability, and robustness. Various parameters of the system can be monitored, and the system can be reconfigured automatically based on the observations. Varied techniques reduce the perceived end-to-end latency and take advantage of software and hardware capabilities that assets connected to the system may possess. One conferencing system allows conference participants to share all or a portion of the display seen on their computer screens. The conferees may be at sites removed from each other, or may view a recorded presentation or archived conference at different times. Conference participants are either "presenters" who can modify the display or "attendees" who cannot modify the display. A pointer icon, which can be labeled to identify the conferee, is displayed on the shared image area. Each conferee can modify the position of his or her own pointer, even when not presenting, so that every participant can see what each conferee is pointing to, should a conferee choose to point to an element of the display. These and other features apply to other data streams shared in the conference or in meetings where there is no shared-image data stream.<H communications, information, Providing, confe, 7426191.html, Patent, pto, 7426191

Title: Providing conference data in a network communications system based on client or server information examined during a conference

Abstract: An improved networked computer communications system handles arbitrary streams of data, and transports at varying speeds those streams where intermediate updates can be dropped if they are obsoleted by later arriving data updates, optimizing the utilization of network and node resources. Complex buffering by system server software allows distributed, parallel, or redundant processing, transmission, and storage for performance, reliability, and robustness. Various parameters of the system can be monitored, and the system can be reconfigured automatically based on the observations. Varied techniques reduce the perceived end-to-end latency and take advantage of software and hardware capabilities that assets connected to the system may possess. One conferencing system allows conference participants to share all or a portion of the display seen on their computer screens. The conferees may be at sites removed from each other, or may view a recorded presentation or archived conference at different times. Conference participants are either "presenters" who can modify the display or "attendees" who cannot modify the display. A pointer icon, which can be labeled to identify the conferee, is displayed on the shared image area. Each conferee can modify the position of his or her own pointer, even when not presenting, so that every participant can see what each conferee is pointing to, should a conferee choose to point to an element of the display. These and other features apply to other data streams shared in the conference or in meetings where there is no shared-image data stream.

Patent Number: 7,426,191 Issued on 09/16/2008 to Salesky, et al.

Inventors: Salesky; Joseph (Cameron Park, CA), Madams; Peter (Moraga, CA), Flower; John (Walnut Creek, CA), Kaul; Clint (San Mateo, CA), Wells; Benjamin (Walnut Creek, CA), Janne; Edward Arthur Ho-Ming (San Francisco, CA)
Assignee: Pixion, Inc. (San Ramon, CA)

Appl. No.: 11/086,506

Filed: March 21, 2005

Related U.S. Patent Documents


Application Number	Filing Date	Patent Number	Issue Date
10600144	Jun., 2003	7197535
09523315	Mar., 2000
08823744	Jan., 2002	6343313
60014242	Mar., 1996

Current U.S. Class: 370/260 ; 370/261; 709/203

Current International Class: H04L 12/16 (20060101)

References Cited [Referenced By]

U.S. Patent Documents


4414621	November 1983	Bown et al.
5241625	August 1993	Epard et al.
5455599	October 1995	Cabral et al.
5485212	January 1996	Frederick
5530795	June 1996	Wan
5550982	August 1996	Long et al.
5563649	October 1996	Gould et al.
5600646	February 1997	Polomski
5617539	April 1997	Ludwig et al.
5649104	July 1997	Carleton et al.
5689553	November 1997	Ahuja et al.
5689641	November 1997	Ludwig et al.
5706290	January 1998	Shaw et al.
5737448	April 1998	Gardos
5740371	April 1998	Wallis
5764731	June 1998	Yablon
5774668	June 1998	Choquier et al.
5790818	August 1998	Martin
5793365	August 1998	Tang et al.
5793980	August 1998	Glaser et al.
5802322	September 1998	Niblett
5822523	October 1998	Rothschild et al.
5859979	January 1999	Tung et al.
5877762	March 1999	Young
5983263	November 1999	Rothrock et al.
6313863	November 2001	Chida
6343313	January 2002	Salesky et al.
6345297	February 2002	Grimm et al.
6560707	May 2003	Curtis et al.
6594699	July 2003	Sahai et al.
6772335	August 2004	Curtis et al.

Other References

Co-Pending U.S. Appl. No. 10/600,144, Joseph Salesky, Real-Time, Multi-Point, Multi-Speed, Multi-Stream Scalable Computer Network Communications System, filed Jun. 19, 2003. cited by other .
Co-Pending U.S. Appl. No. 10/753,702, Joseph Salesky, Real-Time, Multi-Point, Multi-Speed, Multi-Stream Scalable Computer Network Communications System, filed Jan. 7, 2004. cited by other .
Co-Pending U.S. Appl. No. 11/086,507, Joseph Salesky, Real-Time, Multi-Point, Multi-Speed, Multi-Stream Scalable Computer Network Communications System, filed Mar. 21, 2005. cited by other .
Stahl, S., "Conferencing Breakthrough," Distributed by CMF Publications, Jun. 26, 1995, http://www.informationweek.com/533/33iucon.htm, last visited Apr. 13, 2005. cited by other .
"Face Off," Distributed by Byte, http://www.byte.com/art/9510/sec7/art3.htm., Oct 1995. cited by other .
"Transmission Control Protocol," Information Sciences Institute, University of Southern California, 1991, 83 pages. cited by other .
Eleftheriadis, A. et al., "Meeting Arbitrary QoS Constraints Using Dynamic Rate Shaping of Coded Digital Video," Proceedings of the 5th International Workshop on Network and Operating System Support for Digital Audio and Video, 1995, pp. 1-12. cited by other .
Frederick, R., "Experiences with Real-Time Software Video Compression," Xerox PARC, 1994, pp. 1-4. cited by other .
Gajewska, H. et al., "Argo: A System for Distributed Collaboration," Systems Research Center, Digital Equipment Corporation, 8 pages. cited by other .
Jeffay, K. et al., "Adaptive, Best-Effort Delivery of Digital Audio and Video Across Packet-Switched Networks," University of North Carolina at Chapel Hill, Department of Computer Science, 1993, 12 pages. cited by other .
Jeffay, K. et al., "Transport and Display Mechanisms for Multimedia Conferencing Across Packet-Switched Networks," University of North Carolina at Chapel Hill, Department of Computer Science, 1993, 28 pages. cited by other .
Macedonia, M. et al., "Mbone Provides Audio and Video Across the Internet," IEEE Computer, 1994, 12 pages. cited by other .
McCanne, S. et al., "Receiver-Driven Layered Multicast," University of California, Berkeley and Lawrence Berkeley National Laboratory, 1996, 14 pages. cited by other .
Schulzrinne, H. et al., "RTP: A Transport Protocol for Real-Time Applications," Network Working Group, Request for Comments, 1989, 66 pages. cited by other .
Stallings, W., "Data and Computer Communications," Fourth Edition, Copyright 1985, 1988, 1991, and 1994, 12 pages. cited by other .
Turletti, T. et al., "Issues with Multicast Video Distribution in Heterogeneous Packet Networks," Sophia-Antipolis Codex, France, 4 pages. cited by other .
Wakeman, I., "Packetized Video-Options for Interaction Between the User, the Network and the Codec," The Computer Journal, vol. 36, No. 1, 1993, pp. 55-67. cited by other .
Wolf, K. et al., "Multimedia Application Sharing in a Heterogeneous Environment," ACM Multimedia 95--Electronic Proceedings, 1995, 14 pages. cited by other .
Wood, K. et al., "Global Teleporting with Java: Towards Ubiquitous Personalized Computing," The Olivetti & Oracle Research Laboratory, University of Cambridge Computer Laboratory, pp. 1-13. cited by other .
United States District Court for the Northern District of California, Claim Construction Order re: U.S. Patent No. 6,343,313; Jul. 28, 2004. cited by other .
Wikipedia, "Multipoint Control Unit," Dec. 20, 2005 (http://en.wikipedia.org/wiki/Multipoint.sub.--Control.sub.--Unit). cited by other .
Networkworld, "MCU," Oct. 22, 2001 (http://www.networkworld.com/details/762.html?def). cited by other .
Techweb, "MCU," 2003 (http://www.techweb.com/encyclopedia/shared/printArticlePageSrc.jhtml?ter- m=MCU). cited by other.

Primary Examiner: Wilson; Robert W.
Attorney, Agent or Firm: Carr & Ferrell LLP

Parent Case Text

The present application is a continuation application of U.S. patent application Ser. No. 10/600,144, filed Jun. 19, 2003 and entitled "Real-Time, Multi-Point, Multi-Speed, Multi-Stream Scalable Computer Network Communications System"; U.S. patent application Ser. No. 10/600,144 is a continuation application of U.S. patent application Ser. No. 09/523,315, filed Mar. 10, 2000 and entitled "Real-Time, Multi-Point, Multi-Speed, Multi-Stream Scalable Computer Network Communications System," now abandoned; U.S. patent application Ser. No. 09/523,315 is a divisional application of U.S. patent application Ser. No. 08/823,744, filed Mar. 25, 1997, now U.S. Pat. No. 6,343,313, issued Jan. 29, 2002 and entitled "Computer Conferencing System with Real-Time Multipoint, Multi-Speed, Multi-Stream Scalability"; U.S. Pat. No. 6,343,313 claims the benefit of U.S. provisional patent application No. 60/014,242, filed Mar. 26, 1996 all of which the present application incorporates by reference. The present application is also related to U.S. patent application Ser. No. 10/753,702, filed on Jan. 7, 2004 and entitled "Providing Data Updates in a Network Communications System Based on Connection or Load Parameters" and U.S. patent application Ser. No. 11/086,507 filed on Mar. 21, 2005 and entitled "Providing Conferencing Data in a Network Communications System Based on Client Capabilities."

Claims

What is claimed is:

1. A conferencing system comprising: a conference server; at least one client the at least one client including a web browser; and at least one network connection coupling the conference server and the at least one client, the conference server providing conferencing data to the at least one client via the at least one network connection after establishing a client-server connection, the client-server connection established via the web browser at the at least one client having been navigated to a Universal Resource Locator associated with a conference, and wherein one or more characteristics of the provided conferencing data are based on client or server information examined subsequent to both the client-server connection having been established and the client joining the conference.

2. The conferencing system of claim 1, wherein a characteristic of the one or more characteristics of the provided conferencing data comprises a size of at least part of the conferencing data.

3. The conferencing system of claim 1, wherein a characteristic of the one or more characteristics of the provided conferencing data comprises at least some contents within the conferencing data.

4. The conferencing system of claim 1, wherein a characteristic of the one or more characteristics of the provided conferencing data comprises at least some information related to at least part of the conferencing data.

5. The conferencing system of claim 1, wherein a characteristic of the one or more characteristics of the provided conferencing data comprises at least one source of at least part of the conferencing data.

6. The conferencing system of claim 1, wherein a characteristic of the one or more characteristics of the provided conferencing data comprises at least one destination of at least part of the conferencing data.

7. The conferencing system of claim 1, wherein a characteristic of the one or more characteristics of the provided conferencing data comprises a rate of reception of at least part of the conferencing data.

8. The conferencing system of claim 1, wherein a characteristic of the one or more characteristics of the provided conferencing data comprises a rate of transmission of at least part of the conferencing data.

9. The conferencing system of claim 1, wherein a characteristic of the one or more characteristics of the provided conferencing data comprises hardware involved in processing at least part of the conferencing data.

10. The conferencing system of claim 1, wherein a characteristic of the one or more characteristics of the provided conferencing data comprises software involved in processing at least part of the conferencing data.

11. The conferencing system of claim 1, wherein a characteristic of the one or more characteristics of the provided conferencing data comprises at least one protocol used in processing at least part of the conferencing data.

12. The conferencing system of claim 1, wherein the client or server information examined is related to at least a size of at least part of the conferencing data.

13. The conferencing system of claim 1, wherein the client or server information examined is related to at least a calculation of a size of at least part of the conferencing data.

14. The conferencing system of claim 1, wherein the client or server information examined is related to at least some content within the conferencing data.

15. The conferencing system of claim 1, wherein the client or server information examined is related to at least some information related to at least part of the conferencing data.

16. The conferencing system of claim 1, wherein the client or server information examined is related to at least one source of at least part of the conferencing data.

17. The conferencing system of claim 1, wherein the client or server information examined is related to at least one destination of at least part of the conferencing data.

18. The conferencing system of claim 1, wherein the client or server information examined is related to a rate of reception of at least a part of the conferencing data.

19. The conferencing system of claim 1, wherein the client or server information examined is related to a rate of transmission of at least a part of the conferencing data.

20. The conferencing system of claim 1, wherein the client or saver information examined is related to hardware involved in processing at least part of the conferencing data.

21. The conferencing system of claim 1, wherein the client or server information examined is related to software involved in processing at least part of the conferencing data.

22. The conferencing system of claim 1, wherein the client or server information examined is related to at least one protocol used in processing at least part of the conferencing data.

23. The conferencing system of claim 1, wherein the client or server information examined comprises a speed of the server.

24. The conferencing system of claim 1, wherein the client or server information examined comprises a speed of at least one of the clients.

25. The conferencing system of claim 1, wherein the client or server information examined comprises a speed of at least one of the network connections.

26. The conferencing system of claim 1, wherein the client or server information examined comprises a load of the server.

27. The conferencing system of claim 1, wherein the client or server information examined comprises a load of at least one of the clients.

28. The conferencing system of claim 1, wherein the client or server information examined comprises a load of at least one of the network connections.

29. The conferencing system of claim 1, wherein a characteristic of the one or more characteristics is based on at least one modification to a conference environment originated by the server.

30. The conferencing system of claim 1, wherein a characteristic of the one or more characteristics is based at least one modification to a conference environment originated by at least one client.

31. The conferencing system of claim 1, wherein the client or server information comprises at least one modification to a conference environment originated by the server.

32. The conferencing system of claim 1, wherein the client or server information comprises at Least one modification to a conference environment originated by at least one client.

33. The system of claim 1, wherein the at least one client is configured to provide the conference server with a key prior to joining the conference, the key identifying one or more privileges of the at least one client during the conference.

34. The system of claim 33, wherein the one or more privileges includes entering the conference.

35. The system of claim 33, wherein the one or more privileges includes being a presented during the conference.

36. The system of claim 33, wherein the one or more privileges includes access to information about another attendee in the conference.

37. The system of claim 33, wherein the one or more privileges includes changing a conference setting.

38. The system of claim 1, wherein the client or server information is examined at various points during the conference.

39. A method for conferencing between a server and at least one client in a conferencing system, comprising: establishing a network connection between the server and the at least one client, the network connection established via a web browser at the at least one client having been navigated to a Universal Resource Locator associated with a conference; examining client or server information subsequent to both the client-server connection having been established and the client joining the conference; and providing conferencing data from the server to the at least one client alter establishing a client-server connection and the at least one client having joined the conference, wherein one or more characteristics of the conferencing data are based on the examined client or server information.

40. The method of claim 39, further comprising determining one or more characteristics of conferencing data for delivery during a first portion of the conference, the determination based on current capabilities of the at least one client validated after establishing the client-server connection but prior to the at least one client joining the conference.

41. The method of claim 40, wherein the one or more characteristics of conferencing data for delivery during the first portion of the conference are changed in favor of the one or more characteristics of the conferencing data based on the client or server information examined subsequent to both the client-server connection having been established and the client joining the conference.

42. The method of claim 39, wherein the client or server information is examined at multiple points during the conference thereby resulting in an adjustment of the one or more characteristics of the conferencing data provided during the conference.

Description

BACKGROUND OF THE INVENTION

The present invention relates generally to the field of shared computer communications and computer conferencing. In particular, one embodiment of a conferencing system according to the present invention facilitates the conferencing of two or more persons, each with a computer at one or more locations with a shared visual display and additional communication capabilities such as video, shared drawing, audio, text chat, etc. and facilitates the recording and later playback of the communications.

Existing conferencing systems can be described as either video conferencing systems or "whiteboard" systems. In a video conferencing system, a snap-shot of the conference presentation is taken at regular intervals, such as thirty times per second. Given that the image on a computer display is not changing nearly that often, video conferencing wastes large amounts of bandwidth. In a whiteboard system, the presenter at the conference draws within a whiteboard application or imports the output of another program into the whiteboard program for manipulation. When the presenter is ready to present a snap-shot, the presenter presses a "send" button and the whiteboard program updates all the attendees' displays with the image created by the presenter. This type of system, while requiring less bandwidth than video conferencing, is clumsy to use, lacks real-time responses, and limits the presenter to the tools provided with the whiteboard program.

Existing shared-display or shared-image systems rely on interception and communication of display or graphics system commands or depend on conferees' having similar hardware and software platforms. These systems lack flexibility and performance if the network connections are unreliable or have narrow bandwidth, or they require uniform hardware or software installations.

Existing systems that provide single or multiple data stream handling of a nature different than shared-image conferencing depend on wide bandwidth network connections or on all participants having similar platforms.

SUMMARY OF THE INVENTION

An improved general purpose data-stream computer network transport system and, in particular, an improved desktop conferencing system is provided by virtue of the present invention. The desktop conferencing system is used to display a shared collaboration among conference participants ("conferees"), with one or more individuals located at each remote site connected to the conference. Typically, at any particular time some conferees are not able to modify the shared images, and thus they are "attendees," as opposed to "presenters." Preferably, only one conferee is the presenter at any one time. A pointer icon for each conferee can be displayed on the screen, and the conferee is able to modify the location of his or her pointer, even if the conferee is not one who can modify the shared display itself. Each of the pointers can be labeled to distinguish each of the conferees.

In a specific implementation of the desktop conferencing system, conferee client computers ("conferee clients") connect to the "conference server," a computer or several networked computers (any of which may also be used by a conferee as a client computer) running conferencing software, typically by navigating a World Wide Web ("WWW" or "Web") browser through a predetermined Universal Resource Locator ("URL") that indicates a Web page describing the conference. The conference can be set up any time earlier by anyone with access to this server function. At the time of setup, one or more password character strings ("keys") can be specified for the conference. The key that a conferee gives at the time of attempting to connect to the conference server determines whether that conferee will be allowed access to the conference and what the conferee's initial privileges will be for participating in the conference and for modifying the setup of the conference. These privileges include but are not are not limited to the following: entering the conference, being a presenter, having a pointer, seeing the icons or other identifying information of other attendees, hiding or sharing one's own icon or identifying information, changing descriptive information such as the name, time, and purpose of the conference, changing keys, and changing others' privileges. The privileges can be modified during the conference by conferees or others who are so authorized. In general terms, the privileges include those that conferees might enjoy in person at a conventional, physical meeting. In the description below, a conferencing or other communications session provided by the present invention will sometimes be called a "meeting."

A presenter uses his or her computer to begin a conference presentation by connecting to the conference server. Conferencing software on the presenter client computer captures a portion of the screen display of the presenter client and sends the captured region (after possibly compressing it or applying other transformations) to the conference server. The captured region can be anything the presenter client can have displayed on its screen or a portion thereof, whether or not the hardware or other software producing or managing any part of the display is aware of the conferencing system.

When the attendee selects a link from the Web page to begin the conferencing session for that attendee, this action initiates the attendee client conferencing software. The attendee client then obtains a current view of the captured region from the conference server. The position of a pointer icon on a conferee's view of the captured region and an icon specified by the conferee might be communicated to each of the other attendee and presenter clients, so that each of the participants can see what each conferee is pointing at should a conferee choose to point to an element of the shared captured region. A particular conference can include more than one presenter; all conferees may be presenters, or all conferees may be non-presenting attendees. The latter may happen if a conference is set up to review a previously recorded or archived conference.

In a simple embodiment, the entire screen of the presenter is shown to all of the attendees. In a more complex embodiment, multiple subsets of multiple presenters' screens might be made available to all attendees while other subsets of the displays of the presenters are viewable by a subset of the attendees, thus allowing private side "conversations." These side conversations can be flexibly reconfigured during the conference, according to the conferees' privileges; participants in side conversations can have separate pointers whose positions are independent of, and whose labeling icons are distinguished from, those appearing in the general conference.

As each conferee joins a conference, the client and the conference server agree on the capabilities of the client, such as display bit-depth, bandwidth of the connection between client and the conference server, processor speed of the client, and the amount of memory available to each client. These parameters may be modified by the conferee, the client, or the server: this can be done automatically or on demand. If the conference server determines that a client has sufficient computing resources, some of the tasks, such as image data compression (for presenter clients), decompression (for attendee clients), update scheduling (both types of clients), and other image transformations and server management functions can be assigned to the client computers. The client computers might be personal computers, workstations, X-terminals, cable or satellite TV set-top boxes ("STBs"), personal digital assistants ("PDAs"), game playing machines, WebTV.TM.s, network computers ("NCs"), Infopads, visual telephones, and other existing or as yet undeveloped input and/or output devices. These clients might be connected to the server computer or computers (and the server computers might be interconnected) by high or low bandwidth electrical or optical connections, radio, infrared, microwave, telephone modem, or hybrid combinations of these, or other existing or as yet undeveloped data communication technologies.

The system can supply a range of coder-decoder ("codec") facilities for the compression and decompression of images (in order to reduce bandwidth requirements during network transmission) and for the matching of image representations to client display requirements including input or output format transcoding (in order that the shared image appear visually similar to presenter and attendee). In addition, codecs may be provided by the system for such purposes as error-correction, encryption, or audio and video noise reduction, or others. User-provided or proprietary codecs for these purposes and more can also be incorporated into the system. Any of these codecs may be in form of software or specialized hardware. Multiple codecs may be provided for the same function; should the system determine that one is better suited for that function, then it may be selected, and the codec can be changed dynamically when conditions change, such as client requirements, server needs, and network loading.

At least one embodiment of the present invention provides real-time, multi-point, multi-speed transport over a computer network for data streams other than the visual conference shared images described above, including but not limited to audio, video, shared paint and drawing spaces, text chat, and other real-time object streams where intermediate updates may be dropped; in particular, the data streams may combine any or all of these types of data, which may be produced by multiple presenters, and arbitrary data streams may be combined with these. The features of connecting to servers, setting up conferences, keying privileges, passing identifications, accommodating multiple dissimilar platforms and network connections, and configuring subsets of conferees apply equally to these other data streams. In the more general case, the "communications server" connects the "source" and "sink" client machines of the "communicants" during a communication session.

But the system is not limited to real-time; thus, for example, archiving is provided. It is not limited to multi-point; thus, for example, a single user can record for later playback; being scalable means it works well for a few users and provides a similar communications service and experience with many users. It is not limited to multi-speed; thus, for example, data streams where lost information cannot be easily updated by later versions can be accommodated. It is not limited to multi-stream; for the shared screen-image stream (frequently used here as an example) by itself offers great utility. Indeed, it does not require a network: for example, the same computer could be the recording and archiving server for a presenter using it as a client; or the same computer could run presenter client software, attendee client software, and the communication server software connecting them so that a presentation might be previewed from the attendee's point of view.

Although a simple embodiment uses a single computer as the communications server, a more complex embodiment connects several computers in performing the server functions. The server-to-server interconnections can optimize routing by using information provided in the data stream or measured on the network, optimize wide-area network (WAN) usage by connecting clients to nearby servers, provide backup reliability by migrating clients, provide scalability of conference size through splitting the data stream, improve performance and robustness through redundant routing, and distribute functions of the system's transport pipeline (such as compression, decompression, and update scheduling) over several server and client computers. These services can be provided automatically depending on resources of the computers and network (for example, measured net speed and central processing unit, or "CPU," load) and facilities available (for example, announced client characteristics, such as CPU speed, compression and/or decompression hardware, or display parameters). They can also be configured and constrained by the server computer administrators or others with appropriate privileges.

Existing systems do not provide one or more of the following, which are explained in greater detail below: multi-speed at server and client, multiple reconfigurable coder-decoder tranformations and transcodings, storage services (for, e.g., caching, failure recovery, recording, archiving, and playback), keyed access and privilege granting, adaptable servers and clients, multiple servers, adaptive and redundant server-to-server routing, load sharing among clients and servers, adaptive server-to-client matching, client/server and server/server backup and reconnection, multiple protocols for client connections, dynamic reconfiguration of server functions, and scaling beyond single process, host, or network limitations automatically or upon request.

A more complete understanding of the nature, features, and advantages of the invention will be realized by referring to the following description and claims together with the associated drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a desktop conferencing system based on the present invention.

FIG. 2 is a flowchart illustrating the connection of a conferee client computer to a conference server shown in FIG. 1.

FIG. 3 is a block diagram of the data flow in an architecture commonly supporting computer graphical user interfaces.

FIG. 4A is a logic diagram illustrating the comparison of new and old captured images by the presenter client, when full images are compared, and the transmission of the changed information to the conference server.

FIG. 4B is a logic diagram illustrating the comparison of new and old captured images by the presenter client, when a new block is compared with the corresponding block in an old full image, and the transmission of the changed information to the conference server.

FIG. 4C is a logic diagram illustrating the comparison of new and old captured images by the presenter client, when a checksum of a new block is compared with the checksum of an old corresponding block, and the transmission of the changed information to the conference server.

FIG. 4D is a data flow diagram illustrating the updating of the stored old image with a new captured block by the presenter client, and the transmission of the changed information to the conference server.

FIG. 4E is a logic diagram illustrating the updating of the stored old image in various formats with a new delta block by the presenter client, and the transmission of the changed information to the conference server.

FIG. 5 is a state diagram illustrating the operation of the image capture process of the presenter client software.

FIG. 6A is a diagram showing attendee client block clipping.

FIG. 6B is a diagram showing presenter client block clipping.

FIG. 7A is a diagram illustrating the client consistency setting.

FIG. 7B is a diagram illustrating the server consistency setting.

FIG. 8A is a block data flow diagram illustrating the operation of server processes monitoring and filtering a single presenter data stream according to the present invention.

FIG. 8B is a block data flow diagram illustrating the operation of server processes monitoring and filtering multiple input and output data streams according to the present invention.

FIG. 9A is a block diagram illustrating interconnections of several communications servers and communicant clients in a single communications session according to the present invention.

FIG. 9B is a block diagram illustrating interconnections of several communications servers and communicant clients, including migrated and recruited connections, in a single communications session according to the present invention.

FIG. 9C is a block diagram illustrating interconnections of several communications servers and communicant clients, including backup connections for clients and servers, in a single communications session according to the present invention.

FIG. 9D is a block diagram illustrating interconnections of several communications servers and communicant clients, including decomposition of transformation sequences and functional delegation, in a single communications session according to the present invention.

FIG. 9E is a block diagram illustrating interconnections of several communications servers and communicant clients, including distribution and parallelization of output queues and processing, in a single communications session according to the present invention.

FIG. 9F is a block diagram illustrating interconnections of several communications servers and communicant clients, including distribution and parallelization of output queue contents and processing, in a single communications session according to the present invention.

FIG. 9G is a block diagram illustrating interconnections of several communications servers and communicant clients, including multiple and redundant routing, in a single communications session according to the present invention.

FIG. 10A is a block diagram illustrating a multi-layered tree topology for connections of several communications servers with communicant clients in a single communications session according to the present invention.

FIG. 10B is a block diagram illustrating a single-layer tree topology for connections of several conference servers with communicant clients in a single communications session according to the present invention.

FIG. 11 is a diagram of the example architecture for a single server with a single meeting, according to the present invention.

FIG. 12 is a diagram of the example architecture for a server with several meetings running on a single CPU, according to the present invention.

FIG. 13 is a diagram of the example architecture for a single meeting manager directing several servers with several meetings, running on several CPUs, according to the present invention.

FIG. 14 is a diagram of the example architecture for several meeting managers directing several servers with several meetings, running on several CPUs, according to the present invention.

FIG. 15 is a diagram of the example architecture for several meeting managers directing several servers with several meetings, running on the same CPU, according to the present invention.

FIG. 16 is a diagram of the example architecture for a single server with a single meeting, but the meeting is controlled by several instances of a communications session server ("CSS") running on the same CPU, according to the present invention.

FIG. 17 is a diagram of the example architecture for a single meeting manager directing several servers with a single meeting where the meeting is controlled by several instances of a CSS running on the same CPU, with additional CSSs for the same meeting running on other CPUs, according to the present invention.

FIG. 18 is a diagram of the example architecture for several meeting managers directing several servers with a single meeting where the meeting is controlled by several instances of a CSS running on the same CPU, with additional CSSs for the same meeting running on other CPUs, according to the present invention.

FIG. 19 is a diagram of the example architecture for several meeting managers directing several servers with a single meeting where the meeting is controlled by several instances of a CSS running on different CPUs, according to the present invention.

FIG. 20 is a diagram of the example architecture for several meeting managers directing several servers with a single meeting where the meeting is controlled by several instances of a CSS running on different CPUs, according to the present invention. In this diagram, the propagation topology information is shown.

FIG. 21 is a diagram of the graph of the propagation topology information in FIG. 20.

FIG. 22 is a diagram of the graph of the propagation topology information in FIG. 20 together with the information for an additional propagation topology.

FIG. 23 is a time vs. space diagram showing some typical applications of the present invention.

DESCRIPTION OF THE PREFERRED EMBODIMENT

FIG. 1 shows a desktop conferencing system 10 according to one embodiment of the present invention. Desktop conferencing system 10 is shown with three attendee clients 18 and one presenter client 12. Following the arrows, presenter client 12 is connected to attendee client 18 through a conference server 14 and data network 16. The presenter and attendees may also participate in an ordinary telephone call or a conventional conference call using telephones 20 connected through conference switch 22. The voice conferencing might also be carried out through an audio connection on a data network; indeed, telephone network 24 and data network 16 could be the same.

The group of users can comprise from as few as one user, who might record a presentation or lecture or video-mail onto a session archive 23 for later distribution; or two people who wish to share information or collaborate on some work product; or a small group of users, as in a typical business conference call; to many tens of thousands or hundreds of thousands of users using the network as a broadcast rather than interactive medium. In the last case, the voice conferencing might also be broadcast, and could involve one-way telephone conference calls, radio, multicast network audio (such as MBone), or the like.

The presenter client conferencing software, which is usually distributed tightly bound with the attendee client software to facilitate presenter hand-offs from conferee to conferee, captures information (such as image, sound, or other output information) from a program or programs running on the presenter's machine and relays it to the server, as explained in more detail below. The server relays this information to all of the attendee client computers participating in the same session or conference, transforming (in manners and by methods described below) the data as required. A more detailed description of the operation of the system by way of the example of transporting a stream of shared-image data during a conferencing usage of the software now follows.

During a conferencing session, presenter client 12 takes periodic "snap-shots" of the application screen image contained within a rectangular boundary determined by the presenter, breaks the screen shot into smaller rectangular blocks, compares these blocks to information from a previous screen shot. A block that has changed is passed to conference server 14 after it has undergone possibly two transformations and received identification marking ("ID stamps"). The first transformation may form the difference, using a set difference, exclusive-or (XOR), or other difference method, of the new and old block in order to extract information on the changes only. The second transformation may compress the block using a publicly available compression algorithm such as JPEG (Joint Photographic Experts Group) or PNG (Portable Network Graphics), or licensed proprietary methods of compression. The need for the two transformations is determined by the system depending on such parameters as client characteristics, server and network loading, and user requests. The two transformations and possibly many others may be also performed by the server, another client, or another facility available on the network.

The presenter client identifies where the block is in the capture rectangle with a block-location ID stamp; it identifies the time with a time-stamp; it may also identify itself with an origin stamp, and provide other ID stamps as needed. In order to provide synchrony in the system, conference server 14 can issue time synchronization signals. The conference server may also add time-stamps on receipt of blocks, and will need to update time-stamps when a recorded or archived conference is played back.

The changed blocks, however transformed, with ID stamps, are held on the conference server until they have been sent to all attendee client computers 18, or it has been determined by flow control that there is no longer a need to hold them. Flow control between presenter client 12 and server 14 and between server 14 and attendee client 18 determines how often the attendee client receives information updating the image; this flow control, described in more detail below, depends on the characteristics and configurations of the clients, the server, and the network. Attendee client 18 can also send a command to conference server 14 to obtain the latest image change information.

Attendee client 18 uses whatever change information it receives to update its screen display of the shared image. The attendee client may need to decompress the changed block information and to composite differences with previously received image information. The reverse transformations of decompression and composition may instead be performed by other computers.

From time to time, attendee client 18 communicates the position of the attendee pointer (if the conferee has selected this option) to conference server 14, which distributes the pointer position and a chosen identifying icon to each of the other conferee clients, which may then add a representation of the icon or other identifying label at the position indicated by the pointer information to the shared image on the client's display. The purpose of these pointers and labels is to allow conferees to reference particular elements of the shared image, possibly while describing the elements to the other conferees over the audio conference session (via a telephone conference call or an Internet audio conferencing application, for example).

FIG. 2 is a flowchart showing the process of introducing a conferee client 17 (client 17 refers t a generic client which might also be a presenter client 12 or an attendee client 18) to a conference ongoing on server 14, assuming that the conference setup is performed via the WWW. First, the conferee locates a conference listing. This may be done by finding or being told a URL or using a locator service such as ULS.TM. or LDAP.TM.. The conferee also specifies an icon to be used as a pointer label. Then the conferee points a WWW browser to the conference listing, where the server offering this listing or an associated server validates the conferee and provides information that allows the attendee client conferencing software to start and to connect to conference server 14 itself, possibly after further validation. Other information may be passed to the conferee client at this time as well. The connection to a server can also be accomplished in different ways, such as using stored parameters that allow meetings to be resumed after the network connection is temporarily broken, or using client software having a hard-coded list of meetings. Once the attendee client software is running, it communicates commands and pointer icon position to conference server 14, and conference server 14 supplies an initial conference image and later screen updates to client 17 (which is initially an attendee client 18).

An attendee can become a presenter by sending the appropriate attendee-to-presenter command to conference server 14. In the simplest embodiment with a single presenter, a message is sent to the presenter's screen indicating that an attendee wishes to take the presenting role; if the current presenter approves, then the roles are exchanged. In more complex embodiments, there can be a presenter arbitration mechanism, or multiple presenters may be allowed. The ability for a presenter or an attendee to be involved in any particular conferencing session and the assignment of privileges in the conference can be controlled by requiring appropriate keys from the presenter and the attendees.

Referring back to FIG. 1, data network 16 can be provided by dial-up connections, local area networks (LANs), wide area networks (WANs), internets, intranets, the global Internet, or a combination of these or any other computer data communications links. In a specific embodiment, the conferee client computers are personal computers, workstations, or other computing hardware systems, running operating systems such as MacOS.TM., Windows.RTM. 3.1 or 3.11, Windows.RTM. 95, Windows.RTM. NT.TM., Unix.TM., OS/2.TM., NExTStep.TM., BeOS.TM., JavaOS.TM., or the like. There is no requirement that the operating systems or hardware of the conferee clients all be the same.

Conference server 14 matches the form of the image to the attendee clients before sending it. Most computer screen images are represented as a bitmap of pixels whose layout is device-dependent. To be able to send images from one device to another, a transformation or transcoding is often required. The captured image information may be transcoded into a device-independent format for transmission and transcoded back to a device-dependent format for each attendee's screen; if, however, the mix of attendee clients is such that a device-independent format is not needed, no conversion is done. If these or other transcoding operations are needed, they can be carried out at the presenter client's side, the server side, or the attendee client's side, depending on where excess capacity or superior capability exists. The choice of device-dependent vs. device-independent bitmaps (DDB vs. DIB) is made automatically by the server, in response to the number and type of conferee clients. For example, a meeting with just two conferees, each running a Windows.RTM. PC with similar 256-color graphics configurations, may use DDBs and achieve high performance with low overhead However, where differently configured clients are connected in a conference, server 14 transcodes the images to fit the attendee's screen capability.

Multiple codecs may be involved in the transcoding of screen formats as well as other image transformations described herein. It may even happen that different codecs will be used on different blocks in the same image, depending on availability of the codec's host computer, the transformation needs, the loading on client, server, and network, or other conditions relevant to the system's performance.

Server 14 also fits the images to the attendee's CPU capability. Server 14 can be a server operated by the presenter (who would then have full control over the server's resources), or it can be owned or operated by an unrelated third party or even an attendee who never presents. It is possible to have a third party whose involvement is solely as a facilitator of conferences. Operating server 14 is simpler than operating a videoconference hub, since the bandwidth is much lower. One aspect of the present invention is the realization that real-time conferencing can be had without resort to full-motion video transmission, as the item being monitored, a portion of a computer display, does not change as fast as a full-motion video. A similar observation applies to many other data communications streams.

Instead of full-motion video, attendees' screens are updated as the presenter's screen is modified, or less frequently for attendees with slow machines. The screen is modified from left-to-right within a row of blocks, and rows are updated top-to-bottom to improve the perception of low latency.

In some cases, server 14 might be operating without attendees. Such a configuration is useful where the presenter wishes to "record" a session for later playback. Even a session with attendees can be recorded for later playback, possibly including a recording of the voice conferencing. These stored sessions might be stored in session archive 23 or elsewhere. The shared image session can be synchronized with the voice conference by using the time stamps on the block data. When the recorded session is played back, it is an example of conference server 14 operating with attendees but no presenter.

The blocks may be held at the server as full images, as differences ("deltas") from previously received full images, as deltas from previous delta blocks, or as some combination of these, depending on the capabilities of the presenter and attendee clients. Periodically, a server may "checkpoint" the image deltas. To do this, the server requests a full image of a block from the presenter client and uses the full image as a replacement for all the accumulated image deltas for that block. The server might also request the entire captured region from the presenter client, or send the entire region to an attendee client upon request.

The conference server acts as a software-controlled switch that connects the presenter client with the attendee clients, taking into account that the speed of information transfer from the presenter client can change and the speed of transfer to the attendee clients can change and be simultaneously different for different attendees. The workload of the entire system is shared and distributed among the client and server computers, and even other computers that do not perform client or server functions.

Presenter Client Capture Operation

The capture operation and transport technology improves over former approaches by reducing the amount of work required and so enhances performance. In addition, the technique can be tuned to best suit the workload on the hardware and software platforms and network connections involved, manually or automatically. In particular, this tuning dynamically matches the capture operation to the amount of computer power available (while running the other software the conferee may wish to use) and the speed of connection to the network. Existing systems that capture graphics display commands, transmit them, then use them to recreate the original display appear to have great compression, which entails economy of network transmission. But comparison with the description below of the present invention will reveal that the savings are not so great when the task is to communicate data streams which can be updated by later transmissions.

The presenter selects an area of his or her computer display to be shared ("capture region"); it need not be a rectangular area. More than one capture region may be selected at a time and multiple regions may overlap. The selection may be made on a screen display, in a memory representation of a display, or in an aliased representation of either; the selection can be changed at any time. If the client has multiple monitors or multiple displays on a single monitor, independent selection can be made for each. A window provided by the presenter client computer's operating system, or by an application or other program, may be designated as the capture region, and then the capture region can be adjusted automatically if the window is moved or resized. This may be a fixed window, or the capture operation can be set to follow the selection of the current ("top" or "focus") window automatically. In a simple embodiment, the presenter selects a rectangular region on the screen ("capture rectangle"). For efficient transmission, the capture rectangle is broken up into rectangular subregions (blocks) to give good perception of response time. For example, if the presenter has selected all of an 800-by-600-pixel screen display to be within the capture rectangle, then it might be broken up into twelve 200-by-200-pixel square blocks. If the capture rectangle is later adjusted smaller, the blocks are changed to be made up of smaller rectangles, or the capture rectangle is divided into fewer blocks, or both; correspondingly, if the capture rectangle is later adjusted larger, the blocks are changed to be larger rectangles or the capture rectangle is divided into more blocks, or both. For efficient handling of blocks, the blocks are preferably kept between 1000 and 4000 pixels in size. As the blocks are updated on the attendee's screen, they are presented from the top row to the bottom row and from left to right within a row.

The presenter defines the shape of the capture region and can change, control, reposition, and resize it. In some computer systems, when the region is rectangular, the capture rectangle may be marked by a transparent window that stays visible; in other systems, it is appropriate to use four graphical objects that move together to mark the boundary of the capture rectangle.

FIG. 3 shows the display architecture of a typical computer shown with application programs 60(a)-(c), and graphic display mechanisms 62(a)-(c) with graphics commands capture points 64(a)-(c). The graphics display mechanisms 62 send their output to a screen image storage area 66 which in turn presents an image to the user on a computer display 68. Existing techniques of image sharing depend on intercepting graphics display commands (graphic instructions, display commands, graphics subsystem calls, etc.) at graphics commands capture points 64 and sending these commands to another computer which can use them to affect its own display. This appears to have an advantage in that one high-level graphics drawing command (e.g., "draw a blue line from co-ordinates (0,0) to (0,100)") can be expressed in fewer bits of data than what would be required to express the resulting set of pixels on the screen. In this case, 100 blue pixels, say using 24-bit color depth, would require 300 bytes of data, compared with a graphics drawing command that m