Scalably presenting a collection of media objects Number:7,131,059 from the United States Patent and Trademark Office (PTO) owispatent

Title: Scalably presenting a collection of media objects

Abstract: Systems and methods of presenting media objects are described. In one aspect, a group of media objects is selected from the collection based upon media object relevance to one or more data structures of a selected media file of indexed, temporally-ordered data structures. One or more of the selected media file and the media objects of the selected group are transmitted to a client for contemporaneous presentation at a selected summarization level. In another aspect, media objects in the collection are grouped into multiple clusters based upon one or more media object relevance criteria. The media object clusters are arranged into a hierarchy of two or more levels. A selected cluster is transmitted to a client for contemporaneous presentation at a selected summarization level.

Patent Number: 7,131,059 Issued on 10/31/2006 to Obrador

---

Inventors:	Obrador; Pere (Mountain View, CA)
Assignee:	Hewlett-Packard Development Company, L.P. (Houston, TX)
Appl. No.:	10/334,769
Filed:	December 31, 2002

---

Current U.S. Class:	715/501.1 ; 715/500.1
Current International Class:	G06F 15/00 (20060101)
Field of Search:	715/501.1,500.1

---

References Cited [Referenced By]

---

U.S. Patent Documents

	5521841		May 1996		Arman et al.
	5600775		February 1997		King et al.
	5655117		August 1997		Goldberg et al.
	5710591		January 1998		Bruno et al.
	5751289		May 1998		Myers
	5821945		October 1998		Yeo et al.
	5864366		January 1999		Yeo
	5905981		May 1999		Lawler
	5956026		September 1999		Ratakonda
	5983218		November 1999		Syeda-Mahmood
	5995095		November 1999		Ratakonda
	6041147		March 2000		Mead
	6044089		March 2000		Ferriere
	6044365		March 2000		Cannon et al.
	6144375		November 2000		Jain et al.
	6166735		December 2000		Dom et al.
	6199076		March 2001		Logan et al.
	6222532		April 2001		Ceccarelli
	6233367		May 2001		Craver et al.
	6236395		May 2001		Sezan et al.
	6268864		July 2001		Chen et al.
	6301586		October 2001		Yang et al.
	6317740		November 2001		Mukherjea et al.
	6332144		December 2001		deVries et al.
	6411724		June 2002		Vaithilingam et al.
	6426778		July 2002		Valdez, Jr. et al.
	6535889		March 2003		Headrick et al.
	6538665		March 2003		Crow et al.
	6570587		May 2003		Efrat et al.
	6571054		May 2003		Tonomura et al.
	6693652		February 2004		Barrus et al.
	6721741		April 2004		Eyal et al.
	6847977		January 2005		Abajian
	2001/0018693		August 2001		Jain et al.
	2002/0005866		January 2002		Gorham et al.
	2002/0033844		March 2002		Levy et al.
	2002/0069218		June 2002		Sull et al.
	2003/0009469		January 2003		Platt et al.
	2003/0078144		April 2003		Gehrke
	2004/0122731		June 2004		Mannik et al.

Foreign Patent Documents

	1213667		Jun., 2002		EP

Other References

Huang, et al., "Multimedia Search and Retrieval: New Concepts, System Implementation, and Application", IEEE Transactions on Circuits and Systems for Video Technology, vol. 10, No. 5, Aug. 2000, p. 679-692. cite- d by examiner . Benitez, et al., "Object-Based Multimedia Content Description Schemes and Applications for MPEG-7", Signal Processing: Image Communication, vol. 16, Sep. 2000, p. 235-269. cited by examiner . F. C. Li, A. Gupta, E. Sanocki, L. wei He, and Y. Rui, "Browsing digital video." In Proceedings of the SIGCHI conference on human factors in computing systems, pp. 169-176. cited by other . O'Connor et al., "News story segmentation in the fischlar video indexing system," ICIP 2001 Conf. Proceedings, vol. 1 3 of 3 CONF 8,7, pp. 418-421 (Oct. 2001). cited by other . Jan R. Erickson, "Jan's Visit to CHINA," <http://home.sprintmail.com/.about.janerickson/JansChinaWebPage/JansCh- inaWebPage.sub.--Pictures.html>, 9 pages (Oct. 2000). cited by other.

Primary Examiner: Herndon; Heather R.
Assistant Examiner: Rutledge; Amelia

---

Claims

---

What is claimed is:

1. A computer-implemented method of scalably presenting a collection of media objects including a video file that contains a sequence of full-motion video frames and an audio sequence synchronized with the video frames, and media objects that are free of any content contained in the video file, the method comprising: selecting from the collection one or more of the media objects that are free of any content contained in the video file based upon one or more relevance criteria relating the selected media objects to one or more of the video frames of the video file; generating a hierarchical sequence of summarization levels each comprising a respective representation of a different respective selection of temporally-ordered content of the video file and associated content of at least one of the selected media objects linked into the respective representation of the temporally-ordered content of the video file, the summarization levels in the hierarchical sequence being ordered in accordance with their different resource requirements, wherein each of the summarization levels above a lowest summarization level in the hierarchical sequence accommodates more resource constraints than a respective immediately preceding one of the summarization levels in the hierarchical sequence; and in response to a request from a client to browse the collection of media objects, selecting one of the generated summarization levels based upon one or more identified resource constraints, and transmitting to the client the respective representation of the respective selection of temporally-ordered content of the video file and the associated content of the at least one of the selected media objects of the selected summarization level.

2. The method of claim 1, wherein the selecting of one of the summarization levels is based upon at least one resource constraint selected from a communication channel resource constraint and a client resource constraint.

3. The method of claim 2, wherein the communication channel resource constraint corresponds to one or more of a bandwidth constraint, a transmission constraint, and a client reception constraint.

4. The method of claim 2, wherein the client resource constraint corresponds to at least one of a processing constraint and a display constraint.

5. The method of claim 2, wherein the client resource constraint corresponds to a preference indication received from the client.

6. The method of claim 1, wherein the selecting of media objects from the collection is based at least in part on existence of browsable links between the selected media objects and respective ones of the video frames of the video file.

7. The method of claim 6, wherein the transmitting comprises transmitting in accordance with the lowest summarization level all of the video frames of the video file in sequence and transmitting each of the selected media objects in timed relationship with a respective one of the video frames of the video file linked to the respective media objects.

8. The method of claim 7, wherein the transmitting in accordance with the lowest summarization level comprises synchronously transmitting the full-motion video frames and the audio sequence compressed to a first audio compression level.

9. The method of claim 7, further comprising transmitting in accordance with a second summarization level a first subset of the video frames of the video file and each of the selected media objects in timed relationship with respective ones of the video frames in the first subset.

10. The method of claim 9, wherein the transmitting in accordance with the second summarization level comprises synchronously transmitting key sequences of full-motion video frames and the audio sequence compressed to a second audio compression level higher than the first audio compression level.

11. The method of claim 9, further comprising transmitting in accordance with a third summarization level a second subset of video frames of the video file smaller than the first subset and graphical representations of each of the selected media objects in timed relationship with respective ones of the video frames in the second subset.

12. The method of claim 11, wherein ones of the video frames of the video file are designated as key frames, and the transmitting in accordance with the third summarization level comprises synchronously transmitting the designated key frames and the audio sequence compressed to a third audio compression level higher than the second audio compression level.

13. The method of claim 11, further comprising transmitting in accordance with a fourth summarization level a single key frame of the video file.

14. The method of claim 13, wherein the transmitting in accordance with the fourth summarization level comprises synchronously transmitting the single key frame and one or more key sentences that are representative of the audio sequence and are compressed to a fourth audio compression level higher than the third audio compression level.

15. The method of claim 1, wherein the selecting of media objects from the collection is based at least in part on a selected context similarity between the selected media objects and respective ones of the video frames of the video file.

16. The method of claim 15, wherein the selecting of media objects from the collection is based at least in part on at least one context similarity selected from color similarity, motion activity similarity, texture content similarity, audio similarity, data similarity, name similarity, inclusion of similar objects, inclusion of similar persons, and inclusion of similar places.

17. The method of claim 15, wherein the transmitting comprises transmitting in accordance with the lowest summarization level all of the video frames of the video file in sequence and transmitting each of the selected media objects.

18. The method of claim 17, further comprising transmitting in accordance with a second summarization level a first subset of the video frames of the video file and each of the selected media objects.

19. The method of claim 18, further comprising transmitting in accordance with a third summarization level a second subset of the video frames of the video file smaller than the first subset and graphical representations of each of the selected media objects.

20. The method of claim 19, further comprising transmitting in accordance with a fourth summarization level a single video frame representative of the selected video file.

21. The method of claim 1, wherein the selecting of one of the summarization levels is based upon a time constraint specifying a time limit for presenting the video file and the selected media objects.

22. The method of claim 21, wherein the transmitting comprises transmitting representative portions of the video file selected to accommodate the time constraint.

23. The method of claim 1, wherein the selecting of one of the summarization levels is based upon a time constraint specifying a time limit for presenting the video file and the selected media objects and a resource constraint specifying at least one parameter affecting an ability of the client to present ones of the media objects in the collection.

24. A system for presenting a collection of media objects including a video file that contains a sequence of full-motion video frames and an audio sequence synchronized with the video frames, and media objects that are free of any content contained in the video file, the system comprising a media manager operable to perform operations comprising: selecting from the collection ones of the media objects that are free of any content contained in the video file based upon one or more relevance criteria relating the selected media objects to one or more of the video frames of the video file; generating a hierarchical sequence of summarization levels each comprising a respective representation of a different respective selection of temporally-ordered content of the video file and associated content of at least one of the selected media objects linked into the respective representation of the temporally-ordered content of the video file, the summarization levels in the hierarchical sequence being ordered in accordance with their different resource requirements, wherein each of the summarization levels above a lowest summarization level in the hierarchical sequence accommodates more resource constraints than a respective immediately preceding one of the summarization levels in the hierarchical sequence; and in response to a request from a client to browse the collection of media objects, selecting one of the generated summarization levels based upon one or more identified resource constraints, and transmitting to the client content the respective representation of the respective selection of temporally-ordered of the video file and the associated content of the at least one of the selected media objects of the selected summarization level.

25. The system of claim 24, wherein the selecting of media objects from the collection is based at least in part on existence of browsable links between the selected media objects and respective ones of the video frames of the video file.

26. The system of claim 24, wherein the selecting of media objects from the collection is based at least in part on a selected context similarity between the selected media objects and respective ones of the video frames of the video file.

27. A computer-implemented method of scalably presenting a collection of media objects, comprising: grouping respective ones of the media objects in the collection into multiple media object clusters based upon one or more relevance criteria interrelating respective ones of the media objects in each of the media object clusters; selecting one of the media object clusters that includes at least two media objects of different content type; generating from the selected media object cluster a respective hierarchy of two or more summarization levels each comprising a -respective representation of a different respective selection of content of each of the media objects in the selected cluster, the summarization levels in the hierarchy being ordered in accordance with their different resource requirements. wherein each of the summarization levels above a lowest summarization level in the hierarchy accommodates more resource constraints than a respective immediately preceding one of the summarization levels in the hierarchy; and in response to a request from a client to browse the collection of media objects, selecting one of the generated summarization levels based upon one or more identified resource constraints, and transmitting to the client the respective representation of the respective selection of content of the selected summarization level.

28. A system for scalably presenting a collection of media objects, comprising a media manager operable to perform operations comprising: grouping selected ones of the media objects in the collection into multiple media object clusters based upon one or more relevance criteria interrelating respective ones of the media objects in each of the media object clusters; selecting one of the media object clusters that includes at least two media objects of different content type; generating from the selected media object cluster a respective hierarchy of two or more summarization levels each comprising a respective representation of a different respective selection of content of each of the media objects in the selected cluster, the summarization levels in the hierarchy being ordered in accordance with their different resource requirements, wherein each of the summarization levels above a lowest summarization level in the hierarchy accommodates more resource constraints than a respective immediately preceding one of the summarization levels in the hierarchy; and in response to a request from a client to browse the collection of media objects, selecting one of the generated summarization levels based upon one or more identified resource constraints, and transmitting to the client the respective representation of the respective selection of content of the selected summarization level.

---

Description

---

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is related to the following applications, each of which is incorporated herein by reference: U.S. application Ser. No. 10/117,033, filed Apr. 5, 2002, by Pere Obrador, and entitled "Media Object Management;" and U.S. application Ser. No. 10/207,279, filed Sep. 29, 2002, by Pere Obrador, and entitled "Presenting a Collection of Media Objects."

TECHNICAL FIELD

This invention relates to systems and methods of presenting media objects.

BACKGROUND

Individuals and organizations are rapidly accumulating large collections of digital content, including text, audio, graphics, animated graphics and full-motion video. This content may be presented individually or combined in a wide variety of different forms, including documents, presentations, music, still photographs, commercial videos, home movies, and meta data describing one or more associated digital content files. As these collections grow in number and diversity, individuals and organizations increasingly will require systems and methods for organizing and browsing the digital content in their collections. To meet this need, a variety of different systems and methods for browsing selected kinds of digital content have been proposed.

For example, storyboard browsing has been developed for browsing full-motion video content. In accordance with this technique, video information is condensed into meaningful representative snapshots and corresponding audio content. One known video browser of this type divides a video sequence into equal length segments and denotes the first frame of each segment as its key frame. Another known video browser of this type stacks every frame of the sequence and provides the user with information regarding the camera and object motions.

Content-based video browsing techniques also have been proposed. In these techniques, a long video sequence typically is classified into story units based on video content. In some approaches, scene change detection (also called temporal segmentation of video) is used to give an indication of when a new shot starts and ends. Scene change detection algorithms, such as scene transition detection algorithms based on DCT (Discrete Cosine Transform) coefficients of an encoded image, and algorithms that are configured to identify both abrupt and gradual scene transitions using the DCT coefficients of an encoded video sequence are known in the art.

In one video browsing approach, Rframes (representative frames) are used to organize the visual contents of video clips. Rframes may be grouped according to various criteria to aid the user in identifying the desired material. In this approach, the user may select a key frame, and the system then uses various criteria to search for similar key frames and present them to the user as a group. The user may search representative frames from the groups, rather than the complete set of key frames, to identify scenes of interest. Language-based models have been used to match incoming video sequences with the expected grammatical elements of a news broadcast. In addition, a priori models of the expected content of a video clip have been used to parse the clip.

In another approach, U.S. Pat. No. 5,821,945 has proposed a technique for extracting a hierarchical decomposition of a complex video selection for video browsing purposes. This technique combines visual and temporal information to capture the important relations within a scene and between scenes in a video, thus allowing the analysis of the underlying story structure with no a priori knowledge of the content. A general model of a hierarchical scene transition graph is applied to an implementation for browsing. Video shots are first identified and a collection of key frames is used to represent each video segment. These collections then are classified according to gross visual information. A platform is built on which the video is presented as directed graphs to the user, with each category of video shots represented by a node and each edge denoting a temporal relationship between categories. The analysis and processing of video is carried out directly on the compressed videos.

A variety of different techniques that allow media files to be searched through associated annotations also have been proposed. For example, U.S. Pat. No. 6,332,144 has proposed a technique in accordance with which audio/video media is processed to generate annotations that are stored in an index server. A user may browse through a collection of audio/video media by submitting queries to the index server, In response to such queries, the index server transmits to a librarian client each matching annotation and a media identification number associated with each matching annotation. The librarian client transmits to the user the URL (uniform resource locator) of the digital representation from which each matching annotation was generated and an object identification number associated with each matching annotation. The URL may specify the location of all or a portion of a media file.

Methods for transmitting video information over a network have been proposed. For example, in some approaches, an entire video shot (i.e., a single, complete sequence of video images) may be downloaded by a client for browsing. In another approach, one or more static images representative of a shot may be downloaded by a client for browsing. U.S. Pat. No. 5,864,366 has proposed a video information transmission scheme in accordance with which similar video shots of a video file (e.g., a commercial or a news broadcast) are grouped into one or more collections and a subset of frames from each shot is selected for transmission to a user over a network. The selected subset of frames may be downloaded by a user so that each collection may be displayed at a client terminal at the same time. In this way, a dynamic summary of the video information represented by the collections may be presented to a user while satisfying network bandwidth requirements.

SUMMARY

The invention features methods of scalably presenting a collection of media objects.

In one aspect, a group of media objects is selected from the collection based upon media object relevance to one or more data structures of a selected media file of indexed, temporally-ordered data structures. One or more of the selected media file and the media objects of the selected group are transmitted to a client for contemporaneous presentation at a selected summarization level.

In another aspect, media objects in the collection are grouped into multiple clusters based upon one or more media object relevance criteria. The media object clusters are arranged into a hierarchy of two or more levels. A selected cluster is transmitted to a client for contemporaneous presentation at a selected summarization level.

The invention also features systems for implementing the above-described scalable media object presentation methods.

Other features and advantages of the invention will become apparent from the following description, including the drawings and the claims.

DESCRIPTION OF DRAWINGS

FIG. 1 is a diagrammatic view of a media management node coupled directly to a set of local media files and coupled indirectly to multiple sets of remote media files over a local area network and a global network infrastructure.

FIG. 2 is a diagrammatic view of a computer system that is programmable to implement a method of managing media objects.

FIG. 3 is a diagrammatic perspective view of a media file of indexed, temporally-ordered data structures and an automatically-generated selection of key data structures.

FIG. 4 is a diagrammatic perspective view of the media file of FIG. 3 after the selection of key data structures has been modified by a user.

FIG. 5 is a diagrammatic perspective view of an indexed media file containing a sequence of full-motion video frames, a selection of key frames, and a high resolution still photograph.

FIG. 6 is a diagrammatic perspective view of the indexed media file, key frame selection and high resolution still photograph of FIG. 5, along with multiple user-selected media objects that are linked to respective video frames of the indexed media file.

FIG. 7A is a diagrammatic perspective view of the links connecting the key frames, the high resolution still photograph, and the media objects to the indexed media file of FIG. 6.

FIG. 7B is a diagrammatic perspective view of a database storing the indexed media file, key frames, high resolution still photograph, media objects and connecting links of FIG. 7A.

FIG. 8A is a diagrammatic perspective view of a video file mapped into a set of video sequences.

FIG. 8B is a diagrammatic perspective view of a set of video sequences mapped into a common video file.

FIG. 8C is a diagrammatic perspective view of a set of consecutive video sequences mapped into two video files.

FIG. 8D is a diagrammatic perspective view of a set of non-consecutive video sequences mapped into two video files.

FIG. 9 is a flow diagram of a method of presenting a collection of media objects.

FIGS. 10A 10C are diagrammatic views of a multimedia album page during presentation of a video file and a selected group of media objects that are linked to the video frames of the video file.

FIG. 11 is a diagrammatic view of a graphical user interface simultaneously presenting a selected video file and a group of media objects selected based upon a context similarity criterion.

FIG. 12 is a diagrammatic view of a graphical user interface presenting the media objects of FIG. 11 as a video card stack adjacent to the selected video file.

FIG. 13 is a hierarchical arrangement of media objects that are grouped into clusters based upon one or more media object relevance criteria.

FIG. 14 is a flow diagram of an embodiment of a method of scalably presenting a collection of media objects.

FIG. 15A is a diagrammatic view of a selected video file and multiple media objects linked to respective frames of the selected video file.

FIGS. 15B 15E are diagrammatic views of the selected video file of FIG. 15A presented at different levels of summarization.

FIG. 16A is a diagrammatic view of a selected video file and multiple representative portions of the selected video file, including video shots, key frames, and key sequences.

FIGS. 16B 16D are diagrammatic views of the selected video file of FIG. 16A presented at different levels of summarization.

FIG. 17 is a diagrammatic view of an exemplary decision table for selecting a presentation summarization level based on a time constraint and a resource constraint.

DETAILED DESCRIPTION

In the following description, like reference numbers are used to identify like elements. Furthermore, the drawings are intended to illustrate major features of exemplary embodiments in a diagrammatic manner. The drawings are not intended to depict every feature of actual embodiments nor relative dimensions of the depicted elements, and are not drawn to scale.

Media Management System Overview

Referring to FIG. 1, in one embodiment, a media management node 10 includes a media manager 12 that is configured to enable all forms of digital content in a selected collection of media objects to be organized into a browsable context-sensitive, temporally-referenced media database. As used herein, the term "media object" refers broadly to any form of digital content, including text, audio, graphics, animated graphics and full-motion video. This content may be packaged and presented individually or in some combination in a wide variety of different forms, including documents, annotations, presentations, music, still photographs, commercial videos, home movies, and meta data describing one or more associated digital content files. The media objects may be stored physically in a local database 14 of media management node 10 or in one or more remote databases 16, 18 that may be accessed over a local area network 20 and a global communication network 22, respectively. Some media objects also may be stored in a remote database 24 that is accessible over a peer-to-peer network connection. In some embodiments, digital content may be compressed using a compression format that is selected based upon digital content type (e.g., an MP3 or a WMA compression format for audio works, and an MPEG or a motion JPEG compression format for audio/video works). The requested digital content may be formatted in accordance with a user-specified transmission format. For example, the requested digital content may be transmitted to the user in a format that is suitable for rendering by a computer, a wireless device, or a voice device. In addition, the requested digital content may be transmitted to the user as a complete file or in a streaming file format.

A user may interact with media manager 12 locally, at media management node 10, or remotely, over local area network 20 or global communication network 22. Transmissions between media manager 12, the user, and the content providers may be conducted in accordance with one or more conventional secure transmission protocols. For example, each digital work transmission may involve packaging the digital work and any associated meta-data into an encrypted transfer file that may be transmitted securely from one entity to another.

Global communication network 22 may include a number of different computing platforms and transport facilities, including a voice network, a wireless network, and a computer network. Media object requests may be transmitted, and media object replies may be presented in a number of different media formats, such as voice, Internet, e-mail and wireless formats. In this way, users may access the services provided by media management node 10 and the remote media objects 16 provided by service provider 26 and peer-to-peer node 24 using any one of a wide variety of different communication devices. For example, in one illustrative implementation, a wireless device (e.g., a wireless personal digital assistant (PDA)) may connect to media management node 10, service provider 26, and peer-to-peer node 24 over a wireless network. Communications from the wireless device may be in accordance with the Wireless Application Protocol (WAP). A wireless gateway converts the WAP communications into HTTP messages that may be processed by service provider 10. In another illustrative implementation, a voice device (e.g., a conventional telephone) may connect to media management node 10, service provider 26 and peer-to-peer node 24 over a voice network. Communications from the voice device may be in the form of conventional analog or audio signals, or they may be formatted as VoxML messages. A voice gateway may use speech-to-text technology to convert the audio signals into HTTP messages; VoxML messages may be converted to HTTP messages based upon an extensible style language (XSL) style specification. The voice gateway also may be configured to receive real time audio messages that may be passed directly to the voice device. Alternatively, the voice gateway may be configured to convert formatted messages (e.g., VoxML, XML, WML, e-mail) into a real time audio format (e.g., using text-to-speech technology) before the messages are passed to the voice device. In a third illustrative implementation, a software program operating at a client personal computer (PC) may access the services of media management node 10 and the media objects provided by service provider 26 and peer-to-peer node 24 over the Internet.

As explained in detail below, in some embodiments, media manager 12 enables a user to organize and browse through a selected collection of media objects by means of a set of links between media objects. In general, all media objects may be indexed by any other media object in the selected collection. Each link may be browsed from one media object to a linked media object, and vice versa. The set of links between media objects may be generated by a user, a third party, or automatically by media manager 12. These links are stored separately from the media objects in one or more media object linkage data structures that are accessible by the media manager 12.

Media manager 12 may provide access to a selected digital content collection in a variety of different ways. In one embodiment, a user may organize and browse through a personal collection of a diverse variety of interlinked media objects. In another embodiment, media manager 12 may operate an Internet web site that may be accessed by a conventional web browser application program executing on a user's computer system. The web site may present a collection of personal digital content, commercial digital content and/or publicly available digital content. The web site also may provide additional information in the form of media objects that are linked to the available digital content. Users may specify links to be generated and browse through the collection of digital content using media objects as links into and out of specific digital content files. In an alternative embodiment, a traditional brick-and-mortar retail establishment (e.g., a bookstore or a music store) may contain one or more kiosks (or content preview stations). The kiosks may be configured to communicate with media manager 12 (e.g., over a network communication channel) to provide user access to digital content that may be rendered at the kiosk or transferred to a user's portable media device for later playback. A kiosk may include a computer system with a graphical user interface that enables users to establish links and navigate through a collection of digital content that is stored locally at the retail establishment or that is stored remotely and is retrievable over a network communication channel. A kiosk also may include a cable port that a user may connect to a portable media device for downloading selected digital content.

In embodiments in which a user interacts remotely with media manager 12, the user may store the media object linkage data structures that are generated during a session in a portable storage device or on a selected network storage location that is accessible over a network connection.

Referring to FIG. 2, in one embodiment, media manager 12 may be implemented as one or more respective software modules operating on a computer 30. Computer 30 includes a processing unit 32, a system memory 34, and a system bus 36 that couples processing unit 32 to the various components of computer 30. Processing unit 32 may include one or more processors, each of which may be in the form of any one of various commercially available processors. System memory 34 may include a read only memory (ROM) that stores a basic input/output system (BIOS) containing start-up routines for computer 30 and a random access memory (RAM). System bus 36 may be a memory bus, a peripheral bus or a local bus, and may be compatible with any of a variety of bus protocols, including PCI, VESA, Microchannel, ISA, and EISA. Computer 30 also includes a persistent storage memory 38 (e.g., a hard drive, a floppy drive 126, a CD ROM drive, magnetic tape drives, flash memory devices, and digital video disks) that is connected to system bus 36 and contains one or more computer-readable media disks that provide non-volatile or persistent storage for data, data structures and computer-executable instructions. A user may interact (e.g., enter commands or data) with computer 30 using one or more input devices 40 (e.g., a keyboard, a computer mouse, a microphone, joystick, and touch pad). Information may be presented through a graphical user interface (GUI) that is displayed to the user on a display monitor 42, which is controlled by a display controller 44. Computer 30 also may include peripheral output devices, such as speakers and a printer. One or more remote computers may be connected to computer 30 through a network interface card (NIC) 46.

As shown in FIG. 2, system memory 34 also stores media manager 12, a GUI driver 48, and one or more media object linkage structures 50. Media manager 12 interfaces with the GUI driver 48 and the user input 40 to control the creation of the media object linkage data structures 50. Media manager 12 also interfaces with the GUI driver 48 and the media object linkage data structures to control the media object browsing experience presented to the user on display monitor 42. The media objects in the collection to be linked and browsed may be stored locally in persistent storage memory 38 or stored remotely and accessed through NIC 46, or both.

Linking Media Objects

Referring to FIG. 3, in one embodiment, media manager 12 may be configured to automatically generate a selection of key data structures 60, 62, 64 from a media file 66 of indexed, temporally-ordered data structures. Media file 66 may correspond to any kind of digital content that is indexed and temporally-ordered (i.e., ordered for playback in a specific time sequence), including frames of a full-motion video, animated graphics, slides (e.g., PowerPoint.RTM. slides, text slides, and image slides) organized into a slideshow presentation, and segments of audio. Key data structures 60 64 may be extracted in accordance with any one of a variety of conventional automatic key data structure extraction techniques (e.g., automatic key frame extraction techniques used for full-motion video). Media manager 12 also may be configured to link meta data 70 with the first data structure 68 of media file 66. In this embodiment, each of the media file data structures is associated with an index value (e.g., a frame number or time-stamp number for full-motion video). Each of the links between media objects 60 64, 70 and media file data structures 68 is a pointer between the index value associated with the media file data structure 68 and the address of one of the linked media objects 60 64, 70. Each link is browsable from a given data structure of media file 66 to a media object 60 64, 70, and vice versa. The links may be stored in one or more media object data structures in, for example, an XML (Extensible Markup Language) format.

As shown in FIG. 4, in one embodiment, media manager 12 is configured to modify the initial selection of key data structures in response to user input. For example, in the illustrated embodiment, a user may remove key data structure 64 and add a new key data structure 72. In addition, a user may change the data structure of media file 66 to which key data structure 62 is linked. In some embodiments, a user may view a presentation of media file 66 in an editor program designed to allow the user to pause the media file presentation and specify links between selected data structures of media file 66 and one or more media objects.

Referring to FIGS. 5 and 6, in one illustrative embodiment, media file 66 corresponds to a video file sequence 73 of full-motion video frames 74. After automatic key frame extraction and user-modification, two key frames 76, 78 and a high resolution still photograph 80 are linked to video file 73. As shown in FIG. 6, in addition to modifying the selection of key frames 76 80, a user may link other media objects to the video frames 74. For example, the user may link a text file annotation 82 to video file 73. The user also may link an XHTML (Extensible HyperText Markup Language) document 84 to the video frame corresponding to key frame 78. XHTML document 84 may include a hypertext link 86 that contains the URL (Uniform Resource Locator) for another media object (e.g., a web page). The user also may link an audio file 88 to a video frame of video file 73. In the illustrated embodiment, for example, the linked audio file 88 may correspond to the song being played by a person appearing in the linked video frame. The user also may link a full-motion video file 90 to a frame 92 of video file 73. In the illustrated embodiment, for example, the linked video file 90 may correspond to a video of a person appearing in the associated video frame 92. The user also may link to the high resolution still image 80 a text file 94 containing meta data relating to the associated still image 80. For example, in the illustrated embodiment, meta data file 94 may correspond to the meta data that was automatically generated by the video camera that captured the high-resolution still image.

Referring to FIGS. 7A and 7B, in one embodiment, after video file 73 has been enriched with links to other media objects, the resulting collection of media objects and media object linkage data structures (shown in FIG. 7A) may be stored as a context-sensitive, temporally-referenced media database 96 (shown in FIG. 7B). This database 96 preserves temporal relationships and associations between media objects. The database 96 may be browsed in a rich and meaningful way that allows target contents to be found rapidly and efficiently from associational links that may evolve over time. All media objects linked to the video file 73 may share annotations and links with other media objects. In this way, new or forgotten associations may be discovered while browsing through the collection of media objects.

Referring to FIGS. 8A 8D, in some embodiments, all media files in a selected collection are stored only once in data base 96 (FIG. 7B). Each media file (e.g., video file 73) of indexed, temporally-ordered data structures may be split logically into a set of data structure sequences that are indexed with logical links into the corresponding media file. Media objects 98 may be indexed with logical links into the set of data structure sequences, as shown in FIG. 8A. Each data structure sequence link into a media file may identify a starting point in the media file and the length of the corresponding sequence. The data structure sequences may be consecutive, as shown in FIG. 8B, or non-consecutive. In addition, the set of data structure sequences may map consecutively into multiple media files, as shown in FIG. 8C. Alternatively, the set of data structure sequences may be mapped non-consecutively into multiple media files, as shown in FIG. 8D.

Presenting and Browsing a Collection a Media Objects

Referring to FIG. 9, in some embodiments, media manager 12 may be configured to present a collection of media objects to a user as follows. A group of media objects is selected from the collection based upon their relevance to one or more data structures of a selected media file of indexed, temporally-ordered data structures (step 100). As explained in detail below, the relevance criterion for selecting media objects may relate to browsable links between media objects and the selected media file, or the relevance criteria may relate to a selected context similarity between media objects and the selected media file. Next, the media file and the media objects of the selected group are presented contemporaneously (i.e., during a common period of time) to a user for browsing (step 102). The media file and the media objects preferably are presented to the user through a multimedia album page, which is a windows-based GUI that is displayed on display monitor 42 (FIG. 2).

Browsing Media Object Links

In some embodiments, the inter-media-object linking architecture described above may be used by media manager 12 to present the media objects of a collection to a user in a context-sensitive, temporally-referenced way through a multimedia album page 104, which is anchored to a selected media file (e.g., video file 73). For example, media manager 12 may present the data structures of the media file in sequence and, during this presentation, media manager 12 may present media objects in the selected group at times when they are relevant to the data structure (or data structure sequence) currently being presented to the user. In these embodiments, the relevance criterion for selecting the group of objects contemporaneously with the media file relates to the existence of browsable links between the selected media objects and data structures of the media file. The browsable links may be established automatically by media manager 12 and modified by a user, as described above. The browsable links also may be established by media manager 12 just before (e.g., during a preprocessing period) or on the fly during the presentation of the media file by instantiating one or more context matching processes (e.g., a face recognition process, a voice recognition process, or other pattern matching process) between the data structures of the media file being presented and the other media objects in the collection.

Referring to FIG. 10A, in one illustrative example, the media file of indexed, temporally-ordered data structures being presented by media manager 12 corresponds to video file sequence 73. Initially, media manager 12 may present the video file 73 to a user by displaying the first key frame 76 of the video file sequence. In addition, media manager 12 may simultaneously present the text file annotation 82, which is linked directly to key frame 76. Text file annotation 82 may be presented only during the presentation of key frame 76, or it may be presented during the presentation of key frame 76 and for some prescribed time after key frame 76 has been presented. Depending on the size of the display area and the resources of the presentation computer, the text file annotation presentation period may be shorter than the presentation period of the entire video file sequence 73 or it may last throughout the entire video file presentation period. In the illustrated embodiment, text file annotation file 82 is presented only when key frame 76 is presented.

Referring to FIG. 10B, media manager 12 presents the frames of video file 73 to a user in sequence. When one or more media object links to a video frame are detected, media manager 12 presents to the user the linked media objects along with the corresponding linked video frame. In the illustrated example, when video frame 78 is displayed, media manager 12 also displays the text file annotation 84, which is linked directly to video frame 78. A user may jump from multimedia album page 104 to another page (e.g., another multimedia album page, a web page, or some other media object) by selecting hypertext link 86 that is associated with text file annotation 84.

Media manager 12 also may determine that another media object 80 in the collection is relevant to the context of video frame 78. For example, media manager 12 may instantiate a face recognition process to find a match between the person displayed in video frame 78 and the person displayed in media object 80. In the illustrated example, the relevant media object identified by media manager 12 corresponds to high resolution still photograph 80; in other embodiments, however, the identified media object may correspond to a different media object. Depending on the size of the display area and the resources of the presentation computer, media manager 12 may display the identified media object 80 only as long as it is relevant to the context currently being presented (e.g., as long as the same person appears in the presented video frames) or media manager 12 may display the media object 80 for a longer period. In addition to displaying media object 80, media manager 12 displays graphical representations of links to media objects that are linked to media object 80. In the illustrated embodiment, media manager 12 displays a link 106 to a another page for displaying a larger version of the high resolution still photograph, a link 108 to an Internet web page corresponding to the home page of the person displayed in the high resolution photograph, and a link 110 to an audio file relating to the person displayed in the high resolution photograph (e.g., a voice greeting recorded by the displayed person). A user may select one or more of the hypertext links 106 110 to jump from multimedia album page 104 to the corresponding linked media objects.

As shown in FIG. 10C, in the illustrated example, when the context changes, media manager 12 discontinues the presentation of media objects 80, 84. As mentioned above, in other implementations, media manager 12 may present multiple linked media objects at the same time during the presentation of video file 73, depending on the size of the display area and the resources of the presentation computer. If resources are constrained, media manager 12 may remove linked objects to free resources in FIFO (First In, First Out) order. When a new media object link is detected or when a new relevant media object is identified, media manager 12 presents a new media object 90 in the multimedia album page 104. In the illustrated example, media object 90 corresponds to a video file that is represented by a key frame 112 showing a person with a face that matches the person being displayed in a frame 92 of video file 73. Media manager 12 also displays graphical representations of links to media objects that are linked to media object 90. In the illustrated example, media manager 12 displays a link 114 to a multimedia album page corresponding to media object 90, a link 116 to a window for presenting the associated video file, and a link 118 to a high resolution still image of the person displayed in key frame 112. A user may select one or more of the hypertext links 114 118 to jump from multimedia album page 104 to the corresponding linked media objects.

Thus, in these embodiments, during presentation of a selected media file of indexed, temporally-ordered data structures a user may navigate through a collection of associated media objects by activating the links that are displayed to the user in a way that is context-sensitive and temporally-referenced to the playback of the selected media file of indexed, temporally-ordered data structures. In this way, new or forgotten associations may be discovered while browsing through the collection of media objects.

Browsing a Media Object Cluster Hierarchy

In some embodiments, the relevance criteria used to select the media objects that will be presented contemporaneously with the selected media file may relate to a selected context similarity between media objects and the selected media file. The context similarity may correspond to low-level features (e.g., motion activity, texture or color content, and audio content) or high-level features (e.g., meta data, such as keywords and names; objects, such as persons, places and structures; and time-related information, such as playback length and media object creation date). One or more known media object processing techniques (e.g., pattern recognition techniques, voice recognition techniques, color histogram-based techniques, and automatic pan/zoom motion characterization processing techniques) may be used to compare media objects to the selected media file in accordance with the selected context similarity criterion. In response to a user's indication to organize media objects in accordance with a selected context similarity criterion, media manager 12 selects a group of media objects having the highest context similarity to a selected media file. Media manager 12 then contemporaneously presents the selected media file and the media objects of the selected group.

As shown in FIG. 11, in these embodiments, the media objects of the selected group may be presented simultaneously around a selected media file 120. Alternatively, as. shown in FIG. 12, the media objects of the selected group may be presented in the form of a media object stack 122. Individual media objects within the media object stack may be selected for