Interaction between objects and a virtual environment display Number:7,394,459 from the United States Patent and Trademark Office (PTO) owispatent

Title: Interaction between objects and a virtual environment display

Abstract: An interactive table has a display surface on which a physical object is disposed. A camera within the interactive table responds to infrared (IR) light reflected from the physical object enabling a location of the physical object on the display surface to be determined, so that the physical object appear part of a virtual environment displayed thereon. The physical object can be passive or active. An active object performs an active function, e.g., it can be self-propelled to move about on the display surface, or emit light or sound, or vibrate. The active object can be controlled by a user or the processor. The interactive table can project an image through a physical object on the display surface so the image appears part of the object. A virtual entity is preferably displayed at a position (and a size) to avoid visually interference with any physical object on the display surface.

Patent Number: 7,394,459 Issued on 07/01/2008 to Bathiche,   et al.

---

Inventors:	Bathiche; Steven (Bellevue, WA), Kurlander; David Joshua (Seattle, WA), Wilson; Andrew D. (Seattle, WA), Chen; Christina Summer (Redmond, WA), Dehlin; Joel P. (Redmond, WA)
Assignee:	Microsoft Corporation (Redmond, WA)
Appl. No.:	10/834,675
Filed:	April 29, 2004

---

Current U.S. Class:	345/175 ; 345/156; 345/173; 715/700; 715/702; 715/866
Current International Class:	G06F 3/042 (20060101); G06F 3/01 (20060101)
Field of Search:	345/156,175,632,633,168,173 715/700,702,866 463/9,14,30,31,36,37,38,39

---

References Cited [Referenced By]

---

U.S. Patent Documents

4992650	February 1991	Somerville
5436639	July 1995	Arai et al.
5483261	January 1996	Yasutake
6128003	October 2000	Smith et al.
6266061	July 2001	Doi
6522395	February 2003	Barnji
6529183	March 2003	MacLean et al.
6614422	September 2003	Rafii et al.
6710770	March 2004	Tomasi et al.
6720949	April 2004	Pryor et al.
7084859	August 2006	Pryor
2001/0012001	August 2001	Rekimoto
2003/0161524	August 2003	King
2004/0090524	May 2004	Belliveau
2004/0196371	October 2004	Kono
2005/0226505	October 2005	Wilson
2005/0277071	December 2005	Yee
2005/0281475	December 2005	Wilson
2006/0010400	January 2006	Dehlin et al.

Foreign Patent Documents

	0690407		Jan., 1996		EP
	27656077		May., 1998		FR
	9819292		May., 1998		WO

Other References

Shen et al., "DiamondSpin: an extensible toolkit for around-the-table interaction", Apr. 24-29, 2004, Proceedings of the SIGCHI conference on Human factors in computing systems CHI '04, vol. 6, No. 1, pp. 167-174. cited by examiner . "3.6 Interpolation in Two or More Dimensions." Numerical Recipes in C: The Art of Scientific Computing. Chapter 3. Interpolation and Extrapolation. .COPYRGT. 1988-1992, Cambridge University Press. Numerical Recipes Software. pp. 123-128. cited by other . "Bar Code 1,2-Dimensional Bar Code Page." Avaliable http://www.adams1.com/pub/russadam/stack.html. Printed Jan. 20, 2004. 14pp. cited by other . Ambiente article. "InteracTable.RTM.." Dated Jul. 7, 2000. Available http://www.darmstadt.gmd.de/ambiente/activities/interactable.html. Printed Nov. 21, 2003. 3 pp. cited by other . Bier, Stone, Pier, Buston, and DeRose. "Toolglass and Magic Lenses: The See-Through Interface." Proceedings of Siggraph '93 (Anaheim, August). Computer Graphics Annual Conference Series, ACM, 1993, pp. 73-80. 8pp. cited by other . Blickenstorfer, Conrad H. "First Look: Acer TravelMate TM100 with Windows XP Tablet PC Edition." Pen Computing Magazine. Jul. 2002. pp. 44-47. cited by other . "DiamondSpin--Begs for Direct Manipulation Technology Is it the Pen? Sony Leaps Out-of-the-Box Again with Gummi." Mitsubishi/DiamondSpin. CHI 2004 #3. Printed Apr. 30, 2004. 5 pp. cited by other . Dietz and Leigh. "DiamondTouch: A Multi-User Touch Technology." UIST '01 Orlando FLA. .RTM. ACM 2001 1-58113-438 -x/01/11. CHI Letters 3 (2). Nov. 11-14, 2001. pp. 219-226. cited by other . Electronic Check Alliance Processing, Inc. "Gift Cards, How Stored Value Card Systems Work." Available. http://www.electron-cap.com/GiftCards.htm. Printed Jan. 20, 2004 and May 16, 2004. .RTM.2003. 2pp. total (3pp printed). cited by other . Fukuchi and Rekimoto. "Interaction Techniques for SmartSkin." ACM UIST2002 demonstration, 2002. 2pp. cited by other . Grabowski, Robert. "A Miniature Video Laser Range Finder for Small Robots." Available http://www.andrew.cmu.edu/.about.rjg/research/research.sub.--hardware/las- er.sub.--rangefinder.html. Printed May 16, 2004. 8pp. cited by other . Grant and Winograd. "Flexible, Collaborative Organization on a Tabletop." ACM CSCW 2002: Workshop on Co-located Tabletop Collaboration: Technologies and Directions. New Orleans, LA. Nov. 2002. pp. 1-4. cited by other . Horn, Berthold K. P. "Robot Vision." The MIT Press. Available http://mitpress.mit.edu/catalog/item/default.asp?ttype=2&tid=8388. Printed Dec. 17, 2003. 6pp. total. cited by other . Horn, Berthold Klaus Paul. "Robot Vision." Binary Images: Topological Properties. The MIT Electrical Engineering and Computer Science Series. 1986. pp. 66-71 and cover page(s). cited by other . Hunter, Andrew. "Connected Components Analysis (Computer Vision)." www.google.com search results http://www.google.com/search?sourceid=navclient&q=connected+component+ell- ipse. Printed Mar. 7, 2004. Article dated Sep. 24, 2002. 2pp search results, 21pp article. cited by other . "IR Distance Sensor." Available http://www.diyelectronics.com/Accessories/IRDS.html (2pp) and http://www.diyelectronics.com/Accessories/GP2D05.html (1pg). Printed Dec. 30, 2003. 3pp. cited by other . Ishii and Ullmer. "Tangible Bits: Towards Seamless Interfaces between People, Bits and Atoms." Proceedings of CHI '97, Mar. 22-27, 1997, Atlanta, Georgia. .COPYRGT. 1997 ACM 0-89791-802-9/97/03. pp. 1-8. cited by other . Ishii, Wisneski, Orbanes, Chun, and Paradiso. "PingPongPlus: Design of an Athletic-Tangible Interface for Computer-Supported Cooperative Play." Proceeding of CHI '99, May 15-20, 1999, .COPYRGT. 1999 ACM. pp. 1-8. cited by other . Johanson, Kolodny, and Russell. "A Hand pose and Position Tracker for the Interactive Table." CS223B Final Project. Available http://graphics.stanford.edu/.about.drussel/vision/tracker-report.html. Printed Dec. 16, 2003, 6pp. cited by other . Ju, Hurwitz, Judd, and Lee. "CounterActive: An Interactive Cookbook for the Kitchen Counter."Extended Abstracts of CHI 2001, Seattle. Apr. 2001. pp. 269-270. cited by other . Kang, Sing Bing. "Radial Distortion Snakes." IAPR Workshop on Machine Vision Applications (MVA2000), Tokyo, Japan. Nov. 2000. pp. 603-606. cited by other . Kato, Billinghurst, Poupyrev, Imamoto, and Tachibana. "Virtual Object Manipulation on a Table-Top AR Environment" IEEE and ACM Int'l Symposium on Augmented Reality 2000, ISAR '2000, Oct. 5-6, 2000, Munich. 9pp. cited by other . Klemmer, Newman, and Sapien. "The Designer's Outpost: A Task-Centered Tangible Interface for Web Site Information Design." Proceedings of Human Factors in Computing Systems: CHI 2000 Extended Abstracts. The Hague, The Netherlands. Apr. 1-6, 2000. pp. 333-334. cited by other . Klemmer, Newman, Farrell, Bilezikjian, and Landay. "The Designers' Outpost: A Tangible Interface for Collaborative Web Site Design." CHI Letters, The 14.sup.th Annual Symposium on User Interface Soft Technology: UIST 2001. 3(2). pp. 1-10. cited by other . Kobayashi, Hirano, Narita, and Ishii. "A Tangible Interface for IP Network Simulation." CHI 2003, Apr. 5-10, 2003, Ft. Lauderdale, FL ACM 1-58113-630-7/03/0004. 2pp. cited by other . Koike, Sato, and Kobayashi. "Integrating Paper and Digital Information on EnhancedDesk: A Method for Realtime Finger Tracking on an Augmented Desk System." ACM Transaction on Computer-Human Interaction, vol. 8 No. 4, Dec. 2001. .COPYRGT. 2001 ACM 1073-0516/01/1200-0307. pp. 307-322. cited by other . Leibe, Starner, Ribarsky, Wartell, Krum, Singletary, and Hodges. "The Perceptive workbench: Toward Spontaneous and Natural Interaction In Semi-Immersive Virtual Environments." Proceedings of the IEEE Virtual Reality 2000 Conference, Mar. 18-22, 2000. New Brunswick, New Jersey: IEEE Computer Society, 2000. 8pp. cited by other . Leigh and Dietz. "DiamondTouch Characteristics and Capabilities." Mitsubishi Electric Research Laboratories, Cambridge, Massachusetts, USA. Undated. 2pp. cited by other . Magerkurth, Stenzel, and Prante. "STARS--A Ubiquitous Computing Platform for Computer Augmented Tabletop Games." 5.sup.th International Conference on Ubiquitous Computing (Ubicomp '03), Oct. 12-15, 2003, Seattle, Washington. 2pp. cited by other . Malandain, Gregoire. "Connected Components Extraction." Available http://www-sop.inria.fr/epidaure/personnel/malandain/segment/connexe.html- . Printed Dec. 18, 2003. 3 pp. cited by other . Matsushita and Rekimoto. "HoloWall: Designing a Finger, Hand, Body, and Object Sensitive Wall." UIST '97 Banff, Alberta, Canada. .COPYRGT. 1997 ACM 0-89791-881-9/97/10. pp. 209-210. cited by other . Missouri Department Of Revenue. "2D Barcode Technology." Undated. 3pp. cited by other . Moran, Saund, van Melle, Gujar, Fishkin, and Harrison. "Design and Technology for Collaborage; Collaborative Collages of Information on Physical Walls." UIST '99. Asheville, NC. .COPYRGT. 1999 ACM 1-58113-075-9/99/11, CHI Letters vol. 1, 1. pp. 197-206. cited by other . Pangaro, Maynes-Aminzade, and Ishii. "The Actuated Workbench: Computer-Controlled Actuation in Tabletop Tangible Interfaces." Proceedings of UIST 2002, Oct. 27-30, 2002. .COPYRGT. 2002 ACM. 10pp. cited by other . Paradiso, Hsiao, Strickon, Lifton, and Adler. "Sensor systems for interactive surfaces." IBM Systems Journal, vol. 39, Nos. 3&4, 2000. pp. 892-914. cited by other . Patten, Ishii, Hines, and Pangaro. "Sensetable: A Wireless Object Tracking Platform for Tangible User Interfaces." Proceedings of CHI 2001, Mar. 31-Apr. 5, 2001, ACM Press, .COPYRGT. 2001 ACM. 8pp. cited by other . Patten, Recht, and Ishii. "Audiopad: A Tag-based Interface for Musical Performance." Proceedings of Conference on New Interface for Musical Expression (NIME '02). Dublin, Ireland, May 24-26, 2002. 6pp. cited by other . Ramos and Balakrishnan. "Fluid Interaction Techniques for the Control and Annotation of Digital Video." UIST '03 Vancouver, B.C., Canada. .COPYRGT. 2003 ACM 1-58113-636-06/03/0010. pp. 105-114. cited by other . Rekimoto and Ayatsuka. "CyberCode: Designing Augmented Reality Environments with Visual Tags." Proc. of UIST 2000, 2000. 10pp. cited by other . Rekimoto and Matsushita. "Perceptual Surfaces: Towards a Human and Object Sensitive Interactive Display." Proceedings of Workshop on Perceptural User Interactes (PUI'97), 1997, 3pp. cited by other . Rekimoto and Nagao. "The World through the Computer: Computer Augmented Interaction with Real World Environments." Proceedings of UIST'95, 1995. pp. 29-36. cited by other . Rekimoto and Saitoh. "Augmented Surfaces: A Spatially Continuous Work Space for Hybrid Computing Environments." CHI '99, May 15-20, 1999. Pittsburgh, Pennsylvania. .COPYRGT. ACM 1999 0-201-48559-1/99/05. pp. 378-385. cited by other . Rekimoto, Jun. "Matrix: A Realtime Object Identification and Registration Method for Augmented Reality." Proc. of Asia Pacific Computer Human Interaction (APCHI '98), 1998. 6pp. cited by other . Rekimoto, Jun. "Multiple-Computer User Interfaces: `Beyond the Desktop` Direct Manipulation Environments." ACI CHI2000 Video Proceedings, 2000. 2pp. cited by other . Rekimoto, Jun. "Pick-and-Drop: A Direct Manipulation Technique for Multiple Computer Environments." Proceedings of UIST'97, 1997. pp. 31-39. cited by other . Rekimoto, Jun. "SmartSkin: An Infrastructure for Freehand Manipulation on Interactive Surfaces." CHI 2002, Apr. 20-25, 2002, Minneapolis, Minnesota. .COPYRGT. 2001 ACM 1-58113-453-3/02/0004. 8 pp. cited by other . Rekimoto, Ullmer, and Oba. "DataTiles: A Modular Platform for Mixed Physical and Graphical Interactions." SIGCHI'01, Mar. 31-Apr. 4, 2001, Seattle, WA. .COPYRGT. 2001 ACM 1-58113-327-8/01/0003. 8pp. cited by other . Reznik, Canny, and Alldrin. "Leaving on a Plane Jet." 2001 Int. Conf. on Intell Robots & Systems (IROS), Maui, Hawaii, Oct. 2001. 6pp. cited by other . Ringel, Gerh, Jin, and Winograd. "Barehands: Implement-Free Interaction with a Wall-Mounted Display." Short Talks. CHI 2001 Mar. 31-Apr. 5. pp. 367-368. cited by other . Rosenfeld, Zawadzki, Sudol, and Perlin. "Planar Manipulator Display." New York University mrl. NYU Media Research Lab. Available http://cat.nyu.edu/PMD. Printed May 16, 2004. 3pp. cited by other . Rovani, David (Posted by). "My Second Month with the Compaq Tablet." Home>Reviews, TablePCHome.com--Table PC user community, Posted Apr. 10, 2003. Available http://www.tabletpchome.com/Messages.aspx?ThreadID=140. Printed Dec. 30, 2003. pp. 1-2 of 5. cited by other . Schmalstieg, Encarnacao, and Szalavari. "Using Transparent Props for Interaction With The Virtual Table." Presented at 1999 ACM Symposium on Interactive 3D Graphics (13DG '99). Apr. 26-28, 1999, Atlanta, GA. 7pp. cited by other . Scott, Grant, and Mandryk. "System Guidelines for Co-located collaborative Work on a Tabletop Display." Proceedings of ECSCW'03, European Conference Computer-Supported Cooperative Work 2003Helsinki, Finland, Sep. 14-18, 2003. 20pp. cited by other . Shen, Everitt, and Ryall. "UbiTable: Impromptu Face-to-Face Collaboration on Horizontal Interactive Surfaces." .COPYRGT. Mitsubishi Electric Research Laboratories, Inc., 2003. Cambridge, Massachusetts. TR-2003-49. Sep. 2003. 10pp. cited by other . Shen, Lesh, and Vernier. "Personal Digital Historian Story Sharing Around the Table." Interactions. Mar. + Apr. 2003. pp. 15-22. cited by other . Shen, Lesh, Bernier, Forlines, and Frost. "Sharing and Building Digital Group Histories." CSCW'02, Nov. 16-20, 2002, New Orleans, Louisiana. .COPYRGT. 2002 ACM 1-58113-560-2/02/0011. 10pp. cited by other . Shen, Lesh, Moghaddam, Beardsley, and Bardsley. "Personal Digital Historian: User Interface Design." .COPYRGT. Mitsubishi Electric Research Laboratories, Inc. 2001. Cambridge, Massachusetts. 5pp. cited by other . Smart Technologies Inc. White Paper. "DViT Digital Vision Touch Technology." Feb. 2003. 10pp. cited by other . Smart Technologies, Inc. "Rear Projection Smart Board.TM. Interactive Whiteboard" "Smart Board Software". Available http://www.smarttech.com/Products/rearprojection/index.asp and http://www.smarttech.com/Products/sbsoftware/index.asp. Printed Dec. 16, 2003. 5pp. total. cited by other . Starner, Leibe, Singletary, Lyons, Gandy, and Pair. "Towards Augmented Reality Gaming." Available http://www.gvu.gatech.edu/ccg/publications/imagina2000/. Printed Dec. 30, 2003. 27pp. cited by other . Streitz, Gei.beta.ler, Holmer, Konomi, Muller-Tomfelde, Reischl, Rexrogh, Seitz, and Steinmetz. "i-LAND: An interactive Landscape for Creativity and Innovation."Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI'99), Pittsburgh, Pennsylvania, May 15-20, 1999. ACM Press, New York. pp. 120-127. cited by other . Symanzik, Jurgen. "Three-Dimensional Statistical Graphics Based On Interactively Animated Anaglyphs." Published 1993. Available http://citeseer.mj.nec.com/95667.html. Printed Feb. 25, 2004. 7pp. total. cited by other . "The Tablet PC A detailed look at Microsoft's proposed Tablet PC." Pen Computing Magazine: Tablet PC. Available http://www.pencomputing.com/frames/textblock.sub.--tablet.sub.--pc.html. Printed Dec. 30, 2003. pp. 1. cited by other . Tandler, Prante, Muller-Tomfelde, Streitz, and Steinmetz. "ConnecTables: Dynamic Coupling of Displays for the Flexible Creation of Shared Workspaces." Proceedings of the 14. Annual ACM Symposium on User Interface Software and Technoic (USIT'01), ACM Press (CHI Letters 3(2)), 2001, pp. 11-20 (10pp). cited by other . Ullmer and Ishii. "The metaDESK: Models and Prototypes for Tangible User Interfaces." Proceedings of UIST'97, Oct. 14-17, 1997. .COPYRGT. 1997 ACM--ACM 0-89791-881-9/97/10. 10pp. cited by other . Ullmer, Ishii, and Glas. "mediaBlocks: Physical Containers, Transports, and Controls for Online Media." Computer Graphics Proceedings (SIGGRAPH'98), Jul. 19-24, 1998, .COPYRGT. 1998 ACM. ACM-0-89791-999-8-8/98/007. 8pp. cited by other . Ullmer, Ishii, and Jacob. "Tangible query Interfaces: Physically Constrained Tokens for Manipulating Database Queries." Proc. Interact 2003 Conference, 2003. 11pp. cited by other . Underkoffler and Ishii. "Illuminating Light: An Optical Design Tool with a Luminous-Tangible Interface." Proceeding of CHI '98, Apr. 18-23, 1998, .COPYRGT. 1998 ACM. pp. 1-8. cited by other . Underkoffler and Ishii. "Urp: A Luminous-Tangible Workbench for Urban Planning and Design." Proceedings of CHI '99. May 15-20, 1999. .COPYRGT. 1998 ACM. pp. 1-8. cited by other . Underkoffler, Ullmer, and Ishii. "Emancipated Pixels: Real-World Graphics in the Luminous Room." Proceedings of SIGGRAPH '99, Aug. 8-13, 1999, .COPYRGT. 1999 ACM. 8pp. cited by other . Vernier, Lesh, and Shen. "Visualization Techniques for Circular Tabletop Interfaces." To appear in Advanced Visual Interfaces, May 2002, Trento, Italy. .COPYRGT. 2002 Mitsubishi Electric Research Laboratories, Inc. MERL-TR2002-01. Authored Mar. 2002. 10pp. cited by other . Viola and Jones. "Robust Real-time Object Detection." Cambridge Research Laboratory, Technical Report Series. Compaq. CRL 2001/01, Feb. 2001. 30pp. cited by other . "VirtualBoard." Available http://visilab.unime.it/visilab/virtualboard.htm. Printed Dec. 16, 2003. 6pp. cited by other . Von Hardenberg and Berard. Bare-Hand Human-Computer Interaction. PUI 2001 Orlando, FL, .COPYRGT. 2001 ACM 1-58113-448-7-11/14/01. 8pp. cited by other . Wellner, Pierre. "Interacting with Paper on the DigitalDesk." Communications of the ACM. Jul. 1993. EuroPARC tech report EPC-93-195. 17pp. cited by other . Whalen, Tara. "Playing well with Others: Applying Board Game Design to Tabletop Display Interfaces." UIST 2003. Vancouver, Nov. 2-5, 2003, 3pp. cited by other . Wu and Balakrishnan. "Multi-Finger and Whole Hand Gestural Interaction Techniques for Multi-User Tabletop Displays." UIST '03, Vancouver, B.C., Canada. .COPYRGT. 2003 ACM 1-58113-636-6/03-0010. pp. 193-202. cited by other . Hardenberg Von. Christian, "Fingertracking and Handposture Recognition for Real-Tim Human-Computer Interaction", Berlin, Jul. 20, 2001. cited by other . Kijma, Ryugo, et al. "Distributed Display Approach Using PHMD with Infrared Camera", Proceedings of the IEEE Virtual Rality, Orlando, Florida, Mar. 24-28, 2002. pp. 1-8. cited by other . Pavlidis, Ioannis, et al. "A Vehicle Occupant Counting System Based on Near-Infrared Phenomenology and Fuzzy Neural Classification", IEEE Transactions on Intelligent Transportation System, New York, NY, vol. 1, No. 2., Jun. 2000, pp. 78-80. cited by other . Office Action mailed Aug. 30, 2007 cited in related U.S. Appl. No. 10/870,777. cited by other . Office Action mailed Oct. 9, 2007 cited in related U.S. Appl. No. 10/867,434. cited by other.

Primary Examiner: Tran; My-Chau T
Attorney, Agent or Firm: Workman Nydegger

---

Claims

---

The invention in which an exclusive right is claimed is defined by the following:

1. A method for enabling interaction between a virtual environment and a physical object, comprising the steps of: (a) detecting the physical object and its location when the physical object is placed atop a display surface on which the virtual environment is displayed, said physical object being detected on the display surface by optically sensing light reflected from the physical object from below the display surface; (b) within the virtual environment, producing a response to the physical object as if the physical object were included in the virtual environment, enabling an interaction between the physical object and the virtual environment in two different modes, depending upon a type of the physical object placed atop the display surface, said two different modes including: (i) a first mode in which the virtual environment responds to a presence of the physical object on the display surface; and (ii) a second mode in which the physical object that is placed atop the display surface comprises an active object that is capable of performing an active function perceptible to a user, automatically in response to the virtual environment, and wherein the active object, which is placed atop the display surface, interacts with the virtual environment in the second mode by performing said active function atop the display surface automatically in response to the virtual environment; (c) tracking movement of the physical object relative to the display surface by sensing changes in location of the physical object on the display surface; and (d) changing a value of at least one parameter in the virtual environment in response to the movement of the physical object relative to the display surface.

2. The method of claim 1, wherein the step of producing the response in the first mode comprises the step of causing a virtual object in the virtual environment to interact with the physical object.

3. The method of claim 2, wherein the interaction of the virtual object with the physical object comprises the step of automatically preventing the virtual object from moving through a region in the virtual environment where the physical object is located atop the display surface.

4. The method of claim 1, further comprising the step of detecting at least one of a size and a shape of the physical object in the first mode, to enable the predefined response to the physical object within the virtual environment.

5. The method of claim 1, further comprising the step of identifying the physical object based upon the light reflected from the physical object, said light that is reflected being encoded with a predefined pattern that is optically detected below the display surface.

6. The method of claim 5, wherein the physical object to which the virtual environment responds in the first mode comprises a passive object that does not perform any active function.

7. The method of claim 5, wherein the physical object is an active object that actively performs an active function in connection with the virtual environment and the active function is perceptible to the user, said active object and the virtual environment interacting in either or both the first mode and the second mode.

8. The method of claim 7, further comprising the step of producing a signal in the virtual environment in the second mode, said signal being detected by the active object, causing the active object to perform the active function.

9. The method of claim 7, wherein the active function comprises the step of producing one of: (a) a visually perceptible state of the active object; (b) an audibly perceptible state of the active object; (c) a movement of the active object over the display surface; and (d) a vibration of at least a portion of the active object.

10. The method of claim 7, wherein the active object is mobile and moves itself over the display surface to interact with the virtual environment.

11. The method of claim 7, further comprising the step of enabling the active function of the active object to be controlled by the user.

12. The method of claim 7, further comprising the step of enabling the active function performed by the active object to be controlled by a software program that produces the virtual environment.

13. The method of claim 7, wherein electrical power is provided to the active object by one of a battery and a charged capacitor to enable the active object to perform the active function.

14. The method of claim 1, further comprising the step of displaying a projected image limited to the location of the physical object and visible through the physical object at the location of the physical object on the display surface so that the projected image visible through the physical object appears to be part of the physical object, but is part of an overall image of the virtual environment that is displayed on the display surface.

15. The method of claim 1, further comprising the steps of: (a) displaying a virtual entity, wherein the virtual entity comprises one of a virtual object and information on the display surface; and (b) displaying the virtual entity at a position selected to avoid a visual interference between the virtual object and the physical object disposed on the display surface.

16. The method of claim 15, further comprising the step of providing a plurality of prioritized positions at which the virtual entity can be selectively displayed to avoid the interference with the physical object, a position having a higher priority being selected for display of the virtual entity to avoid the interference with the physical entity.

17. The method of claim 15, further comprising the step of determining a position of a specific user who will be visually perceiving the virtual entity, so that the virtual entity is oriented and displayed at a position automatically selected so as to facilitate the specific user readily visually perceiving the virtual entity, while also avoiding visually interfering with the physical object.

18. The method of claim 15, further comprising the step of reducing a size of the virtual entity to avoid interference with the physical object at the position where the virtual entity is displayed on the display surface.

19. The method of claim 15, further comprising the step of reformatting a shape of the virtual entity to avoid interference with the physical object at the position where the virtual entity is displayed on the display surface.

20. A memory medium on which are stored machine executable instructions for carrying out the steps of claim 1.

21. A method as recited in claim 1, wherein said active function comprises vibrating.

22. A method as recited in claim 1, wherein said active function comprises generating sound.

23. A method as recited in claim 1, wherein said active function comprises emitting light.

24. A method for enabling interaction between a virtual environment and a physical object, comprising the steps of: (a) detecting the physical object and its location when the physical object is placed atop a display surface on which the virtual environment is displayed, said physical object being detected on the display surface by optically sensing light reflected from the physical object from below the display surface; (b) within the virtual environment, producing a response to the physical object as if the physical object were included in the virtual environment, enabling an interaction between the physical object and the virtual environment in two different modes, depending upon a type of the physical object placed atop the display surface, said two different modes including: (i) a first mode in which the virtual environment responds to a presence of the physical object on the display surface; and (ii) a second mode in which the physical object that is placed atop the display surface comprises an active object that is capable of performing an active function perceptible to a user, automatically in response to the virtual environment, and wherein the active object, which is placed atop the display surface, interacts with the virtual environment in the second mode by performing said active function atop the display surface automatically in response to the virtual environment; (c) displaying a virtual entity, wherein the virtual entity comprises one of a virtual object and information on the display surface; and (d) displaying the virtual entity at a position selected to avoid a visual interference between the virtual object and the physical object disposed on the display surface.

25. The method of claim 24, further comprising a step of providing a plurality of prioritized positions at which the virtual entity can be selectively displayed to avoid the interference with the physical object, a position having a higher priority being selected for display of the virtual entity to avoid the interference with the physical entity.

26. The method of claim 24, further comprising a step of determining a position of a specific user who will be visually perceiving the virtual entity, so that the virtual entity is oriented and displayed at a position automatically selected so as to facilitate the specific user readily visually perceiving the virtual entity, while also avoiding visually interfering with the physical object.

27. The method of claim 24, further comprising a step of reducing a size of the virtual entity to avoid interference with the physical object at the position where the virtual entity is displayed on the display surface.

28. The method of claim 24, further comprising a step of reformatting a shape of the virtual entity to avoid interference with the physical object at the position where the virtual entity is displayed on the display surface.

29. A memory medium on which are stored machine executable instructions for carrying out the steps of claim 24.

30. A method for enabling interaction between a virtual environment and a physical object, comprising the steps of: (a) detecting the physical object and its location when the physical object is placed atop a display surface on which the virtual environment is displayed, said physical object being detected on the display surface by optically sensing light reflected from the physical object from below the display surface; (b) within the virtual environment, producing a response to the physical object as if the physical object were included in the virtual environment, enabling an interaction between the physical object and the virtual environment in two different modes, depending upon a type of the physical object placed atop the display surface, said two different modes including: (i) a first mode in which the virtual environment responds to a presence of the physical object on the display surface; and (ii) a second mode in which the physical object that is placed atop the display surface comprises an active object that is capable of performing an active function perceptible to a user, automatically in response to the virtual environment, wherein the active function comprises at least one of vibrating, generating sound or emitting light, and wherein the active object, which is placed atop the display surface, interacts with the virtual environment in the second mode by performing said active function atop the display surface automatically in response to the virtual environment, the method further including: (c) tracking movement of the physical object relative to the display surface by sensing changes in location of the physical object on the display surface; and (d) changing a value of at least one parameter in the virtual environment in response to the movement of the physical object relative to the display surface.

31. A method as recited in claim 1, wherein said active function comprises vibrating.

32. A method as recited in claim 1, wherein said active function comprises generating sound.

33. A method as recited in claim 1, wherein said active function comprises emitting light.

34. A memory medium on which are stored machine executable instructions for carrying out the steps of claim 30.

---

Description

---

FIELD OF THE INVENTION

This invention generally pertains to a method and apparatus wherein an interaction occurs between a virtual environment displayed on a surface and objects placed on the display surface, and more specifically, pertains to facilitating an interaction between graphic images displayed on a display surface and passive and/or active objects that are placed on or positioned near the display surface.

BACKGROUND OF THE INVENTION

Virtual environments are typically employed in electronic games that are played on game consoles, personal computers, and other types of computing devices, as well as in other types of applications. A user usually interacts with objects in a virtual environment by manipulating a mouse, joystick, wheel, game pad, track ball, or other user input device that causes the virtual object to move in a specific manner or carry out some other action or function as defined by the software program that produces the virtual environment. The effects of a user interaction with an object in the virtual environment are generally visible on a display. For example, a user might be controlling a virtual object such as a spaceship or race car that is displayed in the virtual environment, so that the virtual object carries out some task. If the virtual object being controlled by the user strikes another virtual object, it may be "destroyed," as shown by graphic images indicating the destruction on the display. In certain games, the game console or computer may control other virtual objects in the environment, either in support of the user's task, or to oppose its completion. Thus, the interaction between virtual objects in a virtual environment is well known.

Another form of user input employs displays that are responsive to the touch of a user's finger or a stylus. Touch responsive displays can be pressure activated, responsive to electrical capacitance, changes in magnetic field intensity, employ surface acoustic waves, or respond to other variables that indicate the location of a finger or stylus on the display. Another type of touch sensitive display includes a plurality of optical sensors spaced-apart around the periphery of the display screen so that the location of a finger or stylus touching the screen can be detected. Using one of these touch sensitive displays, a user can more directly control a virtual object that is displayed. For example, the user may touch the displayed virtual object to select it and then drag the object to a new position on the touch-sensitive display.

However, in most such touch-sensitive displays, the response is only to the touch of the finger or stylus at a point. There is another type of interaction with a virtual environment that might provide a much richer experience for the user. While virtual environments, such as games, often include virtual objects that are displayed on a screen, it would be desirable for the virtual environment to also respond to physical objects that are placed on the display surface. In most prior art touch-sensitive displays, the finger or stylus is not treated as a physical object that is actually within the virtual environment, but instead, is simply an alternative type of pointing device used to make selections or drag elements about on the screen. To be truly interactive to physical objects that are placed on it, a display surface should also be able to detect where a plurality of physical objects are placed on it, as well as detect different types of physical objects, each of which might provide a different interactive experience for the user. However, the capacitive, electromagnetic, optical, or other types of sensors used in conventional touch-sensitive displays typically cannot simultaneously detect the location of more than one finger or object touching the display screen at a time, and thus, would be unable to detect the location or each different type of a plurality of different types of physical objects placed thereon. These prior art touch-sensing systems are generally incapable of detecting more than a point of contact and are unable to detect the shape of an object proximate to or touching the display surface. Even capacitive or resistive, or acoustic surface wave sensing display surfaces that can detect multiple points of contact are unable to image objects on a display surface to any reasonable degree of resolution. Prior art systems of these types cannot detect patterns on an object or detailed shapes that might be used to identify each object among a plurality of different objects that are placed on a display surface.

Another approach that has been developed in the prior art uses cameras mounted to the side and above a horizontal display screen to visually capture an image of a user's finger or other objects that are touching the display screen. This multiple camera mounting configuration is clearly not a compact system that most people would want to use in a residential setting. In addition, the accuracy of this type of multi-camera system in responding to an object that is on or proximate to the display surface depends upon the capability of the software used with the system to visually recognize objects and their location in three-dimensional space. Furthermore, the view of one object by one of the cameras may be blocked by an intervening object.

To address many of the problems inherent in the types of touch-sensitive displays discussed above, a user interface platform was developed in the MIT Media Lab, as reported by Brygg Ullmer and Hiroshi Ishii in "The metaDESK: Models and Prototypes for Tangible User Interfaces," Proceedings of UIST 10/1997:14-17. The metaDESK includes a near-horizontal graphical surface used to display two dimensional geographical information. Above the graphical surface is disposed an arm-mounted flat-panel display that serves as an "active lens" for use in displaying three dimensional geographical information. A computer vision system inside the desk unit (i.e., below the graphical surface) includes infrared (IR) lamps, an IR camera, a video camera, a video projector, and mirrors. The mirrors reflect the graphical image projected by the projector onto the underside of the graphical display surface. The IR camera can detect a distinctive pattern provided on the undersurface of passive objects called "phicons" that are placed on the graphical surface. Magnetic-field position sensors and electrical-contact sensors are also included in the metaDESK. For example, in response to the IR camera detecting the IR pattern (which is transparent to visible light) applied to the bottom of a "Great Dome phicon," a map of the MIT campus is displayed on the graphical surface, with the actual location of the Great Dome in the map positioned where the Great Dome phicon is located. Moving the Great Dome phicon over the graphical surface manipulates the displayed map by rotating or translating the map in correspondence to the movement of the phicon by a user. This paper notes that all of the objects or phicons used with the graphical display surface are passive, so that the display system simply responds to a phicon by producing a specific graphic image, but the phicon does not itself respond to the virtual environment.

A similar approach to sensing objects on a display surface is disclosed in several papers published by Jun Rekimoto of Sony Computer Science Laboratory, Inc. in collaboration with others. These papers briefly describe a "HoloWall" and a "HoloTable," both of which use IR light to detect objects that are proximate to or in contact with a display surface on which a rear-projected image is visible. The rear-projection panel, which is vertical in the HoloWall and horizontal in the HoloTable, is semi-opaque and diffusive, so that objects become more clearly visible as they approach and then contact the panel. The objects thus detected can be a user's fingers or hands, or other objects. However, there is no discussion of active objects being placed on the surface of the display panel to carry out some active function involving an interaction with a virtual environment displayed on the screen.

It would be desirable to enable interaction within the virtual environment between physical objects that are placed on a display surface and virtual objects within the virtual environment, and to enable certain types of physical objects to respond to the virtual environment being displayed in an active and perceivable manner. Ideally, the interaction should go well beyond the use of a physical object to simply select and move a virtual object on a touch-sensitive screen. Although some types of objects placed on a display surface will be passive and only capable of being moved about on the display surface by the user's hand, other types of physical objects should be active and capable of responding to the virtual environment in an interactive manner. For example, it would be preferable if a physical object that looks like some form of vehicle were capable of moving over the display surface without being picked up and replaced at a different position by the user. Thus, a physical object that includes a small electric motor drivingly coupled to drive wheels on the object might be caused by either the user or the computing device creating the virtual environment to move around on the display surface and interact with virtual objects that are displayed thereon. It would also be desirable to employ other types of active functions in different types of active objects to respond to the virtual environment being displayed.

The interaction between physical objects and virtual objects in a displayed virtual environment might take different forms. For example, it would be desirable to be able to position a physical object on the display surface, have the software program executing the virtual environment recognize the physical object and its location, and then respond to the presence of the physical object when moving virtual objects within the virtual environment. Also, the virtual environment might respond to changes that are caused by the user, in the position of the physical object in the virtual environment. Thus, if the user moves a physical object to change its position or orientation on the display surface, in response, the virtual environment software program might respond by changing some function, parameter, or feature related to the virtual environment, such as the volume of sound produced by the virtual environment software.

Sensing the changing area and location of contact of a physical object with the display surface should also preferably result in a corresponding interaction with the displayed virtual environment. For example, if the physical object changes shape with varying pressure, the movement by the user of the object over the display surface should produce a corresponding stroke in the display that represents the changing shape of the object that is in contact with the display surface. These and other forms of interaction between a physical object on the display surface and the graphic images and virtual environment that are displayed thereon can provide a much more enjoyable and realistic experience for the user.

SUMMARY OF THE INVENTION

The present invention enables a blurring of the distinction between the real world of physical objects and a virtual environment in which virtual objects and graphic images are displayed on a display screen. To achieve this result, an interactive display has been developed that includes a display surface formed as the central part of a table top. A virtual environment is projected from below and viewed on the display surface. Any physical object that is disposed on the display surface can be optically detected by an IR camera that is disposed below the display surface and which is responsive to IR light that is reflected from the physical object. Once the physical object has been detected at a specific position using the camera, a software program controlling the virtual environment or graphic images being displayed then facilitates an interaction between the physical object and the virtual environment just as if the physical object were included within the virtual environment. Depending upon the type of physical object that is positioned atop the display surface, the interaction can be in two different modes. Specifically, in a first mode, the virtual environment that is displayed responds to the presence of the physical object on the display surface. In the second mode, the physical object responds to the virtual environment that is displayed, particularly, to changes in the virtual environment or graphic images that are displayed on the displayed surface.

Any movement of the physical object relative to the display surface is tracked by sensing changes in the location of the physical object on the display surface. Accordingly, the response to the physical object may then include changing a value of a parameter in the virtual environment in response to the movement of the physical object relative to the display surface.

The response to the physical object can also be implemented by the interaction of a virtual object in the virtual environment with the physical object. Thus, for example, the virtual object can be prevented from moving through a region in the virtual environment where the physical object is located on the display surface. If the physical object represents a wall in the virtual environment, a virtual object corresponding to a car controlled by a software program displaying the virtual environment would be caused to steer away from and avoid colliding with the wall at the location of the physical object on the display screen. This method can also detect at least one of a size and a shape of the physical object at a specific location on the display surface to enable the predefined response to the physical object within the virtual environment.

The method can also identify the physical object based upon IR light reflected from the physical object. For example, the IR light that is reflected may be encoded with a predefined pattern that is optically detected by the IR camera disposed below the display surface. The physical object can be a passive object that does not perform any active function that is perceptible to the user, or an active object that does. The active object might detect a signal produced within the virtual environment, such as a specific optical signal, causing the active object to perform the active function.

The active function can include producing one or more different types of perceptible conditions, such as emitting light, or producing a tone, or vibrating, or movement of the active object over the display surface. An active object that is mobile will be able to move itself over the display surface and interact with the virtual environment. For example, the active object can be a radio-controlled (RC) toy tank that moves about on the display surface as an electric motor drives treads on the toy tank. Or the toy tank might instead be controlled by modulating IR light from an IR source such as an IR light emitting diode (LED) that is disposed under the display surface, or by visible light pulses or modulation of the visible light produced by the image source used for producing the display seen on the display surface. For this and other types of active object, the active function of the active object can be controlled by the user, or alternatively, by the software program that produces the virtual environment. Electrical power can be provided to the active object by either a battery or a charged capacitor to enable the active object to perform certain active functions. A computing device executing the software can thus control an active object, which can in turn effect the behavior of a virtual object, and further effect the user, or vice versa. Many different interactive scenarios are clearly readily imaginable.

Other types of interaction are possible between a passive object and the virtual environment. For example, an image that is specific to the physical object and is visible through the physical object can be projected on the display surface where the physical object is disposed. The projected image will then be visible through the physical object and will appear to be part of the physical object, since the projected image will move about as the passive object is moved over the display surface.

Since a location of the physical object is determined in this method, a virtual entity can be displayed at a position on the display surface selected to avoid a visual interference between the virtual object and the physical object. The virtual entity comprises either a virtual object or information. Preferably, a plurality of prioritized or ordered positions are determined at which the virtual entity can be selectively displayed to avoid the interference with the physical object. A position that avoids the interference with the physical entity and having a highest priority is selected for display of the virtual entity. Also, the method preferably determines a location of a specific user who will be visually perceiving the virtual entity, so that the virtual entity is oriented and displayed at a position selected to enable the specific user to easily visually perceive the virtual entity. This position will be selected to avoid visual interference between the physical object and the virtual entity. If the virtual entity cannot be displayed without interference with the physical object at a highest priority location, a size of the virtual entity can be reduced to avoid the interference with the physical object.

Another aspect of the present invention is directed to a memory medium on which are stored machine executable instructions for carrying out the steps of the method.

Yet another aspect of the invention is directed to a system for enabling interaction between a virtual environment and a physical object. The system comprises an interactive display that includes a display surface on which the virtual environment is displayed, and a sensor for detecting a physical object placed on the display surface. A processor is coupled to the interactive display and to a memory in which are stored machine instructions. When executed by the processor, the machine instructions cause the processor to carry out a plurality of functions that are generally consistent with the steps of the method described above.

BRIEF DESCRIPTION OF THE DRAWING FIGURES

The foregoing aspects and many of the attendant advantages of this invention will become more readily appreciated as the same becomes better understood by reference to the following detailed description, when taken in conjunction with the accompanying drawings, wherein:

FIG. 1 is a functional block diagram of a generally conventional computing device or personal computer (PC) that is suitable for processing the input and output data used in practicing the present invention;

FIG. 2 is an illustration of the interior of the interactive table showing hardware components of the interactive table, the paths followed by light within the interactive table, and exemplary objects disposed on and above the surface of the table;

FIG. 3 is a flow chart illustrating the logical steps carried out in avoiding visual interference between a physical object on a display surface and a virtual object that is imaged on the display surface, accord with this invention;

FIG. 4 is a functional block diagram illustrating how an active physical object is sensed and interacts with a virtual environment displayed on the interactive display;

FIG. 5 is a functional block diagram illustrating how a active physical object interacts with a virtual environment displayed on the interactive display;

FIG. 6 is a block diagram illustrating how the present invention can be implemented as an interactive display that is coupled to a stand alone personal computer or other computing device;

FIG. 7 is an isometric view of a display surface illustrating a plurality of passive and active objects in regard to their interaction with a virtual environment;

FIG. 8 illustrates an exemplary virtual environment game board for a "TRIVIAL PURSUIT" game, showing how a virtual entity, i.e., a question card, is displayed at a position adjacent a corner of the game board to avoid visual interference from a plurality of player pieces that are disposed in the center of the game board;

FIG. 9 is a further example of the game board shown in FIG. 9, with a question card (reduced in size) displayed at a different position to avoid the interference of a player piece disposed on the display surface at about the position where the question card was displayed in FIG. 9;

FIG. 10 illustrates a wheel hub portion of the game board from FIGS. 9 and 10, wherein the wheel hub represents the possible locations where physical objects (i.e., the player pieces) might be during the play of the game, so that a prioritized list of positions at which the virtual entity (i.e., the question cards) can be displayed to avoid visual interference by the player pieces;

FIG. 11 is an isometric view of a portion of the display surface showing letters projected onto the bottom of transparent blocks through the display surface, so that the transparent blocks, which are identified by unique encoding patterns, can be rearranged to spell a word in a children's spelling program; and

FIG. 12 illustrates how an energy source in an active object can be recharged by a power charging station.

DESCRIPTION OF THE PREFERRED EMBODIMENT

Exemplary System for Implementing Present Invention

With reference to FIG. 1, an exemplary system suitable for implementing various portions of the present invention. The system includes a general purpose computing device in the form of a conventional PC 20, provided with a processing unit 21, a system memory 22, and a system bus 23. The system bus couples various system components including the system memory to processing unit 21 and may be any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. The system memory includes read only memory (ROM) 24 and random access memory (RAM) 25. A basic input/output system 26 (BIOS), containing the basic routines that help to transfer information between elements within the PC 20, such as during start up, is stored in ROM 24. The PC 20 further includes a hard disk drive 27 for reading from and writing to a hard disk (not shown), a magnetic disk drive 28 for reading from or writing to a removable magnetic disk 29, and an optical disk drive 30 for reading from or writing to a removable optical disk 31, such as a compact disk-read only memory (CD-ROM) or other optical media. Hard disk drive 27, magnetic disk drive 28, and optical disk drive 30 are connected to system bus 23 by a hard disk drive interface 32, a magnetic disk drive interface 33, and an optical disk drive interface 34, respectively. The drives and their associated computer readable media provide nonvolatile storage of computer readable machine instructions, data structures, program modules, and other data for PC 20. Although the exemplary environment described herein employs a hard disk, removable magnetic disk 29, and removable optical disk 31, it will be appreciated by those skilled in the art that other types of computer readable media, which can store data and machine instructions that are accessible by a computer, such as magnetic cassettes, flash memory cards, digital video disks (DVDs), Bernoulli cartridges, RAMs, ROMs, and the like, may also be used in the exemplary operating environment.

A number of program modules may be stored on the hard disk, magnetic disk 29, optical disk 31, ROM 24, or RAM 25, including an operating system 35, one or more application programs 36, other program modules 37, and program data 38. A user may enter commands and information in to PC 20, and provide control input through input devices such as a keyboard 40 and a pointing device 42. Pointing device 42 may include a mouse, stylus, wireless remote control, or other pointer, but in connection with the present invention, such conventional pointing devices may be omitted, since the user can employ the interactive display for input and control. As used hereinafter, the term "mouse" is intended to encompass virtually any pointing device that is useful for controlling the position of a cursor on the screen. Other input devices (not shown) may include a microphone, joystick, haptic joystick, yoke, foot pedals, game pad, satellite dish, scanner, or the like. These and other input/output (I/O) devices are often connected to processing unit 21 through an I/O interface 46 that is coupled to the system bus 23. The term I/O interface is intended to encompass each interface specifically used for a serial port, a parallel port, a game port, a keyboard port, and/or a universal serial bus (USB). System bus 23 is also connected to a camera interface 59, which is coupled to an interactive display 60 to receive signals form a digital video camera that is included therein, as discussed below. The digital video camera may be instead coupled to an appropriate serial I/O port, such as to a USB version 2.0 port. Optionally, a monitor 47 can be connected to system bus 23 via an appropriate interface, such as a video adapter 48; however, the interactive display of the present invention can provide a much richer display and interaction with the user for display and input of information and control of software applications and is therefore coupled to the video adaptor. In addition to the monitor, PCs are often coupled to other peripheral output devices (not shown), such as speakers (through a sound card or other audio interface--not shown) and printers. It is also contemplated that PC 20 may be coupled to active objects placed on the display surface through a peer-to-peer radio (e.g. Bluetooth) system (not shown) which would allow it to communicate with the active objects. Alternatively, a controller can be coupled to the PC through an appropriate I/O port, and the controller can be coupled to one more active objects through an appropriate wired, RF link, or other