Title: Universal computing device
Abstract: A universal input device is described. The universal input device provides a common user interface for a variety of different computing platforms including printed documents. Using the present system, one may use the universal input device to control various computing devices as well as capture handwritten electronic ink and have the electronic in be associated with new or stored documents.
Patent Number: 7,009,594 Issued on 03/07/2006 to Wang, et al.
| Inventors:
|
Wang; Jian (Beijing, CN);
Zhang; ChunHui (Beijing, CN)
|
| Assignee:
|
Microsoft Corporation (Redmond, WA)
|
| Appl. No.:
|
284417 |
| Filed:
|
October 31, 2002 |
| Current U.S. Class: |
345/156; 345/179 |
| Current Intern'l Class: |
G09G 5/00 (20060101) |
| Field of Search: |
345/173,156,179,589,634,619
713/176
715/863,503
348/140.1
358/115
382/161,100,185
235/462.45,494,100
434/322
286/46
706/13
|
References Cited [Referenced By]
U.S. Patent Documents
| 4829583 | May., 1989 | Monroe et al.
| |
| 5247137 | Sep., 1993 | Epperson.
| |
| 5294792 | Mar., 1994 | Lewis et al.
| |
| 5587558 | Dec., 1996 | Matsushima.
| |
| 5644652 | Jul., 1997 | Bellegarda et al.
| |
| 5661506 | Aug., 1997 | Lazzouni et al.
| |
| 5686718 | Nov., 1997 | Iwai et al.
| |
| 5754280 | May., 1998 | Kato et al.
| |
| 5756981 | May., 1998 | Roustaei et al.
| |
| 5774602 | Jun., 1998 | Taguchi et al.
| |
| 5822465 | Oct., 1998 | Normile et al.
| |
| 5855483 | Jan., 1999 | Collins et al.
| |
| 5898166 | Apr., 1999 | Fukuda et al.
| |
| 5902968 | May., 1999 | Sato et al.
| |
| 5960124 | Sep., 1999 | Taguchi et al.
| |
| 5995084 | Nov., 1999 | Chan et al.
| |
| 6005973 | Dec., 1999 | Seybold et al.
| |
| 6081261 | Jun., 2000 | Wolff et al.
| |
| 6141014 | Oct., 2000 | Endo et al.
| |
| 6157935 | Dec., 2000 | Tran et al.
| |
| 6202060 | Mar., 2001 | Tran.
| |
| 6226636 | May., 2001 | Abdel-Mottaleb et al.
| |
| 6243071 | Jun., 2001 | Shwarts et al.
| |
| 6278968 | Aug., 2001 | Franz et al.
| |
| 6335727 | Jan., 2002 | Morishita et al.
| |
| 6577299 | Jun., 2003 | Schiller et al.
| |
| 6585154 | Jul., 2003 | Ostrover et al.
| |
| 6731271 | May., 2004 | Tanaka et al.
| |
| 6744967 | Jun., 2004 | Kaminski et al.
| |
| 6752317 | Jun., 2004 | Dymetman et al.
| |
| 6847356 | Jan., 2005 | Hasegawa et al.
| |
| 2001/0024193 | Sep., 2001 | Fahraeus.
| |
| 2002/0024499 | Feb., 2002 | Karidis et al.
| |
| 2002/0069220 | Jun., 2002 | Tran.
| |
| 2002/0148655 | Oct., 2002 | Cho et al.
| |
| 2002/0163510 | Nov., 2002 | Williams et al.
| |
| 2002/0163511 | Nov., 2002 | Sekendur.
| |
| 2003/0063045 | Apr., 2003 | Fleming.
| |
| 2003/0063072 | Apr., 2003 | Brandenberg et al.
| |
| 2003/0146883 | Aug., 2003 | Zelitt.
| |
| 2004/0032393 | Feb., 2004 | Brandenberg et al.
| |
| 2004/0046744 | Mar., 2004 | Rafii et al.
| |
| 2004/0136083 | Jul., 2004 | Wang et al.
| |
| 2004/0140964 | Jul., 2004 | Wang et al.
| |
| 2004/0140985 | Jul., 2004 | Wang et al.
| |
| 2004/0143559 | Jul., 2004 | Ayala.
| |
| 2004/0153649 | Aug., 2004 | Rhoads et al.
| |
| 2005/0024324 | Feb., 2005 | Tomasi et al.
| |
| 2005/0052469 | Mar., 2005 | Crosby et al.
| |
| Foreign Patent Documents |
| 0 407 734 | Jan., 1991 | EP.
| |
| 0 717 368 | Jun., 1996 | EP.
| |
| 0 732 666 | Sep., 1996 | EP.
| |
| 0 865 166 | Sep., 1998 | EP.
| |
| 1 158 456 | Nov., 2001 | EP.
| |
| WO 01/2603/2 | Apr., 2001 | WO.
| |
| WO 01/4868/5 | Jul., 2001 | WO.
| |
| WO 01/7165/4 | Sep., 2001 | WO.
| |
| WO 02/077870 | Oct., 2002 | WO.
| |
Other References
Internet printout—http:/www.anotofunctionality.com: Anoto Functionality—How
does it work?, Sep. 26, 2002.
Internet printout—http:/www.anotofunctionality.com: Anoto Functionality—Anoto
pattern & digital paper, Sep. 26, 2002.
Internet printout—http:/www.anotofunctionality.com: Anoto Functionality—Digital
pens, Sep. 26, 2002.
Internet printout—http:/www.anotofunctionality.com: Anoto Functionality—Digital
pens Use with personal computers, Sep. 26, 2002.
Internet printout—http:/www.anotofunctionality.com: Anoto Functionality—What
is Anoto functionality?, Sep. 5, 2003.
Internet printout—http:/www.anotofunctionality.com: Anoto Functionality—Anoto
pattern & digital paper, Sep. 5, 2003.
Internet printout—http:/www.anotofunctionality.com: Anoto Functionality—Digital
pens Use with mobile phones, Sep. 5, 2003.
Internet printout—http:/www.anotofunctionality.com: Anoto Functionality—Security,
Sep. 5, 2003.
Internet printout—http:/www.anotofunctionality.com: Anoto Functionality—Applications,
Sep. 5, 2003.
Internet printout—http:/www.anotofunctionality.com: Anoto Functionality—E-mail,
Sep. 5, 2003.
Internet printout—http:/www.anotofunctionality.com: Anoto Functionality—Fax,—E-mail,
Sep. 5, 2003.
Internet printout—http:/www.functionality.com: Anoto Functionality—Graphical
SMS, Sep. 5, 2003.
Internet printout—http:/www.functionality.com: Anoto Functionality—Digital
Notes, Sep. 5, 2003.
Internet printout—http:/www.functionality.com: Anoto Functionality—Corporate
applications, Sep. 5, 2003.
Internet printout—http:/www.functionality.com: Anoto Functionality—Freedom
of expression, Sep. 26, 2002.
Internet printout—http:/www.functionality.com: Anoto Functionality—You
to yourself, Sep. 26, 2002.
Internet printout—http:/www.functionality.com: Anoto Functionality—You
to someone else, Sep. 26, 2002.
Internet printout—http:/www.functionality.com: Anoto Functionality—You
to an organization, Sep. 26, 2002.
Internet printout—http:/www.functionality.com: Anoto Functionality—Hints
& tips, Sep. 5, 2003.
Internet printout—http:/www.functionality.com: Anoto Functionality—Hints
& tips Using your digital pen, Sep. 5, 2003.
Internet printout—http:/www.functionality.com: Anoto Functionality—Hints
& tips Using your digital paper, Sep. 5, 2003.
Internet printout—http:/www.functionality.com: Anoto Functionality—
Digital paper, Sep. 26, 2002.
Internet printout—http:/www.functionality.com: Anoto Functionality—
Digital service, Sep. 26, 2002.
Internet printout—http:/www.functionality.com: Anoto Functionality—Hints
& tips Using your digital service, Sep. 5, 2003.
Internet printout—http:/www.anotofunctionality.com: Anoto Functionality—Digital
pens, Sep. 5, 2003.
Internet printout—http:/www.anotofunctionality.com: Anoto Functionality—Digital
paper, Sep. 5, 2003.
Internet printout—http:/www.anotofunctionality.com: Anoto Functionality—Digital
service, Sep. 5, 2003.
Internet printout—http:/www.anotofunctionality.com: Anoto Functionality—Software
and additionals, Sep. 5, 2003.
Internet printout—http:/www.edgereview.com: The Edge—First Look:
Digital Ink n-scribe, Sep. 5, 2003.
Internet printout—http:/www.techtvcom: Nscribe Pen and Presenter-to-Go,
Sep. 5, 2003.
Internet printout—http:/www.pcmag.com: Jot This, Sep. 5, 2003.
Internet printout—http:/www.competitivetech.net: Competitive Technologies'
Investee Introduces N-Scribe Pen, Sep. 5, 2003.
Internet printout—http:/www.flashcommerce.com: n-scribe For Digital Writing,
Sep. 5, 2003.
Internet printout—http:/www.wired.com: The Hot New Medium: Paper, Sep.
5, 2003.
Internet printout—http:/www.gizmo.com: Maxell Digital Pen to Use Anoto
system, Sep. 5, 2003.
Internet printout—http:/www.anoto.com: Pattern, Sep. 5, 2003.
Internet printout—http:/www.anoto.com: Construction, Sep. 5, 2003.
Internet printout—http:/www.anoto.com: Paper space, Sep. 5, 2003.
Internet printout—http:/www.anoto.com: Paper and Printing, Sep. 5, 2003.
Internet printout—http:/www.anoto.com: Page template, Sep. 5, 2003.
Internet printout—http:/www.anoto.com Printers supporting Anoto functionality,
Sep. 5, 2003.
Internet printout—Mimio—Digital Whiteboard Recorder, undated.
Internet printout—http:/www.mimio.com: Capture, Save and Share, Sep. 5, 2003.
Internet printout—http:/www.tabletpccorner.com: Auto Pen Bluetooth, Sep.
5, 2003.
Internet printout—http:/www.wacom.com: intuos2, Sep. 5, 2003.
Internet printout—http:/www.wacom.com: Penable Wacom, Sep. 5, 2003.
Internet printout—http:/www.wacom.com: tablet PC, Sep. 5, 2003.
Internet printout—http:/www.wacom.com: Cintiq—Interactive Pen Display,
Sep. 5, 2003.
Internet printout—http:/www.wacom.com: Intuos2—the Professional Tablet,
Sep. 5, 2003.
Internet printout—http:/www.wacom.com: Graphire2—Have more fun with
digital phones, Sep. 5, 2003.
Internet printout—http:/www.smarttech.com: What's New, Sep. 5, 2003.
Internet printout—http:/www.smarttech.com: SMART Board Interactive Whiteboard,
Sep. 5, 2003.
Internet printout—http:/www.smarttech.com: SMART Board Interactive Whiteboard—Front
Projection Features, Sep. 5, 2003.
Internet printout—http:/www.smarttech.com: SMART Board Interactive Whiteboard—Q&A,
Sep. 5, 2003.
Internet printout—SMART Board™ Interactive Whiteboard Specifications—Model
540, undated.
Internet printout—http:/www.smarttech.com: SMART Board Interactive Whiteboard
Profiles—Sep. 5, 2003.
Internet printout—http:/www.smarttech.com: SMART Board Software Features—Sep.
5, 2003.
Internet printout—http:/www.smarttech.com: Press Releases—SMART lauches
Research Assistance Program—Sep. 5, 2003.
Internet printout—http:/www.smarttech.com: SMART Technologies, Inc. Press
Releases, Sep. 5, 2003.
Internet printout—http:/www.smarttech.com: SMART Technologies, Inc., New
annotation and software functionality on all SMART Board™ Interactive whiteboards,
Sep. 5, 2003.
Internet printout—http:/www.smarttech.com: Carnegie Mellon research ranks
the SMART Board™ interactive whiteboards as fastest, most accurate way
to interact with projected information, Sep. 5, 2003.
Internet printout—http:/www.smarttech.com: SMART Campfire™ whiteboard
camera system effortlessly saves dry-erase whiteboard notes, Sep. 5, 2003.
Internet printout—http:/www.smarttech.com: SMART Technologies, Inc., awarded
another U.S. patent for touch sensitive SMART Board™ technology, Sep. 5, 2003.
Internet printout—http:/www.mimio.com: Turn your whiteboard into an interactive
whiteboard, Sep. 5, 2003.
Internet printout—http:/www.mimio.com: Mimio products, Sep. 5, 2003.
Internet printout—http:/www.mimio.com: Mimio technology, Sep. 5, 2003.
Brochure: VPEN, Revolutionizing human interaction with the digital world™,
OTM Technologies, undated.
Internet printout—http:/www.ammagazine.com: RF Pen Sends Your Scibbles,
Sep. 26, 2002.
Fujieda et al., Developmet of Pen-shaped Scanners, NEC vol. 51 No. 10, 1998.
A. Champaneria, "PADCAM: A Real-Time, Human-Centric Notetaking System" MIT Laboratory
for Computer Science, Mar. 2002.
Internet printout—http:www.is.cs.cmu.edu: Preprocessing in the Npen++ System,
Aug. 8, 2002.
Internet printout—http:www.is.cs.cmu.edu:Final input representation, Aug.
8, 2002.
Internet printout—http:www.is.cs.cmu.edu: Npen++, Aug. 8, 2002.
J.L. Crowley et al., "Things That See", Communications of the ACM, vol. 43, No.
3, Mar. 2000.
B. Ko et al., "Finger Mouse and Gesture Recognition System as a New Human Computer
Interface" Comput. & Graphics, vol. 21, No. 5, pp. 555-561, 1997.
Okada et al., "A High-Resolution Handwriting Character Input Device Using Laser
Beams", 1981.
Sato et al., Video Tablet—2D Corrdinate Input Device with OCD Camera, vol.
J67-D No. 6.
Kai-Fu Lee, "Automatic Spech Recognition—The Development of the SPINX System",
Kluwer Academic Publishers, pp. 1-207, 1992.
Frederick Jelinek, "Statiscal Methods for Speech Recognition", The MIT Press,
pp. 1-283, 2001.
Fujieda et al., "Development of Pen-Shaped Scanners", Nec, vol. 51, No.
10, 1998.
Crowley et al., "Things That See", Communications of the A.C.M., vol.
43, No. 3, pp. 54-64, Mar. 2000.
Sato et al., "Video Tablet—2D Coordinate Input Device With OCD Camera",
Osaka University, vol. J67-D, No. 6, Jun. 1984.
Okada et al., "A High-Resolution Handwiting Character Input Device Using Laser
Beams", Department of Instrumentation Engineering, Faculty of Science and Technology,
vol. 10.4, No. 11.1, 1981.
Ko et al., "Finger Mouse and Gesture Recognition System As a new Human computer
Interface", Computer and Graphics, Col. 21, No. 5, pp. 555-561, 1997.
Champaneria, "PADCAM: A Real-Time, Human-Centric Notetaking System", MIT
Laboratory for Computer Science, Mar. 2002.
OTM Technologies, "VPEN—Revolutionizing Human Interaction With the Digital
World™", pp. 1-6.
Internet Print Out: "N-Scribe For Digital Writing", Mobileinfo.com, News
issue #2001—15 (Apr. 2001), http://www.mobileinfo.com/News 2001/Issue15/Digital-nscribe.htm,
dated Jul. 15, 2002.
Internet Print Out: "Don't Break This Pen", Edgereview.com, by Brian Urbanski
http://www.edgereview.com/ataglance.cfm?category=edge&ID=180, dated Jul. 15, 2002.
Internet Print Out: "DataGlyphs®, Embedding Digital Data", Parc Solutions,
http://www.parc.com/solutions/dataglyphs/, dated Jul. 15, 2002.
Internet Print Out: "Navilite—Optical Opportunities—Bluetooth-enabled
optical transition measurement technology paves the way for an untethered stylus
that can write on any surface", vol. 8, Issue No. 34, Jul. 5-11, 2002, www.computerworld.com,
dated Aug. 15, 2002.
Internet Print Out: "Competitive Technologies' Investee Introduces N-Scribe
Pen—Digital Ink Presents Wireless Pen at Demo 2001", Competitive Technologies,
http://www.competitivetech, dated Sep. 5, 2003.
Internet Print Out: "N-Scibe For Digital Writing", Flash Commerce News,
http://www.flashcommerce.com/articles/, dated Sep. 5, 2003.
Internet Print Out: "The Hot New Medium Paper—How The Oldest Interface
In The Book Is Redrawing The Map Of The Networked World", http://www.wired.com/wired/,
dated Sep. 5, 2003.
Internet Print Out: "Maxwell Digital Pen To Use Anoto System", Gizmo,
http://www.gismo.com/au/, dated Sep. 5, 2003.
Internet Print Out: "Anoto Pen Bluetooth", Tabletpccorner, http://www.tabletpccorner.net,
dated Sep. 5, 2003.
Internet Print Out: "Jot This—Nscribe Pen", Magazine, http://www.pcmag.com/article2/0,4149,31650,00.asp,
dated Jul. 15, 2002.
Internet Print Out: "Jot This—Nscribe Pen", Magazine, http://www.pcmag.com/article2/0,4149,31650,00.asp,
dated Sep. 5, 2003.
Internet Print Out: "RF Pen Send Your Scribbles", Appliance Manufacturing
Magazine, http://www.ammagazine.com, dated Sep. 26, 2002.
Internet Print Out: " Nscribe pen And Presenter-To-Go—Infrared Pen And
New Springboard Module Make Their Debut At Demo 2001 ", Edgereview.com by Brian
Urbanski, http://www.techtv.com.freshgear/pr, dated Sep. 5, 2003.
Internet Print Out: "Don't Break This Ink Pen", Edgereview.com, by Brian
Urbanski, http://www.edgereview.com/ataglance.cfm?category=edge&ID=180, dated Sep.
5, 2003.
Internet Print Out: "Preprocessing In the NPen++ System", http://www.is.cs.cmu.edu/mie/multimodal
npen preproc.html, dated Aug. 8, 2002.
Internet Print Out: "OTM Technologies—V Pen", searched http://www.otmtech.com/vpen3.asp,
pp. 1-7.
Internet Print Out: "Mimio—Products", Mimio, http://www.mimio.com,
pp. 1-8.
Internet Print Out: "SMART Board Interactive Whiteboard", Smarttech, http://www.smarttech.com,
pp. 1-26.
Haynes, "Wacom PL-500", www.wacom.co.uk.
Internet Print Out: "(Electro-Magnetic Resonance) send and position sensing
technology", Wacom, Technology, http://www.wacom-components.com/tech.asp, pp. 1-6.
Internet Print Out: "Cordless Batteryless Pen", Wacom Penabled Components,
http://www.wacom.com/components/index.cfm, dated Jul. 15, 2002.
Internet Print Out: "PL-500—15.1 inch Screen Digital LCD Pen Tablet
System", Wacom Graphics Tablet Solution, http://ap.wacm.co.ip/products/pl/contents/pl500.html,
pp. 1-13.
Internet Print Out: Digital Pens, http://www.anotofunctionality.com/navigate.asp?PageID=96,
pp. 1-70.
European Search Report, Application No.: 03021238.5-1527; Dated Jun. 1, 2005,
Total pages (6).
Dey, et al., "A Fast Algorithm for Computing the Euter Number of an image and
its VLSI Implementation", IEEE; 13th International Conference on VLSI
Design (Jan. 2003).
European Search Report for Application No. EP 03 02 1235, dated Jun. 1, 2005.
|
Primary Examiner: Shankar; Vijay
Assistant Examiner: Dharia; Prabodh
Attorney, Agent or Firm: Banner & Witcoff Ltd
Claims
We claim:
1. An input device for generating data representative of hand written strokes,
the input device comprising:
an image capturing unit for capturing an image of an area of an object over which
the input device is positioned and generating captured image data, said data representative
of hand written strokes being determined from location information extracted from
said captured image data;
a processor processing the captured image data; and
a memory storing data,
wherein the object comprises a display of a computing device including an image
pattern providing location information of an area on the display,
wherein the image of the area over which the input device is positioned includes
image data representative of a location of the area of the object,
wherein the image data representative of a location of the area of the object
includes an image pattern representative of the location of the area of the object, and
wherein the image pattern representative of the location of the area of the object
includes a portion of a maze-like pattern.
Description
RELATED APPLICATIONS
This application is related to U.S. Ser. No. 10/284,412, entitled "Active Embedded
Interaction Code," invented by Jian Wang, Qiang Wang, Chunhui Zhang, and Yue Li,
and to U.S. Ser. No. 10/284,451, entitled "Passive Embedded Interaction Code,"
invented by Jian Wang, Yingnong Dang, Jiang Wu and Xiaoxu Ma, whose contents are
hereby incorporated by reference.
BACKGROUND
1. Technical Field
This disclosure relates to a computer input device for generating smooth electronic
ink. More particularly, the disclosure relates to an input device may be used on
divergent platforms, while providing a common user interface.
2. Related Art
Computing systems have dramatically changed the way in which we live. The
first wave of computers was prohibitively expensive, and was only cost effective
for use in business settings. As computers became more affordable, the use of personal
computers both in the workplace and at home have become so widespread that computers
have become as common as desks in the office and kitchen tables in the home. Microprocessors
have been incorporated in all aspects of our daily lives, from use in television
and other entertainment systems to devices for regulating the operation of our automobile.
The evolution of computing devices, from data crunching devices that occupied
entire floors of large office facilities, to laptop computers or other portable
computing devices, has dramatically impacted the manner in which documents are
generated and information stored. Such portable computing have enabled individuals
to type letters, draft memorandum, take notes, create images, and perform numerous
tasks in places other than the office using these computing devices. Professionals
and nonprofessionals alike are empowered to take perform tasks while on the move
using devices that fulfill their computing needs in any location.
Typical computer systems, especially computer systems using graphical user
interface (GUI) systems, such as Microsoft Windows, are optimized for accepting
user input from one or more discrete input devices such as a keyboard (for entering
text), and a pointing device (such as a mouse) with one or more buttons for activating
user selections.
One of the original goals of the computing world was to have a computer on every
desk. To a large extent, this goal has been realized by the personal computer becoming
ubiquitous in the office workspace. With the advent of notebook computers and high-capacity
personal data assistants, the office workspace has been expanded to include a variety
of non-traditional venues in which work is accomplished. To an increasing degree,
computer users must become masters of the divergent user interfaces for each of
their computing devices. From a mouse and keyboard interface for the standard personal
computer to the simplified resistive stylus interface of personal data assistants
and even to the minimalistic keys of a cellular telephone, a user is confronted
with a variety of different user interfaces that one needs to master before he
can use the underlying technology.
Despite the advances in technology, most users tend to use documents printed
on paper as their primary editing tool. Some advantages of printed paper include
its readability and portability. Others include the ability to share annotated
paper documents and the ease at which one can archive printed paper. One user interface
that is bridging the gap between advanced computing systems and the functionality
of printed paper is a stylus-based user interface. One approach for the stylus-based
user interface is to use resistive technology (common in today's PDAs). Another
approach is to use active sensors in a notebook computer. One of the next goals
of the computing world is to bring the user interface for operating the computer
back to the user.
A drawback associated with the use of a stylus is that such devices are tied
to
the computing device containing the sensor board. In other words, the stylus may
only be used to generate inputs when used in conjunction with the required sensor
board. Moreover, detection of a stylus is affected by the proximity of the stylus
to the sensing board.
There is a need in the art for a portable computing device that may function
as an input device for any one of a variety of computing devices and which may
operate in a variety of situations.
SUMMARY
Aspects of the present invention address one or more of the issues identified
above, thereby providing a common user interface to users across divergent computing
platforms. Aspects of the present invention relate to an input device for generating
electronic ink, and/or generating other inputs, independent of the device for which
the data is intended. The input device may be formed in the shape of a pen, and
may or may not include an ink cartridge to facilitate movement of the input device
in a familiar manner.
The foregoing summary of aspects of the invention, as well as the following detailed
description of various embodiments, is better understood when read in conjunction
with the accompanying drawings, which are included by way of example, and not by
way of limitation with regard to the claimed invention.
BRIEF DESCRIPTION OF THE DRAWINGS
FIG. 1 shows a general description of a computer that may be used in conjunction
with embodiments of the present invention.
FIG. 2 illustrates an input device (including all of the components) in accordance
with an illustrative embodiment of the present invention.
FIG. 3 provides three illustrative embodiments of a camera system for use in
accordance with aspects of the present invention.
FIG. 4 illustrates an illustrative technique (maze pattern) for encoding the
location of the document.
FIG. 5 provides an illustration of a trace pattern from which electronic ink
may be generated.
FIG. 6 shows the hardware architecture of a system in accordance with one embodiment
of the present invention.
FIG. 7 illustrates a further combination of components incorporated in an input
device for generating electronic ink in accordance with another illustrative embodiment.
FIG. 8 illustrates uses of an input device in accordance with several illustrative
embodiments of the present invention.
DETAILED DESCRIPTION OF THE DRAWINGS
Aspects of the present invention relate to an input device that may be used
in a variety of different computing platforms from controlling a desktop or notebook
computer, writing on a whiteboard, controlling a PDA or cellular phone, or creating
ink that may be ported to any of these platforms. The following description is
divided into a number of sections as follows: terms, general-purpose operating
environment, universal pen and camera, active coding, passive coding, internal
sensors, additional components, sample implementations.
Terms
Pen—any writing implement that may or may not include the ability
to store ink. In some examples a stylus with no ink capability may be used as a
pen in accordance with embodiments of the present invention.
Camera—an image capture system.
Active Coding—incorporation of codes within the object or surface over
which the input device is positioned for the purpose of determining positioning
and/or movement of the input device using appropriate processing algorithms.
Passive Coding—detecting movement/positioning of the input device using
image data, other than codes incorporated for that purpose, obtained from the object
or surfaces over which the input device is moved using appropriate processing algorithms.
Input Device—a device for entering information which may be configured
for generating and processing information
Active Input Device—an input device that actively measures signals and
generates data indicative of positioning and/or movement of the input device using
sensors incorporated within the input device.
Passive Input Device—an input device for which movement is detected
using sensors incorporated other than within the input device.
Computing Device—a desktop computer, a laptop computer, Tablet PC™,
a personal data assistant, a telephone, or any device which is configured to process
information including an input device.
General Purpose Operating Environment
FIG. 1 is a functional block diagram of an example of a general-purpose digital
computing environment that can be used to implement various aspects of the present
invention. In FIG. 1, a computer
100 includes a processing unit
110,
a system memory
120, and a system bus
130 that couples various system
components including the system memory to the processing unit
110. The system
bus
130 may be any of several types of bus structures including a memory
bus or memory controller, a peripheral bus, and a local bus using any of a variety
of bus architectures. The system memory
120 includes read only memory (ROM)
140 and random access memory (RAM)
150.
A basic input/output system
160 (BIOS), containing the basic routines
that
help to transfer information between elements within the computer
100, such
as during start-up, is stored in the ROM
140. The computer
100 also
includes a hard disk drive
170 for reading from and writing to a hard disk
(not shown), a magnetic disk drive
180 for reading from or writing to a
removable magnetic disk
190, and an optical disk drive
191 for reading
from or writing to a removable optical disk
192 such as a CD ROM or other
optical media. The hard disk drive
170, magnetic disk drive
180,
and optical disk drive
191 are connected to the system bus
130 by
a hard disk drive interface
192, a magnetic disk drive interface
193,
and an optical disk drive interface
194, respectively. The drives and their
associated computer-readable media provide nonvolatile storage of computer readable
instructions, data structures, program modules and other data for the personal
computer
100. It will be appreciated by those skilled in the art that other
types of computer readable media that can store data that is accessible by a computer,
such as magnetic cassettes, flash memory cards, digital video disks, Bernoulli
cartridges, random access memories (RAMs), read only memories (ROMs), and the like,
may also be used in the example operating environment.
A number of program modules can be stored on the hard disk drive
170,
magnetic
disk
190, optical disk
192, ROM
140 or RAM
150, including
an operating system
195, one or more application programs
196, other
program modules
197, and program data
198. A user can enter commands
and information into the computer
100 through input devices such as a keyboard
101 and pointing device
102. Other input devices (not shown) may
include a microphone, joystick, game pad, satellite dish, scanner or the like.
These and other input devices are often connected to the processing unit
110
through a serial port interface
106 that is coupled to the system bus, but
may be connected by other interfaces, such as a parallel port, game port or a universal
serial bus (USB). Further still, these devices may be coupled directly to the system
bus
130 via an appropriate interface (not shown). A monitor
107 or
other type of display device is also connected to the system bus
130 via
an interface, such as a video adapter
108. In addition to the monitor, personal
computers typically include other peripheral output devices (not shown), such as
speakers and printers. In a preferred embodiment, a pen digitizer
165 and
accompanying pen or stylus
166 are provided in order to digitally capture
freehand input. Although a direct connection between the pen digitizer
165
and the serial port is shown, in practice, the pen digitizer
165 may be
coupled to the processing unit
110 directly, via a parallel port or other
interface and the system bus
130 as known in the art. Furthermore, although
the digitizer
165 is shown apart from the monitor
107, it is preferred
that the usable input area of the digitizer
165 be co-extensive with the
display area of the monitor
107. Further still, the digitizer
165
may be integrated in the monitor
107, or may exist as a separate device
overlaying or otherwise appended to the monitor
107.
The computer
100 can operate in a networked environment using logical
connections to one or more remote computers, such as a remote computer
109.
The remote computer
109 can be a server, a router, a network PC, a peer
device or other common network node, and typically includes many or all of the
elements described above relative to the computer
100, although only a memory
storage device
111 has been illustrated in FIG. 1. The logical connections
depicted in FIG. 1 include a local area network (LAN)
112 and a wide area
network (WAN)
113. Such networking environments are commonplace in offices,
enterprise-wide computer networks, intranets and the Internet.
When used in a LAN networking environment, the computer
100 is connected
to the local network
112 through a network interface or adapter
114.
When used in a WAN networking environment, the personal computer
100 typically
includes a modem
115 or other means for establishing a communications over
the wide area network
113, such as the Internet. The modem
115, which
may be internal or external, is connected to the system bus
130 via the
serial port interface
106. In a networked environment, program modules depicted
relative to the personal computer
100, or portions thereof, may be stored
in the remote memory storage device.
It will be appreciated that the network connections shown are illustrative and
other techniques for establishing a communications link between the computers can
be used. The existence of any of various well-known protocols such as TCP/IP, Ethernet,
FTP, HTTP and the like is presumed, and the system can be operated in a client-server
configuration to permit a user to retrieve web pages from a web-based server. Any
of various conventional web browsers can be used to display and manipulate data
on web pages.
Universal Pen and Camera
FIG. 2 provides an illustrative embodiment of an input device for use in accordance
with various aspects of the invention. The following describes a number of different
elements and/or sensors. Various sensor combinations may be used to practice aspects
of the present invention. Further, additional sensors may be included as well,
including a magnetic sensor, an accelerometer, a gyroscope, a microphone, or any
sensor for that might detect the position of the input device relative to a surface
or object. In FIG. 2, pen
201 includes ink cartridge
202, pressure
sensor
203, camera
204, inductive element
205, processor
206,
memory
207, transceiver
208, power supply
209, docking station
210, cap
211, and display
212. The various components may
be electrically coupled as necessary using, for example, a bus, not shown. Pen
201 may serve as an input device for a range of devices including a desktop
computer, a laptop computer, Tablet PC™, a personal data assistant, a telephone,
or any device which may process and/or display information.
The input device
201 may include an ink cartridge
202 for performing
standard pen and paper writing or drawing. Moreover, the user can generate electronic
ink with the input device while operating the device in the manner typical of a
pen. Thus, the ink cartridge
202 may provide a comfortable, familiar medium
for generating handwritten strokes on paper while movement of the pen is recorded
and used to generate electronic ink. Ink cartridge
202 may be moved into
a writing position from a withdrawn position using any of a number of known techniques.
Alternatively, ink cartridge
202 may be replaced with a cartridge that does
not contain ink, such as a plastic cartridge with a rounded tip, but that will
allow the user to move the pen about a surface without damaging the pen or the
surface. Additionally, an inductive element or elements may be included to aid
in detecting relative movement of the input device by, for example, providing signals
indicative of the input device in a manner similar to those generated by a stylus.
Pressure sensor
203 may be included for designating an input, such as might
be indicated when the pen
201 is depressed while positioned over an object,
thereby facilitating the selection of an object or indication as might be achieved
by selecting the input of a mouse button, for example. Alternatively, the pressure
sensor
203 may detect the depressive force with which the user makes strokes
with the pen for use in varying the width of the electronic ink generated. Further,
sensor
203 may trigger operation of the camera. In alternative modes, camera
204 may operate independent of the setting of pressure sensor
203.
Moreover, in addition to the pressure sensor which may act as a switch,
additional switches may also be included to effect various settings for controlling
operation of the input device. For example, one or more switches, may be provided
on the outside of the input device and used to power on the input device, to activate
the camera or light source, to control the sensitivity of the sensor or the brightness
of the light source, set the input device in a sketch mode in which conversion
to text is not performed, to set the device to store the input data internally,
to process and store the input data, to transmit the data to the an processing
unit such as a computing device with which the input device is capable of communicating,
or to control any setting that might be desired.
Camera
204 may be included to capture images of the surface over which
the pen is moved. Inductive element
205 also may be included to enhance
performance of the pen when used as a stylus in an inductive system. Processor
206 may be comprised of any known processor for performing functions associated
with various aspects of the invention, as will described in more detail to follow.
Similarly, memory
207 may include a RAM, a ROM, or any memory device for
storing data and/or software for controlling the device or processing data. The
input device may further include a transceiver
208. The transceiver permits
information exchange with other devices. For example, Bluetooth or other wireless
technologies may be used to facilitate communications. The other devices may include
a computing device which may further includes input devices.
Power supply
209 may be included, and may provide power if the pen
201
is to be used independent of and remotely from the host device, the device in which
the data is to be processed, stored and/or displayed. The power supply
209
may be incorporated into the input device
201 in any number of locations,
and may be positioned for immediate replacement, should the power supply be replaceable,
or to facilitate its recharging should the power supply be rechargeable. Alternatively,
the pen may be coupled to alternate power supplies, such as an adapter for electrically
coupling the pen
201 to a car battery, a recharger connected to a wall outlet,
to the power supply of a computer, or to any other power supply.
Docking station link
212 may be used to transfer information between
the input device and a second device, such as an external host computer. The docking
station link may also include structure for recharging the power supply
206
when attached to a docking station, not shown, or connected to a power supply.
A USB or other connection may removably connect the input device to a host computer
through the docking station link, or through an alternative port. Alternatively,
a hardwire connection may also be used to connect the pen to a device for transferring
and receiving data. In a hardwired configuration, the docking station link would
be omitted in favor of wiring connecting the input device directly to a host. The
docking station link may be omitted or replaced with another system for communicating
with another device (Bluetooth 802.116, for example).
The input device
201 may further include a removable cap
211 which
may be equipped with a metal tip for facilitating resistive sensing, so that input
device
201 may be used with a device that includes a sensing board, for
example. The shell of input device
201 may be comprised of plastic, metal,
a resin, a combination thereof, or any material that may provide protection to
the components or the overall structure of the input device. The chassis may include
a metal compartment for electrically shielding some or all of the sensitive electronic
components of the device. The input device may be of an elongated shape, which
may correspond to the shape of a pen. The device may, however, be formed in any
number of shapes consistent with its use as an input device and/or ink generating device.
FIGS. 3A-C depict three illustrative embodiments of a camera for use in accordance
with aspects of the present invention. As described, the input device
201
may be used to generate electronic ink by detecting movement of the pen using,
for example, a camera. Camera
321 may be included to capture images of the
surface over which the pen is moved, and through image analysis, detect the amount
of movement of the pen over the surface being scanned. The movements may be correlated
with the document and electronically transpose, add, or associate (e.g. store input
annotations apart from the original document) electronic ink to the document).
As shown in FIG. 3A, in one embodiment, camera
304 includes an image sensor
320 comprised of, for example, an array of image sensing elements. For example,
the camera may be comprised of a CMOS image sensor with the capability of scanning
a 1.79 mm by 1.79 mm square area at a resolution of 32 pixels by 32 pixels. The
minimum exposure frame rate for one such image sensor may be approximately 330
Hz, while the illustrative image sensor may operate at a processing rate of 110
Hz. The image sensor selected may comprise a color image sensor, a grayscale image
sensor, or may operate to detect intensities exceeding a single threshold. However,
selection of the camera or its component parts may vary based on the desired operating
parameters associated with the camera, based on such considerations as performance,
costs or other considerations, as may be dictated by such factors as the resolution
required to accurately calculate the location of the input device.
A light source,
321, may illuminate the surface over which the input device
is moved. The light source may, for example, be comprised of a single light emitting
diode (LED), an LED array, or other light emitting devices. The light source may
produce light of a single color, including white, or may produce multiple colors.
A half mirror
322 may be included within the camera to direct light as desired.
The camera
304 may further include one or more optical devices
323
for focusing light from the light source
321 onto the surface scanned
324
and/or to focus the light reflected from that surface to the image sensor
320.
As illustrated in FIG. 3A, light emitted from light source
321 is reflected
by half-mirror
322, a mirror that reflects or transmits light depending
on direction of the impinging light. The reflected light is then directed through
lens system
323 and transmitted to the reflective surface below. The light
is then reflected off of that surface, through lens system
323, strikes
half-mirror
322 at a transmission angle passing through the mirror, and
impinges on sensing array
320. Of course, cameras including a wide range
of components may be used to capture the image data, including cameras incorporating
a lesser, or a greater, number of components. Variations in the arrangement of
components may also be numerous. To provide just one example, in simplified arrangement,
the light source and the sensing array may be positioned together such that they
both face the surface from which the image is to be captured. In that case, because
no reflections within the camera are required, the half-mirror may be removed from
the system. As shown in FIG. 3B, in a simplified configuration the light source
321 is positioned a distance from the lens
323 and sensor
320.
In further simplified arrangement, as shown in FIG. 3C, the light source may be
removed and ambient light reflecting off the object surface is focused by lens
323 onto the sensor
320.
Thus, variations in the components incorporated into the camera, or their placement,
may be employed in a manner consist with aspects of the present invention. For
example, the placement and/or orientation of the camera and/or cartridge may be
varied from that shown in FIG. 2 to allow for the use of a wide range of camera
and/or ink configurations and orientations. For example, camera
304, or
any of its component parts, may be located in openings adjacent that provided for
the ink cartridge, rather than within the same opening as illustrated. As an additional
example, camera
304 may be positioned in the center of the input device
with the ink cartridge positioned to the side of the camera. Similarly, the light
source
321 may be incorporated within the structure housing the remaining
components of the camera, or one or more components may be positioned separate
from the others. Furthermore, a light projecting feature may also be enabled, using
a light source and/or optical system, with additional structure and/or software,
or modifications to the illustrated components as necessary.
Active Coding
To aid in the detection and/or positioning of the input device, the surface of
an object over which the input device is positioned may include image data that
indicates the relative position of areas of the surface. In one exemplarily embodiment,
the surface being scanned may comprise the display of a host computer or other
external computing device, which may correspond to the monitor of a desktop computer,
a laptop computer, Tablet PC™, a personal data assistant, a telephone, digital
camera, or any device which may display information. Accordingly, a blank document
or other image generated on the screen of a Tablet PC™ may include data
corresponding to a code that represents the relative position of that portion of
the document within the entire document, or relative to any other portion of the
image. The information may be comprised of images, which may include alphanumeric
characters, a coding pattern, or any discernable pattern of image data that may
be used to indicate relative position. The image or images selected for use in
designating the location of areas within the surface of the object may depend on
the sensitivities of the scanning device incorporated into the camera, such as
the pixel resolution of the sensor, and/or the pixel resolution of the image data
contained within the surface being scanned. The location information extracted
from the object may then be used to track movement of the input device over the
object. Using that information, electronic ink or other information corresponding
to movement of the input device may be accurately generated. Location information
may be used to both detect the position within the image at which the input is
to be effected, as well as to provide an indication of movement of the input device
over the object surface. The resulting information may be used interactively with
word processing software to generate changes in a document, for example.
In an alternate embodiment, the object used in combination with the input device
may be composed of paper with positional information included in the background,
for example. The positional information may be incorporated in any form of code,
optical representation, or other form that may be sensed by a sensor associated
with the input device and used to represent the relative location of the specific
site on the paper.
FIG. 4 illustrates an illustrative technique for encoding the location of the
document. In this example, the background of the image may include thin lines that,
when viewed in large groups form a maze-like pattern. Each grouping of lines within
the maze design, comprised of a few thin lines with unique orientations and relative
positions, for example, may indicate the position of that portion of the maze pattern
relative to other portions of the document. Decoding of the maze pattern found
in a captured image may be performed in accordance with numerous decoding schemes.
In one embodiment, a particular arrangement and grouping of lines may be decoded
to generate positional information. In another embodiment, an indication of the
position of the captured data may be derived by extracting a code from the image
corresponding to the sampled pattern, and using that code to address a look-up
table containing data identifying the location of that area. Reference to the coding
technique employing a maze pattern is provided for illustrative purposes, and alternative
active coding techniques, including, but not limited to the visual coding techniques
in U.S. Ser. No. 10/284,412, entitled, "Active Embedded Interaction Code" invented
by Jian Wang, Qiang Wang, Chunhui Zhang, and Yue Li, whose contents are expressly
incorporated by reference for all essential subject matter, may also be used consistent
with aspects of the invention.
Passive Coding
Even in the absence of location codes, images captured by the image sensor may
be analyzed to determine the location of the input device at the time of image
capture. Successive images may be used to calculate the relative positions of the
input device at different times. Correlation of this information may yield an accurate
trace of the input device over the substrate. Using this trace information electronic
ink accurately representing handwritten strokes may be generated, for example.
FIG. 5 provides an illustration of a trace pattern from which electronic ink
may be generated. In this example, a first captured image may contain a portion
of a maze pattern indicative of a first position p
1 of the input device
at a first time, t
1. The next captured image may contain a portion of
the coded image data, a different portion of the maze pattern in this example,
providing location information of a second position p
2 at a second time,
t
2. A third captured image may contain a third portion of the maze pattern,
thereby indicating positioning of the input device at a third position p
3
at time t
3. Using this data, the three points may indicate
