Remote image display method, image capturing device, and method and program therefor Number:7,385,624 from the United States Patent and Trademark Office (PTO) owispatent An omnidirectional camera performs image sensing of a panoramic object 6 surrounding it, and sends the video signal to an image capturing device 1, then the image capturing device 1 extracts a video signal of a partial object 61 and sends the extracted video signal to a camera-equipped portable terminal 3 carried in user's hand at a place distant from the omnidirectional camera 2 to display the image of the video signal on a display surface 32a of the portable terminal, and the user shoots a surrounding object 8 with the portable terminal 3 and sends the surrounding video signal to the device 1. When the shooting direction of the portable terminal 3 is turned as indicated by the broken line, the device 1 detects a change in the shooting direction of he portable terminal 3 from a difference between the surrounding video signals generated before and after the turning of the portable terminal, and extracts, from a panorama video signal, a video signal of a partial object 62 turned from the partial object 61 in correspondence to the difference and sends the extracted video signal to the portable terminal 3 to display the image of the video signal.<H capturing, therefor, Remote, image, di, 7385624.html, Patent, pto, 7385624

Title: Remote image display method, image capturing device, and method and program therefor

Abstract: An omnidirectional camera performs image sensing of a panoramic object 6 surrounding it, and sends the video signal to an image capturing device 1, then the image capturing device 1 extracts a video signal of a partial object 61 and sends the extracted video signal to a camera-equipped portable terminal 3 carried in user's hand at a place distant from the omnidirectional camera 2 to display the image of the video signal on a display surface 32a of the portable terminal, and the user shoots a surrounding object 8 with the portable terminal 3 and sends the surrounding video signal to the device 1. When the shooting direction of the portable terminal 3 is turned as indicated by the broken line, the device 1 detects a change in the shooting direction of he portable terminal 3 from a difference between the surrounding video signals generated before and after the turning of the portable terminal, and extracts, from a panorama video signal, a video signal of a partial object 62 turned from the partial object 61 in correspondence to the difference and sends the extracted video signal to the portable terminal 3 to display the image of the video signal.

Patent Number: 7,385,624 Issued on 06/10/2008 to Takehara, et al.

Inventors: Takehara; Nobuhiko (Yokohama, JP), Watanabe; Tomoki (Yokohama, JP), Anzai; Hiroki (Yokohama, JP), Kishida; Katsumi (Yokosuka, JP)
Assignee: Nippon Telegraph and Telephone Corporation (Tokyo, JP)

Appl. No.: 10/510,954

Filed: January 15, 2004

PCT Filed: January 15, 2004

PCT No.: PCT/JP2004/000223

371(c)(1),(2),(4) Date: October 29, 2004

PCT Pub. No.: WO2004/066632

PCT Pub. Date: August 05, 2004

Foreign Application Priority Data


Jan 17, 2003 [JP]			2003-009277
Jan 17, 2003 [JP]			2003-009278

Current U.S. Class: 348/36 ; 348/155; 348/211.11; 348/211.3; 348/552

Field of Search: 348/36,39,155,211.3,211.11,218.1,222.1,552

References Cited [Referenced By]

U.S. Patent Documents


4786966	November 1988	Hanson et al.
6411275	June 2002	Hedberg
2001/0019355	September 2001	Koyanagi et al.
2002/0175896	November 2002	Vaananen et al.

Foreign Patent Documents


1278384	Dec., 2000	CN
94 00 003	May., 1995	DE
1 130 906	Sep., 2001	EP
06-078341	Mar., 1994	JP
11-205775	Jul., 1999	JP
2000-101992	Apr., 2000	JP
2002-077882	Mar., 2002	JP
2002-185952	Jun., 2002	JP
WO 99/12349	Mar., 1999	WO

Other References

Yamazawa, Kazumasa, "Principle and characteristics and omnidirectional camera with a mirror", Computer Vision and Image Media, vol. 125, No. 21, pp. 155-160, (with English abstract). cited by other .
Taniguchi, Yukinobu et al. "PanoramaExcerpts: Video Cataloging by Automatic Synthesis and Layout of Panoramic Images", IECEJ Journal D-II, vol. J82-D-II, No. 3, p. 390-398. cited by other .
Yamamori, Kazuhiko, "Future Network Technology Series, Media Processing Techniques 4", vol. 20, Denki-Tsuushin Kyoukai, 1.sup.st edition, pp. 208-211, 1999. cited by other.

Primary Examiner: Vu; Ngoc-Yen
Assistant Examiner: Meyers; James A
Attorney, Agent or Firm: Oblon, Spivak, McClelland, Maier & Neustadt, P.C.

Claims

What is claimed is:

1. A remote image display method, comprising: image sensing a panoramic object by a remote image sensing device; sending a video signal of a part of said image-sensed panoramic object via an image capturing device to an image display portion at a place different from said remote image sensing device; playing back and displaying said received video signal by said image display portion as an image of a part of said panoramic object; image sensing surroundings by a camera-equipped portable terminal at a position where a display on said image display portion can be seen, and sending a surrounding video signal to said image capturing device; obtaining variation information about surrounding image by a variation detecting portion in said image capturing devices, based on a previous surrounding video signal and a current surrounding video signal received from said camera-equipped portable terminal; obtaining from said remote image sensing device a video signal of a part of said panoramic object changed corresponding to a change of said surrounding image based on said variation information; and sending said obtained video signal to said image display portion.

2. An image capturing device comprising: a surrounding image receiving portion for receiving a surrounding video signal from a camera-equipped portable terminal; a variation detecting portion for detecting, from said received surrounding video signal, variation information of an image of the received surrounding video signal from an image of a previous surrounding video signal having been previously received by said surrounding image receiving portion, the image of the received surrounding video signal hereinafter being referred to as the current image, the image of the previous surrounding video signal hereinafter being referred to as the previous image; a capturing signal generating portion for generating a capturing signal from said variation information which is used to obtain a video signal of a part of a panoramic object; a signal sending portion for sending said capturing signal to a remote image sensing device that performs image sensing of a panoramic object and outputs a video signal; and an image relay portion for receiving said video signal from said remote image sensing device and for sending the received video signal to an image display portion at the same position as said camera-equipped portable terminal.

3. The device of claim 2, wherein: said variation detecting portion detects, as said variation information, direction information corresponding to a direction of movement of said current image with respect to said previous image or/and zoom information corresponding to a change in a size of a part of said current image corresponding to a part of said previous image; and said capturing signal generating portion generates, as said capturing signal, an extracting reference pixel position signal obtained by correcting a previous extracting reference pixel position signal according to said direction information or/and an image zoom in/out signal corresponding to said zoom information.

4. The device of claim 2, further comprising: a camera direction storage portion which stores identification information of each of plural camera devices provided in said remote image sensing device and angularly spaced apart in a shooting direction of the plural camera devices; and wherein: said variation detecting portion detects, as said variation information, direction information corresponding to a direction of movement of said current image with respect to said previous image; and said capturing signal generating portion determines the shooting direction by referring to the identification information stored in said camera direction storage portion based on a previously generated transmitting camera identification information signal and said direction information, for generating, as said capturing signal, a transmitting camera identification information signal of the camera identification information corresponding to said shooting direction determined by said capturing signal generating portion.

5. The device of claim 2, wherein: said variation detecting portion detects, as said variation information, direction information corresponding to a direction of movement of said current image with respect to said previous image or/and zoom information corresponding to a change in a size of said current image with respect to said previous image; and said capturing signal generating portion generates, as said capturing signal, a shooting direction change signal corresponding to said direction information or/and zoom change signal corresponding to said zoom information.

6. The device of claim 2, wherein: said surrounding image receiving portion receives the surrounding video signal from each of a plurality of camera-equipped portable terminals; and said variation detecting portion is provided with a previous frame memory for storing a surrounding video signal used for detecting the previous variation information for each camera-equipped portable terminal, for detecting variation information corresponding to said each camera-equipped portable terminal having transmitted the surrounding video signal, based on the received surrounding video signal and the surrounding video signal stored in said previous frame memory corresponding to said each camera-equipped portable terminal having transmitted the surrounding video signal; said capturing signal generating portion is provided with a previous signal memory for storing a previously generated capturing signal for said each camera-equipped portable terminal having transmitted the surrounding video signal, for generating a capturing signal corresponding to said each camera-equipped portable terminal from the variation information corresponding to said each camera-equipped portable terminal and said previously generated capturing signal stored in said previous signal memory corresponding to said each camera-equipped portable terminal, and for updating corresponding one of capturing signals in said previous signal memory with said generated capturing signal; said signal portion sends said generated capturing signal and camera-equipped portable terminal identification information for identifying the camera-equipped portable terminal corresponding thereto; and said image relay portion sends the received video signal to said image display means portion of the camera-equipped portable terminal indicated by the camera-equipped portable terminal identification information received together with said video signal.

7. An image capturing device comprising: a surrounding image receiving portion for receiving a surrounding video signal from a camera-equipped portable terminal; a variation detecting portion for detecting, from said received surrounding video signal, variation information of an image of the received surrounding video signal from an image of a previous surrounding video signal having been previously received by said surrounding image receiving means portion, the image of the received surrounding video signal hereinafter being referred to as the current image, the image of the previous surrounding video signal hereinafter being referred to as the previous image; a capturing signal generating portion for generating, from said variation information and a previously generated capturing signal, a capturing signal that is used to obtain a video signal of a part of a panoramic object; a remote image receiving portion for receiving a video signal sent from a remote image sensing device having performed image sensing of a panoramic object; an image capturing portion for capturing the video signal of a part of the panoramic object from said received video signal based on said capturing signal; and an image sending portion for sending said captured video signal to an image display portion at a same position as that of said camera-equipped portable terminal.

8. The device of claim 7, wherein: the video signal received by said remote image receiving portion is a panorama video signal; said variation detecting portion detects, as said variation information, direction information corresponding to a direction of movement of said current image with respect to said previous image or/and zoom information corresponding to a change in a size of a part of said current image corresponding to a part of said previous image; said capturing signal generating portion generates, as said capturing signal, an extracting reference pixel position signal obtained by correcting a previous extracting reference pixel position signal according to said direction information or/and an image zoom in/out signal corresponding to said zoom information; and said image capturing portion captures, from said panorama video signal, one frame of video signal produced by zooming in/out, by said image zoom in/out signal relative to the image of the previously sent captured video signal, a partial image of the panorama image of said panorama video signal that is defined by one frame of video signal or/and the previously generated extracting reference pixel position signal, or said corrected extracting reference pixel position signal with reference to the position on the panorama image that is determined by said corrected extracting reference pixel position signal.

9. The device of claim 7, wherein: the video signal received by said remote image receiving portion is video signals from a plurality of camera devices of said remote image sensing device with their shooting directions angularly spaced apart; and said image capturing device further comprises: a camera direction storage portion having stored herein identification information about said plurality of camera devices and information corresponding to the shooting directions; and said variation detecting portion detects, as said variation information, direction information corresponding to a direction of movement of said current image with respect to said previous image; said capturing signal generating portion determines a shooting direction by referring to the identification information stored in said camera direction storage portion based on a previously generated transmitting camera identification information signal and said direction information, for generating, as said capturing signal, a transmitting camera identification information signal of the camera identification information corresponding to said determined shooting direction; and said image capturing portion captures one of a plurality of video signals corresponding to said generated transmitting camera identification information signal.

10. The device of claim 9, wherein: said variation detecting portion includes a portion for detecting, as said variation information, second direction information corresponding to the direction of movement of said current image with respect to said previous image in a direction perpendicular to the direction of movement of said detected direction information or/and zoom information corresponding to a change in the size of the part of said current image corresponding to a part of said previous image; said capturing signal generating portion generates, as said capturing signal, an extracting reference pixel position signal obtained by correcting a previous extracting reference pixel position signal according to said second direction information or/and an image zoom in/out signal corresponding to said zoom information; and said image capturing portion captures, from said captured video signal, one frame of video signal produced by zooming in/out, by said image zoom in/out signal relative to the image of the previously sent captured video signal, a partial image of the image of said captured video signal that is defined by one frame of video signal or/and the previously generated extracting reference pixel position signal, or said corrected extracting reference pixel position signal with reference to the position on the panorama image that is determined by said corrected extracting reference pixel position signal.

11. The device of claim 7, wherein: said surrounding image receiving portion receives a surrounding video signal from each of a plurality of camera-equipped portable terminals; said variation detecting portion is provided with a previous frame memory for storing a surrounding video signal used for detecting the previous variation information for said each camera-equipped portable terminal having transmitted the surrounding video signal, for detecting variation information corresponding to said each camera-equipped portable terminal, based on the received surrounding video signal and the surrounding video signal stored in said previous frame memory corresponding to said camera-equipped portable terminal; said capturing signal generating portion is provided with a previous signal memory for storing a previously generated capturing signal for said each camera-equipped portable terminal having transmitted the surrounding video signal, for generating a capturing signal corresponding to said each detected camera-equipped portable terminal from the detected variation information corresponding to said each camera-equipped portable terminal and said previously generated capturing signal having been stored in said previous signal memory in correspondence to said each camera-equipped portable terminal, and for updating the corresponding to signal in said previous signal memory with said generated capturing signal; said image capturing portion captures a video signal, for each capturing signal stored in said previous signal memory in correspondence to each camera-equipped portable terminal; and said image sending portion sends said captured video signal for said each camera-equipped portable terminal to an image display portion at a same position as that of said each camera-equipped portable terminal.

12. The device of claim 2 or 7, further comprising: a variation hysteresis storage portion for storing hysteresis information of said variation information; a decision portion for deciding whether the variation information detected by said variation detecting portion is abnormal or not by referring to the hysteresis information stored in said variation hysteresis storage portion; and a portion which, when said decision portion decides that said variation information is abnormal, inhibits a supply of said detected variation information to said capturing signal generating portion.

13. A processing method of an image capturing device, comprising: a first step of deciding whether an operation command for changing the shooting direction of a camera or/and a lens field angle is received or not, the operation hereinafter being referred to as camera operation, the lens field angle being referred to as a zoom amount; a second step of receiving a surrounding video signal of a surrounding object from a camera-equipped portable terminal when it is decided in said first step that said command is received; a third step of detecting, from the received surrounding video signal, variation information indicating a variation of an image of the surrounding video signal from an image of the previous surrounding video signal; a fourth step of generating a capturing signal for capturing a video signal of a part of a panoramic object based on a previous capturing signal and said variation information, the part of a panoramic object hereinafter being referred to as a partial object; a fifth step of sending said generated capturing signal to a remote image sensing device which performs image sensing of said panoramic object; a sixth step of receiving a video signal from the remote image sensing device; a seventh step of sending said received video signal to an image display portion at a same position as that of said camera-equipped portable terminal; and an eighth step of deciding whether a camera operation stop command is received; wherein when the operation command is not received in said third step, processing proceeds to said sixth step, and when the stop command is not received in said eighth step, processing returns to said first step.

14. A processing method of an image capturing device, comprising: a first step of deciding whether an operation command for changing the shooting direction of a camera or/and a lens field angle is received or not, the operation hereinafter being referred to as camera operation, the lens field angle being referred to as a zoom amount; a second step of receiving a surrounding video signal of a surrounding object from a camera-equipped portable terminal when it is decided in said first step that said command is received; a third step of detecting, from the received surrounding video signal, variation information indicating a variation of an image of the surrounding video signal from an image of the previous surrounding video signal; a fourth step of generating a capturing signal for capturing a video signal of a part of a panoramic object based on a previous capturing signal and said variation information, the part of a panoramic object hereinafter being referred to as a partial object; a fifth step of receiving a video signal from a remote image sensing device; a sixth step of capturing a video signal of a part of a panoramic object from said received video signal based on said generate capturing signal; a seventh step of sending said captured video signal to an image display means portion at a same position as that of said camera-equipped portable terminal; and an eighth step of deciding whether a camera operation stop command is received; wherein when the operation command is not received in said first step, processing proceeds to said fifth step, and when the stop command is not received in said eighth step, processing returns to said third step.

15. An image capturing device comprising: a surrounding image receiving portion for receiving a surrounding video signal from each of a plurality of camera-equipped portable terminals; a variation detecting portion, provided with a previous frame memory for storing a surrounding video signal, for detecting variation information corresponding to said each camera-equipped portable terminal having transmitted the surrounding video signal, based on the received surrounding video signal and the surrounding video signal stored in said previous frame memory corresponding to said each camera-equipped portable terminal having transmitted the surrounding video signal; a capturing signal generating portion provided with a previous signal memory for storing a previously generated capturing signal for each piece of user terminal identification information identifying the camera-equipped portable terminal having sent the said variation information signal, for generating, from the detected variation information signal and the previously generated capturing signal of the user terminal identification information stored in the previously memory, a capturing signal to be used for capturing a video signal of a part of a panoramic object, the part of the panoramic object hereinafter being referred to as a partial object; a signal sending portion for sending said generated capturing signal and the corresponding user terminal identification information to a remote image sensing device which performs image sensing of the panoramic object and outputs its video signal; and an image relay portion for receiving the video signal and the corresponding user terminal identification information from said remote image sensing device and sending the video signal to an image display portion of the camera-equipped portable terminal corresponding to the user terminal identification information.

16. The device of claim 15, wherein: said variation information signal represents direction information or/and zoom information of the partial object desired to switch from a previous partial object; and said capturing signal generating portion generates, as said capturing signal, an extracting reference pixel position signal obtained by correcting a previous extracting reference pixel position signal according to said direction information or/and an image zoom in/out signal corresponding to said zoom information.

17. The device of claim 15, further comprising: a camera direction storage portion which stores identification information of each of plural camera devices provided in said remote image sensing device and angularly spaced apart in their shooting direction; wherein: said variation information signal represents first direction information of the partial object desired to change from the previous partial object in a first direction, and/or second direction information of the partial object desired to change from the previous partial object in a second direction perpendicular to the first direction information signal and zoom information of the desired partial object; and said capturing signal generating portion determines the shooting direction by referring to the identification information stored in said camera direction storage portion based on a previously generated transmitting camera identification information signal and said first direction information, for generating, as said capturing signal, a transmitting camera identification information signal of the camera identification information corresponding to said determined shooting direction, and/or a reference pixel position signal obtained by correcting said generated reference pixel position corresponding to said second direction information or/and an image zoom in/out signal corresponding to said zoom information.

18. An image capturing device comprising: a surrounding image receiving portion for receiving a surrounding video signal from each of a plurality of user terminals; a variation detecting portion, provided with a previous frame memory for storing a surrounding video signal, for detecting variation information corresponding to said each camera-equipped portable terminal having transmitted the surrounding video signal, based on the received surrounding video signal and the surrounding video signal stored in said previous frame memory corresponding to said each camera-equipped portable terminal having transmitted the surrounding video signal; a capturing signal generating portion provided with a previous signal memory for storing previously generated capturing signals for each piece of user terminal identification information identifying the user terminal having sent the said variation information signal, for generating, from the received variation information signal and the previously generate capturing signal of the user terminal identification information stored in the previously memory, a capturing signal to be used for capturing a video signal of a part of a panoramic object, the part of the panoramic object hereinafter being referred to as a partial object; a signal sending portion for sending said generated capturing signal to a remote image sensing device which performs image sensing of the panoramic object and outputs its video signal; a remote image portion for receiving the video signal sent from the remote image sensing device having performed image sensing of a panoramic object; an image capturing portion for capturing the video signal of a part of the panoramic object from said received video signal based on the capturing signal for each user terminal identification information stored in said previous signal memory; and an image sending portion for sending said captured video signal to the user terminal corresponding to the user terminal identification information.

19. The device of claim 18, wherein: the video signal received by said remote image receiving portion is a panorama video signal; said variation information signal represents direction information or/and zoom information for the partial object desired to switch from the previous partial object; said capturing signal generating portion generates, as said capturing signal, an extracting reference pixel position signal obtained by correcting a previous extracting reference pixel position signal according to said direction information or/and an image zoom in/out signal corresponding to said zoom information; and said image capturing portion captures, from said panorama video signal, one frame of video signal produced by zooming in/out, by said image zoom in/out signal relative to the image of the previously sent captured video signal, a partial image of the panorama image of said panorama video signal that is defined by one frame of video signal or/and the previously generated extracting reference pixel position signal, or said corrected extracting reference pixel position signal with reference to the position on the panorama image that is determined by said corrected extracting reference pixel position signal.

20. The device of claim 18, wherein: the video signal received by said remote image receiving portion is video signals from a plurality of camera devices of said remote image sensing device with their shooting directions angularly spaced apart; and the device further comprises: a camera direction storage portion having stored herein identification information about said plurality of camera devices and information corresponding to their shooting directions; wherein: said variation information signal represents first direction information of the partial object desired to change from the previous partial object or/and a second direction information of the desired partial object in a direction perpendicular to that of the first direction information or/and zoom information of the desired partial object; said capturing signal generating portion determines the shooting direction by referring to the identification information stored in said camera direction storage means based on a previously generated transmitting camera identification information signal and said first direction information, for generating, as said capturing signal, a transmitting camera identification information signal of the camera identification information corresponding to said determined shooting direction, and/or a reference pixel position signal obtained by correcting said generated reference pixel position corresponding to said second direction information or/and an image zoom in/out signal corresponding to said zoom information; and said image capturing portion captures that one of a plurality of video signals which corresponds to said generated camera identification signal, and for capturing, from said captured video signal, one frame of video signal produced by zooming in/out, by said image zoom in/out signal relative to the image of the previously sent captured video signal, a partial image of the image of said captured video signal that is defined by one frame of video signal or/and the previously generated extracting reference pixel position signal, or said corrected extracting reference pixel position signal with reference to the position on the panorama image that is determined by said corrected extracting reference pixel position signal.

21. A computer readable recording medium embedded with a program for functioning a computer as said image capturing device described in any one of claims 2, 7, 15 and 18.

Description

TECHNICAL FIELD

The present invention relates to a remote image display method for image sensing of a panoramic object by an image sensor, that is, by a camera (hereinafter referred to simply as camera) that outputs video signals, and for displaying the captured images to a user at a place remote from the camera; the invention also pertains to an image capturing device and method therefor, and an image capturing program.

BACKGROUND ART

For the purpose of observing the traffic on the road, it is now a general practice to set a panhead camera to monitor the traffic conditions; video signals obtained by shooting with the camera are sent over a communication line to a monitor set placed at a remote location, where the received video signal are played back to display the traffic conditions on a display. The watcher manipulates, while watching the display, an operating button or lever of a remote controller to send a control signal for changing the shooting direction (horizontal and vertical directions) of the monitor camera, or/and a control signal for changing the zoom amount of a zoom lens of the camera over the communication line to a control mechanism of the monitor camera to control it for changing its shooting direction and shooting condition (refer to, for example, Japanese Patent Application Koukai publication No. 205775/99 (published on Jul. 30, 1999).

There is offered a service what is called video live through a panhead camera set on the roof of a high-rise building; a user accesses the camera from his personal computer via the Internet or similar communication network to occupy the camera for a predetermined time, receives shooting video signals of the camera by the personal computer to play back the video signals to provide a display their images on the display, and while watching the image on the display, the user operates the personal computer to remotely control the camera shooting direction or the zoom amount of the lens to enjoy seeing respective parts of a wide-range panoramic object.

It is possible for a parent to control a personal computer at his home or office to access a camera set up in kindergarten to play back video signal from the camera for display on the personal computer and to remotely control the shooting direction or zoom amount of the camera to observe his child's movements under various circumstances.

Further, when video signals from a security camera set up in a convenience store are played back to display the images on a display at a remote place, it is possible for a watchman to remotely control the shooting direction or zoom amount of the security camera while watching the images being displayed.

Such remote control of a camera to display video signals therefrom can be used for various purposes. In the prior art, the shooting direction of the camera itself is changed by mechanical control, or the control for mechanically moving the zoom lens is effected by a button (key) operation or level operation using a personal computer. Moreover, while one user remotely controls the camera, other users cannot controls the camera; that is, the camera is occupied by one user for a predetermined time, for instance.

Since the control for obtaining a desired image by a conventional remote image display scheme is remote control of the camera itself by operating a button (key) or lever, the operability of the camera is poorer than in the case of manually operating the camera in one's hand or on the panhead to obtain a desired image, that is, the image thus obtained becomes a little different from the desired one in some cases.

An object of the present invention is to provide a remote image display method that permits displaying a desired remote image in the same way a user directly operates the camera by his hand, and an image capturing device and method therefor, and an image capturing program.

DISCLOSURE OF THE INVENTION

According to the remote image display method of the present invention: a remote image sensing device performs image sensing of a panoramic object, and sends a video signal of a part of image-sensed panoramic object via an image capturing device to image display means at a place different from the remote image sensing device; the image display means lays back and display the received video signal as an image of a part of said panoramic object; a camera-equipped portable terminal at a position where the display on said image display means can be seen performs image sensing of its surroundings, and the surrounding video signal to said image capturing device; the image capturing device obtains variation information about the surrounding image from the camera-equipped portable terminal based on a previous received surrounding video signal and the current received surrounding video signal, then obtains from the remote image sensing device a video signal of a part of the panoramic object changed corresponding to a change of the surrounding image based on the variation information, and sends the obtained image signal to the image display means.

The panoramic object is sufficiently wide to such an extent as not to be suitable for displaying as an image of one display frame of the image display means, or it can be displayed as the image of one display frame to some extent but it is equal to the range of a visual angle over which each part of the display needs to be seen in detail, and the field angle is 360 degree or less, and the images to be displayed need not always be continuous. There are two schemes for the image capturing device to obtain the video signal of a changed part of the panoramic object from the remote image sensing device. According to the one scheme: variation information on the image of the current surrounding video signal (hereinafter referred to as the current image) with respect to the image of the previous surrounding video signal (hereinafter referred to as the previous image) received from the camera-equipped portable terminal is detected by variation detecting means from the received surrounding video signal; a capturing signal for obtaining a video signal of a part of the panoramic object is generated by capturing signal generating means based on the detected variation information; the generated capturing signal is sent to the remote image sensing device; and the video signal received by the remote image sensing device is sent by image relay means to the image display means. With this scheme, the image capturing device sends the capturing signal to the remote image sensing device and receives the video signal of the changed part of the panoramic object from the remote image sensing device to thereby obtain an image of the video signal of the changed part of the panoramic object.

According to the other scheme, the user operates the camera-equipped portable terminal in his hand to shoot a part of the panoramic object while watching the image of a part of the panoramic object displayed on the image display means, by which it is possible to remotely shoot and display a desired part of the panoramic object; in this case, the camera can be operated in the same way the user himself operates the camera at the place where he can see the panoramic object directly. Hence, operability of the camera is excellent and no deviation is hard to occur between the desired part of the panoramic object and the displayed image.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram illustrating an example of the system configuration for explaining the principles of the present invention.

FIG. 2 is a diagram for explaining an image sensor device of an omnidirectional video camera and the extraction of an image signal of an each partial object.

FIG. 3A is a diagram showing, by way of example, how captured images move on the image sensor when a camera-equipped portable terminal is turned in the direction of taking pictures.

FIG. 3B is a diagram showing images before and after they move in FIG. 3A.

FIG. 4 is a block diagram illustrating an example of the system configuration of Embodiment 1.

FIG. 5 is a diagram depicting an example of the functional configuration of each of variation detecting means 12 and capturing signal generating means 13 in FIG. 4.

FIG. 6 is a diagram showing an example of the operation procedure of the entire system shown in FIG. 4.

FIG. 7 is a flowchart showing an example of the procedure of an image capturing device 1 in FIG. 4.

FIG. 8 is a block diagram illustrating an example of the system configuration of Embodiment 2.

FIG. 9 is a flowchart showing an example of the procedure of the image capturing device 1 in FIG. 8.

FIG. 10 is a diagram, by way of example, the relationships between a remote image sensing device 2 using plural camera devices and a panoramic object 6.

FIG. 11 is a block diagram illustrating an example of the system configuration of Embodiment 3.

FIG. 12 is a diagram for explaining an image sensor device of the camera device in FIG. 11 and the extraction of a captured signal of an partial object.

FIG. 13 is a block diagram showing a modified form of the remote image capturing device 2 in FIG. 11.

FIG. 14 is a diagram showing an example of a concrete functional configuration of capturing signal generating means 13 and an example of stored contents of camera direction storage means 19 in FIG. 11.

FIG. 15 is a flowchart showing an example of the procedure of a camera determining part 13b in FIG. 14.

FIG. 16 is a diagram showing another example of the stored contents of the camera direction storage means 19.

FIG. 17 is a block diagram illustrating an example of the system configuration of Embodiment 4.

FIG. 18 is a block diagram illustrating an example of the system configuration of Embodiment 5.

FIG. 19 is a diagram showing an example of the outward appearance of a panhead camera in FIG. 18.

FIG. 20 is a block diagram illustrating an example of the system configuration of Embodiment 6.

FIG. 21 is a block diagram illustrating an example of he functional configuration of the image capturing device 1 in FIG. 20.

BEST MODE FOR CARRYING OUT THE INVENTION

Description of Principles of the Invention

The principles of the present invention will be described with reference to FIG. 1.

This example uses an omnidirectional camera as a remote image sensing device 2. The omnidirectional camera is a digital video camera, which is an omnidirectional image sensor capable of image sensing of objects 360-degree surroundings; refer to, for example, Kazumasa Yamazaki, "Principle and Characteristics of Omnidirectional camera with a mirror," Computer Vision and Image Media, 125-21, P. 155-460, Jan. 19, 2001. The omnidirectional camera is commercially available; by which images of a 360-degree panoramic object, with a predetermined reference direction at zero degree as shown in FIG. 2, for instance, are formed on a rectangular two-dimensional image sensor device 7, such as CCD (Charge Coupled Device), from which photoelectric conversion signals corresponding to respective pixels can be extracted.

In FIG. 1 the remote image sensing device 2 is capable of image sensing of a 360-degree surrounding panoramic object 6. In this example a captured video signal of a part of the panoramic object by the remote image sensing device 2, for example, a part (hereinafter referred to as a partial object) 61 in a direction 180 degrees away from the reference, that is, each pixel signal of an area 71 in the image sensor device 7 in FIG. 2 is extracted and sent over a transmission line 4 to the image capturing device 1.

The image capturing device 1 sends the received captured video signal of the partial object over a transmission line 5 to a camera-equipped portable terminal 3 that is located at a place different from the remote image sensing device 2 and serves also as image display means. This example uses a camera-equipped portable telephone as the camera-equipped portable terminal 3, but it may be a camera-equipped PDA (Personal Digital Assistance) or camera-fixed PDA terminal as long as it is equipped with a digital video camera and communicating means and can be manually operated to take pictures. To distinguish between the transmission lines 4 and 5, they will hereinafter be referred to as a remote transmission line 4 and a portable-associated transmission line 5, respectively. Both of the transmission lines 4 and 5 may be dedicated transmission lines or transmission lines that pass through one or more communication networks, such as the Internet, or wireless/wire public telecommunication network.

Incidentally, in this example, the camera-equipped portable terminal 3 is also provided with image display means, and the captured video signal of the partial object 61, sent from the remote image sensing device 2 and received from the image capturing device 1, is played back by the image display means and displayed on its display surface 32a.

A user holds the camera-equipped portable terminal 3 in his hand and takes pictures of a surrounding object 8 by a camera (portable image sensing means) 33 of the terminal 3; as he sees the partial object 61 in the panoramic object 6 displayed on the display surface 32a, if he wants to see another partial object in the panoramic object 6 that he wishes to display at a remote place, then the user operates the camera-equipped portable terminal 3 to direct its camera 33 to take pictures of the partial object desired to see. For example, in the case of taking pictures of a partial object 62 in a direction 45 degree away to the right from the partial object 62, the direction 33a of the camera 33 of the camera-equipped portable terminal 3 is turned 45 degrees to the right as indicated by the one-dot chain line in FIG. 1.

The video signal of the surrounding object being image sensed by the camera 33 of the camera-equipped portable terminal 3 (which video signal will hereinafter be referred to as a surrounding video signal) is sent over the portable-associated transmission line 5 to the image capturing device 1. The image capturing device 1 detects a change in the surrounding image being shot by the camera-equipped portable terminal 3 from the previous and current surrounding video signals.

The range over which the camera-equipped portable terminal 3 is shooting the surrounding object 8 is a shooting range 8b shifted to right from that 8a as shown in FIG. 3A. Accordingly, the received surrounding video signal at the time of seeing the reproduced image of the partial object 61 is the video signal of the shooting range 8a shown at the upper side of FIG. 3B, but the received surrounding video signal with the shooting direction of the camera-equipped portable terminal 3 turned 45 degrees to right is the video signal of the shooting range 8b shown at the lower side of FIG. 3B. Comparison of the both received surrounding video signals indicates that the same image part shifts from right to left in the drawing, from which it is possible to detect that the shooting direction of the camera-equipped portable terminal 3 has been turned to right.

The image capturing device 1 detects variation information about the received surrounding video signal with respect to the previous one, then based on the variation information, changes the video signal to be extracted from the video signal being shot by the remote image sensing device, and instructs the remote image sensing device 2 so that the partial object in the panoramic object 6 corresponding to the extracted video signal changes corresponding to the change in the image of the received surrounding video signal. In the case where the shooting direction of the camera-equipped portable terminal 3 is turned 45 degrees to right in the above example, the image capturing device 2 instructs the remote image sensing device 2 via the remote transmission line 4 to extract the video signal corresponding to the partial object 62 to which the partial object 61 of the panoramic object 6 has turned 45 degrees to right and to extract the video signal of the area 72 to which the area 71 on the image sensor device 7 in FIG. 2 has turned 45 degrees to right.

Upon receiving the instruction, the remote image sensing device 2 extracts the video signal from the area 72 of the image sensor 7, and sends the extracted video signal to the image capturing device 1 via the remote transmission line 4, and the image capturing device 1 sends the received video signal to the camera-equipped portable terminal 3 via the portable-associated transmission line 5. The camera-equipped portable terminal 3 displays on its display surface 32a the reproduced image corresponding to the partial object 62.

The remote image sensing device 2 always shoots the panoramic object 6 and responds to the capturing instruction from the image capturing device 1 to send at all times the video signal of the partial object via the image capturing signal to the camera-equipped portable terminal 3 at the rate of 30 frames per second, for instance. The camera-equipped portable terminal 3 also sends its surrounding video signal to the image capturing device 1 at all times.

As will be seen from the above, according to the method of the present invention, remote shooting and displaying of a desired partial object can be achieved by operating the camera-equipped portable terminal 3 to shoot the part of the panoramic object 6 desired to display while directly watching the panoramic object--this provides enhanced operability of the camera and makes it unlikely to incur the deviation between the desired partial object and the reproduced image of its video signal. For example, in the case of remotely controlling the camera by operating keys of a portable terminal such as a portable telephone, the operability is poor because of the key operation, and since the keys are small and closely packed, an erroneous operation is likely to occur, but the method of the present invention is free from such a possibility.

Embodiment 1

FIG. 4 illustrates in block form an example of the system configuration of Embodiment 1 of the present invention. In this application the same reference numerals denote parts of the same function and no duplicate description will be given of them.

The image capturing device 1 is provided with: surrounding image receiving means 11 that receives a surrounding video signal sent from the camera-equipped portable terminal 3 over the portable-associated transmission line 5; variation detecting means 12 that detects a variation in the image of the received surrounding video signal as variation information; capturing signal generating means 13 that generates a capturing signal based on the detected variation information; sending means 14 which sends the generated capturing signal to the remote image sensing device 2 over the remote transmission line 4; and image relay means 15 that receives the captured video signal sent from the remote image sensing device 2 over the remote transmission line 4 and passes it to the camera-equipped portable terminal 3 over the portable-associated transmission line 5.

The remote image sensing device 2 is provided with: an omnidirectional camera 21 as remote image sensing device means for image sensing of a panoramic object; signal receiving means 23 that receives the capturing signal sent from the image capturing device 1 over the remote transmission line 4; image extracting means 24 that extracts the captured video signal of a partial object from the omnidirectional camera 21; and image sending means 22 that sends the captured video signal to the image capturing device 1 over the remote transmission line 4.

The camera-equipped portable terminal 3 is provided with: image receiving means 31 that receives the captured video signal sent from the remote image sensing device 2 over the portable-associated transmission line 5; image display means 32 that plays back and displays the received captured video signal; a digital video camera 33 that is portable image sensing means for image sensing of surrounding objects; and sending means 34 that sends the surrounding video signal fed thereto from the camera 33 to the image capturing device 1 over the portable-associated transmission line 5.

The image extracting means 24 of the remote image sensing device 2 extracts from the omnidirectional camera 21 a captured video signal of one frame that is determined by the received capturing signal. For example, based on the capturing signal, the image capturing means extracts a sequence of signals corresponding to pixels (x+a.sub.pi, y+a.sub.pi) (where i=0, .+-.1, . . . ,.+-.I, J=0,.+-.1, . . . ,.+-.J, 2I being the number of x-direction pixels corresponding to one frame and 2J the number of y-direction pixels corresponding to one frame) about a pixel position (x.sub.1, y.sub.1) in the image sensor device shown in FIG. 2, and the thus extracted signal is output as a captured video signal of the area 71. That is, in this example the center position of the partial object desired to extract and the zoom amount are specified by the capturing signal. In this example the capturing signal is composed of reference pixel position signals x, y and a zoom in/out signal a.sub.p.

The captured video signal is sent via the image capturing device to the camera-equipped portable terminal 3, and the image of the partial object in the panoramic object being image sensed by the remote image sensing device 2, in this example, the image of the partial object 62 in FIG. 1, is displayed on the image display means 32 of the camera-equipped portable terminal 3.

The variation detecting means 12 of the image capturing device 1 detects, from the received surrounding video signal, information about a variation between the image of the current video signal (which image will hereinafter be referred to as the current image) and the image of the previous video signal received from the camera-equipped portable terminal 3. For example, as depicted in FIG. 5, the received surrounding video signal is stored in a receiving buffer memory 12a, before which the surrounding video signal previously received and stored in the receiving buffer memory 12a is stored in a previous frame memory 12b. The surrounding video signals stored in the memories 12a and 12b are both input to a variation detecting part 12c. The variation detecting part 12c detects the direction of movement from the previous image to the current one or the degree of movement (distance) or/and a change in the size of the object of the current image with respect to the previous image, i.e., the zoom amount, are detected. The variation information can be detected, for example, by techniques for analyzing the camera operation during shooting, that is, what is called camera work. The camera work analysis technique is disclosed, for example, in Yukinobu TANIGUCHI, et at., "PanoramExcerpts: Video Cataloging by Automatic Synthesis and Layout of Panoramic Images," IECEJ Journal D-II, Vol. J82-D-II No. 3, PP. 390-398, March 1999 and a book supervised by Kenji KOGURE, written by Kazuhiko YAMAMORI, "Future Network Technology Series, Media Processing Techniques 4," Denki-Tsuushin Kyoukai, First Edition, Nov, 10, 1999.

A description will be given of an example of the scheme for analyzing camera works of side-to-side operation (pan), up-and-down operation (tilt) and operation of changing the angle of view by the camera lens (zoom). Let the previous image f(x, y) and the current image by f'(x', y'), where (x',y')=(ax+d.sub.x,ay+d.sub.y) that is, let a, d.sub.x and d.sub.y represent parameters for explaining zoom, pan and tilt, respectively. The parameters a, d.sub.x and d.sub.y are calculated which minimize the following square error between the images f(x, y) and f'(x', y'): (1/N).SIGMA..sub.x,y{f(x,y)-f'(x',y')}.sup.2. These parameters a, d.sub.x and d.sub.y are used as the variation information. In other words, the variation detecting part 12c calculates, for example, a, d.sub.x and d.sub.y to obtain the variation information; that is, a is zoom in/out information, and dx and d.sub.y direction information. In the case of detecting only the side-to-side turning of the camera-equipped portable terminal 3, the parameters need only to be set at a=1 and d.sub.y=0; in the case of detecting only the up-and-down turning action, a=1 and d.sub.x=0; in the case of detecting only a change in the zoom, d.sub.x=d.sub.y=0; and in the case of detecting only the turning action, a=1.

The capturing signal generating means 13 generates, from the variation information detected by the variation detecting means 12 and a previous capturing signal, a capturing signal that is used to obtain the captured video signal of the partial object of the panoramic objection. For example, as shown in FIG. 5, in the capturing signal generating means 13, capturing signals x, y and a.sub.p, for instance, generated previously and currently used are stored in a previous signal memory 13a; these signals x, y and a.sub.p and the variation information, for example, d.sub.x, d.sub.y and a.sub.x, are calculated by x+d.sub.x.fwdarw.x in an adding part 13b, y +d.sub.y.fwdarw.y in an adding part 13c, and a.sub.p.times.a.fwdarw.a.sub.p in a multiplying part 13d, and they are complemented to generate new capturing signals x, y and a.sub.p, which are provided to the previous signal memory 13a to update the capturing signals stored therein, and at the same time, they are output to the sending means 14. While the image capturing device 1 does not perform the image capturing processing, there stored in the previous signal memory 13a, as initial values, predetermined capturing signals, for example, the center, x, y, of the sensor device in FIG. 2 and a half value, a.sub.p, of the maximum zoom amount; upon the image capturing device 1 starting the capturing processing, the initial value signals x, y and a.sub.p in the previous signal memory 13a are sent as capturing signals to the remote image sensing device 2.

When the value of the variation information is small, for example, when d.sub.x=1, the image of the partial object of the panoramic object displayed on the camera-equipped portable terminal 3 remains substantially unchanged; to cope with this, it is advisable to provide round-off pats 13e, 13f and 13g, by which if d.sub.x, d.sub.y and a are smaller than their predetermined values, d.sub.x and d.sub.y are rounded off to 0s and a to 1, which are provide to the adding parts 13b and 13c and the multiplying part 13d. The above-mentioned predetermined values set in the discarding parts 13e, 13f and 13g differ among the discarding parts and according to the mode of use of this remote display system. In the case of displaying a partial object in a distant scene, the predetermined values are chosen relatively large, and in the case of using the system to prevent crime in a relatively space, they are chosen relatively small; as for d.sub.x, for instance, it is considered to be rounded down to 0 when it is 5 or smaller at minimum.

In general, updating of the capturing signals to be sent to the remote image sensing device 2 upon each reception of one frame is not preferable since it increases the amount of processing, and even if updating is carried out for each frame, the user hardly perceives a change in the received partial-object image. Accordingly, the capturing signals may be sent to the remote image sensing device 2 at appropriate time intervals. This time interval may be determined properly based on the usage pattern of the system: for example, every 1/10 sec when conditions of the object change relatively rapidly as in the case of traffic monitoring; when the object is scenery, every 1/3 sec, for instance, and in some cases, every several seconds. To this end, an update instructing part 12d is provided, for example, in the variation detecting means 12, and the receiving buffer memory 12a responds to an instruction from the update instructing part 12d to capture the surrounding video signal of one frame at preset update time intervals, and the variation detecting part 12c detects variation information at the update time intervals. Alternatively, the contents of the receiving buffer memory 12a are updated for each frame of surrounding video signals and, upon each detection of variation information, the contents of the receiving buffer memory 12a are transferred to the previous frame memory 12b. When all pieces of variation information detected in the variation information detecting part 12c are zero, that is, when d.sub.x=d.sub.y=0 and a =1 in this example, it is not necessary to generate and send the capturing signals to the remote image sensing device, in which the capturing signals need not be generated, either. The capturing signals may selectively be generated and transmitted in correspondence only to those pieces of the variation information which have undergone changes.

In the remote image sensing device 2, the capturing signals received by the signal receiving means 23 are input to the image extracting means 24 as shown in FIG. 4, and the previously received capturing signals stored in a storage part 24a with the newly received capturing signals. Based on the thus updated capturing signals in the storage part 24a, the image extracting means 24 extracts the video signal of the partial object in the panoramic object being shot by the omnidirectional camera 21. For example, the output of the image sensor device 7 of the omnidirectional camera 21 changes, as shown in FIG. 2, from the state where video signals of a one-fram