Vehicle surroundings monitoring apparatus Number:7,436,982 from the United States Patent and Trademark Office (PTO) owispatent

Title: Vehicle surroundings monitoring apparatus

Abstract: A vehicle surroundings monitoring apparatus, includes: an infrared imaging device that captures an image; a binarized object extraction device that extracts a binarized object by binarizing the image; an extraction region setting device that sets, as an extraction region, a region adjacent to one of an upper end and a lower end of the binarized object extracted by the binarized object extraction device; a search region setting device that sets a search region around the extraction region; a similar object extraction device that, when an object having a shape similar to that of the object in the extraction region is present in the search region, extracts the object in the search region as a similar object; and an identical object determination device that determines whether or not the object in the extraction region and the similar object in the search region are identical.

Patent Number: 7,436,982 Issued on 10/14/2008 to Taniguchi,   et al.

---

Inventors:	Taniguchi; Fuminori (Shioya-gun, JP), Nagaoka; Hobuharu (Nasu-gun, JP), Tsuji; Takayuki (Utsunomiya, JP)
Assignee:	Honda Motor Co., Ltd. (Tokyo, JP)
Appl. No.:	11/150,732
Filed:	June 10, 2005

---

Foreign Application Priority Data

---

Jun 14, 2004 [JP]			2004-175668

Current U.S. Class:	382/104 ; 348/148; 382/291
Current International Class:	G06K 9/00 (20060101); G06K 9/36 (20060101); H04N 7/18 (20060101)
Field of Search:	382/103,104,209,216,282,291 348/148,149

---

References Cited [Referenced By]

---

U.S. Patent Documents

5555312	September 1996	Shima et al.
6531959	March 2003	Nagaoka et al.
7006667	February 2006	Akutagawa
2003/0091228	May 2003	Nagaoka et al.
2003/0137593	July 2003	Watanabe et al.
2003/0138133	July 2003	Nagaoka et al.
2004/0032971	February 2004	Nagaoka et al.
2004/0183906	September 2004	Nagaoka et al.

Foreign Patent Documents

	10 2004 012 811		Nov., 2004		DE
	11-328364		Nov., 1999		JP
	2001-006096		Jan., 2001		JP
	2001-028100		Jan., 2001		JP
	2001-351193		Dec., 2001		JP
	2001-351200		Dec., 2001		JP
	2003-016429		Jan., 2003		JP
	2003-134508		May., 2003		JP
	2003-216937		Jul., 2003		JP

Primary Examiner: LaRose; Colin
Attorney, Agent or Firm: Carrier, Blackman & Associates, P.C. Carrier; Joseph P. Blackman; William D.

---

Claims

---

What is claimed is:

1. A vehicle surroundings monitoring apparatus, comprising: an infrared imaging device that captures an image; a binarized object extraction device that extracts a binarized object by binarizing the image; an extraction region setting device that sets, as an extraction region, a region adjacent to one of an upper end and a lower end of the binarized object extracted by the binarized object extraction device; a search region setting device that sets a search region around the extraction region; a similar object extraction device that, when an object having a shape similar to that of the object in the extraction region is present in the search region, extracts the object in the search region as a similar object; and an identical object determination device that determines whether or not the object in the extraction region and the similar object in the search region are identical.

2. A vehicle surroundings monitoring apparatus according to claim 1, further comprising: another infrared imaging device; and a distance calculating device that, based on two images captured by the two infrared imaging devices, calculates a distance to the object contained in the images, wherein the identical object determination device determines that the object in the extraction region and the similar object in the search region are identical when a distance to the object in the extraction region is the same as a distance to the similar object in the search region.

3. A vehicle surroundings monitoring apparatus according to claim 1, wherein the search region setting device sets a direction to search the search region to one of an upward direction, a downward direction, and upward and downward directions of the extraction region.

4. A vehicle surroundings monitoring apparatus according to claim 1, wherein the search region setting device sets a plurality of search regions in the same direction, and the identical object determination device, when the ratio of the number of images from which the similar object is extracted is equal to or higher than a predetermined ratio, determines that the object in the extraction region and the similar objects in the search regions are identical.

5. A vehicle surroundings monitoring apparatus according to claim 1, wherein the extraction region setting device sets a right extraction region having a horizontal coordinate of a right side end of the binarized object extracted by the binarized object extraction device and a left extraction region having a horizontal coordinate of a left side end of the binarized object, the search region setting device sets the search regions around each of the right extraction region and the left extraction region, the similar object extraction device, when an object having a similar shape as an object in the left extraction region is present within the search region corresponding to the left extraction region, or when an object having a similar shape as the object in the right extraction region is present within the search region corresponding to the right extraction region, extracts the object having the similar shape in the search region as a similar object, and the identical object determination device determines whether or not the object in the left extraction region and the similar object in the search region corresponding to the left extraction region are identical, or whether or not the object in the right extraction region and the similar object in the search region corresponding to the right extraction region are identical.

6. A vehicle surroundings monitoring apparatus according to claim 1, further comprising a pedestrian recognition device that excludes a region that contains the object that has been determined as the identical object by the identical object determination device, and recognizes a pedestrian from a binarized object in a remaining region.

7. A vehicle surroundings monitoring apparatus according to claim 6, further comprising a warning issuing device that issues a warning concerning the pedestrian recognized by the pedestrian recognition device.

8. A vehicle surroundings monitoring apparatus, comprising: an infrared imaging device that captures an image; a binarized object extraction device that extracts a binarized object by binarizing the image; a pedestrian recognition device that recognizes a pedestrian in the binarized object extracted by the binarized object extraction device; an extraction region setting device that sets, as an extraction region, a region adjacent to one of an upper end and a lower end of the binarized object extracted by the binarized object extraction device; a search region setting device that sets a search region around the extraction region; a similar object extraction device that, when an object having a similar shape as the object in the extraction region is present in the search region, extracts the object in the search region and sets the object as a similar object; and an identical object determination device that determines whether or not the object in the extraction region and the similar object in the search region are identical.

9. A vehicle surroundings monitoring apparatus according to claim 8, wherein the vehicle surroundings monitoring apparatus comprises two of the infrared imaging devices, and a distance calculating device that, based on two images captured by the two infrared imaging devices, calculates a distance to the object contained in the images, and the identical object determination device, when a distance to the object in the extraction region is the same as a distance to the similar object in the search region, determines that the object in the extraction region and the similar object in the search region are identical.

10. A vehicle surroundings monitoring apparatus according to claim 8, wherein the search region setting device sets the search region to one of an upward direction, a downward direction, and upward and downward directions of the extraction region.

11. A vehicle surroundings monitoring apparatus according to claim 8, the search region setting device sets a plurality of the search regions in the same direction, and the identical object determination device, when a similar object is extracted by the similar object extraction device from a predetermined ratio of the plurality of search regions or higher, determines that the object in the extraction region and the similar object in the search region are identical.

12. A vehicle surroundings monitoring apparatus according to claim 8, wherein the extraction region setting device sets a right extraction region and a left extraction region as extraction regions, each of the right extraction region and the left extraction region respectively having a horizontal coordinate of the right-side end and the left-side end of the binarized object, that has been recognized as the pedestrian by the pedestrian recognition device, the search region setting device sets the search regions around each of the right extraction region and the left extraction region, the similar object extraction device, when an object having a similar shape as an object in the left extraction region is present within the search region corresponding to the left extraction region, or when an object having a similar shape as the object in the right extraction region is present within the search region corresponding to the right extraction region, extracts the object having the similar shape in the search region as a similar object, and the identical object determination device determines whether or not the object in the left extraction region and the similar object in the search region corresponding to the left extraction region are identical, or whether or not the object in the right extraction region and the similar object in the search region corresponding to the right extraction region are identical.

13. A vehicle surroundings monitoring apparatus according to claim 8, further comprising a warning issuing device that excludes a region that contains the object that has been determined as the identical object by the identical object determination device and issues a warning concerning a pedestrian that is present in a remaining region.

---

Description

---

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a vehicle surroundings monitoring apparatus in which object extraction is carried out by subjecting an image to a binarization processing.

Priority is claimed on Japanese Patent Application No. 2004-175668, filed Jun. 14, 2004, the content of which is incorporated herein by reference.

2. Description of Related Art

Vehicle surroundings monitoring apparatuses have been employed in which objects that may collide with the vehicle, such as pedestrians, are extracted from a picture around the vehicle captured by an infrared camera. More specifically, in the device, the captured infrared image is subjected to a binarization processing, and an area (binarized object) in which bright portions are concentrated is searched for. Then, it is determined whether or not the area is a head portion of a pedestrian by using the aspect ratio (ratio of length to width) and the sufficiency rate of the binarized object, and further calculating the distance between the vehicle and the binarized object using the actual surface area and the position of the centroid in the image. If the area of the head portion of a pedestrian is determined, an area which contains the body of the pedestrian is determined and set. These areas are displayed to be distinguished from the other regions of the image. In this manner, the position of the entire body of the pedestrian in the infrared image is determined, and this information is displayed for the driver so as to effectively assist the vision of the driver (refer to Japanese Unexamined Patent Application, First Publication No. H11-328364, for example).

However, when a vehicle is traveling, due to the influence in change in the geometry of a road ahead, or the pitching of the vehicle, the shape of the binarized object does not necessarily reflect the actual shape of the object.

Consequently, in order to provide an effective visual assistance to the driver, in addition to extracting pedestrians based on physical characteristics that can be distinguished relatively easily, such as a shape identification based on the presence of an object that has an equivalent height as a human height in the object image, a shape identification based on the presence of the head and the torso, and a shape identification based only on the head, or the like, that are used in device described in Japanese Unexamined Patent Application, First Publication No. H11-328364, it is necessary to extract artificial structures, such as road constructions, from a binarized object and then distinguish between objects other than pedestrians and pedestrians.

SUMMARY OF THE INVENTION

The present invention was conceived in view of the above-described background, and an object thereof is to provide a vehicle surroundings monitoring apparatus that can determine a binarized object having an indefinite shape extracted from an infrared image appropriately, and can accurately recognize objects around the vehicle.

In order to address the above-described issues, a vehicle surroundings monitoring apparatus according to a first aspect of the present invention is a vehicle surroundings monitoring apparatus, including: an infrared imaging device that captures an image; a binarized object extraction device that extracts a binarized object by binarizing the image; an extraction region setting device that sets, as an extraction region, a region adjacent to one of an upper end and a lower end of the binarized object extracted by the binarized object extraction device; a search region setting device that sets a search region around the extraction region; a similar object extraction device that, when an object having a shape similar to that of the object in the extraction region is present in the search region, extracts the object in the search region as a similar object; and an identical object determination device that determines whether or not the object in the extraction region and the similar object in the search region are identical.

In the vehicle surroundings monitoring apparatus having the above-described constitution, the binarized object extraction device extracts a binarized object from the image obtained by the infrared imaging device, and the extraction region setting device sets, as an extraction region, the region adjacent to one of the upper end and the lower end of the binarized object extracted by the binarized object extraction device. Then, when an object having a similar shape as the object in the extraction region set by the extraction region setting device is present in the search region set by the search region setting device around the extraction region, the similar object extraction device extracts that object in the search region having the similar shape as a similar object. Since a determination as to whether or not the object in the extraction region and the similar object in the search region are identical is made by the identical object determination device, continuous objects having the same shape can be recognized when the binarized object or an object in the extraction region in the upward and the downward direction thereof has the similar shape as that of an object in the search region.

According to a second aspect of the present invention, in the vehicle surroundings monitoring apparatus according to the first aspect, the vehicle surroundings monitoring apparatus further includes another infrared imaging device; and a distance calculating device that, based on two images captured by the two infrared imaging devices, calculates a distance to the object contained in the images, wherein the identical object determination device determines that the object in the extraction region and the similar object in the search region are identical when a distance to the object in the extraction region is the same as a distance to the similar object in the search region.

In the vehicle surroundings monitoring apparatus having the above-described constitution, when the distance to the object in the extraction region calculated by the distance calculating device that calculates the distance to an object from the parallax between the right image and the left image, and the distance to the similar object in the search region that is also calculated by the distance calculating device are the same, then it is determined that the objects are identical. Thus, a determination as of whether or not the object in the extraction region and the similar object in the search region are identical can be made more accurately since the identical object determination device determines that the object in the extraction region and the similar object in the search region are identical.

According to a third aspect of the present invention, in the vehicle surroundings monitoring apparatus according to the first aspect, the search region setting device sets a direction to search the search region to one of an upward direction, a downward direction, and upward and downward directions of the extraction region.

In the vehicle surroundings monitoring apparatus having the above-described constitution, the search region setting device sets the search region to one of the upward direction, the downward direction, and the upward and the downward direction of the extraction region, and the identical object determination device determines whether or not the binarized object in the extraction region and the similar object in the search region that has been set to one of the upward direction, the downward direction, and the upward and the downward direction of the extraction region are identical. Thus, continuous objects having the same shape arranged in the vertical direction can be recognized when the binarized object in the extraction region and the similar object in the search region that has been set to one of the upward direction, the downward direction, and the upward and the downward direction of the extraction region are similar.

According to a fourth aspect of the present invention, in the vehicle surroundings monitoring apparatus according to the first aspect, the search region setting device sets a plurality of search regions in the same direction, and the identical object determination device, when the ratio of the number of images from which the similar object is extracted is equal to or higher than a predetermined ratio, determines that the object in the extraction region and the similar objects in the search regions are identical.

In the vehicle surroundings monitoring apparatus having the above-described constitution, the search region setting device sets the plurality of search regions around the extraction region, and the similar object extraction device extracts the similar object. In this process, when the ratio of the number of images from which the similar object is extracted is equal to or higher than the predetermined ratio, the identical object determination device determines that the object in the extraction region and the similar objects in the plurality of search regions set around the extraction region are identical even when the similar objects are not present in all of the search regions. Thus, continuous objects having the same shape can be recognized when the object in the extraction region and the objects in the search regions have the similar shape.

According to a fifth aspect of the present invention, in the vehicle surroundings monitoring apparatus according to the first aspect, the extraction region setting device sets a right extraction region having a horizontal coordinate of a right side end of the binarized object extracted by the binarized object extraction device and a left extraction region having a horizontal coordinate of a left side end of the binarized object, the search region setting device sets the search regions around each of the right extraction region and the left extraction region, the similar object extraction device, when an object having a similar shape as an object in the left extraction region is present within the search region corresponding to the left extraction region, or when an object having a similar shape as the object in the right extraction region is present within the search region corresponding to the right extraction region, extracts the object having the similar shape in the search region as a similar object, and the identical object determination device determines whether or not the object in the left extraction region and the similar object in the search region corresponding to the left extraction region are identical, or whether or not the object in the right extraction region and the similar object in the search region corresponding to the right extraction region are identical.

In the vehicle surroundings monitoring apparatus having the above-described constitution, the extraction region setting device, as extraction regions, sets the right and the left extraction region each containing a region corresponding to the right side and the left side of the binarized object, respectively. Then, when an object having a shape similar to that of the object in the extraction regions set by the extraction region setting device is present in each search region corresponding to the right search region and the left search region set by the search region setting device, the similar object extraction device extracts that object in the search regions having the similar shape as a similar object. In this manner, since the identical object determination device determines whether or not an object in either the right extraction region or the left extraction region and the similar object in the search region corresponding to that extraction region are identical, continuous objects having the same shape in either the right end or the left end can be recognized when the object in either the right extraction region or the left extraction region and the similar object in the search region corresponding to that extraction region are similar.

According to a sixth aspect of the present invention, in the vehicle surroundings monitoring apparatus according to the first aspect, the vehicle surroundings monitoring apparatus further includes a pedestrian recognition device that excludes a region that contains the object that has been determined as the identical object by the identical object determination device, and recognizes a pedestrian from a binarized object in a remaining region.

In the vehicle surroundings monitoring apparatus having the above-described constitution, the pedestrian recognition device excludes a region that contains the object that has been determined as the identical object by the identical object determination device, and recognizes a pedestrian from a binarized object in the remaining region. In this manner, a pedestrian can be recognized from remaining objects other than the continuous objects having the same shape that cannot be a pedestrian.

According to a seventh aspect of the present invention, in the vehicle surroundings monitoring apparatus according to the sixth aspect, the vehicle surroundings monitoring apparatus further includes a warning output device that issues a warning concerning the pedestrian recognized by the pedestrian recognition device.

In the vehicle surroundings monitoring apparatus having the above-described constitution, since the warning issuing device issues a warning concerning the pedestrian recognized by the pedestrian recognition device, the presence of the pedestrian around the vehicle can be notified to the driver of the vehicle.

An eighth aspect of the present invention is a vehicle surroundings monitoring apparatus, including: an infrared imaging device that captures an image; a binarized object extraction device that extracts a binarized object by binarizing the image; a pedestrian recognition device that recognize a pedestrian in the binarized object extracted by the binarized object extraction device; an extraction region setting device that sets, as an extraction region, a region adjacent to one of an upper end and a lower end of the binarized object extracted by the binarized object extraction device; a search region setting device that sets a search region around the extraction region; a similar object extraction device that, when an object having a similar shape as the object in the extraction region is present in the search region, extracts the object in the search region and sets the object as a similar object; and an identical object determination device that determines whether or not the object in the extraction region and the similar object in the search region are identical.

In the vehicle surroundings monitoring apparatus having the above-described constitution, the binarized object extracts a binarized object from the image obtained by the infrared imaging device, and the pedestrian recognition device recognize a pedestrian from the extracted binarized object. The extraction region setting device then sets, as an extraction region, the region adjacent to one of the upper end and the lower end of the binarized object that has been recognized as a pedestrian by the pedestrian recognition device. Then, if an object having a similar shape as the object in the extraction region set by the extraction region setting device is present in the search region set by the search region setting device around the extraction region, the similar object extraction device extracts that object in the search region having the similar shape as a similar object. Since a determination as to whether or not the object in the extraction region and the similar object in the search region are identical is made by the identical object determination device, continuous objects having the same shape can be recognized among objects that have previously been determined as pedestrians by the pedestrian recognition device when the binarized object or an object in the extraction region in the upward and the downward direction thereof has the similar shape as that of an object in the search region.

According to a ninth aspect of the present invention, in the vehicle surroundings monitoring apparatus according to the eighth aspect, the vehicle surroundings monitoring apparatus includes two of the infrared imaging devices, and a distance calculating device that, based on two images captured by the two infrared imaging devices, calculates a distance to the object contained in the images, and when a distance to the object in the extraction region is the same as a distance to the similar object in the search region, the identical object determination device determines that the object in the extraction region and the similar object in the search region are identical.

In the vehicle surroundings monitoring apparatus having the above-described constitution, when the distance to the object in the extraction region calculated by the distance calculating device that calculates the distance to a binarized object from the parallax between the right image and the left image, and the distance to the similar object in the search region that is also calculated by the distance calculating device are the same, the identical object determination device determines that the object in the extraction region and the similar object in the search region are identical. Thus, a determination as of whether or not the object in the extraction region and the similar object in the search region are identical can be made more accurately since the identical object determination device determines that the object in the extraction region and the similar object in the search region are identical.

According to a tenth aspect of the present invention, in the vehicle surroundings monitoring apparatus according to the sixth aspect, the search region setting device sets the search region to one of an upward direction, a downward direction, and upward and downward directions of the extraction region.

In the vehicle surroundings monitoring apparatus having the above-described constitution, the search region setting device sets the search region to one of the upward direction, the downward direction, and the upward and the downward directions of the extraction region, and the identical object determination device determines whether or not the binarized object in the extraction region and the similar object in the search region that has been set to one of the upward direction, the downward direction, and the upward and the downward directions of the extraction region are identical. Thus, continuous objects having the same shape arranged in the vertical direction can be recognized among objects that have previously been determined as pedestrians by the pedestrian recognition device when the binarized object in the extraction region and the similar object in the search region that has been set to one of the upward direction, the downward direction, and the upward and the downward directions of the extraction region are similar.

According to an eleventh aspect of the present invention, in the vehicle surroundings monitoring apparatus according to the sixth aspect, the search region setting device sets a plurality of the search regions in the same direction, and when a similar object is extracted by the similar object extraction device from a predetermined ratio of the plurality of search regions or higher, the identical object determination device determines that the object in the extraction region and the similar object in the search region are identical.

In the vehicle surroundings monitoring apparatus having the above-described constitution, the search region setting device sets the plurality of search regions around the extraction region, and the similar object extraction device extracts the similar object. In this process, when the ratio of the number of images from which the similar object is extracted is equal to or higher than the predetermined ratio, the identical object determination device determines that the object in the extraction region and the similar objects in the plurality of search regions set around the extraction region are identical even when the similar objects are not present in all of the search regions. Thus, continuous objects having the same shape can be recognized among objects that have previously been determined as pedestrians by the pedestrian recognition device when the object in the extraction region and the objects in the search regions have the similar shape.

According to a twelfth aspect of the present invention, in the vehicle surroundings monitoring apparatus according to the sixth aspect, the extraction region setting device sets a right extraction region and a left extraction region as extraction regions, each of the right extraction region and the left extraction region respectively having a horizontal coordinate of the right-side end and the left-side end of the binarized object, that has been recognized as the pedestrian by the pedestrian recognition device, the search region setting device sets the search regions around each of the right extraction region and the left extraction region, the similar object extraction device, when an object having a similar shape as an object in the left extraction region is present within the search region corresponding to the left extraction region, or when an object having a similar shape as the object in the right extraction region is present within the search region corresponding to the right extraction region, extracts the object having the similar shape in the search region as a similar object, and the identical object determination device determines whether or not the object in the left extraction region and the similar object in the search region corresponding to the left extraction region are identical, or whether or not the object in the right extraction region and the similar object in the search region corresponding to the right extraction region are identical.

In the vehicle surroundings monitoring apparatus having the above-described constitution of the extraction region setting device, a right extraction region and a left extraction region are set in the positions corresponding to the right end and the left end of the binarized object as extraction regions in a region adjacent to one of an upper end and a lower end of the binarized object that has been recognized as a pedestrian by the pedestrian recognition device. Then, when an object having a similar shape as the object in the extraction region set by the extraction region setting device is present in each search region corresponding to the right search region and the left search region set by the search region setting device, the similar object extraction device extracts that object in the search region having the similar shape as a similar object. In this manner, since the identical object determination device determines whether or not an object in either the right extraction region or the left extraction region and the similar object in the search region corresponding to that extraction region are identical, continuous objects having the same shape in either the right end or the left end can be recognized among objects that have previously been determined as pedestrians by the pedestrian recognition device when the object in either the right extraction region or the left extraction region and the similar object in the search region corresponding to that extraction region are similar.

According to a thirteenth aspect of the present invention, in the vehicle surroundings monitoring apparatus according to the eighth aspect, the vehicle surroundings monitoring apparatus further includes a warning issuing device that excludes a region that contains the object that has been determined as the identical object by the identical object determination device and issues a warning concerning the pedestrian that is present in a remaining region.

In the vehicle surroundings monitoring apparatus having the above-described constitution, the warning issuing device excludes a region in which the object that is determined as the same object by the identical object determination device and cannot be a pedestrian, and issues a warning concerning the pedestrian that is present in the remaining region. Thus, it is possible to notify the driver of the vehicle only of the presence of the pedestrian around the vehicle.

According to vehicle surroundings monitoring apparatus of the first aspect of the present invention, when the binarized object or an object in the extraction region in the upward and the downward direction thereof has the similar shape as that of an object in the search region, continuous objects having the same shape can be recognized.

Accordingly, since a series of objects having the same shape is characteristic only to an artificial structure, a binarized object having an indefinite shape extracted from an infrared image is accurately identified. Accordingly, in a vehicle surroundings monitoring apparatus, the effect of realizing an accurately and easily extracts an artificial structure from objects around the vehicle can be achieved.

According to vehicle surroundings monitoring apparatus of the second aspect of the present invention, a determination of whether or not the binarized object in the extraction region and the similar object in the search region are identical can be made more accurately.

Accordingly, the effect of an enhanced accuracy of extracting artificial structures from objects around the vehicle can be achieved.

According to vehicle surroundings monitoring apparatus of the third aspect of the present invention, when the binarized object in the extraction region and the similar object in the search region that has been set to one of the upward direction, the downward direction, and the upward and the downward directions of the extraction region are similar, continuous objects having the same shape arranged in the vertical direction can be recognized.

Accordingly, the effect of realizing a vehicle surroundings monitoring apparatus that accurately identifies a binarized object having an indefinite shape extracted from an infrared image, and is capable of extracting continuous artificial structures having the same shape in the vertical direction, e.g., utility poles, easily and accurately from-objects around the vehicle can be achieved.

According to vehicle surroundings monitoring apparatus of the fourth aspect of the present invention, even when the similar objects are not present in all of the search regions, continuous objects having the same shape can be recognized when a binarized object in the extraction region and a binarized object in the search region are similar.

Accordingly, continuous artificial structures having the same shape can be extracted even from an object the entire image of which has not been taken by the infrared imaging device due to the influence of the driving state of the vehicle or the environment around the object, and thus the accuracy of extracting artificial structures from objects around the vehicle can be enhanced.

According to vehicle surroundings monitoring apparatus of the fifth aspect of the present invention, by comparing either the right extraction region or the left extraction region and the search region corresponding to that extraction region, continuous objects having the same shape in either the right end or the left end can be recognized when the binarized object in either the right extraction region or the left extraction region and the binarized object in the search region corresponding to the extraction region are similar.

Accordingly, continuous artificial structures having the same shape can be extracted even from an object that is present only in either the right side or the left side of the images taken by the infrared imaging device due to the influence of the driving state of the vehicle or the environment around the object.

According to vehicle surroundings monitoring apparatus of the sixth aspect of the present invention, a pedestrian can be recognized from remaining objects other than the continuous objects having the same shape.

Accordingly, a vehicle surroundings monitoring apparatus can be realized which can accurately determine a binarized object having an indefinite shape extracted from an infrared image, exclude an artificial structure in which a series of objects having the same shape are excluded beforehand, and extract pedestrians accurately and easily from the remaining objects around the vehicle.

According to vehicle surroundings monitoring apparatus of the seventh aspect of the present invention, the presence of the pedestrian around the vehicle can be notified to the driver of the vehicle.

Accordingly, it is possible to notify the driver of the vehicle of the pedestrian.

According to vehicle surroundings monitoring apparatus of the eighth aspect of the present invention, continuous objects having the same shape can be recognized among objects that have previously been determined as pedestrians by the pedestrian recognition device when the binarized object or an object in the extraction region in the upward and the downward direction thereof has the similar shape as that of an object in the search region.

Accordingly, since a series of objects having the same shape is characteristic only to an artificial structure, even when a binarized object having an indefinite shape extracted from an infrared image has been recognized as a pedestrian, an artificial objects determined from a series of objects having the same shape is excluded. Thus, a vehicle surroundings monitoring apparatus that can accurately and easily extracts a pedestrian from objects around the vehicle can be realized.

According to vehicle surroundings monitoring apparatus of the ninth aspect of the present invention, a determination of whether or not the binarized object in the extraction region and the similar object in the search region are identical can be made more accurately.

Accordingly, it is possible to enhance the accuracy of extracting a pedestrian from objects around the vehicle.

According to vehicle surroundings monitoring apparatus of the tenth aspect of the present invention, among objects that have previously been determined as pedestrians by the pedestrian recognition device, the binarized object in the extraction region and the similar object in the search region that has been set to one of the upward direction, the downward direction, and the upward and the downward directions of the extraction region are similar, continuous objects having the same shape arranged in the vertical direction can be recognized.

Thus, a binarized object having an indefinite shape extracted from an infrared image is accurately identified, and a vehicle surroundings monitoring apparatus that is capable of extracting continuous artificial structures having the same shape in the vertical direction, e.g., utility poles, from objects around the vehicle and can recognize a pedestrian easily and accurately from the remaining objects can be realized.

According to vehicle surroundings monitoring apparatus of the eleventh aspect of the present invention, by comparing either the right extraction region or the left extraction region and the search region corresponding to that extraction region, continuous objects having the same shape in either the right end or the left end can be recognized among objects that have previously been determined as pedestrians by the pedestrian recognition device when the binarized object in either the right extraction region or the left extraction region and the binarized object in the search region corresponding to the extraction region are similar.

In the vehicle surroundings monitoring apparatus having the above-described constitution, the search region setting device sets a plurality of search regions around the extraction region, and the similar object extraction device extracts a similar object. As a result, when the similar object is extracted from the predetermined ratio or higher of the plurality of search regions, the identical object determination device determines that the object in the extraction region and the plurality of the similar objects set around the extraction region in the search region are the same object even when similar objects are not present in all of the search regions. Thus, continuous objects having the same shape can be recognized among objects that have been determined as pedestrians by the pedestrian recognition device when the object in the extraction region and the object in the search region have the similar shapes.

According to vehicle surroundings monitoring apparatus of the twelfth aspect of the present invention, by comparing either the right extraction region or the left extraction region and the search region corresponding to that extraction region, continuous objects having the same shape in either the right end or the left end can be recognized among objects that have previously been determined as pedestrians by the pedestrian recognition device when the binarized object in either the right extraction region or the left extraction region and the binarized object in the search region corresponding to the extraction region are similar.

Accordingly, continuous artificial structures having the same shape can be excluded even from an object that is only present in either the right side or the left side of the images taken by the infrared imaging devices, and has been determined as a pedestrian by the pedestrian recognition device. Thus, a pedestrian can be accurately and easily extracted from the remaining objects, and accordingly, it is possible to enhance the accuracy of extracting a pedestrian from objects around the vehicle.

According to vehicle surroundings monitoring apparatus of the thirteenth aspect of the present invention, continuous objects having the same shape can be excluded, and it is possible to notify the driver of the vehicle only of the presence of the pedestrian around the vehicle.

Accordingly, unnecessary output of warning of continuous artificial structures having the same shape can be prevented, and the warning only about the pedestrian can be issued for the driver of the vehicle.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing a configuration of a vehicle surroundings monitoring apparatus according to an embodiment of the present invention;

FIG. 2 is a diagram showing positions of an infrared ray camera, a sensor, a display, etc., installed to a vehicle according to the embodiment of the present invention;

FIG. 3 is a flowchart showing an object detection and alarming operation of the vehicle surroundings monitoring apparatus according to the embodiment of the present invention;

FIGS. 4A and 4B are diagrams showing a gray scale image obtained by using an infrared camera, and a binarized image thereof, respectively;

FIG. 5 is a flowchart showing an alarm determination operation according to the embodiment of the present invention;

FIG. 6 is a diagram showing areas in which collision of a vehicle is likely to occur;

FIG. 7 is a diagram showing areas and sections in front of a vehicle according to the embodiment of the present invention;

FIG. 8 is a flowchart of the artificial structure determining operation according to the present invention;

FIG. 9 is a schematic diagram illustrating of a binarization area upward search of the artificial structure determining process;

FIG. 10 is a schematic diagram illustrating of the binarization area upward search of the artificial structure determining process;

FIG. 11 is a schematic diagram illustrating of a binarization area downward search of the artificial structure determining process;

FIG. 12 is a schematic diagram illustrating of the binarization area downward search of the artificial structure determining process;

FIG. 13 is a schematic diagram illustrating of a binarization area upper left search of the artificial structure determining process;

FIG. 14 is a schematic diagram illustrating of the binarization area upper left search of the artificial structure determining process;

FIG. 15 is a schematic diagram illustrating of a binarization area left downward search of the artificial structure determining process;

FIG. 16 is a schematic diagram illustrating of the binarization area left downward search of the artificial structure determining process; and

FIG. 17 is a picture showing an example of an artificial structure recognized by the vehicle surroundings monitoring apparatus according to the embodiment of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

Hereafter, various embodiments of the present invention will be described with reference to the drawings.

Overall Configuration

FIG. 1 is a block diagram showing a configuration of a vehicle surroundings monitoring apparatus according to an embodiment of the present invention.

In FIG. 1, the reference numeral 1 indicates an image processing unit including a central processing unit (CPU) which controls the vehicle surroundings monitoring apparatus according to the embodiment. To the image processing unit 1, two infrared cameras 2R and 2L capable of detecting far infrared radiations, a yaw rate sensor 3 which detects the yaw rate of the vehicle, a vehicle speed sensor 4 which detects the traveling rate (vehicle speed) of the vehicle, and a brake sensor 5 for detecting brake operation are connected. In this configuration, the image processing unit 1 detects moving objects, such as pedestrians or animals, in front of the vehicle based on an infrared image around the vehicle and signals indicating the driving state of the vehicle, and issues an alarm when it is determined that the possibility of collision is high.

Also, to the image processing unit 1, a speaker 6 for generating an alarm sound, and an image display device 7, which displays images captured by the infrared cameras 2R and 2L and makes the driver of the vehicle recognize objects which may be hit by the vehicle, are connected. The image display device 7 may include, for instance, a display combined with a meter which shows the driving state of the vehicle in terms of numerical values, a navigation display provided with the console of the vehicle, and/or a heads up display (HUD) 7a which displays information at a position in the front window where the sight of the driver is not interfered with.

Moreover, the image processing unit 1 may further include an A/D converting circuit which converts input analog signals into digital signals, an image memory which stores digitalized image signals, a central processing unit (CPU) which performs various operations, a random access memory (RAM) which is used for storing data being operated by the CPU, a read only memory (ROM) which stores programs, tables, maps, etc., performed by the CPU, and output circuits through which driving signals for a speaker 6, display signals for the HUD 7a, etc., are output. Accordingly, signals sent from each of the infrared cameras 2R and 2L, the yaw rate sensor 3, the vehicle speed sensor 4 and the brake sensor 5 are converted into digital signals and are input in the CPU.

Also, as shown in FIG. 2, the infrared cameras 2R and 2L are disposed at the front of a vehicle 10 at symmetrical positions with respect to the center of the vehicle 10 in the width direction so that the optical axis of each of the infrared cameras 2R and 2L become parallel to each other, and the height of the cameras 2R and 2L from the surface of the road become equal. Note that each of the infrared cameras 2R and 2L has characteristics such that the level of output signal becomes higher (i.e., luminance increases) as the temperature of an object increases.

Moreover, the display panel of the HUD 7a is disposed at a position of the windshield of the vehicle 10 where the sight of the driver is not interfered with the display panel.

Next, operation of the vehicle surroundings monitoring apparatus according to the embodiment of the present invention will be explained with reference to the accompanying drawings.

Object Detection And Warning Operation

FIG. 3 is a flowchart showing operations for detecting an object, such as a pedestrian, and for generating an alarm, which are performed in the image processing unit 1 of the vehicle surroundings monitoring apparatus according to the embodiment of the present invention.

In FIG. 3, first, the image processing unit 1 obtains an infrared image (step S1), which is output signals from the infrared cameras 2R and 2L, subjects the signals to an A/D conversion process (step S2), and stores the obtained gray scale image in an image memory (step S3). Note that in this embodiment, the right image is obtained by the infrared camera 2R, and the left image is obtained by the infrared camera 2L. Also, since the horizontal position of the same object in the right image and the left image is shifted from each other when the images are obtained, it is possible to calculate the distance to the object based on the shift (parallax).

After the gray scale image is obtained in step S3, the right image obtained by the infrared camera 2R is used as a reference image, and the image signals thereof is subjected to a binarization process, i.e., a process in which an area whose luminance is higher than a threshold value ITH is set to be "1" (white), and an area whose luminance is less than the threshold value ITH is set to be "0" (black) (step S4).

FIG. 4A shows a gray scale image obtained using the infrared camera 2R, and an image shown in FIG. 4B is obtained by subjecting the gray scale image shown in FIG. 4A to the binarization process. Note that in FIG. 4B, objects indicated in each of the frames P1 to P4 are objects (hereinafter also referred to as "high luminance areas"), which are shown in white in the displayed picture.

After binarized image data is obtained from the infrared image, a process is performed in which the obtained imaged data is converted into run-length data (step S5). Lines that are expressed by the run-length data are made up of areas at pixel level that have been determined to be white in the binarization process. Each of the lines has a width of one pixel in the y direction, and has a length corresponding to the length of a pixel which forms the run-length data in the x direction.

Next, objects in the image data converted to the run-length data are labeled (step S6) so that an extraction process for the objects may be carried out (step S7). That is, among the lines expressed as the run-length data, by regarding a line having a portion superimposed in the y direction as a single object, each of the high luminance areas P1 to P4 shown in FIG. 4B, for example, can be recognized as an object (binarized object).

After the extraction of the objects is completed, the centroid G, the surface area S, and the aspect ratio ASPECT (ratio of length to width) of a circumscribed square are calculated (step S8).

In this embodiment, the surface area S is calculated by assuming the run-length data of an object of label A as (x[i], y[i], run [i], A) (i=0, 1, 2, . . . N-1), and accumulating the lengths of the run-length data (run [i]-1) for the same object (data of N run-lengths). Also, the coordinates (xc, yc) of the centroid G of the object A is calculated by multiplying the length of each run-length data (run [i]-1) by the coordinate x[i] or y[i] of each of the run-length data, further multiplying the resultant value by each other for the same object, and dividing the obtained value by the surface area S.

Moreover, the aspect ratio ASPECT is calculated as the ratio of Dy/Dx where Dy is the length of the circumscribed square for the object in the longitudinal direction, and Dx is the length of the circumscribed square for the same object in the transverse direction.

Note that since the run-length data is expressed as a pixel number (a coordinate number) (=run [i]), it is necessary to subtract one to obtain the actual length (=run [i]-1). Also, it is possible to substitute the position of the centroid of the circumscribed square for the position of the centroid G.

After the centroid, the surface area, and the aspect ratio of the circumscribed square of the binarized object are calculated, and a process for tracking the object over time, i.e., a process in which the same object is recognized every sampling period, is carried out (step S9). In the tracking process over time, objects A and B are extracted at time "k," which is obtained by discretization of time "t," an analog value, using the sampling period, for instance, and it is determined if objects C and D, which are extracted at time (k+1), are the same objects as the objects A and B, respectively. Then, when it is determined that the objects A and B and the objects C and D are the same objects, labels of the objects C and D are changed to labels A and B, respectively, to carry out the tracking process over time.

Also, the positional coordinates of each of the (centroid of) objects thus recognized are stored in memory as time series positional data, and are used for the subsequent operation process.

Note that the processes in steps S4 to S9 explained above are carried out for the binarized reference image (the right image in this embodiment).

Then, the vehicle speed VCAR detected by the vehicle speed sensor 4, and the yaw rate YR detected by the yaw rate sensor 3 are read, and the swinging angle .theta.r of the vehicle 10 is calculated by subjecting the yaw rate YR to time integration (step S10).

On the other hand, a process for calculating the distance "z" between the object and the vehicle 10 (steps S11 to S13) is carried out simultaneously with the process of steps S9 and S10. Since this process for calculating the distance "z" takes a longer time period than the time required in steps S9 and S 10, it is carried out using longer time than steps S9 and S10 (about three times longer than the time for carrying out steps S1 to S10, for instance).

First, one of the objects which are tracked using the binarized image of the reference image (the right image), is selected to extract a search image R1 (in this embodiment, the whole area surrounded by the circumscribed square is referred to as the search image, and the circumscribed square of the search image is referred to as a "target frame") from the right image (step S11).

Then, a search region from which an image corresponding to the search image R1 (hereinafter also referred to as a "corresponding image") is searched, is set in the left image, and the corresponding image is extracted by carrying out a correlation operation (step S12). More specifically, a search region R2 is set in the left image in accordance with each apex coordinate of the search image R1, and a luminance difference summed value C (a, b), which indicates the degree of correlation with the search image R1 within the search region R2, is calculated. The area at which the summed value C (a, b) is minimum is extracted as the corresponding image. Note that the correlation operation is performed using the gray scale image, not the binarized image. Also, when there is a previous positional data for the same object, an area R2a, which is narrower than the search region R2, is set as a search region based on the previous positional data.

Since the search image R1 and the corresponding image R4, which corresponds to this object, are extracted in the reference image (the right image) and the left image, respectively, in step S12, the position of the centroid of the search image R1 and that of the corresponding image R4, and the parallax .DELTA.d (pixel number) are determined, and the distance "z" between the vehicle 10 and the object may be calculated using these parameters (in step S13).

Then, after the calculation of the swing angle .theta.r in step S 10, and the calculation of the distance "z" between the object in step S13 are completed, a swing angle correction process for correcting the positional shift in the image caused by the swing of the vehicle 10 is carried out (in step S14). That is, if the (front portion of the) vehicle 10 is rotated in the left side direction at an angle of .theta.r from time k to (k+1), the range of the image obtained through the cameras shifts in the x direction by .DELTA.x, and this shift is corrected in the swing angle correction process.

After the swing angle correction is completed for the real space coordinate, the coordinate (x, y) in the image and the distance "z" are converted into the coordinate (X, Y, Z) of the real space (in step S15).

In this embodiment, th