Image processing apparatus Number:6,816,193 from the United States Patent and Trademark Office (PTO) owispatent An image processing apparatus includes: an image pickup section for converting an object light into image signals of Bayer RGB array; a color information detecting section for obtaining color information of the object from the image signals obtained by the image pickup section; a first luminance data generating section for finding luminance data for use in edge enhancement processing based on G signals of the image signals obtained by the image pickup section; a second luminance data generating section for finding luminance data for use in edge enhancement processing based on all color signals of the image signals; a selector for providing an output by switching between outputs of the first and second luminance data generating section; and a selection signal generating section for generating selection signal for controlling the switching of the selector based on color information obtained at the color information detecting section. The image processing apparatus is thereby achieved as capable of edge enhancement processing at favorable S/N and with less occurrence of edge noise in high saturation regions and low saturation regions.<H processing, apparatus, Image, processin, 6816193.html, Patent, pto, 6816193

Title: Image processing apparatus

Abstract: An image processing apparatus includes: an image pickup section for converting an object light into image signals of Bayer RGB array; a color information detecting section for obtaining color information of the object from the image signals obtained by the image pickup section; a first luminance data generating section for finding luminance data for use in edge enhancement processing based on G signals of the image signals obtained by the image pickup section; a second luminance data generating section for finding luminance data for use in edge enhancement processing based on all color signals of the image signals; a selector for providing an output by switching between outputs of the first and second luminance data generating section; and a selection signal generating section for generating selection signal for controlling the switching of the selector based on color information obtained at the color information detecting section. The image processing apparatus is thereby achieved as capable of edge enhancement processing at favorable S/N and with less occurrence of edge noise in high saturation regions and low saturation regions.

Patent Number: 6,816,193 Issued on 11/09/2004 to Kohashi, et al.

Inventors: Kohashi; Atsushi (Hachioji, JP); Sakurai; Junzo (Koganei, JP)
Assignee: Olympus Optical Co., Ltd. (Tokyo, JP)

Appl. No.: 695420

Filed: October 25, 2000

Foreign Application Priority Data


Oct 27, 1999 [JP]			11-304875

Current U.S. Class: 348/234 ; 348/235; 348/625; 348/712

Current International Class: H04N 5/14 (20060101)

Field of Search: 348/234,222.1,223.1,235,252,266,289,625,627,631,663,712,612,675,703 382/199

References Cited [Referenced By]

U.S. Patent Documents


5768403	June 1998	Suzuki et al.
5953058	September 1999	Hanagata
6028646	February 2000	Jeong et al.
6400403	June 2002	Saito
6456325	September 2002	Hayashi
2001/0009438	July 2001	Kihara et al.

Foreign Patent Documents


9-261513	Oct., 1997	JP
10-108208	Apr., 1998	JP

Primary Examiner: Garber; Wendy R.
Assistant Examiner: Ye; Lin
Attorney, Agent or Firm: Westerman, Hattori, Daniels & Adrian, LLP

Claims

What is claimed is:

1. An image processing apparatus having a processing section for enhancing edges in an image, said image processing apparatus comprising: image pickup means for converting an object light into image signals of Bayer RGB array; color information detection means for obtaining color information of the object from the image signals obtained by the image pickup means; a first luminance data generation means for finding luminance data for use in edge enhancement processing from G signals out of the image signals obtained by said image pickup means; a second luminance data generation means for finding luminance data for use in edge enhancement processing from all color signals of the image signals obtained by said image pickup means; and luminance data switch means for providing an output by switching between outputs of said first and second luminance data generation means based on the color information obtained at said color information detection means.

2. The image processing apparatus according to claim 1, wherein the color information detected by said color information detection means is saturation information of said image.

3. The image processing apparatus according to claim 1, wherein the color information detected by said color information detection means is hue information of said image.

4. The image processing apparatus according to claim 1, wherein the color information detected by said color information detection means is saturation information and hue information of said image.

5. The image processing apparatus according to any one of claims 1 to 4, wherein said color information detection means detects color information for every pixel signal of said image signals.

6. The image processing apparatus according to any one of claims 1 to 4, wherein said color information detection means detects an average color information of a whole frame of said image signals.

7. The image processing apparatus according to any one of claims 1 to 4, wherein said color information detection means detects an average color information of a whole frame from specific sampling data of said image signals.

8. The image processing apparatus according to claim 5, wherein said luminance data switch means switches the outputs of the first and second luminance data generation means for every pixel signal based on said color information detected for every pixel signal.

9. The image processing apparatus according to claim 6, wherein said luminance data switch means switches the outputs of the first and second luminance data generation means for every frame based on the average color information detected from a whole frame of said image signals.

10. The image processing apparatus according to claim 7, wherein said luminance data switch means switches the outputs of the first and second luminance data generation means for every frame based on the average color information of a whole frame detected from said specific sampling data of image signals.

Description

This application claims benefit of Japanese Application No. Hei 11-304875 filed in Japan on Oct. 27, 1999, the contents of which are incorporated by this reference.

BACKGROUND OF THE INVENTION

The present invention relates to image processing apparatus, and more particularly to an image processing apparatus having a digital image processing section including enhancement of edges in image to be displayed.

When an image is processed for edge enhancement, a technique is generally used to extract edge components within the image so as to add the extracted edge components to the original image. In such case, the edge components to be added are obtained by extracting high-frequency components of luminance signals of the image. In image pickup devices using RGB primary-color filter, the techniques for generating luminance signal to extract such high-frequency components include: (1) technique for generating luminance signal by using only G (green) signals out of all the color signals of RGB within the image; and (2) technique for generating luminance signal by using all the color signals of RGB in the image.

An example of construction of conventional image processing apparatus will now be explained by way of FIG. 1 in which such two techniques are used to generate luminance signal to perform edge enhancement processing. Referring to FIG. 1, numeral 101 denotes an image pickup means; image signals of Bayer RGB array are obtained from the image pickup means 101. The Bayer RGB array image signals are subjected to white balancing at a white balancing section 102. Luminance data are then generated from 2G or (R+2G+B)/2 at a luminance data generating section 103 where R, G, B represent the outputs of the color signals of R (red), G (green), B (blue), respectively. The luminance data are then passed through a high-pass filter 104 to extract edge components and the degree of edge enhancement is adjusted at an edge enhancement adjusting section 105 to form edge signals.

On the other hand, image signals after the white balancing are separated of colors by pixel as a unit into RGB signals at color separation circuit 106 and then subjected to color correction and .gamma. correction at a color correction/.gamma. correction circuit 107. The RGB signals are converted into Y, Cr, Cb signals at YC conversion section 108. The above described edge signals are then added to the luminance signal Y outputted from YC conversion section 108 at an adder circuit 109 to obtain edge-enhanced luminance signal Y.

The above two techniques have their respective merits and demerits. In particular, FIG. 2A shows a part of image consisting of pixels of Bayer RGB array. In such image of Bayer RGB array, two times of image signals S are used in the case of generating luminance signal Y from 4 pixels of RGB (R+2G+B) as shown in FIG. 2B as compared to the case of generating luminance signal Y as shown in FIG. 2C from two pixels of G. The resulting noise N, on the other hand, is only 2.sup.1/2 times from the theory of noise reducer. For this reason, supposing S/N at the time of generating from two pixels of G as 1/1=1, that at the time of generating from 4 pixels of RGB becomes 2/2.sup.1/2 =2.sup.1/2. The case of generating from 4 pixels of RGB becomes 2.sup.1/2 times better. Since, however, the signals of R, B, which contain not much genuine luminance information, are used as luminance signal when 4 pixels of RGB are used, excessively emphasized edges are generated and defects occur especially at the portion of boundary between two highly saturated colors.

As has been described, there are merits and demerits in both the case of using only G signal and the case of using all the color signals of RGB in generating luminance signal. These are summarized in Table 1.

TABLE 1 Luminance signal Edge noise of high Edge noise of low generation method S/N saturation portion saturation portion From only G Bad Small Large From RGB Good Large Small

Prior-art techniques according to some specific documents will now be described. Japanese patent application laid open Hei-10-108208 for example discloses a technique in which contour signals (edge components) are extracted from G signals to perform edge enhancement. Further, Japanese patent laid open application Hei-9-261513 discloses a contour (edge) enhancing method in which two types of contour extracting methods, i.e., MIX (mix) mode and NAM (non-average mix) mode are switched to each other according to the saturation of image. Here, MIX mode refers to generation of contour enhancement signal after addition of a plurality of signals corresponding to the three primary colors of RGB, etc., in the light of image to be displayed; NAM mode refers to generation of contour enhancement signal by adding one selecting the signal emphasized most in the direction of black and one selecting the signal emphasized most in the direction of white out of the respective contour enhancement signals of the above described plurality of signals.

Thus the problem is that there are respective merits and demerits as described above of the technique for generating luminance signal by using only G signals in the image to extract edge components and the technique for generating luminance signal by using all the color signals of RGB in the image to extract edge components. Further, the contour enhancement technique as disclosed in the above Japanese patent laid open application Hei-9-261513 has a problem that, if Bayer RGB array is used as the color filter array of image pickup device, the mere switching between the two types of contour extracting methods according to saturation alone may, depending on the colors of image, fall short of suitable processing due to lack of information. Furthermore, there is a problem of complicated construction in the NAM mode, since it is necessary to perform addition by respectively selecting one emphasized most in the direction of black and one emphasized most in the direction white out of the contour enhancement signals.

SUMMARY OF THE INVENTION

To eliminate the above problems in the conventional methods of edge enhancement processing of an image, it is an object of the present invention to provide an image processing apparatus having a digital image processing section including edge enhancement processing of simple construction having only the respective merits of the prior-art techniques.

In accordance with a first aspect of the invention, there is provided an image processing apparatus having a processing section for enhancing edges in an image, including: image pickup means for converting an object light into image signals of Bayer RGB array; color information detection means for obtaining color information of the object from the image signals obtained by the image pickup means; a first luminance data generation means for finding luminance data for use in edge enhancement processing from G signals out of the image signals obtained by the image pickup means; a second luminance data generation means for finding luminance data for use in edge enhancement processing from all color signals of the image signals obtained by the image pickup means; and luminance data switch means for providing an output by switching between outputs of the first and second luminance data generation means based on the color information obtained at the color information detection means.

By such construction, an output is provided on the basis of color information obtained at the color information detection means by switching between output of the first luminance data generation means for finding luminance data from G signal and output of the second luminance data generation means for finding luminance data from all color signals of the image signals. Hence edge enhancing components can be extracted on the basis of an optimal luminance data corresponding to the color information, an edge enhancement processing is possible with a favorable S/N and with less edge noise in high saturation regions and low saturation regions. The above object is thereby achieved.

It is another object of the invention to provide an image processing apparatus in which switching of the techniques for generating luminance data for use in edge enhancement processing is smooth and the edge enhancement processing is performed without unnaturalness.

In accordance with a second aspect of the invention, there is provided an image processing apparatus having a processing section for enhancing edges in an image, including: image pickup means for converting an object light into image signals of Bayer RGB array; color information detection means for detecting color information of the object from the image signals obtained by the image pickup means; a mixed luminance data generation means for generating mixed luminance data by mixing luminance data for use in edge enhancement processing generated from G signals out of the image signals obtained by the image pickup means and luminance data for use in edge enhancement processing generated from all color signals of the image signals with continuously changing the mixing ratio (including 1:0 and 0:1); and means for setting the mixing ratio at the mixed luminance data generation means based on the color information obtained at the color information detection means.

By such construction, mixed luminance data is generated on the basis of color information by mixing the luminance data generated from G signal and the luminance data generated from all color signals with continuously changing the mixing ratio. Hence an optimal luminance data corresponding to the color information can be smoothly changed, making it possible to perform edge enhancement processing without unnaturalness. The above object is thereby achieved.

It is yet another object of the invention to provide an image processing apparatus in which an image processed of edge enhancement at favorable S/N can be obtained even for an image having frequent edge portions.

In accordance with a third aspect of the invention, there is provided an image processing apparatus having a processing section for enhancing edges in an image, including: image pickup means for converting an object light into image signals of Bayer RGB array; edge information detection means for detecting edge information of the object from the image signals obtained by the image pickup means; a first luminance data generation means for finding luminance data for use in edge enhancement processing from G signals out of the image signals obtained by the image pickup means; a second luminance data generation means for finding luminance data for use in edge enhancement processing from all color signals of the image signals obtained by the image pickup means; and luminance data switch means for providing an output by switching between outputs of the first and second luminance data generation means based on the edge information obtained at the edge information detection means.

By such construction, an output is provided by switching between the luminance data generated from G signal and the luminance data generated from all color signals on the basis of edge information obtained at the edge information detection means. Hence it becomes possible to perform edge enhancement processing on the basis of an optimal luminance data corresponding to the edge information even in an image having frequent edge portions. The above object is thereby achieved.

It is a further object of the invention to provide an image pickup apparatus in which, even for an image having frequent edge portions, the techniques for generating luminance data for use in edge enhancement are smoothly switched so that edge enhancement processing without unnaturalness can be performed.

In accordance with a fourth aspect of the invention, there is provided an image processing apparatus having a processing section for enhancing edges in an image, including: image pickup means for converting an object light into image signals of Bayer RGB array; edge information detection means for detecting edge information of the object from the image signals obtained by the image pickup means; a mixed luminance data generation means for generating mixed luminance data by mixing luminance data for use in edge enhancement processing generated from G signals out of the image signals obtained by the image pickup means and luminance data for use in edge enhancement processing generated from all color signals of the image signals with continuously changing the mixing ratio (including 1:0 and 0:1); and means for setting the mixing ratio at the mixed luminance data generation means based on the edge information obtained at the edge information detection means.

By such construction, mixed luminance data is generated on the basis of edge information by mixing the luminance data generated from G signal and the luminance data generated from all color signals with continuously changing the mixing ratio. Hence an optimal luminance data corresponding to the edge information can be smoothly changed so that edge enhancement processing without unnaturalness can be performed even in an image having frequent edge portions. The above object is thereby achieved.

It is a further object of the invention to provide an image processing apparatus in which an image as processed of edge enhancement at favorable S/N can be obtained corresponding to the characteristic of an object.

In accordance with a fifth aspect of the invention, there is provided an image processing apparatus having a processing section for enhancing edges in an image, including: image pickup means for converting an object light into image signals of Bayer RGB array; color information detection means for detecting color information of the object from the image signals obtained by the image pickup means; edge information detection means for detecting edge information of the object from the image signals obtained by the image pickup means; image region determination means for determining the state of being frequent/infrequent of edge portions (frequency of occurrence per unit area of those regarded as edges, i.e., degree of denseness/sparseness of edges) in the object from the image signals obtained by the image pickup means; a first luminance data generation means for finding luminance data for use in edge enhancement processing from G signals out of the image signals obtained by the image pickup means; a second luminance data generation means for finding luminance data for use in edge enhancement processing from all color signals of the image signals obtained by the image pickup means; luminance data switch means for providing an output by switching between outputs of the first and second luminance data generation means; a first selection signal generating section for generating a selection signal for controlling the switching at the luminance data switch means based on color information obtained at the color information detection means; a second selection signal generating section for generating a selection signal for controlling the switching at the luminance data switch means based on edge information obtained at the edge information detection means; and a selection signal switching section for providing an output by switching between the selection signals of the first selection signal generating section and the second selection signal generating section based on a determination signal from the image region determination means.

By such construction, an output is provided by switching between the selection signal for controlling the switching of the luminance data switch means based on color information and the selection signal for controlling the switching of the luminance data switch means based on edge information on the basis of the determination signal from the image region determination means for determining the state of being frequent/infrequent of the edge portions in an object. Hence the edge enhancement processing at favorable S/N can be performed corresponding to the characteristic of the object. The above object is thereby achieved.

It is a further object of the invention to provide an image processing apparatus in which, corresponding to the characteristic of an object, an image is obtained as processed of edge enhancement processing without unnaturalness where S/N is favorable and the techniques for generating luminance data for use in the edge enhancement processing are switched smoothly.

In accordance with a sixth aspect of the invention, there is provided an image processing apparatus having a processing section for enhancing edges in an image, including: image pickup means for converting an object light into image signals of Bayer RGB array; color information detection means for detecting color information of the object from the image signals obtained by the image pickup means; edge information detection means for detecting edge information of the object from the image signals obtained by the image pickup means; image region determination means for determining the state of being frequent/infrequent of edge portions in the object from the image signals obtained by the image pickup means; a mixed luminance data generation means for generating mixed luminance data by mixing luminance data for use in edge enhancement processing generated from G signals out of the image signals obtained by the image pickup means and luminance data for use in edge enhancement processing generated from all color signals of the image signals with continuously changing the mixing ratio (including 1:0 and 0:1); a first mixing ratio setting means for setting the mixing ratio at the mixed luminance data generation means based on the color information obtained at the color information detection means; a second mixing ratio setting means for setting the mixing ratio at the mixed luminance data generation means based on the edge information obtained at the edge information detection means; and a mixing ratio setting signal switching section for providing an output by switching between setting signals of the first mixing ratio setting means and the second mixing ratio setting means based on a determination signal from the image region determination means.

By such construction, an output is provided, on the basis of the determination signal from the image region determination means for determining the state of being frequent/infrequent of edge portions of the object, by switching between the setting signal of the first mixing ratio setting means for setting the mixing ratio at the mixed luminance data generation means based on the color information and the setting signal of the second mixing ratio setting means for setting the mixing ratio at the mixed luminance data generation means based on the edge information. Hence edge enhancement processing without unnaturalness can be performed at favorable S/N and smoothly corresponding the characteristic of the object. The above object is thereby achieved.

It is a further object of the invention to provide an image processing apparatus in which, when joining a plurality of images, edge enhancement processing can be performed without causing unnaturalness at the joint portions.

In accordance with a seventh aspect of the invention, there is provided an image processing apparatus having a joining processing section for joining a plurality of images to generate one piece of image and a processing section for enhancing edges in an image, including: image pickup means for converting an object light into image signals of Bayer RGB array; color information detection means for obtaining color information of the object from the image signals obtained by the image pickup means; joint determination means for detecting joint information of the image signals obtained by the image pickup means; a first luminance data generation means for finding luminance data for use in edge enhancement processing from G signals out of the image signals obtained by the image pickup means; a second luminance data generation means for finding luminance data for use in edge enhancement processing from all color signals of the image signals obtained by the image pickup means; and luminance data switch means for providing an output by switching between outputs of the first and second luminance data generation means based on the color information obtained at the color information detection means and the joint information obtained from the joint determination means.

By such construction, on the basis of the color information and joint information, an output is provided by switching between the output of the first luminance data generation means for finding luminance data from G signals and the output of the second luminance data generation means for finding luminance data from all color signals. Hence, in joining a plurality of images, edge enhancement processing can be performed without unnaturalness at the joint portions. The above object is thereby achieved.

It is a further object of the invention to provide an image processing apparatus in which, when joining a plurality of images, edge enhancement processing can be performed smoothly and without unnaturalness at the joint portions.

In accordance with an eighth aspect of the invention, there is provided an image processing apparatus having a joining processing section for joining a plurality of images to generate one piece of image and a processing section for enhancing edges in an image, including: image pickup means for converting an object light into image signals of Bayer RGB array; color information detection means for obtaining color information of the object from the image signals obtained by the image pickup means; joint determination means for detecting joint information of the image signals obtained by the image pickup means; a mixed luminance data generation means for generating mixed luminance data by mixing luminance data for use in edge enhancement processing generated from G signals out of the image signals obtained by the image pickup means and luminance data for use in edge enhancement processing generated from all color signals of the image signals with continuously changing the mixing ratio (including 1:0 and 0:1); and means for setting the mixing ratio at the mixed luminance data generation means based on the color information obtained at the color information detection means and the joint information obtained from the joint determination means.

By such construction, on the basis of the color information and joint information, mixed luminance data is generated by mixing the luminance data generated from G signal and the luminance data generated from all color signals with continuously changing the mixing ratio. Hence, in joining a plurality of images, edge enhancement processing can be performed smoothly and without unnaturalness at the joint portions. The above object is thereby achieved.

It is a further object of the invention to provide an image processing apparatus in which, when joining a plurality of images, edge enhancement processing without causing unnaturalness at the joint portions can be performed even for an image having frequent edge portions.

In accordance with a ninth aspect of the invention, there is provided an image processing apparatus having a joining processing section for joining a plurality of images to generate one piece of image and a processing section for enhancing edges in an image, including: image pickup means for converting an object light into image signals of Bayer RGB array; edge information detection means for obtaining edge information of the object from the image signals obtained by the image pickup means; joint determination means for detecting joint information of the image signals obtained by the image pickup means; a first luminance data generation means for finding luminance data for use in edge enhancement processing from G signals out of the image signals obtained by the image pickup means; a second luminance data generation means for finding luminance data for use in edge enhancement processing from all color signals of the image signals obtained by the image pickup means; and luminance data switch means for providing an output by switching between outputs of the first and second luminance data generation means based on the edge information obtained at the edge information detection means and the joint information obtained from the joint determination means.

By such construction, on the basis of the edge information and joint information, an output is provided by switching between the output of the first luminance data generation means for finding luminance data from G signals and the output of the second luminance data generation means for finding luminance data from all color signals. Hence, in joining a plurality of images, edge enhancement processing without unnaturalness at the joint portions can be performed even for an image having frequent edge portions. The above object is thereby achieved.

It is a further object of the invention to provide an image processing apparatus in which, when joining a plurality of images, edge enhancement processing can be performed smoothly and without causing unnaturalness at the joint portions even for an image having frequent edge portions.

In accordance with a tenth aspect of the invention, there is provided an image processing apparatus having a joining processing section for joining a plurality of images to generate one piece of image and a processing section for enhancing edges in an image, including: image pickup means for converting an object light into image signals of Bayer RGB array; edge information detection means for obtaining edge information of the object from the image signals obtained by the image pickup means; joint determination means for detecting joint information of the image signals obtained by the image pickup means; a mixed luminance data generation means for generating mixed luminance data by mixing luminance data for use in edge enhancement processing generated from G signals out of the image signals obtained by the image pickup means and luminance data for use in edge enhancement processing generated from all color signals of the image signals with continuously changing the mixing ratio (including 1:0 and 0:1); and means for setting the mixing ratio at the mixed luminance data generation means based on the edge information obtained at the edge information detection means and the joint information obtained from the joint determination means.

By such construction, on the basis of the edge information and joint information, mixed luminance data is generated by mixing the luminance data generated from G signal and the luminance data generated from all color signals with continuously changing the mixing ratio. Hence, in joining a plurality of images, edge enhancement processing can be performed smoothly and without unnaturalness at the joint portions even for an image having frequent edge portions. The above object is thereby achieved.

It is a further object of the invention to provide an image processing apparatus in which, when joining a plurality of images, edge enhancement processing can be performed corresponding to the characteristic of an object at favorable S/N and without unnaturalness at the joint portions.

In accordance with an eleventh aspect of the invention, there is provided an image processing apparatus having a joining processing section for joining a plurality of images to generate one piece of image and a processing section for enhancing edges in an image, including: image pickup means for converting an object light into image signals of Bayer RGB array; color information detection means for obtaining color information of the object from the image signals obtained by the image pickup means; edge information detection means for detecting edge information of the object from the image signals obtained by the image pickup means; image region determination means for determining the state of being frequent/infrequent of edge portions in the object from the image signals obtained by the image pickup means; joint determination means for detecting joint information of the image signals obtained by the image pickup means; a first luminance data generation means for finding luminance data for use in edge enhancement processing from G signals out of the image signals obtained by the image pickup means; a second luminance data generation means for finding luminance data for use in edge enhancement processing from all color signals of the image signals obtained by the image pickup means; luminance data switch means for providing an output by switching between outputs of the first and second luminance data generation means; a first selection signal generating section for generating a selection signal for controlling the switching at the luminance data switch means based on color information obtained at the color information detection means; a second selection signal generating section for generating a selection signal for controlling the switching at the luminance data switch means based on edge information obtained at the edge information detection means; and a selection signal switching section for providing an output by switching between the selection signals of the first selection signal generating section and of the second selection signal generating section based on the joint information of image from the joint determination means for joint boundary portions of image and based on a determination signal from the image region determination means for those other than the joint boundary portions of image.

By such construction, an output is provided by switching between the selection signal for controlling the switching of the luminance data switch means based on color information and the selection signal for controlling the switching of the luminance data switch means based on edge information on the basis of the joint information of image for joint boundary portions of image and on the basis of the determination signal of the image region determination means for those other than the joint boundary portions of image. Hence, in performing joining processing, the edge enhancement processing can be performed corresponding to the characteristic of the object at favorable S/N and without unnaturalness at the joint portions. The above object is thereby achieved.

It is a further object of the invention to provide an image processing apparatus in which, when joining a plurality of images, edge enhancement processing can be performed corresponding to the characteristic of an object smoothly at favorable S/N and without unnaturalness at the joint portions.

In accordance with a twelfth aspect of the invention, there is provided an image processing apparatus having a joining processing section for joining a plurality of images to generate one piece of image and a processing section for enhancing edges in an image, including: image pickup means for converting an object light into image signals of Bayer RGB array; color information detection means for obtaining color information of the object from the image signals obtained by the image pickup means; edge information detection means for detecting edge information of the object from the image signals obtained by the image pickup means; image region determination means for determining the state of being frequent/infrequent of edge portions in the object from the image signals obtained by the image pickup means; joint determination means for detecting joint information of the image signals obtained by the image pickup means; a mixed luminance data generation means for generating mixed luminance data by mixing luminance data for use in edge enhancement processing generated from G signals out of the image signals obtained by the image pickup means and luminance data for use in edge enhancement processing generated from all color signals of the image signals with continuously changing the mixing ratio (including 1:0 and 0:1); a first mixing ratio setting means for setting the mixing ratio at the mixed luminance data generation means based on the color information obtained at the color information detection means; a second mixing ratio setting means for setting the mixing ratio at the mixed luminance data generation means based on the edge information obtained at the edge information detection means; and a mixing ratio setting signal switching section for providing an output by switching between setting signals of the first mixing ratio setting means and of the second mixing ratio setting means based on the joint information of image signals from the joint determination means for joint boundary portions of image and based on a determination signal from the image region determination means for those other than the joint boundary portions of image.

By such construction, an output is provided as switched between the setting signal of the first mixing ratio setting means for setting the mixing ratio of the mixed luminance data generation means based on color information and the setting signal of the second mixing ratio setting means for setting the mixing ratio of the mixed luminance data generation means based on edge information on the basis of the joint information of image for joint boundary portions of image and on the basis of the determination signal of the image region determination means for those other than the joint boundary portions of image. Hence, in joining a plurality of images, the edge enhancement processing can be performed corresponding to the characteristic of the object smoothly at favorable S/N and without unnaturalness at the joint portions. The above object is thereby achieved.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing an example of construction of conventional image processing apparatus for performing the edge enhancement processing.

FIGS. 2A, 2B and 2C show a portion of Bayer RGB array image, the manner for generating luminance signal from four pixels of RGB and the manner for generating luminance signal from two pixels of G, respectively.

FIG. 3 is a block diagram showing a first embodiment of the image processing apparatus according to the present invention.

FIG. 4 shows saturation in a chromaticity diagram where Cr, Cb color signals of a primary-color filter image are the axes of coordinates.

FIG. 5 shows hue in a chromaticity diagram where Cr, Cb color signals of a primary-color filter image are the axes of coordinates.

FIG. 6 shows "saturation + hue" in a chromaticity diagram where Cr, Cb color signals of a primary-color filter image are the axes of coordinates.

FIG. 7 is a block diagram showing a second embodiment of the invention.

FIG. 8 shows a technique for setting a weighting factor based on saturation.

FIG. 9 shows another technique for setting a weighting factor based on saturation.

FIG. 10 shows yet another technique for setting a weighting factor based on saturation.

FIG. 11 shows a technique for setting a weighting factor based on hue.

FIG. 12 shows another technique for setting a weighting factor based on hue.

FIG. 13 is a block diagram showing a third embodiment of the invention.

FIG. 14 is a block diagram showing a fourth embodiment of the invention.

FIG. 15 shows a technique for setting a weighting factor based on edge information.

FIG. 16 shows another technique for setting a weighting factor based on edge information.

FIG. 17 shows yet another technique for setting a weighting factor based on edge information.

FIG. 18 is a block diagram showing a fifth embodiment of the invention.

FIG. 19 is a block diagram showing a sixth embodiment of the invention.

FIG. 20 is a block diagram showing a seventh embodiment of the invention.

FIG. 21 is a block diagram showing an eighth embodiment of the invention.

FIG. 22 is a block diagram showing a ninth embodiment of the invention.

FIG. 23 is a block diagram showing a tenth embodiment of the invention.

FIG. 24 is a block diagram showing an eleventh embodiment of the invention.

FIG. 25 is a block diagram showing a twelfth embodiment of the invention.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Some embodiments will now be described. FIG. 3 is a block diagram showing a first embodiment of the image processing apparatus according to the present invention. Included in FIG. 3 are: image pickup means 1 for outputting image signals of Bayer RGB array; a white balancing section 2; a color separation circuit 3 for separating the image signals into RGB signals by pixel as a unit; a color-correction/.gamma.-correction circuit 4; YC conversion circuit 5; color information detection means 6 for detecting color information based on Cr, Cb color signals outputted from YC conversion circuit 5; a selection signal generating section 7 for generating a selection signal based on color information signal outputted from the color information detection means 6: a first luminance data generating section 8 for generating luminance data for use in edge enhancement based on G signals of the image signals processed of white balancing at the white balancing section 2; a second luminance data generating section 9 for generating luminance data for use in edge enhancement based on (R+2G+B)/2 of the image signals; a selector 10 for selectively outputting one or the other of the luminance data from the first luminance data generating section 8 and the luminance data from the second luminance data generating section 9 on the basis of the selection signal from the selection signal generating section 7; a high-pass filter 11 for extracting edge components from the selectively outputted luminance data; an edge enhancement magnitude adjusting section 12; and an adder circuit 13 for adding edge signals adjusted of enhancement magnitude to luminance signals Y from the YC conversion circuit 5.

A description will now be given with respect to operation of the first embodiment constructed as the above. It is the same as in a conventional example up to the point that Bayer RGB array signals obtained from image pickup means 1, after processed at white balancing section 2, are separated of colors into RGB signals at the color separation circuit 3, processed of color correction and .gamma. correction at the color-correction/.gamma.-correction circuit 4 and subjected to YC conversion at YC conversion circuit 5. In the first embodiment according to the invention, color information is detected at the color information detection means 6 on the basis of Cr, Cb color signals obtained from YC conversion circuit 5 and, based on such color information, a selection signal is generated at the selection signal generating section 7.

Further, Bayer RGB array image signals from the image pickup means 1 after processed of white balancing are inputted to each of the first and second luminance data generating sections 8 and 9. Luminance data based on 2G signal is generated from the first luminance data generating section 8 and luminance data based on (R+2G+B)/2 signal is generated from the second luminance data generating section 9, both being inputted to selector 10. Based on the selection signal from the selection signal generating section 7, one or the other of the luminance data corresponding to color information is then selectively outputted from the selector 10. The edge components of the selected luminance data are extracted through the high-pass filter 11 and edge signals adjusted of edge enhancement magnitude are generated at the edge enhancement magnitude adjusting section 12. The edge signals are then added to the luminance signals Y from YC conversion circuit 5 at the adder circuit 13 to perform edge enhancement processing.

A suitable luminance data for use in edge enhancement processing is thereby selected corresponding to the color information to generate edge signals. In connection with saturation, in particular, edge signals in the case of low saturation are generated by using the luminance data from the second luminance data generating section 9 and edge signals in the case of high saturation are generated by using the luminance data from the first luminance data generating section 8. An edge enhancement processing is thus performed at favorable S/N and with less edge noise in a low-saturation image or in a high-saturation image. Further, in connection with hue, edge signals are generated by using the luminance data from the second luminance data generating,section in the case of specific hues such as G, magenta (Mg) and by using the first luminance data in the case of other hues. An edge enhancement processing is thereby performed corresponding to hues of the image object at favorable S/N and with less edge noise.

A detailed description will now be given with respect to detection of color information to be performed at the color information detection means 6. First, there are three types of color information to be detected from Cr, Cb signals, i.e., saturation, hue, and saturation and hue of image. Further, the techniques for detecting color information and for switching of luminance data based on the same include: a technique for detecting color information of each pixel of an object image; a technique for detecting an average color information for a whole object image; a technique for detecting an average color information of a whole frame from the color information of specific sampling portions of an object image; and techniques for switching luminance data based on these.

The fundamental technique for detection and switching of color information is to detect and switch color information for every pixel. According to this technique, a suitable edge enhancement processing at favorable S/N is possible even when pixels of chromatic color and achromatic color are mixed within an object image. On the other hand, the technique for switching by detecting an average color information for a whole object has an advantage that an edge enhancement without unnaturalness can be achieved because of the elimination of switching points, though there is a possibility that an accurate processing cannot be performed for some portions when pixels of chromatic colors and achromatic colors are mixed. Further, the technique for switching by detecting color information from specific sampling data is a modification of the above technique for switching by detecting an average color information of the whole object. It is a technique for switching by detecting an average color information of a whole frame by using data of pixels sampled at specific intervals instead of data of the whole frame and has an advantage that the size can be reduced of the operation unit and memory in the color information detection means.

A description will now be given with respect to actual methods of detection in the respective cases where saturation or hue or "saturation +hue" is used as the color information. First, the technique for detecting saturation from Cr, Cb color signals will be described by way of FIG. 4. FIG. 4 is a chromaticity diagram where Cr, Cb color signals are the axes of coordinates in a primary-color filter image so that saturation Cl in the case of Cr, Cb signals being Cr1, Cb1, respectively, is represented by distance from the origin on the chromaticity diagram. In particular, C1 is expressed as:

If the detected saturation C1 is a high saturation, then, a selection signal is generated from the selection signal generating section 7 so that the luminance data from the first luminance data generating section 8 is selected at selector 10. If saturation C1 is a low saturation, on the other hand, a selection signal is generated from the selection signal generating section 7 so that the luminance data from the second luminance data generating section 9 is selected at selector 10.

The technique for detecting hue as color information from Cr, Cb signals will now be described by way of a chromaticity diagram of FIG. 5. As shown in FIG. 5, hue .theta. c is represented by an angle for example from the Cb axis on the chromaticity diagram. (It is supposed that phase is represented on the basis of the first quadrant of the chromaticity diagram.) In particular, hue .theta. c is expressed as:

If the detected hue .theta. c is G or Mg, then, a selection signal is generated so that selector 10 selects the luminance data from the second luminance data generating section 9 which generates luminance data for use in the extraction of edge components based on (R+2G+B)/2 of image signals. If hue .theta. is one of those other than G or Mg, on the other hand, a selection signal is generated so that the selector 10 selects the luminance data from the first luminance data generating section 8 which generates luminance data for use in the extraction of edge components based on 2G of image signals.

The technique for detecting "saturation +hue" as the color information from Cr, Cb color signals will now be described by way of a chromaticity diagram of FIG. 6. As shown in FIG. 6, "saturation +hue" C2 is found as distance from saturation C to a straight line L which connects G and Mg on the chromaticity diagram. A value is thereby obtained as also considering hue. This will be further described below. Supposing G-Mg line L as y=ax (passing through the origin), distance C2 between G-Mg line and point C (Cb2, Cr2) is found as follows. First, since a perpendicular VL to line L from C is a straight line having an inclination of -1/a and passing through point (Cb2, Cr2), it is expressed as y=-x/a+Cr2+Cb2/a. Next, an intersection (Cb3, Cr3) between line L and perpendicular VL is obtained as Cb3=(a.times.Cr2+Cb2)/(a.sup.2+ 1), Cr3=a.times.Cb3 by solving simultaneous equations of L and VL. Hence distance (saturation+hue) C2 between point (Cb2, Cr2) and point (Cb3, Cr3) is expressed by the following expression (3), i.e., can be found from the values of Cb2, Cr2, and a.

In this manner, if "saturation+hue" is used as the color information, a mode is achieved as considering both hue and saturation. Luminance data based on 2G is selected only in the case where it is determined to select luminance data based on 2G in each of using the color information of saturation and using that of hue. Luminance data based on (R+2G+B)/2 is selected in any other cases.

Table 2 summarizes luminance data to be selected for use in edge enhancement processing in the cases as described above respectively using saturation, hue, and "saturation+hue" as the color information to be detected.

[Table 2]

TABLE 2 Subject color Saturation only Hue only Saturation + hue Low G, Mg RGB RGB RGB Saturation Other colors RGB 2G RGB High G, Mg 2G RGB RGB Saturation Other colors 2G 2G 2G

A second embodiment of the invention will now be described. FIG. 7 is a block diagram showing the second embodiment where like components as in the first embodiment shown in FIG. 3 are denoted by like numerals and description thereof will be omitted. In this embodiment, provided as the luminance data generating section is a single mixed luminance data generating section 21 where luminance data based on 2G and luminance data based on (R+2G+B)/2 can be generated and at the same time it is caused to generate mixed luminance data thereof with continuously changing the mixing ratio between the two. Further, a weighting factor generating section 22 is provided as a means for setting a mixing ratio in order to control the mixing ratio at the mixed luminance data generating section 21. The weighting factor generating section 22 generates weighting factor a on the basis of color information detected by the color information detection means 6. It should be noted that this embodiment is applied only to the case where luminance data is switched for every pixel.

At the above mixed luminance data generating section 21, luminance data Y is generated in the manner of:

where a is the weighting factor generated at the weighting factor generating section 22. If factor a is 1, luminance data based on 2G is generated. If factor a is 0.5, luminance data based on (R+2G+B)/2 is generated. For the cases of factor a being between 0.5 and 1, luminance data is generated as a mixture of the two. It is thereby possible to eliminate unnaturalness in edge enhancement at boundary portions where luminance data is switched.

A description will now be given with respect to methods for setting weighting factor a in the intermediate transient region when the weighting factor to be generated is set on the basis of color information. First, setting of weighting factor a based on saturation information will be described by way of FIG. 8. In the technique for setting weighting factor a in the setting example shown in FIG. 8, factor a is set to 0.5 when saturation C is a saturation lower than s (low saturation threshold). Thereby Y=2G .times.a+(R+B) (1-a)=(R+2G+B)/2 is obtained at the mixed luminance data generating section 21 where luminance data based on this is generated. Factor a is set to 1 when saturation C is a high saturation higher than t (high saturation threshold). Y=2G.times.a+(R+B) (1-a)=2G is thereby obtained and luminance data based on this is generated.

When saturation C is s<C<t, i.e., if saturation of image is within a medium range, factor a becomes a=m(C-s)+0.5. Here m is an inclination and is expressed as m=0.5/(t-s). In this manner, weighting factor a is rectilinearly changed in the intermediate transient region and, based on this, the mixed luminance data is generated. In this setting example, since the weighting factor in the intermediate transient region is rectilinearly changed, operation is simple in generating the weighting factor and luminance data in the transient region.

Another example of setting of weighting factor based on saturation information will now be described by way of FIG. 9. The setting technique of weighting factor a in this setting example is identical to the foregoing setting example in that: when saturation C is a low saturation lower than s, factor a is set to 0.5 and luminance data is thereby generated based on Y=2G.times.a+(R+B) (1-a)=(R+2G+B)/2 at the mixed luminance data generating section 21; and, when saturation C is a high saturation higher than t, factor a is set to 1 and luminance data based on Y=2G.times.a+(R+B) (1-a)=2G is thereby generated at the mixed luminance data generating section 21. When saturation C is s<C<t in this setting example, i.e., if saturation of image is in the medium transient region, factor a is set as having a functional relationship {a=F(C)} so that it is changed smoothly as curved from 0.5 to 1.

In this case, luminance data based on Y=2G .times.F(C)+(R+B) {1-F(C)} is generated from the mixed luminance data generating section 21. Accordingly, since weighting factor a is smoothly changed in the intermediate saturation region between the low saturation region and the high saturation region, a smooth change is possible of the luminance data at the boundary of each region and of edge signals based on the same for use in enhancement processing.

Yet another example of setting of weighting factor based on saturation information will now be described by way of FIG. 10. In this setting example, the weighting factor a for the lowest saturation portion of image signals is set to 0.5 and the weighting factor a for the highest saturation portion is set to 1. Weighting factor a between these is set as having a functional relationship {a=F(C)} indicating a curve smoothly connecting the low saturation region, medium saturation region and high saturation region between these irrespective of the threshold values s, t.

In this case, luminance data is generated on the basis of Y=2G.times.F(C)+(R+B) {1-F(C)} from the mixed luminance data generating section 21. Accordingly, weighting factor a can be changed smoothly in the manner of a curve over the regions of saturation and the luminance data based on the same and edge signals for use in enhancement processing can be outputted as even more smoothly changed.

The setting of an intermediate transient region of weighting factor a in the case of using hue as color information will now be described by way of FIG. 11. In the setting technique of weighting factor a in the setting example shown in FIG. 11, the weighting factor a is set to 0.5 when hue .theta. is of the value indicating G or Mg. Luminance data based on Y=(R+2G+B)/2 is thereby generated at the mixed luminance data generating section 21. If by contrast the weighting factor a is set to 1 when hue .theta. is of a value indicating one other than G and Mg, a steep change results and there is a fear that noise or the like occurs at the boundary portions. Hence the weighting factor a is set to 1 when hue .theta. is of a value other than those in the vicinity of G and Mg; and luminance data based on Y=2G is thereby generated. On the other hand, when hue .theta. is of a value indicating the vicinity of C or Mg, the weighting factor a is set so that it is changed along a straight line connecting 0.5 and 1. Luminance data is thereby generated based on what is obtained by causing the mixing ratio of (R+2G+B)/2 and 2G be changed when hue .theta. is in the vicinities of G and Mg and hence a steep change is avoided. In this case, since the weighting factor a in the transient region in the vicinities of G and Mg is caused to change along a straight line, arithmetic of weighting factor a or the like in the transient region can be readily performed.

Another example of setting of weighting factor based on hue information will now be described by way of FIG. 12. The setting of weighting factor a in this setting example is identical to the foregoing setting example in that: when hue .theta. is of a value indicating G or Mg, the weighting factor a is set to 0.5 and luminance data based on Y=(R+2G+B)/2 is thereby generated at the mixed luminance data generating section 21; and, when hue .theta. is of a value indicating one outside the vicinity of G or Mg, factor a is set to 1 and luminance data based on Y=2G is thereby generated at the mixed luminance data generating section 21.

In this setting example, when hue .theta. is in the vicinity of G or Mg, factor a is set as having a relationship changed smoothly in a curve from 0.5 to 1.

In this case, luminance data is generated from the mixed luminance data generating section 21 on the basis of what is obtained by smoothly changing the mixing ratio of Y=(R+2G+B)/2 and Y=2G. Accordingly, since the weighting factor a with hue .theta. being in the vicinities of G and Mg is smoothly changed, a smooth change is possible of the luminance data at such boundary regions and edge signals based on the same for use in enhancement processing.

The method shown in the above setting example is to set weighting factor a to 1 when hue .theta. is of a value indicating the outside the vicinity of G or Mg. However, it is also possible to set so that, when hue .theta. is of a value indicating one other than G or Mg, the weighting factor a is caused to be changed in a curve so that it has an arbitrary value between 0.5 and 1 for those hues other than G or Mg. While setting weighting factor as having arbitrary values for the hues other than G or Mg, a smooth change is thereby possible of luminance data as well as edge signals for use