Method and apparatus for processing image data and semiconductor storage device Number:7,522,748 from the United States Patent and Trademark Office (PTO) owispatent

Title: Method and apparatus for processing image data and semiconductor storage device

Abstract: An image processing device and an image processing method calculate motion vectors with a small amount of calculation. A frame memory stores first and second image data, a memory includes multiple elements, and a control unit. Each element includes a pixel for storing a first pixel value of a first pixel of the first image data, a pixel for storing a second pixel value of a second pixel of the second image data, a minimal pixel difference storage unit for storing a first difference value, a motion vector storage unit for storing phase information, a comparison updating unit for comparing the first difference value stored in the minimal pixel difference storage unit, with a second difference value between the first pixel and the second pixel, updating the first difference value with the second difference value according to the results of comparison, and outputting updating information according to said updating, and an updating unit for updating the phase information according to the updating information.

Patent Number: 7,522,748 Issued on 04/21/2009 to Kondo,   et al.

---

Inventors:	Kondo; Tetsujiro (Tokyo, JP), Shiraki; Hisakazu (Kanagawa, JP), Okumura; Akihiro (Kanagawa, JP)
Assignee:	Sony Corporation (Tokyo, JP)
Appl. No.:	10/640,614
Filed:	August 14, 2003

---

Foreign Application Priority Data

---

Aug 15, 2002 [JP]			2002-274058
Aug 15, 2002 [JP]			2002-274059

Current U.S. Class:	382/107 ; 382/103
Current International Class:	G06K 9/00 (20060101)
Field of Search:	382/103,107

---

References Cited [Referenced By]

---

U.S. Patent Documents

6307888	October 2001	Le Clerc
6317819	November 2001	Morton
6856701	February 2005	Karczewicz et al.

Foreign Patent Documents

	1 096 791		May., 2001		EP
	7-193822		Jul., 1995		JP
	8-116542		May., 1996		JP
	10-191352		Jul., 1998		JP
	2001-126061		May., 2001		JP
	WO 00/45340		Aug., 2000		WO

Other References

Peter M. Kuhn, "Fast MPEG-4 Motion Estimation: Processor Based and Flexible VLSI Implementations", Journal of VLSI Signal Processing, XP-000862772, vol. 23, No. 1, Oct. 1999, pp. 67-92. cited by other . Peter M. Kuhn, "Algorithms, Complexity Analysis and VLSI Architectures for MPEG-4 Motion Estimation", XP-002408643, 1999, pp. III-VI, 24-33, and 50-53. cited by other.

Primary Examiner: Tucker; Wesley
Attorney, Agent or Firm: Oblon, Spivak, McClelland, Maier & Neustadt, P.C.

---

Claims

---

What is claimed is:

1. A semiconductor storage device comprising: a memory including a plurality of memory elements, each memory element having a data structure that includes, a first pixel information portion configured to hold data indicative of a first pixel of an image block, a second pixel information portion configured to hold data indicative of a second pixel of a search region of another image block, an estimation portion configured to hold data representative of a motion estimation value between the first pixel and the second pixel, and a motion vector information portion configured to hold data representative of a motion vector between the first pixel and the second pixel.

2. The semiconductor storage device of claim 1, wherein: said estimation portion is configured to replace another difference value with said motion estimation value when an absolute value of said motion estimation value is less than said another difference value.

3. The semiconductor storage device of claim 1, wherein: the motion estimation value is an absolute value of a difference between the first pixel and second pixel.

4. The semiconductor storage device of claim 1, wherein: the motion estimation value is a summation of an absolute value of a sum of differences of a plurality of pixels.

5. The semiconductor storage device of claim 1, further comprising: an estimation value calculation mechanism configured to calculate the motion estimation value; a comparison mechanism configured to compare the motion estimation value calculated by said estimation value calculation mechanism with a stored estimation value that is stored in said second storage portion; and a mechanism configured to change a first motion information to a second motion information when indicated to do so by a predetermined comparison result of said comparison mechanism.

6. The semiconductor storage device of claim 1, wherein: the second motion information being a minimal estimation value.

7. The semiconductor storage device of claim 5, wherein: the estimation value calculation mechanism includes a first calculation mechanism configured to calculate an absolute value of a difference between the first pixel information and the second pixel information, a second calculation mechanism configured to calculate an absolute value of a difference between the first pixel information and third pixel information, and a summation mechanism configured to add absolute values of differences produced by the first and second calculations mechanisms with the absolute values of differences of other elements selected by a selection mechanism, wherein the estimation value is a summation of an absolute value of a difference of a plurality of pixels.

8. The semiconductor storage device according to claim 1, wherein said data structure of said memory further comprising: a plane information portion that is configured to hold plane information about an object.

9. The semiconductor storage device of claim 8, wherein: the motion estimation value is calculated from the plane information.

10. The semiconductor storage device according to claim 9, further comprising: an estimation value calculation mechanism configured to calculate the motion estimation value according to a predetermined formula; a comparison mechanism configured to compare the motion estimation value calculated by said estimation value calculation mechanism from the motion estimation value stored in said memory; and a mechanism configured to change a pixel value associated with a pixel determined to have moved by said comparison mechanism.

11. The semiconductor storage device according to claim 10, wherein: the estimation value calculation mechanism includes a first calculation section configured to calculate an absolute value of a difference between the first pixel value and second pixel value, a selection mechanism configured to select another memory element in accordance with the plane information, and a summation section configured to add an absolute value of a difference of a first memory element with the absolute value of a difference of the another memory element selected by said selection mechanism, and wherein the motion estimation value is a summation of an absolute value of a difference of a plurality of memory elements.

12. A semiconductor storage device comprising: a data storage unit having portions, each of said portions including, a first storage portion for holding pixel motion information, a second storage portion for holding a motion estimation value, indicative of a frame-to-frame motion between pixels, a third storage portion for holding first pixel information, and a fourth storage portion for holding second pixel information, wherein a first frame that includes the first pixel is different from a second frame that includes the second pixel.

13. The semiconductor storage device of claim 12, further comprising: an estimation value calculation mechanism configured to calculate the motion estimation value; a comparison mechanism configured to compare the motion estimation value calculated by said estimation value calculation mechanism from the motion estimation value stored in said second storage portion, and a mechanism configured to change a pixel value associated with a pixel determined to have moved by said comparison mechanism.

14. The semiconductor storage device according to claim 13, wherein: the motion estimation value stored in said second storage portion is a minimal motion estimation value.

15. The semiconductor storage device according to claim 14, further comprising: a fifth storage portion configured to hold third pixel information, wherein a third frame includes the third pixel information, said third frame being different than said first frame and said second frame.

16. The semiconductor storage device of claim 9, further comprising: a noise addition mechanism configured to add a noise component to an input image, said input image including said first pixel information.

17. The semiconductor storage device of claim 16, further comprising: a readout unit configured to output corresponding pixels for respective elements of different frames, said different frames being separated from one another in time; a class code generating unit configured to receive said corresponding pixels and detect said noise component of said input image and provide a class code output; and a normal equation addition unit configured to receive said input image, said corresponding pixels from said readout unit and said class code output, and determine a coefficient for said class code that is output by said class code generating unit, and output said coefficient to a coefficient storage device.

18. The semiconductor storage device of claim 17, further comprising: a control unit configured to control a sequence of image data to be read out of reference regions in said different frames.

19. The semiconductor storage device of claim 9, further comprising: a readout unit configured to output corresponding pixels for respective elements of different frames, said different frames being separated from one another in time; a class code generating unit configured to receive said corresponding pixels and provide a class code output used in determining a prediction coefficient for said class code; and an estimation computation unit configured to estimate a prediction value from said prediction coefficient and pixels from the respective elements of the different frames.

20. The semiconductor storage device according to claim 12, further comprising: a class code generating unit configured to receive said corresponding pixels and provide a class code output used in determining a prediction coefficient for said class code; and an estimation computation unit configured to estimate a prediction value from said prediction coefficient and pixels from the respective elements of the different frames.

21. The semiconductor storage device of claim 15, wherein: the estimation computation unit includes a first calculation mechanism configured to calculate an absolute value of a difference between the first pixel information and the second pixel information, a second calculation mechanism configured to calculate an absolute value of a difference between the first pixel information and the third pixel information, a selection mechanism configured to select other pixel information in accordance with plane information, and a summation mechanism configured to add absolute values of differences produced by the first and second calculations mechanisms with the absolute values of differences of other elements selected by the selection mechanism, and wherein the estimation value is a summation of an absolute value of a difference of a plurality of elements.

22. The semiconductor storage device of claim 21, wherein: at least one of a memory pixel search region and memory pixel reference region being a 5.times.5 region.

23. The semiconductor storage device of claim 22, further comprising: a motion vector correction unit configured to correct a motion vector based on a frequency with which motion vectors are read out.

24. The semiconductor storage device of claim 23, wherein: said motion vector correction unit is configured to determine the motion vector to be used for characterizing pixel motion by selecting a motion vector having a highest frequency.

25. A method for processing image data for facilitating an estimation of a motion vector, comprising steps of: storing a first pixel information in a first pixel portion of a memory element configured to hold data indicative of a first pixel of an image block; storing a second pixel information in a second pixel portion of the memory element configured to hold data indicative of a second pixel of a search region of another image block; storing an estimation information in an estimation portion of the memory element configured to hold data representative of a motion estimation value between the first pixel and the second pixel, and storing motion vector information in a motion vector portion of the memory element configured to hold data representative of a motion vector between the first pixel and the second pixel; wherein the memory element is one memory element of a plurality, and each of the memory elements has the same data structure.

26. The method of claim 25, further comprising: replacing another difference value with said motion estimation value in said motion vector portion when an absolute value of said motion estimation value is less than said another difference value.

27. The method of claim 25, further comprising: determining the motion estimation value by taking an absolute value of a difference between the first pixel and second pixel.

28. The method of claim 25, wherein: determining the motion estimation value by taking a summation of an absolute value of a sum of differences of a plurality of pixels.

29. The method of claim 25, further comprising: calculating the motion estimation value; comparing the motion estimation value calculated in the calculating step with a stored estimation value that is stored in said second storage portion; and changing a motion information when indicated by a comparison result of said comparing step.

30. The method according to claim 29, wherein: the changing step includes changing said motion information to a minimal estimation value.

31. The method of claim 29, wherein: the calculating step includes calculating an absolute value of a difference between the first pixel information and the second pixel information, calculating an absolute value of a difference between the first pixel information and a third pixel information, and adding absolute values of differences produced by the calculating steps with the absolute values of differences of other elements selected by a selection mechanism, wherein the estimation value is a summation of an absolute value of a difference of a plurality of pixels.

32. The method according to claim 25, further comprising: storing plane information about an object in a plane information portion of the memory.

33. The method according to claim 32, further comprising: calculating the motion estimation value from the plane information.

34. The method according to claim 33, further comprising: calculating the motion estimation value according to a predetermined formula; comparing the motion estimation value calculated by said estimation value calculation mechanism with the motion estimation value stored in said memory, and changing a pixel value associated with a pixel determined to have moved in said comparing step.

35. The method according to claim 34, wherein: the calculating the motion estimation value step includes calculating an absolute value of a difference between the first pixel and second pixel, selecting another memory element in accordance with the plane information, and adding an absolute value of a difference of a first element with the absolute value of a difference of the another memory element selected in said selecting step, and wherein the motion estimation value is a summation of an absolute value of a difference of a plurality of memory elements.

36. A method for processing image data for facilitating an estimation of a motion vector, comprising steps of: storing in a first storage portion of a semiconductor memory pixel motion information; storing in a second storage portion of the semiconductor memory a motion estimation value, indicative of a frame-to-frame motion between pixels, storing in a third storage portion of the semiconductor memory first pixel information, and storing in a fourth storage portion of the semiconductor memory second pixel information, wherein a first frame that includes the first pixel is different from a second frame that includes the second pixel, and the semiconductor memory is one memory element of a plurality, and each of the memory elements has the same data structure.

37. The method according to claim 36, further comprising: calculating the motion estimation value with an estimation value calculation mechanism; comparing the motion estimation value calculated by said estimation value calculation mechanism from the motion estimation value stored in said second storage portion; and changing a pixel value associated with a pixel determined to have moved in said comparing step.

38. The method according to claim 37, wherein: the motion estimation value stored in said second storage portion being a minimal motion estimation value.

39. The method according to claim 38, further comprising: storing third pixel information in a fifth storage portion of said semiconductor memory, wherein a third frame includes the third pixel information, said third frame being different than said first frame and said second frame.

40. The method according to claim 36, further comprising: adding a noise component to an input image, said input image including said first pixel information.

41. The method according to claim 40, further comprising: reading out corresponding pixels for respective elements of different frames, said different frames being separated from one another in time; receiving said corresponding pixels at a class code generating unit, detecting said noise component, and providing a class code output; and determining a coefficient for said class code, and outputting said coefficient to a coefficient storage device.

42. The method according to claim 41, further comprising: controlling a sequence of image data to be read out of reference regions in said different frames.

43. The method according to claim 36, further comprising: outputting corresponding pixels for respective elements of different frames, said different frames being separated from one another in time; receiving said corresponding pixels at a class code generating unit and providing a class code output used in determining a prediction coefficient for said class code; and estimating a prediction value from said prediction coefficient and pixels from the respective elements of the different frames.

44. The method according to claim 37, further comprising: receiving said corresponding pixels at a class code generating unit and providing a class code output used in determining a prediction coefficient for said class code; and estimating a prediction value from said prediction coefficient and pixels from the respective elements of the different frames.

45. The method according to claim 44, wherein: the estimation step includes calculating an absolute value of a difference between the first pixel information and the second pixel information, calculating an absolute value of a difference between the first pixel information and third pixel information, selecting other pixel information in accordance with plane information, adding absolute values of differences produced by the calculating steps with the absolute values of differences of other elements selected in the selecting step, and wherein the estimation value is a summation of an absolute value of a difference of a plurality of elements.

46. The method according to claim 45, wherein: at least one of a pixel search region and a pixel reference region being a 5.times.5 region.

47. The method according to claim 46, further comprising: a motion vector correction unit configured to correct a motion vector based on a frequency with which motion vectors are read out.

48. The method according to claim 47, further comprising: determining the motion vector to be used for characterizing pixel motion by selecting a motion vector having a highest frequency.

49. An apparatus for processing image data for facilitating an estimation of a motion vector, comprising: means for calculating motion vectors; and a memory including a plurality of memory elements, each memory element having a data structure that includes, a first pixel information portion configured to hold data indicative of a first pixel of an image block, a second pixel information portion configured to hold data indicative of a second pixel of a search region of another image block, an estimation portion configured to hold data representative of a motion estimation value between the first pixel and the second pixel, and a motion vector information portion configured to hold data representativc of a motion vector between the first pixel and the second pixel.

---

Description

---

CROSS-REFERENCE TO RELATED PATENT APPLICATION

The present application contains subject matter related to co-pending U.S. patent application, Ser. No. 10/640,380, filed Aug. 14, 2003, entitled SEMICONDUCTOR DEVICE, IMAGE DATA PROCESSING APPARATUS AND METHOD, bearing, commonly owned by Sony Corporation, having a common inventor Tetsujiro Kondo, based on Japanese priority document JP 2002-356530, filed in Japan on Dec. 9, 2002, the entire contents of which being incorporated herein by reference. The present application also contain subject matter related to Japanese priority documents JP 2002-274059, filed in the JPO on Aug. 15, 2002, and JP 2002-274058, filed in the JPO on Aug. 15, 2002, the entire contents of each of which being incorporated herein by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an image processing device, computer program product and an image processing method for generating motion vectors based upon image data, for example.

2. Discussion of the Background

Conventionally, block matching has been known as a method for detecting motion vectors between frames of an image. FIG. 1 is a diagram that illustrates a general image processing device, and FIG. 2 is a flowchart that illustrates the operation of the general image processing device shown in FIG. 1.

In reference to FIGS. 1 and 2, "Block matching" is a process wherein a motion vector is obtained, which indicates what position a certain motion block, in an image at a certain point in time T-1, moves to at a next point in time T.

For example, an image processing device 1s (FIG. 1) comprises frame memory 3s-T, frame memory 3s-T-1, search region memory ms, reference block memory mr, phase block memory mp, computation unit pc, evaluation table memory mv, and minimal value search unit ps.

The image processing device 1s sets a search region at a point in time T so as to perform matching for pixels in a block within the search region with regard to each phase (position) as to a reference block at a point in time T-1, and the phase scoring of the best match is taken as a position of the block at the point in time T. Here, the shift amount of the block between two frames is defined as the motion vector.

Referring to FIG. 2, a flowchart describes the operation of the image processing device 1s. The image data of the search region and the image data of reference block br are each read out from frame memory T and frame memory T-1 wherein continuous input images are stored with regard to time (ST1, ST2). These image data sets are stored in the search region memory ms and the reference block memory mr (ST3).

Next, the block with the same shape and the same size as the reference block br is read out from the search region memory ms for each phase. The block is defined as a phase block bp. The phase is sequentially updated at a point in time when recording to the evaluation table has ended, as described later, and the phase block bp is read out at the phase at the point of time (ST4). The phase block bp read out is stored in the phase block memory mp as shown in FIG. 1, for example.

FIG. 3 (which has two parts--FIG. 3a and FIG. 3b) is a diagram for describing a specific process example beginning with how a readout of the phase block operation is performed with phase shift, up to recording in the evaluation table memory mr. FIG. 3(a) is a diagram which illustrates the state prior to the phase shift, and FIG. 3(b) illustrates the state following the phase shift.

Referring to FIG. 3, the operations up to, and including, recording in the evaluation table memory will be described. Presume that the size of the reference block is 3.times.3 pixels, and the size of the search region is 5.times.5 pixels. Here, the phase block bp read out at the point of the phase 1 is shifted in phase 2 to a position offset by one pixel in the horizontal direction. For example, the shift amount of the phase due to updating is one pixel. Next, the absolute value of the difference is computed by the computation unit pc for each pair of corresponding pixels between the reference block br and phase block bp, the sum of absolute value of the difference S is obtained for all the absolute values of difference computed.

.times..times. ##EQU00001##

The sum of absolute value of the difference S is calculated with Expression (1), for example, using the pixel values ks in the reference block br and the corresponding pixel values kp in the phase block bp (ST5).

Next, the sum of absolute value of the difference S is stored in the evaluation table (ST6). Specifically, the sum of absolute values of difference S is stored in the evaluation table memory mv, as shown in FIG. 1. The evaluation table stores the sum of absolute values of difference S for each phase corresponding to the phase block read out.

In general, with the image processing device 1s, upon the phase being shifted from the shift 1 to the shift 2, the position at which the data is stored in the evaluation table is shifted corresponding to the shift amount as shown in FIG. 3.

Upon the processing from readout of phase blocks bp up to recording in the evaluation table ending for all the search regions (ST7), the minimal value search unit ps searches the minimal value within the evaluation table (ST8).

On the other hand, in the event that judgment is made that the evaluation table does not store the data for all the phases in Step ST7, the flow returns to processing in Step S3.

Thus, the image processing device 1s searches the block within the search region, of which value is closest to that of the reference block br.

As described above, the evaluation table stores the sum-of-absolute-values-of-difference S for each phase corresponding to the phase block bp read out. Accordingly, the phase of the block within the search region, corresponding to the reference block br, can be found by searching the minimal value within the evaluation table. The information with regard to the phase is output as a motion vector.

SUMMARY OF THE INVENTION

An object of the present invention is to address the above-identified and other limitations with conventional devices, methods and computer program product.

The present invention has been made taking the above-described problems into consideration, and it is an object of the present invention to provide an image processing device and an image processing method, which can calculate motion vectors with small amounts of calculation required.

There are three problems, as described below, with the above-described conventional image processing device, and the present invention is configured to address these problems.

1. Procedures are needed to temporarily store values of the phase block and the evaluation table, and so there is a need to provide memory for storing data within a circuit, leading to circuit layout complexity. Furthermore, in the event that motion vectors are calculated at a number of positions within a frame, for example, there is the need to sequentially search a motion vector for each position, leading to a loss of time.

2. A procedure to search for the minimal value is started at the point of all the values being stored in the evaluation table. Accordingly, there is the need to update the phase in two ways: readout of each phase block, and search of the minimal value.

3. In this case, the output is a motion vector, so when reading out pixels extracted based upon the motion vector, for example, another procedure for extracting the pixels based upon the motion vector is necessary.

In order to address the above-identified and other deficiencies with conventional devices and methods, the image processing device according to a first aspect of the present invention, a semiconductor storage device is provided that includes a memory having a data structure that includes a first pixel information portion configured to hold data indicative of a first pixel of an image block, a second pixel information portion configured to hold data indicative of a second pixel of a search region of another image block, an estimation portion configured to hold data representative of a motion estimation value between the first pixel and the second pixel, and a motion vector information portion configured to hold data representative of a motion vector between the first pixel and the second pixel.

According to a second aspect of the invention, a semiconductor storage device is provided that includes a data storage unit having portions, the portions including a first storage portion for holding pixel motion information, a second storage portion for holding a motion estimation value, indicative of a frame-to-frame motion between pixels, a third storage portion for holding first pixel information, and a fourth storage portion for holding second pixel information, wherein a first frame that includes the first pixel is different from a second frame that includes the second pixel.

According to a third aspect of the present invention, a method is provided for processing image data for facilitating an estimation of a motion vector, including steps of storing a first pixel information in a first pixel portion of a memory element configured to hold data indicative of a first pixel of an image block; storing a second pixel information in a second pixel portion of the memory element configured to hold data indicative of a second pixel of a search region of another image block; storing an estimation information in an estimation portion of the memory element configured to hold data representative of a motion estimation value between the first pixel and the second pixel, and storing motion vector information in a motion vector portion of the memory element configured to hold data representative of a motion vector between the first pixel and the second pixel.

According to a fourth aspect of the present invention, a method is provided for processing image data for facilitating an estimation of a motion vector, including steps of storing in a first storage portion pixel motion information; storing in a second storage portion a motion estimation value, indicative of a frame-to-frame motion between pixels, storing in a third storage portion first pixel information, and storing in a fourth storage portion second pixel information, wherein a first frame that includes the first pixel is different from a second frame that includes the second pixel.

BRIEF DESCRIPTION OF THE DRAWINGS

A more complete appreciation of the invention and many of the attendant advantages thereof will be readily obtained as the same becomes better understood by reference to the following detailed description when considered in connection with the accompanying drawings, wherein:

FIG. 1 is a diagram, which illustrates a general image processing device.

FIG. 2 is a flowchart, which illustrates the operations of the general image processing device shown in FIG. 1.

FIG. 3 (which includes FIGS. 3a and 3b) is a diagram for describing a specific example of an operation beginning with reading out a phase block with phase shift up through a step of storing in an evaluation table memory.

FIG. 4 is a functional block diagram that illustrates an image processing device of a first embodiment according to the present invention.

FIG. 5(a) is a diagram that illustrates a memory of the image processing device according to the first embodiment, and FIG. 5(b) is a functional block diagram that illustrates an element within the memory of the image processing device shown in FIG. 5(a).

FIG. 6 (which includes FIGS. 6(a) and 6(b)) shows diagrams for describing updating of the phase for pixel matching and readout of a search region tp, performed by the image processing device shown in FIG. 4. FIG. 6(a) is a diagram which illustrates the phase within the search region. FIG. 6(b) is a diagram which illustrates the relation between the phase and the search region tp.

FIG. 7 is a flowchart for describing the operations of the image processing device 1 shown in FIG. 1.

FIG. 8 is a functional block diagram that illustrates an image processing device of a second embodiment according to the present invention.

FIG. 9(a) is a diagram that illustrates a memory of the image processing device 1a shown in FIG. 8. FIG. 9(b) is a diagram that illustrates a specific example of the memory structure shown in FIG. 8. FIG. 9(c) is a diagram that illustrates the memory structure for storing motion vectors of the image processing device 1a shown in FIG. 8.

FIG. 10 (which includes FIGS. 10(a) and 10(b)) shows diagrams for describing the operations of the image processing device shown in FIG. 8, in particular, the operations for performing updating of the phase and readout of the search region tp. FIG. 10(a) is a diagram which illustrates the phase within the search region sp. FIG. 10(b) is a diagram which illustrates the relation between the phase and the search region tp.

FIG. 11 is a flowchart for describing the operations of the image processing device 1a shown in FIG. 8.

FIG. 12 is a diagram which illustrates blocks extracted from a reference region tr of the frame of the image processing device shown in FIG. 8.

FIG. 13 is a conceptual diagram for describing extraction of corresponding pixels for an image processing device according to a third embodiment of the present invention.

FIG. 14 is a functional block diagram of the image processing device of the third embodiment according to the present invention.

FIG. 15(a) is a diagram that illustrates the memory 2b of the image processing device 1b shown in FIG. 14. FIG. 15(b) is a diagram that illustrates memory for storing the corresponding pixels. FIG. 15(c) is a functional block diagram that illustrates an element 20b of the memory 2b shown in FIG. 15(a).

FIG. 16(a) is a diagram which illustrates the phases of the search region spT+2 (9.times.9) of the frame T+2 and the search region tpT+2. FIG. 16(b) is a diagram which illustrates the phases of the search region spT+1 (7.times.7) of the frame T+1 and the search region tpT+1. FIG. 16(c) is a diagram which illustrates the phases of the search region spT-1 (7.times.7) of the frame T-1 and the search region tpT-1. FIG. 16(d) is a diagram which illustrates the phases of the search region spT-2 (9.times.9) of the frame T-2 and the search region tpT-2. FIG. 16(e) is a diagram for describing updating of the phase and readout processing for the search region tp, performed by the image processing device 1b shown in FIG. 14.

FIG. 17 is a flowchart for describing the operations of the image processing device shown in FIG. 14.

FIG. 18 is a functional block diagram of an image processing device of a fourth embodiment of the present invention.

FIG. 19 is a functional block diagram that illustrates a first specific example of the noise addition unit of the image processing device 1c shown in FIG. 18.

FIG. 20 is a functional block diagram that illustrates a second specific example of the noise addition unit of the image processing device 1c shown in FIG. 18.

FIG. 21 is a functional block diagram that illustrates a third specific example of the noise addition unit of the image processing device 1c shown in FIG. 18.

FIG. 22 is a diagram for describing the principle for generating a class code (detection of noise components) in the class code generating unit of the image processing device 1c shown in FIG. 18.

FIG. 23 shows diagrams for describing the procedures for reading out the reference region tr to the memory for class classification adaptation processing performed by the image processing device 1c shown in FIG. 18. FIG. 23(a) is a diagram that illustrates the state prior to the shift. FIG. 23(b) is a diagram that illustrates the state following the shift.

FIG. 24 is a functional block diagram that illustrates an image processing device of a fifth embodiment according to the present invention.

FIG. 25 is a functional block diagram that illustrates an image processing device of a sixth embodiment according to the present invention.

FIG. 26(a) is a configuration diagram which illustrates the memory of the image processing device shown in FIG. 25, FIG. 26(b) is a diagram which illustrates an element of the memory shown in FIG. 26(a) for storing a prediction pixel, and FIG. 26(c) is a functional block diagram which illustrates the element of the memory shown in FIG. 26(a).

FIG. 27 is a conceptual diagram for describing the operations of an image processing device according to a seventh embodiment of the present invention.

FIG. 28 is a conceptual diagram for describing the operations of plane matching performed by the image processing device of the seventh embodiment according to the present invention.

FIG. 29 is a functional block diagram that illustrates the image processing device of the seventh embodiment according to the present invention.

FIG. 30(a) is a schematic diagram that illustrates the memory of the image processing device shown in FIG. 29. FIG. 30(b) is a partial enlarged diagram of FIG. 30(a). FIG. 30(c) is a functional block diagram that illustrates the element of the memory shown in FIG. 30(b).

FIG. 31 is a diagram for describing the operations of the plane separation unit of the image processing device shown in FIG. 29.

FIG. 32 is a functional block diagram that illustrates a specific example of the plane separation unit of the image processing device 1f.

FIG. 33 is a diagram that illustrates a specific example of a histogram generated by the histogram computation unit of the plane separation unit of the image processing device 1f shown in FIG. 32.

FIG. 34 is a specific example of a histogram for describing the operations of the elimination unit of the plane separation unit of the image processing device shown in FIG. 32.

FIG. 35 is a diagram for describing troughs of a histogram prior to integration by the trough integration unit of the image processing device shown in FIG. 32.

FIG. 36 is a diagram for describing troughs of the histogram following integration by the trough integration unit of the image processing device shown in FIG. 32.

FIG. 37 is a flowchart for describing the operations of the plane separation unit of the image processing device shown in FIG. 32.

FIG. 38 is a flowchart for describing the operations of the image processing device shown in FIG. 29.

FIG. 39 is a functional block diagram that illustrates an image processing device of an eighth embodiment according to the present invention.

FIG. 40 is a functional block diagram of an image processing device of a ninth embodiment according to the present invention.

FIG. 41 is a diagram for describing the operations of the motion vector correction unit of the image processing device shown in FIG. 40.

DETAILED DESCRIPTION OF THE INVENTION

First Embodiment

An image processing device 1 according to the present embodiment detects a motion vector between frames of a moving image having multiple frames. One aspect of detecting the motion vector is by pixel matching, as discussed below.

Pixel Matching

FIG. 4 is a functional block diagram that illustrates a first embodiment of an image processing device according to the present invention. FIG. 5(a) is a diagram that illustrates a memory structure, or data structure, of the image processing device according to the first embodiment. FIG. 5(b) is a functional block diagram that illustrates an element within the memory of the image processing device shown in FIG. 5(a).

As shown in FIG. 4, the image processing device 1 according to the present embodiment includes memory 2, multiple frame memory 3 (shown as 3-T and 3-T-1), readout unit 4, and a control unit 100. The memory 2 generates and stores a motion vector based on predetermined data from the frame memory 3, e.g., pixel values of pixels, and phases (which are also referred to as phase information), as described later. The frame memory 3-T stores the image data (which is also simply referred to as image) input at the point in time T. The frame memory 3-T-1 stores the image data at the point in time T-1. Furthermore, after a predetermined time period, the image data stored in the frame memory 3-T is stored in the frame memory 3-T-1, and the frame memory 3-T stores next image data.

The readout unit 4 reads out a motion vector stored in the memory 2 for each element 20 (FIG. 5(a)). The control unit 100 controls the memory 2, the frame memory 3, and the readout unit 4 according to a computer-implemented process. The control unit 100 controls the frame memory 3 to store an image, for example. The control unit 100 controls the frame memory 3 to output the pixels within a predetermined search region sp and reference region, and the phases of a search region tp of the search region sp and the reference region tr, to the memory 2, for example.

A more detailed description is now made regarding each component. The memory 2 of the image processing device 1 according to the present embodiment has multiple elements 20 in a grid shape, as shown in FIG. 5(a), for example. For simplification, presume that 3.times.3 elements (pixels) 20-1 through 20-9, are provided in the memory 2. The elements 20 are each connected.

A description is now made regarding a case where matching is performed for each element (pixel) 20 at a time. The element 20 has multiple pixels 21, e.g., pixels 21-1 and 21-2, a minimal pixel difference storage unit 22, motion vector storage unit 23, a subtracter 24, an absolute value generating unit 25, a comparison updating unit 26, and updating unit 27, as shown in FIG. 5(b).

Specifically, the element 20 not only memorizes (also referred to as "stores") one pixel, but also stores pixel values for two pixels, for example. Furthermore, the memory structure also computes the absolute value of the difference of the two pixels, and stores the obtained absolute value of the difference. "Matching data" is data with regard to a predetermined matching processing, such as pixel values, absolute values of difference, data indicating results of predetermined computation, and the like, for example.

The pixel 21 has multiple pixels, e.g., the pixel 21-1 and pixel 21-2. The pixel value of a pixel within the search region sp within the frame memory 3-T is input to the pixel 21-1, for example. To the pixel 21-2, the pixel value of a pixel within the reference region tr within the frame memory 3-T-1 is input, for example. The minimal pixel difference storage unit 22 stores the absolute value of the difference. With the minimal pixel difference storage unit 22, updating of the stored absolute value of the difference is performed by the comparison updating unit 26. The motion vector storage unit 23 stores a motion vector as described later. With the motion vector storage unit 23, updating of the motion vector is performed by the updating unit 27. The subtracter 24 outputs the difference value between the pixel value stored in the pixel 21-1 and the pixel value stored in the pixel 21-2, to the absolute value generating unit 25.

The absolute value generating unit 25 outputs the absolute value of the difference (which is also referred to as "absolute-value-of-difference") to the comparison updating unit 26 based upon the difference value between the pixel value of the pixel 21-1 and the pixel value of the pixel 21-2, output from the subtracter 24. The comparison updating unit 26 compares the absolute value of the difference output from the absolute value generating unit 25 with the absolute value of the difference stored in the minimal value difference storage unit 22.

As a result of the comparison, in the event that the absolute value of the difference output from the absolute value generating unit 25 is a smaller value, the comparison updating unit 26 stores the smaller value in the minimal pixel storage unit 22, thereby updating the absolute value of the difference. In the event the absolute value of the difference stored in the minimal pixel difference storage unit 22 according to the above-described comparison results is updated, the comparison updating unit 26 outputs updating information, indicating that the absolute value of the difference has been updated, to the updating unit 27. The updating unit 27 then updates the motion vector stored in the motion vector storage unit 23 based on the updating information output from the comparison updating unit 26 and the phase information. Specifically, in the event the comparison updating unit 26 outputs the updating information indicating that the absolute value of the difference has been updated, the updating unit 27 controls the motion vector storage unit 23 to store the phase information (phase) with regard to the search region tp and the reference region tr at the point in time, as a motion vector, thereby updating the motion vector. As described above, the memory 2 stores the information other than pixel values, such as the motion vectors and absolute values of difference, as addition information.

The subtracter 24, absolute value generating unit 25, comparison updating unit 26, updating unit 27 are all implemented via circuitry hosted on the memory chip. This circuitry may be programmable, programmed logic, fixed circuits, or any combination thereof.

FIG. 6 shows diagrams for describing updating of the phase and readout of the search region tp, with regard to pixel matching for the image processing device shown in FIG. 4. FIG. 6(a) is a diagram that illustrates the phase within the search region. FIG. 6(b) illustrates the relation between the phases and the search region tp. The phase (phase information) is an offset between the centers of the search region sp and the search region tp, for example. For example, the phase is defined as shown in FIG. 6(a). Specifically, as shown in FIG. 6(a), in the event that the center of the search region sp (5.times.5) matches the center of the search region tp (3.times.3), the phase is (0, 0). In the event that the center of the search region sp (5.times.5) is offset from the center of the search region tp (3.times.3) by one pixel toward the right side in the drawing, the phase is (1, 0). With the phases other than the above-described examples, the offset of the center of the search region sp from the center of the search region tp is defined as phase coordinates in the same way.

FIG. 6(b) is a diagram that illustrates updating of the phase and readout of the search region tp. For example, the control unit 100 updates phases, (-1, -1), (0, -1), (1, -1), (-1, 0), (0, 0), (1, 0), (-1, 1), (0, 1), (1, 1), in that order, reads out the pixels within the search region tp corresponding to the updated phase, and outputs the pixels to the corresponding elements 20 within the memory 2. As described later, the phase where the absolute value of the difference is the minimal value is taken as the true motion vector.

Note that the order for readout of pixels, and the order for updating of phases, for the control unit 100, is not restricted to this arrangement. All the pixels should be read out, and the order of readout is not restricted to the above-described arrangement. Here, in FIG. 5, the pixel 21-1 included in each element stores one pixel within the search region tp.

FIG. 7 is a flowchart for describing operations of the image processing device 1 shown in FIG. 1. Description will be made regarding the operations of the image processing device 1, particularly, the operations for generating a motion vector, with reference to FIG. 7. As an example, the control unit 100 stores each pixel of an arbitrary 3.times.3 reference region tr from the frame memory 3-T-1 in the corresponding element 20 within the memory 20 (ST11). Specifically, as shown in FIG. 4, one pixel of the reference region tr is stored in the pixel 21-2 of each element 20. The control unit 100 reads out the pixels within the search region tp (3.times.3) in the frame memory 3-T while updating (shifting) the phase within the search region sp (ST12), and stores each pixel in the corresponding element 20 within the memory 2 (ST13). Specifically, as shown in FIG. 4, the control unit 100 stores a corresponding pixel in the pixel 21-1 of each element 20. With the control unit 100, updating of the phases of the search region tp, and the readout of the pixels within the search region tp are performed nine times in total while shifting the phase by one pixel in the range between (-1, -1) and (1, 1) as shown in FIG. 6, for example, and following performing the processing nine times, the processing ends.

The absolute value of the difference between the pixels stored in the pixel 21-1 and pixel 21-2 is computed by the subtracter 24 and the absolute value generating unit 25 of each element 20. The obtained value is defined as the absolute-value-of-difference "a" (ST14). The comparison updating unit 26 compares the absolute-value-of-difference "a" with the absolute value of the difference (which will be referred to as "absolute-value-of-difference b", for example) stored in the minimal value difference storage unit 22 of the element 20 within the memory 2 (ST15). As a result of the comparison in Step ST15, in the event that the absolute-value-of-difference "a" is smaller, the comparison updating unit 26 stores the absolute-value-of-difference "a" in the minimal value difference storage unit 22, thereby updating the absolute value of the difference (ST16, ST17). Conversely, in the event that the absolute-value-of-difference "a" is greater, the comparison updating unit 26 does not perform updating (ST16), the flow proceeds to the processing for updating of the phase of the next search region tp, and for readout of the pixels within the search region tp.

Note that, in the initial stage, the minimal value difference storage unit 22 of the element 20 does not store the absolute value of the difference. In this case, the minimal value difference storage unit 22 stores the initial absolute values of difference without restriction.

In Step ST17, the comparison updating unit 26 outputs updating information indicating that the absolute value of the difference has been updated only in the case where the absolute value of the difference is updated, via the updating unit 27.

In the event that the comparison updating unit 26 outputs the updating information indicating that the absolute value of the difference has been updated, to the updating unit 27, the updating unit 27 causes the motion vector storage unit 23 to store the phase of the search region tp at this point in time, thereby updating the motion vector (ST18).

At the point that updating of all the phases ends (ST19), with each element 20, the minimal pixel difference storage unit 21-2 stores the minimal value of the absolute value of the difference between the pixels for the search region sp, and the phase (motion vector) wherein the minimal value has been updated is stored in the motion vector storage unit 23. On the other hand, in Step ST19, in the event that updating of all the phases does not end, the flow returns to the processing in Step ST12.

Each motion vector is obtained based upon the absolute value of the difference between the pixels 20, which indicates what position within the search region sp each pixel 20 within the reference region tr moves to at the time T. Finally, the readout unit 4 reads out each motion vector from the corresponding element 20, and outputs the motion vector.

As described above, the multiple frame memory 3 for storing image data, the memory 2 for generating and storing motion vectors based upon the data from the frame memory 3, the readout unit 4 for reading out a motion vector for each element from the memory, and the control unit 100 for controlling these units, are provided. Furthermore, the multiple grid-shaped elements 20 are provided to the memory 2.

The element 20 comprises pixels 21-1-1 and 21-1-2 for storing pixel values of the predetermined pixels of different frame memory, the subtracter 24 for outputting the difference value between the pixel 21-1-1 and the pixel 21-1-2, the absolute generating unit 25 for outputting the absolute value of the difference (absolute-value-of-difference) output from the subtracter 24, the minimal pixel difference storage unit 22 for storing the absolute value of the difference, the comparison updating unit 26 for comparing the obtained absolute value of the difference and the absolute value of the difference stored in the minimal pixel storage unit 22, updating the absolute value of the difference stored in the minimal pixel difference storage unit 22 according to the result of comparison, and outputting updating information indicating that updating has been performed, and the updating unit 27 for making the motion vector storage unit 23 store the phase coordinates at that point as the motion vector so as to update the motion vector in the event that the updating information indicating that the absolute value of the difference has been updated is output from the comparison updating unit 26, thereby enabling the motion vector to be calculated with a relatively small amount of calculation, and to be done so in a timely manner.

Furthermore, two procedures of phase updating for computation of the evaluation table and search of the minimal value for all the phases within the search region sp can be performed at a same time.

Furthermore, with the present embodiment, only the minimal value of the absolute value of the difference is updated, and accordingly, there is no need to generate an evaluation table on another memory for searching the minimal value, for example, thereby enabling the size of memory to be reduced.

Note that while the description of the image processing device 1 according to the present invention has been made with reference to the memory 2 with the size of 3.times.3, the size of memory is not restricted to the example. For example, in the event of providing the memory 2 having elements 20 for all the pixels of one frame, the motion vectors for all the pixels of one frame can be obtained at a time at the point that updating of the phase ends.

Second Embodiment

An image processing device 1 according to the present embodiment detects a motion vector between frames of a moving image having multiple frames. One aspect of detecting the motion vector is by block matching, as discussed below.

Block Matching

FIG. 8 is a functional block diagram that illustrates an image processing device 1a of the second embodiment according to the present invention.

FIG. 9(a) is a diagram that illustrates a memory structure of the image processing device 1a shown in FIG. 8. FIG. 9(b) is a diagram that illustrates a specific example of the memory structure shown in FIG. 8. FIG. 9(c) is a diagram that illustrates the memory structure for storing a motion vector, of the image processing device 1a shown in FIG. 8.

The image processing device 1a according to the present embodiment comprises a memory 2a, multiple frame memory 3a, e.g., frame memory 3a-T, T-1, the readout unit 4, and a control unit 100a, as shown in FIG. 8. With the image processing device 1 of the first embodiment according to the present invention, a motion vector for a certain pixel of interest is obtained with 3.times.3 block matching using 3.times.3 elements, for example.

With the image processing device 1a of the second embodiment, the motion vectors for the 3.times.3 pixels neighboring one to another are obtained at a same time. The memory 2a does not only store one pixel but also stores two pixels in the same way as with the memory 2 of the image processing device 1 according to the first embodiment, and calculates the absolute value of the difference therebetween. Furthermore, the memory 2a stores the obtained absolute value of the difference.

A description is now made regarding a specific example of block matching wherein pixel matching is applied.

The major difference between the image processing device 1a and the image processing device 1 according to the first embodiment is the difference in the memory structure. Accordingly, a description will now be made regarding only the differences.

The memory 2a of the image processing device 1a according to the present embodiment has multiple elements 20a in a grid shape as shown in FIG. 9, for example.

For simplification, 5.times.5 elements 20a-1 through 20a-25 are provided within the memory 2a. The elements 20 are each connected as shown.

A description is now made regarding a case of performing matching for each element 20 at a same time. Here, in the event of not specifying the elements 20a-1 through 20a-25, the element will be simply referred to as element (pixel) 20a. A description will be made taking the search region sp as the phase (-1, -1) through (1, 1) in increments of one pixel.

The memory 2a can transmit the absolute values of difference stored in the eight elements (e.g., elements 20a-1 through 20a-4, and elements 20a-6 through 20a-9) near the element of interest (e.g., element 20a-5), to the element of interest with the wiring between the elements 20a, as shown in FIG. 9(a). Each element 20a has a mechanism for receiving the nine absolute values of difference, and a mechanism for adding the absolute values of the difference. Thus, the elements of interest 20a obtain the sum of absolute values of the differences, and store the obtained sum. Note that each element 20a compares the obtained sum of the absolute value of the difference with the sum of absolute value of the difference stored beforehand, and in the event that the obtained value is smaller, the stored value is updated. In the event that the sum of absolute value of the difference is updated, the stored motion vector is updated with the motion vector corresponding to the obtained sum of the absolute value of the difference, as well.

A more detailed description now follows. The element 20a comprises multiple pixels 21a, e.g., pixels 21a-1 and 21a-2, a pixel difference storage unit 22a-1, a sum-of-difference storage unit 22a-2, the motion vector storage unit 23, the subtracter 24, a absolute value generating unit 25a, a comparison updating unit 26a, the updating unit 27, and the addition unit 28, as shown in FIG. 9(c). The difference between the element 20a and the element 20 according to the first embodiment includes the pixel difference storage unit 22a-1, the sum-of-difference storage unit 22a-2, the absolute value generating unit 25a, the comparison updating unit 26a, and the addition unit 28.

The pixel difference storage unit 22a-1 stores the absolute value of the difference between the pixel 21a-1 and the pixel 21a-2 output from the absolute value generating unit 25a. The pixel difference storage unit 22a-1 outputs the stored absolute value of the difference to the addition unit 28. The sum-of-difference storage unit 22a-2 stores the sum of absolute value of the difference. The sum of absolute value of the difference is updated by the comparison updating unit 26a in the event of predetermined conditions, as described later. The absolute value generating unit 25a outputs the absolute value of the difference between the pixel values of the pixel 21-1 and the pixel 21-2 (which will be also referred to as absolute-value-of-difference) output from the subtracter 24, to the pixel difference storage unit 22a-1.

The comparison updating unit 26a compares the sum-of-absolute-value-of-difference A output from the addition unit 28, with the sum of the absolute val