Title: Programmable image transform processor
Abstract: A programmable image transform system has a programmable addressing and arithmetic blocks. In the programmable addressing block, an input address generator has an input addressing microsequencer and an input addressing memory that stores an input addressing procedure. The microsequencer executes the input addressing procedure to generate addresses from which to request image data. In the programmable arithmetic block, an arithmetic block memory stores an image processing procedure and a microsequencer executes the image processing procedure using the image data to generate transformed image data.
Patent Number: 6,961,084 Issued on 11/01/2005 to Duncan, et al.
| Inventors:
|
Duncan; Kathleen A. (Santa Cruz, CA);
Livingston; Raymond S. (Bonny Doon, CA)
|
| Assignee:
|
ESS Technology, Inc. (Fremont, CA)
|
| Appl. No.:
|
679854 |
| Filed:
|
October 5, 2000 |
| Current U.S. Class: |
348/222.1; 348/231.99; 348/403.1; 375/240.19; 382/248 |
| Intern'l Class: |
H04N 005/22.8; H04N 005/76; H04N 007/12; G06K 009/36 |
| Field of Search: |
348/2211,231.99,403.1,405.1
375/246,240.23,240.19
382/232,246,248
|
References Cited [Referenced By]
U.S. Patent Documents
| 5345552 | Sep., 1994 | Brown.
| |
| 5428804 | Jun., 1995 | Davies.
| |
| 5606520 | Feb., 1997 | Gove et al.
| |
| 5669010 | Sep., 1997 | Duluk, Jr.
| |
| 5701450 | Dec., 1997 | Duncan.
| |
| 5729758 | Mar., 1998 | Inoue et al.
| |
| 5806072 | Sep., 1998 | Kuba et al.
| |
| 5812195 | Sep., 1998 | Zhang.
| |
| 5815680 | Sep., 1998 | Okumura et al.
| |
| 5920343 | Jul., 1999 | Watanabe et al.
| |
| 5923881 | Jul., 1999 | Fujii et al.
| |
| 5926583 | Jul., 1999 | Iwase et al.
| |
| 5982425 | Nov., 1999 | Allen et al.
| |
| 6020920 | Feb., 2000 | Anderson.
| |
| 6020982 | Feb., 2000 | Yamauchi et al.
| |
| 6072936 | Jun., 2000 | Koyama.
| |
| 6100928 | Aug., 2000 | Hata.
| |
| 6597394 | Jul., 2003 | Duncan et al.
| |
Primary Examiner: Christensen; Andrew
Assistant Examiner: Ye; Lin
Attorney, Agent or Firm: Farjami & Farjami LLP
Parent Case Text
CROSS REFERENCE TO RELATED APPLICATIONS
This application claims the benefit of U.S. Provisional Application No. 60/159,000,
entitled "Programmable Image Transform Processor," filed Oct. 7, 1999, that is
incorporated by reference.
U.S. Patent application, titled "Programmable Image Transform Processor for
a Digital Camera," Ser. No. 09/188,871, filed Nov. 9, 1998, incorporated by reference.
U.S. Patent application, titled "Programmable Timing Generator for a Digital
Camera," Ser. No. 09/188,831, filed Nov. 9, 1998, is incorporated by reference.
U.S. Patent application, titled "Programmable Display Controller for a Digital
Camera," Ser. No. 09/188,996, filed Nov. 9, 1998, is incorporated by reference.
Claims
1. An image transform processor for processing image data, comprising:
a programmable arithmetic processor capable of receipt of the image data from
a data source over a data path and processing the image data, the programmable
arithmetic processor comprising a first set of local buffers and a second set of
local buffers, each buffer in the first set of local buffers alternately used for
fetching input image data and each buffer in the second set of local buffers alternately
used for storing output image data; and
a programmable input addresser that controls transfer of the image data from
the data source to the programmable arithmetic processor by providing a source
address onto a source address path, the source address identifying the data source.
2. The image transform processor of claim 1, wherein the programmable input addresser
further controlling transfer of the image data to the programmable arithmetic processor
by providing a storage address to the programmable arithmetic processor, the storage
address identifying a location within the programmable arithmetic processor for
storage of the image data.
3. The image transform processor of claim 1, wherein the data source being a
frame capture processor, the source address identifying the frame capture processor.
4. The image transform processor of claim 1, wherein the data source being a
memory, the source address being a memory address identifying a location of the
image data within the memory.
5. The image transform processor of claim 1, wherein the data source being a
memory, the source address path being a read address bus coupled between the programmable
input addresser and the memory, the source address being a memory address identifying
a location of the image data within the memory.
6. The image transform processor of claim 1, the storage location within the
programmable arithmetic processor being a local buffer.
7. The image transform processor of claim 1 further comprising:
a programmable output addresser controlling transfer of the image data from the
programmable arithmetic processor to a memory by providing a write address onto
a write path, the write address identifying a write address in the memory for storage
of the image data.
8. The image transform processor of claim 7, wherein the write path is a write
address bus electrically connected to the programmable output addresser and the memory.
9. The image transform processor of claim 7, wherein the programmable output
addresser further controlling transfer of the image data by providing a retrieval
address to the programmable arithmetic processor, the retrieval address identifying
a location within the programmable arithmetic processor for retrieval of the image data.
10. The image transform processor of claim 9, wherein the retrieval location
within the programmable arithmetic processor is a buffer.
11. The image transform processor of claim 9, wherein the retrieval location
within the programmable arithmetic processor is at least one buffer of a plurality
of buffers.
12. An image transform processor for processing image data, the image transform
processor comprising:
a programmable arithmetic processor capable of receiving the image data from
a memory over a data bus coupled between the programmable arithmetic processor
and the memory and processing the image data, the programmable arithmetic processor
comprising a first set of local buffers and a second set of local buffers, each
buffer in the first set of local buffers alternately used for fetching input image
data and each buffer in the second set of local buffers alternately used for storing
output image data; and
a programmable input addresser controlling transfer of the image data from the
memory to the programmable arithmetic processor by:
(i) providing a memory address onto a read address bus coupled between the programmable
input addresser and the memory, the memory address identifying a location of the
image data within the memory, and
(ii) providing a storage address to the programmable arithmetic processor, the
storage address identifying a local buffer within the programmable arithmetic processor
for storage of the image data.
13. An image transform processor for processing image data, comprising:
a programmable arithmetic processor capable of receiving the image data from
a memory over a data bus coupled between the programmable arithmetic processor
and the memory and processing the image data, the programmable arithmetic processor
comprising a first set of local buffers and a second set of local buffers, each
buffer in the first set of local buffers alternately used for fetching input image
data and each buffer in the second set of local buffers alternately used for storing
output image data;
a programmable input addresser controlling transfer of the image data from the
memory to the programmable arithmetic processor by:
(i) providing a memory address onto a read address bus coupled between the programmable
input addresser and the memory, the memory address identifying a location of the
image data within the memory, and
(ii) providing a storage address to the programmable arithmetic processor, the
storage address identifying a first local buffer within the programmable arithmetic
processor for storage of the image data; and
a programmable output addresser controlling transfer of the image data from the
programmable arithmetic processor to the memory by:
(i) providing a write address onto a write address bus coupled between the programmable
output addresser and the memory, the write address identifying a write address
in the memory for storage of the image data, and
(ii) providing a retrieval address to the programmable arithmetic processor,
the retrieval address identifying a second local buffer within the programmable
arithmetic processor for retrieval of the image data.
14. An image transform processor comprising:
a programmable input addresser to retrieve an image as a received image in accordance
with a first programmed predefined access pattern, the programmable input addresser
to output the received image in accordance with a second programmed predefined
access pattern;
a programmable output addresser;
a SIMD processor including a controller coupled to a memory storing an at least
one image processing instructions, the SIMD processor having a plurality of processing
elements and a plurality of local buffers arranged in a plurality of levels and
a plurality of processing banks, each processing bank in the plurality of processing
banks being connected in parallel with another processing banks in the plurality
of processing banks, the controller being coupled to each processing element in
the plurality of processing elements and each local buffer in the plurality of
local buffers to control the operation of each processing element and each local
buffer such that the plurality of processing banks simultaneously respond to an
instruction from the controller, the SIMD processor being arranged as:
(i) a first level of the plurality of levels including a first set of local buffers
from the plurality of local buffers;
(ii) a second level of the plurality of levels including a second set of local
buffers from the plurality of local buffers;
(iii) a third level of the plurality of levels including a third set of local
buffers from the plurality of local buffers;
(iv) a fourth level of the plurality of levels including a fourth set of local
buffers from the plurality of local buffers;
(v) a processing level including a set of processing elements from the plurality
of processing elements that generates a processed image from the image stored in
the plurality of local buffers in accordance with an image processing instruction;
each processing bank including one local buffer of the first set of local buffers,
one local buffer of the second set of local buffers, a processing element from
the set of processing elements, one local buffer of the third set of local buffers,
and one local buffer of the fourth set of local buffers, the processing element
of each processing bank storing and retrieving the image in response to the image
processing instruction;
where the processing element associated with each processing bank directly read
from and store to the local buffers of an adjacent processing bank, if any;
each processing bank receiving an image from the programmable input addresser
via the first set of local buffers, each processing bank also receiving the image
from an input block addresser via the second set of local buffers, each processing
bank outputting the processed image to the programmable output addresser via the
third set of local buffers, each bank also sending the processed image to the programmable
output addresser via the fourth set of local buffers;
where the image processing instruction include an instruction that selectively
designate one of the group consisting of the first level and the second level to
receive the image from the programmable input block addresser as a selected input
level, and a non-selected input level, such that simultaneously the selected input
level receives the image while the processing element processes the image from
the non-selected input level;
where the image processing instruction include an instruction that selectively
designate one of the group consisting of the third level and the fourth level to
output the processed image to an output block addresser as a selected output level,
and a non-selected output level, such that simultaneously the selected output level
sends the processed image data while the processing element processes the image
from the non-selected output level; and
the programmable output addresser to receive the processed image from the selected
output level, the output block addresser to output the processed image in accordance
with a programmed predefined output pattern.
15. The image transform processor of claim 14 further comprising:
a Huffman decoder that decodes a encoded image into the image prior to sending
the image to the SIMD processor; and
a Huffman encoder that encodes the image from the SIMD processor.
16. The image transform processor of claim 15 further comprising:
one or more Huffman control registers that causes the Huffman decoder to receive
the encoded image, decode the encoded image to produce the image, and to provide
the image to the SIMD processor, and that causes the Huffman encoder to receive
the processed image from the SIMD processor, encode the processed image and output
the encoded processed image.
17. The image transform processor of claim 14 wherein the SIMD processor further comprises:
a boolean accumulator that has a boolean flag, the controller having a conditional
write instruction that overwrites a value in the local buffers based on a state
of the boolean flag.
18. The image transform processor of claim 14, wherein each processing element
includes a boolean accumulator that has a boolean flag, the controller causing
the processing element to store a result of a comparison operation in the respective
boolean flag.
19. The image transform processor of claim 14 wherein the SIMD processor further comprises:
a base pointer register that stores a base address (BA);
a horizontal counter configuration register that stores a horizontal count (Hcount);
a vertical counter configuration register that stores a vertical count (V count);
and
a row configuration register that stores the length a row (Hdim), where the controller
is responsive to an instruction that specifies a horizontal offset (Hoft) and a
vertical offset (Voft), the controller generating an effective two-dimensional
address (EA) to the input buffers and the output buffers in accordance with the
following relationship:
20. The image transform processor of claim 14 further comprising:
an auxiliary bank including:
one local buffer of the first set of local buffers;
one local buffer of the second set of local buffers; and
one local buffer of the third set of local buffers, where the auxiliary bank
is adjacent an end processing bank of the processing banks, the end processing
bank directly reads data from and stores data to the local buffers of the auxiliary
bank.
21. An image transform processor comprising:
a programmable input block addresser to retrieve image data in accordance with
a first programmed predefined access pattern as retrieved image data;
a SIMD processor including a controller coupled to a memory storing a image processing
instruction, the SIMD processor having processing elements and local buffers arranged
in levels and processing banks, each processing bank being connected in parallel
with other processing banks, the controller being coupled to each processing element
and local buffer to control the operation of each processing element and local
buffer such that the processing banks simultaneously respond to the same instruction
from the controller, the SIMD processor being arranged as:
(i) a first level including a first set of local buffers;
(ii) a second level including a second set of local buffers;
(iii) a processing level including a set of processing elements that generates
processed image data from image data stored in the local buffers in accordance
with the image processing instructions;
(iv) a third level of a third set of local buffers;
(v) a fourth level of a fourth set of local buffers;
a programmable input buffer controller to store the retrieved image data in a
specified level of the local buffers in accordance with a second programmed predefined
access pattern;
where in the SIMD processor:
each bank including one local buffer of the first set of local buffers, one local
buffer of the second set of local buffers, one processing element from a set of
processing elements, one local buffer of the third set of local buffers, and one
local buffer of the fourth set of local buffers, the processing element of each
bank storing and retrieving image data from the local buffers in response to the
instructions;
where the processing elements of the processing banks directly read data from
and store data to the local buffers of an adjacent processing bank, if any;
each bank receiving image data from the input block addresser via the first set
of local buffers, each bank also receiving image data from the input block addresser
via the second set of local buffers, each bank outputting the processed image data
to the output block addresser via the third set of local buffers, each bank also
sending the processed image data to the output block addresser via the fourth set
of local buffers;
where the image processing instructions include instructions that selectively
designate one of the group consisting of the first level and the second level to
receive image data from the input block addresser as a selected input level, the
other level being a non-selected input level, such that simultaneously the selected
input level receives image data while the processing element processes image data
from the non-selected input level; and
where the image processing instructions include instructions that selectively
designate one of the group consisting of the third level and the fourth level to
output the processed image data to the output block addresser as a selected output
level, the other level being a non-selected output level, such that simultaneously
the selected output level sends the processed image data while the processing element
processes image data from the non-selected output level; and
a programmable output buffer controller to cause the local buffers of the selected
output level to output the processed image in accordance with a third programmed
predefined access pattern as output image data; and
a programmable output block addresser to generate addresses to output the output
image data in accordance with a fourth predefined access pattern.
22. An image transform processor for processing image data, comprising:
a means capable of receipt of the image data from a data source over a data path
and processing the image data; and
a means that controls transfer of the image data from the data source to a programmable
arithmetic processing means by providing a source address onto a source address
path, the source address identifying the data source, the programmable arithmetic
processing means comprising a first set of local buffers and a second set of local
buffers, each buffer in the first set of local buffers alternately used for fetching
input image data and each buffer in the second set of local buffers alternately
used for storing output image data.
23. The image transform processor of claim 22 further comprising:
a means for controlling transfer of the image data from the programmable arithmetic
processor to a memory by providing a write address onto a write path, the write
address identifying a write address in the memory for storage of the image data.
24. An image transform processor for processing image data, the image transform
processor comprising:
a means capable of receiving the image data from a memory over a data bus coupled
between a programmable arithmetic processing means and the memory and processing
the image data; and
a means for controlling transfer of the image data from the memory to the programmable
arithmetic processing means by:
(i) means for providing a memory address onto a read address bus coupled between
a programmable input addressing means and the memory, the memory address identifying
a location of the image data within the memory, and
(ii) means for providing a storage address to the programmable arithmetic processing
means, the storage address identifying a local buffer within the programmable arithmetic
processing means for storage of the image data, the programmable arithmetic processing
means comprising a first set of local buffers and a second set of local buffer,
each buffer in the first set of local buffers alternately used for fetching input
image data and each buffer in the second set of local buffers alternately used
for storing output image data.
Description
BACKGROUND OF THE INVENTION
1. Technical Field
The invention relates generally to digital image processing, and particularly
to a programmable image transform processor for digital image processing.
2. Related Art
In photographic cameras, the image-forming light is sensed and recorded directly
on film. Unlike photographic cameras, the electronic still camera uses an electronic
image sensor to sense the image-forming light and a separate recording medium to
record and store the picture. Because the electronic still camera uses digital
technology, the electronic still camera is a type of digital camera.
Typically the electronic image sensor in a digital camera is a solid-state
device such as a charge-coupled device (CCD), charge injected device (CID) or a
complimentary metal oxide semiconductor (CMOS) device. The image sensor connects
to electronic interface circuitry which connects to a storage device and, optionally,
to a display. A typical image sensor has many cells or pixels arranged along vertical
and horizontal dimensions in a matrix. In response to light, the cells generate
a charge or voltage which represents image information. The image sensor senses
an image and stores image information, i.e., a charge or voltage, corresponding
to the sensed light in the cells. Image sensors are made in many sizes such as,
e.g., 400×300, 640×480, 1024×768 and 4096×4096 pixels. The
image information stored in the cells is output serially from the image sensor
using an arrangement of shift registers. The shift registers are arranged along
vertical and horizontal dimensions and are coupled to the cells. The cells and
shift registers require timing, or clock signals, having specific timing requirements,
to output the image information. Each type of image sensor has its own unique timing
requirements. Typically, a single image sensor requires many clock signals to control
the flow of image information in both the horizontal and vertical dimensions. The
clock signals must be synchronized. For example, to output image information from
a 640×480 CCD requires 480 vertical shifts and 640 horizontal shifts for each
vertical shift. Within a single dimension, the clock signals to control the flow
of image information have different phases that must be synchronized. Furthermore,
shifting the information out of the image sensor requires timing signals to synchronize
the image sensor's operation with an analog signal processor (ASP) and an analog-to-digital
(A/D) converter.
The image information sensed by each cell is also called a pixel. For example,
a 640×480 CCD has about 307,200 pixels. After being converted to digital form,
the image information (image data) is stored in a memory, typically an image memory.
Image sensors having a larger numbers of cells produce higher quality images; however,
the more pixel information that is available relates to the amount of processing
and memory resources required to process the pixel information.
Typically, a digital signal processor processes the image data to improve
the quality of the image. Various algorithms well-known in the art are used to
improve the image quality of the image data. Because there is such a large amount
of image data, the image data may be compressed before storage in a storage medium
or memory.
Color imaging increases the complexity of processing the image data. In one
method, the image sensor has a geometric arrangement of cells to respond to three
colors, e.g., red, green and blue. Since each cell senses a particular color, various
algorithms are used to interpolate the missing color information. Alternatively,
two or more image sensors having different color sensitivity may be utilized and
the image information combined.
In digital cameras, processing the data takes time. Analog image data from the
image sensor is processed via the analog signal processor, converted into image
data by the analog-to-digital converter and stored in memory. Furthermore, a digital
signal processor processes the raw image data to improve the quality of the image.
For color images that utilize a single image sensor, "missing" pixel data values
must be interpolated and require even more processing time. Still images are further
processed to compensate and correct for other errors introduced by the optical
system and the image sensor. The compression of the image data adds even more time.
The time required to acquire, process and compress the image data causes an unacceptable
delay when acquiring consecutive images. The delay can take several seconds. This
delay is a problem for photographers who need a continuous shooting capability
to photograph a sequence of images in quick succession. Therefore a process and
apparatus are needed to reduce the delay between consecutive pictures.
Typically, a digital camera has hardware that implements a single digital
image processing procedure or algorithm. If the procedure is changed, the hardware
must be redesigned, which is time consuming and expensive. Therefore, there is
a need in the art for a digital image processing procedure or device that is easily
and quickly modified and that supports numerous digital signal processing procedures
using the same hardware. The digital image processing procedure or device should
also minimize the processing time to allow consecutive pictures to be taken in
quick succession.
In addition, depending on the environmental factors, such as lighting, the image
processing algorithm should be selected or modified to produce the desired image
quality. Furthermore, there is a need to dynamically modify the image processing
algorithm during the image acquisition process.
As the size of the image sensors increases, the amount of image information to
be processed increases. In addition, as image processing algorithms become increasingly
sophisticated, complex processing of the image data consumes more time. Therefore,
there is a need to reduce the image processing time.
SUMMARY
The programmable image transform system may be broadly conceptualized as a device
that separates address generation from arithmetic manipulation, thus improving
the overall efficiency of the device while reducing the time needed to perform
image processing. For example, an image transform processor that processes digital
images may utilize an architecture that includes a programmable arithmetic processor
and a programmable input addresser. The programmable arithmetic processor may be
capable of receiving digital image data from a memory, such as a read only memory
(ROM), electronic erasable programmable read only memory (EEPROM), flash memory
or non-volatile memory, over a data bus for processing. The programmable input
addresser controls the transfer of image data from the memory to a programmable
arithmetic processor. The programmable input addresser provides: (i) a memory address
to a read address bus coupled between the programmable addresser and the memory,
and (ii) a storage address to the programmable arithmetic processor. The memory
address identifies a location of the digital image data within the memory. The
storage address identifies a local buffer within the programmable arithmetic processor
for storage of the digital image data.
The invention also relates to retrieval and storage of image data into a memory
while other image data is being processed. The retrieved image data is placed in
a set of local buffers. To increase the speed of image processing, a single-instruction
multiple-data (SIMD) processor processes the image data in the set of local buffers
and outputs the processed image data to another set of local buffers. For example,
in an image transform processor having buffers, a first portion of input image
data is provided in a first one of the buffers. A first processing operation is
performed on the first portion of the input image data to define a first processed
image data. The first processed image data is stored in a second buffer. A second
processing operation is performed on the first processed image data to define a
second processed image data. While the second processing operation is performed
on the first processed image data, a second portion of the input image data is
provided in the first buffer.
The invention also provides for using the image transform processor for processing
video or other real-time data streams. The image transform processor has four buffer
that are used for storing the video or real-time data. First and second levels
of buffers are alternately used for fetching input data, while third and fourth
levels of buffers are alternately used for storing output data. Thus, image data
can be input, processed and output in every clock cycle.
Other systems, methods, features and advantages of the invention will be or
will become apparent to one with skill in the art upon examination of the following
figures and detailed description. It is intended that all such additional systems,
methods, features and advantages be included within this description, be within
the scope of the invention, and be protected by the accompanying claims.
BRIEF DESCRIPTION OF THE FIGURES
The components in the figures are not necessarily to scale, emphasis instead
being placed upon clearly illustrating the principles of the invention. Moreover,
in the figures, like reference numerals designate corresponding parts throughout
the different views.
FIG. 1 is a block diagram of an electronic digital camera embodying an exemplary
image transform processor.
FIG. 2 is a block diagram of the digital camera of FIG. 1.
FIG. 3 is a diagram of an exemplary image sensor suitable for use with the image
transform processor.
FIG. 4 is a block diagram of a preferred embodiment of the image transform processor
of FIG. 2.
FIG. 5 is a block diagram of a programmable block addresser of the image transform
processor of FIG. 4.
FIG. 6 is a block diagram of the topology of the arithmetic processing block
of FIG. 4.
FIGS. 7A and 7B are exemplary timing diagrams showing the overlapping of data
retrieval, data processing, and data storage operations in the arithmetic processing
block of FIGS. 4 and 6.
FIG. 8 is a diagram of an exemplary two-dimensional array of working blocks.
FIG. 9 is a diagram of exemplary image data showing the pixel blocks of an exemplary
working block.
FIG. 10 is an example of a working block that includes adjacent pixel blocks
in the image data.
FIG. 11 is a diagram showing overlapping working blocks in the image data
FIGS. 12A and 12B are examples of working blocks that include dispersed pixel
blocks in the image data.
FIG. 13 is a block diagram of the buffer owner register and next owner register
of the arithmetic processing block of FIG. 6.
FIG. 14 is a block diagram of an input buffer controller of FIG. 4.
FIG. 15 is a block diagram of a SIMD processor pipeline.
FIG. 16 is a block diagram of a SIMD processor of the arithmetic processing
block of FIGS. 4 and 6.
FIG. 17 is a block diagram of pointer configurations used by an instruction word.
FIG. 18 is a diagram of a circuit that generates an effective address for an instruction.
FIG. 19 is a block diagram of a multiplexor/latch stage of the SIMD processor
pipeline of FIG. 15.
FIG. 20 is a block diagram of an arithmetic stage of the SIMD processor pipeline
of FIG. 15.
FIG. 21 is a block diagram of a descale/write stage of the SIMD processor of
FIG. 15.
FIG. 22 is a block diagram of an accumulator descaler of the arithmetic stage
of the processing element of FIG. 21.
FIG. 23 is a block diagram of an arithmetic logic unit descaler of the arithmetic
stage of the processing element of FIG. 20.
FIG. 24 is a block diagram showing the expandable topology of the arithmetic
processing block of FIG. 6.
FIG. 25 is a block diagram of an arithmetic processing block of FIG. 6 having
multiple master controllers.
FIG. 26 is a flow diagram of an exemplary image transform process of the image
transform processor of FIG. 4.
DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
In FIG. 1, a block diagram of a digital camera
100 embodying the image
transform processor is shown. A lens
102 transmits the image-forming light
104 onto an electronic image sensor (image sensor)
106. The image
sensor
106 is in the digital camera and located at the focal plane of the
lens. The image sensor is typically a charge-coupled device (CCD) or a complementary
metal-oxide-semiconductor (CMOS) sensor. Image sensors differ in the arrangement
of the cells within the image sensor and the type of charge readout. The image
sensor
106 connects to electronic interface circuitry
108. The electronic
interface circuitry
108 also connects to a storage device
110 and
an optional display
112. The electronic interface circuitry
108 controls
the storage device
110 and stores the image sensed by the image sensor
106.
The storage device
110 can include a tape drive, a disk drive, such as a
floppy disk drive, hard disk drive, optical disk drive or magneto-optical disk
drive, or an integrated circuit card with RAM, DRAM, or EEPROM, or non-volatile
memory. The storage device
110 may be inside the digital camera
100
or attached to the digital camera externally. The electronic interface circuitry
108 may also control the display
112 that displays the image sensed
by the image sensor
106. The display
112 can be inside the digital
camera or attached to the camera externally. The electronic interface circuitry
can operate the display
112 in either a viewfinder mode or a review (i.e.,
stored image viewing mode).
In FIG. 2, a block diagram of the electronic interface circuitry of the digital
camera of FIG. 1 is shown. A microprocessor (RISC)
202 is coupled to a memory
controller
203a, a programmable timing generator
204, a frame
capture processor
205, a programmable image transform processor
206,
a storage medium
208 and a programmable display controller
209. The
memory controller
203a is connected to a memory
203. The programmable
display controller
209 is coupled to a display
210. The image sensor
106 is coupled to an analog signal processor (ASP)
211 which connects
to the analog to digital converter (A/D converter)
212. The programmable
timing generator
204 is coupled to the image sensor
106, ASP
211,
the A/D converter
212, the frame capture processor
205, and the microprocessor(RISC)
202. The programmable image transform processor
206 and other elements
read data from and write data to the memory
203 via the memory controller
203a. Preferably, the memory
203 is a high-speed DRAM used
to store the digital image data. The A/D converter
212 supplies digital
image data to the programmable image transform processor
206 that stores
the data in the memory
203. The timing generator
204 supplies timing
signals to the programmable image transform processor
206 and A/D converter
212 to synchronize the transfer of digital image data between the A/D converter
212 and the frame capture processor
205. The frame capture processor
205 supplies the digital image data to the programmable image transform
processor
206. Alternately, the frame capture processor
205 stores
the image data from the sensor directly to the memory
203, and the programmable
image transform processor
206 fetches that data from the memory
203
for further processing. The frame capture processor
205 supports real-time
windowing, histogram, gamma, white balance and auto-focus functions.
The microprocessor(RISC)
202 executes a camera operation procedure that
is stored in memory
203. Alternatively the camera operation procedure can
be stored in a read-only-memory(ROM), or loaded into the memory
203 from
the storage medium
208. Further, in alternate embodiments, the RISC microprocessor
may be substituted a different type of controller, such as a typical microprocessor,
digital signal processor, application specific integrated circuit (ASIC), phase
array logic (PAL), discrete circuits functioning as a controller. The camera operation
procedure comprises an image acquisition procedure. When a user presses a store-image
button (not shown), the camera operation procedure causes the image sensor
106
to acquire an image. The image acquisition procedure causes the microprocessor
(RISC)
202 to control the timing generator
204 to generate vertical
and horizontal clock signals for use by the image sensor
106. The image
sensor
106 outputs image as a series of analog signals corresponding to
the color and intensity of the image sensed by each cell. The sensed image information
is then sent to the ASP
211 and to the A/D converter
212.
The ASP
211 processes the sensed image information before input to the
A/D converter
212. For example, the ASP has a programmable amplifier with
adjustable gain, and also reduces or eliminates noise, such as reset noise, from
the sensed image information using methods well known to those in the art, such
as correlation-double-sampling.
The A/D converter
212 then converts the analog sensed image information
into image data. In an alternative embodiment, the ASP
211 is absent and
no pre-processing of the sensed image data occurs.
The image data is stored in memory
203. Execution of the camera operation
procedure by the microprocessor (RISC)
202 causes the image data to be processed
by the programmable image transform processor
206. The processed image data
is compressed and recorded in memory
203, on a storage medium
208
or transferred to a programmable display controller
209 for output to a
display
210.
In FIG. 3, a block diagram of an exemplary image sensor
302 is shown.
The
image sensor
302 can be a CCD or CMOS device. The image sensor
302
connects to the analog signal processor(ASP)
304 and the A/D converter
306.
The image sensor
302 has cells
308, vertical shift registers
312
and a horizontal shift register
314. Each cell
308 absorbs light
and converts the light energy into an electrical charge. The amount of charge is
a measure of the amount of light energy or radiation absorbed by the image sensor
302. The size of the image sensor
302 determines the quality of the
image. The quality of the image improves as the number of cells
308 increases.
Image sensors are available in many sizes including 400×300, 640×480,
1024×768, and 4096×4096 cells.
The components of the image sensor
302 are arranged along horizontal and
vertical dimensions. An array
310 of cells
308 is arranged in the
vertical dimension. The vertical shift register
312 has register location
316 for storing the charge sensed by the cells
308. Each cell
308
in the array of cells
310 connects to a corresponding register location
316 in the vertical shift register
312.
Free charges move from regions of higher potential to regions of lower potential.
By alternating the voltage on the electrodes (not shown) connected to the cells
308 and the register locations
316 and
318 of the shift registers
312 and
314 in proper phase, a charge packet, i.e., the charge from
the cell
308, can be moved from the cell
308 to a register location
316 in the shift register
312. The charge packet is then moved from
one register location to another register location in the shift registers
312
and
318 until finally output by the image sensor
302.
When appropriate voltages are applied to the cell
308 and the corresponding
register location
316 in the vertical shift register
312, the charge
generated in the cell
308 is transferred out of the cell
308 to the
corresponding register location
316 in the vertical shift register
312.
The programmable timing generator is programmed to output timing or clock signals
to cause the transfer of the charge to occur at synchronized times. When appropriate
voltages are applied to adjacent elements of the vertical shift register
312,
the charge is transferred from to the next registration location. The last element
or output of each vertical shift register
312 connects to a corresponding
register location
318 in the horizontal shift register
314. When
appropriate voltages are applied to the last register location of the vertical
shift register
312 and the corresponding register location
318 of
the horizontal shift register
314, the charge is transferred from the vertical
shift register
312 to the horizontal shift register
314. When appropriate
voltages are applied to adjacent register location of the horizontal shift register
314, the charge is transferred from one register location to another register
location until finally outputted. The output of the horizontal shift register
314
connects to the ASP
304 via an output amplifier
320.
Color imaging is more complex. In one method, the image sensor
302 has
a geometric arrangement of cells to respond to three colors, e.g., red, green and
blue. Alternatively, two or more image sensors having different color sensitivity
are used. The programmable image transform processor of the present invention works
with both methods of color imaging. The programmable image transform processor
performs image transform operations on input data after it has been digitized by
the A/D converter
306.
In FIG. 4, a block diagram of an embodiment of the programmable image transform
processor (ITP)
206 of FIG. 2 is shown. Image transformation and compression
operations, such as discrete wavelet transforms (DWT) and discrete cosine transforms
(DCT) perform two main types of computation: address calculation and arithmetic
computation. Devices such as digital cameras store images, at least temporarily,
in solid-state memory such as a DRAM. The memory is organized into pages of image
data. To acquire image data from the memory, an address is generated. After generating
the address and acquiring the desired image data, the image data is further manipulated.
The ITP
206 separates the address calculation from the arithmetic computation
using parallel hardware. The ITP collects input image data and output image data
in bursts when accessing the same memory page.
The ITP
206 has inputs and outputs for connecting to a read address bus,
a read data bus, a write address bus, a write data bus and control signals. The
ITP
206 connects to the memory, the A/D converter, the timing generator
and the microprocessor (RISC). A DMA controller may be used to access the high
speed image memory. The ITP
206 may a be dynamically configurable to provide
many pipelined data processing paths. In an addressing block
410, a data
path mode register
412 controls an input data multiplexor
414 and
an output data multiplexor
416 to control the flow of image data to and
from a programmable arithmetic processing block
420. The programmable arithmetic
processing block
420 receives the image data, processes the image data and
outputs the processed image data. The microprocessor (RISC) of the digital camera
loads the data path mode register
412 with a specified data flow path information.
In response to the data flow path information being loaded in the data path mode
register
412, the input data multiplexor
414 supplies data from the
microprocessor (RISC), from a frame capture processor, a Huffman decoder
422,
and directly from the DRAM. The frame capture processor provides an analysis of
the image data as it is received from the programmable timing generator. The Huffman
decoder
422 decodes compressed image data that was stored using the Joint
Photographics Experts Group (JPEG) compression format in the external memory.
In response to the data in the data path mode register
412, the output
data multiplexor
416 outputs data from the microprocessor (RISC), processed
image data from the programmable arithmetic processing block
420, or encoded
processed image data from a Huffman encoder
424. The Huffman encoder
424
compresses data from the programmable arithmetic processing block
420 using
a JPEG compression format.
Table one, below, summarizes the data flow for various data path configuration
settings of the data path mode bits of the data path mode register
412.
In table one, the term "ITPBUF" refers to the programmable arithmetic processing
block
420, and in particular to local buffers in the programmable arithmetic
processing block
420.
| TABLE 1 |
| |
| Data Path Configuration Settings |
| |
|
|
Buffer Owners |
| |
|
|
PE = 0, IBC = 1, |
| |
Data Path |
|
OBC = 2, RISC = 3 |
| Mode |
Mode Bits |
Data Flow |
L0 |
L1 |
L2 |
L3 |
| |
| Video |
000100 |
FCP to ITPBUF to |
1/0 |
1/0 |
0/2 |
0/2 |
| |
|
DRAM |
| Frame |
000000 |
DRAM to ITPBUF to |
1/0 |
1/0 |
0/2 |
0/2 |
| Blend |
|
DRAM |
| Process |
000000 |
DRAM to ITPBUF to |
1/0 |
0 |
0 |
0/2 |
| |
|
DRAM |
| Process/En |
000001 |
DRAM to ITPBUF to |
1/0 |
0 |
0 |
0/2 |
| code |
|
HUFF to DRAM |
| Decode/ |
000010 |
DRAM to ITPBUF to |
1/0 |
0 |
0 |
0/2 |
| Process |
|
HUFF to DRAM |
| RISC/ |
111000 |
RISC to ITPBUF to |
3 |
3 |
3 |
3 |
| RISC |
|
RISC |
| RISC |
010000 |
DRAM to ITPBUF to |
3/0 |
0 |
0 |
0/3 |
| replace |
|
RISC to DRAM |
| IBA |
| RISC |
001000 |
DRAM to RISC to |
3/0 |
0 |
0 |
0/3 |
| Replace |
|
ITPBUF to DRAM |
| OBA |
| RISC |
011000 |
DRAM to RISC to |
1/0 |
0 |
0 |
0/3 |
| replace |
|
ITPBUF to RISC to |
| IBA, OBA |
|
DRAM |
| IBA help |
101000 |
DRAM to ITPBUF to |
3 |
3 |
3 |
3/2 |
| RISC |
|
DRAM |
| OBA help |
110000 |
RISC to ITPBUF to |
3 |
3 |
3 |
3/2 |
| RISC |
|
DRAM |
| IBA & |
100000 |
DRAM to ITPBUF to |
1/3 |
3 |
3 |
3/2 |
| OBA help |
|
RISC to ITPBUF to |
| RISC |
|
DRAM |
| |
The programmable addressing block
410 generates addresses and coordinates
handshaking signals to retrieve image data from and to store data to the external
memory. Image data does not flow through the programmable addressing block
410
but flows to the local buffers of the programmable arithmetic processing block
420. The programmable addressing block
410 supplies control signals
to coordinate the transfer of image data with the programmable arithmetic processing
block
420.
The programmable addressing block
410 has an input addresser
430
and an output addresser
440. In the input addresser
430, an input
block addresser (IBA)
442 provides addresses to a read address bus to request
data from an external memory, such as a DRAM, using handshaking control signals,
such as read address available (R_Address Avail) and read address acknowledge (R_Address
Ack). An input buffer controller (IBC)
444 supplies addressing information
to the local buffers of the programmable arithmetic processing block
420
to store the requested image data from the external memory on a read data bus in
buffers in the programmable arithmetic processing block
420 using handshaking
signals. The handshaking signals are read data available signal (R_Data avail)
and read data acknowledge signal (R_Data ack).
In the output addresser
440, an output block addresser (OBA)
446
provides addresses to a write address bus to store data in the external memory
using handshaking control signals such as write address available (W_Address Avail)
and write address acknowledge (W_Address Ack). An output buffer controller (OBC)
448 supplies addressing information to the local buffers of the programmable
arithmetic processing block
420 to transfer the image data from the local
buffers of the programmable arithmetic processing block
420 to the external
memory. The output buffer controller
448 uses handshaking signals to retrieve
the processed image data from the programmable arithmetic processing block
420.
The OBC
448 uses handshaking signals, such as write data available signal
(W_Data avail) and write data acknowledge signal (W_Data ack), to coordinate the
transfer of data from the local buffers of the programmable arithmetic processing
block
420 to the external memory.
The programmable arithmetic processing block
420 receives the image data,
processes the image data and outputs the processed image data. A SIMD master controller
450 controls the operation of the programmable arithmetic processing block
420. Both the programmable arithmetic processing block
420 and the
SIMD master controller
450 communicate with the camera's microprocessor
(RISC)
202 (FIG.
2).
The Addressers
The input block addresser (IBA)
442 and output block addresser (OBA)
446
supply addresses to each address bus. The IBA
442 provides addresses of
requested data to supply to the read data bus, i.e., data to be operated on by
the programmable arithmetic processing block
420. In particular, the IBA
442 generates DRAM memory word addresses for two-dimensional blocks or lines
of image data. The OBA
448 provides addresses of processed data to write
to the write data bus, such as image data to be stored in the image memory.
The Input Block Addresser
Referring to FIG. 5, the input block addresser
442 is configurable
(i.e. programmable). The input block addresser
442 has a microsequencer
460, a control store or instruction memory
462, and pointer registers
A, B, C and D
464. The input block addresser
442 has four loop counters
466, four general purpose registers
468 and four pointer registers
464. The pointer registers A, B, C and D
464 generate the input address
which is output to the read address bus by the multiplexor
476. The input
block addresser
442 also has a base page register
470 and a stack
472 that is part of the control store
462 and a stack pointer
474.
The control store
432 is typically implemented using a static RAM array.
The microsequencer
460 is coupled to the control store
462 and
the pointer registers
464 and generates the input data addresses to access
the memory, such as a DRAM, storing the image data. The microsequencer
460
stores the addresses in the pointer registers
464. The addresses in the
pointer registers
464 are utilized to access the DRAM memory. Data requested
by the input block addresser
442 is stored in a buffer in the programmable
arithmetic processing block
420 (FIG.
4). A multiplexor
476
selects the address in one of the pointer registers
464 to output to the
read address bus based on commands executed by the microsequencer
460.
The control store
462 stores an input block address procedure
478
to be executed by the input addresser microsequencer
460. The input block
address procedure
478 has a sequence of address generation instructions.
The input block addresser
442 has a data request command to initiate read
operations to the image memory and to supply an absolute address to the read address
bus. The microsequencer
460 c
