Method and apparatus for correcting crosstalk and spatial resolution for multichannel imaging Number:7,079,708 from the United States Patent and Trademark Office (PTO) owispatent A multichannel imaging system generates an ensemble of images for each field of view of an object. Each image in the ensemble is intended to contain information from only one source among a plurality of sources for the object. However, due to crosstalk, at least a portion of the signal from a first source appears in a channel intended for a second source. Because the accuracy of the correction will be degraded if the images in an ensemble are spatially misaligned with respect to one another, the spatial offset between images is determined and a correction is applied to substantially eliminate the offset. Then, a correction to the signals is determined to substantially reduce the contributions to the signal in a channel from the signals in other channels. The signal processing can be employed to process the output signals for each of a plurality of different disclosed imaging systems.<H multichannel, resolution, Method, and, appa, 7079708.html, Patent, pto, 7079708

Title: Method and apparatus for correcting crosstalk and spatial resolution for multichannel imaging

Abstract: A multichannel imaging system generates an ensemble of images for each field of view of an object. Each image in the ensemble is intended to contain information from only one source among a plurality of sources for the object. However, due to crosstalk, at least a portion of the signal from a first source appears in a channel intended for a second source. Because the accuracy of the correction will be degraded if the images in an ensemble are spatially misaligned with respect to one another, the spatial offset between images is determined and a correction is applied to substantially eliminate the offset. Then, a correction to the signals is determined to substantially reduce the contributions to the signal in a channel from the signals in other channels. The signal processing can be employed to process the output signals for each of a plurality of different disclosed imaging systems.

Patent Number: 7,079,708 Issued on 07/18/2006 to Riley, et al.

Inventors: Riley; James K. (Redmond, WA); Frost; Keith L. (Seattle, WA)
Assignee: Amnis Corporation (Seattle, WA)

Appl. No.: 224200

Filed: September 12, 2005

Related U.S. Patent Documents


Application Number	Filing Date	Patent Number	Issue Date
10783530	Feb., 2004	7006710
10132059	Apr., 2002	6763149
60286713	Apr., 2001

Current U.S. Class: 382/294 ; 382/299

Current International Class: G06K 9/32 (20060101); G01N 21/00 (20060101)

Field of Search: 382/151,203,254,255,276,287,294,295,299 359/1,3,11,15,237,561 356/73,317,326

References Cited [Referenced By]

U.S. Patent Documents


3922069	November 1975	Kishikawa et al.
4635293	January 1987	Watanabe
4677680	June 1987	Harima et al.
4770992	September 1988	Van den Engh et al.
4786165	November 1988	Yamamoto et al.
5096807	March 1992	Leaback
5141609	August 1992	Sweedler et al.
5153916	October 1992	Inagaki et al.
5159397	October 1992	Kosaka et al.
5159398	October 1992	Maekawa et al.
5159642	October 1992	Kosaka
5247339	September 1993	Ogino
5247340	September 1993	Ogino
5272354	December 1993	Kosaka
5351311	September 1994	Rogers et al.
5422712	June 1995	Ogino
5444527	August 1995	Kosaka
5471294	November 1995	Ogino
5548395	August 1996	Kosaka
5596401	January 1997	Kusuzawa
5633503	May 1997	Kosaka
5644388	July 1997	Maekawa et al.
5674743	October 1997	Ulmer
5695934	December 1997	Brenner
5754291	May 1998	Kain
5760899	June 1998	Eismann
RE35868	August 1998	Kosaka
5831723	November 1998	Kubota et al.
5848123	December 1998	Strommer
5855753	January 1999	Trau et al.
5929986	July 1999	Slater et al.
5959953	September 1999	Alon
6007994	December 1999	Ward et al.
6014468	January 2000	McCarthy et al.
6066459	May 2000	Garini et al.
6116739	September 2000	Ishihara et al.
6156465	December 2000	Cao et al.
6210973	April 2001	Pettit
6211955	April 2001	Basiji et al.
6249341	June 2001	Basiji et al.
6256096	July 2001	Johnson
6330081	December 2001	Scholten
6330361	December 2001	Mitchell et al.
6381363	April 2002	Murching et al.
6522781	February 2003	Norikane et al.
6549664	April 2003	Daiber et al.
6763149	July 2004	Riley et al.
7006710	February 2006	Riley et al.
2001/0006416	July 2001	Johnson
2002/0126275	September 2002	Johnson

Foreign Patent Documents


WO 00/42412	Jul., 2000	WO

Other References

Kubota, Fumio et al. 1995. "Flow Cytometer and Imaging Device Used in Combination." Cytometry: 21:129-132. cited by other .
Kubota, F. 2003. "Analysis of red cell and platelet morphology using an imaging-combined flow cytometer." Clin. Lab. Haem.: 25:71-76. cited by other .
Ong, Sim Heng. 1985. Development of a System for Imaging and Classifying Biological Cells in a Flow Cytometer. Doctor of Philosophy Thesis. University of Sydney, School of Electrical Engineering. (Aug.). cited by other .
Ong, S.H. et al. 1987. "Development of an Image Flow Cytometer." Analytical and Quantitative Cytology and Histology. XIVth International Conference on Medical and Biological Engineering and the VIIth International Conference on Medical Physics, Finland. (Aug.): 375-382. cited by other .
Ong, S.H. and P.M. Nickolls. 1991. "Optical Design in a Flow System For Imaging Cells." Sciences in Medicine: 14:2:74-80. cited by other .
Ong, S.H. and P.M. Nickolls. 1994. "Analysis of MTF Degradation in the Imaging of Cells in a Flow System." International Journal of Imaging Systems & Technology: 5:243-250. cited by other .
Satoh, Kaneo et al. 2002. "Small Aggregates of Platelets Can Be Detected Sensitively by a Flow Cytometer Equipped With an Imaging Device: Mechanisms of Epinephrine-Induced Aggregation and Antiplatelet Effects of Beraprost." Cytometry: 48:194-201. cited by other .
Wang, Fu-sheng and Fumio Kubota. 2002. "A Novel Apoptosis Research Method With Imaging-Combined Flow Cytometer and HITC OR IR-125 Staining." Cytometry: 50:267-274. cited by other .
Wietzorrek, Joachim et al. 1999. "A New Multiparameter Flow Cytometer: Optical and Electrical Cell Analysis in Combination With Video Microscopy in Flow." Cytometry: 35:291-301. cited by other.

Primary Examiner: Alavi; Amir
Attorney, Agent or Firm: Anderson; Ronald M.

Parent Case Text

RELATED APPLICATION

This application is a U.S. divisional patent application based on prior application Ser. No. 10/783,530, filed Feb. 24, 2004 now U.S. Pat. No. 7,006,710, which itself is a U.S. divisional patent application based on prior application Ser. No. 10/132,059, filed Apr. 24, 2002 now U.S. Pat. No. 6,763,149, which in turn is a U.S. conventional patent application based on prior provisional application Ser. No. 60/286,713, filed Apr. 25, 2001, the benefits of the filing dates of which are hereby claimed under 35 U.S.C. .sctn. 119(e) and 120.

Claims

The invention in which am exclusive right is claimed is defined by the following:

1. A method for reducing crosstalk among a plurality of signals from a plurality of sources, each signal being assigned to a separate channel and primarily containing information corresponding to a different source among the plurality of sources, comprising the steps of: (a) applying spatial corrections to correct any misalignment of the signals between channels, such that corresponding signals from different sources in the plurality of channels are aligned; and (b) for each channel, substantially reducing erroneous contributions to the signal assigned to the channel from others of the plurality of signals.

2. The method of claim 1, wherein the step of applying spatial corrections to correct any misalignment of the signals between channels comprises the step of applying spatial corrections at a sub-pixel resolution.

3. An article of manufacture adapted for use with a computer, comprising: (a) a memory medium; and (b) a plurality of machine instructions, which are stored on the memory medium, said plurality of machine instructions when executed by a computer, causing the computer to: (i) correct a signal misalignment between a set of related signals to within sub-pixel resolution, wherein each one of the set of related signals primarily contains information corresponding to a different specific source; and (ii) substantially reduce crosstalk contributions to each of the signals from other of the signals in the set of related signals.

4. An article of manufacture adapted for use with a processor, comprising: (a) a memory medium; and (b) a plurality of machine instructions, which are stored on the memory medium, said plurality of machine instructions when executed by a processor, causing the processor to: (i) correct a signal misalignment between a set of related signals, wherein each one of the set of related signals primarily contains information corresponding to a different specific source; and (ii) substantially reduce crosstalk contributions to each of the signals from other of the signals in the set of related signals.

5. A system for reducing crosstalk among a plurality of related signals in a set, each one of the set of related signals primarily containing information corresponding to a specific different source from among a plurality of different sources, comprising: (a) a memory in which a plurality of machine instructions defining the parent application are stored; and (b) a processor that is coupled to the memory to access the machine instructions, said processor executing said machine instructions and thereby implementing a plurality of functions, including: (i) correcting a signal misalignment between the plurality of related signals, each one of the plurality of related signals in the set is substantially aligned with other of the plurality of related signals in the set; and (ii) for each one of the plurality of related signals in the set, reducing crosstalk contributions from other of the plurality of related signals.

Description

FIELD OF THE INVENTION

The present invention generally relates to a method and apparatus for improving the accuracy of quantitative images generated by multichannel imaging instruments, and more specifically, to correcting errors introduced by crosstalk between channels, with application to a broad range of imaging instruments and particularly, to flow imaging instruments using time-delay-integration image detectors.

BACKGROUND OF THE INVENTION

The parallel advancement of the technology of video microscopy and techniques for preparing and staining biological samples has enabled those working in areas such as fundamental biological science, diagnostic medicine, and drug discovery to gather an ever-increasing amount of information from a single biological specimen. In the fields of cell biology and clinical cytology, for example, specimens may be stained with absorption dyes to define cell morphology, and with fluorescent dyes that attach to molecules bound to specific proteins or nucleic acid chains. Microscopes equipped for exciting and imaging the fluorescent dyes and concurrently imaging cell structures are routinely used for studying complex processes that modify cells on the gross structural level and also at the molecular level. In recent years, computational analysis of images captured from multiparameter microscopes has shown promise for automating large investigative studies such as those conducted by drug discovery and development companies and for automating complex cellular diagnostic tests for clinical medicine. Optimal use of such technology can be attained only if the signals used for image generation are accurately scaled to information about the cells being studied. However, such information can be degraded during the capture process. Specifically, interference can be introduced into a channel dedicated to a first signal due to leakage of a signal intended for a second channel. This type of signal degradation is generally referred to as channel-to-channel crosstalk.

An advancement to computer-based multiparametric imaging is disclosed in commonly assigned U.S. Patents, both entitled IMAGING AND ANALYZING PARAMETERS OF SMALL MOVING OBJECTS SUCH AS CELLS, U.S. Pat. No. 6,249,341, issued Jun. 19, 2001 (filed Jan. 24, 2000), and U.S. Pat. No. 6,211,955, issued Apr. 3, 2001 (filed Mar. 29, 2000), the complete disclosure, specification, and drawings of both of which are hereby specifically incorporated herein by reference. The technology disclosed in these applications extends the methods of computer vision to the analysis of objects either flowing in a fluid stream or moving relative to the imaging instrument on a rigid substrate, such as a glass slide. Instruments based on the inventions of the patent applications cited above deliver improved sensitivity at high spatial resolution through the use of time-delay-integration (TDI) electronic image acquisition, a method wherein signal integration is accomplished by shifting charge packets through an imaging array in synchrony with the motion of the target object being imaged.

The TDI-based flow imaging technology, with its ability to substantially improve signal-to-noise ratio, is of exceptional utility for multiparametric imaging. Each of the channels of a TDI flow imaging instrument can be dedicated to a single light source in the target objects. One such light source, for example, is the fluorescent dye attached to a molecule selected for its specificity for binding to a target protein. Each of a plurality of channels can be dedicated to a particular different dye, and all of the dyes addressed by the instrument may be present in close proximity on a single target cell. Because the dyes may have emission spectra broader than the passbands of the channels that collect their signals, channel-to-channel crosstalk can prevent the accurate estimation of the intensity of the signal from each dye.

Accordingly, it would clearly be desirable to develop a method and apparatus that simultaneously offers speed and accuracy in eliminating such channel-to-channel crosstalk. Preferably such crosstalk reduction can be achieved in conjunction with the TDI-based flow imaging method and apparatus noted above, which are intended for real time collection and processing of images from objects moving in high concentration, at high speed, through the instrument. Accordingly, the crosstalk reduction of the present invention is preferably applicable in real time and in synchrony with the collection of images of the moving targets that include indicators attached to the targets.

SUMMARY OF THE INVENTION

The present invention is directed to enabling an accurate reconstruction of information about objects imaged by an instrument using multiple channels, each channel being generally optimized to receive signals of a type differentiated from other signal types by predefined characteristics. These predefined characteristics may include, but are not limited to wavelength, a modulation of a signal received from a source, a scatter angle, a Doppler shift, and a phase shift (e.g., with respect to a reference phase). The present invention applies to, but is not limited to, instruments for collecting information from electromagnetic waves in all portions of the spectrum, by acoustic waves, by particle flux, and by measurement of object characteristics such as conductivity, chemical reactivity, size, shape, and mass.

One example of an application of the present invention is its use in a multiple-wavelength optical imaging instrument. In such an instrument, each channel is made sensitive to electromagnetic radiation of wavelengths bounded by an upper and lower limit, defining different wavebands for each channel. Typically these limits are determined by the characteristics of one or more filters disposed in a path between a light source and a photodetector servicing a channel. The images in each channel are detected, producing signals that are processed by the present invention to correct errors in alignment between the channels and a reference and then, to correct for crosstalk between the channels.

Thus, the present invention is directed to a method and apparatus that not only corrects for crosstalk between channels, but also ensures that signal data in each channel is properly aligned with signal data in other channels, so that the benefit from the crosstalk correction is not degraded by signal misalignment.

In one preferred embodiment, a method is provided for correcting signal misalignment between individual channels in a multichannel imaging system, such that data in a first channel is substantially aligned with data in other channels. The method also includes the step of reducing erroneous contributions to signal data from a source intended to provide signal data for other channels.

Preferably, the signal data are used to produce an image for display. Accordingly, a preferred embodiment is directed to a method that includes the step of spatially aligning images input in an image ensemble from a plurality of channels, such that each image in the image ensemble is substantially aligned with other images in the image ensemble, and the step of applying spectral crosstalk corrections, to remove the channel-to-channel crosstalk from the image ensemble output.

In one embodiment, the step of spatially aligning images includes the step of utilizing two classes of information, including a first and second class of constants. The first class of constants includes horizontal and vertical spatial offsets, which are derived from an on-line calibration image. The second class of constants is accessed during the step of spatially aligning images, but is not modified. Preferably the second class of constants includes at least one of channel start columns for each image, and inverted source coefficients.

The horizontal and vertical spatial offsets are preferably generated based upon a comparison of each image in an image ensemble with a calibration image. The comparison with a calibration image can be performed when a system for generating the multichannel signal is initialized, and/or periodically during the use of a system for generating the multichannel signal.

The step of generating the horizontal and vertical spatial offsets can include the steps of detecting a boundary of an image, preparing a correlogram based on the boundary and a reference image, determining a peak of the correlogram, and repositioning the image to correspond to a pixel closest to the peak of the correlogram.

Preferably the horizontal and vertical spatial offsets are determined for each pixel of the image, and the detection of the boundary of an image involves the use of a two-dimensional gradient operator to suppress flat surfaces and to enhance object boundaries. In one embodiment, preparing the correlogram based on the boundary and the reference image involves preparing a correlogram in the spatial domain, while in other embodiments the correlogram is prepared in the frequency domain.

In an embodiment in which the correlogram is prepared in the spatial domain, a Fourier Transform is performed on boundary data for the image and the reference image. Those results are multiplied to generate a product, and an inverse Fourier Transform is performed on that product.

To prepare the correlogram based on the boundary and the reference image in the frequency domain, first a Fourier Transform is performed on the boundary data for the image and the reference image. Then a conjugation operation is applied to one of the results of the Fourier Transforms. Next, the result of the conjugation operation is multiplied with the boundary data for the image to generate a product, and an inverse Fourier Transform is performed on the product.

To minimize errors, groups of images in each data channel are preferably processed together, such that a cumulative correlogram is generated for each data channel.

Once the correlogram is complete, the peak of the correlogram defines the aligned position of the image, relative to the reference image employed. The peak of the correlogram is determined by employing a Taylor series expansion, eigenvalues and eigenvectors. The image is then manipulated to align, to the nearest pixel, with that peak. Then, the image is reconstructed by interpolating to a fraction of a pixel, to align within a fraction of a pixel, with the true peak of the correlogram. Preferably, the interpolation involves the step of applying a two-dimensional interpolation.

In one embodiment, the step of applying a two-dimensional interpolation includes the step of computing a new amplitude value for each pixel based on a weighted sum of a group of surrounding pixels. Preferably, the weighted sum is determined by a Taylor series expansion based on a group of nine pixels, eight pixels of which surround a common origin pixel. Coefficients are applied to each pixel value, and the sum of a matrix of the coefficients is equal to 1.0.

The step of reducing erroneous contributions to that channel's measurement by sources intended for other channels preferably involves solving a set of linear equations relating source values to measurement values, wherein each channel is represented by a linear equation. It is also preferred that the set of linear equations are solved for each pixel in each image in each channel. The set of linear equations relating source values to measurement values can be solved by applying singular value decomposition to a matrix form of the set of linear equations.

The signal data, and/or corresponding images, can be spatially aligned in real-time. After the spatial alignment is completed, spectral crosstalk corrections can also be applied in real-time, or after the signal data/images have been stored for a period of time. The signal data/images can also be spatially aligned after having been stored for a period of time.

Another aspect of the present invention relates to a method for correcting errors in a multichannel imaging system, wherein each channel is intended to contain signal information relating to an image of an object that has been produced by only one type of source. The method involves focusing light from the object along a collection path, and dispersing the light that is traveling along the collection path into a plurality of light beams, such that each light beam corresponds to a single source. Each of the light beams is then focused to produce a respective image corresponding to that light beam. A detector is provided, disposed to receive the respective images. The detector generates an output signal corresponding to each image. For each output signal, a signal alignment correction and a crosstalk correction are applied.

In addition to the aforementioned embodiments relating to the method, the present invention also relates to a system having elements that carry out functions generally consistent with the steps of the method described above. One such system relates to a multichannel imaging system for generating an ensemble of images from an object for each field of view, wherein each image in the ensemble contains information from substantially only one type of source. The system includes a collection lens disposed so that light traveling from the object passes through the collection lens and travels along a collection path, and a dispersing component disposed in the collection path so as to receive the light that has passed through the collection lens, dispersing the light into a plurality of separate light beams, each light beam being directed away from the dispersing component in a different predetermined direction. The system also includes an imaging lens disposed to receive the light beams from the dispersing component, thereby producing the ensemble of images. The ensemble comprises a plurality of images corresponding to each of the light beams, each image being projected by the imaging lens toward a different predetermined location A multichannel detector is disposed to receive the plurality of images produced by the imaging lens, and produces a plurality of output signals, such that a separate output signal is produced for each of the separate light beams. Finally, the system includes means for processing each output signal, wherein the means performs the functions of correcting output signal misalignment between individual channels, such that an image generated by an output signal in each channel is substantially aligned with a corresponding image in each other channel, reducing erroneous contributions to that channel's measurement by sources intended for other channels.

The system also preferably includes a display electrically coupled to the means, the display generating an image for each output signal as modified by the means. The means for processing preferably includes a memory in which a plurality of machine instructions defining a signal conditioning application are stored, and a processor that is coupled to the memory to access the machine instructions, and to the display. Execution of the machine instructions by the processor cause it to spatially align images that are displayed, based on the output signals from the multichannel detector, such that each image is substantially aligned with other images. The processor also applies the spectral crosstalk corrections, to remove the channel-to-channel crosstalk from the displayed images.

It is further contemplated that the means for processing the signal alternatively comprise a programmed computer, an application specific integrated circuit, or an oscilloscope.

Yet another embodiment of the system includes a plurality of different detectors, such that an image corresponding to a different source is directed to each detector, and the plurality of different detectors collectively comprise the multiple channels. The detectors employed in this embodiment of the system are preferably pixilated. For example, a TDI detector can beneficially be employed to produce output signals by integrating light from at least a portion of an object over time, while relative movement between the object and the imaging system occurs.

BRIEF DESCRIPTION OF THE DRAWING FIGURES

The foregoing aspects and many of the attendant advantages of this invention will become more readily appreciated as the same becomes better understood by reference to the following detailed description, when taken in conjunction with the accompanying drawings, wherein:

FIG. 1A is a schematic diagram of an image collection and capture system of a multichannel optical imaging instrument that includes a plurality of cameras, with one camera and one filter per channel;

FIG. 1B is a schematic diagram of an image collection and capture system for an optical imaging system accomplishing multiparametric imaging with a single camera and a plurality of filters;

FIG. 2 is a graph of wavelength vs. intensity for three optical signal spectra and idealized passbands of corresponding bandpass filters;

FIG. 3 is a flow chart showing the logical steps generally implemented in the crosstalk correction method of the present invention;

FIG. 4 is a flow chart showing the logical steps in applying spatial and spectral corrections to image signals, in accord with the present invention;

FIG. 5 is a flow chart showing the logical steps in the calibration phase of spatial corrections to image signals, in accord with the present invention;

FIG. 6 is a flow chart showing the steps employed for generating accumulated average correlograms between the data images and the reference image;

FIG. 7 is a schematic diagram indicating the locations of pixels used in determining the correlogram gradient operator, and an equation for that operator;

FIG. 8 illustrates two surface plots and two contour plots of a subregion of an image, including one surface plot and one contour plot before, and one surface plot and one contour plot after application of the correlogram gradient operator;

FIG. 9 is flow chair showing the steps for computing the correlogram between a data image and a reference image in the frequency domain;

FIG. 10 is a schematic diagram depicting three data images and one reference image of objects in a flow stream, showing the objects as misaligned;

FIG. 11 illustrates two grayscale images for two channels of a flow imaging instrument, indicating a vertical misalignment between the two channels;

FIG. 12 is a surface plot of the cross-correlogram for the two images shown in FIG. 11;

FIG. 13 is a schematic diagram illustrating the relative locations and letter designations of pixels used in the computation of surface derivatives for locating a peak of a correlogram;

FIG. 14 is a schematic diagram depicting the relative pixel locations and variable names of the coefficients used in a two-dimensional interpolation for reconstructing the data images;

FIG. 15 illustrates surface plots of the interpolation coefficients for two pairs of image alignment offsets, including a zero offset correction and an offset of 0.3 pixels (each axis);

FIG. 16 illustrates grayscale images showing three channels of a multi-parameter flow imaging system, both before and after the application of spatial and spectral corrections made in accord with the present invention;

FIG. 17 is a plan view of a first embodiment of the present invention in which particles conveyed by a fluid stream (at left side of Figure) are depicted as flowing into the sheet;

FIG. 18 is a side elevational view of the first embodiment shown in FIG. 17;

FIG. 19 is an isometric view of the first embodiment of FIG. 17;

FIG. 20 is an isometric view of a confocal embodiment that includes a slit for spatial filtering of extraneous light;

FIG. 21 is an isometric view showing different locations for a light source in connection with the first embodiment;

FIG. 22 is an alternative to the first embodiment in which a second set of imaging components and a second TDI detector is included for monitoring light from a particle, to avoid interference between Fluorescent In Situ Hybridization (FISH) probes, and showing alternative locations for light sources;

FIG. 23 is an isometric view of an embodiment in which an object is supported by a slide or substrate that moves past a collection lens, showing different locations for a light source;

FIGS. 24A and 24B are respectively a plan view and a side elevational view of an alternative to the embodiment of FIG. 23, which is used to produce a scattered pattern on the TDI detector;

FIG. 25 is a plan view of yet a further embodiment in which light forming a scatter patterned image and spectrally dispersed light from an object are imaged on separate portions of a TDI detector;

FIG. 26 is a plan view of a still further embodiment in which light forming a scatter patterned image and spectrally dispersed light from the object are imaged on two different TDI detectors;

FIG. 27 is a plan view of an alternate embodiment that employs a spectral dispersion component comprising a plurality of stacked dichroic filters employed to spectrally separate the light from an object;

FIG. 28 is a schematic illustration of a detection filter assembly that may optionally be disposed in front of the TDI detector in the embodiment of FIG. 27 to further suppress out-of-band light; and

FIG. 29 is a plan view of another embodiment of the configuration of FIG. 27, wherein the spectral dispersion filter system comprises a plurality of dichroic cube filters orientated at various different angles to create the spectral dispersing effect.

DESCRIPTION OF THE PREFERRED EMBODIMENT

FIGS. 1A and 1B illustrate two configurations of an instrument for implementing the present invention. FIG. 1A shows an embodiment that utilizes a plurality of photosensitive cameras 105. Lens 102 is used to form images on the photosensitive detector arrays of cameras 105. The light along each image formation path 101 is filtered by specially designed mirrors 103 that transmit light in a first range of wavelengths and reflect light in a second range of wavelengths, defining a plurality of different wavebands that are received by individual cameras 105. The signals from cameras 105 are processed by signal processing means 106, which aligns the signals relative to each other, and reduces crosstalk between signals. An optional element is a display 107, on which the plurality of images corresponding to the processed signals can be displayed to a user.

While display 107 will often be beneficially incorporated in such a system, such a display is not required. For example, a user may desire to collect data, process that data with signal processing means 106, and then store the processed data for display at a later time. It is further contemplated that signal processing means 106 and display 107, which are enclosed in a block 109, can be employed to process data (i.e., a multichannel signal) that was previously generated and stored for a period of time. Of course, signal processing means 106 can be employed to process the multichannel signal at any time after the signal has been generated, and the processed signals (aligned and corrected for crosstalk) can then be stored for a period of time before being displayed on display 107 (or further processed). Thus, while the imaging systems shown in FIGS. 1A and 1B represent exemplary preferred embodiments in which the present invention is applied, it should be understood that signal processing means 106 can be incorporated into many other different types of systems that generate a multichannel signal, or even used alone to process previously generated multichannel signals.

Signal processing means 106 ensures that the signals from a multichannel source are aligned relative to each other and reduces crosstalk among the signals from the multichannel source. The manner in which each of these functions is implemented is described in more detail below. Signal processing means 106 preferably comprises a programmed computing device, that includes a microprocessor and a memory in which machine instructions are stored that cause the microprocessor to appropriately process the signals. Alternatively, the signal processing means can comprise an application specific integrated circuit (ASIC) chip that employs hardwired logic for carrying out the processing functions, or a digital oscilloscope that can be configured to provide the required signal processing capability.

An alternative configuration for an imaging system usable with the present invention is shown in FIG. 1B. In this configuration, a single camera 105 is used to form an image in which light from a plurality of sources is filtered and reflected by a set of mirrors 103. Each mirror reflects light in a different waveband, forming a plurality of images in corresponding different regions of the camera's photosensitive array. The light reflected by the first mirror in incident on a first region, while the light transmitted by the first mirror in the stack falls on the face of the second mirror, which reflects light onto a second region of the camera's photosensitive detector. The successive reflection and transmission by the minor stack produces a plurality of spectrally separated images, and single camera 105 produces a multichannel signal corresponding to those images formed on each region of the camera's photosensitive detector. These different signals are processed by signal processing means 106 and optionally displayed on display 107.

Preferably, the light being imaged by the imaging systems of FIGS. 1A and 1B comprises wavelengths entirely encompassed within the passbands of the various channels. In that case, each color of light, such as red, contributes to only one image (e.g., an image of an object that is the source of the color of light in that channel). In many practical applications of the present invention, however, the light that forms images in each channel spans a range of wavelengths broader than the passband of an associated filter for the channel, as shown in FIG. 2. In this example, light 201 from each source is received in three channels 202. The signal conveyed in each channel is then a composite of information for the multiple sources. The object of the present invention is to process the signals from the plurality of channels to deliver the information for each source when displaying the image for that source.

It will be understood that many other multichannel imaging systems can benefit from the reduction in crosstalk provided by the present invention. FIGS. 17 29, which are described in detail below, illustrate examples of additional embodiments of multichannel imaging systems with which signal processing means 106 can be employed, to reduce crosstalk between channels.

In optical imaging systems that produce images of an object, the object may modify incident light by absorption of specific wavelengths, by absorption of the incident light followed by emission of light at a different wavelength, by diffraction of the incident light, or by refraction of the incident light. An object can also emit light without being excited using incident light. Each channel of a multichannel imaging instrument is designed to produce images formed in response to the light from an object produced in one of these ways. The present invention can be employed to enhance the contrast of such images.

In the present invention, the separation of signals from a detector into their proper channels is accomplished by solving a set of linear equations. s.sub.1=.alpha..sub.11m.sub.1+.alpha..sub.12m.sub.2+.alpha..sub.13m.sub.3 s.sub.2=.alpha..sub.21m.sub.1+.alpha..sub.22m.sub.2+.alpha..sub.23m.sub.3- , and s.sub.3=.alpha..sub.31m.sub.1+.alpha..sub.32m.sub.2+.alpha..sub.33m.- sub.3 (1) where m.sub.j is a measurement from channel j, s.sub.i is a characteristic parameter of source i, and .alpha..sub.ij is a weighting coefficient for source i into channel j.

These equations are solved using conventional methods of linear algebra, wherein the variables s.sub.i, and m.sub.j are vectors and the variables .alpha..sub.ij comprise a matrix. The general term for this process is "crosstalk correction," since it removes the information from a source that has spilled over into a channel adjacent to the correct channel. This information spillover is analogous to crosstalk in bundled telephone lines, which results in a listener hearing a conversation from another line.

This set of equations must be solved for each pixel location in an image. It is essential that the images from all of the channels be precisely aligned with one another so that the correct values are entered into the equations for each pixel location. Therefore, a computational process is applied to the images corresponding to the field of view before the crosstalk correction is applied. The "field of view" is the scene captured in an image. The images belonging to a field of view are referred to herein as the "image ensemble" for that field of view.

FIG. 3 illustrates the core processes carried out in the present invention and encompasses various signal processing stages, including calibrating for spatial correction parameters calibrating for spectral correction parameters and producing image ensemble outputs. The image ensemble outputs incorporate both a first spatial correction and a second spectral correction. First, the spatial correction parameters are determined, then the spectral correction parameters are determined, and finally image ensemble outputs are generated that incorporate both the spatial and spectral corrections. It should be noted that FIG. 3 illustrates the data flow used in the present invention, as opposed to a graphical depiction of a sequence of logical steps.

The stage of calibrating for spatial correction parameters is represented by a block 301. In block 301, a calibration image is obtained from actual biological specimens, or artificial specimens (e.g. flow beads). In a preferred embodiment, the calibration image corresponds to a reference channel selected from among a plurality of parallel channels. Most preferably, the reference channel is a bright field channel, with the balance of the channels representing fluorescence channels. In such an embodiment, the calibration image from the reference channel is actually part of the ensemble of images. Thus, the spatial offsets are generated in real time. It is contemplated that a calibration image could be utilized before image ensemble data correlating to actual samples are generated. In such a case, the spatial offset is not "live," but instead is based on the offsets determined with respect to the different channels corresponding to a calibration image that is not part of the image ensemble in regard to sample data. Because such offsets are not determined "live," it is anticipated that offsets determined in such an embodiment will not be as precise as those determined when the calibration image is actually part of the image ensemble corresponding to a particular sample. This approach should be less computationally expensive, and can be beneficially employed when previous data corresponding to similar indicate that little error would be introduced by using stored rather than live offset data.

Referring once again to FIG. 3, the algorithm computes the spatial correction parameters in a block 302. Once the spatial correction parameters are computed, the spectral correction parameters are generated in a block 303. Note that the spectral calibration process requires the use of a control, whereas any sample can be used to provide spatial correction data. The control is preferably a biological specimen or an artificial specimen to which a single known fluorophore has been attached. The fluorophore selected preferably has the characteristic of having a fluorescent signature primarily limited to a single one of the multi-channels. Some small amount of "spill over" from the single fluorophore will likely exist in the other channels. Based on the known spectral signature of the control, and the multi-channel data corresponding to that control, spectral corrections can be determined to eliminate such spill over, or crosstalk. Such a control is also referred to as a single source, because its spectral signature is substantially confined to a single channel. A control can be imaged alone, as part of a calibration phase that occurs before acquiring data from samples. In at least one embodiment, a control is introduced into a batch of samples, so that the calibration can occur during the processing of a batch on samples.

Once the spatial and spectral correction factors have been determined, the signal processing can be performed on the ensemble of images. In a block 304 the ensemble of images are input. The spatial corrections (determined in block 302) are applied to the ensemble of images in a block 305. Next, the spectral crosstalk corrections determined in block 303 are applied to the spatially corrected ensemble of images in a block 306. It is important that the spatial corrections be applied before the spectral corrections are applied. The spatially and spectrally corrected ensemble of images is available as data output at a block 307.

Note that during image processing (in blocks 304 307), the spatial and spectral computation algorithms represented by blocks 302 and 303, respectively, continue to produce measurements that indicate the stability or confidence of the corrections previously applied (with respect to the "live" calibration embodiment described above). Once the algorithm determines that such instability exists, the spatial and spectral calibrations can once again be performed, and the newly generated spatial and spectral correction factors can be applied to additional ensemble of images (until instability is detected, which will cause additional calibrations to be performed). The image processing stages (of blocks 304 307) can be applied in real time as images are collected during system operation, or offline, by accessing stored image files. The images of the image ensemble output in block 307 can be used for further processing and analysis, free of the information degradation caused by crosstalk between channels.

As noted above, stored data, rather than live data, can be used to provide, or augment, the required spatial and spectral correction factors. In addition to stored data derived from a calibration image as described above, other types of stored data can be used to provide additional correction factors. Such data can include stored tables of constants that are derived from measurements and from known characteristics of the objects being imaged. As noted in a block 308, the general positions or channel start columns for the images produced by an instrument such as that shown in FIG. 1B, and inverted source coefficients, as indicated in a block 309, comprise stored information that can be used as correction factors. Such stored constants are not derived from a calibration image.

A flow chart of the general operations of an embodiment of the present invention is shown in FIG. 4. An image ensemble input in a block 401 is the composite signal for a set of images, all of which depict the same field of view, but each of which has been constructed using a signal from a different channel. The number of images in an ensemble is thus equal to the number of channels in the instrument. For example, an image ensemble in which there are four channels is represented in FIG. 10. The reference channel can be any of the channels, although in one preferred embodiment, a bright field channel is preferred over fluorescence channels for use as the reference channel. Referring to FIG. 10, for illustration purposes, the leftmost (first) channel is the reference channel. The first image ensemble would thus include the first cell image from each channel; the second image ensemble would include the second cell image from each channel; and the Nth image ensemble would include the Nth cell image from each channel.

The x (horizontal) and y (vertical) offsets 403 must be established in order for the alignment process to operate on the image ensemble. As noted above in regard to block 301 of FIG. 3, the calibration image is processed to compute spatial X,Y offsets, determining offset values that are input in a block 403 of FIG. 4. The calibration process may be run as a preliminary step in the operation of the instrument and is preferably repeated periodically to accommodate any drift in the image registration caused, for example, by changes in temperature. Details of the process for generating the offsets are illustrated in the flow chart of FIG. 6.

In a block 404, the signals are aligned by selecting one signal in the image ensemble as a reference image and shifting each other signal in the image ensemble by pixel increments to align as closely as possible to the reference signal. Note that shifting by whole pixel increments will rarely fully align the signals, so other steps are needed. For each of the non reference signals, a 3.times.3 interpolation kernel is calculated in a block 405. The interpolation process is described in more detail below, particularly with respect to FIG. 14.

Each non reference signal in the image ensemble is further aligned in a block 406 to a fraction of a pixel by interpolation where the aligned image is represented by expanded integer representation. Expanding the integer representation of the image data to a larger number of bits than the resolution of the captured image enables faster processing to be achieved. This step is particularly important in embodiments that generate spatial correction data "live." In a preferred embodiment of the present invention, the image is originally captured with a resolution of 8-bits, and in block 406, the data are expanded to 16-bits or 32-bits, for ease of processing. This step helps prevent the introduction of round-off errors encountered when processing integers having fewer bits.

Once each signal in the image ensemble is aligned, in a block 407 the crosstalk correction is applied, using a spectral coefficient matrix from a block 408. The crosstalk correction is described in greater detail below. In a block 409, the expanded integer representation is returned to the unexpanded integer representation corresponding to the resolution of the system. The purpose of the return to the unexpanded representation is to reduce the memory required to store the processed image data. In at least one embodiment, the unexpanded representation is an 8-bit digital gray-scale. In a block 402, the signals of the image ensemble have been aligned and crosstalk among the signals has been reduced, and the processing of the image ensemble is complete.

Spatial Alignment

FIG. 5 summarizes the spatial calibration stage. A block 550 represents a sequence of image ensembles. A single image ensemble produces a correlogram ensemble, where each correlogram corresponds to a non-reference channel convolved with the reference channel; this convolution is performed in a block 552. The correlogram ensemble generated from a single image ensemble may not accurately convey the image offsets, especially if those images contain only a few objects. The accuracy and signal-to-noise ratio of the correlogram ensemble can be enhanced by producing correlogram ensembles on a sequence of image ensembles. These correlogram ensembles are accumulated by the N non-reference channels in a block 554. Also in block 554, the accumulated correlograms are used to generate a single correlogram ensemble that represents the average