The dynamic range of a Digital Still Camera (DSC) is greatly limited by SN levels which indicate the noise ratio of analog values obtained from the CCD image sensor and by the precision of converting analog values to digital values. Therefore, detail in dark area tends to be lost in the image taken by the DSC. Particularly, this tendency is large in samples where there are both light areas and dark areas.
As a method of improving the quality of the image, first there is a method of enhancing the contrast such that the brightness range of the digital image extends from the region of high brightness to the region of low brightness.
As a conventional method of enhancing the contrast there is the method of histogram equalization. This method is a method that creates a histogram showing the distribution of brightness of all of the pixels on the original image, and uses the accumulated curves of that histogram as the brightness conversion curve to convert the brightness values of the pixels of the original image to new values.
In this method, in order to use the same brightness conversion curve to convert the brightness value of every pixel in the whole of the original image to a new brightness value, there may be a partial decrease in contrast.
In order to avoid this, a contrast enhancement process that is suitable for a part can be performed. Many procedures to do this have been proposed such as a localized histogram equalization method that divides the image into a plurality of rectangular areas and performs histogram equalization for each of the areas.
For example, as disclosed in Japanese unexamined patent publication No. 2000-285230 and as shown in FIG. 1, a contrast correction unit comprises an image division unit 2001, histogram creation unit 2002 and contrast stretching unit 2003.
The image division unit 2001 divides the input image into rectangles. The histogram creation unit 2002 creates histograms for each of the rectangles. The contrast stretching unit 2003 performs contrast stretching for each of the rectangles.
In the method of this disclosure as well, problems occur in that there are rectangular areas for which the contrast is enhanced too much, and the contrast is not continuous at the boundary between adjacent rectangular areas.
A technique has also been proposed that solves these problems in which a histogram is not used. For example, as disclosed in Japanese unexamined patent publication No. H6-141229, the shutter time and lens stops of the digital camera are changed for each field, to photograph the light areas and dark areas separately. By combining both of the obtained images into one image, halftone densities are presented. This makes it possible to obtain an image having a large dynamic range.
FIG. 2 is a block diagram showing the construction of the image processing apparatus disclosed in Japanese unexamined patent publication No. H6-141229. In this image processing apparatus, the image sensor 2101 performs photoelectric conversion of the light image of the photographed object. An image combination unit 2102 weights and combines two or more images having different electric charge storage periods in the image sensor according to the signal levels of the images. In order to accomplish this, the image combination unit 2102 comprises: a memory 2103, a multiplication unit 2104, level weighting units 2105 and 2106, and an adding unit 2107. The memory 2103 stores image signals. The multiplication unit 2104 multiplies a constant to a signal level. The level weighting units 2105 and 2106 modify the image signals by weights, and the adding unit 2107 adds the image signals.
Also, a speed conversion unit 2108 converts the speed of the image signal, and a level compression unit 21 9 compresses the level of the image signal. Moreover, a timing control unit 2110 controls the timing of each block. This apparatus is for a television imaging system that compresses a television signal to a standard level, so in order to convert the obtained combined image output to speed of a standard television signal, there is a speed conversion unit and a level compression unit. In the case of applying this kind of technique to a digital camera, the speed conversion unit and level compression unit are not necessary.
In the case of the method of combining images obtained for a plurality of electric charge storage periods as described above, it is unlikely that a discontinuity of contrast occurs in the combined image. However, since this method is required to take at least two images in sequence, this is impossible to take the same images in principal. Therefore, when the images are combined, there is a possibility that an image will be created in which the detailed parts of the combined image will be blurred or shifted, although depending on the shutter speed. Also, when it is not possible to cover the entire density range of the image with the density range obtained when photographing the light area and the density range obtained when photographing the dark area, there is a danger that discontinuity will occur in the middle density range.
Also, a method for improving the image quality of a digital image by using the Retinex theory has been disclosed in International Publication No. WO97/45809 or Published Japanese translation of PCT international application No. 2000-511315. When a person observes an object, the problems mentioned above do not occur for detailed areas and colors in dark areas. People are capable of visually perceiving the large density dynamics and colors of the original images. Taking notice of this human visual perception, the concept of center/surround Retinex was introduced by Edwin Land in the publication ‘An Alternative Technique for the Computation of the Designator in the Retinex Theory of Color Vision’, National Academy of Science, Vol. 84, pp. 3078 to 3080 (1986).
This document explains the concept of the Retinex theory and states that in human visual perception, the center view can be represented by an inverse square function having a diameter of from two to four basic units, and that the surround view can be represented by an inverse square function having a diameter of approximately 200 to 250 times that of the center view. Also, the spatial average of the signal intensity in the field of view of both the center view and surround view is defined as being related to the perceived intensity. The method described in the pamphlet or publication is one method of improving the expression of color and lightness in dark areas according to this kind of theory.
FIG. 3 is a block diagram that explains the image processing apparatus described in the pamphlet or publication. Here, the image processing apparatus is explained for a grayscale image as an example, however, that explanation can be expanded to include color images as well.
The processor 2202 and filter 2203 perform adjustment and a filtering process on the values I (i, j) of the pixels (i, j) of the image obtained from the digital image pickup apparatus 2201.
The processor 2202 calculates the adjusted values I′(i, j) given by the following equation 1 for each pixel.I′(i,j)=logI(i,j)−log[I(i,j)*F(i,j)]  (Eq. 1)
Here, F(x, y) is the surround view function that expresses the surround view, and ‘*’ denotes the convolution operation. By setting the normalization coefficient K such that the condition of Equation 2 below is satisfied, the second term of Equation 1 is the weighted average value of the pixel values of the area corresponding to the surround view.K∫∫F(i,j)di dj=1  (Eq. 2)
In other words, Equation 1 is an equation that corresponds to the logarithmic conversion of the ratio of each pixel to the average value in a large area. The surround view function F(i, j) is designed from the correspondence to the model of human visual perception such that the contributing ratio becomes higher closer to the object pixel, and a Gaussian function such as that of Equation 3 below is applied.F(i,j)=exp(−r2/c2)r=(i2+j2)1/2  (Eq. 3)
Here, c is a constant for controlling the adjusted values I′(i, j) for each of the pixel values I(i, j).
When the ratio of the object to with the average pixel value of the area corresponding to the surround view is calculated as the adjusted value I′(i, j), the filter 2203 performs a filtering process on this value, and generates Retinex output R(i, j). This filtering process is a process of converting the adjusted value I′(i, j) in the logarithmic domain to the value R(i, j) in the domain used by the display 2204, and to simplify this process, a process that uses the same offset and gain conversion function for all of the pixels is adopted.
One problem with this method is that the effect by the constant c that controls the surround function is large. For example, when this constant c is a large value and the area corresponding to the surround view that contributes to improvement becomes large, then it is only possible to compensate for colors in large shadows. On the other hand, when this constant c is a small value and only the pixel values near the object pixel affects improvement, then the improvement is limited to a corresponding small area. Therefore, it is necessary to consider a constant c that is suitable to the image being processed. In order to ease this dependence, the document proposes a method in which areas suitable for a plurality of sizes of surround views are prepared. However, it is not clear how many area sizes should be prepared. By preparing many large areas and small areas in order to increase the improvement precision, the processing time becomes very long.
Also, there is a problem in that knowledge from experience is necessary in order to set a suitable offset and gain conversion function.
Furthermore, when there is very small change in the pixel value in the largest area of the areas set by the plurality of constants c, the adjusted value I′(i, j) comes close to 1.0 regardless of the value I(i, j) even when a plurality of areas are prepared. An adjusted value I′(i, j) for an object pixel in the area with little change is often located near the average value of the adjusted values I′(i, j) for the entire input image. Regardless of the offset and gain conversion function, the adjusted values I′(i, j) gather near the center of the histogram. Therefore, particularly in a large area uniformly having highlight brightness, it becomes easy for the brightness to be adjusted in a direction that lowers the brightness and thus worsens the visual appearance. Also, in a large area where the brightness is low, for example in a night scene, color noise and compression noise that occur when taking the photo appear due to excessive enhancement.
USPTO Backgrounds
USPTO Backgrounds