1. Field of the Invention
The present invention relates to image processing methods, image processing devices, programs, storage media, and integrated circuits for enhancing the feeling of depth and the three-dimensional effect of an image in accordance with the foreground region and the background region of a two-dimensional image.
2. Description of the Related Art
To display more natural video on the screen of a large screen FPD (flat panel display) device, for example, users have strongly called for technology that increases the “feeling of depth” and the “three-dimensional effect” in displayed video. In response, three-dimensional televisions and the like that utilize the binocular parallax of humans have been proposed, but it has been pointed out that special dedicated glasses are often required, that there is a large degree of dependence on the image, and that the special devices that are required increase costs. At the present time, one of the selling points of large screen display devices is the technology that achieves a three-dimensional effect in the displayed image (video) by smoothing the gradation characteristics or the color characteristics in the display image of the large screen display device.
It is clear that humans utilize not only binocular parallax but also monocular information such as color information, saturation, brightness, contrast (color information contrast and brightness information contrast), shadows, gradient of texture, and relative size, in order to perceive depth and three-dimensionalness in two-dimensional images.
One conventional technology that utilizes such monocular information is the method of visually increasing the difference between an object and the surrounding background by making distinct the differences between strong areas and other areas of the border component in order to give the image a sense of depth (for example, see JP H10-126708A).
FIG. 71 is a block diagram showing the configuration of a three-dimensional expression circuit 9000 based on the conventional technology. As shown in FIG. 71, the three-dimensional expression circuit 900 is provided with a border enhancing portion 5011 that includes a border extraction portion 5002 for extracting border components from target pixels and nearby pixels from input data made from the luminance signal of an input digital video signal, and an adding circuit 5004 that adds the output of the border extraction portion 5002 and the input data that match the timing of this output (data input to a terminal 5000) to calculate output data B (which correspond to the signal (data) of point B in FIG. 71). The three-dimensional expression circuit 900 is also provided with an averaging circuit 5005 that computes averaged data A (which correspond to the signal (data) of point A in FIG. 71) from a pixel of interest and nearby pixels of the input data, a threshold processing portion 5009 for comparing the border component, which is the output of the border extraction portion 5002, against a freely established threshold value, and a selector that selectively outputs the averaged data A of the averaging circuit 5005 or the output data B of the border enhancing portion 5011 according to the size of the output of the threshold processing portion 5009.
With this device (the three-dimensional expression circuit 900), data whose border has been enhanced over the input data are output for sections with a strong original border component (large edge amount), and data obtained by averaging the input data are output for sections with a weak original border component (flat with a small edge amount), and thus the difference between sections with a strong border component and other sections becomes clear, and images formed by the output data processed by the three-dimensional expression circuit 900 become images that appear three dimensional.
A second separate conventional technology is the technique of adding the perception of near and far by changing the strength of border enhancement in accordance with the attributes (background region or foreground region) of a region that have been detected, where in the image formed by an image signal, regions in which the signal level of a first-order differential signal or a 2nd-order differential signal of the image signal is large are detected as foreground regions, and regions in which that signal level is small are detected as background regions (for example, see JP 2004-159148A).
FIG. 72 is a block diagram showing the configuration of a depth perception enhancement circuit 9100 based on the conventional art.
The depth perception enhancement circuit 9100 enhances the border through edge-added border enhancement in the image formed by the image signal (image signal that is input to the depth perception enhancement circuit 9100), wherein the image signal is a luminance signal, and is provided with a differential circuit 6001 that performs first-order differentiation and secondary differentiation on the image signal that is input (input image signal), a perspective detection portion 6002 that detects near and far regions of the image from the first-order differential value and the 2nd-order differential value that have been calculated by the differential circuit 6001, a coefficient weighting portion 6003 that multiplies the 2nd-order differential value by a coefficient value depending on the result of the detection by the perspective detection portion 6002, and an adding portion 6005 that sums the output signal from the coefficient weighting portion 6003 and the input image signal, which is delayed by a delay portion 6004 in order to adjust the timing.
The perspective detection portion 6002 determines whether a target pixel (pixel corresponding to the input image signal) is a pixel that belongs to a foreground region or is a pixel that belongs to a background region, based on the signal level of a signal S3 that is obtained by quantizing the first-order differential signal DY1 and the signal level of a signal S5 that is obtained by quantizing the 2nd-order differential signal DY2. The perspective detection portion 6002 compares the signal level of the S3 signal with a set value for determining whether or not that pixel belongs to a border region, and if the signal level of the S3 signal is equal to or greater than that set value, then it sets the S4 signal to 1, and in all other cases it sets the S4 signal to 0. For pixels in a border portion of regions where the S4 signal is 1, the perspective detection portion 6002 performs a determination as to whether the signal level of the S5 signal (the signal that is obtained by quantizing the absolute value of the 2nd-order differential signal DY2) is in the foreground or the background by determining whether or not it exceeds a threshold TH. The perspective detection portion 6002 determines that the target pixel belongs to the foreground region if the signal level of the S5 signal is greater than the TH value. The perspective detection portion 6002 also designates a value K1 that is larger than a default value KS for the coefficient KM that is multiplied with the 2nd-order differential signal DY2, and outputs the value that has been designated to the coefficient weighting portion 6003. On the other hand, the perspective detection portion 6002 determines that the target pixel belongs to the background region if the S5 signal is smaller than TH. The perspective detection portion 6002 also designates a value K2 that is smaller than a default value KS for the coefficient KM that is multiplied with the 2nd-order differential signal DY2, and outputs the value that has been designated to the coefficient weighting portion 6003.
In this way, the depth perception enhancement circuit 9100 determines whether a pixel that is believed to lie in the border portion is included the foreground region or is included in the background region by performing a threshold determination of the 2nd-order differential signal DY2 of that pixel. Then, depending on the results of that determination, the depth perception enhancement circuit 9100 increases the coefficient for weighting the 2nd-order differential signal DY2 if the pixel is in the foreground, or reduces that coefficient if the pixel is in the background, and executes the enhancement processing of the border portion, thereby increases the feeling of perspective near the border portion.
With the image processing method of the first conventional example, a preset threshold signal Y and a value X of the border component that is obtained by border extraction are compared to determine whether to output the border-enhanced data or output the averaged data. For this reason, the precision with which the threshold is determined is prone to impact this. Also, with the first conventional example, averaged data are output for sections where the border component is weak, and thus in these sections blurring (phenomenon of blurring) will occur. In the case of originally high resolution image data, the drop in resolution caused by this blurring has a significant impact on the picture quality. Particularly in the case of large-screen televisions, which are becoming increasing HD, it is preferable to execute strong border enhancement so as not to enhance noise because of user's or broadcast-related demands, and under these circumstances it is difficult to perform blur processing such as averaging, even for flat portions with a weak border.
Further, with the image processing method of the second conventional example, the determination of foreground or background is performed only for border portions that have a sufficiently large differential value, and thus the determination of foreground and background is not performed on weak border portions such as texture patterns or on border portions that cannot be suitably extracted due to the image capturing conditions, such as the ambient outside light. In other words, there is a high probability that border extraction with a first-order differential signal will be affected by the precision of the threshold determination. Further, since threshold processing is performed on the 2nd-order differential signal of pixels that are believed to be in the border in order to determine whether they are foreground or background, the threshold determination precision is also prone to have an impact when determining depth information as well. Thus, there is a risk that objects with the same border and at the same distance may have both strongly enhanced and weakly enhanced edges, and there is also a risk that areas where the luminance is not continuous may occur due to only strong edge enhancement and weak edge enhancement being performed at the border portions.
It is not possible to determine whether the 2nd-order differential signal of the luminance is small as the result of blurring caused by the conditions under which the image was captured (focal point shifting or movement) or due to the interpolation that is performed when simply transforming a low resolution image to a higher resolution, or whether that pixel is actually in the background region. For this reason, for example, in the case of decoding an image that has been encoded with an irreversible encoding method and then removing distortion in the decoded image with a low-pass filter or the like, the entire image may be determined to be background, depending on the setting for the threshold determination value of the 2nd-order differential signal, and there is a danger that conventional edge processing will not be carried out appropriately.
The invention solves the issues with the first conventional example and achieves an improvement in the feeling of depth and the three-dimensional effect of the image, and it achieves an increase in the feeling of depth by performing color correction according to depth information without choosing to use either border-enhanced data or averaged data like in the first conventional example. In particular, it is an object to achieve an image processing device, an image processing method, a program, a storage medium, and an integrated circuit that can further enhance the sense of depth of colors that draw human attention by correcting the color contrast effect, which is a visual characteristic of humans, by linking it to the effect due to the depth information.
Also, the color contrast characteristics or the brightness contrast characteristics, which visual characteristics in humans, can be used to determine whether the color or brightness of a target portion (a portion on the image that has been targeted for processing) is greater than the surroundings, even if the image contains blurred sections, for example. Moreover, it has been pointed out that there is a strong tendency for such portions (those portions in which the color or the brightness of the target portion is greater than the surroundings) to be noticed by humans, and by assuming that portions that are readily recognized by humans are the foreground, it is possible to easily estimate the depth information of the image.
The present invention also solves the issues relating to depth information estimation in the second conventional example, and it is an object thereof to achieve an image processing device, an image processing method, a program, a storage medium, and an integrated circuit that can obtain an image with an improved sense of depth without employing border extraction with a differential signal as in the second conventional example, and by using depth information that has been estimated based on the color contrast characteristics or the brightness contrast characteristics in order to further enhance colors that are readily noticed by humans and weaken colors that are not readily noticed humans.
USPTO Backgrounds
USPTO Backgrounds