Achieving good image and video compression efficiency requires choosing a symbolic representation of image color and brightness that closely approximates the differential sensitivity of the human visual system (hereinafter, HVS); otherwise, coding rate is wasted as described in Joan L. Mitchell, et. al., “MPEG Video Compression Standard,” Chapman & Hall, Ltd., London, UK, 1996, incorporated herein by reference. Since perceptual mechanisms of the HVS are complex and non-linear, design of color systems and color spaces for video and image compression, transmission, and display incur a tradeoff of accuracy and precision versus system complexity and practical implementation.
The International Commission on Illumination (usually abbreviated as CIE for Commission internationale de l'éclairage) color space (hereinafter “CIE1931 XYZ”) represents the first attempt (i.e., 1931 refers to the year of publication) to produce a color space based on coarse measurements of human color perception. CIE 1931 XYZ is used in JPEG2000-based codec systems such as those used by Digital Cinema Package as specified in “Digital Cinema System Specification Version 1.2 with Errata,” DCI, LLC, 10 Oct. 2012, but is not otherwise widely used in video transmission systems because of its complexity and the need for a standard illuminant and tristimulus value to be provided by a system implementer (or an end-user). Despite its good approximation to the HVS's color perception sensitivity, CIE1931 XYZ is far from a perfect representation of the HVS's sensitivity to color differences—even with a required tristimulus value and illuminant value being held constant—knowledge gained as a result of research into, and careful measurements of the HVS and its perceptual sensitivities as described in International Color Consortium, “Specification ICC.1:2004-10 (Profile version 4.2.0.0) Image technology colour management—Architecture, profile format, and data structure,” (2006), incorporated herein by reference.
After CIE1931 XYZ, and with the later advent and standardization of color television, the luma (Y), in-phase (I), quadrature (Q) color space (hereinafter “YIQ color space”) was developed in 1953 primarily as a way to encode color signals in a severely band-limited transmission and reception regime. While YIQ is certainly a better approximation of the perceptual tendencies of human visual perceptual characteristics than the RGB color space representation required by RGB phosphor cathode-ray-tube color televisions at the time of their invention and widespread usage, the YIQ color space is by no means ideal. YIQ color space was primarily conceived for practical implementation purposes for the cost-effective analog radio-frequency components available contemporaneously and for maintaining backwards compatibility with existing “black-and-white” television transmission standards.
The YCbCr and Y′CbCr color spaces are derivations of the YIQ color space for purposes of very effective, but yet still very coarse approximations of human visual perceptual color processing and perceptual uniformity. While practical when these color spaces were designed in the early 1980's, designs employing the YCbCr and the Y′CbCr color spaces were limited to simple digital circuits and systems with very limited processing capability and digital memory transfer bandwidth. The YCbCr and the Y′CbCr color space representations form the basis of early and current video compression codec systems that employ the JPEG and MPEG standards of compression. Despite their practicality, the YCbCr and the Y′CbCr color space representations are inefficient because they allocate significant luminance and color depth symbol-rate or bit-rate to perceptually insignificant color differences.
More recently, color spaces and perceptual-difference frameworks that represent a more faithful approximation of HSV perceptual uniformity have been published, such as the CIELAB standard described in “ISO 11664-4:2008(E)/CIE S 014-4/E:2007: Joint ISO/CIE Standard: Colorimetry—Part 4: CIE 1976 L*a*b* Colour Space” (hereinafter, CIELAB). CIELAB accounts for perceptual sensitivities in the lightness and color dimensions, and CIECAM02, described in “CIE 159:2004: A Colour Appearance Model for Colour Management Systems: CIECAM02” (hereinafter, CIECAM02)—which incorporates the aforementioned CIELAB dimensions along with the well-known spatial center-surround retinex effect as observed in E. H. Land, “The retinex theory of color vision.,” Scientific American, 1977.
Even these advanced efforts at defining a perceptually uniform color space suffer from specific observed anomalies. One example is known as the “blue-purple hue constancy” problem, where blue hue colors do not follow a perfectly linear path as lightness traverses the color space from dark to light, as illustrated in detail in Moroney, “Assessing hue constancy using gradients”, Color Imaging: Device-Independent Color, Color Hardcopy, and Graphic Arts V, Reiner Eschbach, Gabriel G. Marcu, Editors, Proceedings of the SPIE Vol. 3963, pp. 294-300 (2000)., incorporated herein by reference. Further, more specific hue-constancy anomalies of CIELAB and its related color spaces have been carefully measured and mapped as taught in Braun et. al., 1998, “Color Gamut Mapping in a Hue-Linearized CIELAB Color Space”, IS&T/SID 6thColor Imaging Conference, pp. 163-168, and are more widespread than just the blue-purple constancy problem. Specifically, a generally observed form of this problem is known as the Bezold-Brücke shift: apparent hue can change with luminance (and vice-versa), and this effect has frustrated efforts to find perceptually efficient color representations for video and image transmission, among other applications.
Many attempts at extending or modifying the CIELAB and related color space representations exist, such as Takamura and Kobayashi, 2002, “Practical extension to CIELUV color space to improve uniformity”, IEE ICIP 002, which teaches alternate conversion matrix coefficients over CIELUV to improve perceptual linearity, and Behrens, “Deficiencies of the CIE-L*a*b* color space and introduction of the SRLAB2 color model”, at “www.magnetkern.de/srlab2.tex” (hereinafter, SRLAB2, incorporated herein by reference). SRLAB2 proposes a whole new color space representation using the chromatic adaptation model of CIECAM02, but which trades off blue-purple (hue) constancy for a reduction in hue angle uniformity and hue-lightness interval lengths uniformity, especially in the skin-tones region of the color space.
Apparently, there exists no perfectly perceptually uniform color space that exhibits all of the characteristics of luminance or lightness uniformity, hue constancy, hue angle uniformity, and hue-lightness interval lengths uniformity, all of which are necessary for an ideal color space representation for video and image coding and transmission.
Since the time of inception of the YCbCr and Y′CbCr color spaces, primarily for simplicity and practicality of implementation, and later for backwards compatibility reasons, most of the video encoder systems today remain standardized upon utilizing the YCbCr color spaces, and not the recent, more complex, but more perceptually uniform color spaces and perceptual-difference-based color representations.
The inefficiency of using YCbCr as a color space basis for video compression extends beyond realizations of color symbol representation, but also luminance as well. For example, CIELAB introduces a non-linear, non-exponential perceptual curve for lightness perception as well, not just color—and in most image and video encoding and display systems, this is not accounted for other than by a simplistic exponential Gamma function. Actual observer measurements have demonstrated that a simple exponential or logarithmic relationship is not adequate to represent perceptual differences of luminance, especially in the low areas of the luminance range, as taught in CIELAB.
Yet, even the most current video and image encoding systems such as the proposed HEVC video encoding standard at the time of this disclosure continue to utilize YCbCr as a color space basis, despite disadvantages including reduced reconstruction quality and wasted coding-rate.
The deficiencies of the selections of color space basis and symbol representation of current video encoding systems is well known in the art, and there have been several attempts to rectify or at least mitigate the negative impact of these codec inefficiencies. Early attempts to use perceptually uniform color space representations as a basis for image and video compression such as those taught in Moroney and Fairchild, “Color space selection for JPEG image compression” Journal of Electronic Imaging 4(4), 373-381 (October 1995) and Drukarev, “Compression-related properties of color spaces”, SPIE Vol. 3024, were frustrated by either the complexity of their application for real-time encoding and decoding, or they exhibited comparatively little benefit. Further, the CIE1931 XYZ color space extensively utilized in the related art is recognized to be a better approximation of the human visual system sensitivity to color differences when compared against RGB, YIQ, or YCbCr, but is also highly non-uniform by as much of a ratio as 80:1, as discussed in Poynton, Charles, “A Technical Introduction to Digital Video.,” John Wiley & Sons, 1996, and the accompanying “Frequently Asked Questions about Colour” at “www.poynton.com/ColorFAQ.html”, (hereinafter, Poynton) both of which are incorporated herein by reference. With CIELAB, this ratio improves to approximately 6:1, but as Poynton points out, CIELAB conversion is very computationally expensive for video and was not suitable for real-time processing at the time of his writing. Uniformity is a key concept for realizing the most efficient coding, and even CIELAB is far from being an ideal perceptually uniform color space, as shown in the prior art.
Other methods, such as those described in U.S. Patent Application Publication No. US 2012/0314943 A1 (hereinafter, “Guerrero”), attempt to achieve a solution by employing either: (1) a quantization factor applied in a color space optimized for a model of a human visual system, which decreases color space entropy and redundancy or (2) a color table lookup step to decrease color space entropy and redundancy but which increases the memory transfer bandwidth requirements of the system because of the need for a table lookup for each pixel at the pre-encoding stage. Both of these methods are effective at marginally reducing color space entropy provided that a suitable HVS model and perceptually uniform color space is utilized—Guerrero utilizes CIE1931 XYZ which is not a perceptually uniform color space, and while casual mention of CIELAB is made, neither is CIELAB an ideal color space for this purpose for the aforementioned reasons. Most importantly, the method disclosed by Guerrero reduces color entropy at the expense of increasing spatial entropy, negating most of the benefits of implementation with standard DCT-based encoders such as JPEG and MPEG. It would be possible to reduce both spatial and color space entropy and redundancy by combining the disclosed quantization factor or color table lookup along with a histogram compression function, but this would require post-processing on the decoder end to expand the histogram, and further would require in-band or out-of-band communication to coordinate the parameters of such histogram companding. Guerrero does not disclose these concepts, and in fact teaches away from them. Further, performing histogram computations at real-time throughput rates on high-definition, 4K, 5K and 8K resolution video in real-time on mass-market computer systems or end-user devices is exceptionally challenging, as taught in U.S. Pat. No. 8,451,384.
The aforementioned in-band or out-of-band communication methods to coordinate pre-filtering and post-filtering of an encoder and decoder, respectively, are well-known in the art, as described in U.S. Pat. No. 6,195,394 (hereinafter the '394 patent). Although the pre-filtering and post-filtering processes described in the '394 patent are directed to the reduction and subsequent restoration of spatial bandwidth, and not reduction and restoration of perceptual color bandwidth, the '394 patent demonstrates that an out-of-band communications method for signaling the presence of, and the configuration of, pre-filtering and post-filtering operations is needed to ensure proper reconstruction of images and video in proximity of the decoder.
What is needed, but has not been provided, is a high-throughput system and method that improves the perceptual quality and/or the transmission or storage efficiency of existing image and video compression or transmission systems and methods that does not impose an excessive burden of processing complexity or memory transfer bandwidth requirements upon either an encoder or decoder device or system. The system and method would impose no requirement to replace the encoder or decoder. The system and method would synchronize pre-filtering of the encoder and post-filtering of the decoder to signal the presence and configuration of the system and method.
USPTO Backgrounds
USPTO Backgrounds