Inter-picture prediction encoding using correlation between pictures is employed in high-efficient, high-compressed encoding of a motion picture. The inter-picture prediction encoding is also employed in encoding standards such as MPEG-2, MPEG-4 (Part 2), and H.264|MPEG-4 (Part 10) AVC (it is simply referred to as H.264 hereafter). In these encoding standards, a present picture is divided into rectangular pixel blocks, each of which consists of a plurality of pixels, and the plurality of pixels are processed per block. The inter-picture prediction encoding generates a motion vector and a predicted block: the motion vector shows motion between a block on the present picture and a block on a reference picture that is a reference target; and the predicted block is a block that has undergone motion compensation based on the block on the reference picture. The inter-picture prediction encoding then performs encoding for the difference between the pixel values of pixels of the predicted block and those of the block on the present picture.
In order to acquire the better encoding efficiency by increasing the accuracy of the predicted picture generated by the above-mentioned motion compensation, each component of the motion vector is expressed in a subdivided unit that is smaller than a full-pel, such as a half-pel and a quarter-pel. In the motion compensation, generation of a predicted picture from a reference picture requires the interpolating processing of pixels, and the calculation method thereof is defined by each of the encoding standards.
According to H.264, one of the encoding standards, 6-tap filtering as a method of the above-mentioned interpolating processing of pixels is specified by Document 1 (Joint Video Team (JVT) of ISO/IEC MPEG & ITU-T VCEG; “Draft ITU-T Recommendation and Final Draft International Standard of Joint Video Specification (ITU-T Rec.H.264|ISO/IEC 14496-10 AVC)”, refer to Section 8.4.2.2).
The 6-tap filtering is explained briefly now. The 6-tap filter includes tap coefficients {1, −5, 20, 20, −5, 1}. Expressing a pixel value of a pixel at i-th location as a[i] (i: integer), among pixels aligned along a horizontal or vertical line on the reference picture, a pixel value for an interpolated pixel at (k+½)th location is calculated as follows. First, according to Equation 1, an intermediate pixel value “b” of the interpolated pixel is calculated. Then, according to Equation 2, normalization and saturation manipulation are performed for the intermediate pixel value “b”, and a pixel value “c” is calculated as the final pixel value of the interpolated pixel.b=a[k−2]−5*a[k−1]+20*a[k]+20*a[k+1]−5*a[k+2]+a[k+3]  (Equation 1)c=Clip((b+16)/32)  (Equation 2)
FIG. 8 shows location relationship of the pixels of the reference picture, that is, the pixels such as full-pels, interpolated half-pels, and interpolated quarter-pels (Document 1: FIGS. 8-4). In the figure, pixels “A” to “U”, each surrounded with a circle, are full-pels. Pixels “b”, “h”, “j”, “m”, “s”, and “aa” to “hh”, each surrounded with a square, are half-pels. The remaining pixels “a”, “c”, “d”, “e”, “f”, “g”, “i”, “k”, “n”, “p”, “q”, and “r”, each surrounded with a square, are quarter-pels.
In this example, the pixel values of the pixels “b” and “s” can be calculated by performing the above-mentioned 6-tap filtering to the pixel values of the pixels aligned along a horizontal line on which each of the pixels “b” and “s” is located. The pixel values of the pixels “h” and “m” can be calculated by performing the 6-tap filtering to the pixel values of the pixels aligned along a vertical line on which each of the pixels “h” and “m” is located. The pixel value of the pixel “j” can be calculated by performing 6-tap filtering in the vertical direction to the pixel values of six pixels “aa”, “bb”, “b”, “s”, “gg”, and “hh”, which are calculated by the 6-tap filtering in the horizontal direction. Alternatively the pixel value of the pixel “j” can be calculated by reversing the order of the filter direction and performing the 6-tap filter in the horizontal direction to the pixel values of the six pixels “cc”, “dd”, “h”, “m”, “ee”, and “if”, which are calculated by the 6-tap filtering in the vertical direction.
The pixel value of a quarter-pel can be calculated by the average of the two pixel values of the adjoining half-pels or full-pels. For example, the pixel value of a pixel “a” is calculated by the average of the pixel value of a full-pel “G” and the pixel value of a half-pel “b”; the pixel value of a pixel “e” is calculated by the average of the pixel value of the half-pel “b” and the pixel value of a half-pel “h”; and the pixel value of a pixel “f” is calculated by the average of the pixel value of the half-pel “b” and the pixel value of a half-pel “j”.
When generating a predicted picture by performing the interpolating processing based on the above-mentioned 6-tap filtering, a lot of processing amount is required in order to calculate a pixel value of an interpolated pixel. For example, when the interpolated pixel is any one of the pixels “f”, “i”, “j”, “k”, and “q”, operation is performed for the pixel values of as many as 36 full-pels. The interpolating processing needs to be performed as many as the number of the pixels included in the predicted blocks, furthermore, the predicted blocks include all the blocks for which inter-picture prediction is performed; therefore, the motion compensation processing requires a tremendous amount of operation in motion picture encoding or decoding.
FIG. 9 is a flow chart showing an example of the conventional pixel interpolating method of calculating a pixel value of an interpolated pixel by the above-mentioned 6-tap filter. In the following, the conventional pixel interpolating method shown in FIG. 9 is explained.
When the interpolating processing starts at Step S0, initial values are allotted to variables “m” and “k” at Step S1. Here, the variable “k” is an index for reading the pixel value of a target reference pixel from an array “a” storing the pixel values of pixels which constitute a reference picture. In correspondence to the above-mentioned FIG. 8, for example, in a case for calculating the pixel value of the interpolated pixel “b”, the variable “k” indicates the full-pel “J”. The variable “m” is an index that indicates a location of the interpolated pixel in the predicted block. A variable “i” is an index indicating the reference point to be used in reading the pixel value of the reference pixel. More specifically, the variable “i” indicates a full-pel located on the upper left of an interpolated pixel to be calculated first. In the example shown in FIG. 8, the variable “i” indicates the pixel “G”.
At Step S2, the pixel values of five reference pixels beginning from the edge of the reference area are read. This processing is a preparatory step for calculating the pixel value of the first interpolated pixel.
At Step S3, the pixel value of the reference pixel located at the variable “k” is read.
At Step S4, an intermediate pixel value “b” of the interpolated pixel is calculated by Equation 1.
At Step S5, according to Equation 2, normalization and saturation manipulation of the intermediate pixel value “b” are performed to calculate the pixel value “c” of the interpolated pixel. The normalization means to divide the intermediate pixel value “b”, which is calculated by Equation 1, by a value “32”. The function Clip (x) is a function suppressing (clipping) a variable “x” within a range of 0 to 255, when the variable “x” exceeds the range of 0 to 255.
At Step S6, the pixel value “c” of the interpolated pixel calculated at Step S5 is outputted.
At Step S7, for the next loop, the variable “m” and the variable “k” are incremented by the value “1”, respectively.
At Step S8, whether or not the number of the generated interpolated pixels has reached the predetermined number “N” is judged. When the judgment result is “No” (it has not reached the predetermined number “N”), the control is returned to Step S3, and Step S3 to Step S8 are repeated. When the judgment result is “Yes” (it has reached the predetermined number “N”), the control is advanced to Step S9, and a series of the pixel interpolating processing ends.
As mentioned above, every time when loop processing from Step S3 to Step S8 of FIG. 9 is performed, the pixel value of one interpolated pixel is calculated.
FIG. 10 shows a block diagram of a conventional pixel interpolating device using a 6-tap filter. FIG. 10 illustrates the 6-tap filter which is constructed so as to be used for the above-mentioned interpolating processing, based on the structure of a tap filter disclosed by Document 2 (Published Japanese patent application No. H07-15734). As shown in FIG. 10, the conventional pixel interpolating device comprises registers (symbolized as “R” in the figure) 1 to 6, multiplying units (symbolized as “M”) 7 to 10, adding units (symbolized as “+”) 11 and 12, and a clipper 13. The registers 1 to 6 constitute a shift register with input of a pixel value “a [k]” of a reference pixel.
The operation of the conventional pixel interpolating device shown in FIG. 10 is explained in the following.
In the shift register of the registers 1 to 6, by performing shift operation in synchronization with a shift control signal (not shown in the figure) and transferring, step by step, the inputted pixel value “a[k]” of the reference pixel, the pixel values of six consecutive reference pixels are stored one by one in the registers. The multiplying units 7 to 10 multiply the respective pixel values of the reference pixels stored in the registers 2 to 5 and the respectively predetermined coefficients (a coefficient “−5” for the multiplying units 7 and 10, and a coefficient “20” for the multiplying units 8 and 9) shown in the figure, thereby obtaining the respective products. The adding unit 11 adds the value stored in the register 1, the value stored in the register 6, and the product values stored in the multiplying units 7 to 10, thereby calculating the total sum. The adding unit 12 adds a constant “16” to the total sum calculated by the adding unit 11. The clipper 13 performs saturation manipulation, and outputs the pixel value “c” of the interpolated pixel. By inputting the pixel values of the reference pixels, step by step, according to the shift control signal, the pixel value “c” of each of the consecutive interpolated pixels can be calculated by the 6-tap filtering using Equation 1 and Equation 2, as mentioned above.
Since the multipliers in the above multiplication are the predetermined constants, the multiplying units 7 to 10 can be constructed by units functioning bit-shift, addition, and sign change. The multiplying units 7 to 10 thus constructed are smaller in circuit scale than the regular multiplier.
In the pixel interpolating device in which the conventional pixel interpolation method is adopted as explained above, there is a limit regarding the high-speed processing in software processing and the power consumption reduction, because the operation amount is constant irrespective of the pixel values of reference pixels and no measure is taken for reducing the operation amount by considering the pixel values.
As a method of improvement in speed, Document 3 (Published Japanese patent application No. 2002-190948) discloses an art to use a look-up table which stores the multiplication result, thereby reducing the operation load. However, this method has a problem of the circuit scale enlarged as the result of installing the look-up table therein.
As explained above, the conventional pixel interpolating method and the conventional pixel interpolating device can not fully deal with the following problem.
In order to meet the request of producing a high-quality motion picture with a high-definition picture and increased frame rate, pixels to be processed per unit time needs to be increased in number. However, in the conventional art, there is a limit for the processing speed; therefore, it is difficult to improve the speed of the motion compensation.
Moreover, when an art of motion picture encoding/decoding accompanied by such pixel interpolating processing is used for portable devices such as a cellular phone, decreasing the power consumption imposes another great concern, since there is a limit for the power supply used in these portable devices.
USPTO Backgrounds
USPTO Backgrounds