The present invention relates to an image processing apparatus and an image processing method and, more particularly, to image processing in which input image data is temporarily stored in a storage device, and the stored image data is subjected to coding.
Although image data including a time-varying image is originally analog data, when this data is digitized, various kinds of complicated signal processing and data compression can be performed on the data and, therefore, the technology of image digitization forms an important field. In an image processing apparatus according to a prior art, input analog image data is subjected to analog-to-digital conversion and compressive coding for recording or transmission, and generally the digitized image data is temporarily stored in a high-speed storage unit such as a memory before being subjected to compressive coding. An example of an image processing apparatus with such temporary storage is disclosed in Japanese Patent Application No. Hei. 7-273461, in which digital image data is temporarily stored in a memory called an image frame memory.
FIG. 10 is a block diagram showing the structure of an image processing apparatus according to the prior art. As shown in the figure, the prior art image processing apparatus is provided with an A/D converter 1001, an image input controller 1002, a memory controller 1003, an encoder 1004, an input image memory 1005, and a rate buffer 1006, and this apparatus receives an analog video signal S1051 and outputs coded data S1057. In FIG. 10, signals shown by solid lines indicate the flow of data to be processed, and signals shown by broken lines indicate the flow of signals for control.
The A/D converter 1001 subjects the input analog video signal S1051 to analog/digital conversion to generate digital image data S1052. The image input controller 1002 generates an image input enabling signal S1061 indicating whether the input digital image data S1052 is xe2x80x9ceffectivexe2x80x9d or xe2x80x9cineffectivexe2x80x9d. The memory controller 1003 controls storage and readout of the digital image data into/from the memory. The encoder 1004 subjects the digital image data S1055 to a predetermined compressive coding process to generate coded data S1056.
The input image memory 1005 temporarily stores the digital image data S1054 for the work of the compressive coding process. In the prior art image processing apparatus, the input image memory is divided into plural regions each region storing a predetermined quantity of digital image data. Here, the input image memory 1005 has two regions, namely, a first region 1005a and a second region 1005b, each region being able to store one frame (one screen) of digital image data.
The rate buffer 1006 temporarily stores the coded data S1056 generated by the coder 1004 and outputs the data so that the output S1057 from the image processing apparatus is output at a constant rate. Hereinafter, the operation of the prior art image processing apparatus so constructed will be described.
When an analog video signal S1051 is input to the image processing apparatus, the analog video signal S1051 is input to the A/D converter 1001, wherein it is subjected to analog/digital conversion. The A/D converter 1001 generates digital image data S1052 and outputs this data to the image input controller 1002. The input analog video signal S1051 includes a signal of an effective region corresponding to a portion of image to be displayed, and a signal of an ineffective region other than the effective region. The image input controller 1002 generates an image input enabling signal S1061 indicating whether the input digital image data S1052 is xe2x80x9ceffectivexe2x80x9d or xe2x80x9cineffectivexe2x80x9d, and outputs both of the digital image data S1053 and the image input enabling signal S1061 to the memory controller 1003.
The memory controller 1003 stores the digital image data S1053 in the input image memory 1005, according to the image input enabling signal S1061 supplied from the image input controller 1002 and an image input request signal S1063 supplied from the encoder 1004 which is described later. When the encoder 1004 goes into the coding executable state and outputs an image input request signal S1063 indicating a request for digital image data to be subjected to coding, to the memory controller 1003, the memory controller 1003 stores the digital image data S1053 in the first region 1005a of the input image memory according to the image input enabling signal S1061 indicating that the digital image data S1053 is effective.
When a predetermined amount of the digital image data S1053 is stored in the first region 1005a, the memory controller 1003 generates a coding start signal S1062 and outputs it to the encoder 1004 so that the encoder 1004 starts coding. Here, the memory controller 1003 generates the signal when one frame of digital image data has been stored.
The encoder 1004 does not perform coding until it receives the coding start signal S1062 directing coding, from the memory controller 1003. When the encoder 1004 has received this signal, it receives the digital image data S1055 stored in the first region 1005a through the memory controller 1003, and performs coding of this data. This coding is carried out according to a predetermined scheme. For example, one frame of digital image data is divided into plural blocks each having a predetermined size, and coding is carried out block by block. When this coding is carried out, the size of each block is generally 8xc3x978 pixels or 16xc3x9716 pixels. Further, xe2x80x9ca pixelxe2x80x9d is discrete unit data as a component of digital image data, and it has a pixel value showing the brightness or color of the image.
The encoder 1004 outputs coded data S1056 generated by the coding, to the rate buffer 1006. The coded data S1056, which has temporarily been stored in the rate buffer 1006, is output to the outside of the image processing apparatus as an output S1057 from the apparatus, for transmission or the like. On the other hand, as the coding is executed, the encoder 1004 generates an image input request signal S1063 indicating that one frame of digital image data to be coded next is to be input, and outputs this signal S1063 to the memory controller 1003.
In the memory controller 1003, the digital image data S1053 is stored in the input image memory 1005 according to the image input request signal S1063 and the input enabling signal S1061 indicating that the digital image data S1053 is effective. As described above, one frame of digital image data is stored in the memory 1005. However, at this time, the memory controller 1003 stores the data in the second region 1005b different from the first region 1005a. 
When one frame of digital image data S1053 (a unit of digital image data) has been stored in the second region 1005b, the memory controller 1003 generates a coding start signal S1062 indicating that coding should be started, and outputs this signal to the encoder 1004. If the encoder 1004 has ended coding of the previous one frame of digital image data (data which were stored in the region 1005a) when it receives the coding start signal S1062 which directs the encoder to start coding, from the memory controller 1003, the encoder 1004 receives the digital image data S1055 stored in the second region 1005b through the memory controller 1003, performs coding of this data, and outputs coded data to the rate buffer 1006.
As described above, in the prior art image processing apparatus, digital image data is stored alternately in the first region 1005a and the second region 1005b possessed by the input image memory 1005, and the stored data is read alternately from these regions to be coded by the encoder 1004.
FIG. 11 is a timing chart showing the processing status in the normal state wherein the above-mentioned processing is carried out normally. In FIG. 11, xe2x80x9cimage input request signal S1063xe2x80x9d indicates the state of the signal S1063 output from the encoder 1004 to the memory controller 1003, and its Hi state shows that the encoder 1004 requests digital image data. Further, xe2x80x9cimage input enabling signal S1061xe2x80x9d indicates the state of the signal S1061 which is generated by the image input controller 1002 to be output to the memory controller 1003, and its Hi state shows that the digital image data is effective and to be stored in the memory.
Furthermore, xe2x80x9cstorage of image dataxe2x80x9d in FIG. 11 indicates regions of the input image memory 1005 where the digital image data S1054 is stored. As described above, under control of the memory controller 1003, the digital image data is stored alternately in the first region 1005a (in the figure, memory (1)) and the second region 1005b (in the figure, memory (2) ), which regions are possessed by the input image memory 1005.
Turning to FIG. 11, xe2x80x9ccoding start signal S1062xe2x80x9d indicates the state of the signal S1062 output from the memory controller 1003 to the encoder 1004, and its Hi state shows that coding should be started. Further, xe2x80x9ccoding processxe2x80x9d indicates a coding process performed by the encoder 1004, and it indicates that the digital image data stored in the first region 1005a (in the figure, memory (1)) or the second region 1005b (in the figure, memory (2)) is being subjected to coding, which regions are possessed by the input image memory 1005.
As shown in the figure, in accordance with the Hi state of the image input request signal S1063, from timing t110, the digital image data whose image input enabling signal S1061 is in the Hi state is stored in the first region, as shown by xe2x80x9cimage data storagexe2x80x9d. Then, in accordance with the Hi state of the coding start signal S1062 shown in the figure, from timing t111, the digital image data is read from the first region to be coded. Further, as the coding is carried out, storage of data in the second region is carried out as shown by xe2x80x9cimage data storagexe2x80x9d. Likewise, from timing t112, storage of data in the first region and readout of data from the second region are carried out. As shown by xe2x80x9cimage data storagexe2x80x9d and xe2x80x9ccoding processxe2x80x9d in the figure, at the timing when storage of data in one of the regions is carried out, readout of data from the other region is carried out.
On the other hand, FIG. 12 is a timing chart showing the processing state where an error has occurred for some reason, and this error disables the normal processing shown in FIG. 11. Also in this case, the processing is carried out in like manner as shown in FIG. 11 until timing t120.
In FIG. 12, xe2x80x9ccoding processxe2x80x9d indicates that the coding of the digital image data stored in the second region 1005b (FIG. 1), which coding has been performed from timing t120, takes time and, therefore, the end of this coding is delayed to timing t121. Accordingly, with respect to the second region 1005b of the input image memory 1005, readout of the digital image data from this region is carried out until reaching timing t121, and thereby storage of digital image data into the second region shown by xe2x80x9cimage data storagexe2x80x9d is not performed although this storage ought to be carried out if the processing has been normally carried out. Therefore, as shown by the broken-line square of xe2x80x9cimage data storagexe2x80x9d in FIG. 12, the digital image data which has not been stored is discarded, i.e., it is not subjected to the coding process.
After the coding process has ended at timing t122, the normal processing is carried out again. As described above, in the prior art image processing apparatus, storage and readout are alternately performed in/from the regions possessed by the input image memory 1005. In this way, the prior art image processing apparatus is able to perform storage and coding of digital image data at their respective timings, and copes with a delay or the like in the coding process by discarding the digital image data. Also in the apparatus disclosed in the above-mentioned Japanese Patent Application No. 7-273461, memory management similar to that mentioned above is performed.
However, the conventional image processing apparatus has the following problems.
First of all, when the image processing apparatus is used in a visual telephone system or as a monitor between an input apparatus such as a video camera and an output apparatus such as a display, it is required to have the property of operating real-time. When the apparatus is applied to such use, the apparatus captures an image from a video camera or the like as a target to be processed and, after processing, coded data is output from the apparatus for transmission or the like. At the reproduction end, the coded data is subjected to decoding and then output as an image. So, if a delay from inputting analog image data to displaying output image data is considerable, a time difference occurs in motions between the real image taken by the camera and the displayed image, resulting in unnatural image display.
As described above, in the prior art image processing apparatus, in order to control data input/output to/from the input image memory, the encoder 1004 starts coding in response to the coding start signal S1062 when the digital image data to be subjected to coding has been stored by a predetermined amount (in the above example, one frame) (FIG. 11). Therefore, in the prior art image processing apparatus, because of a delay between the signal input and the start of coding, it is hard to satisfactorily display the image when the apparatus is applied to such use. This is the first problem.
Further, in the prior art image processing apparatus, as described above, unconditional discarding of digital image data is performed for error processing. Since this unconditional discarding causes absence of data, frequent delays in the coding process result in degradation of image quality. This is the second problem.
Moreover, since the input image memory 1005 is divided into the region where storage is performed and the region where readout is performed, when the amount of digital image data to be processed is large, the memory capacity must be increased according to the data amount. However, to require a bulk memory results in an increase in the cost and, therefore, it is difficult to fabricate an inexpensive apparatus for propagation. This is the third problem.
The present invention is made in view of the above-described circumstances and has an object to provide an image processing apparatus which reduces a delay time from start of signal input to start of coding and, therefore, is suitable for the real-time use.
Further, the present invention has another object to provide an image processing apparatus which reduces the amount of data to be discarded and thereby improves the image quality, even when a delay or the like occurs during coding.
Further, the present invention has still another object to provide an image processing apparatus which reduces the capacity of memory required for temporary storage of digital data and thereby reduces the cost.
Further, the present invention has yet another object to provide an image processing method which reduces a delay time before start of coding, an image processing method which reduces the possibility of discarding data even if coding is delayed, and an image processing method which reduces the capacity of memory required for temporary storage of data.
To attain the above-mentioned objects, an image processing apparatus of a first aspect of the present invention, which is an apparatus for storing input image data in a temporary storage device and subjecting the stored image data to a coding process, comprises: an image input control device for controlling storage of the input image data in the temporary storage device; a storage control device for executing storage of the image data in the temporary storage device under control of the image input control device and, when a predetermined unit storage amount of data has been stored, generating storage information indicating this; a coding device for reading the image data stored in the temporary storage device to subject the read data to a predetermined coding process and, when a predetermined unit processing amount of data has been subjected to the coding process, generating process information indicating this; and a control information generating device for generating first control information used by the image input control device to control the storage, and second control information used by the coding device to control the coding process, in accordance with the storage information generated by the storage control device and the process information generated by the coding device. Thereby, the control information generating device generates control information for controlling the storage and the coding process, according to the image processing status obtained from the storage information and the process information.
According to an image processing apparatus of another aspect of this invention, the control information generating device generates, as the first control information, storage stop information indicating that the storage of the input image data should be stopped, and generates, as the second control information, coding stop information indicating that the coding process should be stopped. According to the status of image processing, the storage is stopped to protect the data which have already been stored, and the coding is stopped to stand by until data to be coded are stored.
According to an image processing apparatus another aspect of this invention, the control information generating device generates, as the first control information, storage stop information indicating that the storage of the input image data should be stopped, and generates, as the second control information, continuous process information indicating how many times the coding device can continuously perform the coding process on the unit processing amount of image data. According to the status of image processing, the storage is stopped to protect the data which have already been stored, and continuous coding according to the status of storage is performed.
According to an image processing apparatus of another aspect of this invention, the control information generating device comprises: a storage information counting device for counting the storage information and holding the result as a storage information count value; a process information counting device for counting the process information and holding the result of the count as a process information count value; an addition control device for outputting an addition enabling signal when the count of the storage information is performed by a predetermined number of times, and outputting an addition disabling signal when the count of the process information is performed by a predetermined number of times; a storage information count value change device for adding a predetermined value to the storage information count value according to the addition enabling signal or the addition disabling signal, thereby generating a storage information count value after processing; a codable unit number generating device for subtracting the process information count value from the storage information count value after processing, thereby generating a codable unit number; a first control information generating device for comparing the codable unit number with a first predetermined value and, when these values match, generating the first control information; and a second control information generating device for comparing the codable unit number with a second predetermined value and, when these values match, generating the second control information. Thereby, the codable unit number indicating the storage status of data to be coded is obtained from the storage information and the process information, and the control information is generated according to the codable unit number to control the storage and the coding.
According to an image processing apparatus of another aspect of this invention, the control information generating device comprises: a storage information counting device for counting the storage information and holding the result as a storage information count value; a process information counting device for counting the process information and holding the result as a process information count value; an addition control device for outputting an addition enabling signal when the count of the storage information has been performed by a predetermined number of times, and outputting an addition disabling signal when the count of the process information has been performed by a predetermined number of times; a storage information count value change device for adding a predetermined value to the storage information count value according to the addition enabling signal or the addition disabling signal, thereby generating a storage information count value after processing; a codable unit number generating device for subtracting the process information count value from the storage information count value after processing, thereby generating a codable unit number; and a first control information generating device for comparing the codable unit number with a first predetermined value and, when these values match, generating the first control information; wherein the codable unit number is used as the second control information. Thereby, the codable unit number indicating the storage status of data to be coded is obtained from the storage information and the process information, and the control information is generated according to the codable unit number to control the storage and the continuous coding.
An image processing method according to another aspect of this invention, which is a method for storing input image data in a temporary storage device and performing coding of the stored image data, comprises: controlling storage of the input image data in the temporary storage device; executing storage of the image data in the temporary storage device under control of the image input control step and, when a predetermined unit storage amount of data has been stored, generating storage information indicating this; reading the image data stored in the temporary storage device to subject the read data to a predetermined coding process and, when a predetermined unit processing amount of data has been subjected to the coding process, generating process information indicating this; and generating first control information used in the controlling to control the storage, and second control information used in the coding to control the coding process, according to the storage information generated in the storage control and the process information generated in the coding. Thereby, in the control information generating, the control information for controlling the storage and the coding process is generated according to the image processing status obtained from the storage information and the process information.
According to an image processing method of another aspect of this invention, in the control information generating, storage stop information indicating that the storage of the input image data should be stopped is generated as the first control information, and coding stop information indicating that the coding process should be stopped is generated as the second control information. According to the status of image processing, the storage is stopped to protect the data which have already been stored, and the coding is stopped to stand by until data to be coded are stored.
According to an image processing method of another aspect of this invention, in the control information generating, storage stop information indicating that the storage of the input image data should be stopped is generated as the first control information, and continuous process information indicating how many times the coding process on the unit processing amount of image data can be continuously performed in the coding is generated as the second control information. According to the status of image processing, the storage is stopped to protect the data which have already been stored, and continuous coding according to the status of storage is performed.
According to an image processing method of yet another aspect of this invention, the control information generating comprises: counting the storage information and holding the result as a storage information count value; counting the process information and holding the result as a process information count value; outputting an addition enabling signal when the count of the storage information is performed by a predetermined number of times, and outputting an addition disabling signal when the count of the process information is performed by a predetermined number of times; adding a predetermined value to the storage information count value according to the addition enabling signal or the addition disabling signal, thereby generating a storage information count value after processing; subtracting the process information count value from the storage information count value after processing, thereby generating a codable unit number; comparing the codable unit number with a first predetermined value and, when these values match, generating the first control information; and comparing the codable unit number with a second predetermined value and, when these values match, generating the second control information. Thereby, the codable unit number indicating the storage status of data to be coded is obtained from the storage information and the process information, and the control information is generated according to the codable unit number to control the storage and the coding.
According to an image processing method of yet another aspect of this invention, the control information generating comprises: counting the storage information and holding the result as a storage information count value; counting the process information and holding the result as a process information count value; outputting an addition enabling signal when the count of the storage information is performed by a predetermined number of times, and outputting an addition disabling signal when the count of the process information is performed by a predetermined number of times; adding a predetermined value to the storage information count value according to the addition enabling signal or the addition disabling signal, thereby generating a storage information count value after processing; subtracting the process information count value from the storage information count value after processing, thereby generating a codable unit number; comparing the codable unit number with a first predetermined value and, when these values match, generating the first control information; and using the codable unit number as the second control information. Thereby, the codable unit number indicating the storage status of data to be coded is obtained from the storage information and the process information, and the control information is generated according to the codable unit number to control the storage and the continuous coding.
USPTO Backgrounds
USPTO Backgrounds