In recent years, image shooting apparatuses such as digital still cameras and digital video cameras increasingly include an imaging sensor having multiple pixels in order to shoot a high-definition still image. However, shooting an image at a high frame rate by using the imaging sensor such as continuous shooting and moving picture shooting increases an amount of data read from the imaging sensor per unit time. More specifically, the amount of data read from the imaging sensor per unit time is determined by a multiplication between the number of pixels in a frame and a frame rate. Accordingly, an imaging sensor in which an amount of data read per unit time is limited makes it difficult to perform image shooting at a high frame rate while reading data of all the pixels.
On the other hand, Patent Reference 1 discloses a technique of achieving a high frame rate while reducing an amount of data read per unit time by thinning pixels to be read from the imaging sensor. Here, a frame indicates a still image at a given time.
Further, Patent Reference 2 discloses an example of a super-resolution technique of generating a high-resolution frame from thinned low-resolution frames.
The following will describe a structure of an image processing device 100 which is structured by combing the above references and an image processing method.
FIG. 1 is a block diagram showing the structure of the image processing device 100. The image processing device 100 shown in FIG. 1 includes: a reading scheme switching unit 101 which receives a shooting mode signal 120 as an input, and outputs a reading scheme indicating signal 121; an imaging unit 102 which receives the reading scheme indicating signal 121 as an input, and outputs an image signal 122; and a super-resolution unit 103 which receives the image signal 122 as an input, and outputs an output image signal 129.
When the shooting mode signal 120 indicates that a current shooting mode is a mode for reading an image at high speed (hereinafter, referred to as high-speed reading mode), the reading scheme switching unit 101 outputs the reading scheme indicating signal 121 indicating thinned-pixels reading to the imaging unit 102. In addition, the reading scheme switching unit 101 outputs the reading scheme indicating signal 121 indicating all-pixels reading to the imaging unit 102 in a case (hereinafter, referred to as regular reading mode) other than the above case.
Here, the high-speed reading mode is, for example, a high-speed continuous shooting mode in which, for example, 8 frames are shot per second, and also a high-frame rate moving picture shooting mode in which, for instance, 120 frames are shot per second. The regular reading mode is, for instance, a regular shooting mode in which continuous shooting is not performed, a low-speed continuous shooting mode in which, for example, 2 frames are shot per second, and a regular moving picture shooting mode in which, for instance, 30 frames are shot per second.
The imaging unit 102 includes an image sensor such as a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor) image sensor. When the reading scheme indicating signal 121 indicates the all-pixels reading, the imaging unit 102 outputs, as the image signal 122, a signal corresponding to all the pixels included in a frame. In addition, when the reading scheme indicating signal 121 indicates the thinned-pixels reading, the imaging unit 102 outputs, as the image signal 122, a signal corresponding to thinned pixels whose number is smaller than the number of all the pixels included in the frame.
When the reading scheme indicating signal 121 indicates the all-pixels reading, the super-resolution unit 103 directly outputs the image signal 122 as the output image signal 129. In addition, when the reading scheme indicating signal 121 indicates the thinned-pixels reading, the super-resolution unit 103 generates a high-resolution frame using temporally continuous frames included in the thinned-pixels read and low-resolution image signal 122, by performing, for instance, a super-resolution process described in Patent Reference 2, and outputs the generated high-resolution frame as the output image signal 129.
The following will describe an internal structure and operations of the super-resolution unit 103 with reference to FIG. 2. FIG. 2 is a block diagram showing the structure of the super-resolution unit 103.
The super-resolution unit 103 shown in FIG. 2 includes: a frame memory 144 which receives an image signal 122 as an input, and outputs a stored image signal 131; a motion detecting unit 143 which receives the image signal 122 and the stored image signal 131 as inputs, and outputs a motion information signal 132; a super-resolution performing unit 142 which receives the image signal 122 and the stored image signal 131 as inputs, and outputs a super-resolution image signal 133; and a signal switching unit 141 which receives a reading scheme indicating signal 121, the image signal 122, and the super-resolution image signal 133 as inputs, and outputs an output image signal 129.
A frame at the time of reading all the pixels or thinned pixels which is included in the image signal 122 is temporarily stored into the frame memory 144, and the frame memory 144 outputs the stored frame as the stored image signal 131.
The motion detecting unit 143 detects a motion between a current frame of the image signal 122 and a previously stored frame of the stored image signal 131, and outputs the motion information signal 132 indicating an amount of motion of one or more objects within the current frame.
The super-resolution performing unit 142 outputs the high-resolution super-resolution image signal 133 by performing a super-resolution process on the thinned-pixels read and low-resolution image signal 122 using the motion information signal 132.
When the reading scheme indicating signal 121 indicates the all-pixels reading, the signal switching unit 141 outputs the image signal 122 as the output image signal 129. Furthermore, when the reading scheme indicating signal 121 indicates the thinned-pixels reading, the signal switching unit 141 outputs the super-resolution image signal 133 as the output image signal 129.
Next, the following will describe a specific operation example of the image processing device 100 with reference to FIGS. 3 and 4. FIG. 3 is a diagram showing operations of the image processing device 100 in a regular reading mode. FIG. 4 is a diagram showing operations of the image processing device 100 in a high-speed reading mode.
A shooting performing signal 124 is a signal which indicates value 1 during a shooting period, and which indicates value 0 in a case other than the above. The reading scheme indicating signal 121 is a signal which indicates value 1 in the case of the all-pixels reading, and which indicates value 0 in a case other than the above. The shooting mode signal 120 is a signal which indicates value 1 in the high-speed reading mode, and which indicates value 0 in the regular reading mode.
First, the operations of the image processing device 100 when the shooting mode signal 120 indicates value 0, that is, in the regular reading mode will be described. As shown in FIG. 3, when the shooting mode signal 120 indicates value 0, the reading scheme switching unit 101 outputs the reading scheme indicating signal 121 indicating value 0. The imaging unit 102 performs the all-pixels reading and at the same time spends time Tf outputting a frame as the image signal 122. Here, because a transfer time of the frame is Tf, an interval to a start of next frame transfer needs to be Tf at minimum.
Next, the operations of the image processing device 100 when the shooting mode signal 120 indicates value 1, that is, in the high-speed reading mode will be described. As shown in FIG. 4, when the shooting mode signal 120 indicates value 1, the reading scheme switching unit 101 outputs the reading scheme indicating signal 121 indicating value 1. The imaging unit 102 performs the thinned-pixels reading and at the same time spends time Tp outputting a frame as the image signal 122. Here, although image quality deteriorates in comparison with an all-pixels read image because the thinned-pixels reading is performed in the high-speed reading mode, due to Tp<Tf, it is possible to shoot an image having a higher frame rate in comparison with the case where the shooting mode signal 120 indicates value 0.
Next, operations of the image processing device 100 will be described with reference to FIG. 5. FIG. 5 is a flow chart of an image processing method of the image processing device 100.
First, the reading scheme switching unit 101 determines whether or not a current shooting mode is the high-speed reading mode (Step S1). When the current shooting mode is the high-speed reading mode (Yes in S1), the reading scheme switching unit 101 indicates the thinned-pixels reading to the imaging unit 102 (Step S5), and when the current shooting mode is the regular reading mode (No in S1), the reading scheme switching unit 101 indicates the all-pixels reading to the imaging unit 102 (Step S2).
When the all-pixels reading is indicated, the imaging unit 102 reads a signal of all the pixels included in a frame, and outputs the signal as the image signal 122 (Step S3). Next, the super-resolution unit 103 directly outputs the read image signal 122 as the output image signal 129. (Step S4).
On the other hand, when the thinned-pixels reading is indicated at Step S5, the imaging unit 102 reads a signal of thinned pixels whose number is smaller than all the pixels included in the frame (Step S6). Next, the super-resolution unit 103 generates a high-resolution frame using temporally continuous frames included in the thinned-pixels read and low-resolution image signal 122, by performing, for instance, a super-resolution process described in Patent Reference 2, and outputs the generated high-resolution frame as the output image signal 129 (Step S7).
FIG. 6 is a diagram showing an example of a conventional super-resolution process. As shown in FIG. 6, for instance, a frame of a high-resolution image is generated using temporally continuous three frames of low-resolution images.    Patent Reference 1: Japanese Unexamined Patent Application Publication No. 2005-109968    Patent Reference 2: Japanese Unexamined Patent Application Publication No. 2000-244851
USPTO Backgrounds
USPTO Backgrounds