The present disclosure relates to an image pickup apparatus, an image pickup apparatus control method, and a program. In more detail, the present disclosure relates to an image pickup apparatus, an image pickup apparatus control method, and a program that carry out exposure control in region units.
An example of the configuration and image pickup process of a CMOS image sensor as one example of an image sensor is described below with reference to FIG. 1. FIG. 1 is a diagram showing the partial configuration of an image sensor (CMOS image sensor) 101. The image sensor (CMOS image sensor) 101 includes a vertical scan circuit 102, a horizontal scan circuit 103, and a plurality of pixels 104 disposed in an array.
When an exposure process is carried out to pick up an image of a subject, charge accumulates in a photodiode inside each of the pixels 104. The charge accumulated in the photodiode in each pixel is outputted via an amp transistor and a transfer transistor to a vertical signal wire 113. The signal current outputted to the vertical signal wire 113 is also supplied to the horizontal scan circuit 103 and, after being subjected to specified signal processing, is outputted via a signal output wire 114 to the periphery.
Since each vertical signal wire 113 is commonly connected to pixels that are aligned in the vertical direction, to read out the signals of the individual pixels, it is necessary to output the signal of only one pixel at a time to the vertical signal wire 113.
With the image sensor (CMOS image sensor) 101, signals can be read out separately from the individual pixels for example by first reading out signals from the respective pixels 104d aligned on the bottom line as shown in FIG. 2A, then reading out signals from a line of pixels 104c as shown in FIG. 2B, and after that reading out signals while successively changing the line being read out. As one example, the control signals for such pixel reads are outputted from horizontal reset wires 111 and horizontal select wires 112 connected to the vertical scan circuit 102 shown in FIG. 1.
Immediately after a read process for the accumulated charge, the individual pixels 104 that construct the image sensor (CMOS image sensor) 101 again start to be exposed. That is, the exposure process for the next image frame starts.
In this way, when the read out process is carried out successively in line units and the exposure process starts immediately afterwards, the exposure start time and end time will differ, that is, a difference in exposure time (or “exposure period”) will be produced, between the photodiodes 104a on the first line and the photodiodes 104d on the last line. This is a characteristic phenomenon of a shutter operation referred to as a “focal plane shutter” or a “rolling shutter”. Note that although only the four lines 104a to 104d are shown in the drawings, this is because only part of an image sensor is illustrated and an actual image sensor will have many more lines, such as several hundred to several thousand lines, with signals being successively read out in line units.
Existing image sensors typically use a configuration where the exposure time, that is, the time from the start to the end of exposure is the same for every pixel.
However, in recent years, a configuration where the exposure time is controlled for individual pixels on an image sensor in accordance with the brightness of the subject has been proposed. By carrying out such exposure control, it is possible to increase the dynamic range of the picked-up images.
A process that expands the dynamic range through control over exposure time (hereinafter referred to as “shutter control”) will now be described. If the exposure time is increased in a bright subject region, pixels become saturated with the accumulated charge, which stops accurate pixel values from being obtained. Meanwhile, if the exposure time is increased in a dark subject region, it becomes easier to obtain accurate pixel values that correspond to the luminance of the subject.
For this reason, in regions where the subject is bright, pixel values of pixels where the exposure time is reduced are acquired as effective pixel values. Meanwhile, in regions where the subject is dark, pixel values of pixels where the exposure time is increased are acquired as effective pixels. One method of expanding the dynamic range is to generate an output image by combining regions with reduced and increased exposure times. Note that when outputting the final pixel values, a pixel value adjusting process is carried out based on the respective exposure times.
In Japanese Laid-Open Patent Publication No. 2010-136205 and Jenwei Gu et al., “Coded Rolling Shutter Photography: Flexible Space-Time Sampling”, Computational Photography (ICCP), 2010, a technique for expanding the dynamic range of picked-up images by setting different exposure times for each line of pixels on an image sensor is disclosed. One example is a configuration that sets a short exposure line and a long exposure line alternately for the lines of pixels on an image sensor
For example, Publication No. 2010-136205 and the Jenwei Gu et al. article disclose a method of carrying out exposure time control in line units to set the exposure time in line units in keeping with the brightness of pixels on a screen. The configuration in the Publication No. 2010-136205 and the Jenwei Gu et al. article carries out exposure time control in units of image regions with a configuration that uses an electronic shutter.
An algorithm that adaptively sets the exposure time in keeping with the luminance of the subject has been described for example in Shuji Shimizu, et al., “A New Algorithm for Exposure Control Based on Fuzzy Logic for Video Cameras”, IEEE Transactions on Consumer Electronics, Vol. 38-3, (1992). In this way, configurations for controlling the exposure time in units of regions, such as lines, in keeping with the brightness of a subject have already been disclosed. However, to carry out exposure control in keeping with subject luminance, it is necessary to decide the exposure time in pixel units before the start of image pickup. Here, there is a problem in that a delay is produced between the following two processes.
Process 1: a process of acquiring a preceding image to be used in a process that decides the exposure time in pixel units, and
Process 2: subsequent to Process 1, an image pickup process that carries out exposure control that reflects the information produced by Process 1.
The longer the delay mentioned above, the greater the difficulty in tracking changes in brightness. As a result, there is the problem that images that are too bright or too dark will be acquired for some time after a change in brightness. In particular, if exposure control is carried out for respective regions such as line units or pixel units, compared to when exposure control is carried out over the entire screen as in the past, it becomes necessary to carry out more severe exposure control. When a single exposure time is decided for the entire screen as in the past, local movement of objects cause very little change in brightness. However, when exposure control is carried out on a region-by-region basis, it is necessary to track such local movement of objects, with the delay in such tracking having a larger effect than before.
The delay in reflecting movement in exposure control will now be described with reference to FIGS. 3 and 4. In FIGS. 3 and 4, time is shown on the horizontal axis and lines of pixels are shown on the vertical axis. As one example, in FIG. 3, the charge read timing shifts in line units as shown by the dotted lines 151a, 151b in the drawing.
The plurality of rectangular blocks shown in FIG. 3 show the exposure time of a single picked-up image frame, and represent the exposure time of line block units that are each composed of a line or a plurality of lines. The exposure process starts immediately after the timing shown by the read line 151a for the preceding picked-up image frame. As shown by the read line 151a, the exposure start time gradually shifts in line units. As shown by the graph in FIG. 3, exposure begins earlier for upper lines but only begins after an increasing delay for lines toward the bottom. For the line at the very top, the time (t1), is the exposure start time, while for the line at the very bottom, the time (t2) is the exposure start time.
The right ends of the plurality of rectangular blocks shown in FIG. 3 are the timing at which a read out process is carried out for the exposed image, with the accumulated charge of the pixels on each line being read at the timing shown by the read line 151b. In this example, since the exposure end time is approximately equal to the read out process time, as shown by the read line 151b in FIG. 3, the read out process is successively carried out for the pixels on each line starting from the first line.
For the line at the very top, the time (t2) is the exposure end time, while for the line at the very bottom, the time (t3) is the exposure end time. Note that in this example, since the exposure start and exposure end of the respective lines shift by the same amount in line units, the exposure time for every line is the same.
In FIG. 4, the exposure process and read out timing corresponding to three consecutive image pickup frames (numbered N to N+2) during image pickup of video images is shown. After read out of the final line in frame N ends, read out of the first line in frame N+1 begins. Acquisition of the entire image of frame N becomes possible at the read out timing of the final line. However, since the exposure of frame N+1 will have already started at such time, it is not possible for exposure control of frame N+1 to reflect the image acquired for frame N, and such control only becomes possible for frame N+2 onwards (i.e., a delay of two frames).
However, since the interval from the read out of the final line in frame N to the start of exposure of the first line in frame N+2 is short, if the calculation of exposure time and communication of control data take some time, it may not be possible to reflect frame N in the exposure control of frame N+2, resulting in an even longer delay.
In this way, if exposure control is carried out in accordance with the brightness of a subject, a delay is produced between the following two processes.
Process 1: a process of acquiring a preceding image to be used in a process that decides the exposure time in pixel units, and
Process 2: subsequent to Process 1, an image pickup process that carries out exposure control that reflects the information produced by Process 1.
Due to such delay, there is the problem that optimal exposure control may not be possible.
USPTO Backgrounds
USPTO Backgrounds