Conventional image sensors, such as CMOS and CCD sensors, integrate all light that impinges on them during the exposure time. This provides sharp images of static objects, but results in spatial blur for objects that move while the shutter is open. Objects that are not in focus, i.e. not positioned in the focal plane are also blurred. The so-called motion blur is proportional to the exposure time and object velocity. Motion blur is particularly troublesome when a camera operates under low light level conditions. Under such circumstances, long exposure times are desired to attain sufficiently high signal-to-noise levels such that the dark areas of a scene can be imaged adequately. Consequently, many cameras suffer from a classic trade-off between motion blur and dynamic range. The exposure times need to be long to capture enough light, but need to be small so as to reduce motion blur. Within the framework of the invention the word camera comprises cameras for taking photographs as well as cameras for video purposes.
A camera and method to reduce blur of objects that are not positioned in the focal plane are known from an article by Nagahara et al “Flexible Depth of Field Photography”, H. Nagahara, S. Kuthirummal, C. Zhou, and S. K. Nayar, European Conference on Computer Vision (ECCV), October, 2008.
In Nagahara et al a camera for taking photographs is shown in which the distance between the sensor and a fixed focus lens is varied. The sensor is swept over a distance during the exposure time. The sweeping distance is arranged to sweep a range of scene depth ranges in order to increase the depth of field. The prior art camera disclosed in Nagahara et al reduces out-of-focus blur. To reduce the out-of-focus blur the sensor is swept along the optical axis to cover certain depth ranges. This concept is also called the ‘sweeping focus camera’.
The sweeping of the focus provides for a compound image, in effect being a combination of a number of images at various focal depths. A point spread function (PSF) characterizing the blur caused by the sweep through various focal positions can be calculated. A point spread function is, in effect, the image a point of an object would make on the sensor. For an object completely in focus the point spread would be zero, and thus the PSF would be a Dirac function. The Fourier transform of this function would be a constant for all frequencies. For a point not in focus the PSF is a spread-out function, for an object in motion while the camera is fixed, the PSF would be spread out over a distance due to the motion. From the PSF one can calculate an inverse point spread function (IPSF). Deconvoluting the compound image with the inverse point spread function allows a sharp image to be obtained and an increased depth of field is obtained. This is due to the fact that, when the sensor is swept, the PSF for static objects at various distances becomes to a considerable degree the same. Thus, deconvolution the original image with one and the same IPSF would allow a sharp image at all distances, or at least an increased range of distance and the an increased depth of field is obtained for static objects.
Although out-of-focus blur and the reduction thereof may be and is important, a major problem, as explained above, exists and remains for moving objects, namely the motion blur, especially for larger exposure times.
Motion blur can be inverted by means of video processing. This is achieved by motion estimation and inverse filtering along the motion trajectory. This is known for instance from U.S. Pat. No. 6,930,676. In practice, however, the results of such a procedure suffer from inaccurate motion vectors, particularly for occlusion areas. One has to know the motion trajectory and deduce motion vectors from them to be able to do the inverse filtering. In many stand-alone cameras used in professional applications, motion vectors may not be available at all. For example, the recordings of many cameras used for surveillance or activity monitoring merely provide input to computer-vision-based analysis procedures (e.g., automatic detection of suspicious objects, fall-detection for elderly, etc). In these scenarios, the quality of the raw input frames is a determining factor for the performance of the detection system. Sufficiently accurate motion vectors may not be available on-the-fly within the camera and post-processing of recorded video is not an option in real-time monitoring systems. For a camera that takes a single snapshot it is fundamentally impossible to accurately determine motion vectors. At occlusion areas estimation of motion is also extremely difficult and inaccurate, if at all possible. At low light conditions the problems increase, due to the lack of light. Another method of getting rid of motion blur is to have the camera follow the moving object. However, this also has a number of significant disadvantages. Although the moving object is not blurred, everything else is. Furthermore, one has to know the direction as well as the speed of the object to accomplish such facts. This methods is thus only possible in situation where a fairly accurate determination of the speed and direction of movement can be made, for instance with formula I races or a ski jump, where the direction of motion is fairly accurately known in advance as well as an relatively accurate estimate of the speed may be made in advance.
Second, most traditional cameras feature an adjustable shutter and aperture that windows the light coming through the lens in the temporal and spatial dimensions. These can typically be characterized as box filters (i.e. a constant sensitivity over a finite interval), corresponding to a sinc modulation in the corresponding temporal and spatial frequency domains. As a result, some high frequencies are fully suppressed during acquisition and cannot be recovered during inverse FIR filtering even when perfect motion information would be available. In practice, inverse filtering should be done with utmost care to prevent the amplification of noise and the introduction of artefacts.
In International Patent Application WO 2010/131142 a system is described in which a sweeping focus set-up is used to accomplish motion invariant imaging. This is achieved by sweeping the focus fast by moving the sensor or lens, of changing the focus of the lens.
Since a sweeping focus is used the acquired image is sharp throughout the focal sweep range, background and foreground are sharp. To the human eye such an all-in-focus image often looks unnatural.
In short, various known ways for reducing motion blur in an image have their shortcomings.
USPTO Backgrounds
USPTO Backgrounds