1. Field of the Invention
The present invention relates to both video and film production systems and methods and particularly to such systems and methods in which image combination and processing is to be achieved.
2. Description of the Prior Art
Video production systems are well known in which video information stored on conventional storage devices, such as analog video tape recorders, is processed in order to edit or treat the information so as to enhance the prerecorded information in the manner desired by the user. Such prior art video production systems known to applicant, however, are limited in that the prerecorded information can only be treated on a frame by frame basis in which the entire video frame of information must be treated. Thus, for example, if it is desired to change the coloration of a given video frame, the coloration of the entire frame must be modified. Thus, if it is desired to increase the red contained in a particular portion of an image of the frame, the entire frame must experience the red color change which increases the coloration of each of the portions of the entire video image by the inclusion of the additional red color change.
This has generally proven unsatisfactory and accordingly, matte techniques have been applied so as to attempt to limit the coloration to only the portions of the image desired. These matte techniques, however, are also limited, such as where they are employed to modify video images in real time in which a plurality of changes are desired in the frame. In addition, in applying such prior art matte techniques, as the portion of the image which is being treated moves in real time the matte or mattes employed must correspondingly move. This creates limitations on this technique when it is desired to process a rapidly changing scene as well as under other conditions. Accordingly, although such matte techniques are satisfactory for certain applications, they have limitations in other areas which have prevented the wide spread acceptance of such an approach as a viable method of specific color correction for portions of a video frame. In addition, another limitation on such prior art matte techniques is that if the video image being treated has a complex shape, considerable difficulty occurs in generating the matte to be used in treating such an image. For the above reasons, as well as others, prior art matte techniques have generally not enjoyed wide scale popularity in video production but rather have been limited to film production and optical printing.
It has also been known in the prior art to digitize video information. However, generally these techniques have not been applied to video production on a wide scale. An example of a digital video production system is the Ampex ESS-2; however, such a prior art digital production system does not employ real time interaction and, moreover, treats the composite video signal as an entity rather than separately processing its various color video components. This prior art digital video production system, however, is strictly a recording and playback device and does not enable pixel by pixel interaction with the video images. Thus, the limitations previously described in the prior art are not resolved by this type of system.
Another type of prior art digital image processing system is the type commercially available from Comtal Corporation of Pasadena, Calif. and distributed under the designation Vision ONE/20. This system is a real time interactive modular system which provides stand alone full feature image processing as well as the option of interfacing to numerous host computers. However, the prior art Comtal system is an image analysis system as well as an image enhancement system and has not been employed in video production. Accordingly, such system is not designed to handle real time interactive video production. For example, the information processed is processed in a 1:1 aspect ratio of previously digitized images, such as provided from a host computer, although the Comtal Vision ONE/20 does have a video digitizer option in which black and white video information may be provided to the system for processing in a 1:1 aspect ratio format, such as for graphic image treatment. In addition, the Comtal processor, since it provides processing on a 1:1 aspect ratio, treats the digital image on a 256 by 256, 512 by 512, or 1024 by 1024 active pixel basis. Thus, although the commercially available Comtal system is satisfactory for many applications it is not satisfactory for real time interactive video production at the video frame rate.
In addition to the above, it is well known in the prior art to modify color video information on a frame by frame basis, such as disclosed in U.S. Pat. Nos. 4,096,523 or Re. 29,787, although these prior art systems do not accomplish such modification on a pixel by pixel basis. There are also prior art image analysis systems in which a color video signal is digitized for image analysis such as disclosed in U.S. Pat. Nos. 3,835,245 and 3,739,078. Other prior art color video treatment systems known to applicant which enable treatment of a color video signal, although not on an active pixel by pixel basis and not enabling real time interactive control of such information treatment on a pixel by pixel basis, are disclosed, by way of example, in U.S. Pat. Nos. 4,017,680; 3,617,636; 3,904,816; 4,026,555; 4,000,510; 3,512,094; 4,060,829; 3,997,912; 3,899,662; 4,122,489; 3,949,416 and 3,969,757.
Thus, although the digitizing of video data for video image treatment is known in the prior art, as is the color modification of video information on a frame by frame basis, and the dot by dot control of static color separation images, other than my prior system described in my aforementioned copending U.S. patent application, there are no prior art systems known to applicant in which real time interactive video production treatment of video images provided at a real time video frame rate is accomplished on an active pixel by pixel basis, nor are there any prior art systems known to applicant in which a single frame of video may be interacted with in real time on a pixel by pixel basis.
Although the interactive video production system described in my copending U.S. patent application Ser. No. 39,184, overcomes many of the disadvantages of the prior art, it does not readily lend itself to simultaneous processing of multiple images on a pixel by pixel basis to produce a composite image in that my prior system employs NTSC encoding and decoding of the R, G, B component signals prior to the recording of the final processed composite image. Thus, although two component images, for example, can be processed in my prior system, to provide a final processed composite image, the first component image must be separately processed, encoded and stored before the second component image can be processed. Thereafter, the processed first component image must then be decoded, and merged with the processed second component image in memory to form the final processed composite image. This procedure can become both cumbersome and time consuming and the decoding and encoding process can possibly degrade the component images. The system of my present invention overcomes these problems by enabling the pixel by pixel simultaneous processing and storage of such multiple images.
In addition, although my prior system accomplishes foreground/background combination on a pixel-by-pixel basis, this is accomplished by image extraction on a frozen frame in refresh memory, as described in my aforementioned copending U.S. patent application, and not on a dynamic real time basis wherein each pixel has its own digital control mask which can be dynamically varied in real time to control the foreground/background image combination ratio for the pixel. Whereas, my prior system is an improvement over conventional chroma key techniques, the system of the present invention is a still further improvement providing in effect what may be termed interactive chroma keying. Thus, wherein prior art chroma key techniques require a predetermined set up of the images to enable proper chroma keying, no previous set up is required in the system of the present invention, and "chroma keying" can be accomplished on a pixel by pixel basis subsequent to the shooting of the original images to be combined in accordance with subsequently made decisions. This enables very subtle changes to be made. Moreover, the combining scheme or format may be based on any desired parameter such as color difference, luminance differences or user defined regions.
Film processing systems in which multiple images are to be combined to form a composite frame or final print are also well known in the art. Such prior art systems are generally known as optical printers, such as commercially available from Oxbury, by way of example. These prior art film optical printing systems are susceptible to several important disadvantages some of which are that these systems are not interactive during the optical printing process requiring all of the component mattes, if employed, and their associated images to have been previously correct; great care must be taken to avoid as much as possible the presence of the matte borders in the composite or final print; there is a finite limit on the number of images which can be combined because of film characteristics and image degradation problems; and the composite image being processed can not be viewed until the process has been completed and the final print developed requiring the entire process to be repeated if any errors are then noted. Moreover, although such prior art optical printing systems can do color correction, it must be done on a matte or frame basis and cannot be done on a pixel by pixel basis nor can it be monitored and changed from a preestablished scheme during the optical printing process. Several of the above disadvantages are directly related to the physical properties of the film media itself and to the photochemistry employed, such as for example in utilizing the photochemical process to separate foreground image from background in the "blue screen" process used in forming the mattes to be employed in making the final composite image.
These disadvantages of the prior art are overcome by the system and method of the present invention.
USPTO Backgrounds
USPTO Backgrounds