The prevalence of consumer electronic devices, such as computers, smartphones, tablets, wearable devices, etc., continues to increase. Such devices often include a display, such as a high-resolution display, and/or an image sensor (e.g., a sensor included in a camera). Image signal processing can be performed in such devices (e.g., to process captured images and/or images stored in electronic memory) using an image signal processor (ISP). Such ISPs can operate, in some implementations, in accordance with a corresponding directed acyclic graph (DAG). For purposes of this disclosure, the term “ISP” will be used to describe such apparatus that are used to implement image signal processing, though other terms may also be appropriate.
Image signal processing (e.g., processing of pixel data corresponding with an image or set of images) by such ISPs can be used in a number of situations. For instance, an image (or series of images, e.g., video) captured by an image sensor (e.g., camera) included in an electronic device can be processed using an ISP, with the fully processed image (or images) being presented on a display of the device and/or stored in electronic memory (e.g., flash memory, dynamic random access memory (DRAM), and so forth) of the electronic device (or operatively coupled with the electronic device). In other instances, for example, an image (or series of images, e.g., video) stored in memory can be processed in an ISP, wherein the fully processed image (or images) (output at the last stage of the ISP) is(are) presented on a display of an associated electronic device and/or is(are) stored in the same memory, or a different memory. The origin and destination of images that are processed by a given ISP can vary based on the particular implementation.
Implementations of ISPs can include a series of cascaded (e.g., pipelined) linebuffers (e.g., static RAM (SRAM) for buffering (storing) image (pixel) data during image signal processing) and associated compute kernels, (e.g., image processing logic for performing image processing functions on the image pixel data). Such image processing functions can include functions that are performed (e.g., in a given sequence) on “stencils” of pixel data corresponding to sub-groups of spatially proximate pixels of a given image or series of images. For instance, such image processing functions can include color interpolation, image sharpening, color correction, and so forth.
Linebuffers, which, as noted above, can be implemented using SRAM memory, can provide local buffering for image pixel data between image processing logic (IPL) stages of an ISP. Pixel data processed by one stage can be written into (buffered for reuse in) a linebuffer for a subsequent processing stage, and so forth, until all processing stages have completed processing of the image (pixel) data). Often, bigger linebuffers (linebuffers that can hold more lines) can be used to facilitate increased throughput via parallelism.
As advances are made in consumer electronic devices, corresponding increases in image resolution, increases in a number of image processing stages (e.g., to facilitate more complex image processing algorithms) and/or requirements for improving image processing throughput can result in an undesirable increase in an amount of memory (e.g., SRAM) used to implement linebuffers in a corresponding ISP. For example, in order to achieve desired performance for a given ISP, an amount of memory used to implement linebuffers for that ISP can increase to an amount that is prohibitive from a cost perspective, (e.g., an amount of silicon area used, product design cost, etc.), and/or a power consumption (e.g., dynamic and/or static power consumption) perspective.
USPTO Backgrounds
USPTO Backgrounds