Image Creation for Non-Artists (OpenCV Project Walkthrough)
Image Creation for Non-Artists (OpenCV Project Walkthrough)

An attempt at exploring how computer vision and image processing techniques can allow non-artists to create Art

The end result

This project stemmed from my predilection of the visual arts — as a computing student, I’ve always envied art students in their studios with splodges of colors and pristine white canvases. However, when I’m faced with a blank canvas and a (few) bottle(s) of paint, I don’t know where to start. Without the hours of training Art students have, I simply can’t go very far unless I’m following some Bob Ross tutorial. That’s why I’ve decided to explore and see if I can code something that will enable me to create art “automatically”.

I did this project a looong time ago for my final year project when I wanted to create some software program that can allow its users to sketch something stickman-like on a pre-selected background, select the category of what was drawn, and the programme will intelligently figure out which image to poisson-blend in. After that, the user can also select a neural transfer style to be applied on the image.

So in this article I will be giving a brief walkthrough of this project. Because there are many parts to it, I won’t be going in-depth with the technical implementations. However, if you’d like me to do so, leave a comment and let me know! :)

Part 1 — Selection of the background image

Because I wanted the user to have a range of images to choose from, I’ve coded a python script that acted as an image search engine. If an image is selected as the user browses through a range of available background images, similar looking background images would be retrieved. The similarity is calculated based on colour histograms in the HSV colour space. A more detailed explanation can be found here.

Here’s the result. If a user selects this background image:

Similar looking images would be retrieved:

Part 2 — Start drawing!

This is the fun part! The program seeks to match whatever you’re drawing with the set of images in the database. So if you draw an image with the wings spread out versus if you draw an oval (for example) the retrieved results would be different. Additionally, because the programme is not as robust, the user would have to specify the category (for example bird or boat, or car). For this particular project, I have six categories. The drawing function is also coded in Python with openCV. Once the outline is drawn, the list of coordinates would be saved. 100 sample points (the number I used for my testing, this can vary — the more points the more accurate but the programme would take longer) was taken from this outline of the drawing.

The drawing did not have to be this specific, it could be just a stickman bird figure too (or stickbird lmao)

Part 3 — Comparing the sample points with pre-processed image descriptors

Initially, I spent a very long time trying to figure out how to use image segmentation methods to get a clean cut of the target image objects so I could crop them out of their original images, figure out which one(s) matched my sketch the most, and them poisson-blend into my result image. However, this was really tricky to do and OpenCV’s image saliency techniques only worked well with certain types of images. For example,

For these images, clean masks can be obtained

However, if the background gets a little complicated or if the subject is not as clear, this happens:

Hence, I decided to use the COCODataset instead.

These are some of the masks of the “birds” category

With these masks, I can use the outline of my sketch to compare against the shape context histogram descriptors of these masks. How it works can be broken down into four steps:

Step 1: Finding the collection of points on shape contour Step 2: Computing the shape context Step 3: Computing the cost matrix Step 4: Finding the matching that minimizes total cost

BY THE WAY, shape context is another fun topic to discuss that’ll take a full article by itself, so I shall not be discussing it here — but do let me know if that’ll be of interest!

Here’s some results:

So it works OKAY. The problem with the Hungarian matching algorithm is that it is extremely time costly — its O(n³)! This is also the part where I’m thinking of implementing some ML/DL techniques to speed up the process.

Part 4 — Blending the closest match into the selected background image

The last part of this project (after using the target mask to “crop” the image out) is to use poisson blending to blend the image into our initially selected background image.

Poisson blending is one of the gradient domain image processing methods. Pixels in the resultant image are computed by solving a Poisson equation.

For each of the final pixels:

if mask(x,y) is 0:

final(x,y) = target(x,y)

else:

final(x,y) = 0

for each neighbor (nx,ny) in (x-1,y), (x+1,y), (x,y-1), (x,y+1):

final(x,y) += source(x,y) - source(nx,ny)

if mask(nx,ny) is 0:

final(x,y) += target(nx,ny)

else:

final(x,y) += final(nx,ny)

All masked pixels with a non-zero value affects each other’s final value. The matrix equation Ax=b is solved to compute everything simultaneously. The size of vectors x and b are both number of pixels in the target image — vector x is what we are solving for and contains all pixels in the final image while vector b is the guiding gradient plus the sum of all non-masked neighbour pixels in the target image. These non-masked neighbour pixels are the values of pixels at the mask boundary and the guiding gradient defines the second derivative of the final mask area. Matrix A is a sparse matrix and computes the gradient of the final image’s masked pixels.

The equation for x is solved to get the final image. If the guiding gradient is zero, we are just solving a Laplace equation and the values at the mask boundary are blended smoothly across. With the gradient of the source image as the guiding gradient, the mask area takes on the appearance of the source image but is smoothly blended at the boundary.

OpenCV’s seamless clone function is an implementation of the algorithm mentioned above.

Some results:

The birds were not in the original background image — they were “blended” in

Part 5 — Applying neural style transfer on the image with pre-trained models

I did a separate article on this here! Do check it out if interested. But in a nutshell, what it does is it takes the image above and apply some cool art style to it. There is of course a range to choose from and the range largely depends on which pre-trained neural models you use.

Here’s some examples:
['Ran', 'Reine']
['Python', 'Art', 'Software Development', 'Technology', 'Computer Science']